Scientific Journals Are Flooded with Texts Written by AI — Is the Solution to Let AI Read Them Too?

Could the scientific conclusions you have just read in a journal paper be completely bogus? A recent journalistic report published in Science this week has revealed that for quite a lot of the articles appearing in some of the medical journals, the answer is an unequivocal yes. The report, by Frederik Joelving from an initiative called Retraction Watch that tracks and reports misconduct in scientific publications including data manipulation, describes how numerous commentary articles that made their way into legitimate medical journals, were written not by an investigator, but by AI. An email by an unnamed writer has disclosed to the authors of the report the fact that three writers have published in one single journal within eight weeks 69 comments, that appear to be inauthentic confabulations created by AI. Retraction Watch is reporting to have found that out of all submissions to one specific journal, half are now of commentaries, as opposed to an empirical study or a review paper. In another journal these texts comprised 70% of submissions. The submission of these texts became so widespread that one journal, the Neurosurgical Review, announced to the research community that it pauses completely any acceptance of letters to the editor and commentary.

It seems clear now that AI can’t write scientific papers, at least for now. But, maybe it could at least read them? And this way help the editors of the world to identify authentic, original writing and thinking?

Of course, the first question to be asked, is who reads and vets these texts, and how could AI-generated manuscripts pass the editors in the first place. The answer is not that AI became so good that it now tricks an experienced scientist. The artificial texts could be identified, as attested by the email that reported them. As a scientific writer, I can add that experienced readers of scientific texts notice quickly the AI parts that were pasted from the AI. Rather, since commentaries are not peer-reviewed, the evidence in the new report discloses a reality where some editors apparently don’t read in full some of the texts that they publish (and where not many scientists, if any, bother to read commentaries). Before we incriminate the editors, we should consider the number of commentaries such editors are flooded with. Some may decide to read until reaching an interesting piece and ignore the rest — an acceptable approach — while others might feel obligated to at least scroll through all the submissions out of respect for the writers and their efforts, and it is probably those who are more at risk of falling in the AI trap.

Well, it seems clear now that AI can’t write scientific papers, at least for now. But, maybe it could at least read them? And this way help the editors of the world to identify authentic, original writing and thinking? News published in Nature in the same week that the Science and Retraction Watch report appeared, disclose aspirations that can definitely lead to such attempts. As reported in Nature, a Swiss company called DeSci labs (the name merging the words decentralized and science) has created a system for assigning a novelty score to manuscripts submitted to journals, which it offers to publishers. According to the developers cited in Nature, “this score could assist journal editors in deciding which studies to publish”, “using an algorithm that compares the combinations of keywords and cited journals in a scientific manuscript.” The said algorithm, Nature discloses, first appeared in a paper archived in arXiv in September 2023, and since then 50 million manuscripts were rated. It is however not obvious at all that editors could in fact apply the new method to identify novelty and originality and thus disqualify AI-written text, nor that the problem can at all be dealt with using technological solutions.

For science, the problem may be deeper than a technical barrier that enough computing would eventually resolve.

A dramatic demonstration of the limitations of AI and algorithms in interpreting scientific texts was recorded in 2023, when a colossal failure of one of the most elaborate and expensive attempts to date to allow AI to read and analyze natural texts, specifically scientific texts, and rate their innovation, took Europe by surprise. In April 2021, the largest funder of research and development in Europe, the EU’s Horizon Europe program, has shifted to AI the entire review process of its popular grant for new technologies, called Accelerator. The EU statements on the astronomical capacities of AI that would allow weighing an idea against the titanic size of the existing knowledge, to quantify the innovation of any proposed idea, were dizzying. Any proposed research was to be profiled “against 180 million scientific publications” and “120 million patent documents”, the science and technology community was told on EU websites. But things weren’t going so well. After only two years of using the AI system for rating research proposals for their innovation, on June 2 2023, in a week’s notice ahead of a deadline of the prestigious grant, the EU announced to the thousands of scientists and developers who were chasing the clock to prepare their research proposal in an AI-compatible format, that it is taking down the entire AI system immediately, and reverting back to a regular fluent narrative read by human peers.

The failure was not really that much of a surprise. In fact, the willingness of an organization as large as the EU to ignore the obvious problems was much more bewildering for some of the developers and scientists who had to describe their novel scientific and technological ideas to the AI system. For one, based on the information a scientist would provide, the AI generated a lengthy synopsis of the proposed research, and this AI-written text was, to put it mildly, complete nonsense. This nonsensical output was provided to the people who had to decide whom to fund.

Just as telling was the process the applicant had to go through. The alleged ability to read a scientific text, was based on requiring the proposers to break down their fluent narrative into sentences, that were to be pasted into hundreds of distinct text fields. As if that wasn’t enough to raise the suspicion that the AI is having difficulties, the sequence of those text fields, would confuse even the savviest scientist and technologist. They seemed to invite circular repetitions — a subject would be described down to its finest details in dozens of fields, but then it had to be described again, with some slight modification. Anyone with some programming experience could see right through this confusing “writing”, and understand very clearly why this ridiculous circularity is happening. The text fields arrangement simply had the familiar structure of loops nested in “if” functions, that a programming code would have. The AI was apparently so great at reading scientific texts, that the way to make it operate was to force the scientists to write as if they were a computer, that is, to break their narrative into hundreds of sentences and rearrange them in the structure of logical loops nested under “if” functions, that could be fed into a machine… (now that’s machine-reading of natural language! said the EU administrator at the end of the sermon of the AI people about the new God, when the PowerPoint oration was over). Alas, even this non-human (and some would say, inhumane) writing procedure couldn’t help the very expensive machine make sense of an innovative scientific idea.

Perhaps an inability of AI to read natural scientific language and interpret it sensibly, is what could save the editors and the readers?

An interesting part of the façade, is that to this day, to the best of my knowledge the EU never admitted the failure of its attempt to let code read scientific texts and rate their innovation, nor did it explain how its administrators were lured in the first place into ignoring the obvious inability of the machine to read sensibly. The cultural urge to believe that machines think or even understand, is extremely strong. Perhaps it is because we gave up hope on finding intelligence in outer space, so we try to hope that we might be able to create it here by ourselves. Maybe it’s basically the eternal fundamental loneliness of humanity, of feeling the only part of a cold, indifferent universe that knows of itself. We want so much something to know us, we tried gods and aliens and when both ghosted us, we looked at our machines, with a new, sad hope in our eyes.

For science, the problem may be deeper than a technical barrier that enough computing would eventually resolve. AI is trained on existing examples, and absorbs those examples, much like a pinpression toy absorbs the shape of your hand or your nose when you place them on the pins, only with digital pins. Layers upon layers of those digital pins, can take the shape of thousands and thousands of examples, which allows to find connections between them, or average the examples and imitate them, for “AI writing”, or to find repetitive patterns in the pressed examples, as attempted in medical sciences today to predict disease progression or to diagnose patients more accurately. In a sense, AI is fundamentally a sponge, needless to say a sophisticated sponge, that absorbs examples and takes their shape, in its micro-matrix. But the very essence of the concept of innovation, is having no previous examples.

If only editors could use AI to sieve the texts and leave only the real ones for reading. The problem is, that despite the courageous attempt reported in Nature, as the EU’s colossal failure has shown, AI is very bad at reading a real scientific text and an algorithm for quantifying the human concept of innovation might be an Icarian demonstration of the line separating human consciousness from machines. But, perhaps an inability of AI to read natural scientific language and interpret it sensibly, is what could save the editors and the readers? It would be interesting to find out whether AI makes more sense of an AI-generated text, compared to natural text. Would AI rate an AI-generated text as more logical, considering that a machine created that text? In that case we only need to reverse our scale — those texts preferred by AI for consideration, should be flagged. And even if a machine cannot show a systematic-enough bias in favor of contributions made by its fellow machines; nonsense might be much easier to detect than reason. At least humans are much better at spotting nonsense than in acknowledging reason. How many in a random audience would laugh from a Monty Python sketch, and how many in that audience would drop their jaws from a new mathematical proof? Few can spot reason, yet most can appreciate nonsense. Instead of rating innovation, we might want to consider rating cluelessness. Perhaps the future of science and beyond is not machine reading, but nonsensing-sensing?