Sacrificing the Soul of Science for GenAI

Trading off curiosity and discovery for mere efficiency is a devil’s bargain

Anthropologist and Novelist Zora Neale Hurston, alongside stock photo of unidentified 19th century scientist.

Scientific exploration is one of the core motivations of our species, and without rigorous academic research, we wouldn’t know so much about the universe and our place within it. The author and anthropologist Zora Neale Hurston once described the scientific impulse this way:

                    "Research is formalized curiosity. It is poking and prying with a purpose. 
                     It is a seeking that he who wishes may know the cosmic secrets of the   
                      world and they that dwell therein." - Zora Neale Hurston

Knowledge isn’t just something we humans acquire; we actively pursue it. A mysterious driving force buried in our DNA makes us inherently inquisitive. On the other hand, LLMs may learn to make inferences from vast amounts of training data, but they have no desire to do anything at all since they are incapable of autonomous motivation.

Curiosity is a quintessentially human endeavor that exists for the express benefit of humankind. We should question, with extreme prejudice, the nascent drive to outsource human scientific inquiry to mere mimicking machines.

To this point, the non-profit consultancy Ithaca S+R recently published a thought-provoking report on the potential impact of generative AI on academic publishing — "A Third Transformation?" It references a small survey of scientific community stakeholders whose answers suggested a general consensus around the prediction that “generative AI will enable efficiency gains across the publication process”, where “writing, reviewing, editing, and discovery will all become easier and faster.”

It’s said that such efficiency gains could dramatically transform academic research and allow for the faster publishing of new papers. Scientists are already using ChatGPT to automate the drudgery of writing research grant proposals. Now, according to this new report, researchers are increasingly hopeful AI will speed up the peer review process by allowing them to pre-review their work to spot discrepancies or citation errors. Some suggest the possibility of replacing the need for human peer review altogether — since a machine is presumed to be far more accurate than a fallible human mind — and the absence of human bias would seem to make AI the perfect meritocratic tool.

But this is easier said than done. So far, generative AI in academia has failed to live up to expectations. In Ithaca S+R’s report, they call out that:

“Generative AI’s accuracy is often poor and at best too inconsistent to be trusted with even modest responsibilities for peer review.”

Moreover, as others have pointed out, the use of generative AI in research could potentially facilitate plagiarism and the falsification of experiment data. As Shijun He and his collaborators point out: “One worry is that it may facilitate plagiarism if researchers abuse the software to produce complete essays without putting in any effort.” As they go on to note, the use of LLM tools have “the potential to facilitate plagiarism if unscrupulous researchers misuse the software to effortlessly produce complete essays.”

Despite these very real concerns, a growing number of scientists are already outsourcing their research writing to LLMs regardless of the risks.

Could the shift to prioritizing the speed of publishing rather than the methodical expansion of human understanding over time lead to bots writing papers that are optimized for reading by other bots rather than humans? If this happens, it would lead to a race to the bottom where human beings are no longer playing a critical role in scientific research:

“If AI becomes the primary reader of this content, it raises the question of whether the work of creating a human-readable scientific record will come to be seen as an unnecessary expense—or as a barrier to improved machine readability.”

The whole point of human-powered research writing and peer review is to uphold the validity and integrity of scientific exploration by ensuring research is reproducible, quantifiable, and objective. LLMs “hallucinate” in unpredictable and hard-to-mitigate ways (“hallucination” is a masking term that marketing departments use to disguise known falsehoods - on a moral level, as Harry Frankfurt tells us, it’s just bullshit).

After all, LLMs have no real-world context or comprehension of any of the scientific principles being evaluated, it’s just a calculator for words. LLMs are full of inherent biases that can’t easily be checked by a human peer or non-human reviewer because its inner workings are a black box: trillions of parameters interpreted by a complex neural network that grows exponentially more complex the larger the model becomes, to the point that even the data scientists who built it have to admit that its internal logic becomes utterly esoteric over time.

Most dangerously, LLMs are designed to exploit certain human cognitive tendencies that make them appear convincing to us even when they’re spouting verifiable bullshit. Even the latest version of ChatGPT wildly overstates what it knows. Leaving LLMs to perform critical research unsupervised could lead us down rabbit holes of ignorance we can never emerge from, with unverifiable counterfeit science that will confuse and confound future generations and stall scientific progress in the aggregate.

As Hurston said, academic research is about “poking and prying with a purpose.” AI serves one master: the profit margins of big tech companies. It can't replicate a human scientist's intuition — that mysterious ability to sense which paths might lead to discovery or which puzzles hold the deepest secrets. In our quest to understand the universe, why would we trade our neural networks for silicon ones?

Work has been certified as genuine Human Intelligence.™ Check the certificate here.