1,682 Participants Can't Tell AI From Humans—and Prefer AI Stories

The line between human and machine creativity is blurring faster than many thought. Researchers at Villanova University presented 1,682 participants with short stories—some written by people, some by ChatGPT—and asked them to judge authorship. Not only did they fail to spot the AI, but they also awarded higher marks to the AI-generated tales. The study, in Judgment and Decision Making, underscores how advanced language models have become at crafting compelling narrative fiction.

A new study from Villanova University, published in the journal Judgment and Decision Making, adds a provocative layer to the debate over AI-generated content: people not only struggle to distinguish machine-written stories from human-authored ones, but they also consistently prefer the AI versions. Across three experiments involving 1,682 participants aged 18 to 81, researchers examined how readers rate short stories when they are told—or misled—about authorship, and whether they can identify AI-generated text. The results are striking. When participants were falsely told that an AI-generated story was human-written, they rated it significantly higher in quality and absorption than actual human-written stories, and even when given no authorship cues, the AI-generated narratives still earned superior ratings. Moreover, participants were unable to reliably identify which stories were crafted by ChatGPT and which by human authors. This suggests that, for narrative short stories at least, the output of today’s large language models can match or surpass average human literary output in perceived quality.

“

Researchers at Villanova University presented 1,682 participants with short stories—some written by people, some by ChatGPT—and asked them to judge authorship.

The study’s methodology was robust. It used three human-written short stories sourced from reputable literary journals or collections, and used ChatGPT to generate three corresponding stories with similar thematic prompts. In the first experiment, 1,682 participants read a story assigned with a true or false authorship label (human or AI) and then rated its quality and how engaging they found it. The subsequent experiments tested participants’ ability to discriminate between a human and an AI story when side-by-side, and also surveyed their self-reported experience with AI platforms and with fiction reading. The finding that AI-generated content received higher scores for both “quality” (how well-written the story was) and “absorption” (how invested the reader felt) challenges long-standing assumptions that human creativity yields a unique, irreplaceable quality. It also raises urgent questions about trust, disclosure, and the economics of content creation.

From a market perspective, the implications are profound. If consumers cannot tell the difference—and indeed prefer AI-generated material—then industries reliant on creative writing, from marketing and journalism to publishing and entertainment, face a seismic shift. Cost-effective, scalable content generation by AI becomes not just a possibility but an attractive competitive advantage. However, the study also reveals a nuanced sensitivity: when participants were explicitly told a story was AI-written, they rated it lower than when they believed it was human. This suggests a disclosure penalty—a trust gap that could harm brands if transparency is mandated or discovered. For businesses, the strategic tension lies in capturing the quality gains of AI while navigating the potential backlash if AI’s role is exposed.