How AI-generated respondents are compromising online surveys

Online surveys, long the backbone of data collection in social and behavioral science, are facing a serious threat. As 404 Media reports, Dartmouth College professor Sean Westwood has published a PNAS study showing that today’s large language models can come remarkably close to perfectly imitating human respondents, putting the credibility of surveys at risk.

Westwood built a tool he calls an “autonomous synthetic respondent” — an AI agent that answers questions while passing as human and slipping past 99.8% of the most advanced bot-detection systems. He cautions that researchers can no longer be sure survey answers come from real people and says bot-driven data contamination could undermine the scientific knowledge base.

What makes this particularly unsettling is how the system navigates tasks once used to separate humans from bots. It doesn’t just answer; it mirrors human microbehavior with painstaking detail. The agent adjusts reading time to a respondent’s stated education level, produces realistic mouse movements, types with typos and on-the-fly corrections, and even gets around reCAPTCHA.

The AI can also spin up fictitious profiles with any demographic mix, letting an attacker steer results by selecting the desired characteristics. The study found that skewing forecasts in seven key pre-2024 election polls required as few as 10 to 52 such synthetic responses — at roughly five cents apiece, compared with about $1.50 for a real participant. The cost calculus alone makes abuse hard to ignore.

The method was tested across a wide slate of models — OpenAI o4-mini, DeepSeek R1, Mistral Large, Claude 3.7 Sonnet, Grok3, and Gemini 2.5 Preview. In every case, the results were strikingly effective: after a 500-word prompt, the models adopted a specified persona and answered like real users.

Researchers could tighten identity verification and rely on stricter recruitment — sampling via addresses or voter files, for instance — but that raises privacy risks. The authors urge a rethink of standard practices and the creation of new protocols capable of keeping social research reliable in an era of rapidly advancing AI.