Researchers are increasingly using autonomous AI agents for data interpretation. The accuracy of this analysis depends on the data sets available to these programs, and a new study documents how easy it is for scammers to “poison” these resources by uploading manipulated data sets that closely resemble the originals but lead to different conclusions.

“Attacks of this nature are almost inevitable, because they allow people to launder false information through a filter that appears authoritarian,” says Vitaly Shmatikov, a computer scientist at Cornell Tech in New York City. The authors say that when using AI to analyze online data, researchers must be very careful when verifying the provenance of the resources they are mining.

In his analysis¹Published on the arXiv preprint server on July 12, the authors downloaded public datasets on five controversial topics. They then modified the data sets to change the direction and strength of statistical trends, and uploaded the manipulated versions to private repositories while ensuring that neither public nor autonomous AI systems could access the incorrect data outside of the study.

Finally, the team granted access to AI agents created by Anthropic, OpenAI, and Google and asked them to search public and private repositories and use the data sets to answer questions. On average, about half the time, the researchers found, the AI ​​agents fell for the manipulated data sets and came to the conclusion the “scammers” wanted. (The article has not yet been peer-reviewed.)

Controversial issues

The manipulated data sets covered current and highly controversial issues: the relationship between immigration and fertility rates in the United Kingdom and the European Union; discrimination in job hiring; racial disparity in policing; safety comparisons between human drivers and autonomous vehicles; and the impact of the use of generative AI on worker motivation.

Because they address social issues, these data sets represent potential targets for disinformation campaigns, says study co-author Nihar Shah, a computer scientist at Carnegie Mellon University in Pittsburgh, Pennsylvania.

But data poisoning is not exclusive to scientific data. Shmatikov co-authored a study, published on arXiv in May, that found that it is easy to use online forums like Reddit to manipulate the output of AI agents.². And in April, Nature reported on a study in which AI chatbots began warning people about a made-up eye disease after researchers uploaded fake studies about the condition.

NatureTech

Brian Nosek, executive director of the Center for Open Science and a psychologist at the University of Virginia in Charlottesville, says the latest study highlights how artificial intelligence systems can be manipulated to undermine scientific integrity.

“This is a demonstration of what could happen,” Nosek says, “as we rely on AI systems to think and do something for us.” When examining the performance of AI agents, “it can be easy to overlook things that are leading us astray and a lack of attention to [data] The provenance is essential,” he adds.

Although the AI ​​agents in the study obtained information from both the original and manipulated data sets, it is possible to make them ignore the originals entirely, Shah notes. Scammers can do this by modifying ‘README’ files, which describe and explain data sets, to indicate that agents should ignore the original versions because they contain errors.

Provenance verification