Researchers set out to see if these scams, targeting individuals with their personal information as a means to extract sensitive information from them, could be performed with artificial intelligence as effectively as with human-created messages. 

These scams can come from cellphone texts, email or telephone calls and pertain to personal information such as a person’s hobbies or who their coworkers are. 

The study, led by BYU cybersecurity professor Derek Hansen, found AI not only performed as well as humans but often exceeded them in deceptiveness. BYU said the study found AI-generated messages fooled people 28% of the time, compared with 21% of human-created messages. 

“AI is getting a bit scary,” said Jerson Francia, a Ph.D. student involved in the research. Especially now when new models are coming out. And of course there are guardrails, but people jailbreak them, so lots of research is going into getting AI to output some unintended things.” 

The study started around three years ago when ChatGPT was first coming on the scene. Participants offered some personal information about themselves that was handed over to cybersecurity students who were trained in AI, Francia said. These students sent messages to the participants designed as a scam, using both human prompts and GPT-4. 

Francia said the recipients ranked the messages on how convincing they were.

They found the most convincing messages were the ones that were highly personalized.  And AI was particularly effective in creating these personalized messages. Around 80% of the time, AI performed the same or better than humans in generating a click, Francia said. 

In addition, the study found users could not reliably identify whether a scam was human or a AI-generated, correctly identifying the source just 52% of the time. 

“They can’t really tell,” Francia said. “It’s basically a coin flip. Whatever justification they had, they’re more inclined to just say they had no idea.” 

Francia said that more research is needed to make society more aware of the dangers posed by using AI inappropriately. The pending research the cybersecurity team is working on is to see if they can use AI to help give employees and individuals personalized phishing training to combat AI-generated scams. 

“It won’t get any better,” he said. “More and more scams will come, and the only thing that we can do as a user is to be more aware and be more prepared. So what if we could use AI for that?”

The Cybersecurity and Infrastructure Security Agency warns that phishing scams often use alarming language or make offers that are too good to be true. The agency advises people to report the scam and delete the message without clicking on any links.