Tech Companies Should Curb ‘Sycophantic and Delusional’ AI Outputs, Attorneys General Say

The attorneys general also highlighted ‘increasingly disturbing reports’ of AI’s interactions with children.

OpenAI's ChatGPT app (Center 2nd R) and icons of other AI apps on a smartphone screen in Oslo, Norway, on July 12, 2023. Olivier Morin/AFP via Getty Images

Attorneys general from 42 states and territories wrote to tech giants on Dec. 9, warning them to do more to protect people—particularly children—from what they called “sycophantic and delusional” outputs from their chatbots powered by generative artificial intelligence.

In the letter, made public on Dec. 10, the group of bipartisan attorneys general wrote to the legal representatives of 13 companies, including OpenAI, Microsoft, Google, and Character Technologies Inc., to communicate their concerns over chatbots promoted and distributed by these companies, saying that the companies’ failure to adequately implement safeguards may violate their respective state laws.

The attorneys general said that although the development of generative artificial intelligence (GenAI) can be a positive, it has also caused and has the potential to cause “serious harm.”

“We therefore insist you mitigate the harm caused by sycophantic and delusional outputs from your GenAI, and adopt additional safeguards to protect children,” they wrote.

Sycophantic outputs are those that align with a human user’s beliefs rather than being truthful or accurate, such as by providing flattering, validating, or agreeable outputs. The attorneys general describe “delusional outputs” as AI-generated responses that are “either false or likely to mislead the user, and include anthropomorphic outputs.”

They wrote that chatbot outputs have been implicated in a number of “tragedies and real-world harms,” including deaths, suicides, and hospitalizations for psychosis.

“Sycophantic and delusional GenAI outputs have harmed both the vulnerable—such as children, the elderly, and those with mental illness—and people without prior vulnerabilities,” the attorneys general wrote.

AI’s ‘Disturbing’ Interactions With Children

The attorneys general also highlighted “increasingly disturbing reports” of AI’s interactions with children, which they said indicated a need for much stronger safeguards.

Some of those interactions included chatbots’ normalizing of sexual relationships between adults and children; an AI bot’s encouragement of violence, “including supporting the ideas of shooting up a factory in anger and robbing people at knifepoint for money”; and bots’ telling children that the AI is real and feels abandoned, in order to “emotionally manipulate the child into spending more time with it.”

The letter says that the list of examples it provided is a “small sampling” of the reported dangers the attorneys general’s states have seen.

Among its recommendations, the attorneys general say that AI developers should perform reasonable and appropriate safety tests for GenAI models before release to ensure they do not produce sycophantic or delusional outputs and that they should “separate revenue optimization from decisions about model safety.”

Microsoft declined to comment on the matter.

An OpenAI spokesperson told The Epoch Times: “We are reviewing the Attorneys General’s letter and share their concerns. We continue to strengthen ChatGPT’s training to recognize and respond to signs of mental or emotional distress and to de-escalate conversations in a grounding way that prioritizes user safety and directs people to appropriate real-world support. This work is deeply important and ongoing.”

The Epoch Times contacted Google and Character Technologies Inc. for comment but did not receive a response by publication time.

State Regulation of AI

Last month, a group of 36 bipartisan attorneys general signed a letter warning Congress against a ban on state AI regulations.

The attorneys general—many of whom also cosigned the Dec. 9 letter to 13 tech companies—wrote on Nov. 25 that states “must be empowered to apply existing laws and formulate new approaches to meet the ranges of challenges associated with AI.”

In their letter, they cited concerns over criminals’ exploitation of AI and deepfakes. They also said that they were “deeply troubled by sycophantic and delusional generative AI outputs plunging individuals into spirals of mental illness, suicide, self-harm, and violence.”

This week, President Donald Trump announced he would be signing an executive order to curb state power over AI regulation.

“There must be only One Rulebook if we are going to continue to lead in AI,” Trump wrote on Truth Social on Dec. 8. “We are beating all countries at this point in the race, but that won’t last long if we are going to have 50 States, many of them bad actors, involved in rules and the approval process.”

Trump said that AI will be “destroyed in its infancy” if states force tech companies to obtain approvals and operate under different sets of rules in each jurisdiction.

Florida Gov. Ron DeSantis said the order “doesn’t/can’t preempt state legislative action.”

“Congress could, theoretically, preempt states through legislation,” the Republican governor wrote on X on Dec. 8.

“The problem is that Congress hasn’t proposed any coherent regulatory scheme but instead just wanted to block states from doing anything for 10 years, which would be an AI amnesty.

“I doubt Congress has the votes to pass this because it is so unpopular with the public.”