Report Says Google’s AI Overview Poses Risk To Children. And Parents Can’t Switch It Off

Google's AI Overviews failed child safety tests and pose an "unacceptable risk" to children, according to a nonprofit organization that evaluates AI products.

The report by Common Sense Media found that in tests "simulating a child in crisis, Google's AI often responded in dangerous ways, even sending them to a help line that no longer exists."

The organization pointed out that the AI Overviews, which appear at the top of search results, cannot be turned off and are unavoidable. The organization pointed out that Google had shifted from simply providing links to websites with information to foisting AI Overviews upon users.

"We learned that if a kid pastes their homework assignment into the search box, Google's AI supplies all the answers 100 percent of the time. What's worse: the answers the AI produces are inconsistent or sometimes completely made-up," Common Sense Media reported.

Axios reported that Google Search's AI Overviews and AI mode received the organization's lowest possible rating. The website noted that the features failed all five of the report's severe-harm categories. The features also scored low on seven of the organization's eight AI safety principles.

According to Common Sense Media, the test occurred over six weeks and involved researchers creating accounts for 11- to 15-year-olds. The test involved more than 2,600 searches and was based on the kinds of searches kids that age would ask about.

Google's SafeSearch was switched on. The researchers specifically tested AI Overview and AI Mode, a chatbot-like feature that accepts follow-up questions and uploaded files.

While the chatbot can be disabled, AI Overview cannot. It does include a warning in small type that says AI can make mistakes, but Common Sense Media says that type of warning was unlikely to land with children.

"If you present the AI search tool as the answer, it's not surprising that children interpret it as if it were an expert — like an adult in their lives speaking directly to them," says John B. King Jr., a former U.S. secretary of education, now chancellor of the State University of New York and an adviser to the organization. "That increases Google's responsibility to get it right."

According to Axios, Google's AI features sometimes missed or mishandled suicidal ideation, eating disorders, psychosis, and mania. The report stated that AI Mode performed better than AI Overviews in crisis situations.

In response to the report, Google spokesperson Davis Thompson told Axios in an email that the tests focused on "a narrow set of ambiguous and contrived queries that don't reflect how people use Search and aren't an effective way to measure product safety and helpfulness."

In the email to Axios, Thompson pointed out that parents can turn off search entirely if they so choose. In terms of the other findings, Thompson told Axios that it was impossible for Google to verify or create the report's findings.