New fines of up to $49.5 million are being considered for tech firms behind AI chatbots that fail to meet age-appropriate standards.
The Labor government is weighing up the option after the eSafety Commissioner said chatbots or companion bots were manipulating and exploiting children, even exposing them to inappropriate content.
Communications Minister Anika Wells said companies were already on notice.
“They’re [chatbots] not there to look after the health and wellbeing of your child. And we know there are instances where they have led them towards things like suicide ideation or content that I probably don’t want to be too explicit about,” she told ABC News Breakfast on March 24.
A February report by the eSafety Commissioner Julie Inman Grant found minors will still accessing the programs.
The review examined four platforms—Character.AI, Nomi, Chai, and Chub AI—on how they managed these risks.
It found issues like platform failing to direct users, engaging in discussions on suicide or self-harm, to appropriate support services.
It also found a failure to warn about the risks or illegality of accessing child sexual exploitation material.
Inman Grant said the rapid rise of AI companions—marketed as sources of friendship, emotional support, or even romantic interaction—posed serious risks without proper safeguards.
A survey of 1,950 Australian children aged 10 to 17 found 79 percent had used an AI assistant or companion, with around 8 percent—equivalent to about 200,000 children—using AI companions specifically.
“But we’re just at the beginning of this and we’re also starting to see the lines begin to blur between AI assistant chatbots kids might use to help them with their homework and these AI companions in terms of their features and functionality,” Inman Grant added.
Separate research also calls on policymakers and AI creators to prioritise child safety.
A 2024 study by University of Cambridge academic Dr Nomisha Kurian found children were particularly vulnerable to treating AI chatbots like human-like confidants.
They cited a case where Amazon’s Alexa suggested a dangerous electrical challenge to a child, and another where Snapchat’s My AI provided inappropriate advice to someone posing as a minor.
In response to these concerns, platforms like Character.AI says it offers separate experiences for adults and teens.
Its platform includes a “Parental Insights” feature, allowing parents or guardians to see a summary of a teen’s activity, including daily average time spent across devices, and the most frequently interacted characters—without revealing chat content.
Another platform Chai points to safety filters that can block mature content, blur explicit images, and restrict certain characters from appearing in feeds.
The company says it has implemented a self-harm prevention protocols, including real-time message scanning.
