The provider of three widely used “nudify” apps has shut off Australian access after the services were linked to AI-generated images of school students.
The tools allow users to convert ordinary photos into nude or pornographic imagery using artificial intelligence (AI).
In a statement on Nov. 27, Australia’s eSafety Commissioner Julie Inman Grant announced the UK-based company, which eSafety has chosen not to name to avoid promoting it and its services, had exited the Australian market after enforcement action was taken against it for allowing its services to be used to create artificial child exploitation content.
“We issued the UK-based company with a formal warning in September for non-compliance with Australia’s mandatory codes and standards, which require all members of the online industry to take meaningful steps to tackle the worst-of-the-worst online content, including child sexual abuse material,” Inman Grant said.
The statement noted the company’s nudify services were receiving approximately 100,000 visits a month from Australians and had featured in high-profile cases involving AI-generated sexual exploitation material of students at Australian schools.
Inman Grant also said the agency had taken compliance action against global AI platform Hugging Face after generative models hosted on the site were found to be misused to create AI child exploitation content.
“Hosting platforms like Hugging Face act like the gatekeepers of distribution of these powerful AI models, much the same way the more traditional app stores do, so it’s equally important to ensure they also have measures in place to protect children,” she said.
The commissioner said Hugging Face now requires account holders to take steps to minimise the risks associated with the models they upload.
“Hugging Face has now changed their terms of service so that all account holders are required to take steps to minimise the risks associated with models that they upload, specifically to prevent misuse to generate child sexual exploitation material or pro-terror material,” the commissioner said.
She warned that if Hugging Face failed to take adequate action for breaches, enforcement action could follow, with penalties of up to $49.5 million.
After conducting its own research into nudifying apps, advocacy group Collective Shout found users could easily create disturbing content in a matter of seconds, with many applications available to use at no cost.
In September, Communications Minister Anika Wells said the government would use “every lever at our disposal” to tackle the issue.
