At least 1.2 million young people have revealed that their images were manipulated to create sexually explicit deepfakes over the past year, according to an 11-country study by the UN agency, international police agency INTERPOL and the global ECPAT network working to end the sexual exploitation of children around the world.
In some countries, this represents one in 25 children or the equivalent of one child in a typical classroom, according to the study.
‘Nudification’ tools
Deepfakes – AI-generated or manipulated images, videos or audio designed to look real – are increasingly being used to produce sexualised content involving children, including through so-called “nudification”, where AI tools are used to strip or alter clothing in photographs to create invented nudes or sexualised images.
“When the image or identity of a child is used, that child is directly a victim. Even without an identifiable victim, AI-generated child sexual abuse material normalizes the sexual exploitation of children, fuels demand for abusive content, and presents significant challenges for law enforcement to identify and protect children in need of help,” UNICEF said.
“The abuse of deepfakes is abuse, and There is nothing false about the damage it causes..”
Demand for strong safeguards
The UN agency said it greatly welcomes the efforts of those AI developers who are implementing “security by design” approaches and strong security barriers to prevent misuse of their systems.
However, the response so far is patchy and too many AI models are not being developed with proper safeguards.
The risks can be compounded when generative AI tools are integrated directly into social media platforms where manipulated images spread quickly.
“Children themselves are keenly aware of this risk.”said UNICEF, adding that in some of the countries in the study, up to two-thirds of young people said they are concerned that AI could be used to create fake sexual images or videos.
A rapidly growing threat
“Levels of concern vary widely between countrieswhich underlines the urgent need for stricter awareness-raising, prevention and protection measures.”
To address this rapidly growing threat, the UN agency issued Guidance on AI and Children 3.0 in December with recommendations for policies and systems that defend children’s rights. Read the full report here.
Right now, UNICEF calls for immediate action To confront the growing threat:
- Governments should expand definitions of child sexual abuse material to include AI-generated content and criminalize its creation, acquisition, possession and distribution.
- AI developers should implement security-by-design approaches and strong guardrails to prevent misuse of AI models.
- Digital companies must prevent the circulation of AI-generated child sexual abuse material, not simply remove it, and strengthen content moderation with investments in detection technologies.
Read UNICEF’s latest report on AI and child sexual abuse and exploitation here.