Deepfake abuse is rapidly emerging as a critical challenge in online safety, with minors increasingly becoming the victims.

A recent report from South Korea found that nearly 60% of Deepfake-related sexual crimes involved minors, with underage victims rising from 53 in 2021 to 181 in 2023—an alarming increase. This surge is also linked to the growing number of minor perpetrators, driven by the increasing accessibility of Deepfake creation tools .

Equally concerning are the cases where students exploit AI technology to bully their female classmates. In one tragic incident, 14-year-old  Mia Janin from the UK was targeted when her peers superimposed her face onto explicit images of another woman. The manipulated content circulated on socials, leading to severe bullying at school. Unable to cope with the relentless harassment, the victim took her own life.

Between 2019 and 2024, the number of Deepfake videos online surged by 550%, with non-consensual explicit content accounting for a significant portion of this rise. The increasing prevalence of Deepfake abuse highlights the critical need for enhanced disinformation security protocols across social media platforms to better protect minors.

Challenges in Identifying Deepfakes

While platforms are making strides to combat digital manipulation, they continue to navigate significant challenges in this rapidly evolving landscape.

The average response time to Deepfake-related incidents remains a critical concern, with many cases going unaddressed for over 72 hours—far too slow to prevent potential harm from viral content. There is a growing need for quicker and more accurate detection, alongside comprehensive strategies, as more cases of Deepfake abuse continue to emerge.

The Impact on Vulnerable Users: From Private Chats to Social Media and Gaming Communities

In 2023, the Wall Street Journal reported a troubling incident at a New Jersey high school, where students used an AI-powered “nudify” app to generate explicit images of over 30 girls. These manipulated images were then shared in group chats and across social media platforms, amplifying the emotional and psychological toll on the victims.

The rise of Deepfake technology has also permeated gaming communities, where voice and video manipulation is increasingly used to target individuals. While some platforms are developing stronger moderation tools, experts agree that more must be done to address the unique challenges posed by AI-generated content in these environments.

Collaborative Strategies for Protecting Minors from Deepfake Abuse

As GenAI technology becomes more accessible, its potential impact on vulnerable populations, particularly minors, raises significant concerns. The implications of AI-generated threats can be severe, ranging from emotional distress to significant reputational damage. 

We encourage trust and safety teams to explore innovative strategies that enhance the protection of their younger users:

1. Efficient Deepfake Detection

Implementing robust Deepfake detection systems, cutting-edge solutions like DeepID, is essential for ensuring comprehensive protection against AI-generated manipulations. These technologies can quickly flag Deepfakes as they emerge, significantly reducing the risk of viral spread. By leveraging state-of-the-art machine learning algorithms, these systems continuously enhance their accuracy in identifying manipulated content, providing a critical line of defense against potential threats.

2. Quick Actioning for Takedown Requests

Speed is critical when dealing with harmful Deepfake content. Once a Deepfake is detected, platforms should immediately initiate rapid takedown processes to minimize exposure. By ensuring a swift response to takedown requests, harmful content can be removed efficiently, reducing the potential risk to users—especially young audiences. This proactive approach helps protect individuals while minimizing the spread of damaging misinformation.

3. Dedicated Moderation Teams

Establishing Trust & Safety teams that specialize in AI-driven content moderation, particularly focused on identifying and managing youth-targeted Deepfakes, can greatly enhance response times and the level of expertise. These teams should be trained to become more familiar with interpreting the results of Deepfake Detection tools. The combination of AI and human judgment helps verify content with greater accuracy, ensuring that cases are handled with the urgency and sensitivity they require.

Protecting the next generation

The threat posed by Deepfakes continues to escalate each year, and without decisive action from platform trust and safety teams, more children will inevitably fall victim to this insidious form of digital manipulation. 

By enhancing Deepfake Detection capabilities and prioritizing Disinformation Security protocols, platforms can take meaningful steps toward safeguarding their youngest and most vulnerable users.  Together, we can work towards creating a safer online environment for everyone.