Discord bans AI-generated child sex abuse material and teen dating

Discord bans AI-generated child sex abuse material and teen dating

SAN FRANCISCO — Discord’s head of trust and safety said Tuesday that the popular chat platform was changing and clarifying its child safety policies including those around around teen dating and AI-generated child sexual abuse material, an announcement that comes after an NBC News investigation last month into child safety on the platform.

John Redgrave, Discord’s vice president of trust and safety, said that Discord was expanding its policies to address generative artificial intelligence that can create fake content and the sexualization of children, specifically banning AI depictions of child sexual abuse and even the sexualization of children in text chats. The Washington Post reported in June that AI-generated child sex images have proliferated across the internet in recent months.

Discord has been a hub for communities devoted to the creation of generative AI images and has hosted several integrations that allow users to generate them. Sexually themed images are frequently created on those servers.

The company said in a blog post announcing the changes that the updated child sexual abuse material policy would include “any text or media content that sexualizes children, including drawn, photorealistic, and AI-generated photorealistic child sexual abuse material. The goal of this update is to ensure that the sexualization of children in any context is not normalized by bad actors.”

Redgrave also said that the company was instituting policy changes and clarifications to explicitly ban teen dating, which experts previously told NBC News posed a significant opportunity for adults looking to exploit or groom children.

Discord wrote in its blog post: “In this context, we also believe that dating online can result in self-endangerment. Under this policy, teen dating servers are prohibited on the platform and we will take action against users who are engaging in this behavior.”

Redgrave said in a presentation at TrustCon, a conference for trust and safety professionals held in San Francisco, that the company saw such online relationships as a major risk for young people.

“We no longer are going to allow teen dating on our platform because we recognize that it is a substantial harm vector for predators to go after teens,” he said.

Discord’s guidelines had already said the company would “remove spaces that encourage or facilitate dating between teens.” In June, an NBC News investigation found hundreds of Discord servers that appeared to promote child abuse material, and some servers that advertised themselves as teen or child-dating servers that solicited nude images from minors. Additionally, NBC News identified 35 cases over the past six years in which adults were prosecuted on charges of kidnapping, grooming or sexual assault that allegedly involved communications on Discord.

NBC News identified an additional 165 cases, including four alleged crime rings in which adults were prosecuted for transmitting or receiving child sexual abuse material via Discord, or for allegedly using the platform to extort children into sending sexually graphic images of themselves, also known as sextortion. 

Redgrave noted that the company was also updating its policies to prohibit instances of older teens grooming younger teens. In the blog post, the company said, “Older teens engaging in the grooming of a younger teen will be reviewed and actioned under our Inappropriate Sexual Conduct with Children and Grooming Policy.”

As part of its policy updates, Discord announced the launch of more tools for parental control. In a new Family Center tool, parents and kids can opt in to have parents receive updates about their kids’ activities on the platform.