Mobicip interview series for Safer Internet Day 2026 · Part 5 of 8
Online Grooming in the AI Age
Watch the recording on YouTube
Summary
Anitha asks about the urgent threats from online grooming and gender-based violence, and how parents can protect their children.
I describe two ways AI is changing the threat. One is the creation of false narratives: fake websites, invented characters, phishing, and fluent text that removes the language mistakes that used to give scams away. The other is the use of AI to create sexualized images of a child, so that sextortion no longer requires grooming the child first. I also describe minors creating this kind of media about other minors, for example with apps that “nudify” classmates.
Key points
- Parents should use these technologies alongside their children, both to model behavior and to teach how the technology works.
- An open channel with parents protects children. A child who feels they can’t confide in the adults around them has nowhere to go when they’re threatened.
- Schools should teach basic online self-defense, with specific instructions. One warning sign is being asked to leave a public chat for a private one.
- Regulators should hold companies accountable for designs that lead children into these traps. One example is prohibiting chat in online games for children under a certain age.
- Companies need trust and safety teams that keep looking for harm and fix the features that cause it. I don’t believe any company wants children to be harmed, but financial pressures can lead them to ignore certain behavior until something tragic happens.
Speakers
- John Zoltner Guest
- Anitha Swaminathan Interviewer, Mobicip
Transcript
The wording is as spoken, with punctuation and obvious errors corrected. See “How this transcript was made” above.
[00:00:00]Anitha Swaminathan: Your work also zeroes in on online grooming and gender-based violence, which heavily affects young kids and teens today. So what are some of the urgent threats that you see, and how can parents protect their kids from online grooming?
[00:00:26]John Zoltner: Yeah, I mean, as far as AI goes, there are two, I would say. One is just misinformation and disinformation. But by misinformation, I would say anything where somebody is using AI in order to create a false narrative. And part of that, it could be in a phishing scheme where they put up a fake website that’s very easy to do using AI, or they create a false character that looks like a real person. They create false text. So they’re able to, for instance, if it’s a person whose English isn’t their first language and they’re in a non-English speaking country, one of the ways that it was most easy to tell if something was a scam or not is that they’d make mistakes in the English because they were supposed to be someone who was from the country of the child speaking English as a native language, but they were making mistakes in that language. Now it’s very easy to correct that language during a dialogue, during a live chat, so that you’re not making those mistakes. It makes it much more difficult to be able to use language in order to catch something that is an attempt to fool you for whatever reason. And that could be for a scam to steal money from you, or it could be for sextortion in order to use the technology, for instance, to create a deepfake of sexualized pictures or videos of a child and then threaten that child with exposing those to other people. And that was a huge problem, beginning especially, it really rose to prominence a couple of years ago. But now it’s, and they would do it by creating confidence, chatting with a child and gradually growing closer so that the child trusted the person who was usually the adult who was chatting with them and who would eventually ask them to do things like take your clothes off and take pictures and would gradually involve them in more incriminating behavior that would cause them to be embarrassed and then demand money in exchange for that. Now it’s possible to do that without even going through that process of grooming them and getting them to do things that later they would regret. So now we see people who are creating deepfakes of a child, showing them in sexualized positions or images, and then threatening to distribute those when the child didn’t have anything to do with those images. So that’s one, using AI to create media or to create narratives that trick a child or anyone into doing something that could be harmful to them in the future, or just to harm them in some way by distributing that kind of media. And the other thing we’re seeing is that minors are creating that kind of media about other minors. So, and it’s natural. I think if the technology exists, there’s no way that children around the world in middle school or in high school aren’t going to use that technology to, for example, nudify classmates and then distribute that among their friends because they feel it’s funny. And it’s a way, sometimes it’s a way of bullying. Sometimes it’s a way of, you know, they feel like it’s fun. But it’s always very damaging to the child who’s the subject of that media. But what practical steps do we take to avoid the situation? Part of it is training. And it’s true, I talked about how parents need to accompany children in using these technologies so that they can both model behavior and teach children about the technology so they understand better how they work. But what we found in grooming is that having an open channel, open dialogue with your child does a lot to protect them against the grooming, because when a child feels like they can’t confide in the adults around them, then they have nowhere to go when they’re threatened in the ways that we’re finding them threatened during sextortion schemes. And that’s where it gets really dangerous, when it happens very suddenly, very quickly that, you know, at one point they’re playing around in a game. Someone asks them in the chat in a game to chat on some other platform. So they go to that other platform where they’re alone one-on-one with the person who’s the groomer in this instance. And they very quickly get to the point where they’re exchanging sexualized imaging or threatening the child that they’ll do that, even if they’ve just created those images with AI. So the children have to know, both through their parents or caretakers, through educators, schools need to be teaching basic online self-defense. They need to teach children to understand, to look for the signs of this behavior, so that, you know, a big warning sign is when someone asks you to leave a public chat and go to a private chat. And I believe in things that are very specific instructions so that when the child sees that, they immediately remember, oh yeah, this could be a grooming situation or this person may not be who they claim to be. And then regulators need to be holding companies accountable for creating their technologies in a way that won’t lead children into these traps. And so an example might be prohibiting chat in online games for children under a certain age. And the companies themselves need to be constantly vigilant through their trust and safety teams and through mechanisms for reviewing complaints and reacting to them. The companies need to be constantly looking for the damage or violence that’s happening through their technologies and fixing the functionalities that are leading to that violence. And I think that, I don’t believe there’s any company out there that wants to see harm come to children, but there are pressing financial motivations that cause them to ignore certain behavior until something tragic happens. And so I think it has to be a combination of the companies, the regulators and the users or parents of the users.
[00:09:37]Unidentified speaker: Thank you.