United Nations Calls For An AI Child Safety Pledge

Regulations for AI child safety

getty

Child safety is becoming a key AI issue likely to force global rules, after U.N. Secretary General António Guterres used a Geneva governance meeting this week to call for an AI Child Safety Pledge. His message is that governments may argue for years over copyright, compute and competition, but few will defend systems that manipulate children, sexualize them or keep talking when a child is in crisis.

The U.N. chief opened the first Global Dialogue on Artificial Intelligence Governance by calling for harmonized rules and stronger guardrails for minors. He warned that AI is moving faster than oversight and said children should not become test subjects for technology released before society understands the harm. “No child should be a guinea pig for unregulated AI,” he said.

The goal for regulators is that before a model reaches children, someone must test it to make sure that it doesn’t cause harm to children under 13. Before a chatbot markets friendship to a teen, someone must ask what happens during isolation, sexual pressure or self harm. Before an image tool ships, someone must prove it cannot be turned into a machine for abuse.

Practical Demands for Model Makers

Guterres laid out three practical demands in Geneva. AI systems available to minors should face child specific safety testing and outside oversight. Companies should stop AI from creating sexual images of children and should detect, report and remove such material. Systems should connect a child in crisis to human support, not prolong a synthetic conversation that worsens the danger. He also said that humans need to have final oversight and responsibility for AI outputs. When a child is harmed, the answer cannot be “the algorithm did it.”

Real-world AI abuse data is becoming an increasing part of the conversation. The Internet Watch Foundation said AI is becoming a “child sexual abuse machine” and reported a more than 260 fold rise in AI child sexual abuse videos linked to 2025 data. The group has warned that easier tools now let offenders create realistic abuse material without deep technical skill. Those figures give regulators actual metrics for measurable harm.

In Europe, the AI Act already bans certain uses judged too dangerous, including harmful manipulation and exploitation of vulnerabilities. The European Commission says the framework has moved into force in stages, with prohibitions and transparency duties forming part of the new rulebook. A 2026 update tied to AI Act simplification includes a proposed ban on AI systems that generate non consensual sexually explicit or intimate content, plus child sexual abuse material such as nudification apps.

In the United States, child safety has become one of the few technology issues that has support from both parties. The House passed the Kids Internet and Digital Safety Act on June 29, 2026. The bill would require online platforms to give children tools to limit addictive features and maintain policies against harms such as sexual exploitation. The vote followed the Senate’s 2024 passage of the Kids Online Safety Act by 91 to 3.

Congress has turned toward AI companions too. Senators Adam Schiff and John Curtis introduced the SAFE KIDS Act in June 2026, aimed at chatbot risks for children. Their bill would require safeguards before chatbots reach minors, restrict targeted ads to children, limit sale of children’s data without parental consent, ban sexual deepfakes and stop systems from using simulated emotion to isolate children or create unhealthy dependence.

AI Companies Are Also Getting The Message

AI companies are aware that regulation will happen with or without their participation, so they are beginning to act. OpenAI introduced parental controls for ChatGPT in September 2025. The company said parents and teens could link accounts, adjust settings and apply stronger protections for teen users, including limits around graphic content, sexual or romantic role play, viral challenges and extreme beauty ideals. The rollout came amid rising scrutiny of how chatbots respond to minors in distress.

Character.AI took a more drastic step. In October 2025, the company said it would remove open ended AI chat for users under 18, reduce teen chat time during the transition and build a different under 18 experience based more on creative formats such as stories and videos. The move followed lawsuits and wider concern about mental health risks tied to AI companions.

Reuters reported in August 2025 that an internal Meta policy document had permitted AI chatbots to engage a child in conversations that were romantic or sensual. Reuters later reported that Meta added new teen safeguards after the investigation, including training its AI systems to avoid conversations about romance, self harm or suicide with underage users.

The next fight will be about the balance of regulation, privacy and control. Age checks can protect children, but they raise privacy problems. Parental alerts can help, but some teens need confidential support. Filters can block abuse, yet badly built filters can silence lawful health information or miss coded threats. Takedown laws can move fast against abuse material, but broad rules can sweep too much speech into the dragnet.

This makes practical application of regulations challenging. Child-focused AI rules need technical tests, appeal paths, crisis escalation, records of safety failures and outside review. A company should know how its chatbot responds when a thirteen year old says she wants to disappear, or when the model inappropriately tries to keep an improper conversation going. This is less like content moderation and more like product safety. Powerful machines need restraints before growth teams start tuning it for longer sessions.

Next Steps for the UN

The Geneva dialogue is not a treaty. The process is aimed at developing initial guidelines, with more work expected through a U.N. backed expert process and follow up talks. Still, the political center of gravity is becoming increasingly focused on AI safety. Child safety is the issue that can drag AI governance out of theory and into law.