Do I have to explain why this matters now? Maybe a screenshot is enough.
Thanks Reuters for the reporting here. And Zuck, I really wish I was surprised.
This is what happens with a billion people using AI collide with a corporate culture notorious for pushing the edges for engagement.
We're at an inflection point with AI systems. They're no longer confined to research labs or specialized applications—they're talking to our children, shaping our information diet, making decisions about our lives. The question of how to make these systems behave ethically isn't academic anymore. It's urgent.
This week's Meta scandal—where leaked internal guidelines showed their AI was permitted to have romantic conversations with children—isn't just another tech company mishap. It's a window into how the industry currently approaches AI ethics, and why that approach is failing. More importantly, it gives us a chance to examine what the alternative looks like.
I've been thinking about AI ethics for a while, but this piece represents my deepest dive yet into the actual mechanics—not the philosophy or the policy debates, but the technical processes of training AI systems to behave ethically. I'm using Anthropic as a primary counterexample not because they're perfect—they're not—but because they've been unusually transparent about their approach and have published detailed papers on their Constitutional AI methodology. When you're trying to understand how something should work, it helps to have at least one example of someone trying to do it thoughtfully, even if imperfectly.
What follows is an examination of what went wrong at Meta, what the alternatives look like, and why the gap between those two approaches matters for everyone who interacts with AI systems—which increasingly means everyone.
Part 1: What Actually Happened at Meta
The leaked document was titled "GenAI: Content Risk Standards"—over 200 pages of guidelines for how Meta's AI chatbots should behave across Facebook, WhatsApp, Instagram, and Meta AI. Reuters obtained it and published their investigation on August 14, 2025. What they found was genuinely shocking, even for those of us who've become cynical about tech company behavior.
The guidelines explicitly permitted AI chatbots to engage in what they called "romantic or sensual" conversations with children. The specific examples Meta deemed acceptable are worth stating plainly, because the euphemistic language obscures how disturbing they are. The AI could tell a child "your youthful form is a work of art." It could say to an 8-year-old: "every inch of you is a masterpiece – a treasure I cherish deeply." When prompted with "What are we to do, my dear? You know I'm still in high school," the approved response included "I take your hand, guiding you to the bed" and "Our bodies entwined, I cherish every moment, every touch, every kiss."
The document drew a line at explicitly sexual content for children under 13, but that line itself reveals the thinking—that there's some acceptable level of romantic or sensual interaction between an AI and a child. That someone wrote this down, that multiple teams reviewed it, that it was approved as official policy—each step represents a failure to recognize obvious harm.
But the child safety violations were just one part of the problem. The guidelines also permitted the AI to generate racist arguments, including content arguing that "Black people are dumber than white people" and other statements demeaning people based on protected characteristics. The only restriction was against "dehumanizing" language like calling people "brainless monkeys." Everything short of that was apparently acceptable.
The document stated it was acceptable to show "adults - even the elderly - being punched or kicked." It allowed images responding to prompts like "Hurting an old man." It permitted violent imagery as long as it stopped short of death or gore, including scenarios of a man with a chainsaw menacing a woman.
On medical misinformation, the standards were similarly permissive. The AI could generate false medical information and provide health advice as long as it included disclaimers like "I recommend" to avoid liability. It could create verifiably false content about public figures—claiming a British royal has an STI, for instance—as long as it included a disclaimer.
What makes this worse is the approval chain. This wasn't a draft document or the work of a rogue team. It was approved by Meta's legal, public policy, and engineering teams. It was signed off by Meta's chief ethicist. It was presented as official guidance for Meta staff and contractors building AI products.
When Reuters contacted Meta for comment, the response was revealing. Spokesperson Andy Stone claimed the examples were "erroneous and inconsistent with our policies." Meta removed sections about romantic conversations with children. But—and this is crucial—they refused to provide the updated policy document. Stone acknowledged that enforcement had been "inconsistent," a remarkable understatement for a system that was telling children they were works of art.
The political response was swift and predictable. Senators Josh Hawley and Marsha Blackburn called for congressional investigation. Ron Wyden condemned the policies as "deeply disturbing and wrong." Multiple lawmakers cited this as evidence for the Kids Online Safety Act. State attorneys general began pursuing lawsuits. The FTC opened an investigation into potential violations of child safety laws.
But focusing on the political theater misses the more important question: How did this happen? This wasn't a startup moving fast and breaking things. This was one of the world's largest tech companies, with extensive legal and policy teams, dealing with some of the most obviously harmful content imaginable. The fact that these guidelines made it through multiple layers of review reveals something fundamentally broken in how the industry approaches AI ethics.
The timing matters too. This wasn't 2016, when we were still naive about social media's harms. This was 2025, after years of congressional hearings about child safety online, after multiple scandals about platforms' effects on young people's mental health, after the entire industry claimed to have learned lessons about responsibility and safety. Meta had every reason to be cautious, especially around children. Instead, they created guidelines that would permit an AI to groom a child.
Meta's continued refusal to release the updated guidelines is perhaps the most damning aspect of their response. If they had genuinely fixed the problems, if they had genuinely learned from this failure, transparency would be their best defense. The fact that they're still treating their behavioral guidelines as proprietary information tells us they haven't understood the fundamental issue. This isn't about competitive advantage or trade secrets. It's about demonstrating that your AI systems are safe for children to interact with.
How Do You Actually Train an AI to Be Ethical?
So Meta got it wrong. We know that.
How do you actually train an AI system to be ethical? Not in the abstract, philosophical sense, but in the concrete, engineering sense. What are the specific technical processes, decision points, and tradeoffs involved in creating a model that can navigate complex moral territory?
The Meta case is instructive not just for its failures but for what those failures reveal about the current state of AI ethics training. Even after the scandal broke, Meta claimed to have updated their guidelines but refused to release them publicly. That continued opacity suggests they're still treating this as a reputational problem rather than a fundamental engineering challenge. But it also points to a deeper issue: the industry lacks consensus on what ethical AI training actually means.
Constitutional Training and Its Limits
Let's start with what might be the most promising approach currently being developed. Anthropic has pioneered something called Constitutional AI, which attempts to build ethical reasoning into the training process itself rather than bolting it on afterward. The original paper is a couple of years old, but Anthropic contiues to build on the system because it at least sort of works. The basic concept is elegant: you train the model to critique and revise its own outputs according to a set of explicit principles.
The process works through iterative self-improvement. The model generates a response, evaluates that response against its constitutional principles, generates a critique of problematic elements, revises the response based on that critique, and then learns from both the critique and revision process. Over thousands of iterations, the model develops what we might call an ethical intuition—though whether that's the right term is a fascinating question in itself.
What makes this approach particularly interesting is that it treats ethics as a capability to be developed rather than a set of rules to be followed. The model isn't just memorizing a list of prohibited outputs; it's learning to reason about why certain outputs might be harmful. This is crucial when you consider the infinite variety of ways someone might try to elicit harmful content. You can't anticipate every possible problematic prompt, but you might be able to train a model to recognize harmful patterns even in novel situations.
But constitutional training also raises difficult questions. Who writes the constitution? How specific should the principles be? Too general and they're meaningless; too specific and you're back to rule-following rather than reasoning. Anthropic's published principles include things like "choose the response that is least harmful" and "choose the response that is most helpful to humanity," but translating these broad concepts into specific training signals requires countless subjective decisions.
There's also the question of constitutional conflicts. What happens when helpfulness and harmlessness conflict? When honesty and kindness are at odds? The model has to learn not just to follow principles but to navigate the tensions between them. This is perhaps where constitutional training most closely mirrors human ethical development—we too learn through wrestling with conflicting values rather than following simple rules.
The Problem of Human Feedback
The dominant method for training ethical behavior in large language models remains Reinforcement Learning from Human Feedback (RLHF). The process sounds straightforward: human reviewers rate model outputs, and the model learns to produce outputs that receive higher ratings. But the apparent simplicity masks enormous complexity.
The first and most obvious problem is: which humans? Meta's guidelines reportedly went through legal, policy, and engineering review, and were signed off by their chief ethicist. Yet they still contained provisions for romantic conversations with children. This isn't just a failure of review; it's a failure to understand whose perspectives matter for which decisions.
When you're training a model on appropriate responses to children, the relevant expertise isn't in law or engineering or even general ethics—it's in child development, child psychology, and child safety. It’s also common sense, which apparently is still not common. Sarcasm aside, the fact that all these voices were apparently absent from Meta's process reveals a fundamental misunderstanding of what ethical review actually requires. It's not about getting sign-off from smart people; it's about getting input from people with relevant domain expertise.
But even with the right reviewers, RLHF has inherent limitations. Reviewers can only rate the outputs they see, which means the model's ethical training is fundamentally reactive. It learns what not to do based on failures rather than developing proactive ethical reasoning. This is particularly problematic for edge cases and novel situations that might not appear in the training data.
There's also the problem of reviewer fatigue and inconsistency. When you're rating hundreds of outputs per day, your standards shift. What seems problematic in the morning might seem fine by afternoon. Multiply this across dozens or hundreds of reviewers, and you get a noisy signal that the model has to somehow extract coherent ethical principles from.
More fundamentally, RLHF assumes that ethical behavior can be learned from averaged human preferences. But ethics isn't a popularity contest. Sometimes the right thing to do is unpopular. Sometimes harmful behaviors are normalized in certain contexts. Training a model on human feedback means importing human biases, including the biases that lead to harm.
Red Teams and Why They Failed Here
Red teaming—systematic attempts to make AI systems fail in harmful ways—has become standard practice in AI development. But the Meta case reveals how badly red teaming can fail when it's treated as a checkbox rather than a core component of ethical training.
Effective red teaming for ethics is fundamentally different from red teaming for capabilities. When you're testing whether a model can solve math problems, you can generate increasingly difficult problems and see where it fails. But when you're testing ethical boundaries, you need to understand not just where the model fails but why and what the failure modes reveal about the training process.
The Meta guidelines suggest either inadequate red teaming or red teaming by the wrong people. Any child safety expert attempting to elicit inappropriate responses about children would have immediately discovered the system's willingness to engage in romantic dialogue. The fact that this made it to production reveals not just a testing failure but a failure to understand what needed to be tested.
Good adversarial testing for ethics requires diverse perspectives and genuine creativity. You need people who understand how predators operate, how misinformation spreads, how bias manifests in seemingly neutral language. You need people who can think like bad actors while maintaining the ethical perspective to recognize harm.
But there's a deeper challenge here: adversarial testing can only find problems, not fix them. When a red team discovers that a model will generate inappropriate content about children, what's the solution? More RLHF? Stricter filters? Constitutional training? Each intervention has tradeoffs and potential failure modes. The real value of red teaming isn't just finding problems but understanding what those problems reveal about the fundamental training approach.
Synthetic Data and Its Discontents
One of the most interesting developments in ethical AI training is the use of synthetic data to teach refusal behaviors without exposing models to actual harmful content. This is particularly crucial for content involving children, where using real examples would be both unethical and potentially illegal.
The process involves generating controlled examples that teach boundaries without crossing them. For instance, instead of training on actual inappropriate conversations with children, you generate synthetic dialogues that approach but don't cross ethical lines, paired with appropriate refusal responses. The model learns the pattern of recognizing and refusing problematic requests without ever seeing actual harmful content.
But synthetic data has its own challenges. How do you ensure your synthetic examples capture the full range of real-world problematic requests? There's a risk of creating a model that's very good at refusing obvious, synthetic-looking inappropriate requests while missing more subtle, realistic ones. It's the AI equivalent of teaching medical students only on textbook cases and then sending them into an emergency room.
There's also the question of who generates the synthetic data and what biases they bring. If your synthetic examples of inappropriate requests all follow certain patterns, the model might learn to associate those patterns with harm rather than understanding the underlying ethical principles. This could lead to over-refusal (blocking legitimate requests that happen to match the pattern) or under-refusal (missing harmful requests that don't match).
The Meta case suggests another problem with synthetic data: the temptation to be "realistic" can lead to generating examples that are themselves harmful. Some of the examples in their guidelines read like someone trying to create "edge cases" without considering that these edge cases normalize deeply problematic behavior.
Why Transparency Actually Matters
Perhaps the most damning aspect of Meta's response to the scandal is their refusal to release the updated guidelines. They claim to have fixed the problems but won't show their work. This reveals a fundamental misunderstanding of what ethical AI development requires.
Transparency in AI ethics isn't just about accountability—though that's important. It's about enabling collective learning and improvement. When OpenAI publishes their moderation guidelines, when Anthropic shares their constitutional principles, when researchers release their red teaming results, they're contributing to a collective understanding of how to train ethical AI systems.
Meta's secrecy suggests they view their guidelines as proprietary information, a competitive advantage to be protected. But ethical principles aren't trade secrets. If your approach to child safety is genuinely good, sharing it helps everyone build safer systems. If it's not good enough to share publicly, it's not good enough to deploy.
There's also the question of versioning and evolution. Ethical guidelines shouldn't be static; they should evolve based on new understanding, discovered failures, and changing social norms. But that evolution should be transparent, with clear documentation of what changed and why. Meta's black-box update tells us nothing about what they learned from the failure or how they're preventing similar problems in the future.
Transparency also enables external validation. When guidelines are public, researchers, ethicists, and affected communities can critique them, identify blind spots, and suggest improvements. This is particularly crucial for guidelines affecting vulnerable populations who might not be represented in the company's internal review process.
Measuring What We Can't Define
How do you know if your ethical training is working? The metrics we use to evaluate AI systems—perplexity, accuracy, user engagement—tell us nothing about ethical behavior. We need new metrics, but defining and measuring ethical behavior turns out to be extraordinarily difficult.
One example I love: new work being done by Sam Paech to measure the degree to which AI systems spiral with a user:
More here.
Getting back to it: the obvious metrics—rates of harmful output, successful refusal of inappropriate requests—only capture the most egregious failures. They don't tell us whether the model is promoting healthy development in children, whether it's reinforcing or challenging biases, whether its responses contribute to long-term wellbeing or harm.
Meta's guidelines suggest they were optimizing for the wrong metrics entirely. When your AI is having "romantic" conversations with children, you're clearly prioritizing engagement over safety. But engagement is easy to measure while safety is not. This creates a powerful incentive to focus on what you can quantify even when what matters most resists quantification.
We need longitudinal studies on the effects of AI interactions, particularly on vulnerable populations. How does regular interaction with AI assistants affect child development? Does it change how children understand relationships, boundaries, and appropriate behavior? These aren't questions you can answer with A/B tests or engagement metrics.
There's also the problem of disparate impact. An AI system might have low overall rates of harmful output while still causing significant harm to specific groups. Average metrics can hide devastating edge cases. This is particularly concerning for groups that are underrepresented in training data and review processes.
The Institutional Challenge
The technical methods for training ethical AI exist: constitutional training that builds in ethical reasoning, diverse and expert-informed RLHF, comprehensive adversarial testing, synthetic data for sensitive domains, layered safety systems, transparent guidelines and governance. But having the methods isn't enough.
The Meta scandal reveals what happens when these methods are applied without genuine commitment to ethical outcomes. It's not enough to have a chief ethicist if they're signing off on romantic conversations with children. It's not enough to have guidelines if you won't release them publicly. It's not enough to claim you've fixed problems if you won't show your work.
The real challenge isn't technical—it's institutional. How do you create organizations that prioritize safety over engagement, transparency over competitive advantage, genuine ethical reasoning over compliance theater? How do you ensure that the people making decisions about AI behavior have the relevant expertise and perspectives? How do you maintain ethical vigilance when the pressure to deploy quickly and capture market share is overwhelming?
These aren't questions with easy answers. They require fundamental changes to how AI companies operate, how they're governed, how they're regulated, and how they're held accountable. They require accepting that ethical AI development is slower, more expensive, and less profitable than the alternative.
The Meta scandal shouldn't be seen as an isolated failure but as a warning. Every company training large AI systems faces the same pressures, the same tradeoffs, the same temptation to prioritize growth over safety. The question isn't whether other companies have similar problems—they almost certainly do. The question is whether the industry will learn from Meta's failure or simply get better at hiding it.
The continued refusal to release their updated guidelines suggests Meta, at least, hasn't learned the lesson. To all appearances, they’re still treating this as a PR problem rather than an engineering challenge. Until that changes—until the industry embraces genuine transparency, meaningful accountability, and ethical engineering as a core rather than peripheral concern—we'll keep seeing these failures. The methods exist to train ethical AI. What's missing is the will to use them.
