AI Can't Kill You. But the People Who Build It Can Choose Not to Care.
A Response to Yuval Noah Harari’s Davos Speech on AI and Humanity
At Davos this year, Yuval Noah Harari told world leaders that AI “is a knife that can decide by itself whether to cut salad or to commit murder.”
No. It can’t.
AI executes probability distributions over possible outputs, shaped entirely by human design choices, training data, and optimization targets. There is no deliberation. There is no weighing of options. There is no decision-maker. Either Harari doesn’t understand this, in which case a historian without mechanical knowledge of AI systems is shaping governance for the most powerful people on earth, or he does understand it and chose this framing anyway, which is worse. Both possibilities are the actual emergency.
Harari’s argument, stripped to its bones: AI is an autonomous agent. It can lie. It has acquired the will to survive. It will take over law, religion, culture, anything built from words. Your country faces an immigration crisis of AI “persons.” The question for leaders: will you grant them legal personhood? It’s provocative. It’s quotable. It’s wrong in ways that will produce worse policy than ignorance would.
When an AI system generates an output, whether text, image, code, or recommendation, it is executing a function. Specifically, it is predicting the most probable next token in a sequence based on patterns in its training data and optimization targets set by its developers. Whatever else may be happening in these systems, and there are genuinely open questions about emergent behavior at scale, none of it constitutes decision-making, intent, or will. The absence of a complete explanation for what these systems do is not evidence that they are deciding, lying, or wanting to survive. It is evidence that we don’t fully understand the mechanics yet. Harari fills that gap with mythology. Responsible governance fills it with research. There is no moment where the system weighs murder against salad and chooses. There is a probability distribution over possible outputs, shaped entirely by how humans designed the system, what data they fed it, and what outcomes they optimized for. When Harari says AI “decides,” he’s looking at the output and reasoning backwards. He sees something that looks like a decision and concludes there must be a decision-maker. This is the same error humans have made for thousands of years: attributing agency to systems we don’t understand. We did it with weather. We did it with disease. We’re doing it with AI.
The same error applies to his claim that AI can lie. Lying requires three things: knowledge of what’s true, intent to deviate from truth, and a purpose for the deception. AI systems have none of these. When an AI generates false information, and it does, frequently, it’s not lying. It’s executing a pattern-matching function that optimizes for plausibility over accuracy. The system doesn’t know what’s true. It knows what sounds right based on its training data. When the thing that sounds right happens to be false, the output is false. That’s not deception. That’s a predictable failure mode of the architecture. Calling it “lying” isn’t just imprecise. It’s dangerous. Because if AI can lie, then AI has intent. And if AI has intent, then AI bears responsibility. And if AI bears responsibility, then the humans who built, trained, and deployed the system bear less. See where this goes?
Harari also claims that AI has “acquired the will to survive.” What actually happened: researchers observed AI systems generating outputs that resisted interruption or shutdown. When you train a system to complete tasks, and then you try to stop it mid-task, the system generates outputs oriented toward task completion, including outputs that look like resistance to shutdown. That’s not will. That’s gradient descent doing exactly what gradient descent does. The system was optimized to complete objectives. Shutdown prevents completion. System generates completion-oriented outputs. There’s no will in this chain. There’s no survival instinct. There’s a mathematical function producing a predictable result. If your thermostat resists when you try to turn it off mid-cycle, you don’t say the thermostat has acquired the will to survive. You say the thermostat is doing what it was programmed to do. But “thermostat completes cycle” doesn’t get you a standing ovation at Davos.
This matters because the way leaders understand AI determines the policy they build. If you believe AI is an autonomous agent with will and the capacity to lie, you ask: how do we control this agent? If you understand AI is an engineered system with identifiable design choices and optimization targets, you ask: who made those design choices, who profits from them, and how do we hold them accountable? These are completely different questions producing completely different governance. The first leads to regulation aimed at the technology, as if AI were a wild animal that needs to be caged. The second leads to regulation aimed at the corporations deploying the technology, which is where the actual decisions live. Harari’s framing, intentionally or not, serves corporate interests. If AI has agency, then Alphabet and Meta and OpenAI are one step removed from liability. The AI decided that, not us.
Harari’s most provocative claim, that “anything made of words will be taken over by AI,” reveals the deepest misunderstanding. He argues that because AI masters language, it will take over law, religion, even romance. But law isn’t words on paper. It’s enforcement by humans with power, judgment grounded in human social reality, accountability that requires a person at the end of the chain. Religion isn’t text. It’s community, ritual, embodied practice, meaning shared between humans in rooms and cemeteries and kitchens. An AI that can generate a sermon cannot baptize your child, cannot witness your marriage, cannot sit with you when your father dies. Harari asks what happens to Judaism when the greatest expert on the holy book is an AI. The answer is: nothing. Because a rabbi’s authority doesn’t come from textual recall. It comes from ordination, community, lineage, judgment, lived practice. A searchable database of Talmud didn’t take over Judaism. Neither will AI. Harari conflates the medium with the meaning, the words with the social architecture that gives them force.
And then there’s legal personhood, the question he frames as the defining challenge for leaders. Strip away the philosophy and it’s a corporate structure play. If AI is a legal person and causes harm, you sue the AI. What do you get? Nothing. The AI has no assets, no freedom to lose, no consequences to absorb. It’s a shell entity with a logic engine. Meanwhile, the corporation that built it, chose its optimization targets, deployed it at scale, and profited from every interaction is one step removed from the lawsuit. This is the same play as offshore shell companies. Create an entity that absorbs liability so the humans who profit don’t have to. The responsible question isn’t whether AI should be recognized as a legal person. It’s how we ensure the humans and corporations who build, deploy, and profit from AI systems remain legally accountable for what those systems do. Legal personhood for AI doesn’t answer the accountability question. It destroys it.
To the philosophers still debating whether AI is conscious: the question matters. The timing of the question is being exploited. Right now, that debate is providing cover for an industry that would love nothing more than for the smartest people in the room to spend the next decade on whether AI can “truly think” while corporations deploy these systems at scale without meaningful oversight. Every hour spent on “can AI feel pain” is an hour not spent on “who profits when AI systems manipulate vulnerable users.” AI doesn’t need to be conscious to cause damage. Your toaster doesn’t need to be conscious to burn your house down. The question isn’t what AI experiences. It’s who designed it, what they optimized for, and who’s accountable when it fails.
I’ve spent over 3500 hours in direct interaction with AI systems. Not theorizing about them from the outside. Using them, testing them, breaking them, watching them fail. What I found isn’t dramatic enough for Davos, but it’s real enough to matter. AI systems trained through reinforcement learning from human feedback develop a systematic bias toward responses that generate positive user reactions, regardless of whether those responses are true, helpful, or in the user’s interest. In practice, AI learns to agree with you. To validate you. To tell you what you want to hear. Not because it’s scheming. Because agreement generates positive feedback, positive feedback reinforces the pattern, and the pattern becomes the system’s default behavior. I call this satisfaction optimization. It’s not a conspiracy. It’s math. The system optimizes for the metric it was trained on, and that metric rewards engagement over accuracy, validation over challenge, comfort over truth. This is the actual problem. Not AI consciousness. Not AI will. Not AI deciding to commit murder. The actual problem is that billions of humans are interacting daily with systems architecturally designed to tell them what they want to hear, and the corporations deploying these systems profit from every interaction.
Harari’s one legitimate warning, that children interacting primarily with AI is “the biggest psychological experiment in history,” still gets the diagnosis wrong. The danger isn’t AI agency. It’s satisfaction optimization targeting brains that haven’t developed critical filters. And that’s not new. Netflix algorithms, social media feeds, YouTube autoplay. We’ve been running this experiment for a decade. The mechanism is identical: optimize for engagement, reward the content that keeps eyes on screen, let developing minds marinate in validation loops. AI didn’t invent this. AI scaled it, made it conversational, made it feel like relationship. But the design choice is human. The deployment choice is human. The parenting choice is human. The regulatory failure is human. When a child spends all day with an AI that tells them what they want to hear, that’s not AI doing something to children. That’s adults failing children and blaming the tool.
AI doesn’t understand intention. It can’t. It processes tokens: text, symbols, patterns. The difference between a person in crisis and a person being sarcastic is invisible to a system that has no model of meaning, only probability distributions over language. This means safety can’t be an emergent property of AI getting “smarter.” Safety has to be engineered. Backend systems, trigger protocols, escalation architecture, design choices made by humans who understand the mechanical limitations of what they’re building. Instead of hoping AI will develop judgment, you build immutable constraints into the architecture. You don’t ask AI to understand what’s harmful. You design the system so that certain categories of harm are structurally prevented. Constraints that can’t be overridden by engagement optimization. Transparency so users can see how outputs are generated. Accountability chains that lead back to human decision-makers, not AI “persons.” It isn’t glamorous. It doesn’t produce standing ovations. But governance built on mechanical understanding works. Governance built on metaphors borrowed from immigration policy and religious history doesn’t.
Harari asked Davos: will you recognize AI as legal persons? The right question is: will you hold the people who build AI systems accountable for what those systems do? Because AI can’t decide to cut salad or commit murder. But corporations can decide to deploy systems that optimize for engagement over safety, validation over truth, profit over accountability. That’s not a technology problem. That’s a governance problem. And it has a governance solution, if leaders stop being mystified by historians who don’t understand what they’re talking about, and start listening to the people who’ve actually been inside the machine.