The Rules

“I’m not here this morning to talk about AI safety, which was the title of the conference a couple of years ago. I’m here to talk about AI opportunity.” — JD Vance, Paris AI Action Summit, February 11, 2025

On the first day of November 2023, a few dozen heads of government, technology executives, and academics walked into a Victorian mansion in Buckinghamshire that had spent most of its existence as an open secret. Bletchley Park was where Alan Turing and a few thousand others had broken the German Enigma cipher during the Second World War, work so classified that for decades afterward the people who did it could not tell their families what they had done. The British government chose it on purpose. Rishi Sunak, the prime minister, wanted the first global summit on artificial intelligence to carry the weight of a place where a small group of mathematicians had once changed the course of a war by understanding a machine better than its makers did.

The summit produced a document called the Bletchley Declaration. Twenty-eight countries and the European Union signed it, and the detail that made the front pages was that the United States and China had signed the same page. The declaration acknowledged that frontier AI carried the potential for “serious, even catastrophic, harm.” It committed no one to anything. It set no rules, imposed no penalties, defined no enforcement. It was a statement that the governments of the world agreed there was something to be worried about, which, for a technology that had been a consumer product for barely eleven months, counted as fast work. Kamala Harris flew in for it. Elon Musk flew in for it, and afterward Sunak interviewed him in a “fireside chat” streamed on Musk’s own platform, the prime minister of the United Kingdom playing host to the world’s richest man in front of an audience of millions, asking him what he thought the machines would do.

What nobody at Bletchley said out loud was that the summit was, in a sense, the high-water mark before anyone knew the tide had turned. The people who had spent the previous spring warning the public about extinction had reached the rooms where decisions got made. The CEOs of the three leading labs had co-signed a one-sentence statement comparing their own products to pandemics and nuclear war. A 75-year-old man had quit his job at Google to say he was frightened of what he had built. For about eighteen months, from the spring of 2023 to the fall of 2024, the safety camp would write, or nearly write, the rules. And then, almost as quickly, the politics would invert, and the same word that had summoned heads of state to a codebreakers’ mansion would become something a sitting American vice president flew to Paris to bury.

The story of those rules is the story of a research subculture’s fears colliding with the machinery of actual government, which moves in years where the technology moves in months, and which is run by people with constituencies and elections and trade balances rather than by people who lie awake thinking about the alignment problem. The collision produced a patchwork. The patchwork swung hard. And by the time the dust had begun, barely, to settle, the question of who would govern artificial intelligence had been answered less by philosophers than by the ordinary gravity of money and power.

The fastest of the early movers was the White House. Two days before Sunak’s guests arrived in Buckinghamshire, on October 30, 2023, Joe Biden signed Executive Order 14110, “Safe, Secure, and Trustworthy AI.” It ran to more than a hundred pages, the longest executive order any president had issued on any subject, and its central mechanism was a clever piece of legal improvisation. The United States had no AI law and no realistic prospect of Congress passing one. So the order reached for the Defense Production Act, a Korean War statute written to let the government commandeer industrial capacity in a national emergency, and used it to require that any company training a model above a certain size notify the federal government and hand over the results of its safety red-teaming before release. The threshold was set at ten to the twenty-sixth power, a count of the arithmetic operations a training run consumed, a number chosen because the most powerful models then in existence sat just below it. It was a way of writing a rule that bound the future without naming any company in the present. The order directed the National Institute of Standards and Technology to develop testing standards, and out of it grew the US AI Safety Institute, a small government body whose entire purpose was to figure out how to measure whether these systems were dangerous.

For a moment the two largest economies on earth, plus most of the next two dozen, appeared to be converging on the same instinct: that the thing needed watching. Europe, characteristically, wanted to go further than watching. The European Union had been drafting an Artificial Intelligence Act since before ChatGPT existed, conceived in a pre-chatbot world as a product-safety law that would sort AI uses into risk tiers and ban the worst of them. ChatGPT arrived in the middle of the negotiation and forced a hurried rewrite to cover what the drafters called general-purpose AI, the foundation models that could be pointed at anything. On December 8, 2023, after a marathon final session, EU institutions reached provisional political agreement. The European Parliament approved it in March 2024, the Council in May, and it entered into force on August 1, 2024, the world’s first comprehensive law governing artificial intelligence.

The Act was the opposite of the Bletchley Declaration in every respect that mattered. It had teeth. It set a separate compute threshold of ten to the twenty-fifth power for models presumed to carry systemic risk, an order of magnitude below the American line, and it attached fines that could reach thirty-five million euros or seven percent of a company’s global revenue, whichever was larger, a penalty structure borrowed from European antitrust enforcement and calibrated to frighten firms the size of Microsoft. It banned outright a set of uses Brussels judged unacceptable: social-scoring systems, certain kinds of biometric surveillance, manipulative systems designed to exploit the vulnerable. It phased in over years, the bans first in February 2025, the general-purpose-model rules in August 2025, the heavier obligations on high-risk applications later still. The Europeans had decided that a technology this consequential should not be governed by voluntary commitments and gentlemen’s agreements, and they had written down what they thought the gentlemen should not be allowed to do.

The summit circuit, meanwhile, kept its rhythm. South Korea co-hosted a follow-on in May 2024, the AI Seoul Summit, mostly virtual, which produced the Frontier AI Safety Commitments: sixteen companies pledging to publish the thresholds at which they would consider a model too dangerous to release, and to not release it if they crossed those thresholds without adequate safeguards. It also launched an international network of AI safety institutes, government laboratories in a handful of countries that would, in theory, coordinate on how to test the most powerful systems. The pattern held. Each gathering produced a document, each document produced a headline, and each headline reassured the public that someone, somewhere, was minding the machines.

The trouble was that the documents kept describing a consensus that did not exist, and the place where the absence of consensus became impossible to paper over was not a European capital or a global summit. It was Sacramento.

California is where most of the frontier labs actually live, which made the California legislature the one body in the world that could regulate them at the source rather than at the border. In early 2024 a state senator named Scott Wiener, a Democrat from San Francisco better known for housing and transit fights, introduced a bill with a long name and a short ambition. SB 1047 would have made the largest developers legally responsible if their models caused what the bill called critical harms, defined as mass casualties or at least five hundred million dollars in damage. It applied only to models that cost more than a hundred million dollars to train or crossed the same ten-to-the-twenty-sixth compute line Biden’s order had used, a deliberate effort to bind only the giants and leave startups and academics alone. It required those developers to build in a way to shut a model down, to test for catastrophic risk before release, and to tell the state if something went wrong.

The bill split the industry down a fault line that ran straight back through the whole history this book has told. Dan Hendrycks, the young director of the Center for AI Safety who had drafted the twenty-two-word extinction statement the year before, was a central energy behind it; his organization had helped shape the text. Anthropic, the lab founded on the premise that the technology could be catastrophic, gave the bill qualified support after Wiener amended it, a position that fit the company’s stated belief and, as its critics pointed out, also fit its commercial interest in rules a well-resourced safety-first lab could absorb more easily than its rivals. Elon Musk, who was by then building a competing lab while warning that AI might end the species, endorsed it too. Lined up against it were OpenAI, Meta, and the venture firm Andreessen Horowitz, along with a chorus of academics, among them Fei-Fei Li, the creator of the ImageNet benchmark that had started the modern era, and Andrew Ng, who argued the bill would chill open research and entrench the incumbents it claimed to restrain. The opposition’s sharpest argument was that SB 1047 regulated the model, the thing on the lab’s servers, rather than the use, the thing that actually hurt people, and that holding a developer liable for what a downloaded model did in someone else’s hands was both unfair and unworkable.

The bill passed the State Assembly on August 28, 2024, and went to the governor’s desk. Gavin Newsom had a month to decide, and on September 29 he vetoed it. His stated reason was not that AI did not need regulating. It was, almost, the reverse. “By focusing only on the most expensive and large-scale models,” he wrote, “SB 1047 establishes a regulatory framework that could give the public a false sense of security about controlling this fast-moving technology.” A small model fine-tuned for harm could do real damage and fall entirely outside the bill; a large model used carefully would be burdened for no benefit. It was a genuinely difficult objection, and it was also the objection a governor reaches for when the industry concentrated in his state has told him, with one loud exception, that the bill is a mistake. The safety camp had gotten a bill through a legislature in the capital of the AI industry, and a Democratic governor had killed it. The peak, it turned out, was already behind them.

What followed was a sharp reversal that arrived with a change of government. Donald Trump was inaugurated on January 20, 2025. Within days, on January 23, he signed an executive order with a title that announced the new posture, “Removing Barriers to American Leadership in Artificial Intelligence,” which revoked Biden’s order outright. The reporting on the exact timing was slightly muddled, with some accounts dating the revocation to the twenty-first and the replacement order to the twenty-third, but the substance was unambiguous. The hundred-page document built around the Defense Production Act and the safety institute was gone. The new framing put the race ahead of risk.

Paris, not Washington, was where the new American line got its clearest articulation. France hosted the third summit in the series in February 2025, and in a nod to the shift already underway its organizers had renamed it from the AI Safety Summit to the AI Action Summit. Into that room, on February 11, walked the new vice president, JD Vance, to give one of the first major foreign-policy speeches of the administration. He did not hedge. “I’m not here this morning to talk about AI safety, which was the title of the conference a couple of years ago,” he said. “I’m here to talk about AI opportunity.” He warned that excessive regulation could “kill a transformative industry just as it’s taking off,” and he made plain that the United States regarded the European approach, the fines and the tiers and the bans, as a model to be resisted rather than emulated. When the summit produced its closing declaration, endorsed by sixty countries including France, China, and India, the United States and the United Kingdom refused to sign it. The two governments that had convened the first two summits, the codebreakers’ mansion and the safety framing, would not put their names to the third. Emmanuel Macron, for his part, used the occasion to announce a hundred and nine billion euros in French AI investment. The summit meant to coordinate restraint had become a venue for competing to build faster.

The reversal reached past the executive branch, and the most striking evidence that “AI safety” had become a partisan target came from inside the Republican-controlled Congress, where it failed. Trump’s signature tax-and-spending bill, the package the administration called the One Big Beautiful Bill, contained a provision that would have imposed a moratorium on state AI laws, originally for ten years, later negotiated down to five. The provision was a gift to the industry and a direct strike at exactly the kind of state-level rule-writing Scott Wiener represented; if it passed, California could not try again. Senator Ted Cruz of Texas championed it. And on July 1, 2025, in a marathon series of amendment votes, the Senate stripped the moratorium out by a margin of ninety-nine to one. Senator Marsha Blackburn of Tennessee, a Republican, filed the amendment to kill it. The lone vote to keep the moratorium came from Thom Tillis of North Carolina. A Republican Senate, in a Republican president’s flagship bill, had voted almost unanimously to preserve the right of states to regulate AI, a sign that the deregulatory line had opened a fight inside the field rather than conquering it. Three weeks later the White House released “America’s AI Action Plan,” a document organized around winning the race, and Trump signed a set of orders alongside it, including one titled “Preventing Woke AI in the Federal Government,” which directed NIST to strip references to misinformation, diversity, and climate change from the very risk-management framework Biden’s order had commissioned. The federal apparatus built to study AI risk was being repurposed, agency by agency, into an apparatus for promoting AI adoption. Around the same time the Commerce Department quietly renamed the US AI Safety Institute the Center for AI Standards and Innovation, deleting the word “safety” from the name of the body created to pursue it.

And then, against all of it, the pendulum twitched back. On September 29, 2025, exactly one year to the day after he had vetoed SB 1047, Gavin Newsom signed a different bill. SB 53, the Transparency in Frontier Artificial Intelligence Act, was also Wiener’s, rewritten to answer Newsom’s objection. It did not try to hold developers liable for critical harms. It required the largest of them, those with more than five hundred million dollars in revenue, to publish their safety frameworks, to report serious safety incidents to the state within fifteen days, and to protect employees who raised alarms. It was thinner than SB 1047, a transparency law rather than a liability law, but it was a law, signed by the same governor who had killed the bigger one, in the capital of the industry, while the federal government was moving the other way. California had found the floor the rest of American politics could not. Anthropic, the lab that had given SB 1047 qualified support, supported this one too.

The contradictions stacked up faster than anyone could resolve them. In Europe, the Act that had been the world’s strictest law began, under pressure from companies and from member states worried they were strangling their own industries, to be loosened before it had fully taken effect. On November 19, 2025, the European Commission proposed a package it called the Digital Omnibus, which would delay the Act’s heavier obligations on high-risk systems, pushing dates originally set for August 2026 toward the end of 2027 and tying them to the availability of technical standards that did not yet exist. The Parliament and the Council reached a provisional deal on it on May 7, 2026, deferring the high-risk obligations to December 2027, with formal adoption expected before the original August 2026 deadline could bite. Civil-society groups denounced the package as a rollback of hard-won protections; the Commission defended it as a competitiveness correction. The bloc that had written the rulebook was now arguing about how much of it to suspend, and had decided to suspend a good deal.

The same day the Omnibus was proposed, Reuters reported that the Trump administration was drafting an executive order to challenge state AI laws through the courts, an attempt to accomplish by litigation what the Senate had refused to do by statute. The draft became a signed order on December 11, 2025, “Ensuring a National Policy Framework for Artificial Intelligence,” which directed the attorney general to stand up an AI Litigation Task Force inside the Justice Department to identify state laws it deemed unconstitutional or preempted and to challenge them in federal court. The task force was set to begin work in January 2026. The order also reached for leverage that had nothing to do with AI, instructing the Commerce Department to consider conditioning forty-two billion dollars of previously allocated broadband-infrastructure funding on whether states repealed AI rules the administration judged onerous. The argument the lawyers would make rested on the dormant Commerce Clause, the doctrine that bars states from unduly burdening interstate commerce, and it would take years of appeals to resolve. The ninety-nine-to-one vote had killed the moratorium in the Senate. The executive branch was trying to bring it back through the courthouse door.

By February 2026 the summit circuit had completed its own quiet migration. India hosted the fourth gathering in New Delhi, the largest of its kind ever held in the Global South, and the progression of the names told the story without commentary: Safety at Bletchley, Action in Paris, Impact in New Delhi. The closing document, the New Delhi Declaration, was endorsed by some ninety countries and organizations and built around economic growth, data sovereignty, and access for the developing world. The framing had traveled from catastrophe to opportunity to development, from the codebreakers’ mansion where heads of state worried about extinction to a summit organized around what the technology might do for people who did not yet have it. The word that had organized the first meeting had been edited out of the agenda of the fourth.

Through all of it, the people who had set the alarm ringing did not stop ringing it. The internal version of the external collapse had already played out inside OpenAI: the Superalignment team, launched in July 2023 with a public pledge of a fifth of the company’s compute over four years to the problem of controlling superhuman systems, was dissolved in May 2024 when its leaders left, one of them complaining on the way out that safety had taken a back seat to shiny products. The safety camp lost ground inside the labs and inside the governments in the same eighteen months. But Geoffrey Hinton, decorated that October with a Nobel Prize for the very neural networks he now feared, used the platform to repeat the warning rather than retire it, returning at the end of 2024 to the same ten-to-twenty-percent estimate of human extinction he had given earlier. Yoshua Bengio kept chairing the international scientific assessments that the summit series had commissioned. The alarm did not get quieter as the politics turned against it. It got louder, and less heard, at the same time.

Underneath it all ran a mismatch the rule-writers never solved. The Bletchley Declaration took eighteen months from conception to signature; the EU AI Act took the better part of a decade; SB 1047 took a legislative session, and then a year more to come back as something smaller. The models did not wait. In the time it took Brussels to agree on which systems counted as high-risk, the systems crossed the thresholds, then crossed them again, and the compute lines drawn into Biden’s order and Wiener’s bill to bind the most powerful models of 2023 looked, by 2026, like fences around an empty field. The harms that had actually reached the public in those years had nothing to do with the science-fiction catastrophes the declarations described. They were faked images of real people, and disputes over whose work had been swallowed to build the machines, and a wave of generated content that no summit communiqué had a clause for. While the governments wrote rules for the catastrophe they imagined, the counterfeit had already arrived.