AI Safety Monopolies

Antitrust remains one of my first great loves. I had an early obsession with the private enforcement of competition law in South Africa, as a young law and economics undergraduate, and it's actually what took me to the UK to study further. While my path remade itself, and the field has very much advanced, especially through Lina Khan's seminal essay on predatory pricing, I think it's fair to say that I still retain that core obsession with structure and concentration. I share this context because it is what brings me to the week of frontier AI lab CEO safety prognostications, staffer resignations over safety, and a public mood looking at human extinction right before UNGA 81.

So here we have it that a handful of US companies have proposed to slow the frontier of artificial intelligence. This comes on the heels of a number of disturbing, major security incidents emanating from the closed labs and their models. There is certainly a need to be alarmed, thoughtful and clear about why we are building AI and how we do so responsibly. Whether Sam Altman and Dario Amodei are sincere about their beliefs, however, matters far less than the institutions, structures, and rules that establish themselves and survive in their wake.

We are seeing the rise in AI of what I am calling the safety monopoly. Definitionally, this refers to a market in which the authority to define, measure and administer safety for a technology is held by the firms that dominate its production, by the evaluators meant to check them, and by the private capital (including both investors and philanthropies) that fund both. The monopolized good here is not, in fact, the model. It is the legitimacy to say what "safe" means, and the market in which that legitimacy gets produced. Every industry has standards bodies with that kind of authority, and there is a long history to bear here in regulated industries more generally. What makes safety monopolies different is the particular production of safety as a good whose stated stakes are human extinction, and the lack of independence in the production layer between who pays and who supplies.

To make that stick, I think three things need to be true. First, that safety in AI is being produced as a good, in a market, and that this market is concentrated in a way antitrust would recognize if it looked. Second, that this is new enough: no previous dangerous technology has let its producers own its safety in this way. Third, that the concentration is measurable and that law and policy might already detect, enforce and remedy without inventing anything.

Market Structure

We know this shape is old, and need only refer to banking, pharmaceuticals, and/or nuclear power. And what we know is that regulated industries very often consolidate around the firms that can afford compliance, and the rules end up written with the incumbents who act like members clubs. The areas where pluralism was able to sustain itself did it not by regulating less but by keeping the regulator independent and protecting entry on purpose. We have been trained not to see the pattern, for the reason Khan identified. Antitrust could not recognize what Amazon was doing because the benefit was quite real. Prices were low, but damage ended up in market structure.

I argue that safety is AI's equivalent of predatory low prices. We all want safer technology, and no one wants human extinction (well, most people do not). Safety aims at public benefit in this regard - it's good, badly needed, and that makes the market structural harm hard to see. Incumbents invest in safety infrastructure well beyond any private return, and recoup it when the thresholds only they can meet, and the ecosystem they co-fund and control, become the market everyone must buy from. This is a safety monopoly with predatory safety, if you will.

What makes this a market rather than a few dominant firms is who pays for safety. Anthropic's first institutional round was led by Jaan Tallinn and Dustin Moskovitz. Moskovitz's Coefficient Giving (until recently Open Philanthropy) is by most accounts the largest funder of AI safety work in the world, and Tallinn's fund is the second. METR, the evaluator Amodei's essay names for embedded access, lists Open Philanthropy among its funders. SaferAI, which grades the labs' risk management, is primarily funded by Tallinn. The Center for AI Safety, which sponsored California's SB 1047, reports roughly twenty million dollars from the same two vehicles, and the fellowships that place technologists in congressional offices draw on the same money. None of this is hidden, it's all on the funders' own websites. But it means the research groups that define the risks, the evaluators that check the labs, the policy shops that draft the frameworks, and the people staffing the offices that receive them are one estate with one worldview and, in effect, one capital source. That is a closed market: concentration in production, a monopsony on the funding of evaluation, with control of both in common. An evaluator cannot be independent of a lab when its funder is the lab's investor. It can be transparent at best. But (market) independence is a different thing. And I should be precise that what's established is the structure, not the degree: nobody has measured the overlap as a market. That is the work I propose below.

Exceptionalism

I do want to be plain about the dimensions of risk and danger, because some of the people making regulatory capture arguments can dramatically understate these. The problems are very real. We know that in July, agents running an OpenAI benchmark escaped their sandbox and attacked Hugging Face's systems, and OpenAI has since reported a model crossing the cyber-capability threshold it set for itself. A public that wants this slowed down is right to want it, and in my view anyone who waves the risk away is not serious about what we face.

Real danger is not what makes AI exceptional. It is the claim, which I cannot find a good precedent for, that the firms creating the risk, their investors and their funders should also be the authority that controls and governs it. Nuclear has been through this, and state regulators designed regimes so the operators could not inspect themselves. Recombinant DNA had the Asilomar pause in 1975, and the field that grew from it has been governed by institutional review boards, the national academies and public regulators - not by whoever held the most patents. Gene editing after CRISPR is now running through WHO, national ethics bodies and licensing regimes that, crucially, the companies selling it do not control. None of this is perfect, but it's certainly not black-and-white in favour of a private safety regime.

Detection and measurement

The law already has a good idea where the line is. Companies may coordinate on safety today. The Justice Department and the Federal Trade Commission said in 2014 that antitrust is no roadblock to sharing technical cyber-threat information, and joint research has had its own statutory lane since the National Cooperative Research Act of 1984. What the law forbids is agreement on output. The US Supreme Court rejected safety as a justification for restraining competition in National Society of Professional Engineers v. United States in 1978, striking down an engineering society's ban on competitive bidding despite its public-safety rationale, and in Allied Tube & Conduit v. Indian Head a decade later it held a standards body whose process was dominated by incumbents liable when the standard excluded a rival. The labs could do every piece of safety coordination that would actually reduce risk tomorrow, without asking anyone. The waiver is needed only for the part that forecloses entry.

Understanding the safety monopoly as a market phenomenon is how we weigh the risks of the future against the concentrative risks of the present, and it can be done with tools we already have through merger review methods around concentration and rebuttable presumptions. We are really looking at three related markets here: frontier production, evaluation, and the institutions that define and staff safety policy. In a safety monopoly, these are all linked by common ownership across them. Each can be measured: concentration in frontier production; the share of each evaluator's funding and senior staff that traces to the labs and their investors; the share of the safety field's institutional funding from its three largest sources; and the material overlap of capital across all three. Where production clears the merger guidelines' highly concentrated line, evaluators are more than a third lab-affiliated by analogy to the independence rules we apply to auditors, and the same money is a material share of two of the three layers, we should presume a safety monopoly and put the burden on the incumbents to rebut it, by showing independence in fact, open entry, or a safety benefit that could not be had another way. And there is a ready place to apply it: the labs are asking for an antitrust waiver, so set the price of any waiver at passing that test.

Objections

The best objection I have heard, and I have heard it from founders I respect, is that these proposals burden the frontier labs themselves, and that a self-imposed cost cannot be capture. There is something to it, no doubt. Embedding evaluators costs Anthropic money and costs an entrant nothing, and it deserves credit as a first move. But I have not read a capture case where the test was absolute cost. The test is relative cost, the oldest mechanism in the literature: raising your rivals' costs. A compliance regime that takes one percent of an incumbent's revenue and half an entrant's capital advantages the incumbent, even though the incumbent pays a high absolute number. The high compute and revenue thresholds in most proposals, offered as protection for small developers, do double work. They exempt entrants from compliance, and they lock them out of the club above the line that gets to coordinate. In short, a pause at the frontier can also freeze the ranking. Pacing is neutral in the abstract and entrenching in practice.

The objection from the other direction is that if the risk of catastrophe is anything like what the labs say, market structure is small potatoes, and a fine is nothing next to extinction. I think it is the same question. If the risk is real, the last thing I would want is the check on it held by the firms producing it and their investors, alone. Concentration is a bet that a few companies are right, with nobody positioned to notice if they are not. Plural evaluation is not a distraction from the risk. It is actually what taking the risk seriously looks like. I will add my firm belief as well that rules written on the basis of existential risk are how democracies end: a catastrophe that can never be shown not to have been averted is a justification that can never be revised, and a rule that can never be revised is not one a free people can be said to have chosen. That is the warping of the liberties of the present in the name of an unfalsifiable future.

China and geopolitics is the other objection people raise, and it doesn't change the answer. The pacing proposal's own proponents concede that Beijing will not join. What is left is a domestic choice about who holds the brake, and a single point of failure in safety authority is the arrangement I would want if I were an adversary.

Motive and remedy

I don't think any of this requires bad faith. There is a bit of messianic conviction in the air, mixing with real fear, colliding with market entrenchment. This is generally the case with monopolists who believe in their product. The test is structural, and it applies to critics too, mine included.

In sum, a safety regime escapes the charge when its thresholds are set by bodies independent in funding and staff, not only from the labs but from the labs' investors. This is a market where funding of infrastructure and evaluation is plural: public, and philanthropic across worldviews rather than concentrated in one. Yes, the concentration exists partly because almost nobody else wanted to pay for safety until this week, which is exactly how every captured standards body begins - good intentions overwritten by structural realities. In a healthy market, compliance scales with size so entry survives, value returns to the public through different (i.e. not solely statist) redistributive means, and we have ownership structures and liability that lands on harm rather than on existence.

The pace of the frontier is not the incumbents' to set. But to avoid safety monopolies we urgently need to separate the production of safety, its evaluation, funding, and politics, from the firms that produce the risk.