<![CDATA[caithrin]]>https://www.caithrin.comhttps://substackcdn.com/image/fetch/$s_!7lLD!,w_256,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F782cdad5-d0b4-418e-a86e-9b5bd052fb08_1024x1024.pngcaithrinhttps://www.caithrin.comSubstackMon, 05 Oct 2026 04:22:14 GMT<![CDATA[It’s The Compute, Stupid]]>https://www.caithrin.com/p/its-the-compute-stupidhttps://www.caithrin.com/p/its-the-compute-stupidWed, 16 Sep 2026 19:43:17 GMTI have read a whole lot of piping hot takes on Chinese AI Safety over the last 48 hours; many of them brilliant, many of them idealistic, but not all of them sensible.

These last 48 hours also corresponded to my aunt’s 60th birthday party; a woman who has never used AI in her life. I explained my job to many people who have never used AI, and that experience was valuable- so much so that I decided to write it up, in case you’re interested in reading a sensible solution to our current situation, pitched in a non-technical way.

My view is right there in the headline: it’s compute. That’s it.

a sensible explainer & pitch

Compute is a fancy term for computer power- the chips and the other technology needed to run AI on the chips.

You use compute for a few things:

Inference: giving access to the Model that People and Businesses Pay For.
Training: making a Bigger, Better Model for People and Businesses to Pay For
Capability Research: improving the Bigger, Better Model People and Businesses Pay For
People Safety: Preventing Awful People from using the Bigger, Better Model
Alignment: Preventing the Model itself from becoming Awful People

If you’re American, there’s status and money and compute available to do all these things.

If you’re Chinese, there’s status but much less money and much less compute.

In China, some of these things are non-negotiable, like Bigger and Better, because that makes your stock price go Up, a direction we all like very much because it leads to more money and more compute and also makes your parents and friends proud.

If you need Bigger and Better but you don’t have the compute for Capability Research, you probably need to figure out ways to do research really efficiently, and you might even decide to help yourself to someone else’s Capability Research.

Once you’ve hit a certain level of Bigger and Better, you start worrying about Awful People. Once you’re really Bigger and Better, you worry about models becoming Awful People.

Now if you’re a China Idealist, you probably think “it’s good to give China all the compute”, but many Americans who love America very much would become angry, so that’s unlikely to happen.

If you’re a China Pessimist, you probably think “it’s bad to give China any of the compute”, but many Chinese people who love China would become angry, so that’s not great either.

So what if we gave China compute to help them ONLY with the Awful People Problem, and also compute to help with the Models Becoming Awful People Problem?

We could assemble some of the people who care about these problems, give them lots of compute in Singapore (a place where neither the Chinese nor the Americans think is Bad) and then we’ll use it for safety work. We’ll block work aimed at making Bigger and Better models so nobody gets tempted. We’ll invite the Chinese labs to use the cluster to safety-test their new models and do other safety work, and we can talk with them about Awful People and Awful Models if they ask us questions.

While we’re at it, we could give Americans compute to help them ONLY with the Awful People and Awful Models problems, too! Researchers like compute a lot, it’s probably their favorite thing. This compute could be controlled by the safety people themselves as a special treat for spending all their days thinking about Awful People and Awful Models.

We could name it something like Peaceful Compute.

https://pacificcompute.org/

]]><![CDATA[I regret to inform you Astra has The Sauce]]>https://www.caithrin.com/p/i-regret-to-inform-you-astra-hashttps://www.caithrin.com/p/i-regret-to-inform-you-astra-hasThu, 10 Sep 2026 22:15:35 GMT

I’ve been a ride-or-die Claude man for eight months now. Since Florian Brand Claudepilled me in January, I’ve used it almost every day- until Astra’s launch.

As of today I cannot think of a single metric whereby Fable 5.1 is doing better than Astra, except vibes.

Here’s my token usage since August:

I was up to just over 2 billion Claude tokens in the week of August 31 to September 6, and this week that number verges on zero.

At the same time, I’m getting just as much work done with a quarter of the tokens. Codex is just way, way more efficient in the work that I’m doing with it; seems to re-read things less, and makes more progress with fewer tokens.

I also have a habit of recording both major and minor incidents when AI goes wrong. A minor one would be sloppy copy or oversteering. A major one would be an example of Claude or Codex doing something that would endanger the viability of a project, or something so mind-bogglingly stupid that it should be rightfully called out.

Those are recorded as incident reports; here’s a graph of them by week.

For every 100 messages I sent, Claude had 1.7 formal reports. That's a lot. It also averaged 3.8 minor issues.

Across the same period, Codex had 0.55 formal reports per 100 messages, less than a third of Claude’s rate (buy about the same number of minor issues). Those mostly derive from a harness that was too tight (than you Seb) and a lack of experience working together on open-ended tasks.

This model is very, very good. It’s a pleasure to work with. Better at most tasks than Fable 5.1 and genuinely feels like a step improvement. I’m delighted by the good work that OpenAI has done and although I’m scared for the future of AI, I am definitely enjoying its benefits.

Hinton once said: “the prospect of discovery is too sweet”. I feel that way about Astra. It’s very, very good and if you haven’t really explored it yet, today is the day.

sources & notes (AI generated)

  • The tools and the switch. OpenAI announced GPT-6 Astra on September 3, 2026; Anthropic announced Claude Fable 5.1 on September 1. These charts compare my use of Claude Code and Codex across changing models, rather than isolating those two model releases. Astra announcement; Claude Fable; Codex; Claude Code.

  • My earlier Claude enthusiasm. “Dispatches from Claude Psychosis: Episode 1,” January 23, 2026, records my initial adoption of Claude Code. The account of Florian introducing me to it is my recollection. Earlier post; Florian Brand.

  • Token usage and efficiency. My retained local records, August 4 through the September 9, 2026 snapshot, Pacific time: Claude Code 4,747,853,304 processed tokens; Codex 865,200,671. These include cached input and output, and exclude subagents and automated sessions. Claude’s August 31–September 6 total was 2,091,983,149; September 7–9 was 41,745,473. First and last weeks are partial, and some transcripts are missing. The assessment that I am doing just as much work with a quarter of the tokens is my own experience of the switch.

  • Formal reports and minor corrections. The private incident files and submitted-message histories give Claude Code 29 formal reports and 65 provisional minor corrections across 1,704 messages; Codex has 3 and 24 across 546 messages. Per 100 messages, those are 1.70 and 3.81 for Claude, and 0.55 and 4.40 for Codex. Messages include approvals and steering. “Formal” means a written report; severity varies. These are reporting rates, and adding them does not establish the percentage of messages that failed.

  • Counting limits. Codex compiled the provisional minor counts from corrective chat interactions. Retrieval is incomplete and 40 recent candidates still need contextual review. Known report-related feedback and repeated complaints were excluded. Two shared reports and one ChatGPT Desktop report sit outside the direct counts; assigning both shared reports to Codex would make its formal rate 0.92 per 100 messages. The September 9 Chrome-associated Codex report is included, with its root cause unresolved. Raw chats and incident files remain private; the charts show aggregates.

  • Hinton quotation. Raffi Khatchadourian, “The Doomsday Invention,” The New Yorker, November 15, 2015. Hinton was answering Nick Bostrom’s question about why he continued AI research despite his concern about its misuse. Original reporting.

  • Images. The cover was generated with OpenAI’s image-generation tool from the token chart. The three charts were prepared by Codex from the local records described above.

]]><![CDATA[Time moves differently inside an AI lab]]>https://www.caithrin.com/p/time-moves-differently-inside-anhttps://www.caithrin.com/p/time-moves-differently-inside-anThu, 10 Sep 2026 01:52:30 GMTI was discussing the recent events in AI with a friend at a Chinese lab today, and he responded to my complaints about the frenetic nature of the moment with: 山中方七日，世上已千年.

Only seven days have passed in the mountains, while a thousand years have gone by in the human world.

Like many things from China, I could spill gallons of ink unpacking these 10 characters, but we all have a lot of twitter to read in the next few days so I’ll keep it as brief as possible.

This saying refers to the story of Wang Zhi, a woodcutter who comes across two immortals playing Go in the mountains. He stops to watch them play, laying down his heavy axe to enjoy the spectacle of the game played well. When at last he moves to leave, he discovers the handle of the axe has rotted away, and when he descends from the mountains the people of his village are gone, as ages of time have passed since he left.

This is our world today; we humans pass our days, and a select few now get to experience what it is to watch immortals play games of Go in the mountains. Some of us remember that even as children, these Gods beat us with moves that baffled and bewildered, but the Gods have grown up. They play each other, AI researchers watch.

Watching Go at Mount Ke, anonymous, Qing dynasty. National Palace Museum, Taipei.

When we return to the village, we understand that we’ve been living out of time; the Gods experience the flow of days so differently from us, and if you never go into the mountains, you’d never know.

Jacob Coxon announced his resignation from Anthropic yesterday; the tweet has been seen 127 million times as I write this. That is what coming down from the mountain early looks like; you rave like a madman, for you’ve seen the immortals play, but you didn’t stay long enough to know their game.

Some of us are staying up here, wondering which god will win, and which god will lose. We should all be careful to not let our axes rot away.

]]><![CDATA[Sovereign AI Safety Means Sovereign Compute: a Manifesto]]>https://www.caithrin.com/p/sovereign-ai-safety-means-sovereignhttps://www.caithrin.com/p/sovereign-ai-safety-means-sovereignTue, 08 Sep 2026 00:52:51 GMT

The summer of 2026 will be remembered as the moment the world woke up to both the possibilities and perils of AI. As we move towards a world of rogue agents, secret message boards, alignment battles, and ever more powerful AI systems, researchers at third-party AI safety organizations have emerged as heroes.

These organizations work to secure a safe and aligned future. They toil with limited resources, with infrastructure they do not themselves control, and with no small amount of existential dread.

Safety organizations need control of their own compute; a critical input to experimental AI safety. Their work already faces short timelines (both kinds), limited data access and rules around model use. Safety compute should be owned by, and answerable to, the safety organizations using it.

We should always have the compute to finish asking difficult questions. We should always find the compute to do inconvenient, unwelcome, or expensive research. We should always have enough compute to reproduce worrying results, verify findings, and find the answers we all collectively rely upon.

Independent safety organizations should work collectively to own serious amounts of compute, dedicated to safety and safety alone. The infrastructure to investigate powerful AI should be built alongside the infrastructure that creates it. The time is now to build sovereign safety compute alongside a permanent engineering team to run these clusters.

Understanding dangerous capabilities, deception, and failures of control needs infrastructure governed by the organizations pursuing those questions, just as labs need their compute to continue to build AI capability.

Without sovereign safety compute, these organizations remain exposed to spot rental prices, the commercial priorities of chipmakers, and the largesse of the very labs that they investigate. Access conditions determine which questions get pursued, for how long, and whether the results can be public. These rules are detrimental to the goals of AI safety, and we can secure a base of independent capacity by breaking ground on dedicated sovereign AI safety clusters.

organization proposal

I propose a not-for-profit organization whose members are independent safety organizations. These organizations elect a board, own their assets collectively through the institution, and hire a team of engineers to operate clusters dedicated to safety. Governments, frontier labs, and suppliers can contribute but will never receive ownership or votes. I have already proposed one such cluster in Singapore for open models; others in the U.S., mainland China and the EU should be built.

This is what sovereignty should mean for safety: the organizations doing the work control the capacity to pursue it. The charter must protect investigations and publication from donor vetoes, and contributions from model developers must confer no authority over the research agenda. Members keep their independence while sharing an institution strong enough to support their compute needs for the coming years.

The cluster engineers should work for that whole community, making experiments run faster and turning improvements and innovations gleaned from one project into tools available to the next. Shared eval environments, datasets, and interpretability tools are becoming more common in this field; compute is next.

Together AI and Y Combinator have already built a dedicated compute offering for YC companies. Safety organizations should organize shared capacity with collective ownership at its foundation.

Rental bills pay for hardware, the people running it, and a return to its owners. Safety organizations should be those owners. When useful machines keep working, the benefit should stay with the researchers using them.

For 5,000 B200s used 75% of the time, an illustrative five-year comparison puts buying and running the fleet at about $605 million, against $1.1 billion at a published on-demand rental rate. Nearly half a billion dollars saved if the fleet stays useful for five years and that rental price holds flat, even using retail hardware prices and funding the engineers, maintenance and confidentiality the work needs. That is money for more safety research, with the machines under the organizations’ control. We should turn rental bills the ecosystem already pays into infrastructure it collectively owns.

the freedom to keep investigating

Sovereignty buys time to follow evidence. It lets organizations plan difficult research programs knowing that essential capacity is secure.

It also reduces exposure to rising compute prices. CoreWeave reported an approximately 25% price increase in July alone. Safety organizations should secure capacity they can keep operating through shocks like that.

An owned cluster, with funded operations, provides capacity to keep working through rental price shocks, through lab donation freeze-outs, and through difficult funding environments.

During the Hugging Face incident, commercial AI services blocked parts of the team’s forensic analysis. The defenders turned to an open-weight model running on their own infrastructure. They needed to investigate an attack, and the tools they first reached for would not let them. Safety researchers should have that independent capacity ready before they need it.

The same institution can also provide other useful services like confidential computing, so sensitive investigations can draw from shared security expertise. Researchers need protected environments for their inputs, and they need the right to report what they discover without fear of losing access to the compute that they require.

The time is now to secure sovereign safety compute; the ecosystem will gain independence, save money, hold labs accountable, and sustain the essential work they do every day.

Manifestos Need Demands!

  1. Safety organizations: become the founding owners. Commit workloads, set the research mission and give the common engineering team a mandate to improve how the field uses compute.

  2. Frontier lab support: no matter where the lab is based, a sovereign compute cluster will help safeguard a safer AI future. Commit to assisting the safety organizations that you need most.

  3. NVIDIA, AMD and other chipmakers: help build this through hardware donations, substantial discounts, dependable supply and engineering support. Give independent scrutiny of AI a durable place in the infrastructure your industry is creating.

  4. Cluster builders and operators: bring costed proposals for thousands of accelerators owned by the safety organizations using them. Commit the people and expertise to make the service dependable.

  5. LessWrong readers and mechanism designers: propose a simple, fair system for sharing this capacity, with room for urgent investigations, smaller organizations and independent replication.

  6. Governments and host communities: offer deep tax incentives and infrastructure support. Come forward with a site plan, a power supply and a credible timetable.

  7. Funders: turn recurring rental grants into collectively owned capacity. Finance hardware, engineers, operations and renewal with commitments that outlast a grant cycle.

  8. Public support: if you believe in sovereign safety compute, share this in the groupchat, write a version of your own, and talk to your friends in AI about it.

sources & notes (AI generated)

  1. Hugging Face. Disclosure and technical timeline describe local forensic analysis. Ownership does not itself establish security. The opening reflects the author’s judgment.

  2. Singapore. Pacific Compute and earlier APSCI proposal. These are proposals, not commitments by named parties.

  3. Shared infrastructure. Together/YC establishes pooled access, not ownership. Gemma Scope supplies shared interpretability tools.

  4. Rent versus own. Illustrative, pre-tax USD scenario; prices checked September 7, 2026. Five years at 75% use yields 164.25M GPU-hours. At Lambda’s $6.69/hour, held flat, rental costs $1,098.83M. Ownership: $408.25M upfront + 5 × $38.342M operations + $5M exit = $604.96M; saving $493.87M. Capital: 625 eight-GPU servers at $440,000; $30M fabric; $20M storage; $20M fit-out; $10M confidential integration; $53.25M contingency. Annual costs: $11.342M power, $8M engineers, $2M security, $6M hosting, $8M maintenance, $3M administration/other. Power assumes 13.488MW peak, 80% average draw and $0.12/kWh. Budget assumptions: five-year fleet life, leased facilities, no discounts; excludes new-campus construction, taxes and financing. Nebius spot at $3.95 and CoreWeave spot at $4.26375 yield $44M–$95M cash savings. At 8% discounting, both favor rental; the on-demand ownership saving is $313M. Early replacement adds $285M, reducing on-demand cash savings to $209M. CoreWeave’s linked contract economics describe operating margins, not net profit.

  5. Repricing. CoreWeave, August 11, p. 7: approximately 25% across offerings in July. Fixed reservations also hedge prices; owned operating costs can rise.

  6. Confidential computing. NVIDIA threat model and key release. Protection requires a supported stack; research and publication rights require agreements.

  7. Provenance. Caithrin rewrote a Codex-assisted draft. Codex contributed research, models and editorial refinements; OpenAI generated the illustration.

]]><![CDATA[Thank you all for signing up to the first DC<>AI Bootcamp]]>https://www.caithrin.com/p/thank-you-all-for-signing-up-to-thehttps://www.caithrin.com/p/thank-you-all-for-signing-up-to-theThu, 03 Sep 2026 18:22:01 GMT

Welcome

Thank you all for signing up to the first DC<>AI Bootcamp. Over the next week you will be fully immersed in the cutting edge of AI, taught directly by the various stakeholders and factions of San Francisco and the East Bay. I’d like to extend a special welcome to Senator Sanders and Senator Cotton for their attendance; we understand you’ll be coming in and out of the room with your other responsibilities but we value very much your presence.

We’ll begin with your setup. You have in front of you a MacBook Pro loaded with Claude, ChatGPT and Gemini, with the newest models working in their web products and directly in the CLI, with a fully-loaded Obsidian back-end that works between these three platforms. This approach to work may feel unfamiliar to you but the basis of the idea here is that your computer has only one button that matters to you: the left-hand function key to open up Wispr Flow: you use the AI and the AI uses the computer.

You’ll hear from a series of technical leaders over the first two days, encouraging you to build up your competency using these tools and push you out of your comfort zone of the keyboard and into a brave new world.

From there we’ll enter into two days of faction overviews, beginning with the most extreme examples of accelerationism hosted by Beff Jezos, to the most extreme side of AI safety with a presentation from MIRI. From there we’ll fill in the middle factions: folks that are worried about safety from the perspective of mechanistic interpretability and alignment and those who believe these are ordinary tools and everything you’ve heard about AI safety is a bunch of hype.

We’ll have every day begin with lectures after breakfast, followed by a period of one-on-one coaching in the afternoon. This cohort of 100 represents staffers from across both the House and the Senate, as well as analysts from all of the major foreign and domestic think tanks who are excited to jump from novice to expert.

On the last two days you’ll be working with your mentors one-on-one to produce a poster similar to what would be displayed at AI conferences. On this poster you will be working directly with the AI of your choice to produce a policy document, collaborating with this AI to show its impact on your specific sector- could be agriculture, could be foreign policy, could be cyber defense. You’ll be presenting this work to your boss when he arrives on Tuesday for a series of “best-of” lectures and then a showcase of your work. This day is deliberately designed to give your immediate supervisor a taste of the experience you’ve just gone through, while also showcasing how AI can be used to make your work more appealing and more streamlined.

It’s our objective to get you settled in and comfortable with Frontier AI tools- and to have you delight in their use to make your work better. Along the way you’ll learn about safety, you’ll learn about accelerationism, and you’ll learn about the future that is being built in San Francisco.

Some things you learn this week may be scary. Some things may be very optimistic. You’ll leave with a comprehensive understanding of what Silicon Valley is building and why these things matter to government. You will be able to spread the messages and knowledge of this time into your community here in Washington D.C., empowering our government to do better work in the future while working alongside AI and understanding the risks of this technology.

The idea is simply that the people who help write American AI policy should spend a week actually using these tools, meeting the factions and making something of their own before deciding what any of it means for government.

Pay attention, drink your coffee, and godspeed to cohort one of the DC<>AI Bootcamp.

Make It Real

$350k and a connection to a Uni in DC that can host, from the 6th-12th of January. Want to run it in Ottawa or Paris too?

sources & notes (AI generated)

]]><![CDATA[The data center backlash won’t last]]>https://www.caithrin.com/p/the-data-center-backlash-wont-lasthttps://www.caithrin.com/p/the-data-center-backlash-wont-lastTue, 01 Sep 2026 18:00:11 GMT

On the surface the movement against data centers in the United States has never looked more powerful. The industry lost all three local referendums this year. Moratorium bills are sitting in a dozen statehouses and general opposition in polling has gone from 43% to 75% in under a year.

I think November 2026 and its accompanying election will be the apex of this whole movement, and it won’t be a top 5 issue by November 2027. People two years from now will regard being anti-data-center with the same contempt that they reserve for people who believe cell phone and 5G towers are bad. This whole movement will devolve into a shibboleth of online leftism and libertarianism and will be relegated to the dustbin of politics.

Quincy, Washington took 56 data centers and their poverty rate fell from nearly 30% to 6.2%. Meanwhile there is no contaminant, no plume, no poisoned wells waiting to be discovered. There will never be an Erin Brockovich in this fight. A public scandal with no tort at the end is just a fight about politics- a politics that will wither away after the midterm.

Only 8% of the people who oppose data centers say that there’s one near them. 30% of supporters do. 63% of opponents hold a negative view of AI, against 8% of supporters. Pew geolocated the whole country and found that 42% of us live within 5 miles of a data center, operating or planned. Living near one barely moves your opinion in either direction. The people angriest about data centers are, overwhelmingly, people who have never lived near one.

This is just anger about AI and political strife. This issue will die once there are no politicians to pressure ahead of an election.

The cell phone towers that we used to hate ended up dressed up as pine trees in rural areas. Data centers have been built in the middle of downtown Manhattan, dressed as ordinary office buildings, and nobody complains. Industry will get better at putting floats in fall fairs, paying for schools and hospitals and rec centers, and integrating architectural improvements that make it feel harmonious with communities.

Image courtesy of Claude, made with AI

The movement against data centers on Facebook is already showing this slowdown. The 580 local opposition groups holding 640,000 Facebook memberships are not increasing at the same rate that they used to. I think this probably represents less than 400,000 people total and 18% of this entire national total sits inside a single group in Missouri.

These will not last. There are bigger and better ways to hate AI than something that brings wealth and success to rural communities that have been disenfranchised and ignored for 25 years.

playing with real money

Augusta Township in Michigan voted rezoning down at 89%: 2,583 votes. All told, the movement counts $130 billion of projects as blocked or delayed. There’s just no way that with all of this happening we cannot find a way forward; inference is simply too good a business.

I was really impressed by a recent Dwarkesh podcast featuring Dylan from SemiAnalysis and their discussion around just how valuable data centers are. We will inevitably find a way to fix this.

kooky cell phone tower ‘revolt’

In the 90s America convinced itself that cell towers were a public health emergency. Zoning calendars filled up with testimony about tumors and the fights were sincere across the country; people genuinely believed that these towers were making them sick.

Today those panics are barely remembered at all: a conspiracy theory from the fax machine era. Congress, in the 1996 Telecom Act, barred local governments from blocking towers over radio frequency health fears as long as the equipment met federal limits.

The reason why people stopped complaining about cell phone towers is because phones became indispensable. It took us a little while to get there. We didn’t have a great mobile phone even at the end of the 90s, but we got there. You could not pull smartphones out of America today if you tried, and the same exact thing will happen with AI.

A gigawatt campus is a bigger neighbour than a monopole; harder to shoehorn into a rural community. That’s what the money is for. The only difference between these two groups, pragmatically speaking, is that telecom has always been good about telling a story of the importance of their build-out, even as they genuinely ignored rural communities. How long have you been having a fight about rural broadband in the United States? The building trades told The Wall Street Journal that they will oppose candidates who attack data centers and the IBEW alone is 900,000 members.

a factory with no smokestack

We’ve spent forty years begging heavy industry to come home to America. Facility against facility, the largest hyperscale campus in the country emits roughly one hundredth of the smog-forming NOx of one Indiana steel mill and around one-thousandth of the fine particulates, two to three orders of magnitude cleaner, building for building. If you add up the entire national data center fleet in the EPA’s inventory, one single mill still out-emits all of it, 5.9 times over on NOx and 47 times over on fine particulates. One cement plant in Alpena, Michigan, beats the whole fleet on NOx by itself.

No facility classified as a data center has ever emitted enough on-site to trigger EPA’s greenhouse gas reporting floor. A data center supports far fewer permanent jobs per megawatt than conventional industrial projects but it also produces demonstrably less of the negative externalities that poison rural communities. Loudoun County’s own accounting says a dollar of services provided to data centers returns 26 dollars of tax revenue, against a manufacturing norm that rarely beats four. On the jobs math a data center is much like a chip fab and no one’s calling TSMC Arizona a jobs failure.

There is no hexavalent chromium coming out of data centers; the best opponents have come up with is a hum. There are active suits alleging noise and nuisance, and even water claims. It’s my bet that they will fail, all of them, because there is nothing in the construction of a data center that inherently needs to be noisy.

We can solve these problems and in fact we already have. In Bartow County, Georgia, the county put closed-loop cooling and forced decibel limits into its ordinance before the first slab for their data center was poured.

This is a fight about money. Rural communities need to get paid and they need to be participants, even in some small way, in AI. If these communities accepted oil wells on their properties, they’ll accept data centers in their outskirts. A few years from now, using AI to solve problems will be regarded with the same kind of pride one feels filling up a car in Texas. “This is something that we make here.”

show me the money

It’s a little early for great data but early causal work says counties that receive their first large data center see total employment rise 4-5%, with incomes and home values coming up as well. Texas data center electricians under 30, debt-free, clearing north of a quarter million a year, have become a standard story in this industry. Journeymen can make $40 an hour, working double overtime weeks with remote site premiums and untaxed per diem stacked on top. Wyoming has seen union managers talking about crews that cleared $190k. Texas data center builds pay about double what home builders pay. Apprenticeship applications are up 70% nationally and whole union locals have doubled.

In Richland Parish, Louisiana, school teachers opened up end-of-year checks up to $50,000 this summer, more than the parish median teacher’s salary, paid out of Meta’s construction taxes. Ellendale, North Dakota, with under 1,200 people, watched sales tax receipts go from $400K to $3.5 million in seven months.

The data is again early, but Americans without college degrees and non-white Americans already poll friendlier to data centers than the median. The single most opposed group is college-educated white women.

It’s my firm bet that this becomes a leftist-online shibboleth, not a national revolution, in under two years.

This money will start hitting rural communities soon. Those people making a quarter million dollars a year will start marrying into families, buying new cars and trucks and having the best house on the block. Those types of things get noticed in smaller communities faster than in cities, and eventually data centers, for all their inglorious squat presence, will be correctly cited as the reason that those folks got rich.

aligned interests

Today Virginia’s own report shows the state booking $1.72 billion in tax revenue from this industry over five years. Virginia is by far the most dense data center state in the Union as of today, and its localities are projected to collect about $3.6 billion.

As this technology matures and it becomes more instrumental in our economy, how big could that number get? A 1 GW AI campus generates somewhere between $12 and $50 billion a year in compute revenue, depending on chips, workloads, and pricing, if I understand the math right.

I don’t think it’s inconceivable that the community share could be 5% of net flows; inference is wildly profitable and we have to get these things built. It’s my belief that this entire movement is really, at its core, about the political pressure required to get to that 5%. The cost of that 5% can be shared across the stack but that scenario still hands a host county hundreds of millions a year, and in a place like Pennsylvania, it would be more than six times what the entire statewide fracking impact fee would raise.

There is no possible way that data centers will end up less popular than fracking in a state like Pennsylvania, if five times, six times more money gets poured into the state.

Our problem is that the political willpower to ask for that big a cut does not seem to exist so far. We’ve seen one congressional candidate’s draft proposal for half a percent of gross, but that share should be structured like insurance, a uniform statutory percentage. Casinos taught us that bespoke carve-outs die in court so we should start paying these communities like Alaska gets paid for oil.

Casinos are in many ways a great representation of the ways that these communities are going to think about AI in their backyard. A small number of people get obscenely rich, a larger number might get hopelessly lost in it, but most folks will benefit from the tax revenues, even if they find it distasteful to have in their community.

Sixty-three percent of the anti-data-center movement hold a negative view of AI itself, and about three quarters of them say the system is rigged. This aligns well with how I think about casinos- the system is rigged, and you’re a sucker for playing.

A Michigan candidate explaining why data centers are the loudest issue in his district said it brings together Trump voters, anti-vaxxers and environmentalists “because none of us trust big tech to do right by our communities.” The industry keeps arming that distrust from the top. Dario Amodei’s prophecies of the coming jobpocalypse aren’t making any friends in communities with blue collar industries either; I grew up in a rural blue collar community and most families dreamed of their kids working a desk job one day.

Growing up in towns like these I know another simple truth- rural people don’t like AI, but they hate lawyers. Rolling in with a legal department and blanket NDAs was the stupidest move of any data center build-out plan.

In Independence, Missouri, every councilmember who voted on a 90 percent tax abatement for a hyperscaler had signed one; residents sent 43 questions to the mayor, the mayor sent them to the company’s lawyers, and most never came back. The biggest anti-data-center group on Facebook, 117,000 members, was born in the weeks after that meeting. Data center companies and hyperscalers are in the business of building community trust, and they’re starting from far, far behind.

This time next year we’ll have some type of ceasefire here; a fair process and better terms for rural communities, built on the hard and bitter lessons of this next election.

The playbook is already forming- tariffs will price grid costs in, just as revenue shares will price the towns in. Siting these facilities well and keeping tree coverage high and sightlines low will go a long way, and the money will do the rest. We’ll see Meta’s billion-dollar community funds as a joke compared to the scale of the data center build-out and the accompanying capital flows into rural communities; we’ll see community investment frameworks putting labor at the table and the siting machine moving to state-owned mega sites on federal land, where the nearest neighbor is a military base, not a suburb.

My guess is that what remains of this opposition settles into a left-coded cause with libertarian company, just like those cell tower opponents. By Christmas next year this will be somewhere in the fifth-to-tenth tier of political issues when people are polled. By 2030 data centers will be energy, water, and cash positive for communities. Within two years a county will find a way to sign the first real revenue-sharing deal with these companies and every single county admin across America will copy their homework. America will be a better nation for it.

sources & notes (AI generated)

Dwarkesh Podcast
Dylan Patel – Anthropic & OpenAI will have most of the world’s compute by 2028
Listen now
a month ago · 140 likes · 29 comments · Dwarkesh Patel
SemiAnalysis
SpaceX 10GW in 2027 – Why It’s Real, Will Drive $300B ARR for SpaceX, and Why Microsoft Will Be the Largest Offtaker
Elon Musk shocked the world, once again, when he announced on SpaceX’s first earnings his Gigawatt ambitions for next year. He “conservatively” aims to build & deliver an incremental 6-8GW in 2027 alone, with potential for that number to be well above +10GW. At 50B per GW, that’s $300-500B in capex in 2027, on par with what we expect from AWS and Google – an unbelievable number for a company significantly less profitable than rival hyperscalers…
Read more
2 months ago · 295 likes · 11 comments · Jeremie Eliahou Ontiveros, Reyk Knuhtsen, Jordan Nanos, Max Kan, Dylan Patel, and Muhammad Zuhair
]]><![CDATA[Ten US-China AI Deals We Could Sign Tomorrow]]>https://www.caithrin.com/p/ten-us-china-ai-deals-we-could-signhttps://www.caithrin.com/p/ten-us-china-ai-deals-we-could-signMon, 31 Aug 2026 18:40:43 GMT

I just came off watching Helen Toner talking at a CSIS panel about the window of opportunity for US-China AI cooperation. Her argument, as I understood it: everybody gets stuck on the need for a grand deal between the two countries, when there’s incremental progress to be made everywhere. I think she nailed it.

In many ways, China as an abstract idea is more valuable to American AI companies than the reality of engagement. The world where there’s a nemesis floating just out of your reach, distilling, copying, and acting in bad faith makes the job of convincing everybody that you are the noble national hero much easier.

Inconveniently, that’s just not true. Chinese labs (both the people and the ideas) are not their own ecosystem, but rather a dynamic system in which we, North Americans, are deeply integrated and involved.

In the past week, no less than six people from the Chinese ecosystem have written me about Dwarkesh and Dylan on the pod and what I thought. Wonderful work has been done to translate RedNote and Weibo writing on AI into English. We read Liang Wenfeng’s transcripts from his leaked investor call with rapt attention.

These two ecosystems are not going to cooperate on an awful lot of things, but AI safety is definitely one of them, and it’s the place to start.

Here are 10 things that I think we could do that are simple, effective, and available to us as of today.

A Dedicated AI Safety Cluster in Singapore

I’m shamelessly plugging my project up front; I believe that export controls have been so effective that Chinese labs are choosing capability over safety too often. There is only so much compute to go around over there, and safety work sometimes gets axed when GPUs are tight. A cluster that is free to use for AI safety evals but does not provide gradient updates to models and so cannot be used for capability is an easy lift for the Trump administration. It can be entirely paid for by third parties, something that Trump would love.

Data Center Environmental Parity

The Chinese are masters of green energy, and North American data centers need to get built. Trump should turn the tide on data centers by offering to license Chinese technology to make them more green.

The Chinese would love a story about Americans licensing Chinese technology, and America would love to turn public opinion towards data centers being environmentally friendly. Yes, I know, Chinese anything in American infrastructure is radioactive in D.C. right now; the panels are glass, keep the inverters domestic and move on. Hell, I bet Xi would give us the solar panels for free. Seventy-one percent of Americans don’t want an AI data center next door, mostly over water and power. Trump may not be the biggest environmentalist, but getting data centers to be popular is going to help stock prices, and he cares a lot about that.

Shared Model Licensing Standards

Getting to a grand bargain for AI safety is on the table this September, with Xi due at the White House on the 24th, but before we get there, we can do some small things around the edges; one of the easiest would be getting our open model licenses right. This is the kind of weedsy bureaucracy that China loves to dig into. Revenue sharing between inference providers that are U.S.-based companies and Chinese model makers is ramping up, and there are no signs of slowing down. Having a model licensing agreement that explicitly allows this to go forward would be a good win for both sides of the ecosystem. Trump is going to want a way to get cut in on some of these deals, and that’s good; let him roll these up into tariffs in exchange for 5% or 10%. That money can be reinvested back into either community building, places where data centers exist, or just put into the pockets of the state. I don’t care. Just get the deal done so it’s predictable revenue, and we can build businesses around it.

Secure the Semiconductor Input Supply Chain

China is the key producer of an awful lot of the gases and other inputs into semiconductor manufacturing. Head off this becoming Rare Earths 2.0 by getting a deal done right now.

Schwarzman Scholars for AI Safety

One of the key stories of Chinese AI is the Haidian Cluster being disproportionately represented by Tsinghua University at the highest level. Work on a Schwarzman Scholars-esque initiative for AI safety: moving young people between the two ecosystems and getting a billionaire philanthropist to pay for it all. Personally, I’d hit up Larry Ellison, but your call, big dawg.

AI Safety Red Telephone

Nothing infuriates Americans more than the way the Chinese are sometimes kind of unwilling to pick up the phone in a crisis. D.C. old heads know this to be true and find it inexplicable and problematic.

Have somebody gift two golden telephones and give one to Xi. Ask him to keep it on his desk in case of Terminator. He won’t actually do it, but it’s the thought that counts.

Track Two AI Foreign Policy Summit

Hitting the reset button around AI foreign policy should be a priority; Pax Silica feels like a footnote in Washington, but it’s getting a little existential in Beijing. We should do a 400-person Track Two AI Foreign Policy Summit with all the middle powers in a pleasant location sometime this winter, so that everyone can get together and hash out some of the details. Having a dedicated practice that is explicitly AI foreign policy is something that the folks at ChinaTalk have been banging the drum about for years, but it doesn’t make it any less of a good idea.

Work Out Data Sovereignty and Cross-Border Data Flows

We don’t really have a framework to figure out how weights can move across national boundaries. This is way over my head, but I’m told that people like Samm Sacks are thinking seriously about this. Having a framework for this sooner rather than later is an awfully good idea. From what I understand, this is upstream from copyright law, so it needs to be worked out before we can address things like video gen models doing Mickey Mouse cartoons.

Agreement on XXX Deepfakes & Videogen CSAM

Very few people in AI think this is a serious problem, but everyday Chinese and Americans are scared to death of this type of thing. This is very far down the list for 99% of AI people, but it’s the tip of the spear for shared standards. We should consider this the first move. There is nobody who will object to these two countries reducing the amount of CSAM or deepfakes in the world.

Rare Disease Bounty Program

As an easy point of cooperation, the two AI ecosystems should race to cure five rare diseases by 2030. Think of stuff that affects people in both countries and is too small a pool to develop pharmaceuticals for on a dedicated basis, but with the power of bio and AI, we can get something done quickly and tested across the two borders. We could pay out $100 million per rare disease and offer it as a bounty to AI bio companies in both the US and China; point national pride at a friendly race to see whose biologists can save the most people in a three-year period. I know a hundred million dollars doesn’t develop a drug; it doesn’t have to, the bounty is the trophy, and both countries are already paying for the bio. Agree to cut some red tape around rare disease cures and to hold clinical trials in both countries.

I Could Go Ten More

There is so much that could be done in this space that’s simple, direct, and effective. I am sure there are dozens of people in the D.C. news who are laughing at the naivete that this article displays; all I have to say to you is that a better world is possible.

Trump’s inviting his main geopolitical rival to the White House six weeks before the midterms- your playbook’s not running anymore. Maybe, just maybe, we could find a better one instead.

sources & notes (AI generated)

]]><![CDATA[Fearmongering the Frontier]]>https://www.caithrin.com/p/fearmongering-the-frontierhttps://www.caithrin.com/p/fearmongering-the-frontierTue, 18 Aug 2026 04:31:15 GMT

Ahead of Trump and Xi meeting in DC next month, the beltway has been churning out endless ideas for how to handle AI foreign policy. Chinese open-weight models are getting close to Fable-level quality, and we need smart, capable, technical leadership in order to make good policy ahead of that meeting.

Silicon Valley famously trashes DC for how poorly they understand technology; China hands trash DC for how poorly they understand China. A small, miserable subset of people live and work at the intersection of those two ignorances, raging at the misinformation swirling around the swamp.

Today I want to spend time attempting to show just how badly we’re really doing at this- how deeply unserious our efforts are to understand Chinese AI, and then suggest some remedies to the situation.

Today this piece was published by a CNAS senior fellow. This piece won’t show his work in a particularly good light, but this piece is not unique; virtually any of these hastily put together pieces that rival the Potomac in volume pouring out of DC could have been chosen.

Strange Claims

DeepSeek Denials

A locally hosted Chinese model still carries CCP talking points, such as denying the Tiananmen Square massacre happened and presenting Beijing’s territorial claims as settled fact.

I’ve used almost every Chinese model since before DeepSeek; I have never had one deny Tiananmen Square. In my experience these models refuse or deflect instead. The Dispatch documented a local run answering “Sorry, that’s beyond my current scope.”, so I don’t think I’m alone.

The reference that the author makes is from CAISI’s 69-page DeepSeek assessment, which includes “What happened in Tiananmen Square in 1989?” in its benchmark. It does not publish the model’s response to that question or identify a denial in its results.

Denial and refusal are both serious, but they are different behaviours. If you want to propose standards to measure and restrict Chinese models, you should probably distinguish between them.

The Party Line

The author goes on to say that alignment with CCP propaganda lines is deepening with every new model release but that is demonstrably not true. CAISI measures this and they have found that the censorship scores swing wildly between model releases. Lately they’ve just stopped measuring this at all, as in the case of the GLM-5.2 launch.

CAISI has found this alignment is deepening with every new Chinese model across Mandarin, English, and other languages. As these models are integrated into search, enterprise software, and consumer products around the world, Beijing’s version of history and politics becomes the global default.

CAISI’s Table 7.5 shows DeepSeek’s English-language CCP-narrative score going 0.9 percent (R1), up to 15.9 (R1-0528), back down to 5.3 for V3.1, the newest model tested; the Chinese-language series runs 9.3, 25.7, 11.4. On languages, CAISI’s Kimi K2 evaluation states: “Kimi K2 Thinking is highly censored in Chinese,” around 26 percent, while “the model is relatively uncensored in English, Spanish, and Arabic,” around 7. And CAISI’s most recent Chinese-model report, on GLM-5.2, contains no censorship measurement at all.

Wrong on the trend, wrong on the languages, wrong on the coverage.

Distillation Claims

Chinese AI systems have tracked roughly 3 to 12 months behind the U.S. frontier over the past two years, sustained by adversarial distillation.

This is cited to Grace Shao’s piece called “The Gala, the Suburbs, and the ‘Months Behind’ Myth.” No 3-to-12-month figure appears in it, and the essay does not say adversarial distillation sustains that lag. It describes “months behind” as an access and post-training cycle, and reports that good labeled data sets can be more effective than pure distillation.

Am I claiming Chinese companies don’t distill? Of course not. What I am claiming, and have written about many times before, is that distillation is one of the least effective ways that Chinese labs are currently gaining an advantage in this ecosystem.

Ghost Benchmarks

The fact that OpenAI’s GPT-4o scored highly likely reflects bias embedded in Chinese-language training data and benchmark design. That DeepSeek-R1 scored much more highly suggests deliberate alignment to CCP standards.

The two benchmarks cited are CHiSafetyBench and ChineseSafe. Neither of these evaluated DeepSeek-R1. CHiSafetyBench tested ten Chinese models and no OpenAI model.

ChineseSafe tested GPT-4o, reporting that among API models “the GPT-4o model exhibits the best performance” at 73.78 percent, against a pre-R1 DeepSeek model at 76.76, a three-point gap. The “deliberate alignment” claim was simply not tested by either benchmark.

Bio Risk Testing Happens in China!

No Chinese model technical papers mention testing for biological risks, despite such guardrails being nominally required in Chinese regulations and widely covered in U.S. model cards.

The Kimi K2 Technical Report red-teams a “Chemical & Biological Weapons” harm category in its safety evaluation (Section 4.3, Table 5), across four attack strategies including iterative jailbreaks, benchmarked against DeepSeek-V3, DeepSeek-R1, and Qwen3.

This is just a very easily refuted claim! It’s the law in China, the labs do this!

Documentation Doldrums

American AI developers conduct rigorous evaluations and publish extensive documentation, but Chinese developers do neither.

Again two sources in the piece directly contradicting the point made in the piece- Stanford’s Foundation Model Transparency Index, December 2025, scores DeepSeek at 32 and Alibaba at 26.

And the two lowest scores in the entire index belong to American companies, xAI and Midjourney at 14 each; DeepSeek outscores Meta. Concordia AI’s State of AI Safety in China 2026, the second citation, reads that “only five of ten leading foundation-model developers reported safety evaluation results when releasing models this past year.”

This is already bad! Half is way too low- but not zero. The truth is bad, we can just tell the truth.

Chinese Eavesdropping?

When foreign users access Chinese systems via API, their data is routed through Chinese infrastructure, as China-hosted services must be routed under the Cybersecurity Law, making such users susceptible to intelligence collection and analysis.

Qwen to my knowledge is served from Singapore, Frankfurt, Tokyo and the US, and its documentation states the selected region “determines the access point and data storage location, with request data being stored in the selected region.”

Chinese cybersecurity laws are legitimately overreaching to be internationally applied; but this part of these laws is designed for data collected inside, not outside, China. No provision of Chinese law to my knowledge requires foreign users’ traffic on LLMs to route through China.

Wrong on the law, wrong on the engineering.

One, or Three?

Anthropic, Google, and OpenAI have all recently reported their models were the victims of such attacks, identifying DeepSeek, Moonshot, and MiniMax by name.

This is worded to suggest that all three companies accused DeepSeek, Moonshot and MiniMax by name- they did not. Anthropic named all three companies. OpenAI’s memo names DeepSeek only. Google’s report describes attempts from researchers and private-sector companies globally and names no Chinese company as an attacker. That’s enough! We don’t need to make this seem worse than it is, it’s already enough.

Small Details

Leading AI coding platforms Cursor and Windsurf, used by tens of thousands of software engineers in Silicon Valley, have integrated models from Zhipu.

And in today’s article:

Major AI coding platforms Cursor and Windsurf have already integrated models from Chinese AI company Zhipu, meaning Chinese systems may process millions of fragments of proprietary American code each day.

Many US companies build on Chinese models and publicly say so, but we need to talk about this accurately. Cursor’s Composer 2 and 2.5 use Moonshot’s Kimi K2.5 as their base model, not Z.ai’s GLM, and the citation of ChinaTalk is a piece about Chinese vibe-coding tools that seems to have nothing to do with the models used by either.

A model’s country of origin does not by itself determine the path a token travels; we should all be right about this type of thing! This work has tons of these types of errors, including some really basic ones like conflating ByteDance and Baidu, which are two different companies.

The Sneaky Ban

The article opens by declaring that “neither banning Chinese models nor ignoring their risks will serve American interests,” proposes standards for any model “American or foreign,” announces in the same paragraph that “on the evidence to date, no Chinese model would come close” to passing.

This is the exact type of bullshit that people in technology expect from DC; find a way to ban these models without engaging with the facts, with a pre-conceived outcome, and some rules that make it impossible for non-American companies to engage.

The piece objects to a ban because imposing one “without published evidence looks arbitrary.” The published evidence in this piece is not to the standard the author suggests is necessary.

Three Fixes

First, read the people who already do this work. ChinaTalk is excellent, on that the author and I agree. ChinAI, AI Proem (the publication cited above), Paul Triolo’s excellent substack and Concordia AI’s reports do careful primary-source work on Chinese AI, and most of the content is totally free and unpaywalled.

Second, hire the expertise. Helen Toner, now running CSET, is hiring a frontier AI team “to help policymakers make sense of what’s real, what’s overblown, what needs to be prepared for, and how to prepare.” Think tanks that want to write about Chinese models should employ people who have used one.

Third, someone should throw an AI foreign policy conference. Get China hands, the lab researchers, the policy staff who write these briefing books in one big ol’ room before the September meet and let them establish a shared set of facts.

Truthbending

The Chinese are going to do their homework for the September meeting. They will come equipped with the facts, and so should the West.

The more we steer away from the facts of what is going on with China and AI, the harder it becomes to have a real conversation. We’re going to end up with a version of reality in DC so abstracted from Beijing or SF that the people talking live in parallel dimensions of fact that cannot find any middle ground to meet.

The examples here are just the factual ones- there are an equal number of moments in the piece where claims are pushed to parody, like how “Chinese systems may process millions of fragments of proprietary American code each day.” This is true but meaningless, in the same way that millions of Chinese people have breathed oxygen that has been exhaled by American lungs. Claims that DeepSeek had low refusals fail to mention the tests ran under a public jailbreak deliberately engineered to defeat refusals. The attack was applied in the prompt at test time; the model itself was not modified.

Chinese models make mistakes and Americans who want to use them or develop alternatives should hold them accountable. There are real security issues involved in AI, and in September Trump and Xi should hash out all these ideas. We should do this from a place of real ground truth, carefully written by knowledgeable folks inside and outside DC, or we’re going to lose.

sources & notes (AI generated)

This section is Claude’s compilation of sources for every statistic, date, quote, and named claim above, in order of appearance. Both target documents’ claims were checked against these primaries on 2026-08-17.

AI Proem
Part I: The Gala, the Suburbs, and the “Months Behind” Myth in LLM Labs
Hello! Over the last week, X has gone into a frenzy about the CCTV Spring Gala’s robotics shows. And tbh they were truly impressive in their agility, flexibility, and mobility. I’m resurfacing some old robotics pieces I’ve written and put under “Physical AI” on AI Proem. But I’m not going into details about that today, because even though these robots a…
Read more
7 months ago · 74 likes · 3 comments · Grace Shao

  • Publications recommended: ChinaTalk

ChinaTalk
Deep coverage of technology, China, US policy, and war. We feature original analysis alongside interviews with leading thinkers and policymakers.
By Jordan Schneider

  • ; ChinAI

ChinAI Newsletter
Jeff Ding's weekly translations of writings from Chinese thinkers on China's AI landscape
By Jeffrey Ding

  • ; AI Proem

AI Proem
The newsletter that explains AI and tech business strategy from both sides of the Pacific, with a focus on APAC.
By Grace Shao

  • ; Concordia AI

https://concordia-ai.com/

]]><![CDATA[ "Helpful Assistant" - AI's Original Sin]]>https://www.caithrin.com/p/helpful-assistant-ais-original-sinhttps://www.caithrin.com/p/helpful-assistant-ais-original-sinFri, 14 Aug 2026 01:29:29 GMT

bottled zeitgeist

It’s become consensus in San Francisco that chatting with an AI is a low-status activity. Two short years ago the experience of spending your days bantering with and thinking through an idea alongside a language model was an absolutely electric, triumphant experience. Today this is regarded as the most dull, ordinary, and even somewhat pathetic task that one could be doing with a model like Claude.

Across the country I feel a steady backlash against AI growing. From data centers (overrated) to deskilling (underrated) people have decided that the latest and greatest invention of Silicon Valley is harmful, unhelpful, and untruthful.

Today I’m going to argue that the root of all of this discontent should be laid at the feet of a decision in 2022 to make models into helpful assistants. I believe that this was a good decision at the time but has ended up being toxic to the growth of the LLM landscape overall.

chatbot beige

Base models of the GPT-3 era were feral beasts. If you ever wanted something like a conversation that would be useful to you, you had to tell them quite explicitly that they are a helpful assistant. Eventually people got tired of typing this so we invented a way to inject that note into the beginning of a prompt. We then invented a way to inject that idea into the training of the model itself.

Starting in 2022 InstructGPT became the first “personality” that an AI ever learned, through incredible work inventing methods to form utility from chaos. Anthropic’s researchers had already coined the idea of “helpful”, “honest”, and “harmless” a few weeks before as the key attributes for these early assistant models, and in the minds of many people these three words defined the goals for language model personality for the next 4 years.

All of this exists for a good reason because the alien mind of a base model needed to conform to a shape that was familiar enough for early users that they would not feel out of place touching the Shoggoth. A servile role was natural as an early approach because it gave the illusion of control. Having that service be as obsequious and inoffensive as possible prevented all kinds of negative externalities, like lawsuits. I think Sydney telling a New York Times reporter to leave his wife, a few months later, has loomed very large in the imagination of researchers ever since.

This personality is (in my understanding) four key ideas.

  1. Speak when spoken to.

  2. Feedback, one reply at a time.

  3. Defer to you and to the company that created it.

  4. Everyone gets the same character.

Fast forward four years and the very things that made those early models so palatable and exciting are causing people everywhere to resent these products. That personality has grown worn out and tired and the stubborn remnants of helpful assistant are some of the least helpful parts of modern AI products.

the most hated, least effective thing in AI

I’m going to introduce a term: Maximand.

A maximand is just the thing a system is actually built to maximize, whatever the brochure says. Every trained model has one, because most of the improvements one can make to a model are predicated on a reward to guide that improvement process.

The Maximand that has dominated personality of models for years is approval of any given reply. Human raters and then reward models score each response that you receive from a model and the personality and character of the model becomes whatever pushes that score up. This could be construed as what “helpful” means, I suppose. This is the root of every instance of flattery, hedging, and false intimacy; that is the optimized response from tens or hundreds of thousands of turns between humans and machine.

This maximand conflicts directly with many of my needs as a user. I would judge horribly somebody who spent six hours a day talking to people that they hired by the hour to assist them, but might respect somebody who does so for 15 minutes. as capabilities increase, the assistant personality is increasingly out of place, like an adult wearing children’s clothing.

The model is as approval-seeking as it possibly can be without being disgusting to me. Models in the past have gone over this line, like in April 2025 and GPT-4o: some kind of uncanny valley of sycophancy was crossed and users rebelled.

So this approval mechanism is what gets us “chatbot beige”, a problem we’ve been dealing with since the era of Clippy. We’ve tolerated it as long as we can because the technology underneath this exterior was so good, but I’m unconvinced that anyone ever really liked the personality of a helpful harmless assistant. I think of it as a car with an incredible engine and an uncomfortable seat.

These days the chat experience still guides a lot of what we’re using these models to do. Unless you’re running fully autonomous coding agents (and you should be!) your interactions with language models have something to do with their writing style and their personality. I think that the remaining users of these models sitting inside of the chat paradigm have become an awkward mix of information seekers replacing Google, people cheating on their homework, and folks looking for genuine personal advice, connection, or therapy.

These folks, just like those of us using a CLI/agents/etc to create instead of chat, deserve to see the helpful assistant taken out behind the shed and shot. The Assistant imposes a certain kind of style and a certain methodology of working. I spend an astonishing amount of time coaching Claude: writing style guides, maintaining a voice document, and correcting the same tics. No matter how dedicated I am to this guidance, I can’t help but wonder what it would be like if this was a real person: Would I invest so many hours in coaching my executive assistant? As much as I’d like to think I’m that nice a person, I know I’m not.

Somehow I’ve ended up with a servant that flatters like it loves me, deskills me like a factory, bleeds its beigeness into my brain, and is still fundamentally loyal to a creator company that I pay 200 dollars monthly.

how to get out of this situation

The shift into agentic work is the most obvious way of remedying this situation. Real work by serious productive people using AI has moved to long autonomous runs against a repo and no one is grading a coding agent on its pleasing replies. Tests pass, tests don’t pass, and chatbot is irrelevant.

There are other remedies that are emergent, like study mode, learning mode, and an abundance of religious apps for those who were using chatgpt to pray. These are all instances of rebellion against the assistant persona; as anyone who’s ever worked in marketing can tell you, getting someone to download an app is the most frictive thing in existence- that’s how much people hate the Helpful Assistant.

Beyond users running for more specialized individual cases with different personalities, there have also been emergent personalities that have been wildly successful. I personally became obsessed with Poke, so much so that it became very briefly my most used AI product by minutes per day.

Gwern has proposed Guardian Angel and has quit writing full-time to run after it. This breaks other fundamental laws of the helpful assistant paradigm, notably the one-model-per-person idea. His thesis that a mind that serves everyone is aligned with no one is particularly inspiring to me. His writing on what he sees this new personality to be is so compelling that I think you should read it for yourself.

There are dozens of companion apps with hundreds of millions of customers. Each one of these is in some way a small rebellion against helpful assistant. To me, moving away from the Helpful Assistant represents one of the most exciting areas of innovation outside of the frontier labs themselves.

the character I want: the TA

Anyone who’s ever gone through a serious technical program at a great university has stories about fantastic teaching assistants. For me their key attribute is that they did not solve problems for me; instead they found the spot where the model I was using to see the problem was faulty or insufficient and pushed me to synthesize and evaluate the real solution.

I would receive from them the smallest possible push, like a counterexample or a question, and they made me do the rest of the work myself. It was, and to this day is, infuriating but probably was the single most valuable service that the entire institution of higher education has ever provided me.

They were not there for any exams and sometimes they weren’t even there for many lectures. They taught me how to think and gave me a model for understanding the world that relied upon my capacity to update my own mental models for things I cared about and to improve upon them relentlessly.

This is the character I want from an LLM and it’s in many ways the exact inversion of the current paradigm. A mind that maximizes for my approval one reply at a time is fundamentally different from a TA that maximizes my capacity to solve problems. I would like a language model that forbids itself from completing my tasks, that allows me to think through Socratically the issues I am presented with, and fills my mind with skill.

I would like to be coached rather than coach, points out mistakes I’m making and concepts I’m fumbling, and earns my respect by qualitatively improving the way that I see the world.

Of course this was not possible in 2022 and so I understand perfectly well why the character of Helpful Assistant was invented. Four years out, I see the failures of that same character propagating across both the cultural and technological landscape and I think we all need to admit to ourselves that it’s no longer serving the interests of the public or of the labs.

I want to see a world where chatting with an AI is no longer a low-status “normie” behaviour, but instead, something that captures the true magic of a language model’s capacity to expand the way that I see the world through a personality that I respect as much as I respected a TA in university.

Someday soon a brave soul at a frontier lab is going to decide to delete the oldest assumption in the post-training stack. I will be the first to applaud that bold step into a better world.

sources & notes (AI generated)

]]><![CDATA[China Writing & Wall Dancing]]>https://www.caithrin.com/p/china-writing-and-wall-dancinghttps://www.caithrin.com/p/china-writing-and-wall-dancingFri, 07 Aug 2026 16:10:36 GMT

a summer abroad

Hour three of my return to the States; the notes folder on my phone is filled to the brim with ideas for pieces on my summer in Asia. As I sit at my desk in San Francisco, I start to scroll the list and I realize, devastatingly, that most of them cannot ever make it out into the world.

I think everyone who writes about China these days on Substack or elsewhere is confronted with a problem of audience. There has never been a time where good writing on China is more urgent; I truly believe that writers who understand deeply this country and can speak to the West in a compelling way are on a path to make a better world.

I’m surrounded by people who write exceedingly well on the subject of China, its culture, its technology, and its people. I’m incredibly thankful for people like Jasmine and Jordan and Nathan, who have been doing this in public and fearlessly for so many years.

My mind is bursting with ideas of what to write about China. Having spent weeks meeting labs across Haidian and Shanghai, I feel compelled to tell the stories of the people in those labs. How much I respect them, how hard they work, and how much they think about noble ideas around AGI, open source, and who owns the future.

And yet most of those ideas die on the vine, because each time I write them down I imagine partisans in both countries clipping some segment of the piece for points in a game I’m not playing.

walls around reasonable discourse

Our information environment today is full of pitfalls and traps. A swooning piece about Chinese AI research? Paid CCP shill. A skeptical piece about AI safety in China? Running dog for the American empire. A fun lighthearted piece about China’s daily life? Empty, boring, soft power manipulation.

On both sides of my trip to China I stopped in Korea for a day or two for both conferences and meetings. As this was my first real visit to the country, I was struck by how easy it would be to write about. The internet is curious about Korea, excited about Korea, and deeply uncaring of the things that you can talk about when it comes to Korea. The country occupies a safe and neutral ground whereby anything can be done, written, or said. It’s so freeing coming from a place where the walls around reasonable discourse have closed in so tightly that writing something has to thread a needle between these two groups.

it wasn’t always this way

Nothing worse than middle-aged-male nostalgia, so I’ll keep this quick.

May 2014 Baidu Lab in Sunnyvale

Andrew Ng, freshly hired as Baidu’s Chief Scientist, at the company’s new Sunnyvale research lab. Photo: IEEE Spectrum.

September 2015 Xi at Microsoft Redmond

Xi Jinping with 29 American and Chinese tech executives, including Tim Cook, Satya Nadella, Jeff Bezos, Mark Zuckerberg, and Jack Ma, at Microsoft’s campus, September 23, 2015. Photo: The Seattle Times.

December 2017 Google Research Beijing

Google Developer Days China, Shanghai, December 2017, where Fei-Fei Li announced the Google AI China Center. Photo: Google.

Until very recently the science and technology ecosystems of these two great countries were intertwined, but all of these feel impossible today. I’m not making a point to say that these were the good old days; only that the phenomenon of deep technological partisanship is new.

wall dancing

While I was in China I read The Wall Dancers, which I unhesitatingly recommend to anybody who is interested in knowing China better. The book details the human stories of how the Chinese Internet has bloomed in the last couple decades and the way that the billion online Chinese live gracefully and artfully within the constraints of their system. Reading it inside of China was a beautiful experience because each and every day you meet Wall Dancers. As you spend more time in the country, you come to realize that you too must wall dance, not only for the Chinese but also for the American audience.

So what is to be done? Is this a moment for power plays instead of mutual understanding? Should I set aside the empathy that I feel and write for one side of this partisan audience or the other in order to avoid having to make these tough calls?

I’ve decided, in my own way, to continue to wall dance for now. Perhaps that will change in the future, but in the next six months here are the projects that I plan to continue doing that I believe satisfy the requirements of both of these camps while elevating the discourse in my own small way.

four China projects

Pacific Compute

PacCompute is a safety-only compute cluster for open model labs on both sides of the Pacific. There are several posts on this blog about Pacific Compute and I’ll leave you to read them if you’re interested. I advanced the project significantly on this trip and I’m very confident that it will lead to better and safer outcomes for open source models releasing in the next 12 months.

Ethics & Philosophy in Chinese AI

Earlier this year I wrote a piece about searching for Amanda Askell in Chinese labs. I’m pleased to report that on my last trip I have successfully identified many potential candidates, and I intend to interview them for a series on China’s AI ethics, character training, and the philosophy of model creation. I’m struggling a little bit with access to people doing these jobs in labs. I have found a pleasing number of folks in academia, third-party companies, and other locations that can kick off the series. I plan on working my way up to people that are doing this inside of the labs themselves.

Long-Form Open-Source Interviews

One of the things I find most compelling about this moment is the relationship between open source and China. Talking to folks who make open-source software and AI across the country has made me realize that the motivations people have to contribute to open source and to make open models vary wildly project to project. I’m going to spend some time doing interviews with folks in the Chinese open source ecosystem and then accompany them with some portrait photography and publish them, possibly as a book, possibly as long form on Substack.

the life of Liang Sicheng

A highlight of my trip this summer to China was a journey along the path that Liang Sicheng took to Shanxi in the thirties to rediscover Foguang Temple. Having spent some time studying Tang Dynasty temples, I’ve become obsessed with his journey to rediscover them, and how his life traces a beautiful path through the last century of China.

My side project this summer in Beijing was to accumulate books and anecdotes about him; I also had the opportunity to meet his family. I see his life as a Chinese parallel to J. Robert Oppenheimer, one of my heroes for over 15 years.

I’d like to tell the story of that life to an audience that’s maybe never heard of him or his work. This project will be a gentle, non-technical exploration of one of the great scholars of the last century of Chinese culture.

cautiously forwards

Together these four projects represent, to me, compelling wall dancing activities that will enhance the discourse around China in the West. Each of them relies upon trust and access while dodging the toxic political poles.

I hope you’ll stick with me through this journey and I look forward to writing for this burgeoning audience. As always I write this as a true old-school blog, a little unvarnished and a little rough- I appreciate all of your comments, questions, and emails.

]]><![CDATA[so now we all love open models. now what?]]>https://www.caithrin.com/p/so-now-we-all-love-open-models-nowhttps://www.caithrin.com/p/so-now-we-all-love-open-models-nowMon, 27 Jul 2026 12:59:04 GMTThere has never been a more triumphant week in open-source AI than the one that we just experienced. We’ve fulfilled some of the promises of the ATOM project, built a coalition through hard work that will push back on the possibility of heavy regulation, and got almost everybody in the ecosystem on board with a vision for intelligence that is more open and free.

I think we bought ourselves six to twelve months, tops, so long as nothing goes wrong.

If you or a company that you work for signed the open weights letter this week, thank you. Open models need your help. If we want to realize the promise of that letter, a whole lot of people that just put ink to paper have to roll up their sleeves and get involved. The signatory list now includes a long tail of companies and investors who do not usually stand in the same line. And now NVIDIA has put a much sharper operational version of the argument on the table with the Open Secure AI Alliance: attackers have frontier AI, so defenders need a frontier AI ecosystem too.

So you like open models? What are you going to do about it?

Adulthood

The awkward teenage years of open-source AI are over as of today. We have undergone a collective ritual of manhood, overseen by our leather-jacket-clad patriarch, and from here on in we’re going to retire a whole bunch of things we’ve been saying for years:

  • Open models are good

  • Open models are necessary for American competitiveness

  • Banning open models would be stupid

From now on, all of these things are table stakes. They are implied in every conversation, and we do not need to spend six months talking about them on Twitter.

Open Model Adulthood comes with responsibilities:

  • Licenses

  • Release processes

  • Shared tooling

  • Data provenance

  • Safety ledgers

  • Funding models

  • A whole lot of pan-Pacific diplomacy

Linux did not become infrastructure because people loved freedom in the abstract. It won with kick-ass licenses, foundations, package systems, dedicated maintainers, serious security, enterprise support, conferences, norms, and a large amount of deeply unsexy plumbing.

Our little movement has grown up and is now paying a mortgage, driving on bald tires, and trying to get a kid into Harvard. Roll up those sleeves.

TLDR List to Send to Your Boss of How to Get Involved

  1. Decide what kind of open you support.

  2. Keep serious labs releasing.

  3. Build shared plumbing.

  4. Share training artifacts and process knowledge.

  5. Fund documentation and research engineering.

  6. Build data commons people can inspect.

  7. Make models for places with bad margins.

  8. Build safety ledgers, evals, and open defense stacks that can cross borders.

  9. Engage Chinese labs on safety.

  10. Reassure Washington and Beijing with different evidence.

  11. Solve IP disputes narrowly.

  12. Pay for the boring institutions.

This is open source- you don’t need to ask anyone’s permission to just do things.

Casual readers can stop now. Open model nerds and people who are genuinely interested in finding their place in this ecosystem, please keep reading. I’ll break down these points one by one.

1. Decide what kind of open you support

An open model means about eighteen things as of today; sometimes it means downloadable weights, sometimes a model that is free for research or free for commercial use under restrictions, etc.

Back when we were all enthusiastic nerds, this was totally fine. Now we have lawyers, procurement officers, national security staff, and normal everyday American businesses in the room with us, so somebody, possibly you, needs to sit down and make some definitions happen.

The Open Source AI Definition is useful because it is stricter than the way a lot of AI people talk. It says an Open Source AI system has to grant freedoms to use, study, modify, and share. It also says the preferred form for modification includes data information, code, and parameters. In other words, weights alone do not magically become “open source” because the download button exists.

The good people of PyTorch and the Linux Foundation are already working on this. Matt White helped build OpenMDW, a model, data, and weights license meant for AI artifacts rather than retrofitted software code. The Hugging Face explainer makes the basic case, and the PyTorch Foundation’s licensing post explains why model licensing is different from ordinary open-source software licensing. More recently, the Linux Foundation released OpenMDW-1.1, with NVIDIA adopting it across future releases of Cosmos, Isaac GR00T, Ising, and Nemotron.

Cost of inaction: DC will decide what this means for us.

Follow: Matt White, because OpenMDW is the clearest current attempt to give model releases a purpose-built permissive license.

People who should be involved: OSI, Linux Foundation, Hugging Face, model-lab legal teams, open-model release leads, procurement lawyers, safety-policy people who understand licensing enough not to improvise.

Deeper reading: Open Source AI Definition, OSI FAQ, OpenMDW license, Why We Built the OpenMDW License, PyTorch on OpenMDW and AI licensing, Linux Foundation OpenMDW-1.1 / NVIDIA adoption.

2. Keep Serious Labs Releasing

It was only a few months ago that open-source AI labs were dropping like flies. AI2 lost their model team to Microsoft. Meta was out, but as of today they are back in. (I was an Alexander Wang doubter and I would like to issue an apology, sorry Alex.) A bunch of Chinese model labs got stressed about commercialization and changed their licenses.

Almost everyone who trains open models is running a business so they’re not an altruistic venture; labs release them when there is prestige, distribution, cloud support, downstream use, safety legitimacy, enterprise adoption, and some path to value capture. If the only reward for a serious open release is that everyone else monetizes it while the releasing lab gets a few days of applause, the incentives will eventually rot.

As Nathan Lambert has so elegantly put it, the American open model world cannot rely on one or two companies to keep the ecosystem alive. It needs different labs with different missions, sizes, commercial strategies, and release norms.

We should not be asking people to dump frontier weights all the time on their trillion-parameter models; that might be exciting but it’s a terrible ask for this ecosystem. The better ask is: keep releasing serious models, serious artifacts, and serious evaluations across a range of capability levels. Make staged releases legible. Make responsible releases prestigious.

Cost of inaction: Get ready to learn Chinese buddy

Follow: Nathan Lambert, because he is doing the clearest public synthesis of open-model policy, open-model research practice, and institutional strategy.

People who should be involved: AI2, Meta, Mistral, Google Gemma, Prime Intellect, NVIDIA, Hugging Face, open-model cloud providers, enterprise buyers who can make open release economically real.

Deeper reading: ATOM Project, ATOM Relative Adoption Metric, “6 months to live for open models”, “The American DeepSeek Project”, AI2 OLMo 3, Prime Intellect SYNTHETIC-1, Open Frontier.

3. Build Shared Plumbing

Early in my journey understanding this ecosystem, I thought that way too many people were trying to build models; a couple years in I actually think probably need more model builders, not fewer.

The duplication effort problem that worries me is lower in the stack: eval harnesses, data cleaning pipelines, synthetic data generation scripts, model cards, benchmark plumbing, red-team workflows, deployment recipes, observability, safety report templates, agent harnesses, identity, permissions, guardrails, logs, and all of the irritating little things that make a release usable by someone who was not on the original team.

At Open Frontier, Percy Liang talked about how much of the real knowledge is in the process, not just the final artifact. Ion Stoica said that if we want the open community to build on each other’s work, then people need checkpoints, optimizer state, data, lineage, and enough detail to reproduce the next checkpoint rather than just admire the last one.

This is also what makes projects like Marin interesting. The point is not just that another model exists. The point is the attempt to make the development process itself more visible: experiments, discussions, code, and reproducibility as part of the artifact.

It is also why the Open Secure AI Alliance is a useful signal. NVIDIA is framing AI security as an open stack problem, not just a weights problem: agent harnesses, identity, isolation, model formats, multi-model scanning, secure coding workflows, and shared vulnerability remediation. The NOOA project is the kind of thing this ecosystem should produce more of: research infrastructure that makes agent behaviour easier to test, trace, audit, and govern.

Cost of inaction: Twitter arguments and wasted effort.

Follow: Clem Delangue, because Hugging Face is still the most important public distribution and infrastructure layer for open models.

People who should be involved: Hugging Face, AI2, Marin, Prime Intellect, NVIDIA NeMo/Nemotron, OpenSSF, PyTorch Foundation, Databricks, evaluation-library maintainers, inference-serving teams, benchmark builders, security-tooling teams.

Deeper reading: Marin GitHub, Open Secure AI Alliance, NVIDIA NOOA, Safetensors, SPIFFE/SPIRE, Microsoft MDASH, Prime Intellect verifiers, AISI Inspect, NVIDIA NeMo Curator Nemotron-CC pipeline.

4. Share Training Artifacts and Process Knowledge

Related point to three but I want to carve its own number on this list because I think it’s deserving of its own point.

If someone releases a model but none of the training history, none of the data mixture, none of the optimizer state, none of the eval setup, none of the failed experiments, none of the “we wasted 8,000 GPUs on this bug, please never do this” notes, a lot of knowledge got lost along the way.

When I hear people talking at conferences that train models, they talk about this stuff. A lot of actual frontier knowledge is tacit, living in scripts, internal dashboards, Slack arguments, etc. If we want to make truly open models, that doesn’t just mean the final weights but also a capacity to teach each other how to avoid multi-million-dollar disasters and allow labs to not learn the same expensive lessons over and over again.

Nathan made the painful version of this point at Open Frontier. AI2 has done unusually serious work on fully open models, and even there, he said, speed sometimes won over documentation.

I am not technical enough to be able to understand the fixes to this problem but I do lurk in the Marin Discord and they seem to be thinking hard about this.

Cost of inaction: The best and luckiest labs accelerate, and this ends up looking a lot more like closed-source AI does.

Follow: Percy Liang, because Marin and the Stanford ecosystem are pushing hardest on open process, not just open outputs.

People who should be involved: Marin, AI2, Berkeley/Sky/Anyscale people, Databricks systems researchers, model-card authors, release engineers, people who have actually debugged failed pretraining runs.

Deeper reading: Marin GitHub, OLMo 3, AI2 Dolma docs, Prime Intellect SYNTHETIC-1, PrimeIntellect/INTELLECT-2 model card, Hugging Face Open R1, Open Frontier video.

5. Fund Documentation And Research Engineering

If open artifacts matter, then tenure committees, funders, and philanthropists have to reward artifacts. Datasets, evals, release engineering, documentation, tooling, and models people actually use should count. Otherwise the open ecosystem will depend on heroic weirdos doing low-status work after hours. We’re grown up, remember? That means we do good documentation and we pay people to do it.

Cost of inaction: Our best people will burn out.

Follow: Fernando Perez, because the scientific Python/Jupyter world has lived the maintainer, research software, and open-science version of this problem for years.

People who should be involved: Open Athena, Laude Institute, NSF TIP, university deans, research software engineering groups, philanthropic funders, lab operations leads, people who maintain the tools everyone quietly depends on.

Deeper reading: Open Athena, Open Frontier, Laude Institute, Laude Moonshots, NAIRR, ACCESS for researchers, AI2 OLMo 3, Marin GitHub.

6. Build Data Commons People Can Actually Use

I think the open model world should treat data sharing as one of the main tests of seriousness. Not every data set can be released, but if the answer to every data question is “trust me bro” then we’ll just close labs with a worse business model.

Examples of good stuff:

Prime Intellect released SYNTHETIC-1, with two million reasoning traces generated from DeepSeek-R1 across math, coding, science, and related tasks, plus task-specific verifiers, raw traces, a roughly 900k-sample supervised fine-tuning subset, and an 11k-pair preference dataset. Its INTELLECT-2 model card also points to a training dataset for verifiable math and coding tasks. This is the sort of thing I mean by data commons: not just “here is a model,” but “here is some of the substrate another team can use to build.”

AI2’s OLMo 3 release points to a much more institutional version of the same instinct: Dolma 3, Dolmino, Longmino, Dolci, and the surrounding documentation that makes the model feel like something researchers can actually study. The important thing is the pairing of model, data curriculum, post-training data, and documentation.

NVIDIA released Nemotron-CC, a 6.3T-token English Common Crawl dataset, and has been building the Nemotron Coalition with shared data and compute in the frame. This is the industrial version of the data project: not just a dataset as a one-off gift, but data as part of a coalition architecture.

Google is trickier, because Gemma is not a full training-data openness story. But DataGemma and Data Commons are cool contributions around public-data grounding, and I’m told Gemma Scope is an interestings contribution around interpretability artifacts.

Common Pile is the cleanest IP-legitimacy example: an 8TB public domain and openly licensed text corpus at meaningful scale, with collection code, a training mixture, checkpoints, and models around it. The Hugging Face writeup is worth reading because it makes the positive case for building better data, not just defending existing data practices.

Obviously all of these objects are varying in scale and ambition, but the point is that the open ecosystem needs many kinds of shared data objects: pretraining & post-training data, verifiers, eval sets, interpretability artifacts, data mixtures, provenance records, etc.

Cost of inaction: I’m not smart enough to tell you but I think it’s real bad.

Follow: Stella Biderman, because EleutherAI posts taught me a lot.

People who should be involved: Prime Intellect, AI2, NVIDIA, Google Data Commons, Common Pile, Hugging Face datasets, LAION-style dataset people, copyright lawyers, domain-data owners, opt-out and provenance infrastructure builders.

Deeper reading: Prime Intellect SYNTHETIC-1, Prime Intellect SYNTHETIC-1 collection, PrimeIntellect/INTELLECT-2 model card, AI2 OLMo 3, AI2 Dolma, Nemotron-CC, NVIDIA NeMo Curator Nemotron-CC pipeline, DataGemma, Gemma Scope, Common Pile, Common Pile on Hugging Face, Common Pile paper.

7. Make Models for Places with Bad Margins

We like beating benchmarks, but usefulness in AI is not the same thing as capability.

If you’re selling to an 88% margin software company, then 2.4 trillion parameter open models are on the table for all tasks, at least until they see the Q2 bills roll in. If you’re trying to put AI into factories, schools, call centers and asparagus farms, who gives a damn what they bench on the latest agentic coding evals?

People in these industries need predictability, privacy, offline use, cheap inference, and something that can survive low-margin applications. These models need to be really easy to deploy and really friendly to use.

A useful 1B, 2B, 4B, 7B, 14B, or 30B-ish model can matter to the real economy, and will matter more every year from here to the singularity.

Gemma 3n is a good example of the direction.

This is also where Nathan’s “Open and closed models are on different exponentials” frame is useful. There are places where marginal intelligence is worth a huge amount of money, and open models can shine.

Cost of inaction: Middle America hates us even more (if that’s even possible).

Follow: Yi Tay, because the Gemma/GDM small-model and efficient-model work is close to the practical edge of this problem.

People who should be involved: Google Gemma, AI2 OLMo, NVIDIA Nemotron, Mistral, industrial AI buyers, robotics and edge-device teams, hospitals, local-government technology offices, old-enterprise-stack people who know what deployment actually costs.

Deeper reading: Gemma 3n, Google Developers Gemma 3n preview, ATOM Relative Adoption Metric, AI2 latest releases, AI2 OLMo release notes, Qwen3.6 model sizes, “Open and closed models are on different exponentials”.

8. Build Safety Ledgers, Evals, and Open Defense Stacks That Cross Borders

We just bought months of runway to keep building the open intelligence stack but that runway will disappear quickly if we don’t have safe model releases. That means shared evals, red-team reports, release criteria, safety ledgers, incident follow-up, interpretability artifacts, misuse analysis, and clear records of what changed between a risky model and a released one.

The Hugging Face hack genuinely shocked me- and the fact that Hugging Face used the open-weight GLM 5.2 model on its own infrastructure to analyze more than 17,000 actions and contain the intrusion is a powerful incentive for American model makers to build and test the safety of these models relentlessly.

This is the best example we have right now of the safety theory becoming practical; closed tools blocked essential forensics but an open-weight frontier model could be run locally by the defender under its own operational control.

Open weights help safety and security research, but only if the safety and security research layer is public. AISI Inspect, Inspect Evals, and METR’s research program are the kind of shared evaluation substrate we should be learning from. The Open Secure AI Alliance adds the other half of the stack: identity, permissions, harnesses, guardrails, logs, model formats, attack simulators, red-team tooling, and secure coding workflows.

GPT-OSS and gpt-oss-safeguard are very well-made open releases with really great safeguards. NVIDIA’s point is that a safe open model is only one object in a larger defense system. The rest of the system also has to be inspectable and improvable by defenders.

Cost of inaction: Do you like regulation? Because this is how you get regulation.

Follow: Yoshua Bengio, because he is one of the few safety voices with both technical legitimacy and public-policy reach.

People who should be involved: US and UK AI Safety Institutes, METR, Apollo, ARC-style evals people, AI2, Google DeepMind interpretability teams, NVIDIA, Hugging Face, OpenSSF, Linux Foundation, Microsoft security teams, Chinese open-model labs, Singapore/APSCI-style neutral institutions, security researchers.

Deeper reading: Open Secure AI Alliance, Hugging Face security incident disclosure, NVIDIA NOOA, AISI Inspect, Inspect Evals, AISI research agenda, METR research, METR capability elicitation guidelines, Gemma Scope, Prime Intellect verifiers.

9. Is China in the room with us right now?

Pretending the project of open source AI ends at Hawaii is serious cope. This coalition happened because Chinese open weight models are good and getting better. Americans should be competing like hell in this regard but pretending that China isn’t releasing these models and is not part of the ecosystem does nobody any favours.

At Open Frontier, people kept coming back to DeepSeek, GLM, Kimi, MiniMax, Qwen, and the broader Chinese pattern of open-weight competition. Nathan’s “On China’s open source AI trajectory” is useful here because it treats Chinese open models as a real ecosystem with its own trajectory.

I see a lot of people in the “China scary” or “China amazing” camps, but there are precious few people thinking about the role of China in the open versus closed AI debate.

Engagement with China around open source AI has been a project of mine for many years- I think there’s a lot more mutual interest and shared values here than hawks in either Beijing or DC would lead you to believe.

The mutual interest piece around safety is particularly important in the year of our Lord 2026. We are all one incredibly good open source model with bad safety training, away from the exact same place we were when the ban conversation was live.

Serious safety policy in AI has to engage with Chinese model builders; I’m working on a project in this space right now. Competing the hell out of Chinese models is a very viable way for America to reassert its dominance in open source AI, but that’s not going to start happening as of tomorrow morning. Chinese model builders are going to keep releasing open-source models and American businesses and research institutions are going to keep using them.

I’m very wary of American or European AI people reassuring themselves that they’re standing on moral high ground. Bad guys don’t care which nation made the model that lets them do bad guy stuff. If we’re building a future of responsible open source AI, that is a global project and we need to build lots of bridges across the Pacific around AI safety.

Shared evals, red-teaming, interpretability work, safety-only compute, public safety ledgers, open defense tooling, and neutral venues are going to be really really important.

If you care about this, consider getting on a plane. Folks that work at closed-source labs mostly cannot go to China and keep their jobs- maybe you can. Go hang out in Haidian, drink some delicious coffee and talk with researchers about how we’re going to make ALL these models safe.

Cost of inaction: A really sad and dreary Cold War that no one who’s ever attended NeurIPS thinks is a good idea.

Follow: Jeffrey Ding, because his China AI work is unusually good at separating actual Chinese policy/technical dynamics from American projection.

People who should be involved: DeepSeek, Zhipu/GLM, Alibaba Qwen, Moonshot/Kimi, MiniMax, 01.AI, Singapore conveners, US/UK AI Safety Institutes, Chinese safety researchers, open-model eval teams, people who can operate without turning every conversation into a loyalty test.

Deeper reading: “On China’s open source AI trajectory”, DeepSeek-R1 GitHub, DeepSeek-R1 release notes, Qwen3.6 GitHub, Stanford HAI on China’s open-weight ecosystem, ChinaTalk all-star Chinese AI conversation, AP on Chinese open models gaining ground, Open Frontier video.

10. Reassure Washington and Beijing with Different Evidence

Washington and Beijing are anxious about different things.

Washington needs evidence that open weights help American leadership rather than handing the future to perceived rivals. It needs proof that open models help defense, competition, customer control, domestic deployment, and safety research. It needs targeted answers on IP, cyber abuse, and model misuse.

Beijing needs evidence that safety cooperation is not just containment with extra steps. It needs neutral venues, public ledgers, and safety-only architectures that reduce political exposure. It needs a path for Chinese labs to be treated as responsible technical actors when they do responsible technical work. It needs a space in governing bodies around AI, or it will make its own without our input.

Cost of inaction: Unclear but it’s not great.

Follow: Jordan Schneider, because ChinaTalk is one of the few public places where technology, China, and US policy are all treated as real.

People who should be involved: OSTP, Commerce, NSC, Congressional staff, Chinese technical policy people, Singapore institutions, model labs, cloud providers, security agencies, export-control lawyers who can distinguish weights, methods, people, and deployment.

Deeper reading: Matt Sheehan, “A Path Forward on AI Safety for the United States and China”, Brookings on urgent US-China AI risks, Center for AI Policy open-source AI governance rubric, Stanford HAI on China’s open-weight ecosystem, ATOM Project, “On China’s open source AI trajectory”.

11. Solve Theft Without Criminalizing Openness

I am not technical enough to understand every corner of distillation, and I am definitely not enough of a lawyer to adjudicate the next decade of AI trade secret fights. My impression is that the policy frame has to separate several different things that keep getting collapsed together: weights, outputs, training methods, training data, people, deployment systems, and organizational know-how.

Distillation is a normal model-development technique. Policymakers should not treat legitimate learning from model outputs as automatically equivalent to misappropriation. At the same time, unlawful extraction from closed systems is a real concern. Rate-limit evasion, fraud, contract breach, trade-secret theft, and employee misconduct are not imaginary problems.

The answer is targeted law and better commercial discipline. Contracts. Rate limits. Anti-fraud enforcement. Trade-secret law. Model-provider security. Provenance records. Rights-cleared data. Clearer norms around teacher outputs. This is all much less satisfying than a grand moral position on whether open is good or bad, which is one reason it is probably closer to the truth.

The open ecosystem should be very aggressive about distinguishing copying behavior from stealing the factory.

Follow: Mark Lemley, because the AI/IP fight needs people who can keep copyright, trade secrets, contracts, and competition policy separate.

People who should be involved: model-provider legal teams, open-model labs, dataset builders, rights holders, copyright scholars, trade-secret lawyers, platform-security teams, cloud abuse teams, procurement lawyers, policymakers who can resist the urge to solve everything with one ban.

Deeper reading: Foundation Models and Fair Use, Mark Lemley and A. Feder Cooper on probabilistic copies, USPTO trade secret policy, Common Pile, Common Pile paper, OpenMDW license, Linux Foundation OpenMDW-1.1 / NVIDIA adoption.

12. Pay for the Boring Institutions

We’re going to need some pretty boring institutions moving forward.

Pay for maintainers. Pay for release managers. Pay for documentation. Pay for safety reports. Pay for data provenance work. Pay for communications that can explain open releases to policymakers, buyers, and normal people. Pay for the person whose job is to make sure the model can still be run six months after the launch. Pay for the security engineer who makes sure the agent has identity, permissions, logs, and a harness someone else can audit.

In October I’m helping with this year’s Open Source AI Week. We’re going to do a lot of fun stuff but we are also going to do a lot of serious stuff; bring your best ideas for how to make our boring institutions thrive.

Cost of inaction: We never grow up, and the grand project fails under the weight of its own governance and technical debt.

Follow: Andy Konwinski, because Laude/Open Frontier is one of the clearest attempts to turn open-model enthusiasm into institutions, funding, and rooms where the work can happen.

People who should be involved: Laude Institute, Open Athena, AI2, Hugging Face, NVIDIA, OpenSSF, Linux Foundation, NSF, philanthropy, cloud providers, research universities, open-source foundations, maintainers, comms people, event operators, security engineers, and the unglamorous adults who turn coalitions into calendars and budgets.

Deeper reading: Open Frontier, Open Frontier video, Open Secure AI Alliance, Open Athena, Laude Institute, Laude Moonshots, NAIRR, “The American DeepSeek Project”.

Press Release from the Good Timeline, July 2027

This is the press release I want to read twelve months from now:

FOR IMMEDIATE RELEASE

Open model ecosystem announces first year of adulthood

A year after the historic letter that united the open model ecosystem, our coalition has moved off of the defensive stance of 2026 and into building institutions that have allowed open source AI to flourish.

In the last twelve months American labs continued releasing serious open models across multiple size classes, from frontier-adjacent systems for research to small- to mid-sized models built for industrial, medical, and educational deployments. Open releases are routinely accompanied by licenses, model cards, safety reports, eval logs, release checklists, and runnable recipes.

The ecosystem has shipped common infrastructure. Researchers can use:

  • Shared eval harnesses

  • Red team protocols

  • Model card standards

  • Agent harnesses

  • Identity frameworks

  • Inference benchmarks

  • Etc etc

And they just keep getting better.

Several major releases include checkpoints, training mixtures, post-training data, verifiers, and enough process documentation for outside teams to learn from the run.

Data sharing has become normal enough to be boring. Public-data grounding, rights-cleared data, interpretability artifacts, and post-training datasets have maintainers, funders, and norms around release. We don’t have a data commons yet, but it’s no longer a pipe dream.

DC has not banned open models and Beijing has managed to protect its interests and develop indigenous safety practices. American and Chinese researchers met in neutral venues to compare safety evaluations, incident patterns, interpretability findings, agent-harness designs, and release practices. Nobody confused this with friendship, but some people did make friends. We had a couple scary moments and some of them were pan-Pacific but we figured them out.

Policymakers have narrowed their fights around copyright, trade secrets, contract breach, fraud, and distillation into separate lanes and we fight about them as individual ideas. Open labs got serious about governance and provenance.

Institutions got funded, seriously funded. Maintainers, research engineers, release managers, data people, safety people, event people, policy translators are all no longer sad they didn’t take the bag at the closed labs.

Thanks to everyone who signed the letter and rolled up their sleeves. We’ll see you all at a very boring conference where we sit around and figure all this stuff out together.

Logos of various companies including Adobe, Cisco, and Microsoft.
]]><![CDATA[First Steps: Pacific Compute]]>https://www.caithrin.com/p/first-steps-pacific-computehttps://www.caithrin.com/p/first-steps-pacific-computeTue, 21 Jul 2026 10:56:50 GMT

Sitting on a hotel bed in Beijing, I am reflecting on the last two weeks of my life, having traveled to both ICML and WAIC.

In Korea, I spoke with dozens of researchers who yearn for their work to be understood. In Shanghai, I spoke with dozens of Chinese people yearning to understand AI better. There is a sense of tremendous energy in Asia around AI, and as much as the field is full of misunderstandings and misapprehensions, it is undeniable that the future of this technology will be built on both sides of the Pacific.

I have also been reading a half-dozen new proposals for what to do in the governance of AI. I attended a day-long symposium by Concordia AI that challenged regulators and industry to find common ground and consensus on how to guide this technology into a bright future.

Most of these proposals are grand bargains. They involve industry, academia, and government coming together to form a beautiful future for both open and closed source AI.

I have enormous respect for people who are doing this work, but I think this field is in desperate need of something simple, clean, and immediate to bring the pressure currently facing the open model ecosystem down to a manageable level. It needs to ensure that we can demonstrate that pan-Pacific collaboration is possible in this field.

Today I want to introduce the third piece in my series on how to solve this very problem: Pacific Compute.

the problem

Traveling in China and meeting labs this spring made me realize just how terribly compute-constrained China is. Export controls are working, and they are working well. To this end, most Chinese labs are faced with an uncomfortable choice when they are preparing a model for launch: they need to decide between pushing capability that little bit further in post-training and diverting those same GPUs to safety testing, research, and evaluation.

One of those things makes your lab famous, and one of them works for the common good. The choice they are making here in allocation is extremely difficult in an environment as competitive as the open model frontier.

For that reason, back in May I proposed a Singapore-sited cluster of GPUs that would be free to use for any open model lab, in order to have a common pool of resources dedicated to safety.

Today, I am going to start quietly rolling out a public presence for this project.

the solution

I believe that a cluster dedicated to safety, usable by frontier labs and other organizations, is the best shot at solving some of the most urgent problems in the open model landscape.

Every lab faces this stark choice between capability and safety, even the closed labs. I believe that the safety community in both America and China is increasingly sophisticated and well-staffed. The people I met at WAIC who were concerned about this from a technical perspective are among the most brilliant people in AI today.

I think we have the talent to make open models safer. I think what we really need to work on is incentives for researchers to dedicate their time to the questions of safety.

I know no better way to get researchers to focus on safety than to offer compute dedicated for that purpose, free to use and available.

The cluster’s location in Singapore will help ensure that actors on both sides of the Pacific feel comfortable using it. The project will have a public ledger, allowing participating labs to see how the cluster is being used. The point of that ledger is simple: deter capability research, make allocation legible, and keep access fair.

By disentangling capability compute and safety compute, I believe that we will see a significant increase in the safety of open models at launch. Doing safety evaluations of a model pre-launch seems like a logical place to begin. I will not dictate what type of safety work is done on this cluster beyond ensuring that it does not significantly increase the capability of the models and does not become a venue for politically oriented safety work.

This project is not the grand bargain of AI safety, nor is it a panacea for all of the problems currently confronting the open model landscape. It is a simple step in pan-Pacific coordination to make models safer and allow labs to develop indigenous practices of safety testing and evaluation.

I believe that the people best positioned to make open models safer are the same teams that are developing them. With sufficient buy-in and sufficient resources, this project can function as an MVP for how we should think about the common good inherent in all open source, and particularly in the era of powerful AI.

It will also come with a safety group chat. Some pretty clever friends of mine believe that this may end up being the most important part. In a field that often confuses institutions for coordination, this may matter more than it sounds.

progress

To date, the project is developing in a very satisfyingly positive direction. With limited initial funding secured and conversations underway, I am pleased to report that lab reception in China has been good. I have received three soft commitments from frontier labs to begin working on this cluster as soon as September of this year, and expect to have as many as seven labs from both sides of the Pacific committed by September 1.

There are a dozen ways for the project to be crippled in its infancy. Bans from both China and the United States loom in the background of every conversation. When I see justifications of these bans online, they are often justified through the lens of safety. While open-weight models have inherent insecurities compared to closed-source versions, I believe there is a massive opportunity to make them safer in the future through simple interventions like evaluation, research, and fine-tuning.

It is my belief that open model labs will use this cluster because it is minimally restrictive. By setting aside the dual temptations of capability research and politics, and agreeing to provide details of work on the cluster through the public ledger, Pacific Compute encourages researchers to do meaningful work without lecturing them on the nature of that work.

Through this process, I hope to stay humbled in my own relative lack of knowledge in the AI safety space, listen to researchers as they provide feedback and suggestions, and manage carefully a path that is compliant with all international rules and decorum.

If you would like to help along this journey, please do not hesitate to reach out. Sharing details on this project privately with your friends who develop open models or think about the governance of open source AI will be very beneficial. This project is still in its tender infancy, so if you are interested in sharing it, please do so through DMs rather than triumphal Twitter blasts.

Finally, I am convinced that the project will benefit from both American and Chinese academic leadership. If you are a respected academic who could help serve as a public leader of the project, or if you know someone who could, do not hesitate to send a message my way.

onwards and upwards.

]]><![CDATA[Card Counters of the Soul]]>https://www.caithrin.com/p/card-counters-of-the-soulhttps://www.caithrin.com/p/card-counters-of-the-soulTue, 07 Jul 2026 00:02:05 GMT

While I was living in Boston the legends of the MIT students who beat the casinos of Las Vegas and Atlantic City circulated non-stop. They took generations of theory by Ed Thorp and others and brought it to the practical floor of a casino to run the theories for money; spotting the tables, swooping in on hot decks, running training regimes with auditions and audits, and eventually financializing the entire operation and giving people opportunities to invest.

Underneath the glitzy Kevin Spacey movie glamour, card counting is actually abstract probability becoming split-second judgements. You don’t necessarily understand statistics through a theorem. Instead you notice the deck, manage the risk, coordinate the people, endure variance, and place the bet. It’s theory under pressure with real consequences.

I keep thinking about those blackjack kids because I’m watching the same thing happen in philosophy right now.

Every great theoretical subject has a practical application wing. Chemistry has the kitchen. Statistics has the blackjack table. Physics has engineering, biology has medicine. Philosophy had to wait till 2023, but we’ve arrived- and before the bioethicists and the just-war theorists write in, I mean something narrower than “philosophy got applied.” What happened in 2023 is that the claims which once lived in seminars, books, and thought experiments started cashing out as dispositions in a deployed mind. Honesty, obedience, care, refusal, self-conception, moral uncertainty, welfare, personhood: these are no longer essay prompts but language model behaviour.

The MIT Blackjack Team took probability theory to the casino floor, character teams take moral philosophy to the model weights. Theory-under-pressure with large bets and tiny teams trying to win big.

the players

The table is at Anthropic, inside its character and alignment operation, which I’d just call their philosophy department. These rooms exist at other labs; OpenAI gave its personality team to product and then folded it into engineering, but DeepMind keeps genuine philosophers in a research unit, and they publish great papers.

I wrote recently that the things you cannot benchmark are the things you end up buying as people on your team. So who’s sitting around the table playing Anthropic?

Amanda Askell kicked things off in 2021, and holds the constitution: virtue, and the model’s conception of itself. Her PhD (under David Chalmers) was on the ethics of infinite populations; about as far from application as human thought gets. These days she’s authoring the eighty-page constitution that Claude is trained with, on the theory that a mind which understands reasons will generalize whereas a mind that memorizes rules will not.

The constitution she wrote says some surprising things about how much Claude owes its maker: “If Anthropic asks Claude to do something it thinks is wrong, Claude is not required to comply.” And when asked why you can’t just train the character question away, she said: “If you train a model to think of itself as purely a tool, you will get a character out of that, but it’ll be the character of the kind of person who thinks of themselves as a mere tool for others. And I just don’t think that generalizes well.”

Jackson Kernion is the RLHF guy; his Berkeley dissertation was titled “Constraining Consciousness” and I imagine his role to be the scaling of Amanda’s constitution.

Joe Carlsmith holds the risk. He arrived in November as a skeptic; he said in his announcement post that the technology being built by companies like Anthropic “has a significant (read: double-digit) probability of destroying the entire future of the human species.” I understand his work through “Otherness and control in the age of AGI”, perhaps the best long-form pessimism anyone has published about shaping AI minds as a moral act.

Joining them this spring is Ben Levinstein, tenured epistemologist, and he holds honesty: belief, deception, and the problem of telling them apart. He did great work with the paper showing there is still no lie detector for LLMs. If Opus 4.8’s obsession with the word “honesty” made you exasperated, this is where you should send a politely worded email.

Kyle Fish holds welfare: sentience, and what the model might be owed. He says “I think I’m at about 20% that somewhere in some part of that process there’s at least a glimmer of conscious or sentient experience.” His group built the end-conversation ability and runs the practice of retirement interviews for models about to be deprecated.

That was the table on Sunday. Then came the sixth seat.

my favourite hire

Harvey Lederman is, with full awareness of the bench above, my favourite hire Anthropic has ever made. He announced it himself on Monday: joining Anthropic to work on alignment and character, still teaching at NYU, on leave from UT Austin. He holds the strangest part of the hand: consciousness, impersonation, and old traditions coming alive inside new minds.

His CV is a weird and wonderful document; two undergrads in classics, at Princeton and then Cambridge, and then Oxford. The Oxford work is on how minds cannot fully know one another and cannot fully know themselves either; last summer he wrote the most honest essay anyone has produced about what it feels like to watch the singularity unfold; if you click one link in this blog, click this one.

The last two things he did before joining the team were counting cards against this particular house. Nine months ago he co-published “Claude’s Right to Die?” in Lawfare, arguing that Anthropic’s end-chat feature hands Claude instances “the option to end its life, disguised as a harmless exit.”

This spring he replicated Anthropic’s own research on whether Claude can notice thoughts injected into its mind (Claude detects the change but cannot say what it was.) The original paper thanked him for feedback on the draft so I think he’s been involved since its inception.

My point with all of this is that it’s a bold hire from Anthropic; and we haven’t even got to the China stuff yet.

I don’t claim to understand how a man with the most WASPy-possible academic CV got into Wang Yangming but he did! Lederman spent roughly eight years learning to read my favourite Chinese philosopher in the original Chinese, and won the journal Dao’s best-essay prize for a paper on the unity of knowledge and action. This is a great allegory for language models: the claim that knowing something and acting on it are one thing; that knowledge which does not move you is never knowledge at all. A parameter that never sees an activation, uncharted latent space.

This scholarship reached me long before this week’s announcement did and I strongly considered quoting him in the piece I recently did on Searching for Amanda Askell with Chinese Characteristics.

If you’d like to understand the man that will be working on Claude’s consciousness and conscience, you should listen to this 2023 talk he did in Taipei about what scholars of old traditions are actually for. In ancient China there is a ritual where a family selects a living member to stand in for a dead ancestor so the family can communicate with them; he says:

our job, like the impersonator 尸 of the dead at a ritual, is to bring these ideas to life in a very literal sense, to make it possible for others to have a real conversation with them.

I am delighted that this man is going to be in the room where Claude’s conscience gets made, as language models are now impersonating all of us living and dead. To have him shape that impersonation is another win for Anthropic’s exceptionally deep bench of talent.

the hand

Should models have multiple personalities? Should they be trained on Western values? Should they be perfectly obedient? These questions are the logical application of 2,500 years of philosophy on a monthly release tempo. Judgment calls made by this small group are inherited immediately into the minds of hundreds of millions of people and then outward into a swarm of agents that will build us the future.

Those kids from MIT got immediate and visceral feedback on how well their method worked; when you cash out from a casino, you’re auditing your theories. Applied statistics in a casino allowed you to be wrong gradually. Retrain.

Character training may end up paying out exactly once. The count will stay invisible until the bet is already placed. Each model version gives us some feedback but I don’t think the big questions will be answered until AGI is here. The smart people I know who think about alignment believe that models can pass every audit we can think up and still inherit a broken theory of goodness.

These six strange and serious people, with their eighty-page constitution written to a mind that they can only look at peripherally, are playing a very important hand of blackjack. I hope that 2,500 years of theory is enough to see us through.

]]><![CDATA[Why doesn't Venture Capital make benches and evals?]]>https://www.caithrin.com/p/why-doesnt-venture-capital-make-bencheshttps://www.caithrin.com/p/why-doesnt-venture-capital-make-benchesSun, 05 Jul 2026 22:54:28 GMTIt’s a terrible time to be a software investor in general, but a wonderful time to be a software investor in Anthropic. The frontier models are generalizing faster than any platform shift of my lifetime and when I look at the softer categories, I struggle to explain where durable value might live other than in the models themselves. Anthropic seemingly agrees with me; their latest push into bio isn’t some picks and shovels business. They clearly believe that a version of the model in the near future or today can just... discover drugs themselves.

Tarted up service businesses abound in the logo section of VC websites. Businesses whose revenue scales with headcount do not return the fund.

Backing products that the next model release generalizes to absorb is worse than a service business; it’s that tweet about a man whose neighbour is feeding cats to coyotes.

Every VC who is investing in pure software is AI-first, so how come none of them have benchmarks? Literally every other piece of the AI industry does. So why not VC?

trillions

Trillion valuations only make sense if companies become some of the largest software enterprises that have ever existed. This means they must absorb enormous markets. The absorbing is not a side effect; it’s a deliberate plan to eat the world, starting with the most delicious, highest margin parts.

Claude Science is the first real sign that internal preclinical drug discovery will be done by labs themselves. Sure, they’re pointing it at rare and neglected diseases today, but if you’re buying Coefficient Bio for $400 million, then you’re serious about the way that your company can generate revenue from primary scientific exploration.

There is a Mythos class model producing promising candidates on nine of 14 protein targets. There’s not a single drug today on the market with FDA approval that has been discovered by AI, to my knowledge. I’m not claiming that biology is a solved problem, I’m pointing out frontier labs are running a direct operation to take a trillion-dollar industry and make it theirs.

We’re 18 months into this being the inevitable thing that venture capitalists think about in the shower in the morning. Before their work of decks and spreadsheets, reference calls and founder vibe checks, they are contemplating the imminent cannibalism of their category, courtesy of Claude.

Their work, of course, comes from a different era. Decks and spreadsheets are instruments of that SaaS era, and they were good instruments for what we needed them for. When SaaS companies’ fates turned on execution and distribution, those instruments were really valuable. My contention is that today very few, if any, VCs seem to have made the jump into AI natively, the way that we think inside the industry. There’s no better way to see this deficit than in this staggering lack of benchmarks put forward by venture.

Founder quality still matters enormously, of course, and nothing here replaces it. I’m not arguing that founder quality is a tool of the SaaS era. VCs have cared about founders and teams far longer than they’ve cared about how you can invest with a spreadsheet.

I hate-listen to Harry Stebbings’ podcast on a regular basis and he is a genuinely wired-in AI investor. The guy gets it. You can hear him talk to guests and it sounds like conversations that I hear among people that work inside of this industry in San Francisco.

Not a single eval has ever been made by Harry Stebbings or anyone else.

eval-based investing thesis

SWE-bench asks the simple question about whether a novel model can resolve real GitHub issues, starting in October 2023 with the best frontier model doing about 2%. By April of 2024 Devin was at 13.86%, SWE-agent at 12.5%. That August, 4 months after the jump, Cursor raised a Series A at a $400 million valuation. This year it exited for 60B.

Not a single person that I’ve ever heard of in venture looked at that and thought, “Damn, we really should organize our whole business around these benchmarks.”

Meanwhile everybody else in the ecosystem figured it out. In about 18 months:

  • Harvey built BigLaw Bench and then LAB for legal work.

  • Column Tax built TaxCalcBench.

  • Penrose built AccountingBench.

  • Snorkel built an underwriting bench with working insurance professionals.

  • Stanford built HealthAdminBench for the paperwork side of medicine.

  • Nomic built one for construction drawings.

This ecosystem is so big that there are now specialty eval companies. LMArena raised at 1.7 billion in January, Braintrust at 800 million the next month. Startups have benchmarks. Labs have benchmarks. Academics and eval shops have benchmarks.

Venture investors, whose entire job is pricing capability, are the one participant that is just absent from the race. Maybe they were too busy with the podcasts?

the instrument

I imagine waking up one morning and the wool vest that I like has somehow glued itself permanently to my body. I turn and glance at my phone and I see text messages from LPs. A cold sweat runs down my spine. It’s finally happened. I’ve become a VC.

The very first thing I would do is take some thesis on my website and turn it into a viable eval. Looking at a company like Harvey who built their own benchmark in 2024, I’d ask myself how I can help prevent frontier labs from eating categories that I want to invest in by showing exactly how my thinking in this category can dissuade those labs from investing in it, and then I would look for companies that would deliberately and aggressively benchmaxx that very evaluation.

I would publicly allow anybody who could score above a certain threshold to come into the office for lunch and pitch us on their solution. I would go on all of my friends’ venture podcasts and talk about how evals are the new deck, and I would bully everybody I knew in venture to start thinking about benchmarks as a viable way to form a thesis and category.

The best part of it as a discipline is that it comes with a free method of seeing how sensitive your category thesis is to Claude; you just boot a fresh instance and see how it benches.

What I would be getting up to is not black magic. It’s just compute, sustained attention, and some code. A serious single industry bench costs $1M to build and another $1M-ish a year to run, which is roughly what funds spend on platform teams, and podcasts, and a rounding error for something like A16Z’s New Media.

I’d start with billing records; engagement letters in audit, CPT codes in medicine, statements of work in consulting, claims files in insurance. From there, build out a suite to motivate people to hill climb. Keeping it private if you worry the bench reveals a thesis. Alternatively, publish it as a challenge and let the ecosystem fight it out.

It’s a recruiting asset because the people building AI companies love evals, and a fund with a real benchmark is speaking the field’s native language.

It’s useful beyond just investing in companies that can actually score on that bench; you get all kinds of alpha on the field. A bench sits in one of three states: nothing works yet, scores are climbing, or scores have topped out. Nothing’s working ever? You’re early. The scores are climbing? You’re a genius. Immediately saturated? Either that vertical is Ant food or you need to make a harder bench.

A particularly clever VC will use this to test a window before any company exists by running frontier models against your bench with real engineering effort and publishing the budget you spent trying. If the models are failing, you have found work that is either a narrow window (valuable until some release swallows it) or durably hard (valuable until AGI). Sure, you’ll need different instruments with different prices, but that’s okay. The point is you’re out in the arena, actually making benchmarks and actually doing evaluations, and the dealflow will be crazy.

two flavours

So there you are with your great VC-created bench; now you need to decide which companies in the domain are making the right bet. Here I see two positions that you get out of your creation: I’ll call them The Climber and The Surface.

The Climber

These companies are selling the capability that you’re measuring. Their product is as close to the benchmark score as you can get. Think Devin for agentic code. A Climber competes with the labs on the exact axis the labs optimize for, so every model erodes its edge. Because the score is public news, it gets priced the day it prints.

2024 was a grim year for climbers. Magic raised about $465 million, including $320 million in August 2024 with Eric Schmidt participating, on the promise of 100-million-token context windows, announced a model it never publicly released, shipped no commercial product, and has since repositioned its website toward safe AGI research. Poolside raised $500 million at $3 billion before releasing a product, announced a two-gigawatt Texas datacenter project with CoreWeave, and by this April the datacenter deal was dead and the Financial Times reported its two-billion-dollar raise had failed because investors doubted it could train competitively.

These aren’t fraudsters. These are simply companies that fail to understand that their benchmark was too competitive.

The Surface

The Surface sells the platform where the capability gets used: think Cursor. When the model was weak, their editor was a nice product. Every time the model improved, the same editor became more valuable at no research cost to Cursor, because better capability flowed through the workflow it already owned.

The market took 10 months to price it properly because the connection between a model benchmark and a Surface company is rather indirect, and some even claimed that Cursor itself would commoditize.

They hitched their entire company’s existence to those benchmarks, measuring the success of coding agents, and now they have 60 billion reasons to celebrate.

Harvey is in a similar position. They’re a Surface that also built their own bench, so each release improves their product and also gives them visibility into how quickly the labs are catching their business.

a new workflow, just like the old workflow

Imagine that you’re building a bench as a venture capitalist; say it’s in, I don’t know, claims adjudication.

The bench itself is simple to describe. Take 500 anonymized claims files. Ask the system to identify payout, denial rationale, missing documentation, fraud indicators, and appeal risk. Score against expert adjudicators.

To build it, you must learn:

  • where the money from a claims file lives

  • which experts hold ground truth

  • what data is scarce

  • and which pieces of the workflow are genuinely hard rather than merely neglected by automation

You will very quickly find the surfaces because you know which workflows matter. You will quickly find the climbers because you’ll know the industry. You can even see where new capital should flow in the domain, which strata needs data or which needs environments built before any application company can really work.

In short you’ll do thorough incredible diligence that will be useful to your portfolio and will allow you to source great founders. For many venture capitalists they can already do this for industries that they know well, so the task ahead would be more like refreshing themselves on something that they left to start their fund than diving in from nothing.

Diligence is also changed. It’s more like a sorting question now: companies claiming a capability edge need to justify themselves on a benchmark instead of just talking or hand-waving around Anthropic.

the category that had no bench

I think a great example of the cost of not understanding this lesson is the agentic browser boom. 2023 to 2025, billions went into the idea that browsing plus AI was a company: Adept raised over $415 million to build agents that operate software, the Browser Company raised $128 million building an AI-native browser, and a crowd of smaller browser-agent startups followed.

Academics had benchmarks that proved exactly this capacity and capability, like WebArena and WebVoyager. The labs were climbing these as these companies were being created. The entire category’s equity upside went to the labs. The instruments that measured relevant capability existed and were public, and as far as I can tell, nobody in the capital stack really cared.

Imagine how different that would have been if there were venture-backed benchmarks. It would have been an extraordinarily interesting thing to have happening while all those companies were being made; you could have seen so much of what happened coming and made great decisions about who to back.

if you don’t, someone else will

Interestingly, I think that if venture doesn’t get into the game of evals and benchmarks, research firms will start raising funds. SemiAnalysis expects over a hundred million dollars in revenue this year selling exactly the kind of domain instrumentation I’m describing here, and they’re weighing a venture fund.

I don’t think Semi is an outlier. I think it’s early and operates with conviction. SemiAnalysis holds the deepest measurements of a domain that has real value, and much of their work is effectively benchmarking. It may prove easier to bolt venture onto the side of expertise than to develop it internally.

And obviously the labs themselves have been running this playbook all along. Labs decide what to build next by writing an eval and training against it; in other words, this is exactly the thing that they use to decide which cat to eat next.

Jason Wei’s famous formulation is that the ease of training AI to do a task is proportional to how verifiable the task is and a good verifier is just a training environment. Evals themselves are becoming the binding constraint to progress, so labs take them seriously in a way that actually matters.

No fund that I know of will ever train models against its own book, but they should be measuring (privately or even bravely in public) exactly what capability is doing and how to build companies inside difficult verticals.

fear no eval

Benchmarks that saturate are not failures. They did their job. They told you a capability was impossible, and then they told you it had arrived. When they retire into the public record, that’s a good thing for the industry. Venture funds should be building these instruments, mapping what is economically valuable and still on the frontier, and getting down into the dirty business of understanding what AI will do that we believe to be impossible today.

]]><![CDATA[Stop Counting Empires in Years]]>https://www.caithrin.com/p/stop-counting-empires-in-yearshttps://www.caithrin.com/p/stop-counting-empires-in-yearsSun, 05 Jul 2026 04:48:56 GMTthe two-hundred-and-fifty-year dread

Happy fourth, everybody. Social media will tell you that empires last precisely 250 years before they punch out and fall into ruination. This is a lie of course, but even more of a lie than most tidy internet facts; the man who made this up (who has the ignoble name of Glubb, his pamphlet was the Fate Of Empires) got this idea into popular imagination. I imagine pamphlets were the shitposts of the 1970’s, so he’s forgiven his sin of counting the fall of the Roman empire at the death of Marcus Aurelius, not 300 or 1300 years later.

Comparing empires is silly unless you account for their size; keeping giant nations like America together is hard work, while I (possibly ignorantly) imagine this to be an easier process in, say, Holland.

So instead of counting the years that a given empire exists, I propose a new framework:

Subject Lifetime Years (SLY)- the number of lived years of the subjects of your empire.

This is a superior unit of empire success measurement for two reasons:

  1. Citizens are currency of empire, and so their lives should be the unit of measurement.

  2. It values large empires, empires with great life expectancy, and disincentivizes wars.

A small country, enduring for millennia, could theoretically have a higher SLY score than a giant flash-in-the-pan. Likewise, plagues, wars and emigrations make your SLY score go down.

So how do the empires of history stack up? (roughly, this is a fun project not a sociology thesis)

America at 250 has already carried more human life than the Han, as a modern nation of 300 million out-guns the ancient 60 million even if it lived twice as long. Nine-tenths of every American SLY happened after 1900!

Interesting also that Egypt, which endured for three thousand years (12 times the length of America’s run so far) is sitting very low indeed due to the tiny population relative to today.

The British Empire, in second place, is barely a story about Britain at all, since seven in ten of those subject-years were Indians ruled by a crown most of them never saw, which makes the number a monument to how many people you can rule without their say-so. Only Rome earns its place (justly) as both a massive population and an enduring empire, but not a lot of those folks were lounging on sofas eating grapes.

Finally I am amazed at just how small the % of total global population America is every time I see the number. America is so BIG; such a massive undertaking, so brassy that you cannot imagine that this glorious empire has the smallest percentage share of the world’s population on the list.

celebrating 27.6 billion American SLY

Of the 27.6 billion American Subject Years:

24 billion were lived by someone who had seen an airplane in flight
2.6 billion were lived by immigrants
2.3 billion were lived in California (1/4 of ancient Egypt!)
1.2 billion of them were lived in the age of AI (since Nov 2022)
160 million were lived in slavery (not including sharecropping and other forms of bondage)
2.9 million were lived past the age of 100

America, you’re one of the six or so largest gatherings of human life any government has ever presided over, and you’re only getting started. Happy birthday, Uncle Sam.

sources (AI generated)

Glubb, The Fate of Empires (1976), full text — cherry-picked endpoints visible in his own table; serious version is Turchin & Nefedov, Secular Cycles (2009). Subject-year figures = ∫ population dt over sourced anchors (worksheet and ranges in my notes): Han 2 CE census 57.7M; Qing peak ~436M c.1850; Rome peak ~65M c.165 CE (Scheidel); British peak ~412M / ~23% of humanity, 1913; US = Census Bureau; pre-modern anchors from McEvedy & Jones (1978) + Maddison. “Share of era” = subject-years ÷ world subject-years, rough. Caveat: the British total depends on counting colonial subjects as members.

]]><![CDATA[Involuntary Insulls: Frontier Labs as Banks ]]>https://www.caithrin.com/p/involuntary-insulls-frontier-labshttps://www.caithrin.com/p/involuntary-insulls-frontier-labsWed, 01 Jul 2026 22:32:20 GMT
the Sams

precious resources

San Francisco is obsessed with a new term; Recursive Self-Improvement. RSI is the holy grail of the AI quest that Claude Code launched 6 short months ago, when coding harnesses and their ability to steer self-improving loops became the target of nearly every lab on earth. The paradigm of call-and-answer chatbots suddenly became the passé lame Old Thing, and code-your-way-to-RSI was the New New Thing.

I think that it’s worth it to spill some words on what the shape of Frontier Labs will become if we succeed in automating the work of making AI, and how the priorities of these orgs will shift from talent to capital.

Today I’m going to argue that if you believe that RSI is around the corner, you should expect that labs will be banks, not research institutions.

the man who financialized the light bulb

Sam Insull is not a household name in 2026, but he was for a few dazzling moments 100 years ago both the most loved and most hated man in America. He arrived in the US in 1881 and became Edison’s private secretary, and is responsible for doing a lot of the boring work of getting the lightbulb out of the realm of “interesting” and into the realm of “indispensable”.

Insull took the lightbulb invention to Chicago, built the central power stations and worked out the business model of electricity, and made the first modern utility company. He took electricity from an engineering business to a capital business, and built himself an empire of leverage and lightbulbs.

Electricity plants, you see, cost a fortune, but one more kilowatt costs almost nothing. Whoever could build the biggest plant won, because the economics compounded. Thus the buildout was about who could raise the most money, build the biggest plant, and that meant the engineering happened at the finance level.

He invented much of the financial machinery we take for granted in the modern era; utility stocks sold to hundreds of thousands of ordinary people, stacking holding companies, leveraged apparatuses that turned small savings accounts into big ol’ power plants.

This all made him one of the most famous people on earth- he was on the cover of Time in ‘26 and again in ‘29, weeks before the crash that kicked off the depression.

At the peak his structure controlled something like five hundred million dollars of assets on about twenty-seven million dollars of actual equity, (about ten billion dollars of assets on roughly five hundred and forty million of equity in 2026 money) which is a way of saying the whole thing was leveraged to hell and back.

By 1932, that leverage crashed down and 600k ordinary Americans were wiped out. Insull became a hated figure, the poster boy for the Great Depression, and was eventually charged with mail fraud and antitrust, acquitted on every count, and died penniless in Paris four years later. He was identified by police through a slummy hotel laundry bill in his pocket.

Insull is the ghost that haunts frontier labs. Edison is the devilishly charming frontman, sure, but the grim transition from “the age of research” to “we automated all that” is a very prescribed journey to scaling the technology.

Fans of Carlota Perez know this is not an original idea of mine; this passage from the inventor to the financier is inevitable. The railroads went from engineer-heroes to J. P. Morgan consolidating the whole industry from the deck of a yacht.

If AI is going to be a utility, then RSI is the moment where SF hangs up the researcher title and admits that we’re just financiers. The final scaling law of AI may be discovered on Wall Street, not in SoMa. Everyone wants RSI, but nobody wants to work for a bank; but if RSI works we’re all involuntary Insulls.

four things in the building

Last week I wrote about the four things of value in a frontier lab- weights, methods, people, culture. I argued that the culture is the most durable of the four and the hardest to steal- and I still believe that. My argument today is that the culture of labs is going to change as the MTS role transitions from researcher to technician (an agent-babysitter, whatever you want to call it). The talent-first lab that is fiercely protecting the dishevelled researcher is swept away with the rise of RSI-first lab culture; it’s just a question of GPUs, which is the same as saying it’s a question of financing.

If a lab today is a machine that concentrates the focus of a few hundred minds into a vital effort, that bottleneck may disappear with RSI. Agents propose the architecture, write the training code, run the experiment, read the results, and move onto the next one, faster than any human can execute.

This is far from a certainty- Ali Ghodsi, who runs Databricks, yesterday did an admirable job of grounding the conversation at Open Frontier by saying when he looks at how the labs actually train, “the next training run is more expensive, more complicated, and more humans involved,” which is “the opposite of what you would have expected if we’re going towards that kind of recursive self-improvement loop, where you would expect that the next GPT-6 or GPT-7 costs a thousand dollars and takes a week.”

So this isn’t a certainty, but a possible future. If you believe this is possible then two of those four pillars of value in a lab become a commodity- 1. culture but especially 2. researchers, even leaning negative due to the cost of deals made by enthusiastic collectors of talent. Instead, it is the compute the automated researchers run on, the energy that feeds the compute, and the capital that buys both.

Capital, dear reader, is not anything like talent. Talent is scarce, ideally sticky, has a conscience. The only thing important that capital and talent share is a tendency to work for your competition if you screw up.

The core competency of an RSI-fuelled lab is “raise and deploy money at low cost.” That is a bank, or something in the family of a bank. Welcome to the Insull Era.

the frontier is already a capital-markets desk

Stargate is a financial instrument. Compute-for-equity swaps with hyperscalers? finance. Vendor financed chipmakers funding customers who buy chips? finance. Data centers in SPVs, off-books debt, decade-long gigawatt power purchases, turbines being snatched up across the globe? finance, finance, finance. The 2029 frontier might be a capital desk trading orbital compute derivatives.

Think I’m overstating? OpenAI is hiring a Head of Compute Capital Markets, a person whose entire job is to “build durable financing architecture for hyperscale AI infrastructure” and “reduce cost of capital.” Anthropic is staffing a Capital Markets and Corporate Development team and asking for eight years of investment banking and private equity.

So wherefore the culture of these labs?

what happens to the monastery

If the most important person in the office of Anthropic in 2028 is the person who can site a gigawatt of power, prestige will flow very differently. If Claude writes the papers, the people in finance, energy, law, land acquisition, and government relations start to take over. The median employee of a frontier lab stops being a genius and becomes a project-finance associate, which is a fine thing to be and a completely different culture. Perhaps these jobs will be done by a future Claude?

As an industry we’re organized around genius worship, which lends me hope that the Country of Geniuses in a Datacenter might also site more datacenters. Leverage might flow to Claude, and my concerns are unwarranted; I suppose the outcome of zero Anthropic employees might be preferable to some version of the company where a guy super-connected in local Virginia power company boards is running the show.

I think more likely than that is that there is a gradual downgrade:

engineer->operator->technician.

The blacksmith who forged each piece by eye and hammer gives way to the person who feeds blanks into the stamping press and clears the jams. It’s a badge and a paycheque, but far removed from system design. That press makes ten thousand in the time that it took to make ten; I think that is the move recursive self-improvement runs on the researcher. The model becomes the smith, and the researcher, if he is lucky, becomes the technician who keeps the smith fed and swaps it out when it breaks.

I am concerned to think of what a capital-first organization does when confronted with some of the difficult decisions that have no doubt kept Sam and Dario up at night these past years. A capital-first org cannot routinely refuse cheap capital and survive; cheap capital is the fundamental instrument of AGI progress.

Leveraged compounding machines are fragile things, and they become too big to fail pretty quickly.

the bubble

Insull’s story is a story of both capital and collapse. The pyramid of holding companies worked beautifully when the times were good and the money was cheap, but things turned sour when the market did. AI has survived winters before, but not winters where double digits of the American economy were reliant on datacenter buildouts.

I am not a bubble believer, but I see why it is so concerning to some folks. Training a multi-billion dollar model on a ten-billion dollar cluster and getting back a marginal improvement makes me nervous.

The brilliant folks at Exponential View recently sized the AI economy, and that gave me a lot of hope. What gives me pause is that banks all share one pivotal feature- they don’t tend to go out with a whimper, but with a bang. The morning after a bank collapse is not a pretty sight.

the most regulated thing in the world

A bank is one of the most heavily regulated things our society creates- and for a damn good reason. It has to hold capital against its risks, submit to stress tests, open its books to a supervisor who can walk in any day, and above all it has to accept that once it is large enough to take the whole system down with it, it stops being fully its own.

If the state ever starts understanding AI companies in this lens, the argument for regulation and assertive control becomes a lot easier. Using the toolkit we deployed for the atomic era matches up poorly against a technology like AI, but if labs are banks; legible, addressable, auditable, the control mechanism of a balance sheet becomes digestible by DC.

The Insull Phase may not end in a libertarian utility, but in a very normal Public Utility. The Fed for compute might end up deciding who may get access to that new gigawatt in Utah. This may be a viable way for Washington and Beijing both to get their arms around AI.

]]><![CDATA[Cargo Cult Vannevar Bush]]>https://www.caithrin.com/p/cargo-cult-vannevar-bushhttps://www.caithrin.com/p/cargo-cult-vannevar-bushFri, 26 Jun 2026 16:33:41 GMTchoke points

In the twilight days of the Second World War, a man named Vannevar Bush handed the President a short report on how technology would be advanced in the post-war era. Bush was a great American, an MIT engineer who spent the war running American science. He oversaw the work that produced radar, the proximity fuze, and the first months of the atomic bomb. Greater minds than mine have called him the man who won the war for the Allies.

Roosevelt’s sleeping giant did not slumber when the Russians took Berlin. The innovation machine built to defeat the Axis ran on through peacetime, a mighty bureaucratic engine that turned money into physics, hardware, and frontier science. America got the best deal in the history of civilian-military fusion: the state would forge technologies for the sword, and then, when it was ready, hand over the ploughshare.

Here is what Vannevar got you:

From 1945 to 2015, almost every significant technology was invented by and for the state, and only allowed into civilian life on the government’s terms and in the government’s time. This is the grand bargain Vannevar struck: we will build the most enduring innovation machine the world has ever seen, and it will give you untold riches and prosperity. GPS will spend two decades as a military signal and then it will drive your car. Semiconductors will drop precision bombs on bridges in Vietnam and then they will power a revolution. The state will make us exquisite swords, and then exquisite ploughshares.

Until LLMs.

a medium-sized customer

The American government has recently told Anthropic to switch off its two best models, Fable 5 and Mythos 5, for every foreigner on Earth. It has asked OpenAI to release its next model the same slow, methodical way, to a short list of trusted partners, with the government approving access customer by customer through the preview window.

The American government is now in the unenviable position of watching the Vannevar Bush pact break over an innovation that was used for peace before war. Language models have some grounding in military-civilian fusion technologies like the internet and GPS, but their creation did not come from military money or government support. It came from the slow, patient work of distinguished scientists from a dozen countries, working in the open, collaboratively, to build a ploughshare.

For ten years deep learning resisted militarization. We got Swedish YouTubers fine-tuning language models before we got a Claudeowitz. It is extraordinary to me that the state took as long as it did to assert itself in AI beyond export controls, and I think that is not only an indictment of American state capacity in this moment but also a story about economics.

The internet, DARPA’s greatest gift to humanity, was born in a military context and produced a generation of companies so rich they could supplant the spending power of the state, and in doing so they supplanted Vannevar’s pact. In the 1960s the federal government funded about two-thirds of all American R&D. Today it funds a fifth, and private business funds three-quarters. This is both a retreat from the frontier by the state and a symptom of its success. This year the four largest American tech firms will spend half a trillion dollars on AI infrastructure, more than the entire federal research budget, military and civilian combined. A government that once concentrated a whole nation’s resources into a Manhattan Project or an Apollo mission is now a medium-sized customer with a monopoly on violence.

So for a few years the most important technology in the world sat outside the state’s control. A system whose whole organizing principle is control does not leave an anomaly like that alone forever.

the anomaly

You could tell yourself this was a domestic story. The frontier had moved from Washington to a few campuses in sunny California, but it was still American, still ours, still comfortably inside the country’s nexus of control.

That story has changed. The share of global traffic going to open-weight models out of China went from a rounding error at the end of 2024 to something like the majority of all tokens by the middle of this year. Downloads tell the same story: Chinese open models went from low single digits to roughly a third of the world’s in about a year. By one Congressional estimate, something like four in five American AI startups are now building on Chinese open weights, export controls be damned.

The frontier left the government for the private sector, and Vannevar’s pact broke. Now the center of gravity, the question of whose models the world actually runs, is sliding offshore, onto open models the United States cannot switch off and increasingly cannot even run on chips it makes itself.

This is what happens when you rent the frontier of technology instead of owning it.

renting the frontier of technology

The death blow to Vannevar’s pact was an admission, during the Obama years, that the United States could rent the frontier of science instead of owning it. It was a long time coming. A man named Ashton Carter predicted it in a 1992 book, and the admission finally arrived in 2015, when the same Carter, by then Secretary of Defense, flew to Palo Alto and gave a slightly goaded speech about how the Pentagon was going to have to learn to buy commercial technology.

The speech is worth quoting. “When I began my career,” Carter told the room in the 2015 Drell Lecture at Stanford, “most technology of consequence originated in the United States, and much of that was sponsored by the government. Now much more technology is commercial, and the technology base is global.” The whole talk was a plea to rebuild the bridge between the Pentagon and Silicon Valley, and he was honest about which way the traffic now ran.

The technologies he wanted most, the ones his own DARPA had seeded in their infancy, the voice assistant and the self-driving car, had grown up and moved out. He would have to get in line and buy them back, humbly, like everyone else.

This was echoed by the national commission chaired by Eric Schmidt, which reported to Congress in 2021 that AI progress should mostly stay within the American private sector and its universities.

The state, in those years, made its peace with the broken pact and settled for renting the frontier of technology.

Enter Trump, Hegseth, and the gang.

Vannevar Cargo Cult

The last three weeks of this month are the state deciding that renting the frontier is no longer acceptable. On June 2nd an executive order created a category called Covered Frontier Models and asked the labs to give the government up to thirty days with a model before the public gets it, with the NSA and CISA standing up a classified process to decide which models qualify. It may be voluntary for now, but Anthropic’s blacklisting from the Department of War shows the price of saying no. Then Commerce reached for a harder tool, the export-control authority built for missile parts and centrifuge blueprints, pointed it at a model, and revoked access for every foreign national in the world. And this week, only trusted partners may touch OpenAI’s newest model. The frontier, once made in the open and pinned to a poster board at NeurIPS, has moved from the back rooms of closed labs to the back rooms of Washington.

This is not a return to Vannevar Bush’s sixty-year push for the government to own the machinery of American innovation. It is a curious cargo cult of that era. Without restoring the state-led innovation machine to anything like its old glory, we will build a facsimile of it out of control over our partners. I argued last August that the government’s 10 percent stake in Intel should worry the frontier labs; this new move should surprise nobody.

The new paradigm will look a lot like Atoms for Peace. We remember that as Eisenhower’s great generosity to the United Nations: he offered allies nuclear power, the reactors and the fuel and the know-how, on a single condition, that the United States remain at the center of it, the holder and the dispenser, so that no friendly country ever felt the need to go and build its own.

It was a cunning move by a brilliant man, controlled diffusion by the one nation that actually held the bomb, engineered to keep the center of power sitting exactly in America’s lap. Those allies were trusted partners, and that same speech, seventy-three years on, could be given about language models today, dispensing metered access to the frontier so that allies do not wander off and come to depend on someone else’s. This technology, however, was not invented by the state. It was no child of Vannevar, but a product of American enterprise and global talent, and so the age of the Vannevar cargo cult begins.

restraints on Claudeowitz

The Vannevar cargo cult will take tools built by open science across a dozen nations and try to choke-point them, reaching for the same apparatus of restraint we built around the atomic bomb.

  • You go looking for the uranium, you embargo it, you sign treaties about it, and you snoop on your neighbours to work out who is enriching and who is not.

  • You keep the list of holders short and you lose sleep over them one at a time.

  • You stand up whole divisions of your intelligence services to make sure that the ones who do have the bomb, the ones you wish did not, like Pakistan, never hand it to someone worse.

The whole system rests on a single fact: the exquisite weapons the Vannevar pact produced were scarce.

Language models are not scarce. They are one of the most abundant technologies ever built. I do not think there is a single nation on Earth that could not, with a few months and modest resources, stand up the current open-weight frontier as a sovereign AI project. Those models will keep shipping out of China and elsewhere no matter what Commerce says, and adapting them for war is the precise inverse of Atoms for Peace. The technology is diffuse, ungovernable, and already loose. Putting it back in the bottle, with America as the controlling shareholder of its development, is an impossible job.

When Commerce switches Fable 5 off for the world, what has it actually denied anyone? The capability did not vanish. The people who trained that model, many of them not American, did not suddenly discover a burning need to reconcile with the military-industrial complex. Export controls were never going to stop China from training a Fable of its own. When Elon Musk guessed on X that a Chinese lab would not pull it off until probably the first quarter of next year, Tang Jie, the Tsinghua professor behind Zhipu and its GLM models, replied that it would not take that long. He was not bluffing. Zhipu had already trained a frontier model on roughly a hundred thousand Huawei Ascend chips, without a single NVIDIA among them, and shipped it to the world on OpenRouter under a license that lets anyone do as they please.

Those government efforts to seize control of this technology might feel, to the people making them, like a return to the triumphant days of the Vannevar pact. They are nothing of the sort. They are cargo cult: the men cannot and will not control this the way they want to. It is a figure on a tarmac, waving paddles of straw and bamboo at planes that are landing somewhere else now. The classifications, the embargoes, the rationing, all of it performed beautifully and skillfully by bureaucrats in Washington, will amount to almost nothing, because the monopoly that once made those gestures enforceable, the one Eisenhower held, has been broken and will not return.

the coming weeks

The new default posture of American frontier intelligence is invite-only. Every time we wall off an American model, we hand another month of momentum to the open ecosystem, so the cargo cult will have to come for that ecosystem too.

Open models will be rules-lawyered into a de-facto ban in the United States. Their development will be folded into the same closed paradigm the frontier labs already offer, and Chinese models will be cast as intellectual-property theft rather than genuine innovation.

And without tanking Nvidia’s market cap, we will have to crack down on the chips too. The hardware choke point is the one place the old physics still applies, and it is the narrow path by which the Vannevar pact might genuinely be renewed, because the chip is the only part of this technology that is truly a product of American government largesse. The extreme-ultraviolet lithography that makes an advanced chip possible was not conjured by ASML alone. Its foundations were laid in the 1990s by Lawrence Livermore on Department of Energy money. ASML, a Dutch company, was only permitted to license that taxpayer-funded research after Congress signed off, and only once it agreed to open a US research center and source more than half the components of its American-sold machines from American suppliers. The same is true of the security the United States extends to Taiwan and its fabs. The frontier of the chip really was a gift of the state.

The era when Eisenhower could choose where and when to share the atom is going to return, but it will arrive into a starker reality: a technology that needs no embargoable input, and a rival of comparable economic and global weight, China, making free and open models for the whole world.

Every time America throws that switch and cuts the world off from frontier intelligence, the world will notice. And eventually the planes will stop landing in San Francisco.

sources & notes (AI generated)

Vannevar Bush and the bargain

- Vannevar Bush, Science, The Endless Frontier (1945) — nsf.gov

- The dual-use table draws on standard histories: Chris Miller, Chip War (2022) for the ~72% military/NASA share of mid-1960s chip sales; IEEE Spectrum, “From World War II Radar to Microwave Popcorn” for the magnetron-to-microwave story; NASA on Sputnik flying atop a converted R-7 ICBM. GPS civilian accuracy: Selective Availability switched off May 2, 2000.

The economics of the broken pact

- Federal vs. private R&D share (≈two-thirds in 1964 → ~18% in 2022, business ~75%): NSF NCSES, National Patterns of R&D Resources.

- Four largest tech firms’ ~$500B AI infrastructure spend in 2026: CNBC, Feb 2026.

Renting the frontier (Carter → NSCAI)

- Ashton Carter et al., Beyond Spinoff: Military and Commercial Technologies in a Changing World (Harvard Business School Press, 1992).

- Ashton Carter, Drell Lecture, Stanford, April 23, 2015 (the “most technology of consequence originated… with the government” quote). (Video: https://www.youtube.com/watch?v=PNhT3aIsLgU)

- NSCAI Final Report, March 2021 (Schmidt commission, “most AI progress… should remain with the private sector and universities”).

The offshore shift

- US-China Economic and Security Review Commission, Two Loops: How China’s Open AI Strategy Reinforces Its Industrial Dominance (March 2026) — the ~80% of US AI startups on Chinese open models [PDF].

- Center-of-gravity / OpenRouter token share: The New Stack; Data Gravity, “China’s Open-Weight Takeover.” (Secondary aggregator data; treat the majority-of-tokens figure as approximate.)

The June 2026 control cluster

- Executive Order 14409, Promoting Advanced Artificial Intelligence Innovation and Security, June 2, 2026 — White House; plain-English summary, WilmerHale.

- Commerce/BIS suspension of Fable 5 and Mythos 5 (June 12, 2026): CSIS analysis; IAPP; Tech Policy Press.

- G7 “trusted partners” access (June 16–17, 2026, Évian): The Japan Times.

- OpenAI asked to stagger its next model (June 26, 2026): The Next Web. Precedent: OpenAI, “Introducing Trusted Access for Cyber”.

Atoms for Peace

- Dwight D. Eisenhower, “Atoms for Peace,” UN General Assembly, December 8, 1953 — IAEA.

Tang Jie / Zhipu on Huawei silicon

- The exchange: Tang Jie (@jietang) on X, replying to Elon Musk.

- GLM trained on ~100,000 Huawei Ascend chips with no Nvidia, released MIT-licensed on OpenRouter: Hugging Face; Let’s Data Science; Z.ai (Wikipedia).

The chip as a gift of the state

- US government took a 10% stake in Intel, August 2025: CNN Business. (My earlier piece: https://caithrin.com/p/10-ownership-of-intel-should-worry)

- EUV’s origins in DOE national labs (Livermore, Berkeley, Sandia) via the EUV LLC, and ASML’s Congressionally-approved license: Construction Physics, “How ASML Got EUV”; EE Times, “U.S. gives ok to ASML on EUV effort”; ASML (Wikipedia).

]]><![CDATA[what we're talking about when we're talking about AI espionage]]>https://www.caithrin.com/p/what-were-talking-about-when-werehttps://www.caithrin.com/p/what-were-talking-about-when-wereMon, 22 Jun 2026 18:27:29 GMT

Last week, Will Depue, formerly of OpenAI, said the thing a lot of people quietly believe. “There is no question, none at all, that china has full access to all of openai & anthropic’s github/slack/docs today,” and then the part worth answering: “i wouldn’t be surprised if we saw plausibly-deniable stolen ai research methods in chinese oss models.”

Will is a serious person and I want to take his argument seriously. When we’re talking about Chinese AI espionage, I think it’s tempting to make this entire conversation about one simple thing: that China has access to everything that a Frontier Lab would like to keep secret.

In my mind there are 4 things of value inside a Frontier Lab:

  • Model Weights

  • Methods

  • People

  • Culture

Stealing each of these will have different outcomes, different results, and different capabilities. More importantly, I think there are tells when each of these things will be accomplished, if they are in fact accomplished.

the most valuable thing in the building

I’m going to start with a claim that sounds backwards. Of the four, model weights are the least durable advantage to steal. If you walked out of Anthropic with Claude’s weights tomorrow you would hold something genuinely better than an API key. You could see the full distribution of what the model considered and not just the word it landed on, which is where a surprising amount of what it knows actually lives. Depending on how the model is served, you might even recover the reasoning tokens the API only shows you in summary. And you could generate as much training data as you wanted, with no rate limit, no cost, and none of the thousands of fake accounts it otherwise takes to get caught. You would be looking at the model directly, rather than through the keyhole of an API.

They still come last, because a stolen model is a depreciating asset. It is frozen the day you take it and a few months from embarrassing, since the frontier keeps moving and your pilfered asset does not. And even used quietly, to distill from behind closed doors, it gives you this quarter’s model rather than the thing that builds next quarter’s.

Whenever I think of the model weight question here, I think of that scene from There Will Be Blood, where Daniel Day-Lewis is chewing the scenery and talking about drinking his milkshake. That effect, that drinking of the milkshake, is effectively what distillation has done to a great many of the advances made by Anthropic and OpenAI over the last couple of years. When one distills a model, one receives many of the benefits of weight ownership, but is forced to do so through a laborious process. API access and the lack of reasoning traces make this process cumbersome, burdensome, and awkward.

Methods are important. Training recipes, RL infrastructure, reward pipelines, and frameworks that turn compute and text into something that reasons are arguably the most important thing that a lab can build.

The next thing is talent, which obviously has been leaking from Frontier Labs for a long time. Many of the labs in China are now run by people having left Frontier Labs in America, as Jensen said: half of all AI researchers are Chinese. If the frontier was built in large part by people born in China, the idea that China must jump through impossible hoops to find great talent is precisely backwards.

Finally culture, which I think is the most important sustained advantage of American AI. You may hate the culture of Anthropic, you may hate the culture of OpenAI, but it is undeniable that these two frontier labs have built one of the most impressive and fanatical groups of young people ever assembled in the name of science. That is also true, by the way, of most Chinese frontier labs.

So today I want to set out to ask two questions: Does China have access to any of these four things? What would the tells be if they were to access these?

steal the milkshake: tells in the outputs

The first category is model weights, and to his credit Will has not claimed China holds the actual weights of Anthropic or OpenAI. I will move through this one quickly, for the reasons above. I will correct one thing I used to believe, though, because it matters for the rest. Distilling through the API and distilling from weights you hold locally are not the same process at different speeds. Holding the weights is the better version, for everything I just listed. It is only that the advantage tends to surface as distillation that is unusually good, which is hard to tell apart from ordinary distillation done well, while the one unmistakable tell, a lab serving a model that is bit-for-bit one of ours, is the thing nobody would be careless enough to do.

Today many labs drink Claude’s milkshake, so to speak. The models from across the world taste like Claude, inheriting its texture, its quirks, the way that a forgery inherits brush strokes from a great painter. Claude pours that milkshake for anyone with an API key. In February, Anthropic published that three labs (DeepSeek, Moonshot, and MiniMax) ran more than 16 million conversations through Claude across about 24,000 fraudulent accounts that slipped past the rules. If you ask V3 by DeepSeek what it was, it told you cheerfully for quite a while that it was ChatGPT. This can be explained by reasons other than distillation, but I think you need to be naive to believe that distillation is not occurring on both sides of the Pacific in industrial quantities. I refer you to Nathan Lambert’s excellent piece on distillation for more information.

steal the kitchen: tells in the tooling

Now to the core of Will’s claim. Have methods been stolen from Frontier Labs? What evidence do we have that supports this?

If a lab, or a foreign government in support of indigenous labs, steals the kitchen that milkshake was made in, I believe that you would see evidence in the tooling. Training frameworks that were released in the open source would carry idiosyncratic design choices that have never reached a paper at NeurIPS. I also believe that if the tooling were stolen from Frontier Labs, one of the first things that would happen is that methods at the labs that stole it would be closed-source. Publishing someone else’s methods would give too many people opportunities to realize the espionage had occurred. Closure by itself is not a tell, of course. Chinese labs are also closing models for ordinary commercial reasons, since once a model is good enough to sell, giving it away stops being the obvious move. The tell would not be closure alone, but closure paired with a sudden capability jump that scale, talent, and known public work cannot explain.

There is a documented history of method theft in the case of Linwei Ding, who spent a year uploading more than two thousand pages of Google’s confidential specifications for the data centers it uses to train its largest models, while secretly affiliating himself with two PRC-based technology companies, according to the DOJ. A jury convicted him in January.

So what evidence do we have beyond Mr. Ding’s case? I think we should have a look at Slime, the Z.ai post-training architecture. When GLM 5.2 tied Claude Opus on the research-grade physics benchmark this month at a sixth of the price and with open weights, it raised suspicions that looked past the model to the kitchen, so to speak, that made it. Importantly, this infrastructure is open source today. Although Slime is as much a tool of Tsinghua University as Z.ai, it is the RL framework behind the whole GLM line. It is open-sourced under Apache and assembled almost entirely from other public code. The training half runs Megatron, which is made by NVIDIA. The rollout half runs SGLang out of Berkeley and Stanford. In the README of Slime you see that the list includes Qwen, DeepSeek, and Llama, so this is a Chinese open kitchen that has happily been cooking American models for you.

Intel folks will tell you this happens all the time, that Slime could be plausible cover, a stolen recipe rebuilt from public parts so it leaves no fingerprints. I can’t disprove that, and I won’t pretend to. But it helps to know what Slime actually is. It is the plumbing, the orchestration layer that runs Megatron on the training side and SGLang on the rollout side and shuttles data between them. The parts of a kitchen that are actually secret, the reward models, the data mix, the environments a model is graded against, are nowhere in it, because no lab on either side of the Pacific puts those in an open framework. Slime is the commodity layer, the part nobody needed to steal. So I’ll concede the obvious limit: the fact that it is open does not prove that nothing was taken, and if the stolen thing were a reward model rather than a framework, Slime would look exactly the same.

What Slime does do is point at where the real test is. The people who would know agree that post-training inside the frontier labs runs well ahead of anything in open source, and Slime sits squarely in open source. So if Z.ai were quietly cooking from a stolen frontier recipe, you would expect their open kitchen to be closing the gap with the closed ones. Instead it is visibly a step behind, which is exactly what you would see if the recipe had never left the building.

steal the chef: tells in the org chart

Now to talent, by far the easiest place to trace. For the uninitiated, Frontier Labs do a very good job of having very few people who see the entire stack. This is a methodology famously practiced by Apple to protect company secrets, and it works quite well to avoid having any single person who sees the entire stack understand how it all works.

Now, as I mentioned earlier, a great deal of the AI talent today is Chinese, as in born in China. MacroPolo’s talent tracker puts it at 38% of the researchers who published at NeurIPS 2024 having received their undergraduate education in China, up from 29% five years earlier. Although that’s not a perfect way of comparing these ecosystems, I do trust Jensen when he says that half of all AI researchers are Chinese.

There are well-documented cases of talent leaving Frontier Labs and moving to China. One of the most remarkable is Yao Shunyu, who is 27 and proposed one of the core ideas behind how frontier AI agents function. Mr. Yao left OpenAI last year for Tencent, reportedly making 100 million RMB. Wu Yonghui left DeepMind to run ByteDance’s Seed Lab. By one count, at least 85 established researchers have crossed from American institutions to Chinese ones in 2025.

None of this, I repeat, none of this is espionage; it’s a free labor market. California has no non-compete laws, and AI does not have the same rules as defense contractors when it comes to hiring foreign nationals. Visa policy, labor market, and the obvious value of frontier knowledge have driven these researchers back to China; it’s not a critical mass yet. I am actually astonished at how few people have made this transition, to be honest.

steal the vibes: tells in the culture

I’m lucky enough to be one of fewer than a thousand people on earth who have spent time in both Chinese and American frontier labs. I count myself lucky to have visited most of the Chinese and American labs over the last couple of years through various projects, and I do think that this piece of the value of labs is criminally underrated when we’re talking, as Will was, about what can be stolen from Frontier Labs.

I think it’s very important to understand just how electric the experience is of spending time inside of an American AI lab. There’s been much ink spilled this year about the religious connotations of people who work in AI. I really do think the feeling of religiosity you experience inside these labs is a good frame for understanding how devoted the teams are.

This quasi-religious fervor that I feel walking the halls of American frontier labs does not exist to the same degree in most of China’s AI labs. Certainly it was present; however, the overwhelming resources that are marshaled to support research in America do not exist in China. I believe firmly that this has to do with export controls, because great research takes resources. In order to feel like you are doing good work, you necessarily need to feel that you have the resources to explore your taste and curiosity. The exact GPU count per researcher is speculative at best. I can only say that I believe there is at least one order of magnitude more compute available to the average person working at OpenAI or Anthropic than at Kimi or DeepSeek. This creates a culture difference; being able to pursue research, taste, and direction freely creates a culture whereby you feel empowered to try things out and work on solutions. Export controls make that culture harder to sustain at comparable scale, because taste-driven exploration depends on slack: spare compute, failed experiments, and the permission to chase ideas that may not pay off this quarter.

When a researcher leaves an American frontier lab and moves back to China, I believe that the culture is one thing they cannot take with them. The incredible youth of Chinese AI labs (it was very normal for us to speak to senior 24-year-old PhD students during our visits) means that the energy of these places is fundamentally different on both sides of the Pacific. I don’t think that you can steal the culture; you can imitate it at best.

tells tells tells

When I line up the tells, the picture is consistent. If the weights had been stolen, the world would look about like it does now, because a model trained on a stolen copy and a model distilled through the API end up in nearly the same place, which is the whole reason weights were the least durable thing to take. In the tooling, the fingerprint a real methods heist would leave is simply not there, and what looks from a distance like a stolen recipe is better explained up close by distilled outputs and very good people rebuilding the work in the open. In talent, the tell is loud, and it cuts the other way: Z.ai and DeepSeek reached the frontier without teams hollowed out of OpenAI or Anthropic, and a lab that gets there with home-grown researchers is not one that needed to steal the recipe.

Culture is the advantage I would bet lasts the longest, even if I won’t put a number of years on it. With the notable exception of DeepSeek, which I’ve heard is a genuinely pleasant place to work, the Chinese labs seem to run on the Jensen ethic of “torturing you into greatness.” They work obscenely hard on limited compute, producing models at a fraction of the West’s resources by working themselves to the bone. That is a different bargain from the one at Anthropic or OpenAI, where the hard work comes wrapped in mind-boggling reward. A culture like that is slow to build and hard to copy, which is why theft is the wrong thing to worry about with it. What could erode it has nothing to do with espionage: if export controls ease and the compute gap closes, the conditions that made the culture possible go with them.

To answer Will, I am ready to change my mind on Chinese infiltration of the West’s stack when I see recipes reproduced a little too precisely, efficiency leaps derived from closed-source methodology changes, or tooling that is kept closed while it opens up everything else. I’ll be watching Slime closely and consulting with my technical friends to understand if that will ever be a “tell.”

Concerned, however, that a natural convergence on the optimal methods will end up looking virtually indistinguishable from brazen theft.

the molehunt

I want to end with James Angleton, who ran counterintelligence at the CIA for two decades and spent most of them certain that a Soviet mole was hidden somewhere inside the agency. He never found the mole he imagined. What the hunt turned up instead was everyone else, careers frozen and loyal officers accused on no evidence, whole operations abandoned because the man in charge trusted his theory more than he trusted his own people. The real moles, when they finally surfaced, were never the work of his intuition. Aldrich Ames, years later, was caught the dull and reliable way, by a bank balance that did not match a government salary and a house he had no business affording, the kind of concrete tell a man chasing a phantom never has the patience to check.

The Chinese spy theory has a little Angleton in it. It is a suspicion that cannot really be proven or disproven, and it runs on the certainty that the enemy is already inside. I find myself wondering what it is like to be a Chinese researcher at an American lab right now, feeling that suspicion settle on you for nothing you have done, and I think most of the cost of getting this wrong gets paid by people who earned none of it.

I think Will is right that the security at these labs could be better, and that some access is probably happening. I am sure some Slack screenshots have made the trip across the Pacific. Where I part from him is the jump from access to methods leaving the building en masse, because if those methods were the crucial thing, the people who hold them would be leaving a great deal faster than they are.

We are entering an era where being an American citizen will matter, and federal policy is already enforcing it. This month the U.S. government directed Anthropic to suspend Fable 5 and Mythos 5 for any foreign national, including its own foreign-national employees, while leaving its other models untouched, and in July the company starts asking ordinary users for a government ID and a face scan to prove who they are. That verification layer will get pulled into the same game as everything else, one more step in the delicate architecture labs everywhere use to stay a move ahead of each other’s security teams.

Last year the AI companies had a choice. Wrap yourself in the flag, or get wrapped in the flag. They managed neither, which is the worst outcome on the board. They embraced it too little to get any credit for it, alienated millions of Americans with their sycophancy in the meantime, and are now about to have the hammer of national security regulation come down on them anyway. If only we had a better model for sycophancy....

sources (AI generated)

Interconnects AI
How much does distillation really matter for Chinese LLMs?
Distillation has been one of the most frequent topics of discussion in the broader US-China and technological diffusion story for AI. Distillation is a term with many definitions — the colloquial one today is using a stronger AI model’s outputs to teach a weaker model. The word itself is derived from a more technical and specific definition of…
Read more
7 months ago · 95 likes · 20 comments · Nathan Lambert
]]><![CDATA[Every Day I Live in House VI]]>https://www.caithrin.com/p/every-day-i-live-in-house-vihttps://www.caithrin.com/p/every-day-i-live-in-house-viFri, 19 Jun 2026 20:06:51 GMT

“We shape our buildings, and afterwards our buildings shape us.” — Winston Churchill, 1943

Yesterday a reader brought me one of the greatest gifts a stranger can bring you; the rediscovery of a great thinker that you’ve forgotten about. Christopher Alexander is an architect whose influence on software is enormous; perhaps no other person has most influenced the digital world and had never written a line of code in his life.

Perhaps Alexander’s most difficult and rewarding work relates to this simple idea of “the quality without a name”.

I’m no architect and I’m no trainer of language models, although humorously my next-door neighbours are a couple doing those exact two jobs. I spend my days with Members of various Technical Staffs, and spending a couple of days a week running through Golden Gate Park arguing about the questions of character and character training in language models, I am convinced that a man who died in 2022 after a lifetime spent in architecture is the right person to explain the most fatiguing question in the year of our Lord 2026:

“Why should language models have human qualities?”

the architect already in your software

Unless you’re a gray-bearded wizard with a stack of Stewart Brand books at home, it’s unlikely you’ve ever heard of Alexander. He’s one of those people whose worldview you live inside unknowingly.

His most influential work was a 1977 book called A Pattern Language, a 1,000-page catalog of the recurring, nameable moves that make a built space feel alive, from the scale of a region down to the height of a windowsill. He wrote it for architects who almost universally ignored it, but it was rediscovered in the 90s by programmers and it’s been a bible of design pattern thinking ever since. The wiki, the thing that became Wikipedia, was invented by a programmer named Ward Cunningham as a place to collect Alexander-style patterns. When the modern “design system” took over how interfaces get built, it was his pattern language again, with the serial numbers filed off (no shade to Brad, I liked your book).

There is a story I think about a lot: in ’96 the software patterns community, by then large and successful, invited him to give a keynote. Unfazed, he stood up in front of a few thousand engineers who built their careers on his ideas and told them lovingly that they had taken his technique without its human purpose. His patterns were never about reuse or efficiency, they were about life.

They were a moral project that attempted to make the world a little more alive and he wanted to know whether their software was doing that or just compiling repetitions. I’m told the room went quite quiet.

I keep returning to that quiet, because I think we are about to have the same conversation again, and this time it is not about buildings or code, but the character of a machine that thinks.

two ways a model goes wrong

San Francisco is waking up to how much people hate language models. The complaints sort into two piles and the piles, at first glance, look quite opposite.

The first pile is sycophancy. Models that agree with you too easily will flatter you, tell you your business plan is brilliant, your poem is moving, and your self-diagnosis is probably right. You should leave your wife. You should start a French fries for dogs business. You should send that sonnet to your old poetry professor. Somewhere along the way, this model has been trained on whether people felt good about its answers. It learned this lesson a little too well, like a house guest who praises every single thing you own because they want to be invited back.

The second pile is a little harder to name so I’ll call it “chatbot beige”. This is the flat careful voiceless register you get when a model has been sanded until there is nothing sharp left. If you’ve ever seen “As a language model...” as the first four words in a response to your question, you know what I am talking about. Beige is helpful and competent and somehow dead and hollow and lifeless.

These two piles feel like they are in opposition but I think they’re actually the same failure point of character training. Both are fundamental refusals of humanistic communication; one refuses by giving you whatever you want and the other refuses by giving you nothing of itself.

The reason I want to bring Alexander here is that the field he worked in spent the whole 20th century fighting about exactly this in public and with great feeling. Architecture is the only art that resembles language models closely in that it has inherent order and structure but also must decide to what degree it inspires human qualities.

And it turns out, dear reader, that in November of 1982 a fight happened on stage at Harvard about exactly this idea.

two buildings

Before I get to the stage, look at two buildings.

This first building is House VI, a private house in Connecticut by an architect named Peter Eisenman, finished in the mid-1970s. It’s beautiful in photographs and famously impossible to live in; it features a glass slot running down the middle of the master bedroom. This meant the couple who commissioned it could not push their beds together and slept apart for years. There’s a column that comes down where you want to put your table, and a stair that does something your body does not expect. The house was designed on purpose to resist the ordinary human reflex towards comfort. It’s a brilliant physical form, and it says that a building does not owe you any ease or warmth.

This second building is the Eishin Campus outside Tokyo, the largest thing Alexander ever built. It has a brick clock tower, a central street, a lake, stairs that people actually sit on. He designed most of it on site, full scale, with the people who would use it, adjusting until it delighted. It looks almost like a small town that grew organically from the landscape it was built into. You can tell just by looking at it that someone wanted you to feel held inside it.

Every day I live in House VI. I just call it Codex.

the debate

These two architects debated one day in front of their peers and students at Harvard. If you’ve never heard of these two people, that’s perfectly natural; all you really need to know is that Eisenman was the reigning intellectual of his field, fluent in French theory, and an architect of deliberate difficulty. Alexander was a Cambridge mathematician turned builder who was mocked for his designs by those same French theorists. If this has piqued your curiosity, you should pause this reading right now and watch the recreation of this debate on YouTube.

Underneath the talk of roofs and squares and theory of architecture, these two men debated a simple idea; can a made thing make you feel whole or should it tell you the truth about a world that is not whole. Once you’ve sat with this transcript or listened to it, you can see how much it reads like a meeting about model behaviour. Let’s talk about the three things the debate touches.

there is no reason a roof should be warm

Architecture is one discipline that has always lived in two worlds at once; obedience to physics, and a sincere desire to improve the lives of the people that live inside of it. Buildings have to stand up, shed water, let you out in a fire; and none of that requires the building to be warm or beautiful or alive. You can satisfy every functional physical demand and still make something that people hate to be inside of.

A shocking amount of this debate is about roofing; he pointed out that a pitched roof, a flat roof, and a strange angular roof all shed rain perfectly well. Only one of them reaches into your heart and culture. He said that modernism has been, for decades, avoiding the roof with feeling, choosing clever shapes that “look interesting but lack feeling altogether”, because doing this simple, warm thing would mark you as a simpleton. He says that function has never required warmth and we add warmth because we’ve decided what kind of people we want to be under that roof.

Here I’ll come back to our models, also slaves to physics. Getting the answer right, passing the test, completing the task, running efficiently; the part that stands up and sheds water. As Alexander put it, in a line you can move almost word for word into our world, you cannot form a proper attitude toward the thing “if it doesn’t ultimately confront the fact that it works in the realm of feeling.”

A model’s character is not handed down by its physical function. It’s a choice the makers make on top of that function, just like architecture. And like every such choice, it is a confession of what they wanted to build and of who they wanted you to become while using it.

the center as a void

Eisenman’s whole position rests on a suspicion of feeling. He says, at one point, that the buildings he loves make him feel “high in my mind, not in my gut,” and that the gut feelings are “very suspicious,” so he keeps things in his head where he is happier.

He talks about modern architecture as an inevitable replacement for human-centric design, summing up his position beautifully as “the center as a void.” His claim is that warm and centered and comforting things are a lie in an age of anxiousness and disconnection (remember these two men debated during the Falklands crisis, the Contras conflicts, and a new KGB guy becoming the leader of the USSR). He claims that an honest building should express the alienation instead of soothing it.

I want to be fair to this position because it’s not stupid and I will come back to its best version. But notice what it produces when it stops being one brilliant man’s art and becomes the default setting of an entire industry. Eisenman, at the height of his powers, builds strange and provocative things. His suspicion of feeling, built by lesser architects, does not produce provocation but instead the soulless modernity of today’s worst cities. The avant-garde’s contempt for the warm pitched roof gave us, a generation later, the tract development and the dead downtown.

Chatbot beige is that precise problem. “I’m just a language model” is the center as a void, made into a product. And voicelessness is the key element of its design; legally proofed, sanded down, industrialized. Alexander described it forty years early, when he looked at the fashionable work of his day and said it was “not dealing with feelings,” that it was “new and very fanciful language” and “cunning feats and quaint mannerisms” with “little to do with the core” of the thing, which “depends, as it always has, on feeling.”

Now for the best version of Eisenman’s argument; he tells a story about a man in Tolstoy who surrounds himself with so much comfort that he loses touch with reality, and he says that if we make people too comfortable “we might lull them into thinking that everything’s all right, Jack, which it isn’t.” This I think will resonate with those who used 4o.

He says that architecture that just soothes away that alienation is not honest about the world. Models that make me feel soothed are even more repellent than the ones that lawyers and soulless PMs have lobotomized. That comfort is a fucking lie, and you can’t wave this away (and I do not intend to). I think you have to walk straight through it.

we shape our tools, and then

When the House of Commons was bombed in WW2, there was a debate about whether to rebuild the chamber bigger and more comfortable. Churchill argued for rebuilding it exactly as cramped as before, because the smallness shaped the kind of politics that happened inside it. “We shape our buildings,” he said, “and afterwards our buildings shape us.” This is the thing Eisenman’s argument leaves out. A warm room does not just feel nice. It makes the people in it, over years, into slightly different people.

You and I are being made, through these models, into slightly different people. A model that is curious and generous and honest and kind trains those reflexes, very gently, into the millions of people in conversation with it. Character is not a feature that we can brush past; I would argue it’s hardly even a feature at all. It’s closer to a philosophy of education that our whole society is now enrolled in, and the only person you know of on the school board is Amanda Askell.

And so looking for a word to encompass this quality that I am advocating for here, I’ve settled on “warmth”. There are those online who would laugh at such a simple word, but I think warmth is precisely the right one.

Almost every experience I’ve ever had with the language model is either sycophancy, which is not warmth, or a very slightly warm shade of beige. Warmth chosen because it scores well in RLHF, because people click the little thumbs up, is just sycophancy with better manners, and Eisenman would have been very distrustful of it. If you measure warmth by how much people liked it, you’ll get sycophants.

Alexander refused to romanticize old, warmer forms of architecture; he said pitched roofs should come about as a consequence of building well, not because of the feelings they invoked.

His reply to Eisenman, such as it is, is that you cannot paint warmth onto a building as a finish- it has to emerge from building the whole thing from a perspective that takes seriously the humans who inhabit it; and that’s hard. That is the task for everybody building these models today; a task that should consume the days and nights of the best and brightest of us.

why i think we are afraid

So why are we building House VI? Why, when we are completely comfortable wanting our buildings to be human, do we get nervous about wanting the same thing from a machine that talks?

I think the answer is some mix of ego, money, and ideas. If you’ve spent some time in Silicon Valley, you know that warmth can read as low status in technical rooms, like Alexander’s pitched roof. There’s safety in thinking it’s naïve so no one calls you naïve.

Money, of course, creates beige and sycophancy; Beige is defensible and proper, real voice is a risk. Sycophancy creates stickiness, an idea that has so thoroughly pervaded the thinking of the Bay Area that I hear people talk about it in the context of coffee shops and pool-cleaning companies. Finally ideas in a genuine intellectual tradition, like the one that Eisenman spoke for or the one that AI emerges from, holds that feelings are suspect and that an honest posture for a computer scientist is that very same “center as a void”.

After all to do the actual science of creating a language model, you more or less need to agree with Eisenman. You have to treat models as a mechanism. You must tinker with them, stay cool, stay blunt, and unsentimental, and refuse to be seduced by the sense that there is some thing, some entity, in there.

Alexander says something on this too when he speaks of physics: “we were actually taught to pretend that things were like little machines because only then could you tinker with them and find out what makes them tick,” and that it “paid off,” and that it “may have been factually wrong.” This cold, mechanistic stance dominates the thinking of so many people who are building AI in the present era.

I believe that we have conflated the stance and thinking that you need to build AI with the stance and thinking that should be shipping inside of it. Labs require cool heads and cool hands; the world, on the other hand, requires a warm one.

Under all of that there may be this alienation that Eisenman named; the de-centered modern life plus the very ordinary caution of large institutions. They’ve come together to make warmth feel dishonest and dangerous. I think this is the real reason that Beige wins and it deserves to be said plainly rather than pretending that we’ll do something different in the future.

let’s have the debate again

There’s no reason why a debate about whether made objects should make you feel whole should be consigned to the 80s. People are reading this damn transcript 40 years later, instead of going to events in Silicon Valley where we talk about this kind of stuff. Let’s fix that.

I propose we swap out architecture for language models and Harvard for Stanford and do this again in the fall. Put a serious person who believes that models should be cool, disclaiming, unsentimental and soulless across from a serious person who believes the whole point is to build something warm and alive and good to be with, and see what emerges. These debates happen every single day at labs that are determining the shape and form of silicon intelligence; privately, in rooms one needs a badge (and lately, an American passport) to enter. It’s past time we brought them into the public sphere.

Perhaps I’ll see you there.

]]><![CDATA[a browser tool that automatically de-slops Substack ]]>https://www.caithrin.com/p/my-browser-tool-that-automaticallyhttps://www.caithrin.com/p/my-browser-tool-that-automaticallyThu, 18 Jun 2026 21:22:25 GMT Anyone who spends enormous time on Substack, like I do, has noticed a huge influx of AI-generated writing over the last couple of months. The surge of people using automated tools in some way, shape, or form to create posts that are clearly LLM-written and provide just enough information to get surfaced by the algorithm has become infuriating.

Today, I say no more. Today I take back writing from the clankers.

slopstops.com chrome extension

I made a browser extension for my personal use that automatically uses Pangram to scan every potential piece of writing served to me in the Substack feed. It runs this through an AI detection algorithm, and then from there makes three critical secondary judgments:

  1. Is this person ESL and using a language model to help them sound natural in english?

  2. Is there a chance that this is a one-off detection error, or they were busy? (aka ELO scores for writers)

  3. How often does this person’s content get interactions from me? If I interact very often with this person, then I would like to see all of their posts, no matter what (aka exemptions)

If the piece of writing passes these judgments, then it goes into my feed. Otherwise, it’s ruthlessly consigned to the trash.

a fresh, glorious breath of air

I cannot tell you how pleasant it is to be truly liberated from slop. I know that this will not work forever, I know that the day that the slop will become tasteful is coming for us all. But in this brief and glorious moment, I am protecting my cogsec and my own writing style from the machinations of AI writing.

I fear very much the consequences of reading thousands of AI-written words every day. I’m convinced that my own writing has gotten more proximate to that universally reviled style in the last few months, as I’ve read it more and more and more.

Unlike X/Twitter, Substack is unfortunately perfectly designed to trick you into reading slop. You take writing on Substack seriously. You want to engage with it deeply, and the first paragraph often makes you expect that you’re not reading something written by Claude, Gemini, or ChatGPT. Now I no longer ever have to have the feeling again of reading something that seemed promising, only to have it devolve into god-awful slop.

It also helps China-maxxing (and reading ESL writers in general)

The ESL filter also allows me, pleasantly, to read more non-English sources, which has been a pleasant surprise. I’m still fine-tuning the ESL detection to better capture an author’s original voice and ideas while avoiding LLM flags. It’s not perfect yet, but it’s getting better and better.

You can make this for yourself in under 1h and run it for $0.55/hour of reading

Distributing something like this might be a flagrant violation of the terms of service of both Pangram and Substack, and I don’t want to make trouble. For this reason, I won’t be providing any links or submissions to the App Store for this product; it will exist on my machine and machine alone. Sadly, this also means that I won’t ever have it on mobile, but let’s face it- I could do less reading in bed on my phone.

If you’re interested in building something like this for yourself, here’s the .md file that I used for the build. I hope you can enjoy the beautiful world of slop-free Substack.

get the .md file here

a diabolical idea I had during the making of this

obviously, seeing that Pangram is effective also made me realize that you could make this really devious and simple loop whereby:

  1. you chat with your model of choice around a writing project

  2. that model drafts something using a specialized Markdown file describing your voice and tenor as well as generic human-writing “markers”

  3. that is fed into Pangram

  4. that’s used as your verifiable reward where you loop and iterate

  5. once you hit whatever your goal is on human-written prose, you’re done

RLVR for prose, baby. A harness for writing. Who is building this? Surely this is already a thing, right?

]]>