The Western relationship with AI has slowly slid into a paradigm that resembles medieval Catholicism.
America has high priests who invoke a god on our behalf through arcane rituals, in a language we cannot comprehend. Everywhere we erect shrines to this god, which consume our resources, tradespeople and state capacity. We tithe an ever-increasing amount of our income to this institution, with the promises of a future utopian pay-off that operates through the power of belief. Our young people aspire to serf-like lives in service of this god, but the powerful insiders enjoy bacchanalian rites. The heretical “open model” breakaway sect are regarded as dangerous Cathars, who should be converted back to the True Faith at every opportunity. The institutions of our culture are warped around it to the point that we can speak of little else, and I have a dreadful feeling that the abundant resources this religion has gleaned will end up being wasted on crusades in the Middle East.
Early this year I wrote how I felt Claude Code could be a Protestant revolution in software. This has not materialized. We managed to integrate AI into our existing power structure without spreading its benefits broadly enough to matter. The gains that seemed so tantalizingly close in January haven’t materialized so far. The direction of travel points to expert mathematics and scientific frontiers but is hampered by mediocre writing, conversation and a lack of what I like to call the “Dwarkesh Test” - an AI that does your taxes.
This logic of medieval church AI seems inevitable in the Bay Area. Labs will hoard the best models, make alliances with power structures in industry and government to create quasi-religious autocracies, and drip out the occasional piece of consumer software that maintains the illusion of general progress.
Imagining AGI
Coming to Beijing this month felt wrenchingly different. Much of this country’s AI work is outside the paradigm requiring absolute belief in AGI. Even the true believers are shockingly pragmatic in their quest for a machine god.
Beijing is currently splintering in perspectives on AI this coming year. I’ll call these two factions Generic Drug Makers and Biohackers.
The first category includes researchers working under two constraints. The GPUs available for training are structurally limited and the Chinese data industry is not ready to fill the gap left behind by the slowdown in distillation options available. These researchers have resigned themselves to making the “generic drugs” of AI.
To make a generic drug is not an ignoble undertaking. In removing a constructed barrier to benefit from a frontier technology, these researchers strip away greedy margins and a maze of IP protections that prevent the democratization of something potentially revolutionary.
But it is not the same as the frontier of drug development. If you have trained your whole life to seize the frontier of an industry, it is difficult to accept that your role is more reverse-engineering the work of others than forging a path for yourself.
The other camp I’ll call biohackers. These young and fervent researchers see a maze of IP and law that holds the secrets of machine intelligence hostage. They push to optimize a personal and idiosyncratic version of AI, motivated in a deeply personal way.
The biohackers’ gamble is drastically different. They bet that a confluence of brilliant research, risk tolerance, and freedom from regulation and lawyerly obedience will yield a path forward.
These twin visions for AGI are deeply different from the Bay Area’s ritualized, constraining AI. One promises abundance to the world through defiance of the intellectual property and secret tradecraft of the west. The other sees this same paralyzing structure and promises a way forward by finding a moonshot path through personal self-improvement.
RSI hype
The biohackers are justifiably obsessed with the holy grail of recursive self-improvement. They see a future where one can hack through the noise and rules of western AI labs and find a path to leapfrog the leaders in this space through a combination of hard work and risk tolerance. This approach has been remarkably successful in the last twenty years of ML. Many of the great leaps forward taken by the industry have come from small dedicated groups of hackers standing up and saying “all of you are wrong.”
The interests of Chinese researchers who see this as the path forward focus overwhelmingly on Ilya and SSI. To China’s believers, Ilya represents a confirmation of the biohacker path. Why else would someone leave OpenAI to work with a handful of brilliant researchers in relative GPU poverty? There must be, they feel, something that “Ilya saw” that gives them hope for this approach.
The generic drug makers see RSI as the inevitable transfer of power from the inventor to the engineer and the technician. Almost every industry goes through this transformation. It begins with wild-haired inventors wielding the protean fire of creation and ends with uniformed lab-coated undergrads manipulating buttons and dials, abstracted from the real work by machines they spend their careers babysitting.
Most AI researchers in China went into AI precisely because they did not want to be technicians. Civil engineering school doesn’t promise you can make novel bridge and tunnel architectural innovations. ML does. To see the frontier moving from inventor to technician so swiftly and brutally horrifies them, but RSI promises to level the playing field. Soon, we’ll all be dressed in our little hats and coats, minding our consoles and dashboards as the great machines whir and grind away.
Safety in the technician’s world
Most of my conversations with researchers turn to safety and the consequences of failures there. The last few weeks have seen the beginning of the Great Cyber Cold War, where proxy battles between US and Chinese models play out across nations who are unlikely to update their software with regularity.
This fight is of great concern to Inventor types. They feel a deep sense of responsibility when creating these models and inventors on both sides of the Pacific feel a mounting sense of dread.
For the modest engineer or humble AI technician, there is a comfortable distance between their actions and these consequences. Their sentiment is « my work is a tiny piece of this inevitable situation, so my responsibility is as minute as my contribution. »
The biohacker faction sees the coming wave of internet instability as a situation that could limit their creative space. Should the Cyber Cold War begin in earnest, the permission space available for radical experimentation may collapse, leaving them with fewer avenues of experimentation than they would ideally enjoy. They’re still deeply invested in open source and its glorious potential for innovation. But they detest the work of the « abliterators » who, by removing the safeguards they ship alongside their models, risk poisoning the legal, technological and political space they happily hack around in.
Both of these factions see the goal of AGI as a project that China has a key role in building. The generic drug makers see China’s role as breaking America’s quasi-religious chokehold on the frontier, a goal many American open-model researchers share. The biohackers see the opportunity to leapfrog America’s bizarre AI ecclesiarchs by breaking consensus beliefs. These beliefs include much of the conventional wisdom of alignment, interpretability and the goals of AI safety. As one researcher told me:
« Everyone in America is trying to solve these problems with psychology and constitutions and moral frameworks. We are going to solve this problem like we solve all problems—with math and physics. »
AI Heresies
One of the most difficult parts of the American AI situation is that there is no control group for the Grand Experiment beyond China. Almost everything important in western AI happens within 20 miles of the Golden Gate Bridge, and California laws are such that people move between labs with breezy regularity. This means that the entire institution of American AI has learned to think in remarkably similar ways. We listen to friends debate Dario’s letters, watch Dwarkesh’s interview with his roommates, watch X hype videos made by the A16Z New Media machine. Our collective experience of takeoff happened this year or the last. Life on the exponential means that besides a few alienating conversations next month at Thanksgiving, we rarely spend time with a) anyone who doesn’t autofill Aschenbrenner when they hear the name Leopold and b) who doesn’t believe that he is basically correct in his thesis but a little off on the timing.
Going to China shows that one can live on the exponential without embracing the cult surrounding it. You spend days with sincere AGI believers who have a very different vision for the relationship between society and AI, revealing a deep desire for a future different from the consensus in Silicon Valley. This dialogue, as muted and suppressed by politics as it is today, seems to me to be the most essential cultural exchange of our era.
Two weeks into this trip I feel more optimistic than ever that AGI will arrive soon. At the same time, I’ve been patiently shown whether my beliefs in AGI are flights of Bay Area fancy or realistic to the technology. SF has a structural advantage in AI development that will not go away, but this trip confirmed for me a feeling that the city no longer holds a monopoly on vision for our AGI future.