Oppenheimer, Sagan, and Saying the Loud Part Quietly
“For a successful technology, reality must take precedence over public relations, for nature cannot be fooled.”- Feynman,
Lighthaven, in a small dark room
Nathan Lambert published an excellent talk on open models and the ATOM Project this week, and I was very interested to see the Safety People Questions afterwards.
Safety People have become a fixture of the bay area AI scene. A lesser derivative of the “thinkboi” (although they come in multiple genders) they are a constant presence in the zeitgeist of AI in SF.
Young, brilliant, and exposed to The Sequences in their teenage years, they earnestly want to save the world from extinction at the hands of AI. They are often deeply technical, very serious, and believe that the key to preventing the AI apocalypse is some combination of research and thinking (and possibly missile strikes on datacenters).
At the end of Nathan’s talk on open models, a line of Safety People formed at the mic. They took turns asking essentially the same question: aren’t open models dangerous? Each one heard the person before them ask this question, but somehow were unable to pivot to another question when their time came at the mic. Gracious as ever Nathan answered them patiently, but this moment really exposed for me the rot at the heart of the safety discourse in AI:
There are not that many ideas on how to get the message out.
The research? The research is amazing, and I support it fully. The problem I am pointing out here is that the insular nature of the AI safety community means there are only a few ways of raising awareness of perceived risk- getting on mic on a recorded lecture seems like an important one to that group.
a modest proposal
There has been one big project in growing the tent of AI safety in 2025; a book called “If anyone builds it, everyone dies”. I see this book as a great representation in all that has gone terribly wrong in Safety People World; a hysterical, wild-eyed and attention grabbing shot at capturing the attention of America before their apocalypse.
I admire that they went there- someone had to. I do think that now that this final step has been taken (surely we can’t go further- “if anyone thinks hard about it, everyone dies?) we should start thinking seriously about how to actually get people to give a shit about AI security risks. This process must start, inevitably, in everyone stopping the use of the word “safety”.
Safety is not interesting. Safety doesn’t inspire. We use LLM-guided recsys to scroll TikTok during workplace-mandated safety videos, and we certainly don’t read books about any other kind of safety for enjoyment. Airports aren’t stocking books about seatbelts and airbags, and even the most scary events of our time don’t precipitate a deep global discussion of safety (see: the pandemic).
The Carl Sagan of AI
The title of America’s Beloved AI Educator is still a jump ball. Dwarkesh is hitting millions of views on his podcast interviewing the luminaries of AI, but he also spends a bunch of time digging into Russo-Japanese naval battles and biotech. Karpathy makes incredible videos on fundamentals of LLMs, but not the kind I can send to both my grandmother and my 13 year old cousin.
We had Oppie for the Atomic Age, and he did more than anyone in advancing ideas in how to make this terrible new technology safe. We had Sagan for the Space Age, and alongside people like Feynman we talked seriously about safety and improved vastly the culture of NASA around this issue (some say too well).
In all these cases, we were in a very similar bind- a group of scientists were driven by zeal and American gumption to birth a new world. The average American had nothing to do but sit back and watch as the marvels were birthed- abundant power, incredible technologies, breakthroughs left and right.
In so many of these prior technological revolutions, the Sword came before the Ploughshare. We made integrated circuits to precision-bomb Vietnam, we dropped Atomics before we warmed our houses with their electricity, and we certainly worried more about the cold-war race with Russia in the space program than about the civilian benefits. DARPA gave us the internet, even the Iron Age was about military application.
In AI, we have the peaceful technology before the military application (coming soon, no doubt). This means that the average person encounters AI with some Hollywood trepidation but quickly learns to trust this technology (indeed, trust it with their most intimate secrets, their government, and their privacy).
This has put AI safety in a deep bind;
the technology doesn’t seem destructive, the way atomics were
the fear doesn’t motivate when you don’t have a “how to act”
the best place to learn about AI is by using AI.
If the Safety People want to win this argument, they need ways to get to the question of AI safety without using the word “safety”, and certainly without having their hands waved in the air in panic. They must seek to take up the mantle of Beloved Public Educator, and through the work of helping America understand this new miracle, they can guide the discourse on safety as they see fit.
“Security is a process, not a product.”
-Bruce Schneier
If I were sitting in an Eames Chair at Open Philanthropy, sipping on a cup of Yorkshire Gold Tea with my Gaziano loafers kicked up on the Ottoman, I would be asking myself one question: How can I make the Carl Sagan of AI?
By funding safety people to not use the word safety. To talk about the opportunities of AI, the incredible technology behind its development, to sit with practitioners and understand the technology deeply, all without the lectures, the navel-gazing discourse, and the hand-waving hysterics.
The next and obvious step in avoiding the Great Risk of AI is simple; learn how to talk about AI Safety without ever saying “Safety”.
