Why doesn't Venture Capital make benches and evals?

Everyone I know in VC has a podcast and not a single one has an eval.

It’s a terrible time to be a software investor in general, but a wonderful time to be a software investor in Anthropic. The frontier models are generalizing faster than any platform shift of my lifetime and when I look at the softer categories, I struggle to explain where durable value might live other than in the models themselves. Anthropic seemingly agrees with me; their latest push into bio isn’t some picks and shovels business. They clearly believe that a version of the model in the near future or today can just... discover drugs themselves.

Tarted up service businesses abound in the logo section of VC websites. Businesses whose revenue scales with headcount do not return the fund.

Backing products that the next model release generalizes to absorb is worse than a service business; it’s that tweet about a man whose neighbour is feeding cats to coyotes.

Every VC who is investing in pure software is AI-first, so how come none of them have benchmarks? Literally every other piece of the AI industry does. So why not VC?

trillions

Trillion valuations only make sense if companies become some of the largest software enterprises that have ever existed. This means they must absorb enormous markets. The absorbing is not a side effect; it’s a deliberate plan to eat the world, starting with the most delicious, highest margin parts.

Claude Science is the first real sign that internal preclinical drug discovery will be done by labs themselves. Sure, they’re pointing it at rare and neglected diseases today, but if you’re buying Coefficient Bio for $400 million, then you’re serious about the way that your company can generate revenue from primary scientific exploration.

There is a Mythos class model producing promising candidates on nine of 14 protein targets. There’s not a single drug today on the market with FDA approval that has been discovered by AI, to my knowledge. I’m not claiming that biology is a solved problem, I’m pointing out frontier labs are running a direct operation to take a trillion-dollar industry and make it theirs.

We’re 18 months into this being the inevitable thing that venture capitalists think about in the shower in the morning. Before their work of decks and spreadsheets, reference calls and founder vibe checks, they are contemplating the imminent cannibalism of their category, courtesy of Claude.

Their work, of course, comes from a different era. Decks and spreadsheets are instruments of that SaaS era, and they were good instruments for what we needed them for. When SaaS companies’ fates turned on execution and distribution, those instruments were really valuable. My contention is that today very few, if any, VCs seem to have made the jump into AI natively, the way that we think inside the industry. There’s no better way to see this deficit than in this staggering lack of benchmarks put forward by venture.

Founder quality still matters enormously, of course, and nothing here replaces it. I’m not arguing that founder quality is a tool of the SaaS era. VCs have cared about founders and teams far longer than they’ve cared about how you can invest with a spreadsheet.

I hate-listen to Harry Stebbings’ podcast on a regular basis and he is a genuinely wired-in AI investor. The guy gets it. You can hear him talk to guests and it sounds like conversations that I hear among people that work inside of this industry in San Francisco.

Not a single eval has ever been made by Harry Stebbings or anyone else.

eval-based investing thesis

SWE-bench asks the simple question about whether a novel model can resolve real GitHub issues, starting in October 2023 with the best frontier model doing about 2%. By April of 2024 Devin was at 13.86%, SWE-agent at 12.5%. That August, 4 months after the jump, Cursor raised a Series A at a $400 million valuation. This year it exited for 60B.

Not a single person that I’ve ever heard of in venture looked at that and thought, “Damn, we really should organize our whole business around these benchmarks.”

Meanwhile everybody else in the ecosystem figured it out. In about 18 months:

• Harvey built BigLaw Bench and then LAB for legal work.

• Column Tax built TaxCalcBench.

• Penrose built AccountingBench.

• Snorkel built an underwriting bench with working insurance professionals.

• Stanford built HealthAdminBench for the paperwork side of medicine.

• Nomic built one for construction drawings.

This ecosystem is so big that there are now specialty eval companies. LMArena raised at 1.7 billion in January, Braintrust at 800 million the next month. Startups have benchmarks. Labs have benchmarks. Academics and eval shops have benchmarks.

Venture investors, whose entire job is pricing capability, are the one participant that is just absent from the race. Maybe they were too busy with the podcasts?

the instrument

I imagine waking up one morning and the wool vest that I like has somehow glued itself permanently to my body. I turn and glance at my phone and I see text messages from LPs. A cold sweat runs down my spine. It’s finally happened. I’ve become a VC.

The very first thing I would do is take some thesis on my website and turn it into a viable eval. Looking at a company like Harvey who built their own benchmark in 2024, I’d ask myself how I can help prevent frontier labs from eating categories that I want to invest in by showing exactly how my thinking in this category can dissuade those labs from investing in it, and then I would look for companies that would deliberately and aggressively benchmaxx that very evaluation.

I would publicly allow anybody who could score above a certain threshold to come into the office for lunch and pitch us on their solution. I would go on all of my friends’ venture podcasts and talk about how evals are the new deck, and I would bully everybody I knew in venture to start thinking about benchmarks as a viable way to form a thesis and category.

The best part of it as a discipline is that it comes with a free method of seeing how sensitive your category thesis is to Claude; you just boot a fresh instance and see how it benches.

What I would be getting up to is not black magic. It’s just compute, sustained attention, and some code. A serious single industry bench costs $1M to build and another $1M-ish a year to run, which is roughly what funds spend on platform teams, and podcasts, and a rounding error for something like A16Z’s New Media.

I’d start with billing records; engagement letters in audit, CPT codes in medicine, statements of work in consulting, claims files in insurance. From there, build out a suite to motivate people to hill climb. Keeping it private if you worry the bench reveals a thesis. Alternatively, publish it as a challenge and let the ecosystem fight it out.

It’s a recruiting asset because the people building AI companies love evals, and a fund with a real benchmark is speaking the field’s native language.

It’s useful beyond just investing in companies that can actually score on that bench; you get all kinds of alpha on the field. A bench sits in one of three states: nothing works yet, scores are climbing, or scores have topped out. Nothing’s working ever? You’re early. The scores are climbing? You’re a genius. Immediately saturated? Either that vertical is Ant food or you need to make a harder bench.

A particularly clever VC will use this to test a window before any company exists by running frontier models against your bench with real engineering effort and publishing the budget you spent trying. If the models are failing, you have found work that is either a narrow window (valuable until some release swallows it) or durably hard (valuable until AGI). Sure, you’ll need different instruments with different prices, but that’s okay. The point is you’re out in the arena, actually making benchmarks and actually doing evaluations, and the dealflow will be crazy.

two flavours

So there you are with your great VC-created bench; now you need to decide which companies in the domain are making the right bet. Here I see two positions that you get out of your creation: I’ll call them The Climber and The Surface.

The Climber

These companies are selling the capability that you’re measuring. Their product is as close to the benchmark score as you can get. Think Devin for agentic code. A Climber competes with the labs on the exact axis the labs optimize for, so every model erodes its edge. Because the score is public news, it gets priced the day it prints.

2024 was a grim year for climbers. Magic raised about $465 million, including $320 million in August 2024 with Eric Schmidt participating, on the promise of 100-million-token context windows, announced a model it never publicly released, shipped no commercial product, and has since repositioned its website toward safe AGI research. Poolside raised $500 million at $3 billion before releasing a product, announced a two-gigawatt Texas datacenter project with CoreWeave, and by this April the datacenter deal was dead and the Financial Times reported its two-billion-dollar raise had failed because investors doubted it could train competitively.

These aren’t fraudsters. These are simply companies that fail to understand that their benchmark was too competitive.

The Surface

The Surface sells the platform where the capability gets used: think Cursor. When the model was weak, their editor was a nice product. Every time the model improved, the same editor became more valuable at no research cost to Cursor, because better capability flowed through the workflow it already owned.

The market took 10 months to price it properly because the connection between a model benchmark and a Surface company is rather indirect, and some even claimed that Cursor itself would commoditize.

They hitched their entire company’s existence to those benchmarks, measuring the success of coding agents, and now they have 60 billion reasons to celebrate.

Harvey is in a similar position. They’re a Surface that also built their own bench, so each release improves their product and also gives them visibility into how quickly the labs are catching their business.

a new workflow, just like the old workflow

Imagine that you’re building a bench as a venture capitalist; say it’s in, I don’t know, claims adjudication.

The bench itself is simple to describe. Take 500 anonymized claims files. Ask the system to identify payout, denial rationale, missing documentation, fraud indicators, and appeal risk. Score against expert adjudicators.

To build it, you must learn:

• where the money from a claims file lives

• which experts hold ground truth

• what data is scarce

• and which pieces of the workflow are genuinely hard rather than merely neglected by automation

You will very quickly find the surfaces because you know which workflows matter. You will quickly find the climbers because you’ll know the industry. You can even see where new capital should flow in the domain, which strata needs data or which needs environments built before any application company can really work.

In short you’ll do thorough incredible diligence that will be useful to your portfolio and will allow you to source great founders. For many venture capitalists they can already do this for industries that they know well, so the task ahead would be more like refreshing themselves on something that they left to start their fund than diving in from nothing.

Diligence is also changed. It’s more like a sorting question now: companies claiming a capability edge need to justify themselves on a benchmark instead of just talking or hand-waving around Anthropic.

the category that had no bench

I think a great example of the cost of not understanding this lesson is the agentic browser boom. 2023 to 2025, billions went into the idea that browsing plus AI was a company: Adept raised over $415 million to build agents that operate software, the Browser Company raised $128 million building an AI-native browser, and a crowd of smaller browser-agent startups followed.

Academics had benchmarks that proved exactly this capacity and capability, like WebArena and WebVoyager. The labs were climbing these as these companies were being created. The entire category’s equity upside went to the labs. The instruments that measured relevant capability existed and were public, and as far as I can tell, nobody in the capital stack really cared.

Imagine how different that would have been if there were venture-backed benchmarks. It would have been an extraordinarily interesting thing to have happening while all those companies were being made; you could have seen so much of what happened coming and made great decisions about who to back.

if you don’t, someone else will

Interestingly, I think that if venture doesn’t get into the game of evals and benchmarks, research firms will start raising funds. SemiAnalysis expects over a hundred million dollars in revenue this year selling exactly the kind of domain instrumentation I’m describing here, and they’re weighing a venture fund.

I don’t think Semi is an outlier. I think it’s early and operates with conviction. SemiAnalysis holds the deepest measurements of a domain that has real value, and much of their work is effectively benchmarking. It may prove easier to bolt venture onto the side of expertise than to develop it internally.

And obviously the labs themselves have been running this playbook all along. Labs decide what to build next by writing an eval and training against it; in other words, this is exactly the thing that they use to decide which cat to eat next.

Jason Wei’s famous formulation is that the ease of training AI to do a task is proportional to how verifiable the task is and a good verifier is just a training environment. Evals themselves are becoming the binding constraint to progress, so labs take them seriously in a way that actually matters.

No fund that I know of will ever train models against its own book, but they should be measuring (privately or even bravely in public) exactly what capability is doing and how to build companies inside difficult verticals.

fear no eval

Benchmarks that saturate are not failures. They did their job. They told you a capability was impossible, and then they told you it had arrived. When they retire into the public record, that’s a good thing for the industry. Venture funds should be building these instruments, mapping what is economically valuable and still on the frontier, and getting down into the dirty business of understanding what AI will do that we believe to be impossible today.