Still, one thing I am confused about: most existing benchmarks target a vertical, rather than an ecosystem. (Arguably even chatbot arena was the vertical of chatbot responses). If a VC makes a specific benchmark, doesn't that narrow who they are interested in too much? Like at that point, don't they just become a company trying to solve the problem?
There's the semianalysis example you give, but isn't a lot of thier benchmarking hardware systems, where the differences are a bit easier to parse?
The ideal scenario for one IMO is to create a good ecosystem measurement if they can. Something like OpenRouter's token percentages or ramp's AI index, which provides a ton of free advertising to both companies. But curious for your thoughts on this
But VCs do do this! They fund RL environment companies, which is one of the hottest startup markets in AI, and then these companies give them investor newsletters. Outsourcing the construction of benchmarks to experts (exactly the people who start such companies) is much more efficient.
I'd like to see it be a force deeper than just the portfolio. You're right, of course, that they fund the category, but I'd love this to be a deeply integrated piece of their process
I guess I'm sceptical about (a) how much more information they can get about the future from this, especially given it's not really their comparative advantage (b) I think plausibly many other forms of data including just a pure personality assessment of founders could be more predictive, especially considering that many future capabilities of AI at VC-relevant time horizons will show on a benchmark as ~0% performance (examples: all of general-purpose robotics right now, or world model bets).
Your recent pieces have been popping off!
Still, one thing I am confused about: most existing benchmarks target a vertical, rather than an ecosystem. (Arguably even chatbot arena was the vertical of chatbot responses). If a VC makes a specific benchmark, doesn't that narrow who they are interested in too much? Like at that point, don't they just become a company trying to solve the problem?
There's the semianalysis example you give, but isn't a lot of thier benchmarking hardware systems, where the differences are a bit easier to parse?
The ideal scenario for one IMO is to create a good ecosystem measurement if they can. Something like OpenRouter's token percentages or ramp's AI index, which provides a ton of free advertising to both companies. But curious for your thoughts on this
Practically speaking, I would like this to replace the current paradigm of developing a thesis- you make the benchmark instead.
ycombinator.com/rfs
Now imagine how amazing these would be as benchmarks!!!!
Oh wow I'm exposing how little I know about VCs. Yeah rfs should be benchmarks, or be benchmarkable fairly easily.
everyone wants to be AI native, nobody wants to make the damn benches
But VCs do do this! They fund RL environment companies, which is one of the hottest startup markets in AI, and then these companies give them investor newsletters. Outsourcing the construction of benchmarks to experts (exactly the people who start such companies) is much more efficient.
I'd like to see it be a force deeper than just the portfolio. You're right, of course, that they fund the category, but I'd love this to be a deeply integrated piece of their process
I guess I'm sceptical about (a) how much more information they can get about the future from this, especially given it's not really their comparative advantage (b) I think plausibly many other forms of data including just a pure personality assessment of founders could be more predictive, especially considering that many future capabilities of AI at VC-relevant time horizons will show on a benchmark as ~0% performance (examples: all of general-purpose robotics right now, or world model bets).