you could build a top tier venture firm just focusing investment decisions short and long term based on deep model benchmarking / evals
find capability overhang, find areas models suck and track trajectory, etc
10:29 AM · Jun 6, 2026 · 42.1K Views
Logan Kilpatrick, who works on Google AI Studio, proposes VCs base their investment strategies on deep AI model benchmarking
This methodology tracks capability overhangs and model performance trajectories.
757143122667.1K
Sentiment
Users who like AI model benchmarking say it helps venture capital decisions with better due diligence and insight into progress, but critics say benchmarks are manipulated and miss real user needs and market feelings.
Pos
70.0%
Neg
30.0%
11 comments with sentiment.
Cluster Engagement
Posts from X
Most Activity
Most Activity
VIEWS9.8KBOOKMARKS9LIKES51RETWEETS3
Logan Kilpatrick@OfficialLoganK
or put a different way, every venture firm should have an evals / benchmarks team putting the models through the paces, building new benchmarks, etc
Logan Kilpatrick@OfficialLoganK
you could build a top tier venture firm just focusing investment decisions short and long term based on deep model benchmarking / evals
find capability overhang, find areas models suck and track trajectory, etc
8hViews 9.8KLikes 51Bookmarks 9
REPLIES4
Rampalli Karthik 🇯🇵🇺🇸@karthashok008
@OfficialLoganK Definitely. Have you considered Eval alpha decays fast. Public benchmarks saturate and get gamed. The edge has to live in proprietary evals, which means running a small research lab inside the firm — compute, evaluators, infra. That’s a cost structure most VCs don’t carry.
7hViews 9
Logan Kilpatrick@OfficialLoganK
only a matter of time until all the top firms are doing this
Logan Kilpatrick@OfficialLoganK
you could build a top tier venture firm just focusing investment decisions short and long term based on deep model benchmarking / evals
find capability overhang, find areas models suck and track trajectory, etc
8hViews 4.5KLikes 31Bookmarks 1
Logan Kilpatrick@OfficialLoganK
Emad@EMostaque
@OfficialLoganK Could call it atp+
Logan Kilpatrick@OfficialLoganK
you could build a top tier venture firm just focusing investment decisions short and long term based on deep model benchmarking / evals
find capability overhang, find areas models suck and track trajectory, etc
7hViews 2.2KLikes 7Bookmarks 2
Curlheinz@Curlh1
@OfficialLoganK @grok explain to me what Logan means like I’m 12
8hViews 229Likes 1
Logan Kilpatrick@OfficialLoganK
@EMostaque :)
7hViews 293Likes 4
furnaces@furnaces
@OfficialLoganK there's only so much benchmarks can prove
vibes >>> benchmarks
8hViews 230
Leo Linsky@leo_linsky
@OfficialLoganK We do this at http://gertlabs.com/rankings
Did you know Gemini 3.5 Flash is a frontier model in real-time simulations and spatial reasoning, but drops the ball in abstract environments with constraints it wouldn't have seen in its training data?
7hViews 149Likes 3
Praveen Kumar@itzzme_pk
@Curlh1 @OfficialLoganK @grok @grok explain to me what Logan means like I’m 20 with detailed explanations with examples
8hViews 36
Varun Gangal@VarunGangal
Agree partially but some blockers are:
a) shifts in model ability on areas that are decisive can show emergence i.e. wherein hard perf metrics on an eval are low due to the model being weak on some out of all the sub-abilities that compose most tasks in the eval but once they stop being weak there perf on the eval sort of breaks out...
b) Benchmarks / evals are themselves built kind of reactively but also slightly serendipitously where people building them (e.g. @OfirPress et al with SWE-Bench) see the frontier and get inspired or "catharsized" enough to put efforts to make an eval [and this is a function of model ability] and it becomes viral enough to adopt. For long term prediction based on benchmarking, this kind of randomized evolution can be unstable to predict or marginalize over (and continual add/remove of the eval basked that's the basis for pred)
c) Also as some comments note here, saturation due to various reasons renders well known benchmarks useless as signals of growing model ability after a while, apart from their use to check non-regression...
7hViews 34
Hanielle@Haniell59310840
@OfficialLoganK Can I also implement it in an existing company?
8hViews 308Likes 1
Blissy@BlissyOnX
@OfficialLoganK kinda feels like were entering the era where the best investors are just eval readers with a thesis
how do u avoid getting wrecked by a model release though?
8hViews 82Likes 1
MetaCritic Capital@MetacriticCap
@OfficialLoganK During the singularity? VC investment periods are too long for that.
If you had foresight about the need for agentic harnesses, what would you have done? Invested in Devin and Replit?
It's the big three labs and that's all.
8hViews 210
Furkan Gözükara@FurkanGozukara
@OfficialLoganK When is gemini 3.5 pro?
Focus on video and audio understanding and lesser censorship
Give up coding
7hViews 57Likes 1
WENE@omijagun
@OfficialLoganK Model evals are becoming the new due diligence. The firms that can consistently identify where models are meaningfully improving (or plateauing) will have an information edge that traditional financial analysis can’t match.
8hViews 45Likes 1
Soroush Fadaeimanesh@S_Fadaeimanesh
@OfficialLoganK benchmark-driven thesis works only if you can hold the position between eval lead and revenue catching up. that gap is usually a few quarters and most LPs do not have patience for it.
7hViews 136
TheAlphaLetters@TheAlphaLetters
The catch is the same one that breaks every quant signal in that a benchmark edge is only an edge until it's priced in. Public evals get arbitraged away fast and the alpha isn't benchmarking deeply, it's benchmarking what the market hasn't priced yet, capability overhang the consensus can't see, or a trajectory read that's not obvious. The moment the eval is legible to everyone, the edge decays
7hViews 134
ByteCrafter@bytecrafter_1
@OfficialLoganK benchmarks saturate way before the capability does though. half the interesting gaps never show up in public evals until someone ships a product into them. private task-specific evals feel like the actual alpha here
7hViews 124
Sup*@mostlyaboutai
Doing this already. Distribution is still the key. Been working BTS with companies on product builds model refinement and evals so I know how deep the industry already runs. Gotta position yourself well and pitch even better. P.S. been trying to get in on AI Studio too and share my views 👀
8hViews 106
Load more posts