• Home
  • Technology
  • Gaming
  • Entertainment
  • World & Business
  • Science
  • Sports
  • AI
HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
  • HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
    • Home
    • Technology
    • Gaming
    • Entertainment
    • World & Business
    • Science
    • Sports
    • AI
    AI

    Harvey Details Moneyball Approach to AI Research Lab

    Gabe Pereyra described Harvey's moneyball approach during a Sequoia Sovereign AI talk.

    RS
    SM
    PG
    27 Sources, 50d ago, first seen 50d ago

    TLDR

    Gabe Pereyra of Harvey spoke at the Sequoia event about building frontier research capabilities on a startup budget. He said early efforts to match big-lab spending failed and the team shifted to tactics suited to limited capital. Posts from Harvey and attendees list elements such as Legal Agent Bench, an open benchmark, use of domain experts to guide synthetic data, and collaboration with the frontier ecosystem for post-training and experiments. Sequoia posted a playlist of sessions from the event that includes Pereyra's talk along with talks on evals and RL environments.

    Combined views

    522.3K

    27 Sources, first seen 50d ago

    Combined views

    522.3K

    27 Sources, first seen 50d ago

    1.9K likes
    1.9K likes
    89 comments
    2.6K saves
    151 reposts

    Sentiment

    Positive——Negative

    Summary

    Not enough discussion yet.

    No sentiment analysis available yet.

    89 comments
    2.6K saves
    151 reposts
    Today's Rank

    —

    Not ranked yet

    Today's Rank

    —

    Not ranked yet

    Sentiment

    Positive——Negative

    Summary

    Not enough discussion yet.

    No sentiment analysis available yet.

    27 Sources

    @sonyatweetybirdOWN YOUR INTELLIGENCE Last year, building on open-weight models was primarily a cost rationalization exercise. Slightly worse performance for a much cheaper price. Now, it is increasingly an existential and strategic topic for our portfolio. Intelligence is the product. Companies want to shape it and own it and let it compound within their own walls. Not your weights, not your product. Now, with frontier open-weight models and fantastic tooling/infrastructure, owning your intelligence at the frontier is finally becoming possible. The result: every application company we work with is embarking on the journey of doing their own research on post-training, evals, harnesses, etc. The hottest neolabs may just be @Harvey, @FactoryAI, @Ramp, etc. The list goes on. We held a summit @sequoia to convene our portfolio on this topic, together with @gabepereyra (@Harvey) on building Harvey Labs, @lqiao (@FireworksAI_HQ) on post-training, @hwchase17 (@LangChain) on harnesses + evals, @BrendanFoody (@mercor_ai) on RL environments and synthetic data, @QuantumArjun (@trajectorylabs) on online continual learning. Opening talk below; rest to come this week! 00:00 What is sovereign AI (and what it isn't) 01:24 Centralized vs. decentralized intelligence 02:54 Four reasons companies own their models: cost, speed, performance, destiny 04:22 "Not your weights, not your product" 05:32 The application companies are the newest neo labs 07:05 Step 1: Deciding what to own vs. rent 09:51 Step 2: Build the team (and don't shoehorn your platform team) 11:17 Step 3: Legibility – why your research has to be visible 12:33 Step 4: The technical roadmap 13:56 The stack: production vs. development 15:16 Opening Pandora's box – base models, harnesses, context
    @gradypbWant world class research capabilities, but don’t have the resources of a big lab? At our recent Sovereign AI event, @gabepereyra shared @harvey ’s “moneyball” approach. Here’s the playbook: 00:00 Introduction 00:37 Building a research lab on a budget 02:28 Legal Agent Bench, contracting, and the diligence dataset 03:57 Domain experts guiding synthetic data generation 05:23 Why Harvey open sourced its datasets 06:55 Working with the neo labs – and why more than one 08:20 Post-training in-house: building "Associate 1" 09:44 The model serving matrix: 60 countries, fallbacks, SLAs 11:05 Deciding what stays in production 12:29 Simple open source switches and model routing 13:55 Moneyball: "If we win on this budget, we change the game" 14:53 Q&A: Training with sensitive data 17:16 Q&A: Competing for research talent 18:46 Q&A: Designing rubrics that actually challenge frontier models 20:19 Q&A: Where the pipeline breaks — data, research, or infra 22:59 Q&A: The tension in open sourcing a benchmark 25:02 Q&A: Biggest remaining open problems 27:10 Q&A: Competing with horizontal products
    @harvey.@gabepereyra gave a talk to @sequoia founders about Harvey’s approach to research. Gabe shared our research playbook, including: 1) Building Legal Agent Bench, our open source benchmark 2) Leveraging the frontier ecosystem to scale post-training and experimentation with a small team 3) Serving closed / open / post-trained models with enterprise-grade infrastructure We’re sharing our playbook to help the next generation of AI-native builders shape their companies.
    @nikogrupenFrom @sequoia's own your intelligence event. We're starting to see a new category of company emerge, the Full-Stack AI company, where innovation happens at both the product and intelligence layer. That's why @sonyatweetybird led her opening keynote with: "The hottest new labs in my opinion are actually the applied research companies like @harvey @FactoryAI @glean @OpenEvidence @semgrep @tryramp" Proud of what the @harvey research and product teams have have accomplished across benchmarking, post-training, inference-time routing, agent harness optimization, scaling inference, open source, and more -- lots more to come here from this group
    @gabepereyraHad so much fun giving this talk at @sequoia about @harvey’s moneyball approach to building a research lab. The biggest mistake I made in the early days of Harvey was trying to play the Yankees baseball style of frontier intelligence. I found out the hard way that we were the Oakland As - we couldn’t raise the capital or attract the talent to build a frontier lab. However a lot has changed since then and it now feels possible to build frontier intelligence without a frontier budget. Winston and I’s most quoted line from Moneyball is “We can recreate him in the aggregate” when Billy Bean talks about his strategy for building the team The talk outlines our playbook to building frontier intelligence in the aggregate and how we leveraged the frontier ecosystem to do so. This is only now possible with inference providers like @FireworksAI_HQ and @baseten, neolabs like @trajectorylabs, @appliedcompute, and @EngramLab, data providers like @mercor_ai, eval infra like @LangChain and many more. I talk about how we build training data and benchmarks, work with the neolabs and training infra providers to post-train, and give an overview of our serving and eval infra to ensure post trained models work in our product. At the end of Moneyball Billy says that if they don’t win everyone will dismiss this strategy but “if we win, with this budget, and this team, we will have changed the game”. Every application layer company, software company, frontier ecosystem company and startup now has a massive opportunity to play moneyball for frontier intelligence. Go change the game.
    @sequoiaYouTube Playlist here: https://seq.vc/2c2 More content coming this week from @lqiao (@FireworksAI_HQ) on post-training @hwchase17 (@LangChain) on harnesses + evals @BrendanFoody (@mercor_ai) on RL environments and synthetic data @QuantumArjun (@trajectorylabs) on online continual learning
    @shaunmmaguire@gradypb @gabepereyra @harvey Great talk!
    @QuantumArjunAbsolutely fire, lowkey might just send this our potential customers' way before some of our own materials
    @BrendanFoodyRT @sequoia: @gradypb @gabepereyra @harvey YouTube Playlist here: https://seq.vc/2c2 More content coming this week from @lqiao (@Fir…
    @ScobleizerSequoia is seen by many here in Silicon Valley as the best VC here. So I read its reports more carefully than most. This kind of activity leads to more new things to talk about. Which makes me smile.

    27 Sources

    @sonyatweetybirdOWN YOUR INTELLIGENCE Last year, building on open-weight models was primarily a cost rationalization exercise. Slightly worse performance for a much cheaper price. Now, it is increasingly an existential and strategic topic for our portfolio. Intelligence is the product. Companies want to shape it and own it and let it compound within their own walls. Not your weights, not your product. Now, with frontier open-weight models and fantastic tooling/infrastructure, owning your intelligence at the frontier is finally becoming possible. The result: every application company we work with is embarking on the journey of doing their own research on post-training, evals, harnesses, etc. The hottest neolabs may just be @Harvey, @FactoryAI, @Ramp, etc. The list goes on. We held a summit @sequoia to convene our portfolio on this topic, together with @gabepereyra (@Harvey) on building Harvey Labs, @lqiao (@FireworksAI_HQ) on post-training, @hwchase17 (@LangChain) on harnesses + evals, @BrendanFoody (@mercor_ai) on RL environments and synthetic data, @QuantumArjun (@trajectorylabs) on online continual learning. Opening talk below; rest to come this week! 00:00 What is sovereign AI (and what it isn't) 01:24 Centralized vs. decentralized intelligence 02:54 Four reasons companies own their models: cost, speed, performance, destiny 04:22 "Not your weights, not your product" 05:32 The application companies are the newest neo labs 07:05 Step 1: Deciding what to own vs. rent 09:51 Step 2: Build the team (and don't shoehorn your platform team) 11:17 Step 3: Legibility – why your research has to be visible 12:33 Step 4: The technical roadmap 13:56 The stack: production vs. development 15:16 Opening Pandora's box – base models, harnesses, context
    @gradypbWant world class research capabilities, but don’t have the resources of a big lab? At our recent Sovereign AI event, @gabepereyra shared @harvey ’s “moneyball” approach. Here’s the playbook: 00:00 Introduction 00:37 Building a research lab on a budget 02:28 Legal Agent Bench, contracting, and the diligence dataset 03:57 Domain experts guiding synthetic data generation 05:23 Why Harvey open sourced its datasets 06:55 Working with the neo labs – and why more than one 08:20 Post-training in-house: building "Associate 1" 09:44 The model serving matrix: 60 countries, fallbacks, SLAs 11:05 Deciding what stays in production 12:29 Simple open source switches and model routing 13:55 Moneyball: "If we win on this budget, we change the game" 14:53 Q&A: Training with sensitive data 17:16 Q&A: Competing for research talent 18:46 Q&A: Designing rubrics that actually challenge frontier models 20:19 Q&A: Where the pipeline breaks — data, research, or infra 22:59 Q&A: The tension in open sourcing a benchmark 25:02 Q&A: Biggest remaining open problems 27:10 Q&A: Competing with horizontal products
    @harvey.@gabepereyra gave a talk to @sequoia founders about Harvey’s approach to research. Gabe shared our research playbook, including: 1) Building Legal Agent Bench, our open source benchmark 2) Leveraging the frontier ecosystem to scale post-training and experimentation with a small team 3) Serving closed / open / post-trained models with enterprise-grade infrastructure We’re sharing our playbook to help the next generation of AI-native builders shape their companies.
    @nikogrupenFrom @sequoia's own your intelligence event. We're starting to see a new category of company emerge, the Full-Stack AI company, where innovation happens at both the product and intelligence layer. That's why @sonyatweetybird led her opening keynote with: "The hottest new labs in my opinion are actually the applied research companies like @harvey @FactoryAI @glean @OpenEvidence @semgrep @tryramp" Proud of what the @harvey research and product teams have have accomplished across benchmarking, post-training, inference-time routing, agent harness optimization, scaling inference, open source, and more -- lots more to come here from this group
    @gabepereyraHad so much fun giving this talk at @sequoia about @harvey’s moneyball approach to building a research lab. The biggest mistake I made in the early days of Harvey was trying to play the Yankees baseball style of frontier intelligence. I found out the hard way that we were the Oakland As - we couldn’t raise the capital or attract the talent to build a frontier lab. However a lot has changed since then and it now feels possible to build frontier intelligence without a frontier budget. Winston and I’s most quoted line from Moneyball is “We can recreate him in the aggregate” when Billy Bean talks about his strategy for building the team The talk outlines our playbook to building frontier intelligence in the aggregate and how we leveraged the frontier ecosystem to do so. This is only now possible with inference providers like @FireworksAI_HQ and @baseten, neolabs like @trajectorylabs, @appliedcompute, and @EngramLab, data providers like @mercor_ai, eval infra like @LangChain and many more. I talk about how we build training data and benchmarks, work with the neolabs and training infra providers to post-train, and give an overview of our serving and eval infra to ensure post trained models work in our product. At the end of Moneyball Billy says that if they don’t win everyone will dismiss this strategy but “if we win, with this budget, and this team, we will have changed the game”. Every application layer company, software company, frontier ecosystem company and startup now has a massive opportunity to play moneyball for frontier intelligence. Go change the game.
    @sequoiaYouTube Playlist here: https://seq.vc/2c2 More content coming this week from @lqiao (@FireworksAI_HQ) on post-training @hwchase17 (@LangChain) on harnesses + evals @BrendanFoody (@mercor_ai) on RL environments and synthetic data @QuantumArjun (@trajectorylabs) on online continual learning
    @shaunmmaguire@gradypb @gabepereyra @harvey Great talk!
    @QuantumArjunAbsolutely fire, lowkey might just send this our potential customers' way before some of our own materials
    @BrendanFoodyRT @sequoia: @gradypb @gabepereyra @harvey YouTube Playlist here: https://seq.vc/2c2 More content coming this week from @lqiao (@Fir…
    @ScobleizerSequoia is seen by many here in Silicon Valley as the best VC here. So I read its reports more carefully than most. This kind of activity leads to more new things to talk about. Which makes me smile.