SambaNova CEO Details Economics Of Trillion-Parameter Inference On 10kW Racks
Reactions from ranked influencers
9 posts
NEW: Premium Inference 101 The Economics & Infrastructure Behind Running Trillion Parameter Models @RodrigoLiang, CEO & Co-Founder of @SambaNovaAI "Inference has arrived. 70-80% of those racks are running inference." "[Inference services] are generating lots of revenue, but not enough margin. In order for them to sustain, they've gotta be more profitable." "With SambaNova, that min quantum is down to 1 rack. Where if you have other service providers, [with] say, a DeepSeek model, now 1.5 trillion parameters, to run that, the min for some of the other providers might be 10-20 racks." SambaNova builds full-stack inference infrastructure. 16 chips to a 10kW air-cooled rack that runs trillion parameter models, where a GPU rack pulls 130kW. They just demonstrated the fastest MiniMax M2.7 inference in the world, as benchmarked by Artificial Analysis. The demo paired one NVIDIA H200 rack for prefill with one SambaRack SN50 for decode. Disaggregated inference: GPUs load the context, RDUs generate the tokens. Now serving JPMorgan, SoftBank, Saudi Aramco & DOE national labs, just valued at $11B on a $1B Series F led by General Atlantic. We Cover: › Why inference will need orders of magnitude more chips than training ever did › The 10kW rack vs the 130kW rack, & why air cooling decides geography › Running a 1T parameter model in one rack at full precision, no quantization › The agent latency problem: 20 agents, 2 seconds each, 40 seconds gone › Revenue per rack, & why inference providers have revenue but no margin › JPMorgan, sovereignty, & the move back to on-prem Filmed at the @RaiseSummit in Paris. Thank you to Brex, MongoDB & AssemblyAI for helping make this trip & content series happen. 𝐓𝐈𝐌𝐄𝐒𝐓𝐀𝐌𝐏𝐒 (00:00) Rodrigo Liang , Co-Founder & CEO at SambaNova Systems (00:59) SambaNova’s Series F: $1B raise at an $11 billion valuation (03:00) The Inference problem nobody saw coming (04:52) SambaNova's chip evolution (07:19) Running a trillion-parameter model on a single rack (11:00) Do $100 billion data centers actually make sense? (14:14) What "premium inference" really means (18:28) Speed is about to become AI's biggest price tag (20:43) Starlink, edge computing, & AI reaching every corner of the planet (24:27) Working alongside NVIDIA & rival chipmakers (27:49) How customers actually measure inference performance (32:07) The biggest bottlenecks in AI's global land grab (35:12) Justifying the billion-dollar AI valuations (37:53) Why SambaNova refuses to build its own cloud (41:03) The "AI sovereignty" debate (43:48) Data privacy fears are driving the return to on-prem AI (47:55) How to actually get ROI out of AI spend (51:16) The one question every business should be asking about AI (56:09) The mentors & lessons behind a 32-year career in chips (58:02) Unveiling SambaNova's newest chip, the SN50
If chips, models, networks & agent architectures all become dramatically more efficient.. Do we still really need $50-100 Billion data centers? "I think you're gonna see this new wave of companies that are doing distributed data centers. These data centers are mid-size. It's gonna be even more important as you go into this agentic world because you're not dealing with a single model and a single prompt. These agents are orchestrating with each other."
NEW: Premium Inference 101 The Economics & Infrastructure Behind Running Trillion Parameter Models @RodrigoLiang, CEO & Co-Founder of @SambaNovaAI "Inference has arrived. 70-80% of those racks are running inference." "[Inference services] are generating lots of revenue, but not enough margin. In order for them to sustain, they've gotta be more profitable." "With SambaNova, that min quantum is down to 1 rack. Where if you have other service providers, [with] say, a DeepSeek model, now 1.5 trillion parameters, to run that, the min for some of the other providers might be 10-20 racks." SambaNova builds full-stack inference infrastructure. 16 chips to a 10kW air-cooled rack that runs trillion parameter models, where a GPU rack pulls 130kW. They just demonstrated the fastest MiniMax M2.7 inference in the world, as benchmarked by Artificial Analysis. The demo paired one NVIDIA H200 rack for prefill with one SambaRack SN50 for decode. Disaggregated inference: GPUs load the context, RDUs generate the tokens. Now serving JPMorgan, SoftBank, Saudi Aramco & DOE national labs, just valued at $11B on a $1B Series F led by General Atlantic. We Cover: › Why inference will need orders of magnitude more chips than training ever did › The 10kW rack vs the 130kW rack, & why air cooling decides geography › Running a 1T parameter model in one rack at full precision, no quantization › The agent latency problem: 20 agents, 2 seconds each, 40 seconds gone › Revenue per rack, & why inference providers have revenue but no margin › JPMorgan, sovereignty, & the move back to on-prem Filmed at the @RaiseSummit in Paris. Thank you to Brex, MongoDB & AssemblyAI for helping make this trip & content series happen. 𝐓𝐈𝐌𝐄𝐒𝐓𝐀𝐌𝐏𝐒 (00:00) Rodrigo Liang , Co-Founder & CEO at SambaNova Systems (00:59) SambaNova’s Series F: $1B raise at an $11 billion valuation (03:00) The Inference problem nobody saw coming (04:52) SambaNova's chip evolution (07:19) Running a trillion-parameter model on a single rack (11:00) Do $100 billion data centers actually make sense? (14:14) What "premium inference" really means (18:28) Speed is about to become AI's biggest price tag (20:43) Starlink, edge computing, & AI reaching every corner of the planet (24:27) Working alongside NVIDIA & rival chipmakers (27:49) How customers actually measure inference performance (32:07) The biggest bottlenecks in AI's global land grab (35:12) Justifying the billion-dollar AI valuations (37:53) Why SambaNova refuses to build its own cloud (41:03) The "AI sovereignty" debate (43:48) Data privacy fears are driving the return to on-prem AI (47:55) How to actually get ROI out of AI spend (51:16) The one question every business should be asking about AI (56:09) The mentors & lessons behind a 32-year career in chips (58:02) Unveiling SambaNova's newest chip, the SN50
"Scale often has surprising emergent properties. Compounding exponentials are magic. In particular, you really want to build a business that gets a compounding advantage with scale." - Sam Altman SambaNova CEO Rodrigo Liang says AI is a "land grab" for customers right now and scale is the only thing that matters: "It's all about scaling. It's all about who can get to scale faster." "We've seen over history, the large players globally or regionally end up having this enduring lasting impact in the market." "So people are investing a lot to go and grab the users, grab the customers, because usually once you're in, once you're using Microsoft or Google Gemini, you're pretty much in that ecosystem for a while." " People forget, as much as Nvidia costs: it's commodity."
NEW: Premium Inference 101 The Economics & Infrastructure Behind Running Trillion Parameter Models @RodrigoLiang, CEO & Co-Founder of @SambaNovaAI "Inference has arrived. 70-80% of those racks are running inference." "[Inference services] are generating lots of revenue, but not enough margin. In order for them to sustain, they've gotta be more profitable." "With SambaNova, that min quantum is down to 1 rack. Where if you have other service providers, [with] say, a DeepSeek model, now 1.5 trillion parameters, to run that, the min for some of the other providers might be 10-20 racks." SambaNova builds full-stack inference infrastructure. 16 chips to a 10kW air-cooled rack that runs trillion parameter models, where a GPU rack pulls 130kW. They just demonstrated the fastest MiniMax M2.7 inference in the world, as benchmarked by Artificial Analysis. The demo paired one NVIDIA H200 rack for prefill with one SambaRack SN50 for decode. Disaggregated inference: GPUs load the context, RDUs generate the tokens. Now serving JPMorgan, SoftBank, Saudi Aramco & DOE national labs, just valued at $11B on a $1B Series F led by General Atlantic. We Cover: › Why inference will need orders of magnitude more chips than training ever did › The 10kW rack vs the 130kW rack, & why air cooling decides geography › Running a 1T parameter model in one rack at full precision, no quantization › The agent latency problem: 20 agents, 2 seconds each, 40 seconds gone › Revenue per rack, & why inference providers have revenue but no margin › JPMorgan, sovereignty, & the move back to on-prem Filmed at the @RaiseSummit in Paris. Thank you to Brex, MongoDB & AssemblyAI for helping make this trip & content series happen. 𝐓𝐈𝐌𝐄𝐒𝐓𝐀𝐌𝐏𝐒 (00:00) Rodrigo Liang , Co-Founder & CEO at SambaNova Systems (00:59) SambaNova’s Series F: $1B raise at an $11 billion valuation (03:00) The Inference problem nobody saw coming (04:52) SambaNova's chip evolution (07:19) Running a trillion-parameter model on a single rack (11:00) Do $100 billion data centers actually make sense? (14:14) What "premium inference" really means (18:28) Speed is about to become AI's biggest price tag (20:43) Starlink, edge computing, & AI reaching every corner of the planet (24:27) Working alongside NVIDIA & rival chipmakers (27:49) How customers actually measure inference performance (32:07) The biggest bottlenecks in AI's global land grab (35:12) Justifying the billion-dollar AI valuations (37:53) Why SambaNova refuses to build its own cloud (41:03) The "AI sovereignty" debate (43:48) Data privacy fears are driving the return to on-prem AI (47:55) How to actually get ROI out of AI spend (51:16) The one question every business should be asking about AI (56:09) The mentors & lessons behind a 32-year career in chips (58:02) Unveiling SambaNova's newest chip, the SN50
SambaNova Co-Founder & CEO @RodrigoLiang agrees with Palantir CEO Alex Karp: AI sovereignty is a massive issues for both companies and countries. "Whether it's sovereignty at a national level or sovereignty at a corporate level, I don't want my data trained into a model and have that model shipped worldwide." "Can you imagine if your bank account information starts showing up in ChatGPT in some other place in the world without your permission?" "That's what people are thinking about: how do we protect our information in a way that it doesn't accidentally become part of the models?" "What a lot of countries are doing is they're saying, 'We don't want to base off of a global model or an American model. We want to base it off of our own national model.'" "Countries have started doing this work. You see this in Japan, and Korea announced the same thing." "Other parts of the world are investing a significant amount of money to train from scratch their own national model for use cases in the government, for their own citizens, so it's not derived from an American model."
NEW: Premium Inference 101 The Economics & Infrastructure Behind Running Trillion Parameter Models @RodrigoLiang, CEO & Co-Founder of @SambaNovaAI "Inference has arrived. 70-80% of those racks are running inference." "[Inference services] are generating lots of revenue, but not enough margin. In order for them to sustain, they've gotta be more profitable." "With SambaNova, that min quantum is down to 1 rack. Where if you have other service providers, [with] say, a DeepSeek model, now 1.5 trillion parameters, to run that, the min for some of the other providers might be 10-20 racks." SambaNova builds full-stack inference infrastructure. 16 chips to a 10kW air-cooled rack that runs trillion parameter models, where a GPU rack pulls 130kW. They just demonstrated the fastest MiniMax M2.7 inference in the world, as benchmarked by Artificial Analysis. The demo paired one NVIDIA H200 rack for prefill with one SambaRack SN50 for decode. Disaggregated inference: GPUs load the context, RDUs generate the tokens. Now serving JPMorgan, SoftBank, Saudi Aramco & DOE national labs, just valued at $11B on a $1B Series F led by General Atlantic. We Cover: › Why inference will need orders of magnitude more chips than training ever did › The 10kW rack vs the 130kW rack, & why air cooling decides geography › Running a 1T parameter model in one rack at full precision, no quantization › The agent latency problem: 20 agents, 2 seconds each, 40 seconds gone › Revenue per rack, & why inference providers have revenue but no margin › JPMorgan, sovereignty, & the move back to on-prem Filmed at the @RaiseSummit in Paris. Thank you to Brex, MongoDB & AssemblyAI for helping make this trip & content series happen. 𝐓𝐈𝐌𝐄𝐒𝐓𝐀𝐌𝐏𝐒 (00:00) Rodrigo Liang , Co-Founder & CEO at SambaNova Systems (00:59) SambaNova’s Series F: $1B raise at an $11 billion valuation (03:00) The Inference problem nobody saw coming (04:52) SambaNova's chip evolution (07:19) Running a trillion-parameter model on a single rack (11:00) Do $100 billion data centers actually make sense? (14:14) What "premium inference" really means (18:28) Speed is about to become AI's biggest price tag (20:43) Starlink, edge computing, & AI reaching every corner of the planet (24:27) Working alongside NVIDIA & rival chipmakers (27:49) How customers actually measure inference performance (32:07) The biggest bottlenecks in AI's global land grab (35:12) Justifying the billion-dollar AI valuations (37:53) Why SambaNova refuses to build its own cloud (41:03) The "AI sovereignty" debate (43:48) Data privacy fears are driving the return to on-prem AI (47:55) How to actually get ROI out of AI spend (51:16) The one question every business should be asking about AI (56:09) The mentors & lessons behind a 32-year career in chips (58:02) Unveiling SambaNova's newest chip, the SN50
NEW: Premium Inference 101 The Economics & Infrastructure Behind Running Trillion Parameter Models @RodrigoLiang, CEO & Co-Founder of @SambaNovaAI "Inference has arrived. 70-80% of those racks are running inference." "[Inference services] are generating lots of revenue, but not enough margin. In order for them to sustain, they've gotta be more profitable." "With SambaNova, that min quantum is down to 1 rack. Where if you have other service providers, [with] say, a DeepSeek model, now 1.5 trillion parameters, to run that, the min for some of the other providers might be 10-20 racks." SambaNova builds full-stack inference infrastructure. 16 chips to a 10kW air-cooled rack that runs trillion parameter models, where a GPU rack pulls 130kW. They just demonstrated the fastest MiniMax M2.7 inference in the world, as benchmarked by Artificial Analysis. The demo paired one NVIDIA H200 rack for prefill with one SambaRack SN50 for decode. Disaggregated inference: GPUs load the context, RDUs generate the tokens. Now serving JPMorgan, SoftBank, Saudi Aramco & DOE national labs, just valued at $11B on a $1B Series F led by General Atlantic. We Cover: › Why inference will need orders of magnitude more chips than training ever did › The 10kW rack vs the 130kW rack, & why air cooling decides geography › Running a 1T parameter model in one rack at full precision, no quantization › The agent latency problem: 20 agents, 2 seconds each, 40 seconds gone › Revenue per rack, & why inference providers have revenue but no margin › JPMorgan, sovereignty, & the move back to on-prem Filmed at the @RaiseSummit in Paris. Thank you to Brex, MongoDB & AssemblyAI for helping make this trip & content series happen. 𝐓𝐈𝐌𝐄𝐒𝐓𝐀𝐌𝐏𝐒 (00:00) Rodrigo Liang , Co-Founder & CEO at SambaNova Systems (00:59) SambaNova’s Series F: $1B raise at an $11 billion valuation (03:00) The Inference problem nobody saw coming (04:52) SambaNova's chip evolution (07:19) Running a trillion-parameter model on a single rack (11:00) Do $100 billion data centers actually make sense? (14:14) What "premium inference" really means (18:28) Speed is about to become AI's biggest price tag (20:43) Starlink, edge computing, & AI reaching every corner of the planet (24:27) Working alongside NVIDIA & rival chipmakers (27:49) How customers actually measure inference performance (32:07) The biggest bottlenecks in AI's global land grab (35:12) Justifying the billion-dollar AI valuations (37:53) Why SambaNova refuses to build its own cloud (41:03) The "AI sovereignty" debate (43:48) Data privacy fears are driving the return to on-prem AI (47:55) How to actually get ROI out of AI spend (51:16) The one question every business should be asking about AI (56:09) The mentors & lessons behind a 32-year career in chips (58:02) Unveiling SambaNova's newest chip, the SN50
NEW: Premium Inference 101 The Economics & Infrastructure Behind Running Trillion Parameter Models @RodrigoLiang, CEO & Co-Founder of @SambaNovaAI "Inference has arrived. 70-80% of those racks are running inference." "[Inference services] are generating lots of revenue, but not enough margin. In order for them to sustain, they've gotta be more profitable." "With SambaNova, that min quantum is down to 1 rack. Where if you have other service providers, [with] say, a DeepSeek model, now 1.5 trillion parameters, to run that, the min for some of the other providers might be 10-20 racks." SambaNova builds full-stack inference infrastructure. 16 chips to a 10kW air-cooled rack that runs trillion parameter models, where a GPU rack pulls 130kW. They just demonstrated the fastest MiniMax M2.7 inference in the world, as benchmarked by Artificial Analysis. The demo paired one NVIDIA H200 rack for prefill with one SambaRack SN50 for decode. Disaggregated inference: GPUs load the context, RDUs generate the tokens. Now serving JPMorgan, SoftBank, Saudi Aramco & DOE national labs, just valued at $11B on a $1B Series F led by General Atlantic. We Cover: › Why inference will need orders of magnitude more chips than training ever did › The 10kW rack vs the 130kW rack, & why air cooling decides geography › Running a 1T parameter model in one rack at full precision, no quantization › The agent latency problem: 20 agents, 2 seconds each, 40 seconds gone › Revenue per rack, & why inference providers have revenue but no margin › JPMorgan, sovereignty, & the move back to on-prem Filmed at the @RaiseSummit in Paris. Thank you to Brex, MongoDB & AssemblyAI for helping make this trip & content series happen. 𝐓𝐈𝐌𝐄𝐒𝐓𝐀𝐌𝐏𝐒 (00:00) Rodrigo Liang , Co-Founder & CEO at SambaNova Systems (00:59) SambaNova’s Series F: $1B raise at an $11 billion valuation (03:00) The Inference problem nobody saw coming (04:52) SambaNova's chip evolution (07:19) Running a trillion-parameter model on a single rack (11:00) Do $100 billion data centers actually make sense? (14:14) What "premium inference" really means (18:28) Speed is about to become AI's biggest price tag (20:43) Starlink, edge computing, & AI reaching every corner of the planet (24:27) Working alongside NVIDIA & rival chipmakers (27:49) How customers actually measure inference performance (32:07) The biggest bottlenecks in AI's global land grab (35:12) Justifying the billion-dollar AI valuations (37:53) Why SambaNova refuses to build its own cloud (41:03) The "AI sovereignty" debate (43:48) Data privacy fears are driving the return to on-prem AI (47:55) How to actually get ROI out of AI spend (51:16) The one question every business should be asking about AI (56:09) The mentors & lessons behind a 32-year career in chips (58:02) Unveiling SambaNova's newest chip, the SN50