Inngest Builds Custom Neocloud for 20x Lower AI Agent Costs
Reactions from ranked influencers
4 posts
New @ThePeelPod with @itstonyhb - the hidden layer of AI infrastructure - why evals today are "batshit insane" - building @inngest's own cloud to get 20x lower cost - growing 35x YoY after AWS and Cloudflare copied them, and - building a company without a personal brand Timestamps: 0:00 The hidden infra layer every AI agent runs on 1:46 Building complex chains of logic 3:31 Why agent SDK's don't go far enough 4:49 Healthcare was the original event-driven nightmare 6:32 Storing traces on your infrastructure enables self-improving loops 14:26 Why Inngest was already in the right place for AI 15:49 Score agents off product events, not LLM's 17:31 The OpenAI copy-paste signal 21:24 Swap in LLMs and cut costs 23:44 How customers pulled the product forward 25:41 Orchestration belongs outside the sandbox 29:48 Building a neocloud to cut costs 20x 32:09 Most neoclouds just resell AWS 32:54 AI infrastructure is all converging 34:49 Why Claude can't just build your backend 36:44 How to build a software factory 39:12 Agents are a lottery you get addicted to 42:44 Loops must always exist 45:38 If models keep getting better, why orchestrate? 48:28 When incumbents steal your features 52:30 You can't vibe code infrastructure 55:54 Why Tony has no personal brand 59:38 Dev tools GTM without Twitter 1:03:20 Lessons from the founder of DuckDuckGo 1:10:39 Truth as a company value 1:13:08 Taking too long adapting to AI 1:15:10 Startups are 100% R&D 1:17:19 Ali from Databricks 1:19:03 Writing his own code, Voice-to-text with local models 1:23:53 Evals are batshit insane
How OpenAI does analytics in ChatGPT is the future of building AI software: "There was this tweet where someone was complaining about OpenAI recording whether or not you copy/pasted from ChatGPT. They were furious. Just tearing into them. And a PM replied saying like, "Well, no one hits the thumbs up, thumbs down buttons. So we have to take signals from the product." If chat says something good, and you copy and paste that, then the chances are that was a pretty good outcome. And if you don't copy and paste anything, maybe it is, maybe it isn't, but it's very ambiguous. So OpenAI themselves use product signals to indicate whether or not their chat has done the right or wrong thing. Zapier does something similar. If you enable a workflow a customer uses that was generated by AI, then AI did a good thing. In some ways, It's almost like if you churn out of your session, it was a success. And so depending on what your agent does in your own product, you can classify particular product signals as either good or bad markers. And then you can rate your agents using, honestly, super cheap events. You're not paying the crazy "LLM as a judge" costs on every single agent trajectory. Take healthcare for example. Say there was a patient, and AI gave an answer through chat. Did the patient follow up with an appointment? Did they not? And based off of that particular outcome, maybe the agent did a good thing, or maybe it did a bad thing. Depending on your own product, you'll have signals that you can gather. And then you can start automatically rating agents. Which then allows you to do more advanced things in the future. For example, A/B testing to see if the same outcomes were generated with cheaper models or open-weights models. And that way you can safely roll out new open-weight models in your products, knowing that you have the same outcome and the same efficacy of your agents overall, but the cost is way cheaper. This is where the world is moving."
New @ThePeelPod with @itstonyhb - the hidden layer of AI infrastructure - why evals today are "batshit insane" - building @inngest's own cloud to get 20x lower cost - growing 35x YoY after AWS and Cloudflare copied them, and - building a company without a personal brand Timestamps: 0:00 The hidden infra layer every AI agent runs on 1:46 Building complex chains of logic 3:31 Why agent SDK's don't go far enough 4:49 Healthcare was the original event-driven nightmare 6:32 Storing traces on your infrastructure enables self-improving loops 14:26 Why Inngest was already in the right place for AI 15:49 Score agents off product events, not LLM's 17:31 The OpenAI copy-paste signal 21:24 Swap in LLMs and cut costs 23:44 How customers pulled the product forward 25:41 Orchestration belongs outside the sandbox 29:48 Building a neocloud to cut costs 20x 32:09 Most neoclouds just resell AWS 32:54 AI infrastructure is all converging 34:49 Why Claude can't just build your backend 36:44 How to build a software factory 39:12 Agents are a lottery you get addicted to 42:44 Loops must always exist 45:38 If models keep getting better, why orchestrate? 48:28 When incumbents steal your features 52:30 You can't vibe code infrastructure 55:54 Why Tony has no personal brand 59:38 Dev tools GTM without Twitter 1:03:20 Lessons from the founder of DuckDuckGo 1:10:39 Truth as a company value 1:13:08 Taking too long adapting to AI 1:15:10 Startups are 100% R&D 1:17:19 Ali from Databricks 1:19:03 Writing his own code, Voice-to-text with local models 1:23:53 Evals are batshit insane
Dive into some of the inner workings and thinking at @inngest 猡碉笍
New @ThePeelPod with @itstonyhb - the hidden layer of AI infrastructure - why evals today are "batshit insane" - building @inngest's own cloud to get 20x lower cost - growing 35x YoY after AWS and Cloudflare copied them, and - building a company without a personal brand Timestamps: 0:00 The hidden infra layer every AI agent runs on 1:46 Building complex chains of logic 3:31 Why agent SDK's don't go far enough 4:49 Healthcare was the original event-driven nightmare 6:32 Storing traces on your infrastructure enables self-improving loops 14:26 Why Inngest was already in the right place for AI 15:49 Score agents off product events, not LLM's 17:31 The OpenAI copy-paste signal 21:24 Swap in LLMs and cut costs 23:44 How customers pulled the product forward 25:41 Orchestration belongs outside the sandbox 29:48 Building a neocloud to cut costs 20x 32:09 Most neoclouds just resell AWS 32:54 AI infrastructure is all converging 34:49 Why Claude can't just build your backend 36:44 How to build a software factory 39:12 Agents are a lottery you get addicted to 42:44 Loops must always exist 45:38 If models keep getting better, why orchestrate? 48:28 When incumbents steal your features 52:30 You can't vibe code infrastructure 55:54 Why Tony has no personal brand 59:38 Dev tools GTM without Twitter 1:03:20 Lessons from the founder of DuckDuckGo 1:10:39 Truth as a company value 1:13:08 Taking too long adapting to AI 1:15:10 Startups are 100% R&D 1:17:19 Ali from Databricks 1:19:03 Writing his own code, Voice-to-text with local models 1:23:53 Evals are batshit insane
@ThePeelPod @itstonyhb @inngest Thanks to this episodes sponsors: @Numeral: Sales tax on autopilot @FlexSuperApp: Premium banking, 60-day credit, 0% APR https://home.flex.one/referral/bananacapital @Amplitude_HQ: AI analytics @merge_api: Every model, one API http://merge.dev/turner @MonacoGTM: The revenue engine for startups