• Home
    • Technology
    • Gaming
    • Entertainment
    • World & Business
    • Science
    • Sports
    • AI
    HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
    AI
    Reaction

    Quail and the case for specialized AI-SQL inference

    A user reports that a query took 6.84 hours with hand-tuned vLLM, versus a 15-minute “speed-of-light” estimate.

    Shreya ShankarSS
    Rohit RastogiRR
    Josh RosenJR
    55 Sources, 20d ago, first seen 20d ago

    TLDR

    A user reports substantial host overhead on several queries run through hand-tuned vLLM baselines. They say vLLM also discarded cached data it later needed, causing 50 million extra tokens of processing. One query took 6.84 hours, compared with a 15-minute “speed-of-light” estimate. The user argues that AI-SQL warrants specialized inference: nearly all requests are known up front, allowing better cache planning, and the priority is finishing the entire query quickly rather than serving interactive requests. They say those differences motivated Quail, a specialized inference engine.

    Combined views

    271.5K

    55 Sources, first seen 20d ago

    1.4K likes59 comments1.6K saves464 reposts

    Combined views

    271.5K

    55 Sources, first seen 20d ago

    1.4K likes59 comments1.6K saves464 reposts

    Sentiment

    Positive80.7%19.3%Negative

    Summary

    Many accounts welcomed Quail’s co-planning of queries with LLM inference and its agent design on existing code because it enables high throughput like 1B tokens per minute on one H100, while some flagged added costs.

    Based on 116 sentiment-bearing replies from 86 accounts across 14 conversations.

    Featured Source

    Sentiment

    Positive80.7%19.3%Negative

    Summary

    Many accounts welcomed Quail’s co-planning of queries with LLM inference and its agent design on existing code because it enables high throughput like 1B tokens per minute on one H100, while some flagged added costs.

    Based on 116 sentiment-bearing replies from 86 accounts across 14 conversations.

    55 Sources

    Shreya Shankar@sh_reyaAI-SQL is such a special case workload, where you know nearly all requests up front (and can plan KV a lot better), and you care more about throughput, or executing the *entire* query as quickly as possible. And, requests are entirely prefill. Not like the interactive request setting that general-purpose inference engines are optimized for. Turns out it makes sense to build a specialized inference engine here, motivating Quail.20d
    Rohit Rastogi@RohitRastoamazing stuff! really deep integration between inference engine and query engine. they use vllm internals as a library, but replace the general purpose serving scheduler with one designed for batched, high throughput, prefill only queries used for ai filters/joins. their custom scheduler allows for workload aware kv cache management and their custom execution layer has attention kernel implementations that avoid repeatedly reading the same shared kvs across multiple queries. reminds me a bit of workload aware inference schedulers for agent workloads (https://arxiv.org/pdf/2511.02230) that have session aware kv ttls to avoid evicting cache during environment interactions. will likely see more custom inference schedulers as we get more creative with how llms are used. great timing for this project, especially with the launch of jev, showing the utility of classification in llm workflows. there are so many obvious applications of infra like this for production agent observability, which can be used for harness improvement and data curation for post training. fenic took a lot of inspiration from docetl and lotus. we were primarily focused on exploring the ai dataframe/query interface and barely scratched the surface with logical plan optimizations. excited to see where ai in data processing libraries/dbs go as the physical execution layer become more specialized too. wonder the extent to which there is room for datafusion integration to enable joint optimization across the inference engine and query engine.20d
    Josh Rosen@JoshARosenAmazing example of co-engineering a harness, in this case a specialized SQL-engine-shaped harness, along with the LLM. And the results speak for themselves. They optimized the deterministic planning code with the interference-based execution for unmatched results. Not to mention very cool syntax being able to mix semantics IFs with conventional SQL.20d
    » teej@teej_mwoah20d
    Aditya Parameswaran@adityagpAll existing AI-SQL systems focus on optimizations at the logical + physical level, leaving the inference stack alone - some exciting new work from @sh_reya and @charles_irl shows there is lots of low hanging fruit in that layer. Awesome!20d
    Venkat Krishnamurthy (primary: Bluesky)@nivikshaSuper cool!! Can’t wait to try this out - cc @ToddMostak20d
    Shadaj Laddad@ShadajLAnd now @sh_reya is presenting the world premiere of Quail, a domain-specific LLM inference engine for AI-SQL!20d
    Qian Li@qianl_csVery cool to see query planning/execution and inference engine co-design. @sh_reya20d
    Connor Shorten@CShorten30RT @qianl_cs: Very cool to see query planning/execution and inference engine co-design. @sh_reya20d
    Fergus Finn@finn_fergusWas great to collaborate with @sh_reya and @charles_irl on this awesome project. I think there are still lots of great opportunities out there for 'pushing down' workloads further into the inference stack19d
    • Home
    • Technology
    • Gaming
    • Entertainment
    • World & Business
    • Science
    • Sports
    • AI

    55 Sources

    Shreya Shankar@sh_reyaAI-SQL is such a special case workload, where you know nearly all requests up front (and can plan KV a lot better), and you care more about throughput, or executing the *entire* query as quickly as possible. And, requests are entirely prefill. Not like the interactive request setting that general-purpose inference engines are optimized for. Turns out it makes sense to build a specialized inference engine here, motivating Quail.20d
    Rohit Rastogi@RohitRastoamazing stuff! really deep integration between inference engine and query engine. they use vllm internals as a library, but replace the general purpose serving scheduler with one designed for batched, high throughput, prefill only queries used for ai filters/joins. their custom scheduler allows for workload aware kv cache management and their custom execution layer has attention kernel implementations that avoid repeatedly reading the same shared kvs across multiple queries. reminds me a bit of workload aware inference schedulers for agent workloads (https://arxiv.org/pdf/2511.02230) that have session aware kv ttls to avoid evicting cache during environment interactions. will likely see more custom inference schedulers as we get more creative with how llms are used. great timing for this project, especially with the launch of jev, showing the utility of classification in llm workflows. there are so many obvious applications of infra like this for production agent observability, which can be used for harness improvement and data curation for post training. fenic took a lot of inspiration from docetl and lotus. we were primarily focused on exploring the ai dataframe/query interface and barely scratched the surface with logical plan optimizations. excited to see where ai in data processing libraries/dbs go as the physical execution layer become more specialized too. wonder the extent to which there is room for datafusion integration to enable joint optimization across the inference engine and query engine.20d
    Josh Rosen@JoshARosenAmazing example of co-engineering a harness, in this case a specialized SQL-engine-shaped harness, along with the LLM. And the results speak for themselves. They optimized the deterministic planning code with the interference-based execution for unmatched results. Not to mention very cool syntax being able to mix semantics IFs with conventional SQL.20d
    » teej@teej_mwoah20d
    Aditya Parameswaran@adityagpAll existing AI-SQL systems focus on optimizations at the logical + physical level, leaving the inference stack alone - some exciting new work from @sh_reya and @charles_irl shows there is lots of low hanging fruit in that layer. Awesome!20d
    Venkat Krishnamurthy (primary: Bluesky)@nivikshaSuper cool!! Can’t wait to try this out - cc @ToddMostak20d
    Shadaj Laddad@ShadajLAnd now @sh_reya is presenting the world premiere of Quail, a domain-specific LLM inference engine for AI-SQL!20d
    Qian Li@qianl_csVery cool to see query planning/execution and inference engine co-design. @sh_reya20d
    Connor Shorten@CShorten30RT @qianl_cs: Very cool to see query planning/execution and inference engine co-design. @sh_reya20d
    Fergus Finn@finn_fergusWas great to collaborate with @sh_reya and @charles_irl on this awesome project. I think there are still lots of great opportunities out there for 'pushing down' workloads further into the inference stack19d
    Today's Rank

    —

    Not ranked yet

    Today's Rank

    —

    Not ranked yet