Reactions from ranked influencers
11 posts
This is super cool:
We had a team of agents rebuild SQLite from its 835-page manual. It created a replica in Rust which passed 100% of a held-out test suite. Interestingly, cost varied 15x depending on which model mix we used.
We had a team of agents rebuild SQLite from its 835-page manual. It created a replica in Rust which passed 100% of a held-out test suite. Interestingly, cost varied 15x depending on which model mix we used.
Incredible! ✨
We had a team of agents rebuild SQLite from its 835-page manual. It created a replica in Rust which passed 100% of a held-out test suite. Interestingly, cost varied 15x depending on which model mix we used.
Great read! SQLite is also represented in ProgramBench, but in this case study, the model is given a massive PRD, no access to a reference executable. The many long horizon coding benchmarks released recently make it an opportune time to study effects of multi-agent.
We had a team of agents rebuild SQLite from its 835-page manual. It created a replica in Rust which passed 100% of a held-out test suite. Interestingly, cost varied 15x depending on which model mix we used.
Fascinating.... especially the massive cost savings by segmenting tasks among different caliber AI models.
We had a team of agents rebuild SQLite from its 835-page manual. It created a replica in Rust which passed 100% of a held-out test suite. Interestingly, cost varied 15x depending on which model mix we used.
Recommended reading. (bookmark it) Pay attention to the prices and what a combination of models can unlock for you. You don't need the absolute best model for everything. I am not that impressed with the Rust SQLite replica. A totally new and improved SQLite (or something more creative and new) could be a more interesting test. Takeaways: - Use a frontier model for decomposition, architecture, key design choices, and trade-offs. - Have cheaper, faster workers execute well-defined, narrow implementation tasks. - Do not have planners implement or workers make broad design decisions. Use a swarm or harness with a recursive task tree: planners break the original spec into subtrees, delegate them, and workers operate at the leaves. Cursor argues this helps mostly because each agent has a constrained context, rather than simply because many agents run in parallel. Pay extra attention to biases and failures that might emerge from poor/inefficient agent coordination.
We had a team of agents rebuild SQLite from its 835-page manual. It created a replica in Rust which passed 100% of a held-out test suite. Interestingly, cost varied 15x depending on which model mix we used.