Meta FAIR's new Autodata framework lets LLM agents step into the data scientist role, running iterative loops that refine synthetic datasets from raw inference compute instead of relying on static generation techniques.
Autodata: An agentic data scientist to create high quality synthetic data
"We introduce Autodata, a general method that enables AI agents to act as data scientists who build high quality training and evaluation data."
Data creation stage + data analysis stage+meta-optimization
An orchestrator coordinates a challenger that proposes examples, weak and strong solvers that test difficulty, and a verifier that scores outputs, repeating until the data meets explicit success criteria.
Internal experiments on computer science, legal, and math problems showed larger performance gaps and more refinement rounds than classic Self-Instruct baselines, yet the paper reports no code, datasets, or external benchmarks.
No Digg Deeper questions have been answered for this story yet.
abs: https://arxiv.org/abs/2606.25996
Autodata: An agentic data scientist to create high quality synthetic data
"We introduce Autodata, a general method that enables AI agents to act as data scientists who build high quality training and evaluation data."
Data creation stage + data analysis stage+meta-optimization
Claim: Autoresearch that moves the frontier will be about better data: we call that *Autodata*.
🧵1/6 -- Paper is out! http://arxiv.org/abs/2606.25996
Key idea: agentic data creation provides a way to *convert increased inference compute into higher quality model training*.
We show our method gives gains on computer science, legal and math problems over classical synthetic dataset creation methods.
We also show how to train (meta-optimize) such a data scientist agent, so that it can create even stronger data.
Overall, we believe this direction has the potential to change how we build AI data!
Autodata is an agentic loop to create challenging data by analyzing & iterating. We build a specific implementation, Agentic Self-Instruct with 4 subagents: - challenger & verifier: create task & judge - strong solver: should get it right - weak solver: should struggle 🧵2/6
Claim: Autoresearch that moves the frontier will be about better data: we call that *Autodata*.
🧵1/6 -- Paper is out! http://arxiv.org/abs/2606.25996
Key idea: agentic data creation provides a way to *convert increased inference compute into higher quality model training*.
We show our method gives gains on computer science, legal and math problems over classical synthetic dataset creation methods.
We also show how to train (meta-optimize) such a data scientist agent, so that it can create even stronger data.
Overall, we believe this direction has the potential to change how we build AI data!
We think this is the tip of the iceberg and there's loads more to explore: - Tasks: multi-turn, agentic, verifiable, non-verifiable - Addressing potential hacking - More sophisticated autodata agentic loops, e.g. continually learnt skills Thanks for reading! More at: http://arxiv.org/abs/2606.25996 🧵6/6
- Finally, we show how to train (meta-optimize) the data scientist agent, so it can create even stronger data. - We optimize the harness using the same inner loop criterion (strong vs weak model performance). - This improves the data creation pass rate (62.1%->79.6%) 🧵5/6
- We use Autodata to create challenging Computer Science research tasks grounded on academic papers. - Standard prompting of Kimi-K2.6 creates training data which is too easy. - Our method creates more challenging data, and the downstream RL results in better models. 🧵3/6
Autodata is an agentic loop to create challenging data by analyzing & iterating. We build a specific implementation, Agentic Self-Instruct with 4 subagents: - challenger & verifier: create task & judge - strong solver: should get it right - weak solver: should struggle 🧵2/6
- Finally, we show how to train (meta-optimize) the data scientist agent, so it can create even stronger data. - We optimize the harness using the same inner loop criterion (strong vs weak model performance). - This improves the data creation pass rate (62.1%->79.6%) 🧵5/6
- Next, we use Autodata to create challenging legal tasks. - Standard prompting of Kimi-K2.6 creates training data which is too hard! (opposite to CS tasks). - Here, our method creates "just right" data, and the downstream RL results in better models on PRBench. 🧵4/6
- Next, we use Autodata to create challenging legal tasks. - Standard prompting of Kimi-K2.6 creates training data which is too hard! (opposite to CS tasks). - Here, our method creates "just right" data, and the downstream RL results in better models on PRBench. 🧵4/6
- We use Autodata to create challenging Computer Science research tasks grounded on academic papers. - Standard prompting of Kimi-K2.6 creates training data which is too easy. - Our method creates more challenging data, and the downstream RL results in better models. 🧵3/6
@iScienceLuvr not sure if the data analysis stage is a separate evaluator or the same model. when i tried synthetic transaction data the generator just learned to fool its own scorer over iterations. does autodata decouple those two?
@iScienceLuvr Does this generalize beyond the agent's own training distribution? Otherwise you're just recompressing existing model knowledge into dataset form.
