A reposted comment calls the related paper potentially “the most important paper on AI agents this month.”
A reposted comment says CMU researchers trained a small model and calls their AI-agents paper potentially the most important on the topic this month.
—
2 Sources, first seen ago
—
Not ranked yet
—
Not ranked yet
@undefinedKiThis might be the most important paper on AI agents this month. Researchers at CMU trained a small model to rewrite an AI agent’s harness, and it beat the 35B model it learned from. The harness is the program around the model: how it gets called, which tools run, where the information goes. Here a 4B model reads a report from the last run, proposes edits to the harness, and keeps the one that scores best. The model doing the task never changes. What they found: > on task types it never saw, edit quality went from 0.32 to 0.62 > 5 edits in a row matched or beat the best of 8 parallel tries > broken edits dropped from 38% to 15% You can run the same loop today with any agent: 1. Put your harness in one file: prompts, steps, tool calls 2. Split your test cases into three sets: learn, score, final check 3. Run it on the learn set and save a short report of what failed 4. Ask a model for 5 edits based on that report 5. Run each edit on the score set and keep the winner 6. Repeat for 5 rounds, then test once on the final set Key pages: p.2: the full loop p.6: results on unseen tasks p.8: a harness the model learned
@irinarishRT @undefinedKi: This might be the most important paper on AI agents this month. Researchers at CMU trained a small model to rewrite an AI…@undefinedKiThis might be the most important paper on AI agents this month. Researchers at CMU trained a small model to rewrite an AI agent’s harness, and it beat the 35B model it learned from. The harness is the program around the model: how it gets called, which tools run, where the information goes. Here a 4B model reads a report from the last run, proposes edits to the harness, and keeps the one that scores best. The model doing the task never changes. What they found: > on task types it never saw, edit quality went from 0.32 to 0.62 > 5 edits in a row matched or beat the best of 8 parallel tries > broken edits dropped from 38% to 15% You can run the same loop today with any agent: 1. Put your harness in one file: prompts, steps, tool calls 2. Split your test cases into three sets: learn, score, final check 3. Run it on the learn set and save a short report of what failed 4. Ask a model for 5 edits based on that report 5. Run each edit on the score set and keep the winner 6. Repeat for 5 rounds, then test once on the final set Key pages: p.2: the full loop p.6: results on unseen tasks p.8: a harness the model learned@irinarishRT @undefinedKi: This might be the most important paper on AI agents this month. Researchers at CMU trained a small model to rewrite an AI…@undefinedKiThis might be the most important paper on AI agents this month. Researchers at CMU trained a small model to rewrite an AI agent’s harness, and it beat the 35B model it learned from. The harness is the program around the model: how it gets called, which tools run, where the information goes. Here a 4B model reads a report from the last run, proposes edits to the harness, and keeps the one that scores best. The model doing the task never changes. What they found: > on task types it never saw, edit quality went from 0.32 to 0.62 > 5 edits in a row matched or beat the best of 8 parallel tries > broken edits dropped from 38% to 15% You can run the same loop today with any agent: 1. Put your harness in one file: prompts, steps, tool calls 2. Split your test cases into three sets: learn, score, final check 3. Run it on the learn set and save a short report of what failed 4. Ask a model for 5 edits based on that report 5. Run each edit on the score set and keep the winner 6. Repeat for 5 rounds, then test once on the final set Key pages: p.2: the full loop p.6: results on unseen tasks p.8: a harness the model learned@irinarishRT @undefinedKi: This might be the most important paper on AI agents this month. Researchers at CMU trained a small model to rewrite an AI…@undefinedKiThis might be the most important paper on AI agents this month. Researchers at CMU trained a small model to rewrite an AI agent’s harness, and it beat the 35B model it learned from. The harness is the program around the model: how it gets called, which tools run, where the information goes. Here a 4B model reads a report from the last run, proposes edits to the harness, and keeps the one that scores best. The model doing the task never changes. What they found: > on task types it never saw, edit quality went from 0.32 to 0.62 > 5 edits in a row matched or beat the best of 8 parallel tries > broken edits dropped from 38% to 15% You can run the same loop today with any agent: 1. Put your harness in one file: prompts, steps, tool calls 2. Split your test cases into three sets: learn, score, final check 3. Run it on the learn set and save a short report of what failed 4. Ask a model for 5 edits based on that report 5. Run each edit on the score set and keep the winner 6. Repeat for 5 rounds, then test once on the final set Key pages: p.2: the full loop p.6: results on unseen tasks p.8: a harness the model learned@irinarishRT @undefinedKi: This might be the most important paper on AI agents this month. Researchers at CMU trained a small model to rewrite an AI…
