Reaction
Fresh-task testing for AI agents that edit their own harness
A post warns that bad harness edits can overfit, erase old skills or spread to other users.
TLDR
A post argues that agents editing their own prompts, memory, tools and code should retain changes only when they beat the current setup on fresh, independent tasks. It cites a case compiled from prior work: GPT-5.4-nano with an evolved harness and skills scored 77.58% on SpreadsheetBench, versus 76.20% for GPT-5.5 with skill tuning alone; costs were not matched. The post also recommends keeping evaluators, permissions and emergency stops beyond agents’ reach.
Combined views
—
1 Source, first seen ago
