Cheaper AI inference puts more pressure on the software around the model

As the cost of running AI models falls, the harder product question is moving outward: what happens before a prompt reaches the model and after the model returns an answer?

The New Stack calls that surrounding layer the harness. It includes the context an agent receives, the tools it can call, the way tasks are routed and the checks applied to its output.

The publication ties that argument to several product changes. Zed's Delta organizes agent-written code around shared threads rather than ordinary pull requests. OpenRouter now offers U.S.-only processing routes for business and enterprise customers. Anthropic combined Claude Chat and Cowork into one interface to remove an upfront decision about where a task belongs.

The economics strengthen the case. Vercel's September AI Gateway Production Index said the average price per token fell 23.2% in August, its third monthly decline. Open-weight models accounted for 56% of gateway token volume but 14% of estimated spending, while closed models drew most of the money. Those figures describe Vercel's gateway traffic, not the entire AI market.

Cheaper inference does not guarantee reliable agents. The New Stack cites Real-SWE, a small benchmark using 10 tasks from private company codebases. The best tested setup solved 38.8% of attempts, and none exceeded 40%. Missed requirements and integration failures remained common.