Harness Hub announced a Pi-specific LoRA adapter for Qwen3.8-27B on November 17, 2026, reporting higher task-success rates and 55β62% fewer tool calls per task across two benchmark comparisons. The results come from its own evaluations with the modelβs thinking mode off, and benchmark scores helped select the released checkpoint. SourceSource
A harness is the coding-agent interface and workflow around the model. Here, the idea is to train the model for the interface people want to use, rather than asking them to switch to accommodate the model. As announcement participant @LysandreJik put it: βkeep the harness, adapt the model.β The LoRA is a trained add-on for Qwen3.8-27B, not a replacement base model. The model gets homework; the user gets to keep Pi. SourceSource
What the benchmark gains actually cover
The model card reports pass@1 results, meaning one attempt per task, comparing the base model and adapter under Pi with thinking mode off. Temperature, the setting that controls output randomness, was 0.8. Thinking was also off during training. Source
The team chose checkpoint 160, a saved model version, using scores on SWE-bench Pro and an 18-task subset of Terminal-Bench 2.1. The headline benchmarks therefore werenβt kept untouched during checkpoint selection. That does not establish that their tasks were training examples. Source
All figures are reported by the model card. The tool-call reductions compare average calls per task across each evaluation, including unsuccessful runs. They donβt mean every individual task used 55β62% fewer calls. The SWE-bench result covers the hard-51 subset, not the entire benchmark. Source
The cardβs Terminal-Bench table lists 86 tasks, while its prose says βall 89 tasks.β The captured card does not explain the three-task difference. Source
For the 68 Terminal-Bench tasks not used to choose the checkpoint, the card separately reports success rising from 39.7% to 55.9%. Exclusion from checkpoint selection does not establish independence from all training data. Source
Fewer tool calls are welcome; they are not a stopwatch or a bill. The card supplies no measured time or monetary savings, and the available material includes no independent replication. Source
Training, deployment and what comes next
According to the announcement thread, each reinforcement-learning trial, or rollout, was a real Pi session in a Hugging Face sandbox, rather than a reimplementation of Piβs agent loop inside the trainer. The model card says training rewarded correct task completion and included a tool-efficiency bonus, making fewer calls an explicit objective. SourceSource
On a task fixing Ansibleβs human_to_bytes function, the card says the base model reached the 100-step limit, while the adapted model fixed the problem in eight tool calls. Less rummaging, more repairing is an appealing outcome. One showcase task is not a promise that the next bug takes eight calls. Source
For deployment, the card documents ways to run it with vLLM, transformers serve, llama.cpp and Llama.app on macOS. The adapter requires Qwen3.8-27B and does not work with other base models. Training used BF16, a 16-bit number format, while the GGUF version packaged for local tools such as llama.cpp has not been re-evaluated. Those local-running options are no guarantee of identical benchmark scores. Source
In its November 17 announcement thread, the team said more harness-specific adapters would follow and promised a comprehensive recipe post for Friday, November 20, 2026. SourceSource
Comments