Granular SQL verification steps failed to improve GLM's correctness
"GLM produces cleaner code" — ❌ Not supported
Pass@1 is 6 pp lower. More verification ≠ more correct.
"GLM verifies more" — ✅ Partially confirmed
But it's atomized differently. GLM fires one sql_execute per check. Opus batches the same checks into fewer dbt show --inline calls. Same coverage, different shape.
Users are excited about GLT 5.2 because it represents promising advancements they want to integrate into tools like Coco's harness for customer delivery.
No Digg Deeper questions have been answered for this story yet.
"GLM verifies more" — ✅ Partially confirmed
But it's atomized differently. GLM fires one sql_execute per check. Opus batches the same checks into fewer dbt show --inline calls. Same coverage, different shape.
"GLM uses 2× more tokens" — ✅ Confirmed
860M vs 439M billing tokens. More turns + atomic API calls + lower prompt-cache reuse (53% vs 96%).
On tasks both models solve, GLM uses ~17% more calls — not 2×.
The 2× framing is a whole-run average driven by tail tasks where GLM spirals. Not representative of typical behavior.
GLM failure mode #2: Over-verification of wrong axes.
One task: 411 tool calls, 24 minutes. Checked row counts, distributions, nulls, column types, DuckDB/Snowflake parity. Failed 0/3.
Opus solved it with 49 calls in 9 min.
The real GLM edge: dual-platform validation.
The spec requires passing both DuckDB and Snowflake. GLM more consistently validates both targets. This is the causal factor behind several GLM-only wins.
"GLM produces cleaner code" — ❌ Not supported
Pass@1 is 6 pp lower. More verification ≠ more correct.
GLM failure mode #1: Early give-up.
When GLM can't see the write path from reads alone, it exits without attempting. One task: 22 turns, 5 file reads, 0 writes, stop.
The real GLM edge: dual-platform validation.
The spec requires passing both DuckDB and Snowflake. GLM more consistently validates both targets. This is the causal factor behind several GLM-only wins.
Bottom line: Verification volume doesn't predict correctness.
GLM's worst losses come from verifying the wrong axes exhaustively. Its other failure mode — early give-up — is orthogonal to verification quantity.
On tasks both models solve, GLM uses ~17% more calls — not 2×.
The 2× framing is a whole-run average driven by tail tasks where GLM spirals. Not representative of typical behavior.
GLM failure mode #2: Over-verification of wrong axes.
One task: 411 tool calls, 24 minutes. Checked row counts, distributions, nulls, column types, DuckDB/Snowflake parity. Failed 0/3.
Opus solved it with 49 calls in 9 min.
GLM failure mode #1: Early give-up.
When GLM can't see the write path from reads alone, it exits without attempting. One task: 22 turns, 5 file reads, 0 writes, stop.
Overall though, we are super excited for what GLT 5.2 represents and can't wait to tune Coco's harness more for it and to get it in front of our customers!
