OpenAI releases coding test results for 20 languages and COBOL migration

Two tests, two different results

In MultiLangBench, a model must build a SQLite-compatible database engine from scratch. OpenAI tests the resulting database against a reference SQLite implementation. At maximum reasoning effort, GPT-6 Astra averaged an 87.4% test pass rate across 20 languages, compared with 26.8% for GPT-5.6 Sol. OpenAI reported increases in COBOL from 10.9% to 72.1%, Fortran from 21.4% to 83.5%, and Visual Basic from 27.5% to 89.4%.

The separate migration evaluation asks models to rewrite COBOL programs in C++, Java, Kotlin, Python or Rust. A rewrite counts as successful only if it passes every hidden test, with observable outputs matching byte for byte. At low reasoning effort, OpenAI reported 86.6% accuracy for GPT-6.1 Sol, versus 5.4% for GPT-5.6 Sol at the same setting, at 61% of the API cost. That accuracy figure measures successful migrations, not the test pass rate from MultiLangBench.

Why a passing rewrite still needs review

A program can run without errors yet change a business result. OpenAI gives the example of COBOL arithmetic that discards excess decimal digits unless rounding is requested: a rewrite that rounds instead could produce different financial results. The company warns that applying these capabilities to production software still requires expert review and testing against the system's business rules and dependencies.

OpenAI said developers could try its latest models on coding tasks through its API and that it was releasing the MultiLangBench evaluation setup and test suite on GitHub.

Many accounts questioned OpenAI's claims of major advances in migrating legacy COBOL and Fortran systems, citing undocumented code risks and unconvincing benchmarks, while a few welcomed potential modernization for banks and governments.

Based on 353 sentiment-bearing replies from 334 accounts across 6 conversations.

Many accounts questioned OpenAI's claims of major advances in migrating legacy COBOL and Fortran systems, citing undocumented code risks and unconvincing benchmarks, while a few welcomed potential modernization for banks and governments.

Based on 353 sentiment-bearing replies from 334 accounts across 6 conversations.