A new Oracle PL/SQL benchmark is headed to EMNLP’s Industry Track

The benchmark’s creator says it will be presented as a poster at EMNLP’s Industry Track. They translated MBPP and MBPP+ into PL/SQL to test how models and agents handle simple programming tasks. They later walked back a claim that their quality checks made the MBPP+ translation “nearly perfect,” while saying a score of at least 87% is achievable on MBPP+ PL/SQL and the test harness has no faults.