Announcement
A new Oracle PL/SQL benchmark is headed to EMNLP’s Industry Track
Its creator says they translated MBPP and MBPP+ into PL/SQL using rule-based mappings of unit tests to test models and agents.
TLDR
The benchmark’s creator says it will be presented as a poster at EMNLP’s Industry Track. They translated MBPP and MBPP+ into PL/SQL to test how models and agents handle simple programming tasks. They later walked back a claim that their quality checks made the MBPP+ translation “nearly perfect,” while saying a score of at least 87% is achievable on MBPP+ PL/SQL and the test harness has no faults.
Combined views
1.3K
5 Sources, first seen ago