Flawed Regex Creates Rsthon Prompts in MultiPL-E MBPP

Cody Blakeney, a research engineer at Arcee AI, shared examples from the multiPL-E MBPP dataset where a naive regex replace swapped py for rs. The substitution produced prompts asking researchers to write a rsthon function and similar mangled terms such as shthon and swifthon. He questioned how many papers had already reported results from these corrupted evals. Another engineer replied with an additional screenshot of the same pattern. The posts show the errors were visible in the benchmark files for years.