RealBench Benchmark Tests LLMs on Realistic Repository-Level Coding Tasks
At least a first glance (granted no deep reading was done), this seems to be a more realistic repository-level benchmark compared to Meta's RepoBench (because AI has to build code directly from requirements/specs rather than input/output pairs). https://arxiv.org/abs/2604.22659
Leo Boytsov@srchvrsThe authors evaluate modern LLMs on generating entire repositories or modules within repositories and show that current models struggle significantly with large-scale, structured software engineering tasks. While models can often identify the correct modules and architecture implied by UML diagrams, they frequently produce incorrect, inconsistent, or logically broken implementations (see the attached problem). ↩️
PS: sadly authors use pretty old models, which might not represent well performance of frontier models (GPT 5.5 and the latest Claude).
Leo Boytsov@srchvrsAt least a first glance (granted no deep reading was done), this seems to be a more realistic repository-level benchmark compared to Meta's RepoBench (because AI has to build code directly from requirements/specs rather than input/output pairs). https://arxiv.org/abs/2604.22659