Kilian Lieret

@klieret

Meta Superintelligence, prev. Princeton. SWE-bench multilingual/multimodal, (mini-)SWE-agent, SWE-smith, CodeClash, ProgramBench, SWE-sweep

following
followers

Astra reportedly tops SWE-sweep but finds less than 10% of real-world bugs tested

A post says OpenAI and Anthropic models doubled the number of bugs found while halving costs on SWE-sweep.

4 Sources, 3h ago, first seen 3h ago

DeepSeek Releases V4.1-Flash-1017, Claiming Improved Efficiency and Benchmark Gains

DeepSeek says the model is live on its website, app and API, and optimized for DeepSeekHarness and Tencent CodeBuddy.

147 Sources, 6h ago, updated 6h ago

Some ProgramBench model evaluations reportedly cost over $10,000

A user says evaluating some models on ProgramBench now costs more than $10,000 and shares a ranking based on tasks per dollar.

2 Sources, 22d ago, first seen 22d ago

Muse Spark 1.3 reportedly lifts average test pass rate from 47.0% to 68.6% at xhigh

A user comparing versions 1.1 and 1.3, both at xhigh, says 1.3 used 39% fewer tokens across the benchmark and brought the number of programs at least 95% resolved from eight to 33.

12 Sources, 21d ago, updated 21d ago

Featured in

Astra reportedly tops SWE-sweep but finds less than 10% of real-world bugs tested

DeepSeek Releases V4.1-Flash-1017, Claiming Improved Efficiency and Benchmark Gains

Some ProgramBench model evaluations reportedly cost over $10,000

Muse Spark 1.3 reportedly lifts average test pass rate from 47.0% to 68.6% at xhigh

Featured in

Astra reportedly tops SWE-sweep but finds less than 10% of real-world bugs tested

DeepSeek Releases V4.1-Flash-1017, Claiming Improved Efficiency and Benchmark Gains

Some ProgramBench model evaluations reportedly cost over $10,000

Muse Spark 1.3 reportedly lifts average test pass rate from 47.0% to 68.6% at xhigh