https://www.vals.ai/benchmarks/programbench

@scaling01 Lives up to its name ig

@scaling01 You can't lose a battle if you refuse to fight it!

@scaling01 @usr_bin_roygbiv

@scaling01 Absolutely should not be rated as NA. Should be rated at 0

@scaling01 and still they CHARGE for it

@scaling01 Guardrails so high we can’t even use it for anything except one prompting knock off games.

@scaling01 Lollll

@scaling01 Acc is 0/0, we've literally reached singularity.

@scaling01 I haven’t been able to get it to do any besides answer basic ass questions

@scaling01 haha similar to my experience with the model thus far

unfortunately, because when it doesn't refuse its great actually

@scaling01 You can't beat someone who is not competing. Smart move from Antrophic.

@scaling01 “you can just not do things” -claude fable 5

@scaling01 Hey, at least it was nice enough not to charge you 🤣

@scaling01

@scaling01 wait why would you run fable on benchmarks??

Wouldn't the whole thing leak?

@scaling01 @claudeai @AnthropicAI are geniuses to come up with Fable yet they are extra paranoid and greedy at the same time.

@scaling01 Only if @elder_plinius released his prompt we would actually be getting proper benchmarks on this thing.

@scaling01 It still leads in every other benchmark that http://Vals.ai has for coding so one could assume it still would win there too?

@scaling01 Accuracy 0%