Stella Biderman, EleutherAI executive director, accuses Sakana AI of publishing fraudulent performance claims for its Fugu Ultra model

What’s wrong with this post? Well they say “Our ‘Fugu Ultra’ model matches the performance of Fable and Mythos, delivering frontier capability without the risk of export controls.”

but this claim is obviously nonsense. Their model is a closed source router for clouded models

This comment seems to have upset a lot of people unfamiliar with Sakana’s history, so here’s a more detailed description of why I view their claims as false by default and view the company as largely fraudulent.

@BlancheMinerva @SakanaAILabs I had a positive prior on them, but then the series of really bad things they did has completely shifted my prior.

And i still can't reconcile it with David Ha having done nice and very attention-to-detail-y work before Sakana, it's so weird!

At this point you should not trust any claims that @SakanaAILabs makes. Between the blatantly false claim in this tweet and the series of fraudulent “AI Scientist” papers they put out last year, my default assumption is that the opposite of what they say is true.

@BlancheMinerva @SakanaAILabs Just run it you can know the truth, I can say I already see improvement in my own replication. https://github.com/trotsky1997/OpenFugu/tree/main/results

In their AI CUDA Engineer work they reported results that were not just bugged but obviously impossible. As both Tri Dao and I quickly noticed, they claimed speed ups for real workloads that outperformed the theoretical max of the GPUs.

Now let’s talk about the AI scientist saga:

As soon as people started using the system it became clear that the claims about it and the examples they released were not remotely grounded in the model’s actual abilities. Here’s one example: https://arxiv.org/abs/2502.14297v3

Lucas asked o1 to fix the bugs in their core and found it was immediately able to do so, but resulted in showing that the method was actually 3x slower rather than 150x faster. Again, this is for an eval that was obviously bugged because it was reporting impossible results.

@iarthsingh @SakanaAILabs Why doesn’t it make sense to respond to a company frequently publishing fraudulent results and having a PR group that regularly lies by distrusting them? Or do you think one of those claims is false?

Another reason to believe that their AI scientist claims are non-sense: Sakana hasn’t done much good work. If they really had an AI scientist over a year ago, surely they’d have been able to publish important technical research using said scientist!

peer review because they withdrew the paper before it got an accept or reject decision. So maybe its 43rd percentile score would have gotten in. Maybe it wouldn’t. But claiming it passed peer review is just blatantly false.

The paper, by the way, is hot garbage.

and they can’t at all guarantee that the models they route through won’t be export restricted. Some people are also alleging that the evals are unfair, but I haven’t looked into that.

@BlancheMinerva @SakanaAILabs Any empirical evidence to your statements ?

@BlancheMinerva @SakanaAILabs Your statements don’t make sense and are biased I think.

@BlancheMinerva @SakanaAILabs i think the Nature publication counted for a lot in my book, wonder if you mistrust that process also?

@BlancheMinerva @SakanaAILabs is fraud really the right word here? I am aware of the issues with some of their papers but errors and sloppy work is not the same thing

@di_zhang_fdu @BlancheMinerva @SakanaAILabs you built open source for fugu? thats insane! keep the good work man.

anyway you can claimed fees from $OpenFugu though your github

4bF2hKbWG3vrCFBbqiUjx75JHsEZWTxoGSCtmW7Cpump

@tactipus_ @BlancheMinerva Yeah how do you say it’s fraud??

@BlancheMinerva @SakanaAILabs what about ai sci was fraudulent? i missed the discussion

These findings can’t be squared with the claims Sakana made in their marketing materials such as “near human accuracy” in reviewing papers (when tested on 10 papers, it had a 50% precision, 20% recall, and 28.6% F1-score) or the ability to write and run code without human input.