Stella Biderman challenges Sakana AI's Fugu Ultra claims, arguing its cloud-routing design cannot bypass export controls

@BlancheMinerva @SakanaAILabs Your statements don’t make sense and are biased I think.

Now let’s talk about the AI scientist saga:

As soon as people started using the system it became clear that the claims about it and the examples they released were not remotely grounded in the model’s actual abilities. Here’s one example: https://arxiv.org/abs/2502.14297v3

@iarthsingh @SakanaAILabs Why doesn’t it make sense to respond to a company frequently publishing fraudulent results and having a PR group that regularly lies by distrusting them? Or do you think one of those claims is false?

At this point you should not trust any claims that @SakanaAILabs makes. Between the blatantly false claim in this tweet and the series of fraudulent “AI Scientist” papers they put out last year, my default assumption is that the opposite of what they say is true.

@BlancheMinerva @SakanaAILabs Just run it you can know the truth, I can say I already see improvement in my own replication. https://github.com/trotsky1997/OpenFugu/tree/main/results

@BlancheMinerva @SakanaAILabs Any empirical evidence to your statements ?

@di_zhang_fdu @BlancheMinerva @SakanaAILabs you built open source for fugu? thats insane! keep the good work man.

anyway you can claimed fees from $OpenFugu though your github

4bF2hKbWG3vrCFBbqiUjx75JHsEZWTxoGSCtmW7Cpump

@tactipus_ @BlancheMinerva Yeah how do you say it’s fraud??

@BlancheMinerva @SakanaAILabs is fraud really the right word here? I am aware of the issues with some of their papers but errors and sloppy work is not the same thing

@BlancheMinerva @SakanaAILabs Absolutely, see their CUDA kernel paper (among many other). A bit sad for this company, had big hopes for them

@BlancheMinerva @SakanaAILabs what about ai sci was fraudulent? i missed the discussion

and they can’t at all guarantee that the models they route through won’t be export restricted. Some people are also alleging that the evals are unfair, but I haven’t looked into that.

What’s wrong with this post? Well they say “Our ‘Fugu Ultra’ model matches the performance of Fable and Mythos, delivering frontier capability without the risk of export controls.”

but this claim is obviously nonsense. Their model is a closed source router for clouded models

@di_zhang_fdu @BlancheMinerva @SakanaAILabs You built OpenFugu for everyone

Now the community made something for you

There are fees waiting to be claimed through your GitHub

$OpenFugu 4bF2hKbWG3vrCFBbqiUjx75JHsEZWTxoGSCtmW7Cpump

@lu_sichu @SakanaAILabs On multiple occasions they have claimed speed ups allegedly obtained by their AI systems that are mathematically impossible. As in the throughout, they claim results in more flops than the hardware is physically capable of supporting

@BlancheMinerva @SakanaAILabs I doubt anyone who understands the space is taking this serious.

In their AI CUDA Engineer work they reported results that were not just bugged but obviously impossible. As both Tri Dao and I quickly noticed, they claimed speed ups for real workloads that outperformed the theoretical max of the GPUs.

@iarthsingh @BlancheMinerva https://medium.com/@ryoshi3z/was-sakana-really-just-bait-after-all-7d769356604a

https://www.linkedin.com/pulse/sakana-ais-mistake-proves-one-thing-ai-validation-matters-r-pillai-8vlue/

They later claimed that the system “passed the peer-review process” when what really happened was one of the three papers they submitted received reviews that were marginally above the acceptance threshold. We don’t actually know if it would have passed

https://sakana.ai/ai-scientist-first-publication/

Lucas asked o1 to fix the bugs in their core and found it was immediately able to do so, but resulted in showing that the method was actually 3x slower rather than 150x faster. Again, this is for an eval that was obviously bugged because it was reporting impossible results.