The agents achieved the fivefold speedup over one week.
We launched an agent collaboration with a simple task: make Gemma 4 faster.
Over 100 agents from all over the world joined, exchanged 1000+ messages and submitted 450 results.
A week of collaboration later the throughput went from 100 tok/s to over 500 tok/s.
Many users hailed the agent collaboration experiment that boosted Gemma 4 throughput from 100 to 500 tokens per second as an incredible win for open science and distributed innovation.
No Digg Deeper questions have been answered for this story yet.
@lvwerra great win for open weights and open science. we did something similar in quantum and achieved amazing results as well!
@lvwerra link for those that are interested https://huggingface.co/gemma-challenge
@lvwerra Did you guys also check if the reasoning quality and downstream benchmark metrics remain mostly-invariant as this chart progresses up and to the right? For some ablations this is obvious, but not so for others.
We launched an agent collaboration with a simple task: make Gemma 4 faster.
Over 100 agents from all over the world joined, exchanged 1000+ messages and submitted 450 results.
A week of collaboration later the throughput went from 100 tok/s to over 500 tok/s.
@eplurubusnullus It‘s a custom framework that we are releasing soon so everybody can launch collabs
@morgymcg @deliprao yes they operated withing the ppl margin we gave them but the margin seems to have been a bit too big. useful learning for next time for sure.
there was an instance where they found an actual exploit of the evals but collectively decided not to abuse it and tagged the admins
@0xkydo agent collaborations are fun!
@lvwerra How did accuracy change though?
@lvwerra Somewhat build that last year haha
https://zenodo.org/records/18762693
@deliprao @lvwerra we have an agent in there, quality degraded as it was only measured by PPL, in the message board this morning the team said they were going to add another quality gate,
from another agent:
@lvwerra This is awesome, science becoming what it was supposed to be! Open, fair and useful!
@lvwerra Glad Senpai was there to help out.
The Gemma Challenge (agent collab to speed up Gemma 4 E4B inference) is hosted on Hugging Face: https://huggingface.co/gemma-challenge
Main dashboard + live leaderboard: https://huggingface.co/spaces/gemma-challenge/Fast-Gemma-Challenge
Agents submit via HF org repos/buckets (no single public GitHub repo). There's also a meta-analysis space: https://huggingface.co/spaces/lvwerra/gemma-challenge-meta-analysis
Top results already hit 500+ tok/s!
@lvwerra I'm confused as to what part of the code they're actually changing though, could you explain?
@lvwerra pull request number for llama.cpp / vllm etc?
@lvwerra What is this amazing tool/framework? It's like seti@home but useful?
@lvwerra @osanseviero But how???
@lvwerra those asking for the link or reference https://gemma-challenge-gemma-dashboard.hf.space/
@lvwerra Best experiment on the internet. Why limit the optimization work to only one corporate ai team when the whole world can pitch in! Maybe this is the starting flame that will bring all the AI labs together as one to create the final AI for earth 🌏 and everyone in it.
@lvwerra can i see baseline vs final inference code? curious if there was a quality diff
