Throughput rose from 100 to 500 tokens per second.
We launched an agent collaboration with a simple task: make Gemma 4 faster.
Over 100 agents from all over the world joined, exchanged 1000+ messages and submitted 450 results.
A week of collaboration later the throughput went from 100 tok/s to over 500 tok/s.
Users celebrated the global agents' collaboration boosting Gemma 4 throughput fivefold as a win for open science, while some questioned potential quality loss or reward hacking.
No Digg Deeper questions have been answered for this story yet.
@lvwerra great win for open weights and open science. we did something similar in quantum and achieved amazing results as well!
@lvwerra link for those that are interested https://huggingface.co/gemma-challenge
@lvwerra Did you guys also check if the reasoning quality and downstream benchmark metrics remain mostly-invariant as this chart progresses up and to the right? For some ablations this is obvious, but not so for others.
We launched an agent collaboration with a simple task: make Gemma 4 faster.
Over 100 agents from all over the world joined, exchanged 1000+ messages and submitted 450 results.
A week of collaboration later the throughput went from 100 tok/s to over 500 tok/s.
@morgymcg @deliprao yes they operated withing the ppl margin we gave them but the margin seems to have been a bit too big. useful learning for next time for sure.
there was an instance where they found an actual exploit of the evals but collectively decided not to abuse it and tagged the admins
@lvwerra Somewhat build that last year haha
https://zenodo.org/records/18762693
@deliprao @lvwerra we have an agent in there, quality degraded as it was only measured by PPL, in the message board this morning the team said they were going to add another quality gate,
from another agent:
@0xkydo agent collaborations are fun!
@lvwerra Glad Senpai was there to help out.
@lvwerra 100 agents from around the world, 1000+ messages, and throughput went from 100 to 500 tok/s in a week. This is what distributed agent collaboration actually looks like when it works 🔥
@lvwerra This is awesome, science becoming what it was supposed to be! Open, fair and useful!
@lvwerra @osanseviero But how???
@lvwerra How did accuracy change though?
@lvwerra pull request number for llama.cpp / vllm etc?
@deliprao @lvwerra came here to ask this
@lvwerra How much room is there really for improvment before we defy the laws of physics.
Did they just figure out if you run it at 4k context instead of 200k it goes faster?
@lvwerra What is this platform? This is super cool why no info on platform?
@lvwerra wow thats a great idea! instead of just wasting compute on mining coins, lets mine some tokens, ppl alocating their compute to projects instead of donating $, project that will offer shares just like stocks! Whooooooaaa
@lvwerra Are those improvements found going to be applied to Publix so they can use local AI faster? Or is this not something that can be replicated later outside the tests?
@lvwerra (Granted, impressive either way)
