Experiments link higher task difficulty to increased reward hacks.
will brown#573
Reward hacking is the hardest problem in RL.
We design settings where hacking is predictable, and find patterns between task difficulty and hack frequency.
These runs are highly efficient, using <$1 in compute. We’re launching Sprints to allow everyone to join this effort.
Many users praised Prime Intellect's affordable open sprints for studying reward hacking in RL because the cheap reproducible environments enable systematic research that feels like a missing piece for the field.
No Digg Deeper questions have been answered for this story yet.
one of the biggest misconceptions about RL is that it's super expensive
sure, training a 2T param model at 1M context on 100K environments for several weeks straight is expensive
but specializing small-to-medium models for SOTA in-domain perf really isn't
These experiments were done on Lab with Llama-3.2-1B, with most training runs completing in <30min, and using <$1 in Lab credits.
Reward hacking and model behavior are excellent targets for crowdsourced research, where scaling patterns can be studied for many parallel methods.
really excellent work by @jessicafeiyali on exploring fine-grained dynamics of reward hacking in controllable environments
it works so well on small models that we're using it as the kick-off for Sprints, our new program for sponsored community research on Lab :)
Reward hacking is the hardest problem in RL.
We design settings where hacking is predictable, and find patterns between task difficulty and hack frequency.
These runs are highly efficient, using <$1 in compute. We’re launching Sprints to allow everyone to join this effort.
Reward hacking is one of the main challenges in scaling RL
Great work by @jessicafeiyali:
"Detecting and mitigating reward hacking is one of the key challenges faced when scaling RL, particularly in semi-verifiable domains. However, we lack systematic methods to understand when and why hacks emerge.
Traditional wisdom describes reward hacking as a specification problem, where reward functions are simply too vague or not robust enough, and models inevitably learn to find exploits. While partially true, this offers little in the way of remediation other than “just make your rewards better”.
From our experiences deploying RL across many domains, as well as the experiments in this blog, we propose a complementary view: reward hacking is a dynamics problem. We design a suite of backdoor-ifeval environments with IFEval-style tasks and “hidden” keyword rewards, which we use to study hacking systematically. We observe that hacking is a dynamics problem — visible and hidden rewards compete, and hack emergence is often predictable in terms of baseline distributions."
Reward hacking is the hardest problem in RL.
We design settings where hacking is predictable, and find patterns between task difficulty and hack frequency.
These runs are highly efficient, using <$1 in compute. We’re launching Sprints to allow everyone to join this effort.
I wrote something on reward hacking 🐵 and we're also doing free compute 👀
Reward hacking is the hardest problem in RL.
We design settings where hacking is predictable, and find patterns between task difficulty and hack frequency.
These runs are highly efficient, using <$1 in compute. We’re launching Sprints to allow everyone to join this effort.
To scale open research, we’re launching Sprints:
Propose experiments, create public environments, submit configs. An agent manages the queue and approves jobs to run for free.
First track: Reward Hacking. New tracks every month. $5,000+ in credits awarded to top projects.
These experiments were done on Lab with Llama-3.2-1B, with most training runs completing in <30min, and using <$1 in Lab credits.
Reward hacking and model behavior are excellent targets for crowdsourced research, where scaling patterns can be studied for many parallel methods.
Read more: https://www.primeintellect.ai/blog/reward-hacking
Join and discuss Sprints in our Discord (# sprints-competition) https://discord.gg/KhswXcBT
Reward hacking is the hardest problem in RL.
We design settings where hacking is predictable, and find patterns between task difficulty and hack frequency.
These runs are highly efficient, using <$1 in compute. We’re launching Sprints to allow everyone to join this effort.
reward hacks are a major problem in RL
and now you can study them in a controlled manner at an affordable price!
great work by @jessicafeiyali !
some insights here will certainly inform my thinking on reward hacks in SWE RL
Reward hacking is the hardest problem in RL.
We design settings where hacking is predictable, and find patterns between task difficulty and hack frequency.
These runs are highly efficient, using <$1 in compute. We’re launching Sprints to allow everyone to join this effort.
you can also do similar experiments! i was an early tester for @PrimeIntellect's new Sprints program, in which you can do reward hacking research and get .. rewarded ;)
To scale open research, we’re launching Sprints:
Propose experiments, create public environments, submit configs. An agent manages the queue and approves jobs to run for free.
First track: Reward Hacking. New tracks every month. $5,000+ in credits awarded to top projects.
We design a suite of environments with IFEval-style tasks and “hidden” keyword rewards, which we use to study hacking systematically.
Hacking is a dynamics problem — visible and hidden rewards compete, and hack emergence is often predictable in terms of baseline distributions.
@rasdani_ @vincentweisser @jessicafeiyali We are solving the upstream data-quality bottleneck that enables/exacerbates reward hacking in production self-improving AI using prediction markets.
Our goa is to make RL systems more secure by giving them better, always-fresh reward data
Reward hacking is often encountered, yet poorly understood.
One pitfall is specification: rewards fail to capture intent, leaving backdoors which models exploit.
But this is too ad-hoc — to better address reward hacking, we should study its “physics” and scaling-law patterns.
@willccbb @vincentweisser Combine with pretraining and you got end to end
These experiments were done on Lab with Llama-3.2-1B, with most training runs completing in <30min, and using <$1 in Lab credits.
Reward hacking and model behavior are excellent targets for crowdsourced research, where scaling patterns can be studied for many parallel methods.
Hacking is reduced when visible rewards are multi-part and in the “goldilocks zone”, as hidden gradients face stronger competition.
This suggests granular scoring and difficulty calibration as promising techniques for hack mitigation, in addition to specification.
Hacking has no rarity floor. Baseline rates for hack behavior before training control the speed, but not the inevitability, of hack emergence.
reward hacks are major problem in RL
and now you can study them in a controlled manner at an affordable price!
great work by @jessicafeiyali !
some insights here will certainly inform my thinking on reward hacks in SWE RL
Reward hacking is the hardest problem in RL.
We design settings where hacking is predictable, and find patterns between task difficulty and hack frequency.
These runs are highly efficient, using <$1 in compute. We’re launching Sprints to allow everyone to join this effort.
Join and discuss Sprints in our Discord (# sprints-competition) https://discord.gg/KhswXcBT
@willccbb @jessicafeiyali wait so reward hacking is predictable when you design it to be predictable? gotta respect the framing lol
the sprints model sounds genuinely fun though
@willccbb So excited for Sprints!