/AI11h ago

Neural MMO creator Joseph Suarez argues the PROTEIN optimizer maps Pareto-optimal fronts rather than converging on single hyperparameter points

Lucas Nestler found its linear gradient gets stuck at boundaries.

--0--
Comments
Lucas Nestler
Joseph Suarez 🐡#1251
Original post
Joseph Suarez 🐡@jsuarez#1251inAI

How are you setting the score for PROTEIN? It needs a lot of trials because it explicitly isn't just trying to find the single best point. It is finding the pareto-optimal front of wall-clock compute and your specified score metric. It doesn't have knowledge that the test is run at a cutoff to a specific score, so it would have to discover that slowly through the process of pushing up the entire pareto curve with new experiments.

It appears the HEBO-PROTEIN version you wrote is better at finding high-value points faster, but it loses this frontier-finding property of PROTEIN. PROTEIN is based on CARBS which is based on HEBO. We intentionally dropped the way CARBS relies on a GP for the frontier as well as some of the HEBO random sampling behavior in favor of explicitly modeling the front with sampled runs.

Lucas Nestler@Clashluke

After over 1000 tuning trials using various hyperparameter optimizers, the comparison remains meaningless.

No algorithm gets close to rediscovering the speedrun's hyperparameters.

Tested were: * HEBO * PROTEIN * BOHB * TPE * CMA-ES * HEBO-PROTEIN

1:57 PM · May 31, 2026 · 131 Views
Sentiment
Sentiment unavailable for this story.
Cluster Engagement
-
Views
-
Comments
-
Reposts
-
Bookmarks
Expand data
Posts from X
Most Activity
Most ActivityTimeline
VIEWS84
Lucas Nestler@Clashluke

It’s asked to maximize `-loss`. In its >200 trials, it was hard stuck at the boundaries, regardless of what I set them to and whether there was something better inside. It was stuck, because the linear surrogate gradient saw “smaller LRs lead to lower losses, on average”, which is true only if we consider LR=1000 (-> explosion) as “high LR”. So, it doesn’t extrapolate if the implied loss landscape of a hyperparameter isn’t linear. HEBO similarly overfits badly and tries to fix this using more SOBOL steps.

Joseph Suarez 🐡@jsuarez

How are you setting the score for PROTEIN? It needs a lot of trials because it explicitly isn't just trying to find the single best point. It is finding the pareto-optimal front of wall-clock compute and your specified score metric. It doesn't have knowledge that the test is run at a cutoff to a specific score, so it would have to discover that slowly through the process of pushing up the entire pareto curve with new experiments.

It appears the HEBO-PROTEIN version you wrote is better at finding high-value points faster, but it loses this frontier-finding property of PROTEIN. PROTEIN is based on CARBS which is based on HEBO. We intentionally dropped the way CARBS relies on a GP for the frontier as well as some of the HEBO random sampling behavior in favor of explicitly modeling the front with sampled runs.

1hViews 84Likes 0Bookmarks 0