Prime Intellect's Will Brown argues that automating robust evaluation is the essential precursor to safe recursive self-improvement

optimizers and architectures are wonderful nerdsnipes, and RSI will find some cute tweaks for sure, but the big levers are evals, data, and kernels. but data and kernels are evals problems, so it’s really just evals. that, and bringing the damn GPUs online.

@willccbb Did you see this? Thought it was interesting.

https://alignment.openai.com/beneficial-rl/

evals evals evals

at @primeintellect we are hard at work scaling both evals and GPUs for the masses

come hang http://primeintellect.ai/careers

@willccbb totally agree. was just trying to explain to some folks last week that evals are where they should be spending their efforts. you get value from strong evals in a number of different ways.

@willccbb This is why I suspect the interp work at ant is what led to their large improvements. Something something steering vectors during RL to improve exploration and desired behavior.

@willccbb i love working on the things that never make anyone's lists of things!

@willccbb Basically

@leerob nice! very cool results + great signs for alignment generalization

@leerob @willccbb Bruh after cursor will you join spacex?

@willccbb I am not a ML person per se but isn’t the ability to effectively automate evals effectively true AGI?

Not sure if you’re in that camp or not.

@willccbb One thing comes my mind is property based testing if we consider `eval : model :: tests : code`, so basically finding properties/invariants manually but automatically generating eval dataset based on them. Something like that?

@willccbb @PrimeIntellect @willccbb Does remote mean remote from within the US or remote from anywhere?

@willccbb The bottleneck may not be model capability but measurement capability. If we can't reliably evaluate behavior, every capability gain just increases uncertainty faster than confidence. Prime Intellect is betting eval infrastructure scales before intelligence does.

@EdSealing the bitter lesson way to improve exploration in RL is to avoid entropy collapse + sample more

@willccbb Cells Interlinked

@willccbb You mean proof-type formalization? Any interesting work you've seen there?

@willccbb formalize robust eval and half the safety policy arguments dissolve. nobody wants to do the work though

@EdSealing my instinct is that steering vectors function like a more surgical version of prompt conditioning, and are useful for interpreting behavior + example generation, but the big lever is basically pure RL but with really well-calibrated soft rewards

@willccbb i need to adjust this more after a recent Pliny paper but you get the idea

https://app.primeintellect.ai/dashboard/environments/anthone/channel-switching-eval