Reactions from ranked influencers
8 posts
@a1zhang @PrimeIntellect I had trouble understanding this one. * Does base Transformers get to use tools? * Does the RLM get strictly less conditioning than a standard harness?
The experiments compare an LLM to an LLM with a harness (not just a python interpreter, which likely doesn't do anything on its own for these tasks). The tasks are set up so the eval setup has some explicit difference w.r.t. the train setup, namely either the length of the task or the topic of the task. On paper if you ignore any details about the harness or its trajectories, we just show that RL training on the harness (root LM only) generalizes better to these evals tasks vs. RL training on the base LM. My claim to you earlier is that arbitrarily using some standard harness won't see the same lift effect, but that doesn't really matter atm. The point about shaping the "token-by-token" input is an observation about the trajectories that the harness is now shaping for its individual LM calls to see. The point being, that our trajectories from our experiments show that for most eval rollouts that the LLM produces in our evals, there exists some train rollout that is very *similar* token-wise to it -- this is not the same thing as two coding agent trajectories following the same pattern btw, what I'm trying to say is that at a token-level they are very similar. And sure, this relies on some kind of smoothness assumptions about the fact that similar token-for-token inputs yield similar output distributions, but I don't think this is an unreasonable assumption to make on most prompts. This obviously doesn't happen for the base Transformer, and it won't happen for harnesses that append all observations in context either. The individual prompt it sees between train and test are quite different by nature of the tasks being different. Keep in mind that for the harness + LM it also is given the same "different" input in train / test, but the LM within the harness doesn't see this prompt. It's over training that the harness converges to behavior s.t. the LM calls within the harness only see things it's seen during training.
Fair, and so this is why we compare to directly training the LLM vs. LLM with the harness. But obviously it's a small experiment w.r.t. the larger argument, which must be shown through proper scaling experiments. I think LLMs do exhibit input generalization (by existence, I could construct some unique prompt no one has ever seen before and it could maybe solve it), the question is how well and how efficient is this process -- is this true across new domains and axes like length. Fundamentally if you believe model architecture choices influence scaling laws (even at convergence) and the generalization capabilities of a model, then it's not hard to buy that we could improve how well LLMs generalize at the harness layer as well
@a1zhang @PrimeIntellect Sorry, I don't really understand. The experiment seems to compare a raw LLM to an LLM with a python interpreter. I'm lost as to why that shows LLMs need token-by-token input?
@srush_nlp @PrimeIntellect sees things in test that it sees during training* in the most literal, token-for-token sense
@a1zhang @PrimeIntellect I'll give it another read post review 😀