Sorry, I don’t mean “I’m cheating” to imply I’m benchmaxxing or actually cheating.

I mean if what data went into a model was as clear cut as a single number for all scales, etc. than the auto researchers probably would already outpace humans.

My point is that defining good isn’t even easy to do for what we want out of base/midtrained models. So we cannot produce an easy metric, and as such we cannot give an auto researcher an easy hill to climb.

It’s not that the work isn’t hill climbing, it’s that without something to give you a dead reckoning I don’t know that the auto researchers are really up to the task yet.

I think people are taking results from things with really easy and meaningful metrics like loss / perplexity and assuming it applies more broadly than it really does.

I promise I can out data the auto researchers right now. I’m cheating because it’s a messy objective, but that’s the point right?

@code_star the difference between academic wins and actual robustness hits different when the metric cant be gamed

which way is ur cheating leaning?

@code_star messy objectives are where the real gaps live tbh

thats what the loss/perplexity crowd never wants to touch

@rugbist_ Making a good model

@code_star that is such a valid point, quality beats raw metrics everytime.