I think people are taking results from things with really easy and meaningful metrics like loss / perplexity and assuming it applies more broadly than it really does.
I promise I can out data the auto researchers right now. I’m cheating because it’s a messy objective, but that’s the point right?
I feel like manual hill-climbing research now would be like trying to write code to beat neural networks 10 years ago
Users backed the argument that manual methods beat automated research on messy objectives because quality beats raw metrics like loss or perplexity.
Sorry, I don’t mean “I’m cheating” to imply I’m benchmaxxing or actually cheating.
I mean if what data went into a model was as clear cut as a single number for all scales, etc. than the auto researchers probably would already outpace humans.
My point is that defining good isn’t even easy to do for what we want out of base/midtrained models. So we cannot produce an easy metric, and as such we cannot give an auto researcher an easy hill to climb.
It’s not that the work isn’t hill climbing, it’s that without something to give you a dead reckoning I don’t know that the auto researchers are really up to the task yet.
I think people are taking results from things with really easy and meaningful metrics like loss / perplexity and assuming it applies more broadly than it really does.
I promise I can out data the auto researchers right now. I’m cheating because it’s a messy objective, but that’s the point right?
I think people are taking results from things with really easy and meaningful metrics like loss / perplexity and assuming it applies more broadly than it really does.
I promise I can out data the auto researchers right now. I’m cheating because it’s a messy objective, but that’s the point right?
I feel like manual hill-climbing research now would be like trying to write code to beat neural networks 10 years ago
@code_star the difference between academic wins and actual robustness hits different when the metric cant be gamed
which way is ur cheating leaning?
@code_star messy objectives are where the real gaps live tbh
thats what the loss/perplexity crowd never wants to touch
@rugbist_ Making a good model
@code_star that is such a valid point, quality beats raw metrics everytime.