Frontier AI labs are reportedly spending $10 billion to $15 billion each on training data acquisition

@xlr8harder It is not as useful or valuable as one may think especially as it gets more and more specialised

It is just “easy” to spend money on this vs say GPUs

@xlr8harder @xeophon you don’t have to

closed model needs to be good at every use case

open model needs to be good at your use case

just do last-mile training on your own tasks and data

@xlr8harder you know it, i know it, everyone knows it

@xeophon I'm not sure I actually do (unless the answer is that it doesn't.)

Crowd sourcing could do some of this but all of the efforts so far have been quite lackluster.

Oh, just distillation, I guess?

@xlr8harder ding ding ding

@EMostaque Maybe, but open datasets are still quite far behind.

@xlr8harder @xeophon open models don’t need to beat closed models outright, they just need to be close enough that you can bridge the gap and then some, relatively quickly and cheaply

for anything served at scale, you can amortize out the training cost pretty quickly, and retrain regularly

@xlr8harder @xeophon crowdsourcing will come in due time; have learned a lot of lessons here, you need activation energy to be super low and QC to be really high, but these are solvable with the right tooling

@umi33563 @xlr8harder @xeophon needs to be easy for people to contribute

@willccbb @xlr8harder @xeophon >you need activation energy to be super low ?

@willccbb @xlr8harder @xeophon I think the problem is how to fuse the local specialized models and tooling into a fluid user toolbox. Document editing, content generation, up to date data are all hard to switch between without custom building something (afaik) and building your own context stack.

@willccbb @xeophon This is related to a question I've been exploring. Say you need to do meaningful domain adaptation, the kind of thing you'd probably need CPT for. Is there any good way to do this without wrecking post-trained behavior?

This seems like it would be very valuable.

@willccbb @xlr8harder @xeophon Ah, oh, fuck. I was thinking about literal activations in network (cus I'm neckdeep in that rn)

@xlr8harder We need a decentralized mercor. Data is the next ai bottleneck

@xeophon @xlr8harder Curious to see this DM ?