Neural Network Scaling Expands Beyond Money Using Clever Data Generators
#488kache@YACINEMTBYou can also scale the total amount of training by just stacking compute multipliers in ways that are not immediately obvious. We have all of these incredible data generators in video games that are now easily decompiled by LLMs. Have you thought about pushing that to the limit?
kache@yacineMTBScaling neural networks is not a function of purely money. There are all these other dimensions which you can scale that people don't really pay attention to because spending money is the easier thing. But if you don't have money you can scale data, using clever data generators
#488kache@YACINEMTBIf you extract the state_i to state_i+1 of a video game, and autonomously ruthlessly optimize that while verifying that the state replay matches the real game how much data could you generate? How many frames per second or states per second could a single CPU core generate?
kache@yacineMTBYou can also scale the total amount of training by just stacking compute multipliers in ways that are not immediately obvious. We have all of these incredible data generators in video games that are now easily decompiled by LLMs. Have you thought about pushing that to the limit?
#488kache@YACINEMTBWe are late game in language models but we're still not late game in neural networks in general. As as our technology improves, we are unlocking all of this experimentation that was not worth doing. If you're just a little bit creative you'll be able to do some goofy stuff
kache@yacineMTBIf you extract the state_i to state_i+1 of a video game, and autonomously ruthlessly optimize that while verifying that the state replay matches the real game how much data could you generate? How many frames per second or states per second could a single CPU core generate?
#488kache@YACINEMTBYou can build your own hardware at the limit you can have a bunch of lidar modules on a 3D printed harness and a stereoscopic camera and you can take that data and then train a very small model that produces the depth. It's very very easy a 4090 is a supercomputer
kache@yacineMTBWe are late game in language models but we're still not late game in neural networks in general. As as our technology improves, we are unlocking all of this experimentation that was not worth doing. If you're just a little bit creative you'll be able to do some goofy stuff
#488kache@YACINEMTBWould cost you probably less than $100 to build this is like a raspberry pi Pico attached to some $5 aliexpress modules. I wondered myself why no one actually does stuff like this and I think it's because they don't understand how things work they don't understand how easy it is
kache@yacineMTBYou can build your own hardware at the limit you can have a bunch of lidar modules on a 3D printed harness and a stereoscopic camera and you can take that data and then train a very small model that produces the depth. It's very very easy a 4090 is a supercomputer