Experts Call for Robotics MMLU Benchmark to Test Generalist Models

If eval is bound to real world, then we need useful hardware, humanoid has achieved huge success with G1 as standard platform

Where is that robot for manipulation?

@JieWang_ZJUI what I dream of: a simple low cost humanoid robot that you can build at home w/ a 3d printer or purchase that acts as a research platform

@JieWang_ZJUI yes

@JieWang_ZJUI if you reduce the size to the limit, you reduce the cost and kinetic energy and they last much longer. You can build one using the micro servos that they use for model aircraft

@JieWang_ZJUI because right now unitree academic version is effectively the MMLU

@yacineMTB I think it’s hard because tech is not there for durable DIY humanoids, maybe 5 yrs later we can distill insights into cheap and reliable custom robots

@yacineMTB @JieWang_ZJUI

@yacineMTB Model aircraft took years of development, as aircraft industry is mature enough to produce know how about what works what’s not. The solution is definitely feasible but I will say for certain skill, humanoid gets long way to go

(I can’t image a cheap robot doing sim2real well