• Home
  • Technology
  • Gaming
  • Entertainment
  • World & Business
  • Science
  • Sports
  • AI
HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
AI
Reaction

Direct real-world robotics training versus generated simulators

A post argues that direct pixel-space training has an advantage because even good simulators are costly and lossy.

clem 🤗C🤗
Andrew Carr 🤸AC
Chris PaxtonCP
6 Sources, 21h ago, first seen 21h ago

TLDR

A post argues that too much emphasis is being placed on coding models generalizing to robotics by generating simulators. It favors training directly in pixel space using the physical world, saying that generating simulators is slow and that simulation remains expensive and lossy even with a good simulator. The author compares the approach to simulating the entire internet to train a language model.

Combined views

—

6 Sources, first seen 21h ago

— likes— comments— saves— reposts

Combined views

—

6 Sources, first seen 21h ago

— likes— comments— saves— reposts

Sentiment

Positive——Negative

Summary

Not enough discussion yet.

No sentiment analysis available yet.

Sentiment

Positive——Negative

Summary

Not enough discussion yet.

No sentiment analysis available yet.

6 Sources

Pim de Witte@PimDeWitteToo much emphasis is being placed on coding models generalizing to robotics just by “generating simulators” - other than it being slow, training natively in pixel space using the world’s compute (our universe) directly is a massive advantage as you don’t need to go through simulating at all. You can have a good simulator, but simulating is still expensive. And lossy. Imagine having to simulate the entire internet just to train an LLM!21h
Chris Paxton@chris_j_paxtonSome thoughts on a robotics conference: the research community increasingly is looking at enabling contact-rich manipulation with dexterous hands, and has the tools to do so https://itcanthink.substack.com/p/dexterity-is-the-frontier-thoughts?r=3uj5js&utm_campaign=post&utm_medium=web17h
clem 🤗@ClementDelangueIf you’re a researcher, please consider sharing your RL environments openly! Something you built for one experiment could help the entire field move forward. I know we’re in dark days for sharing from frontier labs in the US, but I believe this can have a huge impact on the field and the world!16h
Andrew Carr 🤸@andrew_n_carrEvery profitable modality right now went through the capture generate cycle. Basically most of the time the data has existed has been during capture. The internet captured language, IDEs captured code, cameras captured video, and so on. Robotics requires kinematics data though, and the number one way to catch that was with super expensive motion capture rigs or with actual deployed hardware. Neither of those scale to the level required to hit performance thresholds though!!! Scaling laws still dominate - maybe even more so in robotics. This really means we need a new kind of capture technology for kinematics, some way to translate an existing modality into the shape needed for robotics.15h
Phil Chen@philhchenFrontier models still struggle to accurately caption egocentric videos. Dense video captioning remains a task where a specialized model, our collaboration with @MeckaAI, outperforms general-purpose foundation models. Many interesting problems still to solve in robotics.14h
    • Home
    • Technology
    • Gaming
    • Entertainment
    • World & Business
    • Science
    • Sports
    • AI

    6 Sources

    Pim de Witte@PimDeWitteToo much emphasis is being placed on coding models generalizing to robotics just by “generating simulators” - other than it being slow, training natively in pixel space using the world’s compute (our universe) directly is a massive advantage as you don’t need to go through simulating at all. You can have a good simulator, but simulating is still expensive. And lossy. Imagine having to simulate the entire internet just to train an LLM!21h
    Chris Paxton@chris_j_paxtonSome thoughts on a robotics conference: the research community increasingly is looking at enabling contact-rich manipulation with dexterous hands, and has the tools to do so https://itcanthink.substack.com/p/dexterity-is-the-frontier-thoughts?r=3uj5js&utm_campaign=post&utm_medium=web17h
    clem 🤗@ClementDelangueIf you’re a researcher, please consider sharing your RL environments openly! Something you built for one experiment could help the entire field move forward. I know we’re in dark days for sharing from frontier labs in the US, but I believe this can have a huge impact on the field and the world!16h
    Andrew Carr 🤸@andrew_n_carrEvery profitable modality right now went through the capture generate cycle. Basically most of the time the data has existed has been during capture. The internet captured language, IDEs captured code, cameras captured video, and so on. Robotics requires kinematics data though, and the number one way to catch that was with super expensive motion capture rigs or with actual deployed hardware. Neither of those scale to the level required to hit performance thresholds though!!! Scaling laws still dominate - maybe even more so in robotics. This really means we need a new kind of capture technology for kinematics, some way to translate an existing modality into the shape needed for robotics.15h
    Phil Chen@philhchenFrontier models still struggle to accurately caption egocentric videos. Dense video captioning remains a task where a specialized model, our collaboration with @MeckaAI, outperforms general-purpose foundation models. Many interesting problems still to solve in robotics.14h
    Today's Rank

    —

    Not ranked yet

    Today's Rank

    —

    Not ranked yet