• Home
  • Technology
  • Gaming
  • Entertainment
  • World & Business
  • Science
  • Sports
  • AI
HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
    • Home
    • Technology
    • Gaming
    • Entertainment
    • World & Business
    • Science
    • Sports
    • AI
    AI

    The prospect of large-scale real-to-simulation systems

    A user sees large-scale real2sim—bringing the real world into simulation—as within reach, noting how hard it once was to imagine OpenAI putting serious resources into 3D/4D.

    Michael BlackMB
    Angjoo KanazawaAK
    Ken GoldbergKG
    11 Sources, ,

    TLDR

    A user expresses optimism about large-scale real2sim alongside OpenAI’s commitment of serious resources to 3D/4D. They say these systems build on tools they helped create and lessons learned, while emphasizing that much remains to be done and new problems remain to explore.

    Combined views

    201K

    11 Sources, first seen 21d ago

    Combined views

    201K

    11 Sources, first seen 21d ago

    1.2K likes
    21d ago
    first seen 21d ago
    1.2K likes
    24 comments
    532 saves
    80 reposts
    24 comments
    532 saves
    80 reposts

    Sentiment

    Positive——Negative

    Summary

    Not enough discussion yet.

    No sentiment analysis available yet.

    Sentiment

    Positive——Negative

    Summary

    Not enough discussion yet.

    No sentiment analysis available yet.

    Today's Rank

    —

    Not ranked yet

    Today's Rank

    —

    Not ranked yet

    11 Sources

    Angjoo Kanazawa@akanazawaA few years ago, it was hard to imagine OpenAI putting serious resources into 3D/4D. Now large-scale real2sim is within reach! These systems build on the tools we’ve built and what we’ve learned. Progress!! So much more to do, and a whole new set of problems to explore.21d
    Georgia Gkioxari@georgiagkioxari1. Problems that can be solved by scale because data exists at scale will be solved by large-scale approaches like Astra. 2. Producing data at scale is the challenge. 3D/4D was not available at scale; it only became so because researchers in the past 10 years unlocked it (generative 3D, GS, etc) -- same thing for quasi-static manipulation. 3. There are **many** unsolved problems in CV that we haven't gotten to scale yet; that's the next frontier of problems to go after. The goal is to unlock them so that we can scale.21d
    Michael Black@Michael_J_Black@georgiagkioxari While Astra is not “natively” 3d/4d, it understands enough to help us scale 3d/4d data collection. That’s what’s exciting.19d
    Jitendra MALIK@JitendraMalikCV3D/4D reconstruction is one of the great achievements of computer vision. With roots in photogrammetry, multi-view geometry was much studied (e.g. Hartley & Zisserman). In the last decade, it was successfully combined with deep learning. Shape priors which come from recognition got incorporated (e.g. for humans, in HMR) and recent systems like VGGT and SAM 3D can do amazing stuff. This progress is directly relevant for robotics, which (duh) operates in the 3D world. Many robotics papers in the learning era weren't exploiting 3D structure, which IMHO is just wasting valuable signal. But there has been a strong real2sim2real tradition for years e.g. here are two projects from my group (and there are many others, I am not claiming exclusivity). I am glad that people have rediscovered this capability with Astra. But please do give credit to human researchers, not just AI models. https://zhec.github.io/rhoi/ https://www.videomimic.net/17d
    Ken Goldberg@Ken_GoldbergThank you computer vision researchers for solving robotics so many years ago!17d

    11 Sources

    Angjoo Kanazawa@akanazawaA few years ago, it was hard to imagine OpenAI putting serious resources into 3D/4D. Now large-scale real2sim is within reach! These systems build on the tools we’ve built and what we’ve learned. Progress!! So much more to do, and a whole new set of problems to explore.21d
    Georgia Gkioxari@georgiagkioxari1. Problems that can be solved by scale because data exists at scale will be solved by large-scale approaches like Astra. 2. Producing data at scale is the challenge. 3D/4D was not available at scale; it only became so because researchers in the past 10 years unlocked it (generative 3D, GS, etc) -- same thing for quasi-static manipulation. 3. There are **many** unsolved problems in CV that we haven't gotten to scale yet; that's the next frontier of problems to go after. The goal is to unlock them so that we can scale.21d
    Michael Black@Michael_J_Black@georgiagkioxari While Astra is not “natively” 3d/4d, it understands enough to help us scale 3d/4d data collection. That’s what’s exciting.19d
    Jitendra MALIK@JitendraMalikCV3D/4D reconstruction is one of the great achievements of computer vision. With roots in photogrammetry, multi-view geometry was much studied (e.g. Hartley & Zisserman). In the last decade, it was successfully combined with deep learning. Shape priors which come from recognition got incorporated (e.g. for humans, in HMR) and recent systems like VGGT and SAM 3D can do amazing stuff. This progress is directly relevant for robotics, which (duh) operates in the 3D world. Many robotics papers in the learning era weren't exploiting 3D structure, which IMHO is just wasting valuable signal. But there has been a strong real2sim2real tradition for years e.g. here are two projects from my group (and there are many others, I am not claiming exclusivity). I am glad that people have rediscovered this capability with Astra. But please do give credit to human researchers, not just AI models. https://zhec.github.io/rhoi/ https://www.videomimic.net/17d
    Ken Goldberg@Ken_GoldbergThank you computer vision researchers for solving robotics so many years ago!17d