The talk analyzes scaling and training challenges beyond language models face.
@VictoriaLinML This talk was incredibly insightful. Thank you for preparing and sharing!
The video of my Stanford CS25 guest lecture, From Language Models to Native Multimodal Intelligence, is now online.
I discussed how the core ideas behind LLMs has shaped multimodal AI, from architectures to training paradigms and scaling, and where the next challenges may lie. 🧠🌐 🎥: https://www.youtube.com/watch?v=NDdc39KYqDU
Many users praise the Stanford CS25 lecture on multimodal AI for delivering insightful discussion on specialized capabilities like visual reasoning.
No Digg Deeper questions have been answered for this story yet.
The success of multimodal LLMs has inherited many architectural and training principles from the LLM paradigm. By compressing text, vision, and audio into tokens, we can build AI systems that go beyond reading text to seeing, hearing, and responding to richer human inputs.
The video of my Stanford CS25 guest lecture, From Language Models to Native Multimodal Intelligence, is now online.
I discussed how the core ideas behind LLMs has shaped multimodal AI, from architectures to training paradigms and scaling, and where the next challenges may lie. 🧠🌐 🎥: https://www.youtube.com/watch?v=NDdc39KYqDU
On the other hand, multimodal AI spans a vast design space, where systems for digital information (static inputs over longer time horizons) and physical-world applications (real-time perception and action) tackle problems of different natures and require different technical approaches.
There has been continuous exploration of how to represent multimodal information and natively integrate it with the LLM backbone. For instance, - Chameleon: https://arxiv.org/abs/2405.09818 - Transfusion: https://arxiv.org/abs/2408.11039
Multimodal architecture design is also evolving, with promising directions such as sparsity and modality-specialization. - MoT: https://arxiv.org/abs/2411.04996
Thanks @stevenyfeng @_KaranPS_ and the class for having me. Hope the discussion is helpful. You could also find the slides at: https://drive.google.com/file/d/1zJVesVqH9R_K7Tscd5eNAQ9g38Uk_vOK/view?usp=sharing
A personal take ☕: The complexity of the problem space demands models specialized for specific capabilities (e.g. visual reasoning, knowledge processing or embodiment) rather than a single uniform design for everything. Seamless integrating these capabilities would be truly exciting.
@VictoriaLinML hot take: multimodal progress is bottlenecked by data curation, not architecture. the transformer already handles it fine
@VictoriaLinML Good job Victoria!
