New episode of the Information Bottleneck with Phillip Isola @phillip_isola (MIT) is out 🥳🥳🥳 - What makes a good representation? Phillip has been a legend for me for many years. So much of how I think about representation learning comes from his work, so getting to ask him all my questions was a personal highlight.
We talked about why totally different models end up learning the same representations, and whether there's a single "true" representation of the world they're all converging on.
Phillip also argues that pre-training is far from over. His new "neural thickets" paper shows pre-trained weights already sit close to solutions for downstream tasks, which is why fine-tuning with a few parameters works at all.
We also talked about why he thinks the most interesting thing to do right now is artificial life. Drop LLM agents in an open environment with no task and study what emerges, like biologists with a new organism.
Also on YouTube, podcast feeds, and our website!