Reactions from ranked influencers
8 posts
what is a multimodal LLM thinking as it watches a video? Gemma 4 12B reads raw image patches, as if they were tokens. It was never trained to predict anything at these 'tokens' - but this video shows what it would predict if you did sample from its next token prediction head
this is cleaner to do with Gemma 4 12B's encoder-free architecture. Most VLMs would encode through a dedicated vision network, so there wouldn't be such a clean mapping from patch -> token
at each patch I am showing a word cloud of the top 3 token distribution. But this is after dividing out the global distribution of tokens over the whole image. This helps to show what each individual square predicts that the rest of the image doesn't
This is a wonderful visualization.
what is a multimodal LLM thinking as it watches a video? Gemma 4 12B reads raw image patches, as if they were tokens. It was never trained to predict anything at these 'tokens' - but this video shows what it would predict if you did sample from its next token prediction head
in this frame you can see it is reading the text "National Galleries of Scotland"