Gemma LLMs Generate Objects Or Textures Via Image Optimization
Reactions from ranked influencers
4 posts
Deep dreams on modern LLMs are so cool (optimizing an image to maximize P(target caption)) Gemma 12B (left) has no vision encoder — it reads pixels like token embeddings — and stamps recognisable objects around the canvas. E4B (right) has one, and drifts to texture instead.
these runs are doing gradient descent to optimize the probability of the caption given an input image, represented as an FFT. Higher frequencies are suppressed in the FFT, and the RGB colours are decorrelated and normalized to a target standard deviation, then tanh
final frames
of course 12B is a bigger model, so it is not a fully fair comparison. But it's an interesting glimpse into how these models see.