Vikhyat K, Moondream AI co-founder, recommends analog sci-fi video to explain generative AI linear algebra

Story Brief

The hour-long video visualizes Jacobian matrices, pullbacks, and SVD.

Non-Euclidean Therapy for AI Trauma [Analog Archives]PATIENT ALICE: An Artificial Intelligence suffering from hallucinations of a lost puppet show. These hallucinations need to be erased. GENERATIVE MODEL TYPE: Diffusion-based. PRESCRIBED TREATMENT: A Latent Space Editing method that involves the Pullback, the Jacobian Matrix, Eigenfaces and SVD. ------------------------- 2. NOTE: read the Errata (section 4) below or in the pinned comment for a few important corrections of explanations in the video. This video experimented with a different approach, combining a science fiction mystery story told by analog VHS effects with an explanation of a paper about latent space editing in diffusion models. Link to the paper this was based on: "Unsupervised Discovery of Semantic Latent Directions in Diffusion Models" https://arxiv.org/abs/2302.12469 Math-only version: https://www.youtube.com/watch?v=EMJsYBD-dNk&feature=youtu.be ------------------------- 3. Timestamps: 00:00 - Patient Introduction -- [No Math] 02:08 - Manifolds and Pushforwards 06:38 - The Three Functions 09:55 - The Lost Show -- [No Math] 12:32 - Diffusion Models and the U-Net 14:13 - Matrix Multiplication 20:01 - The Jacobian Matrix 26:17 - The Pullback and the Dot Product 28:42 - A treat before treatment -- [No Math] 31:09 - The Treatment -- [No Math] 36:13 - Finding the Error 39:31 - Correlations in Matrices 42:13 - Superposition 45:23 - W^T W 47:24 - Eigenvectors of W^T W 49:38 - The Trauma -- [No Math] 51:43 - Singular Value Decomposition 54:30 - Reunion -- [No Math] ------------------------- 4. ERRATA IMPORTANT NOTE: A matrix is only called a "change of basis" for invertible, square matrices; in general, it's a transformation. Only those matrices can rotate the "full space", while other matrices may only rotate a "part of the space". There is one line in this video where this term is used, but it is only used colloquially to draw an intuitive analogy with "units", and is not used accurately. For simplification, the visual in this also only involves rotation, which can give the wrong impression. As the introduction to these transformations involve oversimplified examples, this is an important correction to the terms used loosely in this video. 4.1. This video refers to needing to find singular vectors to "compute the Jacobian" but that's poorly worded; it actually means for calculations involving the Jacobian, such as "for J^T * J to compute the SVD and get editable directions". The Jacobian matrix, as stated in the paper, is computed via "a sum-pooled feature map of the bottleneck representation of H" to reduce the number of parameters to compute. 4.2. This video states that the SVD is used to compute both the Jacobian and the semantic directions. But in the paper, the SVD is actually only used to obtain the semantic directions. To compute the SVD, in section 3.2 of a subsequent paper, the Power Method is used to compute the singular vectors without computing (M^T)(M). An updated version will fix this by stating that "dimensionality reduction" should be performed on J before multiplying with it; this was not done in the paper, so the video will take a different approach than it for the sake of explaining dimensionality reduction. This would fix the "circular issue" of needing the Jacobian J to compute J^T J to get the eigenvectors used to compute J. Link to this paper "Understanding the Latent Space of Diffusion Models through the Lens of Riemannian Geometry": https://arxiv.org/abs/2307.12868 4.3. Other Fixes (for the future update): 37m35s: This could be misleading; singular vectors are neglected, not dropped, so the matrix would have the same dim, but with a reduced rank (see: https://stats.stackexchange.com/questions/107533/how-to-use-svd-for-dimensionality-reduction-to-reduce-the-number-of-columns-fea) 46m: this transpose should switch the elements to row a c and row b d 53m20s: this should say "square root of eigenvalues" 55m55s: this is an audio glitch. it should be 'you weren't such a' 4.4. Subtitle additions: 37m: added- "it’s too many for them to calculate (for finding J^T * J for the pullback metric)" 54m15s: added- "There’s algorithms I still need (to get the eigenvectors without calculating M^T M)" #some4

Source Posts on X

Cluster sources

vik@vikhyatkTECH

if anyone is wondering wtf a jacobian is, i recommend watching "Non-Euclidean Therapy for AI Trauma" https://x.com/vikhyatk/status/2074270269234201072/photo/1

Vikhyat K, Moondream AI co-founder, recommends analog sci-fi video to explain generative AI linear algebra

Related Stories

Source Posts on X

Digg Deeper