@probablyspiking @zacharynado Whats your take on how to do this with the transformer
You have X : [BTDin] dY: [BTDOut]
Define how you will do L and R for K-FAC
Answer to your question is within this question.
Actually hard question. KFAC is based on Kronecker approx of the covariance matrix: E[gg^T] where g is the grad for a single sample. That would work for CV, but in next token prediction you solve T tasks simultaneously. Even per-sample gradients from hooks are avg of T losses.
@probablyspiking @zacharynado Whats your take on how to do this with the transformer
You have X : [BTDin] dY: [BTDOut]
Define how you will do L and R for K-FAC
Answer to your question is within this question.