AIA finer-grained approach to transformer checkpointing
A user suggests applying selective activation checkpointing (SAC) to smaller units within a transformer block: normalization plus attention, or normalization plus MLP, when residual-add results are always saved.
TLDR
One user observes that SAC is conventionally applied to whole transformer blocks, but many policies always save the result of a residual addition. They say those saved results are the sole inputs to the attention and feed-forward (MLP) components, allowing SAC to be applied to those parts individually. In a follow-up, they clarify that each unit should include normalization too, since you usually want to recompute it.
A finer-grained approach to transformer checkpointing
A user suggests applying selective activation checkpointing (SAC) to smaller units within a transformer block: normalization plus attention, or normalization plus MLP, when residual-add results are always saved.
TLDR
One user observes that SAC is conventionally applied to whole transformer blocks, but many policies always save the result of a residual addition. They say those saved results are the sole inputs to the attention and feed-forward (MLP) components, allowing SAC to be applied to those parts individually. In a follow-up, they clarify that each unit should include normalization too, since you usually want to recompute it.