The model decodes 512×512 latents into legible 2048×2048 images.
Sanja Fidler#497
The latent-vs-pixel debate misses the point.
GPT Image 2 shows what users notice: pixel-level fidelity. Latent models show what scales: compact semantic structure.
We connect them by replacing VAE/RAE decoders with a Pixel Diffusion Decoder.
Code and Model available: https://research.nvidia.com/labs/sil/projects/pid/
🧵(1/N)
Users praise NVIDIA's Pixel Diffusion Decoder for Latent Image Models because it bridges the latent-vs-pixel debate by letting each handle its strengths in semantics and fidelity.
No Digg Deeper questions have been answered for this story yet.
@xuanchi13 Very impressive results, and also, nice thread and videos!
Check out more results at: https://research.nvidia.com/labs/sil/projects/pid/
Huge kudos to @YifanLu17525599 for his outstanding work.
Joint work with an amazing team at @NVIDIAAI: @wilson_over, @jayzhangjiewu, @zianwang97, @HuanLing6, @FidlerSanja
(N/N)
PiD can also decode partially denoised latents.
The base latent model can stop early, and PiD finishes the image in pixel space.
Less latent sampling. More pixel detail.
(9/N)
Most high-res pipelines still treat decoding as reconstruction: latent → VAE decode → super-resolution → high-res image
PiD does: low-res latent → high-res pixels
one generative decoding stage.
(2/N)
The key insight: decoding is not just inversion.
Latents carry the what and where. Pixels carry the final visual truth: text, texture, edges, hair, skin, foliage.
PiD lets each side do what it is best at.
Beyond that, in omni models where understanding + generation meet:
keep semantics in the model, post-train the decoder for pixel fidelity.
(3/N)
How does PiD do it?
Instead of reconstructing a low-res image first, PiD injects the latent directly into a pixel-space diffusion transformer. A lightweight adapter aligns the latent with pixel tokens, while σ-aware gates decide how much to trust it.
Latent guidance, pixel synthesis.
(4/N)
PiD works across latent spaces: FLUX.1 / FLUX.2 / SD3 / Z-Image VAEs + semantic latents like DINOv2 and SigLIP / RAE
Different latent worlds, one decoder-side idea.
(6/N)
This becomes especially natural with RAEs.
RAEs gave us compact semantic latents. PiD gives them a 2K generative decoder.
DINOv2 / SigLIP-style latents are semantically rich, but low-level appearance is under-specified. PiD turns that semantic structure into high-res pixels.
(5/N)
PiD is not limited to 2K. We also scale the same idea to 4K decoding: latent → 4096² pixels This is where pixel priors matter most: small text, faces, hair, crowds, foliage, texture.
(7/N)
The decoder also becomes fast. 512² latent → 2048² image ~210ms on GB200. <1s on RTX 5090 with 13GB peak memory.
(8/N)
@xuanchi13 @aria_agi validate
@handsomeblob @xuanchi13 The latent vs pixel debate misses it. Users see fidelity. Models learn semantics. A pixel diffusion decoder bridges both: replace VAE with direct high resolution synthesis. Compact structure scales, pixel quality ships. First, the insight. Second, the execution.
@xuanchi13
@xuanchi13 arguing over latent vs pixel tech is endless. real users just want results.
@xuanchi13 @ruu_1010
@xuanchi13 so basically we stop arguing and just let pixels and latents each play their lane
@xuanchi13 Hello 👋👋... @FrankOsei146876
@xuanchi13
@xuanchi13
@giffmana Thanks Lucas! ☺️