Diffusion Model

A diffusion model generates data by starting with pure noise and removing a little at each step. It learned to do this by being trained on the reverse: adding noise to real images until nothing remained.

Training is the easy direction. Take a real image, add a known amount of noise, ask the model to predict what was added. Do that enough times and the model can walk the process backwards from noise alone.

Think of it like this. Think of watching a photograph dissolve into static, frame by frame, thousands of times. Eventually you learn the dissolve so well that you can run it backwards and pull a photograph out of static.

How it works

• Forward process adds noise to real data in small steps until it is unrecognisable. This part needs no learning.
• The model learns to predict the noise that was added at each step.
• Generation runs it backwards. Start from pure noise, subtract predicted noise repeatedly, and an image emerges.
• Latent diffusion works in a compressed space rather than on pixels, which is what made it fast enough to be practical.

Trade-offs

• Excellent quality and diversity, with none of the mode collapse GANs suffer.
• Slow generation. Many steps means many forward passes, unlike a GAN's single pass.
• Stable training, because it is ordinary supervised learning on a well-defined target.
• Steps hit diminishing returns. More is not indefinitely better.

Common mistakes

• "Diffusion models are always slow." Latent diffusion and faster samplers cut steps dramatically. The naive version was slow.
• "More steps always means better images." Quality plateaus, and past that you are only paying for compute.
• "They beat GANs at everything." They are more stable and more diverse. GANs still generate in a single pass, which matters where latency does.