Denoising Diffusion Implicit Models (DDIM): A Verbose Guide

3.1 A family of non-Markovian inference processes

Consider a family \(\mathcal{Q}\) of inference distributions indexed by a vector \(\sigma \in \mathbb{R}^T_{\ge 0}\):

\[q_\sigma(x_{1:T}\mid x_0) := q_\sigma(x_T\mid x_0) \prod_{t=2}^{T} q_\sigma(x_{t-1}\mid x_t, x_0) \tag{6}\]

with \(q_\sigma(x_T\mid x_0) = \mathcal{N}(\sqrt{\alpha_T}x_0, (1-\alpha_T)I)\) and, for \(t>1\),

\[q_\sigma(x_{t-1}\mid x_t, x_0) = \mathcal{N}\!\left(\sqrt{\alpha_{t-1}}\,x_0 + \sqrt{1-\alpha_{t-1}-\sigma_t^2}\cdot \frac{x_t - \sqrt{\alpha_t}\,x_0}{\sqrt{1-\alpha_t}},\ \sigma_t^2 I\right) \tag{7}\]

The mean is chosen precisely so that, by induction (proved as Lemma 1 in the paper’s Appendix B and reproduced below), \(q_\sigma(x_t\mid x_0) = \mathcal{N}(\sqrt{\alpha_t}x_0, (1-\alpha_t)I)\) holds for every \(t\) — i.e., the marginals match the DDPM ones exactly, no matter what \(\sigma\) is. Applying Bayes’ rule gives the corresponding “forward” conditional

\[q_\sigma(x_t\mid x_{t-1}, x_0) = \frac{q_\sigma(x_{t-1}\mid x_t,x_0)\,q_\sigma(x_t\mid x_0)}{q_\sigma(x_{t-1}\mid x_0)} \tag{8}\]

which is also Gaussian, but now depends on both \(x_{t-1}\) and \(x_0\) rather than only on \(x_{t-1}\) — this is what makes the process non-Markovian. The magnitude of \(\sigma\) controls how stochastic this process is: as \(\sigma \to 0\), observing \(x_0\) and any single \(x_t\) pins down \(x_{t-1}\) almost exactly (it becomes near-deterministic).

Proof sketch of Lemma 1 (marginal matching by induction): Assume \(q_\sigma(x_t\mid x_0) = \mathcal{N}(\sqrt{\alpha_t}x_0, (1-\alpha_t)I)\) holds for some \(t\). Using the Gaussian marginalization identity \(q_\sigma(x_{t-1}\mid x_0) = \int q_\sigma(x_t\mid x_0)\, q_\sigma(x_{t-1}\mid x_t,x_0)\, dx_t\) and the standard formula for marginalizing a linear-Gaussian model (Bishop, PRML, Eq. 2.115), the resulting mean is

\[\mu_{t-1} = \sqrt{\alpha_{t-1}}\,x_0 + \sqrt{1-\alpha_{t-1}-\sigma_t^2}\cdot \frac{\sqrt{\alpha_t}x_0 - \sqrt{\alpha_t}x_0}{\sqrt{1-\alpha_t}} = \sqrt{\alpha_{t-1}}\, x_0\]

(the correction term vanishes because the two \(\sqrt{\alpha_t}x_0\) terms cancel), and the resulting covariance is

\[\Sigma_{t-1} = \sigma_t^2 I + \frac{1-\alpha_{t-1}-\sigma_t^2}{1-\alpha_t}(1-\alpha_t) I = (1-\alpha_{t-1}) I.\]

So \(q_\sigma(x_{t-1}\mid x_0) = \mathcal{N}(\sqrt{\alpha_{t-1}}x_0, (1-\alpha_{t-1})I)\), completing the induction (base case \(t=T\) holds by construction). \(\blacksquare\)

3.2 The matching generative process

Given a noisy \(x_t\), the natural strategy is: first predict what \(x_0\) probably was, then plug that prediction into \(q_\sigma(x_{t-1}\mid x_t, x_0)\) to sample \(x_{t-1}\). Concretely, since \(x_t = \sqrt{\alpha_t}x_0 + \sqrt{1-\alpha_t}\epsilon_t\) (Eq. 4), a network \(\epsilon_\theta^{(t)}(x_t)\) trained to predict the noise \(\epsilon_t\) gives us a denoised prediction of \(x_0\) for free:

\[f_\theta^{(t)}(x_t) := \big(x_t - \sqrt{1-\alpha_t}\cdot \epsilon_\theta^{(t)}(x_t)\big) / \sqrt{\alpha_t} \tag{9}\]

The generative process is then defined, with prior \(p_\theta(x_T) = \mathcal{N}(0,I)\), as

\[p_\theta^{(t)}(x_{t-1}\mid x_t) = \begin{cases}\mathcal{N}\big(f_\theta^{(1)}(x_1),\ \sigma_1^2 I\big) & t=1\\[4pt] q_\sigma\big(x_{t-1}\mid x_t,\, f_\theta^{(t)}(x_t)\big) & t>1\end{cases} \tag{10}\]

i.e., Eq. (7) with the true \(x_0\) replaced by the model’s prediction \(f_\theta^{(t)}(x_t)\). Training uses the variational objective

\[J_\sigma(\epsilon_\theta) := \mathbb{E}_{q_\sigma(x_{0:T})}\big[\log q_\sigma(x_{1:T}\mid x_0) - \log p_\theta(x_{0:T})\big] \tag{11}\]

At first glance it looks like a different model must be trained for every choice of \(\sigma\), since each \(\sigma\) defines a different objective \(J_\sigma\). The paper’s central theoretical result rules this out:

Theorem 1. For all \(\sigma > 0\), there exist \(\gamma \in \mathbb{R}^T_{>0}\) and \(C \in \mathbb{R}\) such that \(J_\sigma = L_\gamma + C\).

Proof (sketch). Expand \(J_\sigma\) and complete the square: each summand for \(t>1\) becomes a KL divergence between two Gaussians with the same variance \(\sigma_t^2\), which collapses to a squared-error term,

\[\mathbb{E}\big[D_{\mathrm{KL}}(q_\sigma(x_{t-1}\mid x_t,x_0)\,\|\,p_\theta^{(t)}(x_{t-1}\mid x_t))\big] = \mathbb{E}\left[\frac{\|x_0 - f_\theta^{(t)}(x_t)\|_2^2}{2\sigma_t^2}\right]\]

Substituting the definitions of \(x_0\) and \(f_\theta^{(t)}\) from Eq. (4) and (9), the \(x_t\)-dependent terms cancel and this reduces exactly to a rescaled noise-prediction error,

\[= \mathbb{E}_{x_0,\epsilon}\left[\frac{\|\epsilon - \epsilon_\theta^{(t)}(x_t)\|_2^2}{2 d\, \sigma_t^2 \alpha_t}\right]\]

where \(d = \dim(x_0)\). The \(t=1\) term reduces analogously. Hence, setting \(\gamma_t = 1/(2d\sigma_t^2\alpha_t)\) for every \(t\) gives \(J_\sigma(\epsilon_\theta) \equiv L_\gamma(\epsilon_\theta)\) for all \(\epsilon_\theta\), i.e., \(J_\sigma = L_\gamma + C\). \(\blacksquare\)

The payoff of Theorem 1 is enormous: since \(L_\gamma\)’s optimum does not depend on the weighting \(\gamma\) whenever parameters are not shared across timesteps (the global optimum is reached by independently minimizing each term of the sum), the \(L_1\)-trained DDPM network is already the optimal solution for every \(J_\sigma\). A single pretrained DDPM checkpoint can serve as the noise predictor for any member of this entire family of generative processes — no retraining required. The only thing that changes across family members is how you use the trained network at sampling time.

4.1 Denoising Diffusion Implicit Models (DDIM)

From Eq. (10), one can sample \(x_{t-1}\) from \(x_t\) via:

\[x_{t-1} = \sqrt{\alpha_{t-1}}\underbrace{\left(\frac{x_t - \sqrt{1-\alpha_t}\,\epsilon_\theta^{(t)}(x_t)}{\sqrt{\alpha_t}}\right)}_{\text{"predicted }x_0\text{"}} + \underbrace{\sqrt{1-\alpha_{t-1}-\sigma_t^2}\cdot \epsilon_\theta^{(t)}(x_t)}_{\text{"direction pointing to }x_t\text{"}} + \underbrace{\sigma_t \epsilon_t}_{\text{random noise}} \tag{12}\]

with \(\epsilon_t\sim\mathcal{N}(0,I)\) independent of \(x_t\), and \(\alpha_0 := 1\). Different choices of \(\sigma\) give different generative processes using the same \(\epsilon_\theta\):

• If \(\sigma_t = \sqrt{(1-\alpha_{t-1})/(1-\alpha_t)}\sqrt{1-\alpha_t/\alpha_{t-1}}\) for all \(t\), the forward process becomes Markovian again and the generative process reduces exactly to the original DDPM.
• If \(\sigma_t = 0\) for all \(t\), the forward process becomes deterministic given \(x_{t-1}\) and \(x_0\) (except at \(t=1\)), and the coefficient on the random noise term in Eq. (12) vanishes. The resulting model is an implicit probabilistic model — samples are a deterministic function of the initial latent \(x_T\). This is the Denoising Diffusion Implicit Model (DDIM), pronounced /ˈdɪm/, so named because it is an implicit model trained with the DDPM objective, even though its “forward process” is no longer a diffusion in the classical sense.

4.2 Accelerated Generation Processes (Sampling with Fewer Steps)

Because the denoising loss \(L_1\) never depended on the forward process having exactly \(T\) steps — only on the marginals \(q_\sigma(x_t\mid x_0)\) being fixed — we can define the forward process over just a subsequence \(\{x_{\tau_1}, \dots, x_{\tau_S}\}\), where \(\tau\) is an increasing subsequence of \([1,\dots,T]\) of length \(S \ll T\), while requiring \(q(x_{\tau_i}\mid x_0) = \mathcal{N}(\sqrt{\alpha_{\tau_i}}x_0, (1-\alpha_{\tau_i})I)\) to still match the marginals. The generative process then only needs to walk backwards along \(\tau\) — this reversed subsequence is called the sampling trajectory. Since sampling cost is linear in the number of steps taken, using \(S \ll T\) steps gives a proportional speedup.

Formally (Appendix C), the accelerated inference process factorizes as

\[q_{\sigma,\tau}(x_{1:T}\mid x_0) = q_{\sigma,\tau}(x_{\tau_S}\mid x_0)\prod_{i=1}^{S} q_{\sigma,\tau}(x_{\tau_{i-1}}\mid x_{\tau_i}, x_0) \prod_{t\in\bar\tau} q_{\sigma,\tau}(x_t\mid x_0) \tag{52}\]

where \(\bar\tau := \{1,\dots,T\}\setminus\tau\). Intuitively, \(\{x_{\tau_i}\}\) together with \(x_0\) form a chain (used to actually produce samples), while the remaining \(\{x_t\}_{t\in\bar\tau}\) together with \(x_0\) form a star graph (only needed to keep the variational bound well-defined — see Figure 2 below). A similar argument to Theorem 1’s proof shows this accelerated objective is also equivalent (up to reweighting) to \(L_\gamma\), so — again — no retraining is required; only the sampling procedure changes.

Two practical choices for picking the subsequence \(\tau\) (Appendix D.2) are used in the experiments: linear (\(\tau_i = \lfloor ci\rfloor\)) and quadratic (\(\tau_i=\lfloor ci^2\rfloor\)) spacing, with \(c\) chosen so \(\tau_S \approx T\). The paper uses quadratic spacing for CIFAR-10 and linear spacing for the other datasets, each found empirically to give marginally better FID than the alternative.

4.3 Connection to Neural ODEs

Rewriting the DDIM update (Eq. 12, with \(\sigma_t=0\)) in terms of a small step \(\Delta t\) reveals a resemblance to an Euler discretization of an ODE:

\[\frac{x_{t-\Delta t}}{\sqrt{\alpha_{t-\Delta t}}} = \frac{x_t}{\sqrt{\alpha_t}} + \left(\sqrt{\frac{1-\alpha_{t-\Delta t}}{\alpha_{t-\Delta t}}} - \sqrt{\frac{1-\alpha_t}{\alpha_t}}\right)\epsilon_\theta^{(t)}(x_t) \tag{13}\]

Reparameterizing with \(\bar x := x/\sqrt{\alpha}\) and \(\sigma(t) := \sqrt{(1-\alpha(t))/\alpha(t)}\) (an increasing, continuous function with \(\sigma(0)=0\)), taking \(\Delta t \to 0\) turns Eq. (13) into the ODE

\[d\bar x(t) = \epsilon_\theta^{(t)}\!\left(\frac{\bar x(t)}{\sqrt{\sigma^2(t)+1}}\right) d\sigma(t) \tag{14}\]

with initial condition \(\bar x(T) \sim \mathcal{N}(0,\sigma(T)^2 I)\) for large \(\sigma(T)\). Because this is an ODE (not an SDE), it can, with a fine enough discretization, be simulated forward in time (\(t=0\to T\), encoding \(x_0\to x_T\)) just as easily as backward (\(t=T\to0\), generating \(x_0\) from \(x_T\)). This is a property DDPMs fundamentally lack, since their reverse process is stochastic.

Proposition 1. The ODE in Eq. (14), evaluated at the optimal \(\epsilon_\theta^{(t)}\), is equivalent to the probability-flow ODE corresponding to the “Variance-Exploding” (VE) SDE of Song et al. (2020) — a continuous-time generalization of NCSN-style score-based models.

Proof idea. Reparameterize with \(\bar x(t) = \bar x(0) + \sigma(t)\epsilon\), matching the VE-SDE convention. The DDIM-derived ODE (Eq. 45 in the appendix) is \(\frac{d\bar x}{dt} = \frac{d\sigma}{dt}\,\epsilon_\theta^{(t)}\big(\bar x/\sqrt{\sigma^2+1}\big)\). Independently, the probability-flow ODE for a VE-SDE is \(d\bar x = -\tfrac12 g(t)^2 \nabla_{\bar x}\log p_t(\bar x)\, dt\) with \(g(t)^2 = d\sigma^2(t)/dt\). Using the fact that the optimal noise predictor and the optimal score function are related by \(\nabla_{\bar x}\log p_t(\bar x) = -\epsilon_\theta^{(t)}(\bar x/\sqrt{\sigma^2+1})/\sigma(t)\) (a standard denoising-score-matching identity), substituting into the probability-flow ODE and simplifying recovers exactly \(\frac{d\bar x}{dt} = \frac{d\sigma}{dt}\epsilon_\theta^{(t)}(\cdot)\) — identical to the DDIM ODE. \(\blacksquare\)

Although the two ODEs are mathematically identical, their discretizations differ: DDIM’s Euler steps are taken with respect to \(\sigma(t)\), while the probability-flow ODE’s Euler steps (Eq. 15 in the paper) are taken with respect to \(t\) directly. These coincide when \(\alpha_t\) and \(\alpha_{t-\Delta t}\) are close, but diverge for large steps — which is precisely the regime accelerated sampling operates in, explaining why the two integration schemes behave differently at low step counts in practice.

6.1 Sample Quality vs. Efficiency

Measuring Fréchet Inception Distance (FID; lower is better) while varying the number of sampling steps \(\dim(\tau) \in \{10,20,50,100,1000\}\):

Similarly, on the larger LSUN datasets:

(For reference, the 1000-step DDPM baseline FIDs are 6.36 for Bedroom and 7.89 for Church.)

Takeaways: as \(\dim(\tau)\) grows, quality improves for every method (expected trade-off between compute and quality). DDIM (\(\eta=0\)) is consistently the best or tied-best at every step count, and its advantage is largest exactly where it matters most — few steps. DDPM-style stochastic sampling (\(\eta=1\), \(\hat\sigma\)) degrades sharply as \(\dim(\tau)\) shrinks; \(\hat\sigma\) in particular becomes extremely poor at low step counts because FID is highly sensitive to the added noisy perturbations it introduces. Wall-clock time scales linearly with the number of sampling steps (Figure 4 in the paper), so DDIM’s ability to hit “1000-step quality” using only 20–100 steps translates directly into a \(10\times\)–\(50\times\) real speedup; e.g., the 100-step DDPM on CelebA is roughly matched in FID by the 20-step DDIM.

6.2 Sample Consistency

Because DDIM’s generative process is a deterministic function of \(x_T\), fixing \(x_T\) and varying the sampling trajectory \(\tau\) (i.e., varying how many/which intermediate steps are taken) produces images that share the same high-level content, with only fine details varying. Even a 20-step trajectory already captures most of the high-level structure that a 1000-step trajectory would produce. This strongly suggests that \(x_T\) alone functions as a semantically meaningful latent code — an unusual property for a diffusion-style model, and one that DDPM’s stochastic sampling cannot offer (identical \(x_T\) under DDPM yields highly diverse outputs).

6.3 Latent-Space Interpolation

Since high-level content is encoded in \(x_T\), the authors test whether interpolating in the latent space, as is common for GANs, produces meaningful interpolation in image space. Two random \(x_T^{(0)}, x_T^{(1)} \sim \mathcal{N}(0,I)\) are combined with spherical linear interpolation (slerp), which is the natural way to interpolate points that are meant to lie roughly on a Gaussian’s typical “shell”:

\[x_T^{(\alpha)} = \frac{\sin((1-\alpha)\theta)}{\sin\theta}\, x_T^{(0)} + \frac{\sin(\alpha\theta)}{\sin\theta}\, x_T^{(1)}, \qquad \theta = \arccos\!\left(\frac{x_T^{(0)\top}x_T^{(1)}}{\|x_T^{(0)}\|\,\|x_T^{(1)}\|}\right) \tag{67}\]

Each interpolated \(x_T^{(\alpha)}\) is then decoded with DDIM. The resulting sequences show smooth, semantically coherent transitions (pose, identity, background attributes blending gradually) — a property DDPM cannot replicate directly, since re-using the same \(x_T\) under DDPM’s stochastic decoder does not pin down a consistent \(x_0\).

6.4 Reconstruction from Latent Space

Because DDIM corresponds to an ODE, it should be approximately invertible: encode \(x_0\to x_T\) by simulating Eq. (14) forward, then decode \(x_T\to x_0\) by simulating it backward, and measure the reconstruction error.

Reconstruction error shrinks steadily and monotonically as the number of discretization steps \(S\) increases — exactly the behavior expected of an Euler-integrated ODE as its step size shrinks, and analogous to what invertible flows / Neural ODEs exhibit. DDPMs, being fundamentally stochastic in their generative process, cannot do this at all.

Appendix A: The discrete/categorical case (briefly)

For a one-hot categorical \(x_0\) with \(K\) classes, define \(q(x_t\mid x_0) = \mathrm{Cat}(\alpha_t x_0 + (1-\alpha_t)\mathbf{1}_K)\) (a mixture that gradually forgets the true class in favor of the uniform distribution \(\mathbf{1}_K/K\)). An analogous non-Markovian \(q(x_{t-1}\mid x_t,x_0)\) is defined as a three-way mixture over “keep \(x_t\)”, “reveal \(x_0\)”, and “go fully uniform,” with mixture weights chosen so the marginals match. The KL divergence between this and the model’s own reverse conditional \(p_\theta(x_{t-1}\mid x_t)\) reduces to (an upper bound of) a standard multi-class classification loss, mirroring the Gaussian case’s reduction to squared error. The authors leave a full empirical study of this discrete variant to future work, but note it suggests DDIM-style acceleration is not intrinsically tied to continuous, Gaussian diffusions.

Figure 1 — Graphical models: Markovian diffusion (left) vs. non-Markovian inference (right)

In the original DDPM, each \(x_t\) is generated only from \(x_{t-1}\) (a first-order Markov chain), and the reverse/generative process retraces the same chain backward. In the non-Markovian family the paper introduces, every \(x_{t-1}\) is conditioned jointly on both \(x_t\) and \(x_0\) — shown by the extra arrows from \(x_0\) fanning out to every latent variable.

graph LR
  subgraph "Markovian diffusion (DDPM)"
    direction LR
    A0["x0"] --> A1["x1"] --> A2["x2"] --> Adots["..."] --> AT["xT"]
    AT -.generative process.-> Adots -.-> A2 -.-> A1 -.-> A0
  end

graph LR
  subgraph "Non-Markovian inference (DDIM family)"
    direction LR
    B0["x0"] --> B1["x1"]
    B0 --> B2["x2"]
    B0 --> Bdots["..."]
    B0 --> BT["xT"]
    B1 --> B2
    B2 --> Bdots
    Bdots --> BT
    BT -.generative process.-> Bdots
    Bdots -.-> B2
    B2 -.-> B1
    B1 -.-> B0
  end

Reading the diagrams: solid arrows denote the (non-)Markovian forward/inference dependencies; dashed arrows denote the direction the trained generative process actually samples in (from \(x_T\) down to \(x_0\)). In the right-hand diagram, note that \(x_0\) has a direct edge into every \(x_t\) — this is exactly the extra conditioning that makes the process non-Markovian, and it is what allows the reverse chain to legally “jump” over intermediate steps (Figure 2, next).

Figure 2 — Accelerated generation with \(\tau = [1, 3]\)

When sampling is accelerated, the generative process only visits a subsequence \(\tau\) of the original \(T\) timesteps (here, as a minimal illustrative example with \(T=3\), only \(x_1\) and \(x_3=x_T\) are visited; \(x_2\) is skipped entirely at sampling time, existing only conceptually as part of keeping the variational bound consistent).

graph LR
  subgraph "Accelerated sampling trajectory, tau = [1, 3]"
    direction LR
    X0["x0"] --> X1["x1 (tau_1, sampled)"]
    X0 --> X2["x2 (skipped at sampling time)"]
    X0 --> X3["x3 = xT (tau_2, sampled)"]
    X1 --> X3
    X3 -.DDIM step.-> X1
    X1 -.DDIM step.-> X0
  end

Reading the diagram: the solid edges from \(x_0\) show which latents remain tied to \(x_0\)’s marginal constraint; the edge \(x_1 \to x_3\) (and its dashed reverse) is the direct jump the accelerated sampler takes — going straight from \(x_3\) to \(x_1\), then from \(x_1\) to \(x_0\), entirely skipping \(x_2\). This is the mechanism, generalized to \(S \ll T\) chosen steps out of \(T\), that gives DDIM its \(10\times\)–\(50\times\) wall-clock speedup.