Harnessing Low Dimensionality in Diffusion Models: From Theory to Practice: Lecture II: Sampling Theory for Diffusion Models

Harnessing Low Dimensionality in Diffusion Models: From Theory to Practice: Lecture II: Sampling Theory for Diffusion Models

Speakers: Yuxin Chen, Qing Qu, Liyue Shen

Conference: ICML 2025

YouTube: https://slideslive.com/39043829

Overview

This talk, the second lecture in a comprehensive tutorial on diffusion models, delves into the intricate mathematical foundations governing the sampling stage of these powerful generative models. Presented by Yuxin Chen, with co-speakers Qing Qu and Liyue Shen, the lecture focuses on demystifying the practical efficacy of two widely adopted sampling algorithms: Denoising Diffusion Probabilistic Models (DDPM) and Denoising Diffusion Implicit Models (DDIM). The core objective is to bridge the gap between abstract theoretical understanding and the remarkable empirical performance observed in real-world applications.

The presentation meticulously unpacks the challenges inherent in moving from continuous-time, exact-score theoretical frameworks to discrete-time, imperfect-score practical implementations. Chen highlights two pivotal questions: how discretization error impacts sampling fidelity and how score estimation error affects the quality of generated samples. Crucially, the talk reveals a profound mechanism through which DDPM and DDIM automatically adapt to the intrinsic low-dimensional structure of data, offering a robust theoretical explanation for their surprising efficiency even when operating in high-dimensional ambient spaces. This work is significant for providing rigorous convergence guarantees under minimal assumptions, thereby solidifying the theoretical bedrock for the next generation of diffusion model research and deployment.

Background

▶ Watch: Introduction: Mathematical foundation for diffusion model sampling. (0:00)

Diffusion models operate through a two-stage process: a forward diffusion process and a reverse sampling process. The forward process progressively adds Gaussian noise to a data distribution, transforming it into a pure noise distribution. This process, denoted as $X_t$, is typically straightforward and well-understood. The complexity and generative power lie in the reverse process, denoted as $Y_t$, which aims to gradually denoise a pure noise sample back into a high-fidelity data sample resembling the target data distribution.

The feasibility of this reverse process hinges on the accurate estimation of the Stein score function, defined as the gradient of the log probability density function of the noisy data distribution at each time step. While theoretically, an exact score function ($s^*_t$) allows for a perfect reversal of the forward process, practical implementations rely on learned score estimates. Early theoretical work, notably building on Yang Song's contributions, established a crucial connection between discrete-time diffusion processes and their continuous-time counterparts: stochastic differential equations (SDEs) for the forward and reverse processes, and ordinary differential equations (ODEs) (specifically, the probability flow ODE) for deterministic sampling. These continuous-time frameworks, backed by classical results from SDE literature, demonstrated that reliable data generation is feasible provided the exact score function is known.

However, a significant chasm existed between these continuous-time, exact-score theoretical guarantees and the realities of practical implementation. Real-world diffusion models operate in discrete time steps and use neural networks to estimate the score function, inevitably introducing discretization error and score estimation error. Understanding how these errors propagate and impact the final sample quality, and why models like DDPM and DDIM perform so well despite these imperfections, became a pressing theoretical challenge. This lecture specifically addresses this challenge by developing a mathematical framework to analyze the sampling efficiency and fidelity of DDPM and DDIM in the discrete-time, imperfect-score regime, decoupling it from the score learning problem which was the focus of the preceding lecture.

Key Findings

▶ Watch: Lecture outline: convergence theory, low-dimensionality, acceleration. (3:20)

The talk presents several pivotal findings that advance the theoretical understanding of diffusion model sampling:

1. General Convergence Theory for DDPM and DDIM:

• Under mild assumptions on data distribution (polynomially bounded support size, no smoothness or log-concavity requirements), a comprehensive theory is established for both DDPM and DDIM.
• For DDPM, the theory provides KL divergence and Total Variation (TV) distance based convergence guarantees. It shows that the number of iterations required for a target accuracy $\epsilon$ scales with the ambient dimension $D$ of the data. Specifically, to achieve a KL divergence bounded by $\epsilon^2$, $D/\epsilon^2$ iterations are needed, and for TV distance bounded by $\epsilon$, $D/\epsilon$ iterations suffice.
• DDPM is shown to degrade gracefully as score estimation error increases, highlighting its robustness to imperfect score functions.
• For DDIM, convergence guarantees are established primarily using Total Variation distance, as KL divergence can be unbounded in this deterministic setting. Crucially, DDIM's TV distance bound not only depends on the mean square score estimation error but also on an additional mean Jacobian estimation error, indicating its potentially higher sensitivity to the quality of score estimates compared to DDPM.

1. Automatic Low-Dimensional Adaptation:

• The most striking finding is that both DDPM and DDIM possess an automatic low-dimensional adaptation capability. While general theory suggests iteration counts scale with the ambient dimension $D$, practical observations show much faster convergence (hundreds of iterations vs. millions predicted by $D$).
• This discrepancy is resolved by demonstrating that if the target data distribution possesses an intrinsic dimension $k$ (significantly lower than $D$, e.g., $k=43$ for ImageNet's $D=150k$), the total number of iterations required to achieve a TV distance of $\epsilon$ scales as $k/\epsilon$, rather than $D/\epsilon$. This represents a dramatic speedup that aligns with empirical observations.

1. Criticality of Coefficient Choices:

• The talk reveals that the specific choice of coefficients (e.g., $\eta_t$, $\sigma_t$ for DDPM/DDIM, which are derived from the forward process noise schedule $\alpha_t$) is paramount for achieving this low-dimensional adaptation.
• The original coefficient choices proposed in DDPM and DDIM papers are shown to be precisely those that enable this adaptation, effectively setting a specific "orange term" in a complex lower bound to zero. Arbitrary alternative coefficient choices can destroy this property, leading to iteration counts that scale with $\sqrt{D}$ rather than $k$, effectively losing the low-dimensional advantage.

1. Mathematical Mechanism for Adaptation:

• The adaptation is explained through Tweedie's formula, which allows the drift term of the reverse-time SDE/ODE to be decoupled into a linear component and a non-linear component. The non-linear component is shown to act as a projection onto the low-dimensional data manifold, effectively pushing samples towards the intrinsic data structure.

These findings collectively provide a robust theoretical explanation for the observed efficiency and performance of DDPM and DDIM, particularly in high-dimensional settings where data often lies on lower-dimensional manifolds.

Technical Deep Dive

▶ Watch: General convergence theory: DDIM and DDPM basics. (4:00)

The technical exposition begins by precisely defining the DDPM and DDIM sampling algorithms.

The forward process starts with $X_0 \sim P_{data}$ and at each step $t$, generates $X_t = \alpha_t X_0 + \sqrt{1-\alpha_t} Z_t$, where $Z_t$ is Gaussian noise and $\alpha_t$ are parameters defining the noise schedule.

The DDPM sampler is described as a stochastic sampler. Starting from $Y_T \sim \mathcal{N}(0, I)$, each subsequent iterate $Y_{t-1}$ is computed as a linear combination of three components: the current iterate $Y_t$, its estimated score function $s_t(Y_t)$, and an additional Gaussian noise term. The update rule is given by:

$Y_{t-1} = c_1 Y_t + c_2 s_t(Y_t) + \sigma_t Z'$, where $Z' \sim \mathcal{N}(0, I)$, and $c_1, c_2, \sigma_t$ are coefficients dependent on the noise schedule.

The DDIM sampler, in contrast, is a deterministic sampler. Its update rule is similar but omits the additional Gaussian noise term, making its trajectory fully deterministic after initialization.

$Y_{t-1} = c_1 Y_t + c_2 s_t(Y_t)$.

The talk then revisits the well-known continuous-time interpretation. The discrete forward process corresponds to a simple SDE. Crucially, classical results (dating back over 40 years) show that a reverse-time SDE exists that precisely reverses the forward SDE, sharing the same pathwise distribution. The drift term of this reverse-time SDE explicitly includes the true score function, $s^_t(X_t)$. DDPM is precisely a time-discretized version of this reverse-time SDE. Similarly, a probability flow ODE (a deterministic counterpart) exists, and DDIM is its time-discretized approximation. These continuous-time results guarantee perfect data generation if* the exact score function is available.

The core technical challenge lies in analyzing the discrete-time processes with imperfect score estimates. Two primary error sources are identified:

1. Discretization error: Arising from approximating continuous SDE/ODE paths with discrete steps.
2. Score estimation error: The difference between the estimated score function $s_t(Y_t)$ and the true score $s^*_t(Y_t)$.

To develop a general convergence theory, the speaker outlines necessary assumptions. The data distribution $P_{data}$ is assumed to have a polynomially bounded support size, but no strong assumptions like smoothness or log-concavity are imposed, making the theory widely applicable. For score estimates, the theory primarily uses the mean square estimation error $\mathbb{E}[\|s_t(Y_t) - s^*_t(Y_t)\|^2]$, denoted as $\epsilon_{score}$. For DDIM, this is insufficient, and an additional assumption on the mean Jacobian estimation error of the score function is required due to its deterministic and potentially less smooth trajectory.

The convergence guarantees are summarized:

• DDPM: For a total of $T$ steps, the KL divergence between the generated distribution and the true data distribution is upper bounded by $D/T + \epsilon_{score}$. For TV distance, it's bounded by $\sqrt{D}/T + \epsilon_{score}$. Translating this to iterations needed: to achieve a KL divergence bounded by $\epsilon^2$, $T \sim D/\epsilon^2$ iterations are needed; for TV distance bounded by $\epsilon$, $T \sim D/\epsilon$ iterations. These results were established by Benton et al. (for KL) and by Gen Li and Yuling Yan (for TV). A key property is DDPM's graceful degradation with increasing $\epsilon_{score}$.
• DDIM: Only TV distance guarantees are provided, as KL divergence can be unbounded. The TV distance bound depends on $D/T$, $\epsilon_{score}$, and the additional Jacobian error term, highlighting its distinct sensitivity.

The speaker briefly touches upon analysis strategies, mentioning the use of change of measure results from SDE literature for KL divergence, and stochastic localization for handling discretization error when mapping to continuous time. A second, more direct strategy involves analyzing the discrete-time process itself, which led to the state-of-the-art TV-based theories for both DDPM and DDIM.

The most significant technical contribution is the theory of low-dimensional adaptation. The gap between theoretical $D/\epsilon$ iteration counts and practical hundreds of iterations is explained by the data's intrinsic dimension $k$. This is formally defined via the log of the covering number of the data distribution's support, a metric that encompasses various low-dimensional structures like subspaces and manifolds. The new theory demonstrates that for data with intrinsic dimension $k$, the TV distance for DDPM/DDIM scales as $k/\epsilon$ iterations, a dramatic improvement over $D/\epsilon$.

This adaptation capability is rooted in the structure of the reverse-time SDE's drift term. Using Tweedie's formula, the drift term (which includes the score function) can be decomposed into a linear component and a non-linear component resembling a conditional mean or posterior mean of the original data $X_0$. If $X_0$ lies on a low-dimensional manifold, this non-linear term effectively acts as a projection operator, continuously pushing the sample trajectory towards that manifold.

Crucially, this low-dimensional adaptation is not automatic for any choice of sampler coefficients. The coefficients $\eta_t$ and $\sigma_t$ (for DDPM) and similar terms (for DDIM) are fully determined by the forward process noise schedule $\alpha_t$. The original choices in DDPM and DDIM papers were derived by optimizing variational lower bounds and are precisely those that enable this adaptation. A lower bound analysis shows that if these coefficients are chosen improperly, the TV distance can scale with $\sqrt{D}$, indicating a complete loss of the low-dimensional adaptation. This implies that the specific design of DDPM and DDIM is not arbitrary but mathematically optimized to leverage intrinsic data structures.

Experimental Setup & Results

▶ Watch: Detailed explanation of the DDIM deterministic sampler algorithm. (6:00)

This talk primarily focuses on the theoretical underpinnings of diffusion model sampling. As such, it does not present specific experimental setups, datasets, or empirical results. Instead, the focus is on developing rigorous mathematical proofs and convergence guarantees that explain observed practical phenomena, such as the efficiency of DDPM and DDIM in high-dimensional settings where data exhibits an intrinsic low-dimensional structure. The speaker references the empirical observation that diffusion models generate high-quality samples within hundreds of iterations, even for complex datasets like ImageNet (with an ambient dimension $D$ of approximately 150,000 pixels), contrasting this with the significantly higher iteration counts ($D/\epsilon$) predicted by general, worst-case theoretical bounds. The core contribution is to provide the mathematical framework that reconciles this gap, rather than presenting new experimental data.

Practical Implications

▶ Watch: Understanding DDPM/DDIM through stochastic differential equations. (6:40)

The theoretical insights presented in this talk have profound practical implications for anyone working with diffusion models:

• For Practitioners and Model Builders: The most significant takeaway is the strong theoretical justification for the observed efficiency of DDPM and DDIM. Even when generating high-resolution images or complex data in very high ambient dimensions ($D$), the models effectively operate on a much lower intrinsic dimension ($k$). This explains why hundreds, rather than millions, of sampling steps suffice for high-quality generation. This understanding provides confidence in the fundamental design of these models and validates current practices.
• Importance of Coefficient Design: The talk unequivocally highlights the critical role of the sampler coefficients (e.g., $\eta_t, \sigma_t$ for DDPM/DDIM). These coefficients, which are determined by the forward noise schedule ($\alpha_t$), are not arbitrary. The speaker demonstrates that only specific choices, precisely those proposed in the original DDPM and DDIM papers, enable the crucial low-dimensional adaptation. Deviating from these carefully derived coefficients can lead to a dramatic loss of efficiency, causing sampling steps to scale with $\sqrt{D}$ rather than $k$, effectively negating the benefits of intrinsic low dimensionality. This means that future research into novel sampler designs must carefully consider these coefficients to maintain performance.
• Robustness to Imperfect Score Estimates: The finding that DDPM degrades gracefully with increasing score estimation error is highly valuable for real-world deployment. In practice, the score function is always estimated by a neural network, which will inevitably have some error. DDPM's stability to this imperfection makes it a more robust choice for many applications, especially where perfect score function learning is challenging.
• Tradeoffs between DDPM and DDIM: The theory clarifies the different requirements for convergence guarantees for DDPM and DDIM. DDIM's reliance on Jacobian error estimates, in addition to mean square score error, suggests it might be more sensitive to the quality and smoothness of the learned score function. This provides a theoretical basis for choosing between stochastic (DDPM) and deterministic (DDIM) samplers based on the specific application and the reliability of the score model.
• Guidance for Future Research: By explicitly identifying the mathematical mechanisms behind low-dimensional adaptation and the critical role of coefficients, the talk opens avenues for designing even more efficient sampling algorithms. Understanding these principles can inform the development of adaptive noise schedules, improved discretization schemes, and potentially novel samplers that further optimize for intrinsic data structures. The mention of "training-free acceleration" in the omitted third part hints at further practical gains.
• Resource Planning for Infra Teams: Knowing that iteration counts scale with intrinsic dimension $k$ rather than ambient $D$ helps in more accurately predicting the computational resources and time required for generating samples, leading to more efficient infrastructure planning.

In summary, this talk transforms empirical observations into rigorous mathematical principles, empowering practitioners to make more informed design choices and providing a solid foundation for future innovations in diffusion model sampling.

Key Takeaways

• Mathematical Foundation for Sampling: The talk provides a rigorous mathematical framework for understanding the sampling stage of diffusion models, specifically DDPM and DDIM, addressing the practical challenges of discrete time and imperfect score estimates.
• Convergence Guarantees: Comprehensive convergence guarantees (via KL divergence and Total Variation distance) are established for DDPM and DDIM under minimal assumptions, showing how sampling fidelity relates to the number of steps, ambient dimension, and score estimation error.
• Automatic Low-Dimensional Adaptation: A major discovery is that DDPM and DDIM automatically leverage the intrinsic low-dimensional structure ($k$) of data, leading to a dramatic speedup in sampling efficiency where iterations scale with $k$ rather than the high ambient dimension ($D$).
• Criticality of Sampler Coefficients: The specific choice of sampler coefficients (derived from the forward process noise schedule) is shown to be absolutely essential for preserving this low-dimensional adaptation property; arbitrary choices can lead to significantly worse performance.
• DDPM Robustness: DDPM exhibits graceful degradation with increasing score estimation error, making it a robust choice for real-world applications where perfect score functions are unattainable.
• Bridging Theory and Practice: This work effectively bridges the gap between general, worst-case theoretical bounds and the remarkable efficiency observed in practical diffusion model applications, providing a deeper understanding of "why" these models work so well.

About the Speaker(s)

Yuxin Chen is the primary speaker for this specific lecture, focusing on the mathematical foundations of diffusion model sampling. He is a prominent researcher in the field, likely a professor, whose work centers on developing rigorous theoretical guarantees for machine learning algorithms, optimization, and signal processing. His expertise is evident in the detailed mathematical analysis and the depth of the convergence theory presented.

Qing Qu and Liyue Shen are listed as co-speakers for the broader tutorial, suggesting they are also leading experts in diffusion models. Qing Qu's contributions were mentioned by Yuxin Chen as preceding this lecture, likely covering the score learning aspect of diffusion models. Together, the speakers present a comprehensive overview, from the practical challenges of score estimation to the theoretical underpinnings of efficient sampling.