An analytic theory of creativity in convolutional diffusion models

An analytic theory of creativity in convolutional diffusion models

Speakers: Mason Kamb, Surya Ganguli

Conference: ICML 2025

YouTube: https://slideslive.com/39043899

Overview

This talk, presented by Mason Kamb and Surya Ganguli at ICML 2025, introduces a groundbreaking analytic theory aimed at explaining the origins of combinatorial creativity and spatial consistency failures in convolutional diffusion models. The central premise is that these seemingly disparate phenomena—the ability of models like DALL-E 2 to blend concepts like "Napoleon cat" and "nebula dog," alongside common artifacts such as extra limbs or distorted features—stem from a single underlying mechanism. This mechanism is rooted in the fundamental architectural constraints of convolutional neural networks (CNNs), specifically translational equivariance and locality.

The researchers propose that by studying the simplest yet non-trivial generative image models—those built on fully convolutional backbones—they can analytically derive how these architectural biases lead to both imaginative generalization and predictable errors. Their theory offers a unified, highly predictive explanation for the behavior of these models, demonstrating an unprecedented level of theory-experiment agreement in deep generative image models. This work is crucial for understanding the foundational principles governing how diffusion models generalize, providing insights that could inform the design of more robust and controllable generative AI systems.

Background

▶ Watch: Introduction: Combinatorial creativity & consistency issues (0:00)

Diffusion models have emerged as a dominant paradigm in generative AI, capable of producing remarkably high-quality and diverse images. The core idea behind these models involves a forward process that progressively adds noise to an image, transforming it into pure Gaussian noise. This process can be described by a stochastic differential equation (SDE). The generative capability comes from learning the reverse process, an ordinary differential equation (ODE) that starts from noise and iteratively denoises it back into a coherent image, effectively advecting probability mass towards the training data distribution.

A critical component of the reverse process is the score function, denoted as $\nabla \log p_t(x_t)$, which encapsulates the complex structure of the data distribution. Tweedie's theorem provides a powerful simplification: the score function is exactly equal to the conditional expectation of the noise given a corrupted signal. This means that if a neural network can be trained to optimally predict the noise added to a partially corrupted image, it effectively learns the score function necessary for the reverse process.

However, a fundamental paradox arises here. If a diffusion model were to perfectly learn this optimal denoising function, it would, by definition, perfectly recover the true training distribution. For a discrete dataset, this distribution is a sum of delta functions centered on each training example. Such a perfectly optimized model would therefore only memorize and reproduce its training examples, exhibiting no generalization or creativity. This directly contradicts the observed phenomenon of combinatorial creativity, where diffusion models can generate novel combinations of attributes (e.g., "bear Times Square") that could not plausibly have appeared together in the training data. The resolution of this paradox, the authors argue, must lie in the fact that diffusion models are implicitly biased or constrained, preventing them from optimally achieving their training objective. It is this underfitting that enables generalization and creativity.

To investigate this underfitting, the researchers focused on the architecture of early diffusion models, specifically those parameterized by fully convolutional neural networks (CNNs), eschewing more modern components like self-attention. While highly expressive, CNNs are inherently limited by two key architectural constraints:

1. Translational Equivariance: If an input image is shifted by a certain number of pixels, the output feature map (and ultimately the generated image) will shift commensurately. This implies that the model's internal representations do not encode absolute positional information.
2. Locality: The output at any given pixel location depends only on a local window of pixels in the input, defined by the model's receptive field. This arises from the finite width of convolutional kernels and the finite depth of the network, meaning information flow is spatially constrained.

By examining how an optimal denoiser would behave under these specific constraints, the talk aims to uncover the mechanistic origins of combinatorial creativity and spatial consistency failures.

Key Findings

▶ Watch: Reverse process, score function, and Tweedie's theorem (2:30)

The talk presents several key findings that establish a unified analytic theory for creativity and consistency issues in convolutional diffusion models:

• Unified Mechanism: The core insight is that both combinatorial creativity (the ability to mix and match attributes from disparate training examples) and spatial consistency issues (e.g., generating too many fingers or limbs) arise from the same underlying mechanism: the combined effects of locality and translational equivariance inherent in convolutional neural networks.
• Bayes Optimal Denoising under Constraints: The researchers analytically derived the Bayes optimal denoising strategy for a model constrained by locality and translational equivariance. They demonstrated that, under these conditions, each pixel independently forms its own "belief" about which training example it originated from, based only on its local context. This local, uncoordinated decision-making is the source of combinatorial mixing.
• Automatic Combinatorial Creativity: The theory predicts that convolutional diffusion models will exhibit combinatorial creativity by default. As the denoising process unfolds, different regions of the image are drawn towards different training examples (or patches thereof) that best match their local context, leading to novel combinations without explicit intervention.
• Prediction of Spatial Consistency Issues: Concurrently, the theory explains spatial consistency failures. Because pixels make local decisions without global awareness, the model cannot "count" elements across the entire image. This leads to stochastic and potentially incorrect numbers of repeated features, such as extra limbs or distorted objects, which are famously observed in diffusion model outputs.
• High Theory-Experiment Agreement: The analytical predictions were rigorously tested against actual small CNN-based diffusion models (ResNet and U-Net) trained on standard computer vision datasets (MNIST, Fashion-MNIST, CIFAR-10, CelebA). The results showed strikingly accurate correspondence, with quantitative correlations frequently exceeding 90%. This represents an exceptionally high level of agreement between theory and experiment for deep generative models.
• Emergent Locality Scale: The theory identifies a crucial hyperparameter: the effective locality scale used by the model during denoising. The experiments revealed that models dynamically adapt this scale, using a larger scale at high noise levels (where individual training examples are not yet resolved) and progressively decreasing it as noise reduces. This time-decreasing scale is critical for preventing the model from memorizing specific training examples too early in the denoising process.
• Insights into Attention-Enabled Models: Even though the primary focus was on CNNs, preliminary investigations into self-attention-enabled models (like Diffusion Transformers) showed that while they achieve slightly more non-local coherence, many of their observed defects could still be traced back to the principles of the local theory. This suggests that locality remains a powerful inductive bias even in more complex architectures, with self-attention acting as a non-local corrective mechanism.

Technical Deep Dive

▶ Watch: Paradox: Optimal denoising vs. combinatorial creativity (3:40)

The technical core of this work lies in understanding how the fundamental architectural inductive biases of convolutional neural networks—translational equivariance and locality—constrain the learning process of a diffusion model, thereby enabling generalization and simultaneously introducing characteristic failure modes.

At a high level, diffusion models learn to reverse a gradual noise-adding process. The forward process $q(x_t|x_0)$ adds Gaussian noise to an initial image $x_0$ over time $t$. The reverse process $p(x_{t-1}|x_t)$ aims to denoise $x_t$ back to $x_{t-1}$. This reverse process is governed by the score function $\nabla \log p_t(x_t)$, which can be estimated by training a neural network to predict the noise $\epsilon$ added to a noisy image $x_t = \sqrt{\bar{\alpha}_t} x_0 + \sqrt{1-\bar{\alpha}_t} \epsilon$. According to Tweedie's theorem, the optimal noise prediction $\mathbb{E}[\epsilon | x_t]$ is directly related to the score function.

The critical insight is that if a model could perfectly learn this optimal noise prediction, it would perfectly reconstruct the true data distribution $p(x_0)$. For a finite training dataset, this distribution is effectively a sum of delta functions at each training example. A perfect denoiser would thus simply reproduce memorized training examples, exhibiting no creativity. Therefore, the observed combinatorial creativity must arise from the model's inability to perfectly fit this objective, due to its inherent biases.

The researchers then analyze the Bayes optimal denoiser under the specific constraints of CNNs:

1. Locality: A CNN's receptive field means that the output at any pixel $(i,j)$ depends only on input pixels within a local window around $(i,j)$. This implies that the model cannot "see" the entire image when making a denoising decision for a specific pixel or patch. In a Bayes optimal sense, this means that instead of forming a single, global Bayes weight over the entire training dataset for the whole image, each pixel (or local region) must form its own independent Bayes weight about which training example it most likely originated from, based only on the information available within its local context. This breaks the global coherence of belief about the underlying true image.

1. Translational Equivariance: This constraint means that the model's internal representations are blind to the absolute position of features in the image; shifting the input merely shifts the output. In the context of optimal denoising, this is tantamount to "deleting a pixel's knowledge of its location within the image." Consequently, the Bayes optimal strategy under this constraint requires summing over every possible translation of every training set image. Essentially, the model considers every possible patch from every training image, regardless of its original position, as a potential source for the current local noisy patch.

Combining these two constraints, the equivariant local denoiser operates as follows:

• Each local region (pixel or patch) in the noisy image independently evaluates its likelihood of originating from any local patch within any training image, considering all possible translations.
• During the reverse denoising process, as noise is reduced, each local region is drawn towards the training example patch that it most strongly "believes" it came from, based on its local Bayes weights.
• Crucially, because these decisions are made locally and independently, different regions of the generated image can be drawn towards patches from different training examples. This uncoordinated merging of locally optimal "beliefs" from diverse sources is precisely the mechanism that generates combinatorial creativity. For instance, one region might locally resemble a "cat" from one image, while an adjacent region might resemble "Napoleon's hat" from another, leading to a "Napoleon cat" even if no such image existed in the training data.

This same mechanism also explains spatial consistency issues. If a model locally decides to generate a "pant leg" based on its local context, it has no global mechanism to "count" how many pant legs already exist in other parts of the image. This leads to a stochastic number of features, resulting in artifacts like "pants with too many legs" or "shirts with too many arms." The model is locally optimal but globally incoherent.

The theory also introduces the concept of an emergent locality scale. While the receptive field defines the maximum possible locality, the effective scale used by the model is dynamic. At high noise levels, many distinct training patches become indistinguishable. The model can safely use a large locality scale without prematurely collapsing to a single training example. However, as noise decreases, individual patches become more distinct. To prevent the model from memorizing specific training examples (which would lead back to the "no creativity" problem), the effective locality scale must progressively decrease. This allows the model to continue mixing and matching at finer granularities without committing to a single training instance, thereby avoiding memorization. The authors observed that models indeed exhibit this time-decreasing locality scale, which is crucial for maintaining generalization.

Experimental Setup & Results

▶ Watch: Key constraints: Translational equivariance and locality (5:50)

To validate their analytic theory, the researchers conducted experiments using simplified yet representative diffusion models.

Models Studied:

The primary models investigated were small, fully convolutional neural networks:

• ResNet backbone: A variant of the Residual Network architecture.
• U-Net backbone: A U-shaped architecture commonly used in image-to-image tasks, often forming the basis for many diffusion models.

These models were chosen specifically because they rely purely on convolutional operations, embodying the translational equivariance and locality constraints central to the theory, rather than incorporating self-attention mechanisms.

Datasets:

Experiments were performed across four standard computer vision datasets, covering a range of complexity and image types:

• MNIST: Handwritten digits (grayscale, simple structures).
• Fashion-MNIST: Images of clothing items (grayscale, more complex shapes than MNIST).
• CIFAR-10: Small color images of common objects (low resolution, diverse categories).
• CelebA: Faces of celebrities (higher resolution, more intricate details, but still relatively constrained pose/expression).

Methodology:

The core experimental approach involved a direct comparison between the theoretical predictions and the actual behavior of the trained neural networks. For a given input noise seed, the researchers:

1. Calculated the output predicted by their analytic theory (the Bayes optimal denoiser under locality and equivariance constraints).
2. Generated an image using the trained neural network (ResNet or U-Net) from the same initial noise seed.
3. Compared these two outputs.

Key Qualitative Results:

The visual correspondence between the theoretical predictions and the neural network outputs was described as "strikingly accurate." Across all datasets, the image pairs (theory vs. experiment) showed remarkable alignment, even for generated images that were "very very far from any individual image in the training set." This strongly supported the theory's ability to predict both novel combinations and characteristic errors.

For instance, on Fashion-MNIST, the theory accurately predicted spatial consistency issues such as:

• Pants with "too many legs."
• Shirts with "too many arms."
• Shoes with "too many toes."

These defects were clearly visible in both the theoretical and experimental outputs, underscoring the common mechanistic origin.

Key Quantitative Results:

Beyond qualitative observation, the agreement was also quantified. For most configurations and metrics analyzed, the correlations between the theory and the experiment were in the 90%+ range. This exceptionally high quantitative agreement provides strong evidence for the validity and predictive power of the proposed analytic framework.

Insights from Self-Attention Enabled Models:

While the primary focus was on CNNs, the researchers also conducted preliminary tests with attention-enabled models. They found that even for these more complex architectures, the local theory could reproduce outputs with "reasonable amount of correlation." Attention-enabled models did exhibit "a little bit more non-local coherence," allowing them to "carve out a semantically coherent foreground image" from the more "textural background" typically produced by purely local models. However, crucially, some of the defects observed in self-attention models (e.g., a "dog face that has three eyes") could still be "traced back to the local theory." This suggests that while self-attention adds non-local corrective capabilities, the fundamental inductive biases of locality might still exert a strong influence, or that self-attention itself can sometimes introduce its own forms of local misinterpretations.

Overall, the experimental results provide compelling empirical validation for the analytic theory, demonstrating its ability to accurately predict the complex emergent behaviors of combinatorial creativity and spatial inconsistency in diffusion models constrained by convolutional architectures.

Practical Implications

▶ Watch: How locality impacts Bayes optimal denoising strategy (6:00)

The analytical theory presented in this talk offers profound practical implications for researchers, infrastructure teams, model builders, and deployers working with generative AI.

Firstly, by providing a mechanistic explanation for combinatorial creativity, this work moves beyond treating diffusion models as black boxes. Understanding that creativity stems from the interplay of locality and translational equivariance allows practitioners to approach model design and behavior with a more principled, interpretable framework. This is crucial for building trust in generative AI and for debugging unexpected outputs.

For model builders and architects, the findings highlight the critical role of architectural inductive biases. If the goal is to enhance combinatorial creativity, deliberately designing models that leverage local, independent decision-making could be a strategy. Conversely, if the aim is to mitigate spatial consistency issues (like extra limbs or distorted objects), the theory points towards the necessity of incorporating mechanisms that enforce global coherence and allow for "counting" or understanding relationships across distant parts of an image. While self-attention models show some improvement in non-local coherence, the fact that their defects can still be traced to local theory suggests that even they do not fully overcome this fundamental challenge. This might spur research into novel architectural components or training objectives that explicitly enforce global consistency constraints.

Infrastructure teams and those involved in model deployment can benefit from a better understanding of the inherent tradeoffs. The talk suggests that combinatorial creativity, as we currently observe it, might be inextricably linked to spatial inconsistencies. Achieving higher levels of novel generation might inherently come with a certain rate of "failures" in realism or anatomical correctness. This understanding can inform quality control processes, the design of post-processing pipelines (e.g., for correcting artifacts), and the setting of realistic expectations for model performance. It implies that simply scaling up existing architectures might not eliminate these issues if the underlying architectural biases remain.

The concept of an emergent, time-decreasing locality scale also has practical relevance. If models implicitly learn to adapt their effective receptive field during denoising to avoid memorization, this dynamic behavior could be explicitly leveraged or controlled. For example, curriculum learning strategies could be designed to guide the model's locality scale, potentially optimizing the balance between generalization and fidelity.

Limitations and Tradeoffs: It's important to acknowledge that the theory is developed for simpler, CNN-based diffusion models. While the conceptual picture is believed to extend more generally, its direct applicability to state-of-the-art models like DALL-E 2 or Diffusion Transformers (which heavily rely on self-attention) needs further investigation. These advanced models incorporate many other components (e.g., conditioning mechanisms, larger datasets, higher resolutions) that could introduce additional complexities or modify the impact of locality. The "not very good" sample quality of the models studied in the experiments also indicates that while the theory explains how creativity and errors arise, it doesn't necessarily dictate high-fidelity generation in general. Nevertheless, by offering a foundational, interpretable theory for a complex emergent behavior, this work provides a crucial stepping stone towards designing more controllable, predictable, and ultimately more capable generative AI systems.

Key Takeaways

• Combinatorial creativity and spatial consistency issues in diffusion models stem from a unified mechanism rooted in architectural biases.
• This mechanism is the combined effect of locality (pixels make decisions based on local context) and translational equivariance (model ignores absolute position) in convolutional neural networks.
• Locality forces pixels to form independent "beliefs" about their origin, leading to automatic mixing and matching of features from disparate training examples, thus inducing creativity.
• The lack of global coordination due to locality also explains spatial consistency failures, such as generating extra limbs or distorted features.
• An analytic theory accurately predicts the behavior of CNN-based diffusion models, demonstrating over 90% correlation between theoretical predictions and experimental outputs.
• Models exhibit an emergent, time-decreasing locality scale during denoising, which is crucial for preventing memorization of training examples.
• Even self-attention-enabled models, while offering some non-local coherence, still exhibit defects traceable to the principles of the local theory, suggesting locality remains a powerful underlying factor.

About the Speaker(s)

Mason Kamb and Surya Ganguli are researchers who presented this work at ICML 2025. Their collaboration focuses on developing an analytical understanding of complex phenomena in deep learning, particularly in the context of generative models like diffusion models. Their research aims to demystify the emergent behaviors of AI systems by deriving theoretical frameworks that explain their performance and limitations.