The ML Technique Every Founder Should Know...

Diffusion is a foundational machine learning framework that enables learning any probability distribution of data across any domain, provided sufficient data is available. It is particularly effective for mapping high-dimensional inputs to high-dimensional outputs, even with limited training data. For example, it can map 30 images within a 3-million-dimensional space to another 3-million-dimensional space.

The core process involves taking a data sample, such as an image, and progressively adding noise to it to create a sequence of increasingly noisy versions. The challenge lies in reversing this easy process: teaching a model to denoise from random static back to the original data. The trained model learns to reverse this noising process, making it an effective denoiser.

Diffusion models find applications in a surprisingly wide range of fields beyond their origins in image processing. Initially applied to image datasets like CIFAR-10, their utility has expanded significantly. DeepMind utilized diffusion for protein folding, contributing to a Nobel Prize-winning achievement, and the diffusion policy paper demonstrated its capability in driving cars.

Other notable applications include predicting weather, generating images and videos (like Stable Diffusion, Sora), and developing new models in life sciences, such as predicting small molecule binding to proteins (e.g., DiffDock). The technology's versatility allows a single core methodology to be deployed across diverse scientific and engineering challenges, from text generation in continuous/discrete diffusion LLMs and robotic policies to failure sampling and even code generation. The only remaining holdouts where diffusion has not yet surpassed alternative state-of-the-art models are LLMs and gameplay (e.g., AlphaGo's Monte Carlo Tree Search).

The 2015 paper by Josha established the foundational components of modern diffusion models. Subsequent innovations focused on refining specific elements, such as the noise addition process ('how we add noise at what weight'), loss functions, and model architectures. Early work explored various loss functions, including predicting the previous data state, predicting the added error, or predicting the velocity of change.

Improvements involved 'hill climbing' on metrics like the Fréchet Inception Distance (FID), iteratively finding 'easier' objectives for the model to learn. Predicting the actual data proved hard, predicting the error was easier, and predicting velocity became even simpler, culminating in approaches like flow matching. This evolution also saw architectural shifts from UNets to diffusion transformers with cross-attention mechanisms, consistently improving performance as measured by FID. Mathematically, these advancements often led to simpler equations and more concise code.

A crucial component of stable diffusion model training is the carefully designed 'noise schedule.' Intuitively, one might linearly interpolate between an image and noise, gradually adding noise. However, this approach proves massively unstable because the instantaneous error added is very small initially but rapidly increases towards the end of the process, requiring models to handle vastly different error magnitudes.

Effective noise schedules (known as beta schedules) aim to introduce a relatively constant amount of error at each time step. This avoids instability by ensuring the model doesn't encounter disproportionately small or large changes at different points in the diffusion process. The cumulative sum of this error often follows a non-linear, 'one-minus-sigmoid' type curve. Getting this schedule right significantly contributes to the overall stability and performance of the diffusion model.

Flow Matching introduces a remarkably simple yet powerful method for training diffusion models. Instead of learning an intricate, circuitous path through intermediate noised states, Flow Matching focuses on learning a direct 'global velocity' vector between the noise and the original data. The model is trained to predict this velocity, regardless of its current position along the path from noise to data, encouraging it to consistently move towards the original data.

The training objective for Flow Matching is simple: minimize the loss between the predicted velocity and the actual global velocity (calculated as noise minus data). This method reduces the core training code to mere lines, making the procedure highly stable and less complex. Crucially, the model (which could be an RNN, UNet, or diffusion transformer) is abstracted, meaning the same five lines of code can apply to images, weather data, stock market data, robotics trajectories, proteins, or DNA, demonstrating its universal applicability.

The 'squint test' refers to evaluating AI architectures by comparing them to biological intelligence, acknowledging that evolution doesn't always lead to direct mimicry (e.g., airplanes don't flap wings like birds, but share the principle of two wings). Applying this to AI, LLMs operate by producing 'one token at a time' and lack the recursive, dynamic, and backtracking processes characteristic of human thought. Brains, in contrast, exhibit massive recursion, continuous learning, and parallel processing between hemispheres.

Diffusion models offer two key advantages against this 'squint test' that LLMs currently lack: the inherent leveraging of randomness (akin to biological systems where neurons are probabilistically random) and the ability to conceptualize and decode into larger chunks rather than single tokens. While not a complete solution for general intelligence, diffusion's use of randomness and its potential for more holistic output generation move closer to biological learning and thought processes, unlike the current bottleneck of sequential token generation in LLMs.

For those actively training machine learning models, irrespective of the application, diffusion procedures should be a fundamental consideration. The technique's capability to learn latent spaces and its broad applicability make it essential for enhancing training loops across diverse problem domains. This applies from image generation to robotics, emphasizing that founders and researchers should integrate or at least evaluate diffusion in their model development.

For those not directly involved in training models, it is crucial to update prior expectations regarding the capabilities of AI technologies. The rapid advancement in diffusion-based generative AI, particularly in image generation (Midjourney vs. Sora, Vox, Flux, SD3), illustrates that 'scaling up' is often the primary driver of progress. This principle, when applied to proteins, DNA, robotics, and self-driving cars, suggests that these capabilities will inevitably advance dramatically. The core procedure of diffusion is also continually improving, becoming simpler and more effective, indicating a future where robust AI solutions will become commonplace, driving significant economic transformation. Founders have opportunities to build companies either by developing new diffusion-based models or by leveraging existing ones.