How AI is trained: Pre-training, mid-training, and post-t...

LLM training is segmented into pre-training, mid-training, and post-training, each with distinct objectives. Pre-training focuses on foundational language understanding through next-token prediction on vast datasets like internet text, books, and papers. This initial phase establishes a broad knowledge base, with recent advancements incorporating synthetic data to improve efficiency and quality.

Mid-training, formerly part of pre-training, specializes in refining specific capabilities. For example, it can focus on long-context documents to prevent catastrophic forgetting, a common issue in neural networks where learning new information can lead to forgetting older knowledge. This phase uses the same algorithms but applies them to more specialized datasets.

Post-training involves refinement stages such as supervised fine-tuning, Direct Preference Optimization (DPO), and Reinforcement Learning with Human Feedback (RLHF). This phase concentrates on skill acquisition and problem-solving using the knowledge gained during pre-training, rather than acquiring new factual knowledge. It helps the model unlock its potential by teaching it how to apply its learned information effectively.

Pre-training data has evolved beyond raw internet scrapes to include sophisticated synthetic data. This isn't just AI-generated content; it involves rephrasing existing high-quality documents (e.g., Wikipedia articles into Q&A formats) or summarizing texts to create better-structured data. This refinement allows LLMs to learn more efficiently, as structured and grammatically correct data facilitates faster learning compared to noisy, uncurated sources.

Techniques like Optical Character Recognition (OCR) are used to extract text from PDFs and other digital documents, converting trillions of tokens into usable training data. This process, exemplified by tools like DeepSea and CR, expands the available pool of potential data. Labs create vast funnels of candidate data, then filter it to train models on a smaller, higher-quality subset, often measured in tens to hundreds of trillions of tokens. The selection of data directly impacts model performance, with higher quality data leading to superior outcomes.

Data quality is a critical factor influencing model performance and training speed. Higher quality data allows LLMs to train faster and achieve better results, even with less computational power. Clean, structured data with correct grammar and punctuation enables the model to learn the correct patterns from the outset, rather than spending resources correcting errors from messy input.

Optimizing data quality involves scientific methods, such as training classifiers to prune vast datasets like Common Crawl into high-quality subsets tailored for specific tasks. For instance, to train a reasoning model proficient in math and code, the data mix must be reconfigured to include relevant sources. This involves sampling small amounts from various sources (GitHub, Stack Exchange, Reddit, Wikipedia), training small models on these mixes, and then measuring performance to determine the optimal dataset composition. If evaluation metrics change, the optimal dataset mix also changes, necessitating continuous re-evolution of data strategies.

Unexpected sources like Reddit, when properly filtered, can provide valuable training data. PDFs, particularly from scientific archives such as Semantic Scholar, also constitute a significant source of high-quality, openly accessible information. Labs like AI2 actively scrape and process these documents to extract valuable data for pre-training.

Frontier labs often invest heavily in skilled researchers dedicated to finding, cleaning, and integrating new, better data. This labor-intensive process is crucial for making an impact in AI development. While algorithmic breakthroughs are celebrated, many practical advancements stem from improving data input or enhancing infrastructure to accelerate experiments.

The legal and ethical implications of using training data are becoming increasingly contentious. Companies often guard their training data secrets due to competitive and legal reasons. A distinction exists between training on unlicensed data, such as general scrapes from the internet (e.g., Common Crawl), and licensed data, where explicit consent for use has been granted.

Cases like Anthropic's lawsuit, where they faced a $1.5 billion penalty for using pirated books alongside legitimately purchased ones, highlight the legal risks. The debate centers on whether purchasing content, like an Amazon Kindle book, grants the right to use it for training LLMs. This 'gray zone' creates uncertainty for developers and emphasizes the need for a compensation scheme, similar to how Spotify redefined music streaming royalties, to address authors' intellectual property rights.

The current focus on general-purpose LLMs, exemplified by models like ChatGPT, only scratches the surface of AI's potential. A significant future development lies in domain-specific LLMs, which will be trained on proprietary data within specific industries such as pharmaceuticals, law, or finance. This approach could unlock new capabilities not currently achievable with generalized models.

Access to highly specialized data, like clinical trial results, is a major barrier. As LLMs become more commoditized, these industries will likely hire AI experts to build in-house models tailored to their unique datasets. This shift will lead to another wave of scaling and innovation, driven by the ability to leverage previously unavailable, context-rich information, pushing the boundaries of what LLMs can achieve in targeted applications.