A curated, chronological reading list tracing the architectural and algorithmic breakthroughs that built modern generative AI from the perceptron to DeepSeek-R1.
Turing didn’t ask if machines could think he asked if they could fool us into thinking they do.
Alan Turing reframed the unanswerable philosophical question “can machines think?” into an empirical, operational test: a human judge converses blind with a machine and a human, and tries to tell them apart. This sidestepped centuries of metaphysical debate by reducing intelligence to observable behavior rather than internal experience.
The paper set no benchmark scores, but its conceptual legacy is the entire field of AI evaluation every modern chatbot eval, from human preference studies to LLM-as-judge, is a descendant of the Turing Test.
The first machine that could learn from its mistakes, one weight adjustment at a time.
Frank Rosenblatt formalized the perceptron as a single-layer network that adjusts connection weights based on classification errors, drawing loosely on biological neuron models. This established the core mechanic iterative, data-driven weight updates that underlies all modern neural network training.
While limited to linearly separable problems, it proved empirically that machines could “learn” representations from data rather than being explicitly programmed with rules.
The algorithm that finally let neural networks go deep.
Rumelhart, Hinton, and Williams introduced backpropagation: an algorithm that efficiently computes the gradient of an error function with respect to every weight in a multi-layer network by propagating the error backward from output to input. This solved the core bottleneck that had stalled neural network research for decades.
Backpropagation remains the training backbone of essentially every deep network used today, from CNNs to transformers making it arguably the single most consequential algorithm in deep learning history.
Words became vectors, and language modeling was never the same.
Yoshua Bengio and co-authors proposed representing words as continuous, dense vectors (embeddings) learned jointly with a neural network that predicts the next word, replacing sparse, discrete n-gram counting methods. This let the model generalize across semantically similar words instead of treating each one as an isolated symbol.
The approach directly overcame the curse of dimensionality plaguing n-gram models, and its embedding-based paradigm became the foundation for word2vec, GloVe, and ultimately the input layers of every modern transformer.
AlexNet didn’t just win a competition it ended an era of hand-engineered computer vision.
Krizhevsky, Sutskever, and Hinton trained a deep convolutional neural network on GPUs using ReLU activations and dropout regularization, scaling depth and data in ways previously considered computationally impractical. This was one of the first demonstrations that deep CNNs could outperform classical feature-engineering pipelines at scale.
AlexNet achieved a top-5 error rate of 15.3% on ImageNet, more than 10 points better than the runner-up, igniting the modern deep learning boom and the shift toward GPU-accelerated training as standard practice.
Skip connections let networks get radically deeper without falling apart.
He, Zhang, Ren, and Sun introduced residual (“skip”) connections that let layers learn a residual mapping relative to their input rather than a full transformation, directly addressing vanishing/degrading gradients in very deep networks. This simple architectural change made it possible to stably train networks with hundreds or thousands of layers.
ResNet became a foundational building block reused across vision and even non-vision architectures, and its residual connection concept directly inspired the design of transformer layers.
Generative modeling met probabilistic inference, and the VAE was born.
Kingma and Welling introduced the Variational Autoencoder, which jointly trains an encoder and decoder by optimizing a tractable Evidence Lower Bound (ELBO) that balances reconstruction accuracy against a KL-divergence regularizer pulling the latent distribution toward a simple prior. This reparameterization trick made gradient-based optimization of latent-variable generative models practical for the first time.
VAEs established the latent-space generative paradigm later extended into structured domains like clinical patient simulation (e.g., VAMBN), and their compression principle underlies the autoencoders used in modern latent diffusion models.
Two competing networks, one zero-sum game, and suddenly machines could dream up photorealistic images.
Ian Goodfellow and co-authors framed generative modeling as a minimax game between a generator that synthesizes fake samples and a discriminator that tries to distinguish them from real data, with both networks improving through adversarial competition. This bypassed the need for explicit density estimation entirely.
GANs produced strikingly photorealistic outputs but suffered from unstable training and mode collapse, spawning an entire family of stabilizing variants (Conditional GAN, CycleGAN, Rev-GAN, MUNIT) that remain reference points for image-to-image translation tasks.
Add noise, then learn to undo it and you’ve got a new way to generate images.
Ho, Jain, and Abbeel modeled image generation as a reverse Markov chain: a forward process gradually corrupts data with Gaussian noise over many timesteps, and a neural network is trained to predict and remove that noise step by step to reconstruct clean samples. This reframed generation as iterative denoising rather than single-shot synthesis.
DDPMs delivered superior sample quality and training stability compared to GANs, at the cost of slow, multi-step inference — a tradeoff later mitigated by accelerated samplers like DDIM.
Move diffusion out of pixel space and into a compressed latent space, and suddenly it runs on a laptop GPU.
Rombach and team trained a perceptual autoencoder to compress images into a low-dimensional latent space, then applied the diffusion process and cross-attention text-conditioning entirely within that compressed manifold rather than on raw pixels. This dramatically cut the computational cost of high-resolution diffusion.
This latent-space approach became the technical backbone of Stable Diffusion, enabling high-fidelity text-to-image generation on consumer-grade hardware instead of requiring industrial compute clusters.
Recurrence and convolution were thrown out, and self-attention rebuilt sequence modeling from scratch.
Vaswani and colleagues introduced the Transformer, an architecture built entirely on self-attention layers that compute relationships between all tokens in a sequence in parallel, eliminating the sequential bottleneck of RNNs. Multi-head attention and positional encodings let the model capture both local and long-range dependencies efficiently.
The Transformer became the universal backbone for nearly all subsequent generative AI language, vision, audio, and video models making it perhaps the single most influential architecture paper of the decade.
At 175 billion parameters, the model learned new tasks just by reading a few examples in the prompt.
OpenAI scaled a decoder-only Transformer to 175B parameters and showed it could perform new tasks via in-context learning conditioning on a handful of examples at inference time — without any gradient updates or fine-tuning. This was a structural shift away from the pretrain-then-fine-tune workflow toward a single, dynamically adaptable model.
GPT-3 demonstrated strong few-shot performance across a wide range of NLP benchmarks and effectively launched the modern era of prompt engineering as a primary interface for using large models.
Most “huge” models were actually undertrained — and a smaller model fed more data beat them all.
Hoffmann and the DeepMind team trained over 400 models across varying parameter counts and data budgets to derive power-law scaling relationships, finding that for a fixed compute budget, parameters and training tokens should scale in equal proportion (roughly 20 tokens per parameter). They validated this by training Chinchilla, a 70B-parameter model, on 1.4 trillion tokens.
Chinchilla outperformed larger models like Gopher and GPT-3 despite having fewer parameters, fundamentally changing how the industry allocates compute between model size and data volume.
Compute-optimal training met open data, and suddenly state-of-the-art language models could run on a single workstation.
Touvron and the Meta AI team trained a family of models from 7B to 65B parameters exclusively on publicly available datasets, applying compute-optimal training principles at scale rather than relying on proprietary data. This prioritized inference efficiency and reproducibility over sheer parameter count.
LLaMA-13B outperformed the much larger GPT-3 on most benchmarks while being small enough to run on local hardware, catalyzing the modern open-source LLM ecosystem.
Self-attention’s quadratic cost finally met a linear-time challenger.
Gu and Dao built on Structured State Space Models (S4) and introduced a selection mechanism making the SSM parameters input-dependent, allowing the model to selectively propagate or discard information along the sequence. A custom hardware-aware parallel scan algorithm computes this efficiently without materializing huge intermediate states in GPU memory.
Mamba scales linearly (O(L)) in sequence length, delivering up to 5x higher inference throughput than comparable Transformers while matching or exceeding their quality on long-sequence tasks.
Images and text were mapped into the same space, unlocking zero-shot vision.
Radford and the OpenAI team trained a dual-encoder model contrastively on 400 million image-text pairs scraped from the internet, optimizing a symmetric contrastive loss that maximizes cosine similarity between correctly paired images and captions while minimizing it for mismatched pairs. This produced a shared multimodal embedding space without needing task-specific labels.
CLIP enabled strong zero-shot image classification and became a critical steering mechanism for text-to-image generative models like DALL-E 2.
A two-stage pipeline turned CLIP’s embeddings into vivid, controllable images.
Ramesh and colleagues built a two-stage system: a “prior” model that converts a text prompt into a CLIP image embedding, followed by a diffusion-based decoder that generates the final image conditioned on that embedding. This decoupled semantic understanding (via CLIP) from pixel-level synthesis (via diffusion).
The approach produced highly realistic, diverse images with strong text alignment, demonstrating that pretrained multimodal embeddings could effectively steer generative diffusion decoders.
The U-Net got replaced by a Transformer, and image quality scaled predictably with compute.
Peebles and Xie swapped the convolutional U-Net backbone common in latent diffusion models for a Transformer operating directly on latent image patches, using mechanisms like adaptive layer normalization, cross-attention, and extra conditioning tokens to inject class or text information. This unified diffusion architecture with the same scalable Transformer recipe used in language modeling.
DiT showed that increasing GFLOPs through greater depth, width, or token count reliably improves sample quality, achieving state-of-the-art FID scores and laying the architectural groundwork for video models like Sora.
A 1.3B-parameter aligned model beat a 175B-parameter raw GPT-3 alignment mattered more than scale.
Ouyang and the OpenAI team established the three-stage RLHF pipeline: supervised fine-tuning on human demonstrations, training a reward model on human-ranked response pairs, and then optimizing the policy against that reward model using PPO with a KL penalty to prevent drift from the reference model. This directly optimized for human-judged helpfulness rather than raw next-token prediction.
Human evaluators preferred outputs from the 1.3B InstructGPT model over the 175B base GPT-3, proving that targeted alignment can outperform brute-force scale on real-world usefulness.
RLHF’s reward model and PPO loop got replaced by a single, elegant classification loss.
Rafailov and colleagues proved that the RLHF reward-maximization objective has a closed-form analytical solution, allowing preference alignment to be reformulated as a single-stage binary classification loss computed directly on preferred/dispreferred response pairs against a frozen reference policy. This eliminated the need to train a separate reward model or run reinforcement learning sampling loops.
DPO matches or exceeds RLHF’s alignment quality while being far more stable and computationally lighter, making it a widely adopted default for open-source model fine-tuning.
Showing the model how to “think out loud” unlocked reasoning that scale alone couldn’t.
Wei and colleagues demonstrated that prompting sufficiently large models (≥100B parameters) with exemplars containing explicit, step-by-step intermediate reasoning dramatically improves performance on arithmetic, commonsense, and symbolic reasoning tasks. No architecture or weight changes were needed — only the prompting strategy.
Chain-of-thought prompting became a standard technique for eliciting multi-step reasoning from LLMs and directly foreshadowed later reasoning-focused training approaches like DeepSeek-R1.
Reasoning emerged on its own once the model was rewarded purely for getting verifiable answers right.
DeepSeek-AI trained DeepSeek-R1-Zero using large-scale reinforcement learning (Grouped Relative Policy Optimization, GRPO) directly on a base model with no supervised fine-tuning step, relying on rule-based, verifiable rewards like compiler feedback and mathematical correctness. The final DeepSeek-R1 added a small “cold-start” dataset of expert rationales plus multi-stage RL to fix readability and language-mixing issues.
The work showed that self-reflection, error correction, and verification can emerge purely from RL incentives, and that these reasoning patterns can be distilled into much smaller models — a major shift away from human-demonstration-heavy training pipelines.
A language model trained on protein sequences learned to predict 3D atomic structure.
Researchers trained a large protein language model on evolutionary sequence data at scale, treating amino acid sequences analogously to natural language tokens, and showed that the model’s internal representations capture enough structural information to predict atomic-level 3D protein folds. This eliminated much of the need for explicit physics-based structural modeling.
The model achieved structure prediction accuracy competitive with specialized folding pipelines while running at far greater scale and speed, with direct applications in computational biology and drug discovery.
A general-purpose LLM was benchmarked against real medical licensing exam standards and held its own.
Google researchers evaluated large language models on a curated benchmark of clinical question-answering datasets, including U.S. medical licensing exam-style questions, assessing not just accuracy but factuality, potential for harm, and bias in generated answers. This represented one of the first rigorous, multi-dimensional evaluations of LLMs in a high-stakes clinical context.
The model demonstrated that LLMs can encode substantial clinical knowledge and generate plausible medical answers, while also highlighting persistent gaps in safety and factual reliability that limit unsupervised clinical deployment.