The AI Glossary That Actually Goes Deep
The field of artificial intelligence has its own language and most glossaries explain it backwards. They define hallucinations before explaining what a language model is. They describe fine-tuning before covering how neural networks learn. This article works the other way: sequentially, from the foundational mechanics to the emergent failures.
The thesis is simple: these twelve concepts build on each other. Understanding them in order is the difference between someone who drops terminology in meetings and someone who can reason about what AI systems actually do and fail at.
Deep Learning: The Foundation Everything Else Runs On
Deep learning is a class of machine learning that uses neural networks with many layers to learn hierarchical representations of data directly from raw inputs, without hand-engineered features.
The name comes from the "depth" of the network: stacking layers of simple computational units (neurons) so that each layer learns progressively more abstract patterns. A network trained on images starts by detecting edges, then textures, then shapes, then objects. At no point does a human encode what an "edge" or a "shape" is. The network figures that out from examples.
This matters because it made general-purpose pattern recognition tractable. Before deep learning, computer vision required enormous amounts of manual feature engineering. A researcher had to specify what features mattered (edge gradients, color histograms, HOG descriptors) before training could begin. Deep learning automated that step.
The cost is interpretability. Because the representations are learned, not specified, no one fully knows what a 70-billion-parameter model is doing in layer 43. That opacity is not a bug that will get patched — it is structural to the approach.
Distillation: Making Large Models Small Without Losing Their Judgment
Distillation is a training technique where a smaller model (the student) is trained to reproduce the output distribution of a larger, more capable model (the teacher), rather than being trained directly on hard labels from a dataset.
The key insight is that a large model's probability outputs carry more information than the final prediction. When a teacher model assigns 0.6 probability to "cat," 0.3 to "dog," and 0.1 to "lynx," it is encoding something about how the concepts relate. Training the student on those soft targets transfers that relational knowledge, not just the answer.
Geoffrey Hinton, Jeffrey Dean, and Oriol Vinyals formalized this in their 2015 paper on knowledge distillation, and it became one of the most practically important techniques in the field. DeepSeek-R1, released in January 2025, used distillation from its larger reasoning model to produce a series of smaller models that punched well above their parameter count.
If you are working with small language models, distillation is the reason a 3B-parameter model trained against a 70B teacher can frequently outperform a 7B model trained from scratch. As explored in How to Train a Small Language Model for Your Niche, the practical mechanics of this process matter more than the theoretical framing.
Generative Adversarial Networks (GANs): The Two-Player Game That Changed Image Generation
A generative adversarial network (GAN), introduced by Ian Goodfellow and colleagues in 2014, is a framework where two neural networks — a generator and a discriminator — are trained simultaneously in opposition.
The generator learns to produce data that fools the discriminator. The discriminator learns to tell real data from generated data. Neither has a fixed target: the generator's goal is to beat the current discriminator; the discriminator's goal is to beat the current generator. The result is a feedback loop where both networks improve, and the generator eventually produces data convincing enough that the discriminator cannot reliably distinguish it from real samples.
GANs dominated image generation research from roughly 2016 to 2021. StyleGAN2, released by NVIDIA in 2020, could generate photorealistic human faces that were indistinguishable from photographs in blind tests. The limitation was control: GANs are notoriously difficult to condition precisely. You could ask for "a face" and get a remarkable one; you could not easily ask for "a 35-year-old woman with red hair in profile view and soft studio lighting" and get exactly that.
Training instability is the structural problem. The adversarial dynamic can collapse in two ways: the generator figures out a small set of outputs that always fools the discriminator (mode collapse), or the discriminator becomes too accurate too early and the generator's gradients vanish. Both are common, and both are solved differently depending on the architecture.
Diffusion Models: The Slow-Denoising Process That Beat GANs at Their Own Game
A diffusion model is a generative model that learns to reverse a gradual noise-addition process. During training, real data is corrupted step by step until it becomes pure noise. The model learns to predict and remove that noise at each step. At inference, the model starts from pure noise and iteratively denoises toward a coherent output.
The crucial difference from GANs: diffusion models do not require an adversary. They are trained with a simple denoising objective — predict the noise added at each step — which is stable and scalable. DALL-E 2 (2022), Stable Diffusion (2022), and Imagen (2022) all used diffusion as their core generation mechanism, and by 2023 they had largely displaced GANs from the frontier of image generation.
The tradeoff is inference speed. Because generation requires many denoising steps (often 20–50 or more), diffusion models are slower than GANs at inference time. A GAN generates an image in a single forward pass. A diffusion model runs the denoiser tens of times. Techniques like DDIM sampling, flow matching, and consistency models have substantially reduced this gap, but it remains a design consideration.
The same denoising principle has since been extended beyond images. Diffusion models now generate audio (Stability AI's Stable Audio), video (Sora, Veo 2), 3D shapes, protein structures, and robot motion trajectories. The mechanism generalizes because noise can be added to any continuous signal.
Understanding how Gemini Omni actually works under the hood illustrates how modern foundation models increasingly combine these generative mechanisms in a single architecture, rather than specializing in one.
Fine-Tuning: Adjusting a Pretrained Model for a Specific Task
Fine-tuning is the process of continuing to train a pretrained model on a smaller, task-specific dataset, adjusting its weights to improve performance on a target distribution without training from scratch.
The economics are straightforward. Pretraining a large language model costs tens of millions of dollars in compute. Fine-tuning the resulting model on a few thousand examples of customer support conversations costs a fraction of that, and the fine-tuned model will outperform the base model on that domain. The pretrained weights encode general knowledge of language and the world; fine-tuning specializes them.
There are two dominant forms:
Full fine-tuning updates all parameters in the model. It is the most expressive but the most expensive, and for large models it typically requires significant GPU memory.
Parameter-efficient fine-tuning (PEFT), of which Low-Rank Adaptation (LoRA) is the most widely used variant, freezes the base model weights and trains only a small set of adapters. LoRA adds low-rank matrices to attention layers; training those matrices alone can achieve results comparable to full fine-tuning at 10–30% of the compute cost.
Fine-tuning is not a cure for gaps in the base model's knowledge. If a model does not know about an event that happened after its training cutoff, fine-tuning on text that mentions the event will not reliably teach it to reason correctly about that event. It adapts style and format more reliably than it installs new factual knowledge.
Hallucination: When a Language Model Confidently Invents Facts
Hallucination, in the context of AI language models, refers to outputs that are factually incorrect, unsupported by the model's training data, or fabricated entirely, presented with the same fluency and apparent confidence as correct information.
The term borrows from psychology but the mechanism is different. A human hallucination involves perceiving something not present in sensory input. A model hallucination involves generating a token sequence that is statistically coherent but semantically false. The model is not "confused" in any mentalistic sense. It is doing exactly what it was trained to do: predict the next likely token. It is just that the likely token is sometimes wrong.
The root cause connects to the throughline stated at the top of this article. Language models are trained to maximize the likelihood of their training data, not to represent a ground truth model of the world. This works well when the training data is dense with correct, consistent information. It fails when the model is asked about low-density regions: obscure facts, recent events it wasn't trained on, specific numbers, or precise citations.
Hallucination rates vary substantially by task. Models are more reliable on tasks that are heavily represented in training data (English prose, common programming languages, well-documented topics). They are less reliable on tasks where training data is sparse or inconsistent (minority languages, niche technical domains, recent events).
This is not a temporary limitation that will be fixed with the next model generation. It is a structural consequence of how generative models are trained. Retrieval-augmented generation (RAG), chain-of-thought prompting, and verification pipelines are mitigation strategies, not solutions. The problem of building systems that separate "things the model genuinely knows" from "things the model will confidently generate" remains open.
The trust failure mode is the same one that makes prompt injection worse than SQL injection: downstream systems, users, and other models cannot reliably distinguish a model's confident correct output from a confident incorrect one.
Chain of Thought: Making Reasoning Visible
Chain of thought (CoT) prompting is a technique, first documented by Wei et al. at Google Brain in a 2022 paper, where a language model is prompted to produce intermediate reasoning steps before arriving at a final answer, rather than jumping directly from input to output.
The empirical result was striking: on math and reasoning benchmarks, large models prompted with CoT outperformed much larger models prompted without it. A 540-billion-parameter model with CoT outperformed a 175-billion-parameter model answering directly on a suite of multi-step math problems.
The intuition is that for tasks requiring multiple logical steps, generating the steps in sequence lets each step condition the next. Without CoT, the model compresses all of that reasoning into a single output generation, which exceeds what its feedforward network can reliably handle in one pass.
What CoT is not: a guarantee of correct reasoning. Models prompted with chain of thought can produce fluent, internally consistent reasoning chains that lead to wrong conclusions. The steps look like reasoning without being verified as correct. This is a known failure mode, and it is why verification-based approaches (where a separate model or process checks the steps) have become part of production reasoning pipelines.
Modern reasoning models — o1, o3, DeepSeek-R1, Claude 3.7 Sonnet's extended thinking — can be understood as fine-tuning CoT at scale: training models to produce long internal reasoning traces before answering, with the training objective shaped to reward correct final answers. The internal trace is CoT scaled up and made part of the model's learned behavior rather than a prompting trick.
AI Agents: Putting a Language Model in a Loop with Tools
An AI agent is a system that uses a language model as its reasoning core, connected to a set of tools or actions, to complete multi-step tasks by iteratively deciding what to do next based on observations from the environment.
The definition is intentionally mechanical. "Agent" has accumulated a great deal of marketing noise. The structural definition cuts through it: a language model that can call tools, observe results, and decide on next steps, in a loop, until a task is complete or a stopping condition is met.
The components are:
- A model: does the reasoning, planning, and decision-making.
- Tools: functions the model can call — web search, code execution, API calls, file reads, database queries.
- Memory: context passed between steps. This is the agent's working memory for a session; it does not persist without external storage.
- An environment: where the agent's actions have effects.
Agents are not autonomous in any philosophically meaningful sense. They are a language model in a loop. The model does not have goals. It generates the next action token by token, conditioned on a context window that includes the task description, previous steps, and tool results. What looks like planning is token prediction at each step.
The practical implication: an agent that produces three plausible-looking steps and then fails on step four does not "know" it failed unless the tool call returns an error that lands in context. Self-correction requires that failures are observable and the model has been trained or prompted to recognize and handle them.
The Agent Loop: The Execution Cycle That Drives Autonomous Tasks
The agent loop is the core runtime pattern used to execute agentic tasks: a repeating cycle of observe, think, act, repeat.
In the most common formulation:
- Observe: The agent receives the current state — the task, any previous messages, and the results of prior tool calls.
- Think: The model generates a response, which may include reasoning and a tool call.
- Act: If the response includes a tool call, the runtime executes it and returns the result.
- Repeat: The tool result is appended to context and the model reasons again.
- Stop: The loop terminates when the model outputs a final answer (no tool call) or a maximum step limit is reached.
The agent loop is where most production failures occur. Context window limits constrain how many steps an agent can reason across before earlier observations are truncated or summarized. Compounding errors — where an early wrong decision leads the agent further into a wrong subtask — are common and hard to detect without human-in-the-loop checkpoints. Tool call latency accumulates across steps, making long-horizon agents slow in ways that are non-obvious from prototyping.
Multi-agent systems, like the Sakana Fugu orchestration system built around Japan's sovereign AI strategy, run multiple agent loops in parallel or in hierarchies, with one orchestrator agent delegating to specialist subagents. This distributes the problem but multiplies the failure modes: coordination errors, inconsistent state, and cross-agent hallucinations become new risks.
Artificial General Intelligence (AGI): The Term Nobody Defines Consistently
Artificial General Intelligence (AGI) refers, in most usages, to an AI system capable of performing any cognitive task that a human can perform, at human level or above, across domains, without task-specific training for each new domain.
The problem with AGI as a concept is that the definition has drifted. OpenAI's internal definition, as reported by The Verge in 2024, ties AGI to a system that can generate $100 billion in profit autonomously. DeepMind uses a multi-level taxonomy, where AGI occupies the highest rung of a five-level scale. These are not the same thing.
For practical purposes, what matters is the distance between current systems and the AGI definition you are using:
Current frontier models (GPT-4o, Claude Sonnet 4.5, Gemini 2.5 Pro) are genuinely capable across a wide range of language tasks. They fail at consistent multi-step physical reasoning, sustained memory across sessions, reliable causal inference, and learning from new experiences without retraining. Whether those gaps put them "before AGI" depends entirely on which definition you are using.
The term matters because it shapes regulatory framing, safety research priorities, and public expectations. It does not have a scientific consensus definition, and any article — including this one — that uses it without specifying which definition should be read with that caveat in mind.
The Honest Take
Three things this article simplifies in ways that a builder should know about.
First, these concepts do not evolve in isolation. Diffusion models borrowed the denoising objective from score-based models. Chain of thought emerged partly from observations about in-context learning, itself a behavior that wasn't explicitly trained. The clean sequential presentation here is pedagogically useful and historically inaccurate. Real research is messier.
Second, the hallucination section treats hallucination as a failure of factual grounding. That is the most common framing, and it is useful. But there is an active debate about whether some hallucinations are better described as failures of calibration (the model is confident when it shouldn't be), failures of instruction following (the model misunderstood what was asked), or failures of retrieval (the model had the right knowledge but retrieved the wrong pattern). These distinctions matter for which mitigation you reach for first.
Third, the agent loop description treats memory as a solved problem. It isn't. Most production agents use a combination of in-context memory (what's in the context window), external retrieval (RAG over a database), and summarization to compress long conversations. Each has error modes that compound in multi-step tasks. Agent memory is one of the most actively researched and unsolved problems in the field as of July 2026.
The Bottom Line
These twelve concepts are not isolated vocabulary items. They form a dependency stack: you cannot understand fine-tuning without understanding what pretraining establishes; you cannot understand hallucination without knowing what the training objective actually optimizes for; you cannot understand AGI debates without knowing what current agents can and cannot do in a loop.
The through-line is that optimization for a training objective is not the same as optimization for truth or correctness. That gap — between what the model was trained to produce and what would actually be right — is where nearly every failure mode on this list originates.
Frequently Asked Questions
What is the difference between deep learning and machine learning?
Machine learning is the broader category: any system that improves performance on a task through data, including decision trees, linear regression, and support vector machines. Deep learning is a subset that uses multi-layer neural networks. All deep learning is machine learning; most machine learning is not deep learning.
Is fine-tuning the same as training?
Fine-tuning is a type of training, specifically the continuation of training on a smaller task-specific dataset after pretraining. The distinction matters because the starting point is different: a fine-tuned model begins from pretrained weights, not random initialization. Starting from a pretrained model converges faster and with less data.
Do diffusion models always produce images?
No. Diffusion models produce samples from whatever data distribution they were trained on. They have been applied to images, audio, video, protein structures, molecular conformations, robot trajectories, and text (though text diffusion is less established). The underlying mechanism — learn to reverse a noising process — is domain-agnostic.
Are GANs obsolete?
For general-purpose image generation at the frontier, largely yes — diffusion models have displaced them. GANs remain competitive in specific high-speed, low-latency applications where a single-forward-pass generator is a practical requirement. Several real-time video synthesis systems still use GAN-based architectures.
What is the difference between an AI agent and a chatbot?
A chatbot responds to inputs and produces outputs without taking actions in the world. An agent can call tools, execute code, browse the web, read and write files — and does so iteratively across multiple steps toward a goal. The agentic distinction is about action and iteration, not just conversation.
Is chain of thought the same as extended thinking in Claude or o-series models?
CoT prompting is a technique applied at inference time. Extended thinking and o-series reasoning are models fine-tuned to produce long internal reasoning traces as a learned behavior. They produce CoT-style outputs, but the mechanism is baked into the model weights, not prompted in. The distinction matters for reproducibility and reliability.
Can hallucinations be fully eliminated?
No, with current architectures. Hallucinations can be substantially reduced through retrieval-augmented generation, constitutional AI training, and verification pipelines — but not eliminated. The structural reason is that language models generate text by predicting likely token sequences, not by querying a verified knowledge base. Fundamentally different architectures would be required to eliminate rather than mitigate the failure.
What does AGI mean for AI safety research?
The AGI framing shapes safety research priorities significantly. Teams that believe AGI is near prioritize alignment research (how to ensure a very capable system acts in accordance with human values). Teams that believe AGI is far prioritize near-term harms from current systems (bias, misuse, hallucination at scale). Both research programs are running simultaneously, with different funding bases and different urgency framings.