nomenclates – draft mode!

An AI agent typically wraps a language model in a loop: the model receives a goal and the current state (including results from its own prior actions), decides on a next action from a set of available tools, executes it, observes the outcome, and repeats until it judges the goal met or it hits a limit. This differs from a single-turn LLM call mainly in that control flow — which action happens next — is decided by the model at runtime rather than fixed in advance by a developer.

Design points that vary a lot between agent systems: how much autonomy the agent has before it must check in with a human (a spectrum from "propose and wait for approval" to "act and report afterward"), how it maintains memory across steps (context window vs. external storage it can read and write), what tools it's given access to and how those are sandboxed, and how failure is handled when an action doesn't produce the expected result. Because agents chain multiple model calls and tool executions, errors can compound step over step in a way a single response can't — a wrong intermediate action can send the whole run down the wrong path, which is why guardrails, action logging, and human checkpoints matter more here than in single-turn use.

AI is a field, not one technique. It spans symbolic approaches (explicit rules and logic, dominant through the 1980s), statistical machine learning, and today's connectionist methods — neural networks trained on data, particularly deep learning. The term was coined for a 1956 Dartmouth workshop proposal by McCarthy, Minsky, Rochester, and Shannon, who bet that "every aspect of learning... can in principle be so precisely described that a machine can be made to simulate it."

Useful distinctions: narrow AI (a system good at one task, which is everything that exists today) versus artificial general intelligence (a system with human-like flexibility across tasks, which doesn't); and AI as an engineering goal versus AI as a research field studying intelligence itself. Current state-of-the-art systems are almost entirely machine-learning based, trained on large datasets rather than programmed with hand-written rules.

The mechanism computes attention as a weighted sum over "value" vectors, where the weights come from comparing a "query" vector against a set of "key" vectors — commonly via a scaled dot product, softmax-normalised so the weights sum to 1. In self-attention, queries, keys, and values are all derived from the same input sequence (via separate learned projection matrices), which is what lets a transformer relate any two positions in a sequence regardless of how far apart they are, unlike RNNs, where information from distant positions has to survive being passed step-by-step through a fixed-size hidden state.

Multi-head attention runs several attention computations in parallel with independently learned projections, then concatenates the results. Each head can specialise — some empirically track syntactic relationships, others track coreference (like the "it" example above) — and the model combines their outputs rather than relying on one fixed notion of relevance.

Attention predates the transformer: Bahdanau et al. (2014) introduced it for RNN-based machine translation, letting a decoder look back at encoder states instead of compressing the whole source sentence into one fixed vector. Vaswani et al.'s 2017 "Attention Is All You Need" showed attention alone, without any recurrence, was sufficient — and better, given enough data and compute. The main cost: standard self-attention is O(n²) in sequence length, since every token compares against every other token, which is the central scaling bottleneck for long-context models.

Backpropagation computes the gradient of a loss function with respect to every weight in a network by applying the chain rule of calculus layer by layer, from the output back to the input. For a layer computing z = Wx + b followed by activation a = f(z), the gradient of the loss L with respect to W is:

∂L/∂W = (∂L/∂a · f'(z)) xᵀ

and the term in parentheses — the "error signal" — propagates to the previous layer by multiplying through Wᵀ. Each layer reuses the error signal computed by the layer after it, which is what makes the algorithm efficient: computing gradients for all parameters costs roughly the same as one forward pass, not one pass per parameter.

The method was described for neural networks by Rumelhart, Hinton, and Williams in 1986, though the underlying reverse-mode automatic differentiation traces back further, to Linnainmaa's 1970 thesis and Werbos's 1974 dissertation. Backpropagation itself doesn't decide how to use the gradients — that's the job of an optimiser such as gradient descent or Adam, which apply the computed gradients to update weights.

Two well-known failure modes: vanishing gradients, where the chain-rule product shrinks toward zero across many layers (common with sigmoid/tanh activations, mitigated by ReLU, residual connections, and normalisation layers), and exploding gradients, where it grows unboundedly (mitigated by gradient clipping).

A CNN layer applies learned filters (kernels), typically small (3×3 or 5×5), via convolution across the spatial dimensions of the input, producing a feature map per filter. Because a filter's weights are shared across every spatial location, the layer is translation-equivariant — a detected pattern is recognised the same way regardless of where in the image it appears — and needs vastly fewer parameters than a fully connected layer over the same input.

Standard architecture stacks: convolution → nonlinearity (typically ReLU) → pooling (max or average, downsampling the spatial resolution) → repeat, with the number of feature channels typically increasing as spatial resolution shrinks. Deep stacks build a receptive field that grows with depth, so late layers respond to large, complex patterns built from the simple edge/texture detectors learned in early layers — a hierarchy that generally isn't hand-designed, it emerges from training.

CNNs became the dominant approach in computer vision after AlexNet's 2012 ImageNet win, building on earlier work (LeCun's LeNet, 1989/1998) applied to digit recognition. Residual connections (ResNet, 2015) let much deeper stacks train reliably by giving gradients a shortcut path around convolutional blocks. Since roughly 2020, vision transformers have challenged CNNs' dominance on large-data image tasks, though CNNs remain competitive and considerably cheaper on smaller datasets, thanks to the inductive bias baked into the convolution operation itself.

A deep network is a composition of many parametrised, differentiable layers, trained end-to-end by backpropagation: compute the error at the output, and propagate gradients backward through every layer to update its weights. Depth matters because each layer can build more abstract representations on top of the last — this is representation learning, distinguished from classical ML, where a human engineers the input features by hand.

Architecture follows the structure of the data. Convolutional networks (CNNs) exploit spatial locality for images. Recurrent networks (RNNs) and their successors process sequences by carrying state across steps. The transformer architecture (2017), which processes an entire sequence at once using attention instead of stepping through it, now dominates both language and, increasingly, vision. Deep learning's practical rise tracks three enablers arriving together around 2012: large labelled datasets, GPU-accelerated matrix computation, and refinements (ReLU activations, dropout, batch normalisation) that made very deep networks trainable without their gradients vanishing or exploding.

An embedding maps a discrete or high-dimensional input to a dense vector in a continuous space of much lower dimension, typically anywhere from tens to a few thousand dimensions. The mapping is learned, not designed: a model adjusts the vectors during training so that geometric relationships in the space (distance, direction) mirror relationships that matter for the task — semantic similarity, co-occurrence, or whatever the loss function rewards.

Word2vec (Mikolov et al., 2013) is the canonical early example: it learns word vectors by training a shallow network to predict a word from its surrounding context (or vice versa), and the resulting vectors capture analogies as vector arithmetic — the classic result being king − man + woman ≈ queen. Later embedding methods extended the same idea to subword units (fastText), full sentences and documents, and — in modern LLMs — every token gets an input embedding vector, and internal layers effectively refine that embedding into successively more contextualised representations as it passes through the network.

Beyond language, embeddings are how retrieval-augmented generation (RAG) systems work: documents get embedded once and stored in a vector database, a query gets embedded at search time, and the system retrieves the documents whose vectors are closest by some distance metric (cosine similarity is the usual choice). Embedding quality is measured by how well distance in the vector space tracks actual similarity — a poorly trained embedding space can place unrelated items close together, which silently degrades every downstream system built on it.

Fine-tuning continues gradient-descent training on a pretrained model's weights, using a smaller task-specific dataset instead of the original training corpus. Full fine-tuning updates every weight; parameter-efficient methods like LoRA and adapters freeze most of the network and train a small set of added parameters instead, cutting compute and storage cost with little accuracy loss.

Push it too hard and the model forgets general skills it had before — catastrophic forgetting — so fine-tuning runs use a low learning rate and often just a few passes over the data. Instruction tuning and RLHF are both fine-tuning: the first on (instruction, response) pairs, the second on a reward signal derived from human preferences, typically applied to a model that's already been instruction-tuned.

A GAN consists of a generator G, which maps noise z (usually sampled from a simple distribution) to synthetic data G(z), and a discriminator D, which outputs the probability that a given sample is real rather than generated. Training alternates updates to each network against a minimax objective:

min_G max_D  E[log D(x)] + E[log(1 − D(G(z)))]

D is trained to maximise this — correctly classifying real data as real and generated data as fake — while G is trained to minimise it, i.e., to make D(G(z)) as close to 1 as possible. At the theoretical optimum, G recovers the true data distribution and D outputs ½ everywhere, unable to distinguish real from fake at all.

In practice, this adversarial setup is notoriously hard to train stably. Mode collapse — where G learns to produce only a narrow subset of plausible outputs, sacrificing diversity for looking convincing — and vanishing gradients when D becomes too confident too early are the two classic failure modes. Architectural and objective refinements were developed specifically to address these: DCGAN (2015) established stable convolutional architectures; Wasserstein GAN (2017) replaced the original objective with the Earth Mover's distance for smoother, more informative gradients; StyleGAN (2019) introduced style-based generation for fine-grained control over generated image attributes.

Originally proposed by Goodfellow et al. in 2014, GANs were, for several years, the dominant approach to high-fidelity image synthesis. Since roughly 2021, diffusion models have overtaken them on many benchmarks for sample quality and training stability, though GANs retain an advantage in inference speed — a single forward pass through G, versus the many denoising steps a diffusion model needs.

Gradient descent minimises a differentiable loss function by repeatedly moving parameters in the direction of steepest decrease — the negative of the gradient. The update rule is:

θ ← θ − η∇L(θ)

where θ is the parameter vector, L is the loss, and η is the learning rate. Learning rate governs the trade-off directly: too high and the optimiser overshoots or diverges; too low and training crawls, or stalls in a shallow local minimum or saddle point.

Computing the true gradient over an entire training set is expensive, so practical training almost always uses stochastic gradient descent (SGD) — estimating the gradient from a random mini-batch — or a variant. Momentum accumulates a running average of past gradients to dampen oscillation across ravines in the loss surface. Adam and its relatives adapt the learning rate per parameter using running estimates of the gradient's mean and variance, which is why they're the default for training large neural networks despite lacking SGD's convergence guarantees in the convex case.

Because loss surfaces for deep networks are non-convex, gradient descent finds a local minimum, not necessarily the global one — but empirically, for over-parameterised networks, most local minima found this way perform comparably well.

Language models generate text by predicting likely next tokens, not by checking claims against a source of truth. Hallucination is a byproduct of that: the model has no built-in mechanism to distinguish a well-formed false statement from a well-formed true one, so fluency and factual accuracy come apart. Surveys of the problem typically split causes into data issues (training corpora contain errors, contradictions, or gaps), training objective mismatch (next-token prediction rewards plausible text, not verified text), and inference-time factors (sampling temperature, prompts that push the model outside its reliable knowledge).

Common mitigations include retrieval-augmented generation (grounding answers in retrieved source documents rather than parametric memory), lower-temperature or constrained decoding, fine-tuning on factuality-labelled data, and having the model cite sources it can be checked against. None of these eliminate hallucination outright — RAG reduces it but still fails when retrieval misses the right passage or the model ignores the retrieved context in favor of its own priors. Evaluation is its own open problem: benchmarks like TruthfulQA and hallucination-rate leaderboards measure it approximately, and there's no universal ground truth for "did the model make this up."

Inference is the forward pass through a trained model: input goes in, parameters (fixed at this point — no gradient computation, no weight updates) are applied, output comes out. For an LLM specifically, inference is autoregressive — the model predicts one token, appends it to the sequence, and repeats, which is why longer outputs take proportionally longer to generate and why generation speed is usually measured in tokens per second.

Because inference runs far more often than training, a whole layer of engineering exists just to make it cheap and fast: quantisation (running the model at lower numerical precision — int8, int4 — to cut memory and compute at some accuracy cost), KV-caching (reusing previously computed attention keys/values instead of recomputing them for every new token), batching multiple requests together to use hardware more efficiently, and dedicated inference hardware or runtimes (TensorRT, vLLM, and similar) tuned for this workload specifically rather than for training.

Two deployment patterns cover most use cases: batch inference, where a large set of inputs gets processed together, offline, with no one waiting on an individual response (e.g. overnight scoring of a dataset); and online/real-time inference, where a single request needs a response within a latency budget of milliseconds to a few seconds (e.g. a chat reply or a fraud-detection check on a live transaction). The two impose different tradeoffs — batch favors throughput, online favors latency — and production systems are usually built around one or the other, not both.

LLMs are typically transformer-based neural networks trained on next-token prediction over web-scale text corpora, with parameter counts ranging from low billions to over a trillion. Capabilities like in-context learning and instruction-following emerge from scale and fine-tuning (e.g. RLHF) rather than being explicitly programmed.

ML is the subfield of AI concerned with algorithms that fit a model to data and generalise to new, unseen inputs. Three broad regimes: supervised learning (train on labelled examples — inputs paired with correct outputs), unsupervised learning (find structure in unlabeled data, e.g. clustering), and reinforcement learning (learn from reward signals through trial and error).

The central technical problem is generalisation, not memorisation. A model that perfectly reproduces its training data but fails on new data has overfit — it memorised noise instead of learning the underlying pattern. Guarding against this drives most of the field's methodology: held-out test sets, regularisation, cross-validation. Classical algorithms (linear regression, decision trees, support vector machines) still see wide use; deep learning is the subset built on multilayer neural networks, and has become the default approach for problems with enough data — images, text, audio — since roughly the early 2010s.

Classic NLP breaks into tasks: tokenisation (splitting text into words or subwords), part-of-speech tagging, parsing (recovering grammatical structure), named-entity recognition, machine translation, sentiment analysis, and question answering. The field's dominant approach has shifted twice: hand-written rules and grammars through the 1980s, then statistical methods trained on labelled corpora through the 2000s, then neural methods — first RNNs, now near-universally transformer-based models — from the 2010s onward.

Language resists the clean assumptions that make other ML problems tractable: the same sentence can be ambiguous ("I saw her duck"), meaning depends on context outside the sentence itself, and grammar varies by language in ways that don't transfer. Modern large language models blur NLP's traditional task boundaries — a single model now handles translation, summarisation, and question answering without task-specific architectures — but they still inherit the field's older problems: bias in training text, factual unreliability, and uneven performance across languages with less available data.

The core building block, dating to McCulloch and Pitts's 1943 model, is a unit that takes a weighted sum of its inputs and passes the result through a nonlinear activation function (historically a sigmoid; today usually ReLU or a variant). Units are arranged in layers — an input layer, one or more hidden layers, an output layer — with weighted connections between them. Stack enough layers and you have deep learning; the boundary is informal, not a fixed layer count.

Training means adjusting every weight in the network so its output error shrinks, done via backpropagation (computing the gradient of the error with respect to each weight using the chain rule) combined with an optimiser like stochastic gradient descent. The "neural" framing is a loose analogy, not an engineering claim: artificial units are far simpler than biological neurons, and backpropagation has no established biological counterpart. Modern architectures — convolutional, recurrent, transformer — differ mainly in how connections are structured, not in this basic unit.

Overfitting occurs when a model's capacity is high relative to the signal available in the training data, letting it fit noise rather than the underlying distribution. The tell is a growing gap between training loss (keeps falling) and validation loss (falls, then rises) as training continues.

Common mitigations:

• Regularisation — L1/L2 penalties on weights, dropout, or early stopping, all of which constrain how tightly the model can fit the training set.
• More or better data — augmentation, or simply collecting more examples, to make memorisation a worse strategy than generalising.
• Reduced capacity — fewer parameters, fewer layers, or heavier weight sharing (as in convolutional networks), so there's less room to memorise.
• Cross-validation — holding out data specifically to detect the train/validation gap before it shows up in production.

Overfitting sits opposite underfitting, where the model is too simple to capture even the real pattern, on a curve usually described as the bias-variance tradeoff: high-capacity models have low bias but high variance (prone to overfitting), low-capacity models have the reverse.

One counterintuitive finding from deep learning: very large networks can be pushed well past the point of zero training error and still generalise well — the "double descent" phenomenon — which complicates the classical picture where capacity beyond the data's information content is assumed to hurt.

A prompt sets the context a model conditions its output on, so changes to wording, structure, and ordering shift the probability distribution over possible responses — sometimes by a lot, even when the underlying meaning looks unchanged to a human reader. Established techniques include few-shot prompting (showing example input/output pairs), chain-of-thought prompting (asking the model to reason step by step before giving a final answer), role or system prompts (framing the model's persona or constraints), and structured formats like XML tags or JSON schemas to make outputs easier to parse.

Prompt engineering is not a substitute for defining what "correct" means for your task. Effective practice starts with success criteria and a way to test against them, then treats the prompt as one lever among several — alongside model choice, retrieval, and fine-tuning — for hitting those criteria. Prompts also don't transfer perfectly across models: wording tuned for one model's training and instruction-following behavior can underperform on another, which is why vendors publish model-specific prompting guides.

RL formalises the problem as a Markov decision process: an agent observes a state, takes an action, receives a reward, and moves to a new state, repeating the loop. The agent's policy — its rule for choosing actions — gets updated to increase expected cumulative reward, often via a learned value function that estimates how good a state or action is.

Unlike supervised learning, there's no labelled "correct action" for each state — only a reward signal, which can be sparse and delayed, arriving many steps after the action that caused it. That creates the credit-assignment problem (which past action deserves the reward?) and the exploration-exploitation tradeoff (try something new, or stick with what already works?). Algorithms like Q-learning, policy gradients, and PPO tackle these in different ways. Notable applications: AlphaGo, robotic control, and — via RLHF — steering language models toward outputs people prefer.

RLHF runs in two stages after a base model has been pretrained and usually instruction-tuned. First, human labelers rank several model outputs for the same prompt, and that comparison data trains a separate reward model to predict which output a person would prefer. Second, the reward model scores the main model's outputs, and a reinforcement learning algorithm — PPO in OpenAI's InstructGPT paper — updates the model's weights to raise its expected reward.

A KL-divergence penalty keeps the updated policy close to the original model, so it doesn't just learn to game the reward model with outputs that score well but read like nonsense — reward hacking. RLHF is the technique behind ChatGPT's and InstructGPT's shift from "predict plausible text" to "produce what a human rater would rate as helpful and honest." Newer methods like DPO (Direct Preference Optimization) skip the separate reward model and RL loop entirely, training directly on the preference pairs — a simpler, cheaper alternative gaining ground on RLHF.

A RAG system has two main parts: a retriever and a generator. The retriever encodes a query and a document corpus into vector embeddings and finds the passages most similar to the query (typically via nearest-neighbor search over a vector index). Those passages get concatenated into the generator's prompt, and the language model conditions its output on both the query and the retrieved text.

RAG was introduced by Lewis et al. (2020) as a way to combine a pre-trained parametric model (the LLM's weights) with a non-parametric memory (a retrievable document index), giving the system access to knowledge that can be updated by swapping the index rather than retraining the model. This matters for two practical problems: reducing hallucination by grounding answers in retrieved source text, and answering questions about information outside the training data — recent events, private documents, or domain-specific corpora — without fine-tuning.

Retrieval quality bounds generation quality: if the retriever misses the relevant passage or ranks it low, the generator never sees it, and no amount of prompting fixes that. Production systems tend to invest in chunking strategy, embedding model choice, hybrid search (combining vector similarity with keyword search), and re-ranking on top of the base retrieve-then-generate loop.

A supervised model learns a function mapping inputs to outputs from a labelled dataset of (x, y) pairs. Training minimises a loss function that measures the gap between the model's predictions and the true labels — cross-entropy for classification, mean squared error for regression — by adjusting model parameters through gradient descent.

The label is what makes it "supervised": every training example carries the correct answer, so the model gets direct, per-example feedback. That's also the method's main cost — labelled data is expensive to collect — which is why unsupervised and self-supervised pretraining often come first, with a smaller supervised fine-tuning stage on top.

Modern LLMs use subword tokenisation rather than whole-word or character-level schemes, because whole-word vocabularies can't handle typos, rare words, or new terms (unbounded vocabulary size), while character-level tokenisation produces very long sequences for a given amount of text. Subword methods split the difference: common words stay as single tokens, rare words get broken into smaller reusable pieces, and any string can be represented from a fixed, finite vocabulary.

The dominant algorithm is byte-pair encoding (BPE) and its variants (WordPiece, used by BERT; SentencePiece/Unigram, used by many multilingual models). BPE starts from individual characters (or bytes) and iteratively merges the most frequent adjacent pair into a new symbol, repeating until the vocabulary reaches a target size — typically 32K to 100K+ tokens. Byte-level BPE (used by GPT-2 and later OpenAI models) operates on raw UTF-8 bytes instead of Unicode characters, which guarantees any input, including emoji and unseen scripts, can be tokenised without an out-of-vocabulary fallback.

Tokenisation choices have real downstream effects: languages poorly represented in the training corpus used to build the tokeniser end up needing more tokens per word than English does, which inflates their effective cost and context usage. Tokenisers also explain some well-known LLM failure modes — a model asked to reverse a word or count its letters often struggles because it never sees individual characters, only the multi-character tokens they were merged into.

The transformer was introduced in "Attention Is All You Need" (Vaswani et al., 2017) as a sequence-to-sequence architecture built entirely from attention mechanisms, dropping the recurrence (RNNs/LSTMs) and convolution that previous sequence models relied on. Removing recurrence means every position in a sequence can be processed in parallel during training, which is what made it practical to train on the scale of data and compute that produced modern LLMs.

The original architecture has an encoder stack and a decoder stack, each built from repeated layers combining multi-head self-attention with a position-wise feed-forward network, residual connections, and layer normalisation. Since input tokens are processed in parallel with no inherent notion of order, positional information gets injected separately — originally via fixed sinusoidal position embeddings, later via learned or relative position encodings (e.g. RoPE).

Three architectural variants dominate today: encoder-only (BERT and similar, good for classification and embeddings), decoder-only (GPT, Claude, Llama — autoregressive, generate one token at a time, current default for general-purpose LLMs), and encoder-decoder (T5, the original translation-focused design, still used for some translation and summarisation tasks). Self-attention's compute and memory cost scales quadratically with sequence length, which is the main reason long-context models need architectural tricks — sparse attention, sliding windows, KV-cache optimisations — to stay efficient.

Unsupervised methods find structure in unlabeled data without a target output to optimise against. Common tasks: clustering (k-means, hierarchical clustering — group similar points), dimensionality reduction (PCA, t-SNE — compress high-dimensional data while preserving structure), and density estimation (model the distribution the data was drawn from).

Because there's no label to check predictions against, "correctness" is defined by the algorithm's own objective — minimising within-cluster distance, maximising reconstructed variance — rather than by matching a ground truth. Self-supervised learning sits adjacent to this: it generates its own labels from the data (predict the next word, fill in a masked patch), giving supervised-style training signal without human-labelled examples.