A practical learning resource

Inference engineeringStart with the system. Understand every token.

One complete foundation module, a map of 13 topics, and interactive labs. Modules 2–13 are a roadmap and are not published lessons.

Start Module 1Sources & assumptionsFree · no sign-up

Scope: dense, decoder-only Transformers unless stated otherwise. GB = 10⁹ bytes; GiB = 2³⁰ bytes. FLOPs count work; FLOP/s measures a rate. Hardware settings are illustrative.

FOUNDATION1 module available · 12 topics on the roadmap

Master map

Select a topic to see its prerequisites and what it unlocks. ● Published lesson · ○ Roadmap only.
13 layers · 1 system
LAYER 1Foundation

Foundations

Prerequisites
Core question
Unlocks
Algorithm
Tensor ops
Kernel
Hardware
Memory
Runtime
Serving
User latency

Module 1 · Hardware & performance foundations

Build the resource model behind every inference decision.
LESSON 1 OF 7
PrerequisitesPython, tensors, basic Linux
You will learnHow compute, memory, and movement set latency
UnlocksTransformer execution, prefill, and decode
SYSTEM TOUR

Inference is a resource-conversion system

10–15 min

Mental model: an inference server turns request bytes into tensor operations, moves those tensors through a memory hierarchy, and spends finite compute and bandwidth to emit tokens before a deadline.

The inference engineer’s first question is not “Which optimization is popular?” It is “Which finite resource is limiting this workload now?” Select each stage below and identify its resource, metric, and failure mode.

STAGE 1

Request

CPU · network
Why it exists
Watch
Typical failure

A continuous-batching sketch

Prefill processes prompt tokens and supplies logits for the first output token. Standard decode then advances each active sequence one token at a time.
Prefill Decode Tick 1: Request A enters prefill.
Prefill projection work—≈ 2 × parameters × prompt tokens × batch
One decode step: projection work—≈ 2 × parameters × active sequences
Weight memory—parameters × bytes
Work estimates count dense weight projections only. They omit attention score/value products, normalization, sampling, and serving overhead; they cannot predict TTFT or TPOT. Timeline ticks show scheduling order, not equal elapsed durations. Speculative decoding can emit multiple tokens per iteration.
COMPUTE

From matrix multiplication to FLOPs

20–25 min

Why it exists: Transformer layers are dominated by matrix multiplications. Counting their operations gives a first estimate of work; counting their bytes reveals whether hardware can stay fed.

Intuition: each output element is a dot product. Multiplying an M×K matrix by a K×N matrix produces M×N outputs, each requiring K multiplies and K−1 additions. Counting a multiply-add as two operations gives approximately 2MKN FLOPs.

A · 16×4096activations
×
W · 4096×11008weights
=
C · 16×11008output
Work—2 × M × K × N
Minimum tensor bytes—A + W + C once, same element size
Arithmetic intensity—FLOPs per byte

Ideal traffic at one memory boundary: each input is read once and output written once. Accumulators may use wider precision; tiling, caches, rereads, and quantization metadata change traffic. Integer operations are not floating-point FLOPs.

# One linear projection: [tokens, hidden] @ [hidden, out]
y = x @ weight
# Approximate work
flops = 2 * tokens * hidden * out
UnderneathThe runtime chooses a GEMM kernel, tiles operands, and maps tiles to thread blocks and Tensor Cores.
Performance implicationLarger M reuses weights across more tokens, increasing arithmetic intensity.
Common mistakeHigh theoretical FLOPs do not help when the kernel waits for bytes or uses an unsupported shape.
GPU EXECUTION

How a tensor operation reaches the GPU

25–30 min

Mental model: a GPU is a factory of many streaming multiprocessors (SMs). On NVIDIA GPUs, kernels schedule thread blocks onto SMs; warps of 32 threads issue instructions. Other vendors use different execution terminology.

Python opRuntime dispatchKernel launchThread blocksWarps on SMsCUDA / Tensor Core math
PER THREAD

Registers

fastest

Hold live values for each thread. Register pressure can reduce occupancy or spill data into much slower memory.

Capacitytiny
Inference useaccumulators, fragments
Failure signalspills, low occupancy
Threads → warpsThreads execute in groups. Branch divergence makes a warp serialize different paths.
Blocks → SMsIn the usual CUDA execution model, a block runs on one SM. Registers and shared memory limit how many blocks can reside together.
Tensor CoresSpecialized matrix units deliver high low-precision throughput when shapes and layouts match supported paths.
More occupancy≠automatically faster·enough active warps must hide latency without starving each warp of registers
MEMORY & PRECISION

Build a model-fit ledger

20–25 min

Why it exists: a model “fits” only after weights, KV cache, activations, temporary buffers, graphs, allocator fragmentation, and runtime reservations all fit at the same time.

First-principles rule: weight bytes = parameters × bytes per parameter. BF16 stores each weight in 2 bytes, so a 7B model needs about 14 GB (13.0 GiB) before overhead. Quantized storage also needs scales, sometimes zero-points, and possibly unquantized tensors.

Weight memory—parameters × bytes
Ideal GPU-count lower bound—equal weight shards + per-GPU reserve
KV / workload room—after weights + reserve

This count assumes perfectly divisible weights and a reserve on every GPU. It is not a deployable TP configuration: head divisibility, replicated tensors, KV, activations, and communication buffers still need checking.

14 GB weights + 8 GB runtime = 22 GB used
PrecisionLower-bit formats reduce capacity and bandwidth pressure, but need supported kernels and acceptable quality.
FragmentationReserved memory can exceed live tensors; inspect allocated versus reserved memory before declaring a leak.
Production implicationUnused memory is not waste when it is the safety margin for long prompts, bursts, and temporary buffers.
DATA MOVEMENT

PCIe, NUMA, and network boundaries

15–20 min

Mental model: every boundary is a toll road. HBM feeds the GPU, PCIe connects host and device, an intra-node fabric connects GPUs, and the network connects nodes. The slowest unavoidable transfer can dominate the whole request.

Ideal transfer time—size ÷ bandwidth + latency
Effective throughput—including fixed latency
4 serial transfers (example)—no overlap; not a universal layer count
Remote clientNICCPU memoryPCIe rootGPU HBMSM / Tensor Core

Rates are illustrative sustained, one-direction bandwidths. 100 Gb/s = 12.5 GB/s before protocol overhead. Aggregate or bidirectional figures are not point-to-point payload bandwidth.

NUMAA CPU core accessing memory attached to another socket pays extra latency and consumes inter-socket bandwidth. Pin workers and memory near their GPU/NIC.
CollectivesTensor parallelism exchanges activations repeatedly. Topology matters because collectives synchronize the slowest rank.
OverlapMoving bytes can be hidden only when independent compute exists and the runtime schedules both concurrently.
ROOFLINE LAB

Compute-bound or bandwidth-bound?

20–25 min

Why it exists: arithmetic intensity connects algorithmic work to hardware capability. It is FLOPs divided by bytes moved. Compare it with the hardware ridge point: peak compute throughput (FLOP/s) divided by memory bandwidth (B/s).

Arithmetic intensity—FLOP / byte
Ridge point—peak FLOP/s ÷ bandwidth
Predicted lower bound——
Ideal roofline · logarithmic axes

Ceiling = min(peak compute, intensity × bandwidth). Minimum time = max(work ÷ peak compute, bytes ÷ bandwidth). Use the compute peak for the actual precision and dense/sparse path, and count bytes at the same memory boundary. The dot is a model ceiling, not a measured result. Launch, cache, and communication limits are omitted.

Memory-bound responseReduce bytes, reuse weights across more tokens, quantize, fuse operations, or increase bandwidth.
Compute-bound responseUse faster kernels, supported low precision, better tiling, fusion, or more compute capacity.
Reality checkThe roofline is an upper bound. Launch overhead, synchronization, shape inefficiency, and queueing make real time worse.
CHECKPOINT

Prove the mental model

25–35 min + project
QUESTION 1 OF 5

Loading…

Score 0 / 5

PROJECT 1

Trace and benchmark one Transformer forward pass

2–4 hours
ObjectiveRun a small Transformer locally and produce a resource ledger from token IDs to logits.
ArchitectureTokenizer → PyTorch model → profiler trace → latency/memory report. CPU is acceptable; a GPU adds kernel and memory observations.
ImplementationUse model.eval() and torch.inference_mode(): inference mode does not disable dropout by itself. Warm up, synchronize CUDA before and after wall-clock timing (or use CUDA events), and record tensor shapes, parameter bytes, wall time, and peak allocated memory.
ExperimentsSweep prompt lengths 16/64/256/1024 and device-supported precisions. Separate cold start from steady state.
Expected observationsWeight memory stays mostly fixed; activation work grows with tokens; first-run compilation and cache effects distort cold measurements.
Debugging exerciseExplain low GPU utilization: is the model too small, is the CPU/tokenizer gating launches, or are asynchronous launches being timed without waiting for completion?
Optimization exerciseTry inference mode, supported reduced precision, and batching. Predict the limiting resource before measuring.
Final challengeWrite a one-page recommendation: which change helped, why, what metric confirmed it, and what trade-off it introduced.
Hypothesis→measurement→bottleneck→change→verification

Score at least 4/5 before marking the module complete.

Performance playground

Change one assumption and watch memory, work, and communication move.
EDUCATIONAL CALCULATORS
Allocated KV cache—entire batch
Per token—one sequence
16-token pages—sequence pages across all layers
L×tokens×KV heads×head dim×2 (K+V)×bytes×batch

Example budget: 80 GB GPU − 14 GB weights − 4 GB runtime = 62 GB for KV. Full attention, equal sequence lengths, no sharing or sharding; count KV heads, not query heads. Round each sequence up to 16-token pages. Sliding-window layers, MLA, sharing, replication, and quantization metadata need different accounting.

Prefill work—parallel token processing
Decode steps—serial across output
Bottleneck—requires hardware + traffic measurements
Prefill · projection FLOPs
Decode · projection FLOPs

Approximation: 2 × dense parameters × tokens. Attention work is omitted and can dominate at long contexts. FLOP totals alone cannot identify compute or bandwidth limits. Prefill exploits token parallelism; standard decode has sequential dependencies. No prefix cache or speculative decoding is modeled.

Raw weight payload = parameters × bits / 8. Actual storage includes scales, zero-points, packing, and unquantized tensors. GPU counts assume perfect sharding and exclude KV, activations, runtime, and safety margin. Less storage does not guarantee faster inference or acceptable quality; test supported kernels on the target workload.

Weights / GPU—BF16 ideal shard
Ideal speedup—compute ceiling
Modeled step time—ideal compute split + serial all-reduces

Ring model: each all-reduce costs 2(N−1) × [phase latency + payload/(N × bandwidth)]. Inputs are assumptions, not benchmarks. No overlap, contention, kernel-efficiency changes, or replicated work. Production TP must satisfy model/head divisibility; KV heads may be replicated when TP exceeds their count.

Offered load / capacity—not GPU utilization
Excess arrivals in 10 s—fluid backlog lower bound
Capacity regime—p99 cannot be inferred here
Demand
Modeled capacity

Capacity = sustained output tokens/s ÷ average output length. Use a measured rate at the same prompt/output mix, concurrency, and latency target. This fluid model starts empty with constant arrivals/capacity and no burst distribution. It does not simulate batching policies, predict p99, or prove an SLO.

Metric compass

A metric is useful only when you know its clock, population, and failure mode.
MEASURE → EXPLAIN → ACT
BENCHMARK PROTOCOL

Make “tokens/s” mean something

Declare model + hardware
Fix input/output mix
Warm up
Sweep concurrency
Find saturation
Report distributions
Profile bottleneck
Repeat + compare
ControlModel, precision, engine version, hardware, prompt/output distribution.
MeasureTTFT, TPOT/ITL, E2E, output tok/s, req/s, queue, p50–p99.
RejectA single throughput number without load, latency, lengths, or measurement window.

Debug lab

Form a hypothesis before revealing the diagnosis.
EVIDENCE FIRST
SCENARIO A

SignalObservedQuestion
Choose the first thing you would test:
Likely cause
Confirm with
Try next

Progressive build path

Project ideas for practice. Only Project 1 has a detailed brief here; the others and the capstone are roadmap exercises, not completed lessons.
PROJECT ROADMAP
1 · WhyNeed and intuition
2 · SeeVisual + numbers
3 · FormalizeMath + tensors
4 · BuildCode + runtime
5 · ExecuteKernels + hardware
6 · MeasureLatency + throughput
7 · BreakMistakes + debugging
8 · OptimizeTrade-offs
9 · ProduceServing implications
10 · ProveExercise + interview
CAPSTONE

Design a production inference platform

Senior-level review

Choose model placement, parallelism, batching, KV policy, quantization, replicas, load balancing, autoscaling, observability, rollouts, and failure handling under a concrete traffic, latency, hardware, and budget envelope.

Workload70B model; 1,800 input / 320 output tokens median; 45 requests/s peak.
Hardware16 × 80 GB GPUs; two 8-GPU nodes; fast intra-node, slower inter-node fabric.
SLO + budgetp95 TTFT ≤ 900 ms; p95 TPOT ≤ 45 ms; no more than 16 GPUs. Feasibility is a hypothesis: specify GPU model, workload tails, engine, and precision, then benchmark. Revise the target if evidence does not support it.
Framework support depends on version, hardware, and model. Check the linked primary documentation in Sources before choosing an engine. TGI documentation currently labels the project as maintenance mode.

Sources & assumptions

Primary references behind the lessons. Hardware ceilings and calculators support reasoning; verify deployments with measured workloads and quality evaluations.

Scope limits. Work models assume dense decoder Transformers and ordinary autoregressive generation. MoE uses active rather than total parameters for most compute estimates; MLA, sliding attention, multimodal inputs, speculative decoding, and distributed caches need additional terms.

Progress. Lesson position and checkpoint results are saved only in this browser when storage is available. The checkpoint is a self-study milestone, not a certification.

Reviewed: 25 September 2026. Report a technical error or missing assumption.