LLM Inference at Scale: 1M Context on GPUs — Illustrated Guide (14 Parts)

How do you run Llama-70B with a 1 million token context window on A100 GPUs? The model weights need 140 GB, the KV cache needs 328 GB, and the naive attention matrix would require 128 TB. This series walks through every technique needed to make it work — with step-by-step calculations and animations at every stage.

The Reader Journey

Phase 1 (Parallelism): "Why doesn't it fit?" → "How to distribute it"
Phase 2 (Quantization): "How to compress it" → "Even more headroom"
Phase 3 (Production): "How to actually serve it" → "Working code"

Phase 1: Parallelism

✓

Part 1: The GPU Memory Problem

Why Llama-70B with 1M tokens doesn't fit on a single GPU. The three bottlenecks: weights, activations, and attention.

✓

Part 2: Weight Sharding (FSDP)

Distributing 140 GB of weights across GPUs using Ring AllGather and ReduceScatter.

✓

Part 3: Sequence Sharding (Ulysses)

Splitting the 1M token sequence with All-to-All communication so each GPU handles a subset of attention heads.

✓

Part 4: Ring Attention

Distributed attention by rotating KV blocks around a ring. No head count limit — scales to any number of GPUs.

○

Part 5: Putting It Together (USP)

FSDP + Ulysses + Ring Attention orchestrated in one forward pass.

Phase 2: Quantization

○

Part 6: Why Quantization — The Memory Multiplier

Number representations, what's lost, memory savings, interaction with USP.

○

Part 7: Post-Training Quantization (PTQ)

Absmax, zero-point, per-group, outlier problem, GPTQ, AWQ.

○

Part 8: Weight-Only vs Weight-Activation Quantization

W8A16, W4A16, W8A8, SmoothQuant, dynamic vs static.

○

Part 9: Advanced Quantization — GGUF, AQLM, QuIP#

Sub-4-bit, vector quantization, quality vs compression tradeoffs.

Phase 3: Production

○

Part 10: Throughput & Latency

Batch sizing, QPS, prefill vs decode, TTFT, inter-token latency. How quantization + USP multiply savings.

○

Part 11: Serving Mixed Workloads

Continuous batching, variable-length sequences, request scheduling, SLA management.

○

Part 12: KV Cache Management

PagedAttention, prefix caching, eviction strategies, offloading to CPU/disk.

○

Part 13: Multi-Node Deployment

Network topology, fault tolerance, monitoring, choosing the right parallelism config.

○

Part 14: Code Walkthrough

Reference USP implementation, quantization integration, benchmarks, end-to-end deployment.

Cite this series

If you find this series useful in your work or writing, feel free to cite it:

@misc{fofadiya2026llminference,
  author       = {Darshan Fofadiya},
  title        = {The Illustrated Guide to LLM Inference at Scale},
  year         = {2026},
  url          = {https://darshanfofadiya.com/llm-inference/},
  note         = {Accessed: 2026}
}

Or in plain text:

Darshan Fofadiya, "The Illustrated Guide to LLM Inference at Scale," 2026. https://darshanfofadiya.com/llm-inference/