-
Why A Single GPU Is Never Enough: A Map Of 5D Parallelism
Why one GPU is not enough: the 16Ψ memory ledger, the memory and time limits, and the five dimensions of parallelism covered in this series.
-
The Price Of All-Reduce
The six collective primitives, a derivation of the 2(N−1)/N·S cost of ring all-reduce, and measurements from a real 8-GPU PCIe machine.
-
From DP To DDP: Buckets, Overlap, And A 25 MB Sweet Spot
Why data parallelism is mathematically equivalent to large-batch training, and how DDP reduces communication overhead through gradient bucketing and overlap, measured with a full bucket_cap_mb sweep on 8 GPUs.
-
ZeRO: Three Ledgers, Zero Redundancy
How ZeRO places local updates between reduce-scatter and all-gather, divides optimizer-state memory by N without extra communication, and differs from the theoretical ledger in real DeepSpeed measurements.
-
FSDP Under The Hood: ZeRO-3 The PyTorch Way
How FSDP1 and FSDP2 shard parameters, how gather, prefetch and reshard fit into one step, and how reshard_after_forward reduces resident memory from 11.8 GiB to 1.3 GiB on 8 GPUs.
-
One Matrix, Two Cuts: Tensor Parallelism From Scratch
Why tensor parallelism uses a column cut followed by a row cut, why weight gradients remain local, how its communication compares with data parallelism, and what an 8-GPU implementation measures.
-
Splitting The Sequence: Megatron-SP And Ring Attention
How sequence parallelism removes the activations replicated by tensor parallelism, how Ring Attention distributes long contexts, and why context parallelism scales on the same PCIe machine where TP does not.
-
The Geometry Of Pipeline Bubbles
A derivation of the (p−1)/(m+p−1) pipeline-bubble fraction, how 1F1B reduces activation memory without changing that fraction, and measurements showing the per-microbatch costs omitted by the formula.
-
The bf16 Bargain: A Numerics Ledger For Mixed Precision
Why mixed-precision training uses bf16 for computation and fp32 for persistent state, why fp16 needs loss scaling, and how a small-learning-rate experiment exposes updates lost without an fp32 master copy.
-
Optimizers In A Sharded World: Muon's O(mn) Vs RMNP's Zero
How the data required by an optimizer interacts with parameter sharding: RMNP uses row-local normalization and needs no communication under row sharding, while Muon gathers full matrices, producing a measured 67× preconditioning-time gap on 8 GPUs.
