distributed-training
an archive of posts with this tag
| Jul 13, 2026 | Optimizers In A Sharded World: Muon's O(mn) Vs RMNP's Zero |
|---|---|
| Jul 13, 2026 | The bf16 Bargain: A Numerics Ledger For Mixed Precision |
| Jul 12, 2026 | The Geometry Of Pipeline Bubbles |
| Jul 11, 2026 | Splitting The Sequence: Megatron-SP And Ring Attention |
| Jul 10, 2026 | One Matrix, Two Cuts: Tensor Parallelism From Scratch |
| Jul 09, 2026 | FSDP Under The Hood: ZeRO-3 The PyTorch Way |
| Jul 08, 2026 | ZeRO: Three Ledgers, Zero Redundancy |
| Jul 07, 2026 | From DP To DDP: Buckets, Overlap, And A 25 MB Sweet Spot |
| Jul 06, 2026 | The Price Of All-Reduce |
| Jul 05, 2026 | Why A Single GPU Is Never Enough: A Map Of 5D Parallelism |