并行策略 on Yiwen Caihttps://yiwen-cai.github.io/tags/%E5%B9%B6%E8%A1%8C%E7%AD%96%E7%95%A5/Recent content in 并行策略 on Yiwen CaiHugo -- gohugo.iozh-cncaiyiwen.cs@foxmail.com (Yiwen Cai)caiyiwen.cs@foxmail.com (Yiwen Cai)© 2026 Yiwen CaiFri, 10 Jul 2026 21:12:36 +0800分布式训练并行策略：CS336 Lecture 7 笔记https://yiwen-cai.github.io/notes/systems/cs336-distributed-parallelism/Mon, 15 Jun 2026 00:00:00 +0000caiyiwen.cs@foxmail.com (Yiwen Cai)https://yiwen-cai.github.io/notes/systems/cs336-distributed-parallelism/从单 GPU 扩展到多 GPU/多机并行：集合通信原语（all-reduce = reduce-scatter + all-gather）、NVLink/InfiniBand 互联，以及 DDP、FSDP/ZeRO、Tensor/Pipeline/Sequence Parallelism 的取舍与实践法则。