Train Your Large Model on Multiple GPUs with Fully Sharded Data Parallelism

On January 1, 2026, an article was published discussing how to train large language models using Fully Sharded Data Parallelism (FSDP) across multiple GPUs. The article outlines the advantages of sharding, a method that divides model parameters across different devices to improve training efficiency.

Sharding, originally a concept from database management systems, involves splitting a large model into smaller units known as shards. In machine learning applications, sharding helps manage model parameters by distributing them across several GPUs. This differs from traditional pipeline parallelism, where only portions of operations are shared, as FSDP requires the gathering of shards to form complete matrices for processing.

The method described in the article indicates that with FSDP, each GPU holds only one shard of the model. This contrasts with data parallelism, where each GPU maintains a full copy of the model. Instead, FSDP synchronizes both the model and the data at every training step, which leads to increased communication overhead but significantly lowers memory usage, making it possible to train larger models.

FSDP operates by having multiple processes exchange data to temporarily unshard the model. A typical workflow involves an all-gather request that allows all processes to share their shards, thus forming a complete module for processing. The article explains that after processing, the unsharded module is discarded to save memory. During the backward pass, different gradients are computed for each micro-batch of data, which are then averaged across processes to update the model parameters accordingly.

To enhance efficiency, the article mentions that PyTorch employs prefetching techniques to overlap the communication and computation phases. This means while one module is being computed, shards for the next module are exchanged, optimizing the training process.

For those looking to implement FSDP in PyTorch, the article provides guidance on using the torchrun command to initiate the training script with multiple processes. It stresses the importance of applying the fully_shard() function on both the model and its submodules to ensure effective sharding throughout the model architecture. This method is crucial for managing large models that cannot fit on a single GPU, thereby enabling scalable training solutions.