PyTorch Provides Infrastructure for Pipeline Parallelism to Train Large Models

PyTorch offers infrastructure to enable pipeline parallelism for training large language models that are too large to fit on a single GPU.

Pipeline parallelism functions by creating a model as a pipeline of stages. In a transformer model, each stage consists of a transformer block. These blocks are chained together, where each block takes one tensor as input and produces one tensor as output. This process is mathematically equivalent to executing the model.

To keep all GPUs busy and avoid idle time, PyTorch uses the concept of micro-batches. Instead of processing a single batch of size N, the system splits the batch into n micro-batches of size N/n. This allows one stage to process a subsequent micro-batch while another stage is still processing the current one. Results are aggregated after all micro-batches are processed.

Model preparation involves creating the model for one stage or creating a full model on a fake meta device to avoid out-of-memory errors. When using a meta device, weights are not allocated. The model is then partitioned into stages, such as dividing decoder layers into thirds across different ranks. Components not required for a specific stage are set to None, and the model code must be modified to skip these components during the forward pass.

Transferring a partial model from a meta device to a real GPU requires the to_empty() method to allocate weight tensors. Weights are then initialized using a reset_all_weights() function.

Execution is handled via the torchrun command, which launches multiple processes. Each process is assigned a unique rank and a local rank to identify its GPU device. The distributed environment must be initialized using the torch.distributed module. Users then create a stage object to define the model's stage, its device, and the total number of stages. PyTorch provides various schedules for processing micro-batches, with ScheduleGPipe serving as the default.

The pipeline parallelism API in PyTorch is experimental and may change. The described methods were tested on PyTorch 2.9.1, and different versions may not work.