Measuring Inference Performance of LLMs

A new chapter focuses on the measurement of inference performance for large language models (LLMs). It highlights the necessity of accurate measurement to avoid complications in models that do not enhance speed or worsen user-visible latency while improving throughput.

The chapter outlines various performance metrics relevant to LLMs. Users are primarily concerned with the time taken to see the first token and the speed at which the complete answer is delivered. Operators are interested in the number of requests the hardware can handle, memory usage, and the cost associated with each generated token. Researchers may focus on the impact of optimizations on the quality of the model's output.

Commonly used metrics for measuring performance include latency numbers. For LLMs, metrics should consider different aspects, such as prefill and decode requests, which can have varying performance profiles despite the same total token count. Tail latency is also crucial, as noticeable delays could affect user experience. High-percentile latencies like p90, p95, and p99 should be reported alongside mean or median latencies to provide a clearer picture of performance in worst-case scenarios.

The simplest method for measuring performance employs time.perf_counter(), a high-resolution timer in Python, which is more accurate than time.time(). When measuring, it is important to account for one-time costs that shouldn't skew results, such as module imports or initial code executions. Therefore, benchmarks should include warmup iterations to ensure steady-state measurements.

For GPU-based LLM inference, it is essential to synchronize GPU operations using torch.cuda.synchronize() to obtain accurate timing metrics. This process ensures that wall-clock measurements reflect the actual time taken for GPU work.

Additionally, CUDA events can measure GPU execution time without Python overhead, providing valuable insights for kernel optimization. Tools like PyTorch Profiler or Nsight Systems can further aid in detailed GPU profiling by reporting on memory usage and GPU performance.

Memory measurement is also vital for understanding how many users a system can support, with GPU memory often being a limiting factor. Reporting allocated versus reserved memory can help in capacity planning, particularly when measuring memory alongside token usage.