Mixture of Experts Architecture in Transformer Models Introduced

Mixture of Experts architecture has been introduced as a method to enhance the efficiency of transformer models while managing computational complexity. This architecture allows for the selective activation of parameters, enabling large models to maintain performance without proportionally increasing computational costs.

The concept of Mixture of Experts (MoE) was first introduced in 1991 and gained renewed attention with recent models like Switch Transformer and Mixtral. In MoE, only a subset of parameters is activated for each input, which allows for the creation of extensive models that are computationally efficient. This targeted approach helps manage the challenges associated with the traditionally high computational demands of transformer models.

In MoE architecture, multiple expert models are used to handle input data, and a gating mechanism is employed to decide which expert should be activated for a given input. The architecture modifies the multi-layer perceptron (MLP) block while keeping the attention block shared among the experts. This design enables a variety of combinations across transformer layers, ensuring scalability without drastically increasing the number of parameters.

The MoE architecture consists of three main components, where the operation begins with the router producing logits from the output sequence of the attention block. After applying a softmax transformation, the top-k operation selects which experts to engage, allowing for a more efficient computational process. This model works effectively even with a small value of k, such as 2.

A specific implementation of MoE in transformer models includes a sequence of transformer layers where each layer features both an attention sublayer and an MoE sublayer. The implementation is designed to process input independently across sequence vectors, enhancing the model's ability to handle diverse tasks efficiently. Recent advancements also propose the inclusion of shared experts, which operate without the need for a router, further optimizing the model's performance.