Technical
Mixture of Experts(MoE)
Definition
A neural-network design that routes each token to a few specialized sub-networks (experts), growing model capacity without growing per-token compute.In-Depth Explanation
In a Mixture-of-Experts model, an MoE layer replaces the standard feed-forward block with many expert networks plus a gating "router." For each token the router selects the top-k experts, so only a fraction of the model's parameters activate on any given forward pass — this is "sparse activation." The result is that models can hold far more total parameters (even into the trillions) while keeping inference cost close to a much smaller dense model. Many frontier LLMs now use MoE for efficient scaling.
Real-World Example
Mixtral 8x7B has eight experts but activates only two per token, giving large-model quality at lower compute cost.
4 views0 found helpful