Mixture of experts
A model design that splits the network into many specialized subnetworks and activates only a few per input, so a very large model can run at the cost of a much smaller one.
A mixture of experts (MoE) breaks a model into many smaller “expert” subnetworks plus a router that, for each input, sends the work to only a few of them. The model can hold a huge number of parameters while activating just a fraction at a time, which keeps the compute cost per query down.
The trade-off is complexity: training and serving MoE models is trickier than a plain dense neural network. The design became prominent in efficient large language models, including the systems behind DeepSeek-R1.