Mixture of Experts (MoE) is a model structure that replaces the feed-forward network with several expert sub-networks, with a router assigning computation per token. The key point is not how many experts there are, but sparse activation: each token passes through only a few experts, while the rest stay idle for that step. Total parameters can therefore be very large, while the compute per inference step is counted only on the activated part. Grasping that sentence means grasping all of MoE's trade-offs.
The router is the smallest yet most critical part of MoE. It is a small learnable network that reads each token's representation, scores every expert, and keeps only the top-k scorers. The selected experts each process the token, and their outputs are combined weighted by the router scores. Attention layers, embeddings, and the router itself are shared by all tokens; only the expert feed-forward layers are sparse. Some designs add a shared expert that every token passes through, handling common patterns and easing the load on routed experts.
Dense models and MoE: keep the two bills separate
A dense model runs every parameter for each generated token, so doubling parameters roughly doubles both compute and memory. MoE separates the two: total parameters decide how many weights must fit in memory, while activated parameters decide how fast each step runs. On the same card, an MoE model can often hold far more total parameters, but that does not mean it runs like a small dense model, because all weights must still reside in memory, and loading and switching experts costs time too. Reading only total parameters misleads, and reading only activated parameters misleads as well; the two numbers must be read together.
Three common misconceptions need separating. First, experts are not departments divided by subject. Routing learns a division of labour over statistical patterns; nobody labels an expert a "math expert" or "code expert", and forcing an expert by topic is unreliable. Second, more experts do not automatically add capability. If the router sends most tokens to a few experts, the idle ones are wasted memory, which is why training uses load-balancing constraints to prevent this collapse. Third, MoE does not automatically save money. It saves compute per token, in exchange for larger memory, more complex deployment, and harder-to-debug training; serving must also pay for expert parallelism and communication.
MoE fits settings with large total-capacity needs, tight per-step latency budgets, and the resources for large memory and distributed deployment, such as large-scale online inference. Running MoE on a personal device starts with the memory bill: total weights decide whether it fits, activated size decides how fast it runs, and both conditions are required. Conversely, when the parameter scale is modest and deployment is simple, a dense model is usually less trouble, with more mature debugging and quantization tooling. When choosing an architecture, look at the constraints first, then at the parameter numbers.
