Model2026-09-01

Understanding How AI Models Put Their Parameters to Work

#dense-model#mixture-of-experts#state-space-model#model-architecture#parameter-activation

Parameter count alone doesn't tell you much about a model, because how those parameters actually get used at inference time varies a lot from one architecture to the next. Some models fire every parameter on every incoming token. Others fire only a fraction of them. Still others don't hold onto individual tokens at all — they compress everything that came before into a running state and carry that forward instead. Split models along this line and you get three families: Dense, MoE (Mixture of Experts), and SSM (State Space Model).

The question separating these three architectures is a simple one: of all the parameters a model holds, what percentage actually does work on a given token? That percentage feeds directly into inference cost, memory footprint, and latency, which is why sorting models by how they activate parameters turns out to be more useful in practice than sorting them by total parameter count. In a Dense model that percentage is effectively 100%. In MoE, only a slice of the total gets switched on for any given token. SSM doesn't even frame the problem as dividing parameters across tokens — it changes how the sequence itself gets processed, which reshapes the whole compute-and-memory curve.

This post walks through what problem each architecture set out to solve, how they compare on functionality and performance, and which deployment environment each one actually fits best.

Where it all starts: Dense

Dense is the simplest structure there is: every parameter in the model gets involved on every token, no matter what the input is. It's been the default since neural language models first started scaling up, and plenty of models still stick with it today.

Dense earned its long run as the default because of that simplicity. With every parameter always in play, training is stable, inference latency is easy to predict, and there are fewer variables to account for when designing distributed training or serving infrastructure. Improving a Dense model is just as straightforward: pour in more parameters along with proportionally more data and compute, and performance climbs along a predictable curve — a pattern early scaling research confirmed widely enough that growing Dense architectures became the industry's default playbook for a good stretch of time.

The catch is that this same simplicity turns directly into cost. Compute for training and inference grows almost linearly with parameter count. On top of that, a Dense model built on attention carries the added burden of compute scaling near-quadratically as sequence length grows. In other words, whether you're trying to make the model smarter or trying to give it a longer context window, the cost climbs right along with you — that's Dense's fundamental limitation.

Given these traits, Dense fits best where model size stays relatively small to mid-scale, and where predictable latency and simple deployment matter more than squeezing out maximum performance. On-device and edge environments that can't easily support complex routing infrastructure are a good example, as are settings that need to keep fine-tuning and serving pipelines as simple as possible.

Between big capacity and low compute: MoE

MoE starts from Dense's core dilemma: grow the model's capacity and its compute cost grows right along with it. The idea behind MoE is to break that link. Put a number of specialized sub-networks — experts — inside the model, and have a router pick only a handful of them to activate for each token. Do that and you can grow total parameter count, and with it the model's representational capacity, while keeping the actual compute spent per token low.

The core payoff is that MoE decouples total capacity from compute cost. Total parameter count can run far higher than a Dense model's, but because only a slice of those parameters activates per token, you get more model capacity for the same compute budget. There's also a quality upside: as different experts specialize toward different patterns or domains through training, MoE models sometimes outperform Dense ones on particular tasks.

That said, the structure drags in a new kind of complexity. Learning which expert to route a token to is prone to instability, and load imbalance — traffic piling up on a handful of experts — has to be actively managed. Serving gets more complicated too. Even though only a fraction of parameters actually computes anything for a given token, you can't know in advance which token will go to which expert, so every expert still has to sit in memory, and spreading experts across multiple devices (expert parallelism) brings communication overhead that has to be designed for directly.

Given these traits, MoE fits best where a system needs maximum model capacity and already has, or can build, the distributed serving infrastructure to support it. That means building frontier-scale models while keeping average per-token compute cost down, or covering a wide mix of task types with a single model that benefits from per-expert specialization.

Breaking through the sequence-length wall: SSM

SSM takes issue with something entirely different from what Dense and MoE argue over. Both Dense and MoE process sequences through attention, and attention computes the relationship between every pair of tokens in the sequence — which means compute and memory both grow near-quadratically as the sequence gets longer. SSM emerged from the recognition that this ceiling becomes a genuine bottleneck for applications where the sequence itself runs long: long documents, real-time streaming, and the like.

Instead of recomputing the relationship between every pair of tokens each time, SSM's approach compresses everything the sequence has seen so far into a state of fixed size and carries that forward. Each new token updates this state rather than being compared against everything that came before, so compute grows much more gently with sequence length than it does under attention, and the memory required for autoregressive generation stays constant regardless of how long the sequence gets.

That said, this advantage comes with a real cost attached. Because the state is compressed to a fixed size, tasks that need to reach back and retrieve a specific earlier token exactly, or copy it verbatim, tend to see lower accuracy than an attention-based architecture that can compare every pair of tokens directly. The very thing that makes compression useful is also where detail can get lost.

Given these traits, SSM fits particularly well where the input sequence itself is extremely long — long documents, logs, genomic data — and where real-time streaming or edge inference needs to hold latency and memory tightly in check. On the other side, for tasks that need to precisely trace back to a specific point somewhere in the sequence, attention-based architectures are still generally considered hard to fully replace.

The axis that actually separates them: what gets switched on, and how much

Line the three architectures up and it becomes clear that each is optimizing for something different. Dense optimizes for simplicity and predictability, accepting that compute cost rises in lockstep with parameter count in exchange for keeping training and serving complexity low. MoE optimizes for decoupling capacity from compute cost, and takes on the new complexity of routing and distributed infrastructure as the price for that. SSM optimizes for scaling gracefully with sequence length, and gives up some precision in exactly pinpointing individual tokens as the price for that.

It's also worth noting these three aren't mutually exclusive. Recent work increasingly combines attention-based layers with SSM layers, then stacks MoE-style feedforward layers on top — an attempt to cherry-pick the strengths of all three. That trend is a good sign that these architectures aren't really competing alternatives so much as design axes you can mix inside a single model.

In summary

Sort AI models by how they activate parameters and the problem each architecture was built to solve comes into sharp focus. Dense started as the simplest possible default and still holds its ground wherever predictability and simple deployment matter. MoE grew out of the attempt to decouple model capacity from compute cost and earns its keep in environments with the distributed serving infrastructure to support it. SSM grew out of the attempt to break through the sequence-length wall and stands out wherever long context and real-time processing matter most.

None of the three is a complete answer on its own. Dense carries a heavy cost burden once you're chasing scale, MoE has to absorb the complexity of routing and infrastructure, and SSM still has ground to make up on precisely pinpointing individual tokens. Choosing an architecture, in the end, isn't about finding the one model that solves everything — it's about first figuring out which axis your own deployment environment actually needs most: compute cost, model capacity, or sequence length.

References

← Blog