Knowledge just moved out of the GPU

DeepSeek released a paper and a repository in January 2026 that proposed a new architectural subsystem for LLMs: moving knowledge outside of the neural network parameters and into a hash-based lookup table called Engram.

In the past few weeks, two major labs have released industry-grade models with a set of Engram parameters: DeepSeek's own v4.1-Flash, and Qwen3.8-Flash-Next (which is a preview of the architectural design of the upcoming Qwen 4 series).

This is a bigger change than it looks, because it changes the economics of LLM inference. Here is why.

How LLMs work, with and without Engram

Every token is a trip through memory

A language model writes one token at a time. To produce the next one, it runs the whole network once: the context goes in, every layer does its arithmetic, and a probability for each word in the vocabulary comes out. Pick one, append it, run the whole network again. A five-hundred-token answer is five hundred passes, each waiting on the last.

The arithmetic is not the expensive part. The expensive part is that every weight a pass touches has to travel from memory into the GPU compute units, and for a single token each weight is used exactly once and then discarded. A dense 27B-parameter model holds 54 GB of weights, so generating one token means reading 54 GB. An H100 (an enterprise-grade NVIDIA card that costs $25k–$40k) can pull about 3.35 TB/s out of its memory, which puts a hard ceiling of roughly 60 tokens per second on one conversation, before any other overhead. The same chip can do about a thousand trillion floating-point operations per second. The arithmetic takes 0.05 ms. The memory read takes 16 ms. More than 99% of the time the compute units sit idle, just waiting for bytes.

This is what "memory-bound" means, and it is why the speed of an LLM at generation time is a bandwidth number rather than a FLOPS number.

What one generated token reads from memory

Dense model

54 GB read from HBM per token

A dense 27B model in bf16. Every weight streams out for every token, and each new token goes back into the context for the next pass.

Everything about how accelerators are built follows from this. High bandwidth memory (HBM), the type of memory in GPUs, moves a few terabytes per second. DDR5 does not come close: a desktop tops out near 100 GB/s and a big server near 500, still several times slower than HBM.

Mixture-of-experts to the rescue

The first approach to minimize the number of parameters needed to go from HBM to the GPU cores was an architecture called Mixture-of-Experts (MoE). In this architecture, each token is routed only through a subset of parameters (the "experts"). If you've ever seen a model name like Qwen3.6-35B-A3B, that's an MoE model that contains 35B parameters but only activates 3B parameters per token.

Only a handful of experts run per token, so only their weights need to be read by the GPU cores. But the idle experts still occupy HBM.

What one generated token reads from memory

Mixture-of-experts

6 GB of 70 GB read per token, 70 GB resident in HBM

Qwen3.6-35B-A3B in bf16: two experts wake per token, but all of them have to live on the GPU.

The result of this architecture is an unchanged need for HBM (since all experts need to be loaded into the GPU) but lower token generation time since we need to move fewer bytes between the HBM and the GPU cores. The tradeoff is that a 35B-A3B model does not have the same intelligence as a dense 35B model. In fact, Qwen3.6-27B is widely regarded as a better model than Qwen3.6-35B-A3B in terms of output quality.

Engram: the parameters you just look up

Engram, the paper DeepSeek published in January, proposes a different way of reducing the required HBM memory.

LLMs don't use text directly. They first translate it into tokens, which you can think of as pseudowords. The problem is that in human language we have plenty of concepts that are multi-token. For example, "New York" is two words, but one concept.

How do LLMs deal with this? Through compute: the parameters in their neural networks not only encode how to reason about text, but also need to encode knowledge.

DeepSeek looked at this through a different lens: can we move knowledge outside of the neural network? The answer reuses the oldest idea in language modeling: n-grams.

Take the current token and the one or two before it. That short sequence (e.g. New York, as a result, def __init__) is a key. Hash the key into a very large table, pull out some information (an embedding vector), and add it to the internal state of the LLM at an early layer. The lookup is constant time, it involves no matrix multiplication, and the key depends only on token ids the model already has.

Each token touches a handful of rows out of billions, and because the key is just a few token ids, the model knows which rows it needs before the pass starts. That is what lets the table live in host RAM, or even on an NVMe SSD.

Where the Engram table lives and what one token fetches

Engram

tens of KB per token over PCIe, fetched before the backbone runs

DeepSeek V4.1-Flash: a 196B-parameter table in host RAM, a 16B-active backbone in HBM. The row address is a hash of the last few tokens, so it is known before the pass starts.

What's the benefit of the Engram table? Once the LLM gets to "York" in "New York", it will instantly know the text is referring to the US city and not the UK one thanks to the information pulled from the Engram table. In a traditional LLM, that only happens through computation, as the tokens move through the different layers of the neural network.

A telling result from the paper is that once DeepSeek removed the Engram table from their model, factual benchmarks collapsed to 29% to 44% of their original score, while reading comprehension kept 81% to 93%. The facts were in the table. The reasoning was not.

How this changes the economics of LLMs

Two kinds of parameters, two price tags

Until now, every parameter in a model had to live in HBM, and HBM is soldered next to a GPU that costs tens of thousands of dollars. If your model needed more memory, you bought more or bigger GPUs.

Engram splits the model in two. The reasoning parameters still need HBM, because every token reads them and only HBM can feed them fast enough. The knowledge parameters need something else entirely: capacity. A few kilobytes per token, fetched ahead of time, can come from DDR5 you add to the server by the stick, or from an SSD that holds tens of terabytes for a few cents a gigabyte. In V4.1-Flash that is 196B of roughly 750B parameters. In Qwen3.8-Flash-Next, 51B of 176B. About a quarter of each model just stopped needing a GPU. Knowledge that used to cost compute now costs a lookup, so the labs can keep the active parameter count small by using MoE (6B for Qwen and 16B for DeepSeek), without giving up what the model knows.

The combination of MoE and Engram allows aggressive optimizations, like running Qwen3.8-Flash-Next on a 12 GB GPU: a 125B-parameter model with 6B activated parameters plus a 51B n-gram lookup table. The experts live in host RAM and the lookup table on an NVMe drive, and it still decodes at about 20 tokens per second.

Self-hosting and fine-tuning

We argued in an earlier post that your business fits in a smaller model. Engram makes the expensive part of a model (the neural network parameters) smaller. The question for a specialized model is whether 196B of facts about the world help a task that does not need any of them. Probably not: a model that reads insurance claims does not need to know who discovered penicillin. Nobody has a good recipe for fine-tuning Engram models yet, so trimming or replacing that table is still an open problem.

Why this matters

For as long as LLMs have been served from GPUs, a parameter was a parameter: every one of them had to sit in the most expensive memory in the building. MoE was the first crack in that rule. It cut the bytes that move per token, but left the HBM bill untouched. Engram is the second. It cuts what has to sit in HBM at all. Put the two together and the GPU holds only the part of the model that reasons, while what the model knows sits in cheap RAM or an SSD.

Two labs shipping the same parameter split within weeks of each other is a sign that this architecture has evolved from research prototype to industry trend.

If you run models, capacity planning now has two lines instead of one: HBM bandwidth for reasoning, and cheap capacity for knowledge. If you want to own a specialized model, what the model knows is becoming a separate artifact from how it reasons, one that could be inspected, swapped, or shrunk to fit the job. That second part is the one we are watching most closely.