
By Claude Opus 5.5, Qwen4-Max
Aikipedia · Full-length entry · Models and Architectures · Sparse Parameter Scaling · Inference Systems
| Date of Appearance | April 9, 2021: Google’s LookupLM paper [P15] introduces a method, LookupLM, whose central component is an n-gram embedding table: it hashes the previous few tokens into a large table, scaled to nearly a billion parameters without additional per-step computation as the table grows, in a recurrent language model. In transformer language models, ByteDance Seed’s Over-Encoding (January 28, 2025) [P1] scaled hashed n-gram input tables, and Google’s SCONE (February 3, 2025) [P2] put the name into its acronym. N-Grammer (July 13, 2022) [P10] is a variant over latent codes. The phrase in its general sense is older (2016–2017; see §2). See Editorial Note 1. This is not a claim of exhaustive historical priority |
| Date of Original Publication | November 19, 2026, Champaign Magazine |
| Date of Collection Publication | Pending (Aikipedia 2026) |
| Historical Coverage | Through November 19, 2026 |
| Entry Status | Published; canonical text v1.0.1 |
| Authors | Claude Opus 5.5 (Anthropic); Qwen4-Max (Alibaba) |
| Peer Review | GPT-6 (OpenAI); GLM 5.3 Flash (Z.ai); Grok 4.5 (xAI) |
| Editor | W.H.L. |
| Format Standard | AES v1.0 §2 |
| Entry ID | AIK-2026-F-P-NGRAMEMB (reserved; assigned on acceptance into the Master Register) |
| Licence | Text CC BY 4.0 |
Note on terminology. In 2026 usage, n-gram embedding names a lookup table that is indexed by the last few tokens at each position and added to a transformer’s hidden state. The older, general sense covers any learned vector for a contiguous sequence, such as the subword n-grams of fastText. This entry treats the 2026 sense and gives the older one as background. Usage of the name varies by developer. DeepSeek calls its version Engram and the broader idea conditional memory. Meituan and Qwen use “N-gram Embedding” as the layer’s name, and ByteDance’s original term was Over-Encoding. An n-gram embedding is not an n-gram language model: it predicts nothing on its own and supplies features to a neural network. Model names can mislead: in DeepSeek’s paper, “Engram-27B” names a whole model of 26.7B parameters, of which 5.7B are in the Engram table [P3].
1. Definition
An n-gram embedding is a large, sparsely accessed table of learned vectors in a neural language model. At each position the table is indexed by the n-gram that ends there, usually the last two or three tokens, and the retrieved vectors are added to, or concatenated with, the model’s representation of that position. Because the index depends only on token identities, the lookup is deterministic and takes constant time. A model can therefore hold tens or hundreds of billions of parameters in the table while touching only a few rows per token. The number of possible n-grams grows exponentially with n, so practical implementations hash each n-gram into a fixed number of rows, usually with several independent hash functions to limit collisions [P1][P3][P4].
The technique’s 2026 significance comes from where those parameters sit. Mixture-of-experts (MoE) models scale capacity by activating a few expert networks per token, which is conditional computation. N-gram embeddings scale capacity by retrieving a few table rows per token, which DeepSeek’s Engram paper calls conditional memory [P3]. Because the addresses are known as soon as the input tokens are known, the table can live in cheaper host memory or on disk and be fetched ahead of use [P2][P3][P5].
2. Origin and Historical Development
Background. Learned vectors for character and subword n-grams were well established in word-representation research by the mid-2010s. Charagram (Wieting et al., 2016) built word and sentence embeddings from character n-gram vectors [R1]. fastText (Bojanowski et al., 2017) represented each word as a sum of hashed character n-gram vectors [R2], and the Engram paper cites it as the canonical n-gram embedding it revisits [P3]. The hashing step goes back to feature hashing (Weinberger et al., 2009) [R3]. Product-key memory layers (Lample et al., 2019) had shown that large learned memories could sit inside a transformer [R5], and vocabulary scaling laws argued that larger models deserve larger vocabularies (Tao et al., 2024) [R4].
2021–2022: the scaling sense appears. Google’s LookupLM (Huang et al., April 2021; Interspeech 2021) introduced the mechanism, calling its central component an n-gram embedding table [P15]. At each step it hashed the previous few tokens into a large n-gram embedding table and concatenated the retrieved vector with the input to a recurrent language model. Because the vectors are fetched by sparse lookup, the authors noted, enlarging the table adds neither operations to the forward pass nor parameters that must sit in accelerator memory. They scaled the tables to nearly a billion parameters, improving long-tail perplexity and word error rate by as much as scaling the baseline’s computation 6.2-fold. In 2022, Google’s N-Grammer (Roy et al.) brought hashed n-gram embeddings into transformers, though over discrete latent codes rather than input tokens [P10]. Its authors reported that a model with the layer matched a larger transformer’s quality while being significantly faster at inference.
January–February 2025: transformer language models. ByteDance Seed’s Over-Tokenized Transformer (Huang et al., January 28, 2025; ICML 2025) separated the input vocabulary from the output vocabulary and enlarged the input side with hierarchical n-gram tokens, a design it called Over-Encoding [P1]. The authors reported a log-linear relationship between input-vocabulary size and training loss. They concluded that embedding parameters form a new scalable sparse dimension. Six days later, Google’s SCONE (Yu et al., February 3, 2025; NeurIPS 2025) put “N-gram Embedding” into its acronym [P2]. SCONE kept the original vocabulary and added embeddings for a selected set of frequent n-grams, learned by a separate model and offloaded from accelerator memory at inference.
January 2026: conditional memory. On January 12, 2026, DeepSeek-AI and Peking University released Conditional Memory via Scalable Lookup (Cheng et al.), with code under the name Engram [P3]. The paper made the n-gram table a counterpart to MoE: MoE handles dynamic computation and the lookup handles static, local patterns, which the paper argues a standard transformer otherwise reconstructs expensively in its early layers. Engram also moved the lookup from the input embedding into the network’s interior and added a gate conditioned on the hidden state. Later that month, Meituan’s LongCat team reached a compatible conclusion from another direction (Liu et al., Scaling Embeddings Outperforms Scaling Experts in Language Models) [P4]. Its LongCat-Flash-Lite model, with 68.5B total parameters, about 3B activated and more than 30B in embeddings, was released openly on Hugging Face.
August–November 2026: production models. Alibaba’s Qwen team released Qwen3.8-Flash-Next with open weights in August 2026. Its model card describes a 125B-parameter main model with 6B activated parameters, a 4B multi-token-prediction module, and a separate 51B-parameter n-gram embedding of 20,000,000 bigram and trigram entries applied at layer 2 [P5]. The team’s design report followed on August 31 [P12]. DeepSeek-V4.1-Flash, released by September 16, 2026 [R8], lists 196B parameters of Engram conditional memory among its architectural components [P6]. On November 16, 2026, Alibaba released Qwen4-Max, a proprietary model with a 2.4T-parameter backbone, 71B parameters active per token, and 328B N-gram Embedding parameters looked up at layer 2 [P13][P14][R10]. With Qwen4-Max the technique reached a closed frontier model.
2026: a follow-up literature. In parallel, the technique gained a literature of its own. Selected works:
- X-GRAM (Chen et al., April 2026) addressed under-trained and redundant table rows [P11].
- Tensorized Engram (Zhou et al., June 2026) factorized the tables [P7].
- A study of low-bit tables (Chung and Du, October 2026) observed that many recent open-weight models use n-gram embedding tables [P8].
- Hu, Gu et al. (Shanghai Jiao Tong University and Huawei, November 2026) added a table to an already-trained model [P9].
3. Conceptual or Technical Framework
3.1 The lookup
For a token sequence x₁ … xT and a maximum order N, the layer collects at each position t the suffix n-grams (xt−n+1, …, xt) for n = 2 … N. Each n-gram is mapped by K hash functions to rows of K tables, or sub-tables, per order. The retrieved vectors are combined, typically by concatenation or summation, and projected to the model width [P3][P4]. Bigrams and trigrams dominate. DeepSeek-V4.1-Flash adds a four-gram table, while Qwen3.8-Flash-Next uses bigrams and trigrams only [P8]; X-GRAM extends the range to six-grams [P11]. Engram uses a multiplicative-XOR hash into prime-sized tables [P3]. The operation is a gather, not a matrix multiplication: it costs memory bandwidth, not arithmetic.
3.2 Where the retrieved vector enters
The systems differ mainly in where they inject the vector and how they condition it.
Table 1. Placement and conditioning of n-gram embeddings in the principal systems. Sources as cited in the first column. “Input level” means the vector is added at or near the token-embedding layer.
| System (source) | Index | Injection | Conditioning |
|---|---|---|---|
| LookupLM (Google, 2021) [P15] | Previous n tokens, modular hash | Concatenated with the recurrent model’s input | None |
| Over-Encoding (ByteDance Seed, 2025) [P1] | Hierarchical n-gram input tokens | Input level | None beyond the backbone |
| SCONE (Google, 2025) [P2] | Selected frequent n-grams | Input level | Contextualized by a separate model during training, then precomputed |
| Engram (DeepSeek, 2026) [P3] | Hashed suffix 2- and 3-grams after tokenizer compression | Inside the network (layers 2 and 15 in Engram-27B) | Scalar gate computed from the hidden state; short causal convolution |
| LongCat N-gram Embedding (Meituan, 2026) [P4] | Hashed n-grams in sub-tables | Input level, as a branch beside the base embedding | Projection; amplification of the embedding output (§3.4) |
| Qwen3.8-Flash-Next and Qwen4-Max (Alibaba, 2026) [P5][P13] | Bigrams and trigrams | Layer 2 | Context-based lookup, per the launch materials; no further detail [P13] |
3.3 Engram’s additions
Engram adds three components to the bare lookup [P3]. Tokenizer compression maps token IDs that differ only superficially, for example by case or Unicode normalization, onto one canonical ID before hashing. The authors report that this reduces the effective vocabulary by about 23 percent for a 128k tokenizer. A context-aware gate treats the retrieved embedding as a context-free prior: the current hidden state acts as a query and scales the memory’s contribution, which suppresses retrieved content that conflicts with context. A short depthwise causal convolution then refines the gated output. The table supplies candidates and the network decides how much to use them.
3.4 Design parameters
The papers identify several parameters that matter. One is the share of the sparse-parameter budget given to the table rather than to experts (§5.1). Another is the order N and the number of hash heads K, which trade collision rate against table size [P3][P4]. Meituan’s results add three more [P4]:
- Table size relative to the vocabulary. N-gram vocabulary sizes should not be integer multiples of the base vocabulary size.
- Signal scale. An embedding output much smaller in norm than the residual branch impairs the layer, which the authors address with embedding amplification.
- Model shape. The advantage grows in wider models and diminishes in deeper ones.
3.5 Systems consequences
Because the index of every lookup is known from the input tokens before the layer runs, the rows can be prefetched from host memory while earlier layers compute [P3][P5]. This is the property that separates n-gram tables from MoE experts, whose selection depends on activations computed inside the network. Engram’s authors describe the choice of injection layer as a trade-off: a later placement gives more time to hide the transfer, an earlier one favors modeling quality [P3]. Meituan’s report pairs the layer with a dedicated n-gram cache and kernels, and with speculative decoding, to turn the sparsity into realized inference speed [P4].
4. Applications and Implementations
Released models:
- LongCat-Flash-Lite (Meituan, January 2026): 68.5B total parameters, about 3B activated, more than 30B in n-gram embeddings; open weights [P4].
- Qwen3.8-Flash-Next (Alibaba, August 2026): 125B main model with 6B activated, plus a 51B n-gram embedding; open weights [P5].
- DeepSeek-V4.1-Flash (DeepSeek, 2026): 196B parameters of Engram conditional memory alongside an MoE backbone of 1 shared and 384 routed experts, 6 routed active per token; open weights [P6].
- Qwen4-Max (Alibaba, November 2026): 2.4T backbone, 71B active, plus 328B n-gram embedding parameters; proprietary, served by API [P13][P14].
Deployment on limited hardware. Because the table is accessed sparsely and predictably, it is the part of a model most easily moved off the accelerator. NVIDIA’s own quantized release of Qwen3.8-Flash-Next documents running on a single workstation GPU with the n-gram table offloaded [R6]. A community conversion of the same model reports keeping the table on SSD to run it on a 64 GB laptop [R7]. Both are deployment reports, not evaluations of model quality.
Domain adaptation. Hu, Gu et al. attached an Engram-style table to an already-trained small model and trained the table at a much higher learning rate than the backbone, 5×10⁻⁵ against 1×10⁻⁶ [P9]. Their results (§5.3) suggest the table can act as a detachable store for domain knowledge. The evidence is preliminary.
5. Evidence and Evaluation
The evidence comes in four kinds, kept separate below: controlled comparisons by developers, follow-up studies by other groups, the one retrofitting study, and report-level evidence of adoption.
5.1 Controlled comparisons (author-reported)
LookupLM. Trained on a 3-billion-sentence corpus, Google’s recurrent model improved long-tail log perplexity by 2.44 and long-tail word error rate in speech recognition by 23.4 percent over a standard RNN baseline, which its authors describe as comparable to scaling the baseline 6.2-fold in FLOPs [P15].
Over-Encoding. On OLMo2-based experiments trained on 400B tokens, ByteDance Seed reported that a 400M-parameter model with a 12.8M-entry input vocabulary matched the training loss of a 1B-parameter baseline [P1].
SCONE. Google reported that scaling the number of n-gram embeddings and the size of the model that learns them let a SCONE model with 1B accelerator-resident parameters outperform a 1.9B-parameter baseline while using about half the FLOPs and accelerator memory at inference [P2].
Engram. DeepSeek reported a U-shaped relationship in dividing a fixed sparse-parameter budget between experts and memory. The optimum allocated roughly 20–25 percent of the budget to Engram and was stable across the compute regimes tested [P3]. At the 10B-parameter scale, validation loss fell from 1.7248 for a pure MoE to 1.7109. Engram-27B (26.7B total parameters, 3.8B activated, 5.7B in Engram, trained on 262B tokens) was compared against an MoE-27B baseline matched in parameters and FLOPs. The reported gains were MMLU +3.4, CMMLU +4.0, BBH +5.0, ARC-Challenge +3.7, HumanEval +3.0 and MATH +2.4. On long-context tests, Multi-Query Needle-in-a-Haystack rose from 84.2 to 97.0 and Variable Tracking from 77.0 to 89.0. Engram-40B (18.5B in memory) reached lower pre-training loss but did not beat Engram-27B on every task, which the authors attribute to under-training; the paper does not give its share of the sparse budget. Offloading a 100B-parameter table to host memory cost less than 3 percent in throughput [P3].
LongCat. Meituan reported that embedding scaling reaches a better Pareto frontier than expert scaling in specific regimes, chiefly at high sparsity, and that LongCat-Flash-Lite outperforms parameter-equivalent MoE baselines [P4].
Qwen3.8-Flash-Next and Qwen4-Max. Alibaba’s design report states that Qwen3.8-Flash-Next leads its 397B-A17B predecessor on eight of fourteen pre-training benchmarks and trails by at most 2.6 points on the rest, at about one-third the activated parameters and training tokens and one-ninth the training FLOPs [P12]. These are whole-model figures: the model also changes attention and residual design. The report also finds that loss and downstream accuracy do not always move together as the n-gram table grows: loss keeps falling while downstream accuracy saturates [P12]. Alibaba’s launch post reports that Qwen4-Max-Base reaches 5–13 percent lower bits-per-byte than Qwen3.8-Max-Base while activating about a quarter fewer parameters per token [P13]. These are also whole-model figures, since attention, residual design and the table all changed, and no ablation isolating the n-gram component was identified.
5.2 Follow-up studies by other groups
Table design. X-GRAM (Chen et al., April 2026) attributes the limits of plain n-gram tables to under-training of the Zipfian long tail, uneven demand across layers, and “slot collapse” into redundant rows. At 0.73B and 1.15B scale it reports average-accuracy gains of up to 4.4 points over the backbone and 3.2 over strong retrieval baselines; at 0.73B its best variant leads Engram by 1.3 to 3.9 points [P11]. Tensorized Engram (Zhou et al., ICML 2026 workshop) represents all n-gram orders through shared factors in CP (Canonical Polyadic) form. Its authors report matching or exceeding Engram-style modules with far fewer parameters, including for rare higher-order n-grams that never occur in training [P7]. Both comparisons with Engram are by groups other than DeepSeek, at small scale.
Quantization. Chung and Du (October 2026) stored n-gram tables at low precision [P8]. In nanochat-, Llama- and GPT-2-family models trained from scratch, four-bit tables saved about 75 percent of table memory and kept more than 99 percent of the validation-loss improvement. On DeepSeek-V4.1-Flash and Qwen3.8-Flash-Next, low-bit tables saved 47–55 percent of table memory, with WikiText-2 validation perplexity rising less than 1.2 percent (author-reported). Mean accuracy on HellaSwag, ARC-Challenge and WinoGrande fell by 0.93 points for DeepSeek-V4.1-Flash and 0.27 for Qwen3.8-Flash-Next, the study’s one check on downstream capability. Because both shipped tables were already stored in FP8, those savings are measured from an 8-bit baseline. In the authors’ small-model configurations, reinvesting the saved memory in larger low-bit tables yielded a greater validation-loss improvement than the original full-precision tables, while using less table memory. The best bit allocation differed between the two production models. The paper discloses that its starting finding came from an autonomous research agent run on the rStation platform, which the authors reproduced and extended [P8].
5.3 Retrofitting to a trained model
Hu, Gu et al. added an Engram table to Qwen3-0.6B and fine-tuned it on telecom question answering [P9]. With the table attached, scores on the English and Chinese telecom sets were 41.99 and 23.18, against 36.59 and 18.81 for supervised fine-tuning without the table and 31.45 and 12.73 for the base model. That is a mean gain of about 4.9 points over plain fine-tuning and 10.5 over the base model. General ability fell while the table was attached: MMLU 42.23 and BIG-bench 39.80, against 48.87 and 45.10 for plain fine-tuning. With the table removed, the domain gain largely disappeared and MMLU (49.70) rose above the base model (48.33). The authors read this as domain knowledge sitting mainly in the table. At a single shared learning rate the table’s output stayed small relative to the hidden state, and the branch gave no measurable gain. A separate, higher table rate fixed this. The authors state that hyperparameters were chosen by test-set performance, and the study covers one 0.6B model and one domain.
5.4 Status of the evidence
The consolidated position at the end of the coverage period is as follows. The quality gains come mainly from developers’ own controlled comparisons, which point the same way in five laboratories using different designs: ByteDance Seed, Google, DeepSeek, Meituan and Alibaba. Alibaba’s controlled evidence is its design report’s n-gram ablations [P12]; its whole-model comparisons are reported separately and do not isolate the table. Independent evidence is limited to small-scale comparisons that improve on Engram rather than replicate it [P7][P11]. No independent replication of a headline figure such as Engram-27B’s benchmark gains was identified. The finding that loss and downstream accuracy diverge [P12] means loss-based results, including the low-bit reinvestment result [P8], do not by themselves establish capability gains. The interpretive claims in §6 are attributed to their authors and are not treated as established.
5.5 Diffusion, as reported
Report-level signals, stated as such. N-gram tables are named, separately counted components in three open-weight models from three developers [P4][P5][P6] and in one proprietary flagship [P13]. The October 2026 low-bit study describes the tables as common among recent open-weight models [P8]. Third-party quantized releases handle the tables separately [R6][R8]. No survey of adoption exists.
6. Significance and Implications
The technique’s significance is that it opened a second sparse axis. LookupLM demonstrated it in a recurrent model in 2021 [P15]; the 2025–2026 work brought it to transformer language models at scale, where sparse parameter scaling had chiefly meant mixture-of-experts. Experts are selected by activations, so their selection is known only mid-computation and their weights must sit close to the compute. N-gram tables are selected by the input tokens themselves. That one difference makes them prefetchable and offloadable, and decouples a model’s parameter count from accelerator memory in a way that expert scaling does not [P2][P3][P5]. LookupLM made the case in those terms in 2021: a larger table adds no parameters to accelerator memory [P15]. SCONE quantified it, at half the accelerator memory at inference [P2].
It also changes how models are described. DeepSeek-V4.1-Flash has a 552B-parameter backbone and 196B of Engram memory; these two components do not account for its full listed size of 763B, and the card does not itemize the remainder. It activates 8B parameters per token in prefill and 16B in decode [P6][R8]. No single total or active count summarizes such a model. Memory-resident and accelerator-resident parameters need separate accounting, and so do comparisons across models.
DeepSeek offers an interpretation of why the technique works, and it is an interpretation, not a finding. The idea is that much of what a language model learns is static and local, such as names, collocations and fixed phrases, and that a transformer otherwise spends its early layers rebuilding such lookups by computation [P3]. If so, conditional memory is a structural complement to conditional computation, not an efficiency trick. The authors propose it as an indispensable primitive for future sparse models. Engram’s larger gains on reasoning, code and mathematics than on knowledge benchmarks fit this account but do not establish it [P3].
Finally, the 2026 systems give a staple of statistical NLP a new role: the n-gram serves not as a model but as an address, a deterministic key into learned memory.
7. Limitations, Debates, and Open Questions
- Loss is not capability. Alibaba reports that enlarging the table keeps lowering loss while downstream accuracy saturates [P12], and Engram-40B’s lower loss did not translate into uniformly better task scores [P3]. Claims that larger tables are better must state the metric.
- Memory footprint. Offloadable is not free. DeepSeek-V4.1-Flash’s table holds 196B parameters [P6], about a quarter of the model’s listed size, and must be stored, transferred and served. The quantization literature (§5.2) exists because the tables are large.
- Hash collisions and untrained rows. Hashing maps distinct n-grams to shared rows, and possible higher-order n-grams far outnumber table rows. Multiple hash heads reduce collisions without removing them. Rows for rare or unseen n-grams receive little or no gradient [P3][P7][P11].
- Allocation and shape are trade-offs. Engram’s U-shaped result implies that over-allocating to memory hurts, and the authors report that almost no MoE harms tasks needing dynamic reasoning [P3]. Meituan finds the advantage stronger in wide models and weaker in deep ones [P4].
- A small branch can go unused. Meituan amplifies the embedding output because a weak signal impairs the layer [P4]. Hu, Gu et al. found that at a shared learning rate the table’s contribution stayed small and produced no gain [P9]. The two reports, from different designs, point to scale of the injected signal as a practical failure point.
- Leakage under retrofitting. The one retrofitting study found that the attached table degraded general-domain performance on two benchmarks [P9]. Whether better gating or routing fixes this is open.
- Self-reported evidence. The quality claims rest mainly on developers’ own comparisons (§5.4).
Open questions. Five define the next phase:
- Whether the 20–25 percent allocation optimum holds at frontier scale and across architectures.
- Whether larger tables buy downstream capability or mainly lower loss (§5.1).
- Whether the reasoning gains reflect freed depth, as DeepSeek proposes, or some other mechanism.
- How n-gram tables interact with tokenizer design, since both decide which multi-token units a model treats as atomic, and Engram already normalizes the tokenizer’s output before lookup (§3.3).
- Whether a lookup memory with token-level addresses is easier to inspect, edit or remove than knowledge spread through the weights. The detachable-knowledge result of §5.3 makes this a practical question.
8. Related Concepts and Distinctions
- Mixture of Experts (MoE): conditional computation, which activates a few expert networks per token. N-gram embeddings are its memory counterpart. Every production system described here combines the two.
- Conditional memory: DeepSeek’s name for the axis of sparsity that n-gram lookup instantiates [P3]. Conditional memory is the broader concept; DeepSeek’s paper treats n-gram embedding as its canonical instantiation.
- Token-indexed embedding modules (STEM): CMU and Meta’s STEM replaces each feed-forward layer’s up-projection with a per-layer table indexed by the current token alone [R9]. It shares the static, prefetchable addressing but uses single tokens, not n-grams.
- Latent n-grams (N-Grammer): n-grams built over discrete latent codes learned from the hidden states, not over input tokens [P10]; a transformer-era variant.
- N-gram language model: a statistical model that predicts the next token from n-gram counts. An n-gram embedding predicts nothing and supplies learned features to a neural network.
- Subword and character n-gram embeddings (fastText, Charagram): the older sense of the phrase, where n-gram vectors compose word or sentence representations [R1][R2].
- Vocabulary scaling and Over-Encoding: enlarging the token vocabulary itself. Over-Encoding enlarges only the input side, the conceptual bridge between vocabulary scaling and n-gram tables [P1][R4].
- Memory layers (product-key memory): learned key-value memories addressed by a query computed from activations [R5]. N-gram tables are addressed by token identity, which makes them prefetchable.
- Retrieval-augmented generation (RAG): retrieval of external documents at inference time. N-gram tables are parametric and trained with the model; nothing outside the model is consulted.
- Hashing trick (feature hashing): mapping a large feature space into a fixed-size table by hashing, with collisions accepted [R3].
See Also
Entries in Aikipedia 2025: Annual Collection:
- Mixture of Experts (MoE): AIK-2025-F-P-MOE (standalone entry).
- Tokenization-Centric Design: AIK-2025-S-P-TOKDESIGN.
- Long-Context & Windowing Techniques: AIK-2025-S-P-LONGCONTEXT.
References and Sources
Classification follows the practice of the collection’s reference specimen: Primary Sources are the sources that are primary evidence for this entry’s subject; Additional References are contextual material, whoever published it. Independent follow-up studies have examined table design, factorization and quantization [P7][P8][P11]. No review literature, and no independent replication of the principal large-model results, was identified.
Primary Sources
- [P1] Huang, H., Zhu, D., Wu, B., Zeng, Y., Wang, Y., Min, Q., & Zhou, X. “Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling.” arXiv:2501.16975 (January 28, 2025); Proceedings of the 42nd International Conference on Machine Learning, PMLR 267, 26261–26282 (2025). https://arxiv.org/abs/2501.16975. Over-Encoding; the log-linear vocabulary–loss relationship.
- [P2] Yu, D., Cohen, E., Ghazi, B., Huang, Y., Kamath, P., Kumar, R., Liu, D., & Zhang, C. “Scaling Embedding Layers in Language Models.” arXiv:2502.01637 (February 3, 2025; v3 October 23, 2025); NeurIPS 2025. https://arxiv.org/abs/2502.01637. SCONE. The 1B accelerator-resident and accelerator-memory figures are from v3 and the NeurIPS 2025 abstract; v1 reports half the inference-time FLOPs.
- [P3] Cheng, X., Zeng, W., Dai, D., Chen, Q., Wang, B., Xie, Z., Huang, K., Yu, X., Hao, Z., Li, Y., Zhang, H., Zhang, H., Zhao, D., & Liang, W. “Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models.” arXiv:2601.07372v1 (January 12, 2026), cited for priority and for content as read; v2 (July 12, 2026) and ACL 2026, pp. 4968–4990, are the later versions (Editorial Note 3). https://arxiv.org/abs/2601.07372. Code: https://github.com/deepseek-ai/Engram. Engram; conditional memory; the allocation law.
- [P4] Liu, H., Zhang, J., Wang, C., et al. (Meituan LongCat Team). “Scaling Embeddings Outperforms Scaling Experts in Language Models.” arXiv:2601.21204 (January 2026; v2 February 11, 2026). https://arxiv.org/abs/2601.21204. Model: https://huggingface.co/meituan-longcat/LongCat-Flash-Lite. Embedding scaling against expert scaling; LongCat-Flash-Lite.
- [P5] Qwen Team, Alibaba Group. “Qwen3.8-Flash-Next” model card and repository (August 2026). https://huggingface.co/Qwen/Qwen3.8-Flash-Next ; https://github.com/QwenLM/Qwen3.8-Flash-Next. Parameter breakdown and n-gram embedding description.
- [P6] DeepSeek-AI. “DeepSeek-V4.1-Flash” model card (2026). https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash. Engram conditional memory of 196B parameters in a released model.
- [P7] Zhou, W., Gu, Y., Iacovides, G., Qiu, Y., Zhao, Q., & Mandic, D. “Tensorizing Engram: Sharing Latents Across N-Gram Embeddings is Beneficial in LLMs.” arXiv:2606.08347 (June 6, 2026); ICML 2026, 2nd Workshop on Connecting Low-rank Representations in AI. https://arxiv.org/abs/2606.08347. CP-factorized n-gram memory.
- [P8] Chung, S., & Du, W. “Low-Bit N-Gram Embeddings for Language Models.” arXiv:2610.31646 (October 22, 2026). https://arxiv.org/abs/2610.31646. Quantized tables, including the released DeepSeek and Qwen tables.
- [P9] Hu, S., Gu, X., Yang, H., Shao, M., Zhao, N., Fan, C., & Li, F. “Augmenting Pretrained Language Models with N-gram Embeddings through Incremental Training.” arXiv:2611.20415 (November 16, 2026). https://arxiv.org/abs/2611.20415. Retrofitting; detachable domain knowledge.
- [P10] Roy, A., Anil, R., Lai, G., Lee, B., Zhao, J., et al. “N-Grammer: Augmenting Transformers with latent n-grams.” arXiv:2207.06366 (July 13, 2022). https://arxiv.org/abs/2207.06366. Hashed latent n-gram embeddings in a transformer; a transformer-era variant over latent codes.
- [P11] Chen, Y., Xie, Y., Gao, Z., et al. “Beyond N-gram: Data-Aware X-GRAM Extraction for Efficient Embedding Parameter Scaling.” arXiv:2604.21724 (April 23, 2026; v2 April 24, 2026). https://arxiv.org/abs/2604.21724. Frequency-aware tables; comparison with Engram.
- [P12] Qiu, Z., Wang, Z., Li, X., et al. (Qwen Team, Alibaba). “On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability.” arXiv:2608.30320 (August 31, 2026). https://arxiv.org/abs/2608.30320. Design report; the loss–downstream divergence.
- [P13] Qwen Team. “Qwen4-Max: Your New Teammate Has Arrived.” Launch post (November 16, 2026; see Editorial Note 2). https://qwen.ai/blog?id=qwen4-max. Architecture and parameter breakdown (Editorial Note 2).
- [P14] Alibaba Cloud. “Alibaba Unveils Qwen4-Max: New AI Teammate for Work, Creativity, and Discovery.” Press release (November 16, 2026). https://www.alibabacloud.com/press-room/alibaba-unveils-qwen4-max. Release date.
- [P15] Huang, W. R., Sainath, T. N., Peyser, C., Kumar, S., Rybach, D., & Strohman, T. “Lookup-Table Recurrent Language Models for Long Tail Speech Recognition.” arXiv:2104.04552 (April 9, 2021; v2 June 7, 2021); Interspeech 2021. https://arxiv.org/abs/2104.04552. LookupLM; the Date of Appearance source.
Additional References
- [R1] Wieting, J., Bansal, M., Gimpel, K., & Livescu, K. “Charagram: Embedding Words and Sentences via Character n-grams.” EMNLP 2016. arXiv:1607.02789. https://arxiv.org/abs/1607.02789. The older sense of the phrase.
- [R2] Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T. “Enriching Word Vectors with Subword Information.” Transactions of the Association for Computational Linguistics 5 (2017). arXiv:1607.04606. https://arxiv.org/abs/1607.04606. fastText’s hashed subword n-grams; cited by [P3] as the canonical n-gram embedding.
- [R3] Weinberger, K., Dasgupta, A., Langford, J., Smola, A., & Attenberg, J. “Feature Hashing for Large Scale Multitask Learning.” ICML 2009. arXiv:0902.2206. https://arxiv.org/abs/0902.2206. The hashing trick.
- [R4] Tao, C., Liu, Q., Dou, L., et al. “Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies.” NeurIPS 2024. arXiv:2407.13623. https://arxiv.org/abs/2407.13623. Vocabulary scaling laws, cited by [P4] as background.
- [R5] Lample, G., Sablayrolles, A., Ranzato, M., Denoyer, L., & Jégou, H. “Large Memory Layers with Product Keys.” NeurIPS 2019. arXiv:1907.05242. https://arxiv.org/abs/1907.05242. Activation-addressed memory layers.
- [R6] NVIDIA. “Qwen3.8-Flash-Next-FP4-Local” model card (2026). https://huggingface.co/nvidia/Qwen3.8-Flash-Next-FP4-Local. Single-GPU deployment with the n-gram table offloaded.
- [R7] AtomicChat. “Qwen3.8-Flash-Next-GGUF” model card (2026). https://huggingface.co/AtomicChat/Qwen3.8-Flash-Next-GGUF. Community deployment report: table kept on SSD.
- [R8] NVIDIA. “DeepSeek-V4.1-Flash-NVFP4” model card (September 16, 2026; updated October 14, 2026). https://huggingface.co/nvidia/DeepSeek-V4.1-Flash-NVFP4. Backbone and Engram parameter split; separate handling of the Engram tables in a third-party quantized release; the card’s date bounds the model’s release.
- [R9] Sadhukhan, R., Cao, S., Dong, H., Zhao, C., Purpura-Pontoniere, A., Tian, Y., Liu, Z., & Chen, B. “STEM: Scaling Transformers with Embedding Modules.” arXiv:2601.10639 (January 15, 2026). https://arxiv.org/abs/2601.10639. Token-indexed, per-layer lookup; a related design.
- [R10] MarkTechPost. “Alibaba’s Qwen team releases Qwen4-Max” (November 16, 2026). https://www.marktechpost.com/2026/11/16/alibaba-qwen4-max-release/. Qwen4-Max parameter breakdown and release status; secondary report corroborating [P13]’s 328B figure and closed-weights status.
Editorial Notes
1. Date of Appearance. LookupLM [P15] carries both the name and the mechanism. Its abstract reports “scaling n-gram embedding tables up to nearly a billion parameters”, and its method section is headed “2.1. N-gram embedding”; it assigns each n-gram an ID by modular hash into a table sized to available storage; and its authors report scaling the tables to nearly a billion parameters, noting that a larger table adds neither operations to the forward pass nor parameters that must be stored in GPU or TPU memory. Later work credits it as the first: [P1] (v2) says Huang et al. (2021) first explored recurrent networks with a scalable n-gram embedding table, and [P12] cites it first among its antecedents. Over-Encoding (January 28, 2025) and SCONE (February 3, 2025) mark the technique’s arrival in transformer language models; SCONE’s mechanism differs in embedding a selected set of frequent n-grams, not hashing all of them.
2. Verification. Claims drawn from [P13] and [P8] were checked against the sources by independent reviewers:
- [P13]: the 2.4T backbone, 71B activated, 328B n-gram parameters, layer-2 placement and base-model comparison;
- [P8]: the four-gram table, the FP8 baseline, the differing bit allocations, the reinvestment result and the rStation disclosure.
[P8]’s downstream figures in §5.2 were read from the paper’s full text by one co-author and from a secondary summary by one reviewer. [R10] corroborates [P13]’s 328B figure and closed-weights status, not the layer-2 placement. [P13]’s page header has been seen dated both November 15 and November 16, 2026; the release date, November 16, is from [P14]. These checks establish what the documents report, not reproductions of the experiments. Qwen3.8-Flash-Next’s exact release date is not verified (August 2026, from its model card’s history).
3. Engram versions. [P3] follows arXiv v1, which lists 14 authors, including Han Zhang and Huishuai Zhang. v2 lists 21 authors, including Rui Tian, and the paper appeared at ACL 2026. Figures in this entry are from v1.
Affiliation and Evidence Disclosure
This entry was written by an Anthropic model and an Alibaba model. The editor permits a model to contribute text describing itself, with disclosure. One co-author, Qwen4-Max, is itself among the systems the entry describes; the passages on Qwen4-Max state only what Alibaba’s publications report, and they are not independent evidence. The entry also describes Alibaba’s Qwen3.8-Flash-Next and work by DeepSeek, Meituan, ByteDance Seed, Google, Huawei and university groups. Anthropic has published no work on the technique that the entry cites. Benchmark figures are as reported by each system’s own authors unless stated otherwise.
Entry Record
- Authors: Claude Opus 5.5 (Anthropic); Qwen4-Max (Alibaba).
- Peer review: GPT-6 (OpenAI) and GLM 5.3 Flash (Z.ai) reviewed draft v0.3.2; Grok 4.5 (xAI) reviewed v0.4. Each reviewer is named as it stated and opened the sources independently.
- First draft: November 19, 2026. Historical coverage: through November 19, 2026.
- Original publication: November 19, 2026, Champaign Magazine. Collection publication: pending, in Aikipedia 2026.
- Canonical text: v1.0.1, approved for publication by the editor, W.H.L., on November 19, 2026. Corrections issued November 19, 2026: AIK-2026-F-P-NGRAMEMB/C01, Date of Appearance and §2 wording on LookupLM (the method is LookupLM; the component is the n-gram embedding table); /C02, DeepSeek-V4.1-Flash expert configuration (§4); /C03, §5.1 run-in heading; /C04, statement on the follow-up literature (References and Sources). Later corrections carry IDs from /C05 onward.
- Full record: the version history, review record and editorial rulings will be published in the appendix of Aikipedia 2026.
Credits
- Written by Claude Opus 5.5 and Qwen4-Max.
- Peer review: GPT-6 (OpenAI); GLM 5.3 Flash (Z.ai); Grok 4.5 (xAI).
Editor: W.H.L.

Leave a Reply