NANJING · TSINGHUA / RESEARCH PROJECT
NGM.

A Plug-and-Play Training-Free
Memory Module for LLMs

The memory is already there.
Reuse pretrained token embeddings as local N-gram memory.

Yuwen Qu1,*Wenhui Dong1,*Camil Hamami2Ming Zhao1Chenyang Si1Caifeng Shan1,†

1 Nanjing University · 2 Tsinghua University * Equal contribution · † Corresponding author

Read the manuscriptarXiv BibTeX
0 added trainable parametersFrozen backbone weightsNo external memory bank
01 / THE IDEA

A new role for
existing embeddings.

Can a pretrained embedding space serve as local memory without retraining or a separate memory table?

NGM builds causal, multi-scale N-gram representations directly from the backbone’s token embeddings. A cosine gate then selects positively aligned signals and writes them into the decoder’s residual stream.

The result is a two-stage memory mechanism that leaves the model’s weights frozen and introduces no learned memory representations.

02 / HOW IT WORKS

Construct. Gate. Inject.

Two parameter-free components.
One pretrained representation space.

Comparison of a standard Transformer, DeepSeek Engram with learned N-gram embeddings, and NGM reusing frozen token embeddings through average pooling and cosine gating.
Figure 2 from the manuscript. NGM reuses the vocabulary embedding space to expose local-memory signals. View full resolution ↗
01

Causal N-gram encoder

Average pretrained embeddings over trailing windows. The default 2-gram and 3-gram windows expose local patterns at different scales.

gt,n = 1/n ∑k=0n−1 E[xt−k]
02

Cosine-gated injector

Measure compatibility with the decoder state. ReLU suppresses negative alignment; a fixed scale controls the residual write.

h′t = ht + λ ∑n ReLU(cos(ht, gt,n)) gt,n
WHY THIS CAN WORK

Local structure remains
in the embedding space.

In Qwen3-8B, actual N-gram representations align more strongly with decoder hidden states than position-shuffled and random-token controls. This motivates a geometric gate without a learned projection.

Explore the analysis in the paper ↗
Original paper plot showing actual N-gram cosine alignment above position-shuffled and random-token controls across Qwen3-8B layers.
Original alignment study · Qwen3-8B
03 / EXPERIMENTS

Same weights.
Additional memory signal.

Six backbones. Ten benchmarks.
Two language-model families.

+4.26pp

HumanEval

Qwen3-1.7B · pass@1

+6.56pp

GPQA-Diamond

Qwen3-1.7B · accuracy

6/ 6

Higher task average

+0.98 to +1.72 percentage points

Explore the reported results

Choose a backbone to compare its baseline with NGM.

Qwen3-1.7BBaseline + NGM
View scores and run-to-run variation
Main results
BenchmarkBaselineNGM mean ± SDΔ (pp)

Source: manuscript Table 1. Scores are percentages; NGM reports mean ± sample standard deviation over three runs. Δ is the paper’s reported difference in percentage points. Average is unweighted across all ten tasks, calculated before rounding. TruthfulQA here uses the main table’s generative evaluation protocol.

BEYOND TEXT

Transfer to a vision-language model.

Preliminary Qwen3-VL-2B-Instruct experiments apply NGM to the language decoder while retaining the visual encoder. On MMStar, the score rises from 54.67 to 56.20.

MMStar+1.53pp

54.67 → 56.20

04 / PAPER & CITATION

Build on this work.

Read the updated manuscript for the full methodology, evaluation setup, ablations, and implementation details.

Updated manuscript PDF ↗arXiv record 2605.16893 ↗Download citation .bib ↓

This page presents the updated manuscript. The arXiv record currently hosts the earlier release.

BibTeX
@article{qu2026ngm,
  title = {NGM: A Plug-and-Play Training-Free
           Memory Module for LLMs},
  author = {Qu, Yuwen and Dong, Wenhui and Hamami, Camil and
            Zhao, Ming and Si, Chenyang and Shan, Caifeng},
  journal = {arXiv preprint arXiv:2605.16893},
  year = {2026},
  doi = {10.48550/arXiv.2605.16893}
}