The five question sets of the HippoRAG 2 evaluation, with its corpora and protocol: NQ, PopQA, MuSiQue, 2Wiki and HotpotQA, 1,000 questions each. Every system uses the same embedding model, reranker, reader and judge, so the only difference is how it finds the passages, and whether it learns.
LeanMemory's first answer to every question, before any learning. Conventional RAG 58.5, state-of-the-art RAG 62.2: the same models and the same questions.
Two study rounds over the questions it was asked, and nothing else: no answers, labels or gold passages. The second round added 0.4 points; the first, 9.3.
Reference answers for the questions it still missed, given the way a person would teach it. 85.7 on rewordings it had never seen; the two RAG systems stay at their first answers.
Studying the questions it was asked lifts LeanMemory most on the multi-hop sets, where an answer has to be assembled from several passages. Teaching lifts every set.
The average of answer accuracy and recall at 5, in per cent. Each LeanMemory bar starts at its first answer to every question and adds what it gained from studying the questions it was asked and from being taught the answers it still missed. Conventional and state-of-the-art RAG do not learn, so their bars are their first answers.
| Question set | Conventional RAG | SOTA RAG | First sight | Study 1 | Study 2 | Taught |
|---|---|---|---|---|---|---|
| NQ simple | 63.2 | 65.2 | 64.8 | 69.2 | 69.2 | 84.3 |
| PopQA simple | 58.2 | 58.0 | 57.9 | 61.6 | 61.7 | 76.8 |
| MuSiQue multi-hop | 35.3 | 42.9 | 43.5 | 54.5 | 55.3 | 83.6 |
| 2Wiki multi-hop | 59.0 | 59.6 | 59.9 | 81.9 | 82.7 | 94.4 |
| HotpotQA multi-hop | 76.9 | 85.1 | 86.0 | 91.6 | 91.6 | 98.4 |
| Average | 58.5 | 62.2 | 62.4 | 71.7 | 72.1 | 87.5 |
In per cent. SOTA RAG is the state-of-the-art recipe. The last four columns are LeanMemory: its first answer, after one and two study rounds, and after teaching. Neither RAG system learns, so each has one figure, its first answer.
Accuracy asks whether the answer is right. Recall at 5 asks whether the gold passages are among the five the reader gets.
Teaching takes accuracy to 98.4 on average because a taught answer travels with the passages that support it, so the reader has what it needs on the page. Recall at 5 rises less, from 67.7 to 76.6, since it counts only the five passages and never LeanMemory's own notes.
The answer is correct, graded strictly against every accepted answer.
| Question set | Conventional RAG | SOTA RAG | First sight | Study 1 | Study 2 | Taught |
|---|---|---|---|---|---|---|
| NQ simple | 64.8 | 66.2 | 65.8 | 69.9 | 69.8 | 97.3 |
| PopQA simple | 66.3 | 66.6 | 66.2 | 69.9 | 69.9 | 99.4 |
| MuSiQue multi-hop | 19.9 | 26.2 | 27.3 | 44.5 | 45.7 | 97.1 |
| 2Wiki multi-hop | 46.8 | 47.4 | 47.3 | 76.6 | 77.7 | 99.0 |
| HotpotQA multi-hop | 68.8 | 77.8 | 78.9 | 86.4 | 86.4 | 99.3 |
| Average | 53.3 | 56.8 | 57.1 | 69.5 | 69.9 | 98.4 |
In per cent. SOTA RAG is the state-of-the-art recipe. The last four columns are LeanMemory: its first answer, after one and two study rounds, and after teaching. Neither RAG system learns, so each has one figure, its first answer.
The share of the gold passages among the five passages the reader gets.
| Question set | Conventional RAG | SOTA RAG | First sight | Study 1 | Study 2 | Taught |
|---|---|---|---|---|---|---|
| NQ simple | 61.6 | 64.2 | 63.9 | 68.5 | 68.5 | 71.3 |
| PopQA simple | 50.1 | 49.5 | 49.5 | 53.3 | 53.4 | 54.2 |
| MuSiQue multi-hop | 50.6 | 59.6 | 59.6 | 64.3 | 64.7 | 70.1 |
| 2Wiki multi-hop | 71.3 | 71.8 | 72.4 | 87.1 | 87.7 | 89.7 |
| HotpotQA multi-hop | 85.0 | 92.4 | 93.2 | 96.8 | 96.8 | 97.6 |
| Average | 63.7 | 67.5 | 67.7 | 74.0 | 74.2 | 76.6 |
In per cent. SOTA RAG is the state-of-the-art recipe. The last four columns are LeanMemory: its first answer, after one and two study rounds, and after teaching. Neither RAG system learns, so each has one figure, its first answer.
Time to find the passages for one question, measured on the first 100 questions of each set, asked one at a time, each system on its own fresh engine with nothing cached. It includes embedding the question and the reranker, both on a single consumer GPU, an NVIDIA RTX 5070 Ti. Writing the answer takes the same time for every system and is not included.
LeanMemory finds the passages in 453 ms at the median; the state-of-the-art recipe takes 1.6 s. Most of that is the model call that splits the question into sub-questions, which also varies with the provider's load (1.2 to 1.6 s median across our runs, while the other two systems moved by under 20 ms). LeanMemory read each document once when it was added, so it needs no model at query time. Plain vector search is faster still, at 26 ms, and scores lowest.
Milliseconds. The first 100 questions of each set, asked one at a time, each system on its own fresh engine with nothing cached.
| Question set | Conventional RAG | SOTA RAG | LeanMemory |
|---|---|---|---|
| NQ | 26 / 29 | 1,359 / 2,492 | 255 / 308 |
| PopQA | 25 / 26 | 1,608 / 3,389 | 567 / 701 |
| MuSiQue | 28 / 31 | 1,826 / 3,317 | 407 / 506 |
| 2Wiki | 23 / 27 | 1,607 / 2,590 | 664 / 772 |
| HotpotQA | 27 / 30 | 1,585 / 2,675 | 371 / 514 |
| Average | 26 / 29 | 1,597 / 2,893 | 453 / 560 |
Milliseconds per question: median / 90th percentile. SOTA RAG is the state-of-the-art recipe.
After learning, LeanMemory's page also carries its own notes: the answers it verified or was taught, with the passages they cite. A one-passage page therefore still holds what it has learnt. Before learning there are no notes, and the tables below give that starting point beside the two RAG systems.
After studying the questions it was asked, LeanMemory's one-passage page answers 61.8 of them; the state-of-the-art five-passage page answers 56.8, the conventional one 53.3. Learning adds gold passages to the page rather than re-ordering its first slot, so recall at 1 hardly moves while recall at 2 and 5 rises.
Correct answers when the reader is given only the first 1, 2 or 5 passages, graded by the same strict judge.
The share of the gold passages among the first 1, 2 and 5 passages found.
| Conventional RAG | State-of-the-art RAG | LeanMemory, first sight | After studying | After teaching | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Question set | k = 1 | k = 2 | k = 5 | k = 1 | k = 2 | k = 5 | k = 1 | k = 2 | k = 5 | k = 1 | k = 2 | k = 5 | k = 1 | k = 2 | k = 5 |
| NQ | 48.8 | 55.7 | 64.8 | 55.5 | 60.0 | 66.2 | 55.2 | 59.8 | 65.8 | 67.5 | 70.0 | 69.8 | 96.2 | 96.9 | 97.3 |
| PopQA | 58.3 | 62.8 | 66.3 | 64.2 | 64.7 | 66.6 | 64.3 | 64.5 | 66.2 | 70.3 | 70.2 | 69.9 | 98.9 | 98.8 | 99.4 |
| MuSiQue | 6.0 | 13.6 | 19.9 | 6.0 | 17.6 | 26.2 | 6.0 | 17.4 | 27.3 | 27.7 | 37.3 | 45.7 | 89.8 | 94.1 | 97.1 |
| 2Wiki | 4.0 | 32.6 | 46.8 | 4.5 | 33.7 | 47.4 | 4.5 | 33.4 | 47.3 | 66.9 | 73.7 | 77.7 | 88.6 | 95.4 | 99.0 |
| HotpotQA | 27.6 | 55.5 | 68.8 | 30.8 | 68.9 | 77.8 | 31.1 | 69.5 | 78.9 | 76.7 | 84.1 | 86.4 | 89.0 | 96.4 | 99.3 |
| Average | 28.9 | 44.0 | 53.3 | 32.2 | 49.0 | 56.8 | 32.2 | 48.9 | 57.1 | 61.8 | 67.1 | 69.9 | 92.5 | 96.3 | 98.4 |
Correct answers, in per cent, when the reader gets the first 1, 2 or 5 passages. Before learning, LeanMemory's page holds no notes, so its first-sight figures sit beside the two RAG systems.
| Conventional RAG | State-of-the-art RAG | LeanMemory, first sight | After studying | After teaching | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Question set | k = 1 | k = 2 | k = 5 | k = 1 | k = 2 | k = 5 | k = 1 | k = 2 | k = 5 | k = 1 | k = 2 | k = 5 | k = 1 | k = 2 | k = 5 |
| NQ | 21.6 | 36.0 | 61.6 | 24.3 | 39.7 | 64.2 | 24.1 | 39.7 | 63.9 | 27.5 | 44.0 | 68.5 | 29.6 | 48.4 | 71.3 |
| PopQA | 25.7 | 43.2 | 50.1 | 31.9 | 45.2 | 49.5 | 30.7 | 45.1 | 49.5 | 30.8 | 48.9 | 53.4 | 29.0 | 49.4 | 54.2 |
| MuSiQue | 27.8 | 38.7 | 50.6 | 32.1 | 46.4 | 59.6 | 32.1 | 46.2 | 59.6 | 31.4 | 49.6 | 64.7 | 32.2 | 53.9 | 70.1 |
| 2Wiki | 41.9 | 63.9 | 71.3 | 43.2 | 65.8 | 71.8 | 43.3 | 65.8 | 72.4 | 42.6 | 77.4 | 87.7 | 42.6 | 79.4 | 89.7 |
| HotpotQA | 44.0 | 71.3 | 85.0 | 47.4 | 82.8 | 92.4 | 47.5 | 83.1 | 93.2 | 46.5 | 86.6 | 96.8 | 46.9 | 87.9 | 97.6 |
| Average | 32.2 | 50.6 | 63.7 | 35.8 | 56.0 | 67.5 | 35.5 | 56.0 | 67.7 | 35.7 | 61.3 | 74.2 | 36.1 | 63.8 | 76.6 |
The share of the gold passages among the first 1, 2 or 5 passages, in per cent. The first-sight figures come from a retrieval-only run before any learning.
The published evaluation, run unchanged: the same question sets, corpora and reader setting as the paper, and the same models for every system.
NQ and PopQA (simple questions), MuSiQue, 2WikiMultihopQA and HotpotQA (multi-hop), as released for the HippoRAG 2 paper. Each question is asked against its set's whole corpus of 6,119 to 11,656 Wikipedia passages, and the reader gets the top 5 passages, as in the paper.
Conventional RAG: the top 5 passages by embedding similarity. State-of-the-art RAG: the 2025 production recipe, with the question split into sub-questions by a model, keyword and embedding results fused, and a cross-encoder reranker, top 5. LeanMemory: its memory walk with the same reranker, top 5.
Embedding BAAI bge-m3, reranker BAAI bge-reranker-v2-m3, reader and judge DeepSeek V4 Flash. The reader answers only from the passages it is given and says unknown otherwise.
After a pass over the questions, LeanMemory studies them: a slow reader finds the passages that answer each question, checks the answer, and writes the question and three rewordings into memory. No answers, labels or gold passages are used. Study 2 is a second round.
For the questions still missed after two study rounds, the set's reference answer is given to LeanMemory the way a person would teach it; it finds the passages that support the answer before learning it. Asked the same question again, it answers as taught, which is the point of teaching. The held-out rewordings show that what was taught is reached by meaning, not by wording.
Conventional and state-of-the-art RAG: the top 5 passages. LeanMemory: the top 5 passages plus its own notes, the answers it verified while studying or was taught, each with the passages it was built from. Before any learning no notes exist, so its reader starts from the same 5 passages as the others; recall at 5 always counts the 5 passages.
The rewordings were written before the first pass and never studied or taught. They score within two points of the questions as asked, and the reader's page stays small: eight entries on average after teaching.
| Question set | First sight | Study 1 | Study 2 | Taught | As asked |
|---|---|---|---|---|---|
| NQ | 66.3 | 69.6 | 69.7 | 83.9 | 84.3 |
| PopQA | 55.8 | 61.3 | 61.4 | 76.7 | 76.8 |
| MuSiQue | 41.0 | 51.7 | 52.3 | 79.2 | 83.6 |
| 2Wiki | 60.5 | 81.0 | 82.1 | 92.3 | 94.4 |
| HotpotQA | 82.8 | 90.9 | 90.8 | 96.2 | 98.4 |
| Average | 61.3 | 70.9 | 71.3 | 85.7 | 87.5 |
Score in per cent on rewordings written before the first pass and never studied or taught, at each stage. The last column is the score on the questions as asked, after teaching.
| Question set | First sight | Study 1 | Study 2 | Taught |
|---|---|---|---|---|
| NQ | 5.02 | 7.39 | 7.41 | 7.27 |
| PopQA | 5.00 | 6.78 | 6.79 | 7.34 |
| MuSiQue | 5.00 | 8.32 | 8.45 | 9.35 |
| 2Wiki | 5.00 | 9.58 | 9.62 | 9.59 |
| HotpotQA | 5.00 | 6.30 | 6.30 | 6.34 |
| Average | 5.00 | 7.67 | 7.71 | 7.98 |
Mean entries per question at each stage. Before learning the page is the 5 passages (on a handful of NQ questions the continuation of a passage rides along).
LeanMemory is one of the LeanTools products: one endpoint, one key, from your first call to production.