We ran BEAM, LoCoMo and LongMemEval through mem0's open-source benchmark harness and changed one thing: the memory search. GPT-5 answers and GPT-5 judges, with the harness's own prompts and scoring. Neomonic is ahead on BEAM, level on LoCoMo and behind on LongMemEval. Every question, answer and verdict is published below.
Each neomonic figure is the mean of two or three complete runs. The mem0 figures are the ones mem0 publishes in its README (platform v3, GPT-5). We did not run mem0 ourselves.
| Benchmark | Questions | Memories (top-k) | neomonic, mean | Each run | mem0, published | Result |
|---|---|---|---|---|---|---|
| BEAM 10M | 200 | 50 | 49.0% / 0.448 | 48.5% / 0.452 · 49.5% / 0.443 | 45.5% / 0.413 | ahead +3.5 |
| BEAM 1M | 700 | 200 | 72.3% / 0.663 | 72.0% / 0.656 · 72.6% / 0.669 | 70.1% / 0.641 | ahead +2.2 |
| BEAM 10M | 200 | 200 | 51.8% / 0.487 | 52.5% / 0.482 · 51.0% / 0.492 | 50.5% / 0.486 | ahead on pass rate level on score |
| BEAM 1M | 700 | 50 | 67.4% / 0.625 | 67.4% / 0.625 · 67.3% / 0.624 | 67.1% / 0.604 | level on pass rate ahead on score |
| LoCoMo | 1,540 | ours 100, mem0 200 | 92.9% | 93.2% · 92.5% | 92.5% | level, half the memories |
| LoCoMo | 1,540 | 50 | 92.1% | 92.3% · 91.8% | 91.8% | level |
| LongMemEval | 500 | 50 | 93.7% | 467 · 472 · 466 correct of 500 | 94.8% | behind −1.1 |
BEAM reports two numbers: the share of questions that pass (per-question score of 0.5 or more) and the mean score from 0 to 1. LoCoMo and LongMemEval report the share judged correct. Run-to-run spread is up to 1.5 points, so a gap smaller than that is a tie, and the LoCoMo rows are reported as one.
The harness is mem0's: github.com/mem0ai/memory-benchmarks, at commit 4b61c5d. Only the memory search was replaced with neomonic's. The answer prompt, the judge prompt and the scoring code are the harness's, unchanged. GPT-5 writes every answer and GPT-5 judges every one.
Each conversation turn is cut into pieces of at most 1,500 characters. Pieces are ranked by BM25 and by cosine similarity on nomic-embed-text embeddings, and the two rankings are fused by reciprocal-rank fusion (k = 60). No BEAM labels were read and nothing was tuned on BEAM. At 50 memories this sends GPT-5 about 13k tokens of memory per question, and about 52k at 200.
Neomonic's evidence ranker (hopnet) scores every turn of the conversation against the question. It was trained leave-one-conversation-out: each of LoCoMo's ten conversations was ranked by a model that never saw it in training.
The same ranker, trained 5-fold, so no question is ranked by a model trained on it. On top of the ranking, a note step (gpt-5-mini) reads the top turns and writes dated, verbatim quotes. It is routed by the question's wording alone, never by its type label. In each run 186 of the 500 questions get notes and the other 314 get ranked turns only. The notes count toward the same 50-memory budget.
The harness's GPT-5 answerer sees only what the memory search returns, cut at the stated top-k. The judge sees the question, the gold answer and the generated answer. We changed neither prompt.
Per-category results at the cutoffs in the headline table. BEAM and LongMemEval figures are means over the runs, and LoCoMo shows each run separately.
| Question type | 10M · top-50 | 10M · top-200 | 1M · top-50 | 1M · top-200 |
|---|---|---|---|---|
| abstention | 42.5% | 37.5% | 57.1% | 57.9% |
| contradiction resolution | 55.0% | 45.0% | 55.7% | 47.1% |
| event ordering | 12.5% | 17.5% | 57.9% | 68.6% |
| information extraction | 57.5% | 70.0% | 77.1% | 83.6% |
| instruction following | 82.5% | 87.5% | 79.3% | 87.9% |
| knowledge update | 70.0% | 75.0% | 69.3% | 70.7% |
| multi-session reasoning | 17.5% | 22.5% | 59.3% | 75.7% |
| preference following | 72.5% | 70.0% | 97.1% | 97.9% |
| summarization | 50.0% | 57.5% | 61.4% | 72.9% |
| temporal reasoning | 30.0% | 35.0% | 59.3% | 60.7% |
BEAM 10M has 20 questions per type, so one question moves a 10M cell by 2.5 points over two runs. BEAM 1M has 70 per type.
| Category | Questions | top-100 run 1 | top-100 run 2 | top-50 run 1 | top-50 run 2 |
|---|---|---|---|---|---|
| single-hop | 841 | 94.8% | 94.4% | 93.3% | 93.5% |
| multi-hop | 282 | 92.9% | 91.5% | 91.1% | 89.7% |
| temporal | 321 | 94.4% | 94.1% | 93.8% | 93.1% |
| open-domain | 96 | 77.1% | 74.0% | 81.2% | 78.1% |
| all | 1,540 | 93.2% | 92.5% | 92.3% | 91.8% |
Open-domain has only 96 questions, so a single answer moves it by about one point. In run 2 the same retrieval was judged at both cutoffs.
| Question type | Questions | Correct, runs 1 / 2 / 3 | Mean |
|---|---|---|---|
| single-session user | 70 | 66 / 68 / 68 | 96.2% |
| single-session assistant | 56 | 54 / 54 / 54 | 96.4% |
| single-session preference | 30 | 29 / 29 / 30 | 97.8% |
| knowledge update | 78 | 76 / 77 / 75 | 97.4% |
| temporal reasoning | 133 | 128 / 128 / 128 | 96.2% |
| multi-session | 133 | 114 / 116 / 111 | 85.5% |
| all | 500 | 467 / 472 / 466 | 93.7% |
Multi-session questions, which need counting or combining facts across many sessions, account for most of the misses.
Every mem0 figure here comes from mem0's own README (platform v3, GPT-5). We did not run mem0 ourselves. mem0 reports those figures from its own harness with GPT-5 answering and judging, which is the setup we used. The runs were made at different times, and possibly at a different harness commit.
Two runs of the identical setup differ by up to 1.5 points. That is why every figure is a mean, every individual run is shown, and gaps inside that spread are called level rather than wins.
LoCoMo is a tie. Neomonic reaches mem0's score while sending half as many memories (100 against 200), but the score itself is level. LongMemEval is behind by 1.1 points.
These are answer-accuracy results. They make no claim about speed or latency against mem0.
For every run in the table: the harness's final score file, and every question with its gold answer, GPT-5's answer and the judge's verdict and reason. The text of the retrieved memories is left out for size, but their ids and scores are kept. Each score was recounted from the per-question rows and matches the reported figure exactly. The manifest lists sha256 hashes and maps each file to its row.
About 17 MB in all. The judged files are gzipped JSON. The full harness outputs, including retrieved memory text, are available on request.