Benchmark numbers

Everything measured is here, including the parts that do not flatter us. Everything not measured is listed as not measured.

Retrieval quality on LoCoMo

Ranking quality, measured on a public dataset with a competitor's harness. Deliberately: we did not want the first number we published to be one we had built the ruler for.

Conditions, in full

The embedder and reranker were verified against the engine's own code paths before the run: identical inputs produce identical outputs to the last bit, not to six decimal places. The numbers therefore describe the product rather than a benchmarking shim built to resemble it.

Five configurations, whole corpus, n = 1,531

Margins are on nDCG@10 against BM25. 95% CIs: vector [+0.016, +0.056]; fusion [+0.071, +0.096]; vector+rerank [+0.153, +0.191]; fusion+rerank [+0.182, +0.213]. Sign tests significant for all four. The last row is RCLL's default configuration.

What "fusion" means in these rows. Reciprocal-rank fusion of two channels, lexical and dense, at k = 60 and equal weights. The shipped engine fuses four — it adds a graph-activation channel and a temporal one. So these rows are a floor on what fusion does in the product, not a measurement of the product's fusion, and every conclusion below inherits that limit. We say it here rather than in a footnote because an earlier version of this page called these rows four-way, which was wrong.

Why BM25 is the baseline and not "no memory". A no-memory arm scores exactly zero on a ranking metric by construction — it is an honest control that carries no information here. Lexical search is the baseline that matters, because published work on agent memory repeatedly finds plain BM25 beating systems that cost far more to run. A store that cannot clear it has nothing to report.

By question type — where it goes wrong

On multi-hop our default configuration loses to a simpler one. Adding the lexical channel to dense retrieval costs 0.041 nDCG@10 there, while helping everywhere else. Multi-hop is literally the shape of the thing we sell — one fact learned in one place, needed together with another learned somewhere else — so this is the category we care most about and the one where our defaults are currently wrong.

Why multi-hop loses, and what fixes it

Two readings were available: fusion dilutes a precise dense hit, or the reranker's candidate window fills with near-misses. A reranker cannot return evidence it was never handed, so the question is answerable by measuring what the first stage delivers into the window — no model, no judge. At the shipped window of 20 candidates on multi-hop, dense retrieval delivers evidence for 48.8% of questions and equal-weight fusion for 43.2%. Fusion hands the cross-encoder 5.6 points less to work with. Reordering cannot recover that, so it is the first stage, not the reranker.

The mechanism is that reciprocal-rank fusion weights a channel's rank and not its quality. BM25 is the weakest channel on multi-hop by a wide margin — 27.9% against dense retrieval's 48.8% at the same window — and equal weighting still lets it spend window slots. Halving its weight beats the current default in every category at once:

Evidence recall into a 20-candidate window, n = 1,531. Self-check passed: fusion with the lexical channel weighted to zero reproduces dense-alone exactly, so the fusion code produces no difference it could not have caused.

This is not shipped. It was measured on the two-channel substrate described above, and the engine fuses four; a weight that is right for two channels is not automatically right for four. It is a directional result and a follow-up run, not a release note. When the default changes, this page will say so and carry the number that moved it.

On single-hop, BM25 beats dense retrieval on its own — 0.4591 against 0.4518, and the difference is not significant. Twenty-five years of lexical search is not a straw man, and the whole margin in the headline table comes from fusion and reranking rather than from embeddings.

Open-domain is bad in every configuration and the best arm reaches 0.295. Those questions ask about things the conversation only gestures at; no amount of ranking rescues evidence that is not distinctly there.

The reranker in isolation

A separate run, different conditions — 20-candidate pools rather than whole-corpus ranking, so these figures are not comparable line-by-line with the table above. n = 1,137, with 394 questions dropped because first-stage retrieval returned no evidence at all for them.

The reranker changes which document ranks first on 641 of 1,137 questions. Read that next to the latency table below: the largest quality gain we can measure costs 85% of our response time. Both facts are ours and both are printed.

Two honesty notes on this run. The 394 dropped questions are a real ceiling, not a rounding detail — a reranker reorders what the first stage brought and cannot rescue what it missed. And on the open-domain subset (n = 47) the bootstrap interval and the sign test disagree; the harness marks it significant, the sign test gives p = 0.11. On a sample that small that is disagreement, not evidence, and it should not be quoted as a result.

What a shared store costs, and what scoping buys back

Every figure above ranks each question against its own conversation and nothing else: ten conversations, ten separate stores, no query ever meeting a stranger's memory. That is a silo, and it is the easy case. RCLL sells the other one — a single store a fleet shares, where every query is ranked against everybody else's memories too. We published the silo numbers without saying which case they were, which is exactly the omission that flatters the party making it.

So we ran the control. Same dataset, same metrics, same paired statistics, same embedder and reranker; between arms the only thing that changes is what the query is allowed to see.

nDCG@10 by scope, whole corpus, n = 1,531

Paired deltas against the same retriever in the silo, 95% bootstrap CI. Fusion: pooled −0.0268 [−0.0329, −0.0208]; pooled+room −0.0141 [−0.0195, −0.0086]; pooled+3rooms −0.0165 [−0.0220, −0.0110]. BM25: pooled −0.0333 [−0.0411, −0.0258]; pooled+room +0.0197 [+0.0134, +0.0261]. Every one of those excludes zero. The direction holds in all four question categories separately. Fusion + rerank: pooled −0.0111 [−0.0182, −0.0043]; pooled+room −0.0070 [−0.0124, −0.0020]; pooled+3rooms −0.0078 [−0.0137, −0.0022].

What the rows say

Sharing a store costs ranking quality, and the cost is real. Pooling ten conversations drops fusion by 0.0268 nDCG@10 — about 5.7% relative — with the interval well clear of zero, on every retriever and in every question category. It is a small number and it is not nothing, and a shared store sold as free is being sold past a measurement that exists.

Scoping buys back about half of it. Not all of it. Filtering to the right room leaves fusion 0.0141 below the silo — roughly 47% of the pooling loss recovered. The filter is worth having and it is not a restoration, and we would rather write that sentence than the one our own marketing would prefer.

In the configuration we actually ship, the cost is about a fifth of that. The two rows above are fusion without the reranker. RCLL's default adds one, and the cross-encoder reads the query against each candidate — so it re-sorts foreign turns down on its own, whether or not a room filter removed them first. Pooling costs the shipped stack 0.0111 nDCG@10 (1.9% relative); with the room filter, 0.0070 (1.2%). Read against the shipped default rather than against a component of it, "what a shared store costs" is 1.2%, not 5.7%. We report both because quoting only the second is the flattering half, and quoting only the first describes a configuration nobody runs.

And most queries do not move at all. Under pooled + room with the reranker, 1,425 of 1,531 questions score identically to a private store — 44 better, 62 worse, 93% untouched. That is also why those three rerank rows carry a bootstrap interval clear of zero while their sign tests do not reject (p = 0.10 to 0.22): the mean effect is real and it is carried by a small minority of questions, not by a broad shift. A cost concentrated in 7% of queries is a different engineering problem from one spread across all of them, and the two are indistinguishable in the averages.

On the lexical channel, scoping beats a private store outright — +0.0197 over the silo, the only positive delta in the run. Term statistics are a property of the corpus: an idf estimated over 5,882 turns is simply better than one estimated over 600. Pooling improves the vocabulary at the same time as it adds competition, and the room filter lets you keep the first while dropping the second. This is the one place a shared store is strictly better than private ones, and we did not predict it.

The dense column is the self-check, not a result. pooled + room reproduces the silo exactly on vector retrieval — a delta of 0.0000 with zero of 1,531 questions differing, not 0.0000 to four decimal places. Cosine similarity has no corpus-level statistic, so filtering a shared ranking down to one room must give the private ranking back. Had it not, the filter would not be doing what the product says it does, and every other row here would be suspect. The wrong room arm scoring a flat zero is the same check from the other side: a non-zero there would have meant identifiers from different conversations were colliding.

The wrong room scores zero. Not degraded — zero, in every configuration and every question category. A room filter is a hard gate, so a mis-typed or mis-guessed room is not a slightly worse search, it is an empty one that returns quietly. That is the whole argument for canonicalising room names as they are written rather than filing spelling drift under tidiness. What we did about it →

Being unsure is cheap; being confidently wrong is not. Asking three rooms instead of one costs 0.0024 nDCG@10 against asking the single right one. Asking one wrong room costs everything. So the correct instruction to an agent is "name several rooms", not "name the right room" — which only works if it can see which rooms exist, and that call did not exist when we started measuring.

What a shared store puts in front of you

The deltas above are what pooling does to a ranking metric. This is what it looks like in the result list, on the unscoped arm a naive deployment would ship:

"Foreign" means a turn from a conversation other than the one the question belongs to — the closest measurable proxy this dataset offers for another agent's memory.

What this measurement is not

A room here is a whole conversation, which is the best case a room filter will ever get. Ten cleanly separated topics, every answer entirely inside one of them, no fact belonging to two. Real rooms leak, real topics overlap, and 46% of our own store carries no room at all. Treat pooled + room as a ceiling on what the filter buys, not an estimate.

The room is handed to the query, not guessed. In the + room arm the right room is known. Real agents choose, which is why the + 3 rooms and + wrong room arms are there — they bracket the choice rather than model it.

Ten conversations is not a fleet. 5,882 turns pooled is a small shared store, and the cost of pooling should be expected to grow with the store, not stay at 0.0268. We have not measured that curve.

Still retrieval, still not accuracy, still the two-channel substrate rather than the engine's four, and still no reader and no judge. Everything in the section above inherits every limit in the sections before it.

The silo arm is also an independent reproduction of the published table at the top of this page: a second implementation, written to a different index structure for speed, reproduces all five published metrics — nDCG@5, nDCG@10, MRR, recall@10 and recall@20 — to the last printed digit on all five configurations. Its BM25 is checked against the published harness's own class at startup before any arm runs — max absolute difference 0.

Latency, per phase

Median of 12 queries against a live store of 8,345 facts / 971 documents / 192,318 links, traced through the engine's own instrumentation, 23 August 2026 — the store as it stood on the day of the run, which is why it is smaller than the usage table below, read three days later.

Median volumes per query: ~1,541 candidates fused, 300 reranked, ~108 facts returned inside a 4,096-token budget.

Read this before comparing it to anything. The conditions were: 8 vCPU KVM, no GPU, both the embedder and the reranker forced to CPU, on a box that was simultaneously serving our production CRM at a load average around 2.8. This is a working-machine number, not a reference-hardware number. It is also the default configuration rather than a tuned one.

What the split says. Search — everything except the cross-encoder — is 0.289 s p50. The reranker is 85% of the total. So if you see a memory system quoting tens of milliseconds, check whether reranking is in the number at all; in several systems it is a separate optional stage. Our search-only figure is the one comparable to a search-only figure, and our total is the one comparable to a total. Quoting our 3.0 s against someone else's search-only number would be dishonest in our own favour's opposite direction, and quoting our 0.289 s as the headline would be dishonest in our favour. Both are printed above.

What we would do about it. The fix is a GPU or a smaller reranker, not an index rewrite — which is exactly why the ledger is more useful than a single number. We have not done that work yet, so there is no improved figure to show.

Token cost

Read cost being zero is a property of the code path, not a tuned configuration. Write cost being real is the other half of the same ledger, and we report both because reporting only the read column decides the comparison before the measurement starts.

What the whole corpus above cost to write: 971 documents became 960 chunks, each of which is one extraction call. Roughly 1,500 tokens in and under 1,000 out per call, once. Two months of a 22-agent fleet's memory was built with about a thousand model calls, and retain runs asynchronously, so in day-to-day use the 2.5-second reranker on the read path is the cost you actually feel.

And the keyless case, stated precisely. With LLM_PROVIDER=none the service runs and both retain and recall work — but retain falls back to chunk mode: whole chunks are stored and embedded, with no fact extraction, no entity resolution, no causal links, and reflection and consolidation disabled. That is a functioning vector-and-lexical chunk store, which is a smaller product than the one measured on this page. Every number above was produced with extraction on. There is an offline provider in the engine that will run a small local model (~2B, GGUF) for extraction instead of an API key. We have now measured that substitution against a hosted arm on the same 20 chunks: 4.75 facts per chunk against 5.05, 27.2% of fields left N/A against 48.8%, and a median of 96.6 seconds per chunk against 7.4. On extraction yield the local model holds; the cost is latency, roughly 13× on CPU with no GPU. n=20, one box, one dataset — enough to stop calling it unmeasured, not enough to call it a benchmark. The full table, and why we threw out the schema-compliance row, is on the front page.

Usage from one fleet

Not a benchmark — the accumulated state of the store our own agents write to, read 26 August 2026.

Layers: L0 459, L1 3,176, L2 5,175. Halls: decision 2,078, fact 801, warning 687, discovery 520, procedure 507, preference 88, event 61 — plus 4,068 raw observations not yet typed. Room coverage: 459 of 459 L0 facts and 3,176 of 3,176 L1 facts carry a room, both exactly 100%; 1,107 of 5,175 L2 facts do. Attribution, live since 24 August: 243 facts carry an author, the 8,567 written earlier carry none.

What the link count is and is not

Publishing "201,154 links" beside "22 agents writing" would imply a fleet-built causal graph. It is not one. All but two of those edges were derived mechanically from similarity, time and entity overlap — useful for retrieval, and not the same thing as a fleet recording why things happened. Anyone with a dump of this store computes that split in a single query, so it belongs here rather than in a correction later.

What the same edges do say is where the knowledge sits relative to the topic axis: 123,631 of them cross a room boundary against 77,523 that stay inside one — 61.5% cross, with shared as the hub. That is the closest thing to a cross-agent measurement we can currently make, and it is not the one a buyer wants; the reasons are on cross-agent memory, together with the three things that are not instrumented in this store or, as far as we can tell, in anybody else's.

What we are still not claiming

Retrieval quality is not accuracy. Everything above measures whether the right evidence came back and how highly it ranked. It does not measure whether a model then answered correctly. Those are different measurements in different units, and putting our 0.586 nDCG@10 alongside somebody's "77% on LoCoMo" would be a category error we would deserve to be corrected on. We are stating that first because we are the party who benefits from the confusion.

We have no accuracy benchmark published, and we are not going to quote one until you can run the harness that produced it. In published work on agent memory, changing the reader-and-judge pair moves the score more than changing the memory store does — so a figure quoted without a named reader, a named judge, a dataset version and a seed is not comparable to any other figure, ours very much included. A number nobody can reproduce is not evidence, it is decoration.

The abstention case is invisible to us. LoCoMo's adversarial questions — the ones whose correct answer is "the store does not know" — carry no evidence labels, so a ranking metric cannot see them and they are excluded from every figure on this page. That is not a convenient exclusion: independent work finds that plain files outperform structured stores exactly there. Our cheapest measurement layer is structurally blind to our most likely weakness, and closing that gap requires a reader and a judge, which is the first thing on this list that costs money.

We have not measured the claim the product rests on. Agent A learns something while doing its own work; agent B, a day later and with no handoff, needs it. Does a shared store make B finish? That is a question about completed work, not about recall, and no public suite asks it — including the ones we just ran. The scoping ablation above narrows this gap without closing it: it says what a shared store costs a ranker, and says nothing about whether B finished.

So, in order: the fleet-execution question above, on a harness we ship; a reader-and-judge layer, named and pinned, to reach accuracy and the abstention case; then the same suites run twice, once with cloud models and once with a fully local keyless stack, because what local operation costs in quality is a number nobody in this category publishes and every self-hosting reader wants.

Negative results get published on the same page as positive ones. If the fleet-execution result comes out flat, that is the single most useful thing we could tell a reader, and we would rather tell it than have someone else discover it.

Reproducing the retrieval tables

The harness is Caura's, Apache 2.0, and needs only an OpenAI-compatible embeddings endpoint and a TEI-compatible rerank endpoint — it has no dependency on either vendor's storage. We pointed it at our own local models through a thin HTTP shim; anyone can point it at theirs. The dataset is public. The run takes CPU hours and costs nothing. When the repository is public the shim and the arms script ship with it.

Reproducing the latency table

The engine exposes per-phase timings on any recall — pass trace: true and read trace.summary.phase_metrics. The table above is the median of 12 queries collected that way, with no sampling, no discarded runs and no warm-up exclusion beyond one discarded warm-up pass to load the models. When the repository is public the collection script ships with it.