Canopy
Let a model do the tidying overnight, and let the reader walk a tree no more than seven wide.
Canopy, replayed
Hierarchies make long stores navigable for people. A curator model can build one at write time, once, and every later read pays less. Whether the reads also get more accurate is the open part.
1. The idea
Experiment 9 of 10 in the Engram series. ← Coral · Engram overview · Glyph →
The store keeps the raw record as the source of truth and builds a tree on top of it. The bottom tier is the original entries. The middle tier groups them into topics. The top tier is a few short summaries. A curator model, running offline and slowly, does the grouping and writes the summaries. A reader starts at the top, picks which of at most seven children to open, and repeats until it reaches entries. Seven is borrowed from the hexagonal grid the idea is named after, where a cell has six neighbours.
Canopy · three tiers
The tree is derived and disposable. Rebuild it and the grouping may come out differently, which the design accepts: a question about two topics should still find both by their shared ancestry whichever way the boundary falls.
2. Where it came from
The archived H3 was a curator (Haiku), three tiers of markdown nodes, and node ids made from a hash of the title so that a stable cluster keeps its identity across rebuilds. It was tested once, on the first twenty questions of one conversation from the LoCoMo benchmark, using only the first 200 turns. BM25 got none of them and H3 got one. Each query took about two minutes.
The audit found that the two numbers were not the same kind of thing. BM25’s score was whether the answer string appeared in the retrieved text, a recall measure. H3’s was whether a model’s written answer contained the gold, an accuracy measure. A BM25 score of zero for a keyword method on that data is odd enough that a matching bug is the likelier explanation. The match rule accepted a prediction contained in the gold answer, so a very short answer could pass. The “seven” was never enforced: the curator made five to fifteen clusters of any size. And the script’s docstring called its metric the benchmark’s official one, which it was not.
3. Hypothesis
Stated before running. Navigating a curated tree and then ranking the entries it reaches with the same dense scorer beats Dense alone at recall@5 on LongMemEval_S for the question types that ask about a topic across several sessions, and costs more per question.
The cost is not a footnote. Curating a haystack takes a model call per question, and a hundred questions means a hundred haystacks. I would count a method that needs minutes per query as unusable for an agent, and the table will say what each query cost.
4. Design
- Curator
- A language model builds the tree for each haystack once, with a fixed prompt and a hard cap on calls per run, written into the run record. The prompt is the archived one: summarise without inventing, name clusters by what they are.
- Width
- Seven, enforced this time. A node with more than seven children is split before the tree is used.
- Navigation
- The reader descends from the top and the entries it reaches are ranked by the dense scorer, so the output is a ranked list of at most k session ids.
- Where it runs
- LongMemEval_S, tier 0 only. The curation bill is why: it is paid once per haystack. The ~500-session LongMemEval_M run goes ahead only if the cost cap allows it. FactConsolidation is not run for Canopy.
- Seeds
- Curation is not deterministic, so if the budget allows it the tree is built twice per haystack and the two scores are reported.
5. How it is tested
The measurement is session recall@5, recall@10 and NDCG@10 against the official LongMemEval ground truth, which is a retrieval measure and the same one Dense and BM25 are scored on. The comparison is Canopy against Dense, paired, with the cost per question beside it. The published structure-augmented rows (RAPTOR, GraphRAG, MemoRAG) are in the table for context. They are MemoryAgentBench numbers, a different benchmark, which is why Canopy’s own MemoryAgentBench cells are n/a.
The standing rules for every Engram experiment apply here unchanged.
- Fixed, seeded pilot sets: 100 LongMemEval_S questions (83 scored under the official skipping rules) and the 100 questions each of FactConsolidation single-hop and multi-hop at 6K. The seed is 20261008 and the sets are committed.
- Equal k. Every method, ours and the anchors alike, returns at most the same k ids, and the runner asserts it.
- No training or warm-up on scored data. Anything learned uses a seeded split disjoint from the pilot sets, and the code asserts the disjointness.
- Official metrics: recall_any@k and NDCG@k for retrieval, and the official SubEM with the official FactConsolidation prompt for answers.
- 95% confidence intervals on every number (Wilson for 0/1 metrics, bootstrap for NDCG), and paired tests against BM25 and Dense on the same questions (paired bootstrap and McNemar).
- Stochastic methods run on five seeds; the table reports the mean and the range.
6. Results
Canopy · results
| Method | LongMemEval-S recall@5 | LongMemEval-S NDCG@10 | MAB FC-MH 6K answer in top 10 | MAB FC-SH 262K SubEM | MAB FC-MH 262K SubEM |
|---|---|---|---|---|---|
| Canopy (K=7 curated topics), n=16 | 1.000 | 0.895 | n/a | n/a | n/a |
| Dense on the same 16 questions | 0.938 | 0.890 | n/a | n/a | n/a |
| BM25 (ours) | 0.928 | 0.875 | 0.27 | 75.0 | 0.0 |
| Dense bge-small (ours) | 0.952 | 0.885 | 0.39 | 72.0 | 0.0 |
| BM25 (published, MAB Table 3, GPT-4o-mini) | n/a | n/a | n/a | 48.0 | 3.0 |
| Text-Embed-3-Small (published, MAB Table 3, GPT-4o-mini) | n/a | n/a | n/a | 28.0 | 3.0 |
| RAPTOR (published, MAB Table 3, GPT-4o-mini) | n/a | n/a | n/a | 14.0 | 1.0 |
| GraphRAG (published, MAB Table 3, GPT-4o-mini) | n/a | n/a | n/a | 14.0 | 2.0 |
| MemoRAG (published, MAB Table 3, GPT-4o-mini) | n/a | n/a | n/a | 21.0 | 7.0 |
How to read thisCanopy’s rows are one run on the pilot sets with the new harness; no archived number is carried over. The LongMemEval columns are session retrieval (83 questions). The FC-MH 6K column is retrieval only: does any of the 10 returned facts hold the answer (100 questions; a 95% interval on a single cell is about ±0.1). The 262K SubEM columns are answer accuracy, which we ran for the BM25 and dense anchors only; the published rows sit at that context. “not run” means exactly that.
7. What would change our mind
- If Canopy does not beat Dense at equal k, a tree built by a model has not made this store easier to search, and the cost buys nothing.
- If it wins on recall and costs minutes a question, it is a result about cost, not usefulness, and the page will report it that way.
- If two builds of the same haystack differ by more than Canopy differs from Dense, non-determinism is the finding.
Sources
- H3 hierarchical memory · three tiers, a curator model, K=7 neighbourhoods, non-deterministic by design
- The archived H3 benchmark · BM25 against curated navigation on one LoCoMo conversation, 20 questions
- Archived scorecard · the first round’s hand-written results table, with no raw logs behind it
- The Engram bench · one Memory interface, fixed pilot sets, official metrics and prompts
- Published rows · every number copied from the two papers’ tables, with the table named
- Rebaseline plan · the ten experiments and their improved designs
Experiment 9 of 10 in the Engram series. ← Coral · Engram overview · Glyph →
Canopy is one of ten experiments in Engram. Its rows come from one run on the pilot sets.