Loom
When two facts disagree, the newer one is usually right. Loom asks what happens when it is not.
Loom, replayed
On FactConsolidation the newest fact wins by design, so recency is hard to beat there. Loom is the experiment that asks the harder question, and it needs a test that Tide cannot pass by counting.
1. The idea
Experiment 7 of 10 in the Engram series. ← Horizon · Engram overview · Coral →
Facts are not independent. “Person B lives in City R” sits among facts about where B works, where B’s family lives and what B did last year. If a new fact claims B lives somewhere else, most of that surrounding web disagrees with it, and the claim is weakly supported. If instead several other facts back the new claim, the old one is the odd one out. Loom measures support and lets the better-supported side win.
Loom · who has more support
Mutual-kNN means two facts count as agreeing only if each is among the other’s nearest neighbours, which limits how much support a fact that sits near everything can collect.
2. Where it came from
In April this was called protein binding: chunks attach to each other at shared-entity “sites”, and a contradiction disrupts the fold and earns low trust. Four scripts built it up. One stored small arithmetic webs (a division, a multiplication, an addition that agree) and asked whether an update that breaks the web loses. A second carved sites as it ingested facts. A third asked whether original facts sit in denser neighbourhoods than their updates. The project page’s one-line verdict on all of it: perfect in the lab, six for six, and useless at scale, because the locks were too vague.
The audit adds the reasons. The toy cases were small enough to tune. The author set the expected winner for each, and the gradual-evidence test counted a tie as a correct answer. Entities were capitalised words. The density test hard-coded example chunk indices. The one scored number for the family, “knit/loom”, was 42% against cosine’s 39%, in the one-feature test that returned more than k chunks.
3. Hypothesis
Stated before running, in two parts. First, on FactConsolidation, Loom at equal k does not beat Tide, because the dataset’s rule is that the later fact wins and support is a noisier proxy for that rule than the date is. I would be pleased to be wrong.
Second, on the gradual-override micro-test, Loom flips a belief only after a plausible amount of contrary evidence, and Tide never does: it flips on the first newer fact, whatever backs it. That is the property worth having in a memory that is sometimes told something false. This is the part of Loom that Tide cannot pass by counting.
4. Design
- Support
- The agreement mass of a fact: the summed similarity to the facts that are mutual nearest neighbours of it. Computed over the embeddings the Dense anchor uses.
- Conflict
- Two retrieved facts that are close in meaning and differ in content are a candidate conflict. The winner is the one with more support; the other is dropped from the result.
- Equal k
- Resolution removes losers and the list is refilled from the ranking to k. It never returns more than Dense.
- The micro-test
- A small fact web, a contradicting update, and 0 to 5 further facts that back the update. We record how many it takes to flip the answer. A tie is a miss, not a hit, and the winners are fixed by a rule written down before the run.
- The ablation
- Support term off: the same pipeline picking by date. It should land on Tide.
5. How it is tested
Loom is read against Tide as well as Dense, because on this dataset Tide is the right control. The micro-test is run separately from the benchmark: its output is a flip point per memory, with the number of cases and their construction rule published beside it. Hand-built cases are suggestive, not evidence, and the page will call them that.
The standing rules for every Engram experiment apply here unchanged.
- Fixed, seeded pilot sets: 100 LongMemEval_S questions (83 scored under the official skipping rules) and the 100 questions each of FactConsolidation single-hop and multi-hop at 6K. The seed is 20261008 and the sets are committed.
- Equal k. Every method, ours and the anchors alike, returns at most the same k ids, and the runner asserts it.
- No training or warm-up on scored data. Anything learned uses a seeded split disjoint from the pilot sets, and the code asserts the disjointness.
- Official metrics: recall_any@k and NDCG@k for retrieval, and the official SubEM with the official FactConsolidation prompt for answers.
- 95% confidence intervals on every number (Wilson for 0/1 metrics, bootstrap for NDCG), and paired tests against BM25 and Dense on the same questions (paired bootstrap and McNemar).
- Stochastic methods run on five seeds; the table reports the mean and the range.
6. Results
Loom · results
| Method | LongMemEval-S recall@5 | LongMemEval-S NDCG@10 | MAB FC-MH 6K answer in top 10 | MAB FC-SH 262K SubEM | MAB FC-MH 262K SubEM |
|---|---|---|---|---|---|
| Loom (support by mutual-kNN agreement) | 0.952 | 0.885 | 0.30 | not run | not run |
| Loom, recency decides conflicts | 0.952 | 0.885 | 0.57 | not run | not run |
| Loom, no subject regex | 0.952 | 0.885 | 0.39 | not run | not run |
| BM25 (ours) | 0.928 | 0.875 | 0.27 | 75.0 | 0.0 |
| Dense bge-small (ours) | 0.952 | 0.885 | 0.39 | 72.0 | 0.0 |
| BM25 (published, MAB Table 3, GPT-4o-mini) | n/a | n/a | n/a | 48.0 | 3.0 |
| Text-Embed-3-Small (published, MAB Table 3, GPT-4o-mini) | n/a | n/a | n/a | 28.0 | 3.0 |
| HippoRAG-v2 (published, MAB Table 3, GPT-4o-mini) | n/a | n/a | n/a | 54.0 | 5.0 |
| GPT-5-mini 400K, full context (published, MAB Table 3) | n/a | n/a | n/a | 78.0 | 28.0 |
How to read thisLoom’s rows are one run on the pilot sets with the new harness; no archived number is carried over. The LongMemEval columns are session retrieval (83 questions). The FC-MH 6K column is retrieval only: does any of the 10 returned facts hold the answer (100 questions; a 95% interval on a single cell is about ±0.1). The 262K SubEM columns are answer accuracy, which we ran for the BM25 and dense anchors only; the published rows sit at that context. “not run” means exactly that.
7. What would change our mind
- If Loom does not beat Tide on FactConsolidation, that is the expected result and not a failure, provided the micro-test shows a graded response.
- If the micro-test shows the same flip point whatever the support, support is not what drives the choice.
- If Loom loses to the support-off ablation, the “binding” in the name has been a date all along.
Sources
- isolated-ab · the first round’s one-feature-at-a-time test: chrono bonus, entity walk, ring, ant, manifold and knit, each added to cosine top-5
- Archived scorecard · the first round’s hand-written results table, with no raw logs behind it
- The Engram bench · one Memory interface, fixed pilot sets, official metrics and prompts
- Published rows · every number copied from the two papers’ tables, with the table named
- Rebaseline plan · the ten experiments and their improved designs
Experiment 7 of 10 in the Engram series. ← Horizon · Engram overview · Coral →
Loom is one of ten experiments in Engram. Its rows come from one run on the pilot sets.