Horizon
The first attempt bent the space with a function nobody trained, and it lost by a wide margin. That was the wrong test of the right question.
Horizon, replayed
Hierarchies grow exponentially and flat space is bad at holding them. Hyperbolic space is built for the job. The question is whether a learned hyperbolic space helps recall for conversation and fact memories, or only for textbook trees.
1. The idea
Experiment 6 of 10 in the Engram series. ← Bloodhound · Engram overview · Loom →
In flat space, the number of points that fit at a given distance grows polynomially. In a hyperbolic disc it grows exponentially, which is how a tree behaves. Put a tree in the disc with its root at the middle and the leaves near the rim and distances come out right: siblings are close to each other, cousins are far. A memory is not exactly a tree, but “general at the centre, niche at the edge” is a hierarchy.
Horizon · the disc
The embedding is trained: it is fitted so that memories that were similar stay close and memories that were general end up near the origin. Retrieval ranks by hyperbolic distance.
2. Where it came from
The archived manifold took ordinary embeddings and squashed them into the ball with a tanh function. Nothing was learned. On synthetic clusters, retrieval at five scored 0.348 for hyperbolic distance and 0.296 with “dynamics” added, against 0.876 for cosine. Over twenty cycles of use the hyperbolic version stayed near 0.21 and cosine near 0.95. The project’s own note says it plainly: a naive projection hurts retrieval badly, and cosine is the right baseline until there is a hyperbolic embedding model.
So the honest result was negative, and also uninformative about hyperbolic space itself: an untrained projection is a strawman. The later variants (a PCA reduction to eight dimensions, an encoder trained by write-read-verify) left no result files, and the project page that advertised “−74% to +4%” has no log behind either number. The synthetic data was also built so that cluster membership was the ground truth, which makes cosine close to an oracle.
3. Hypothesis
Stated before running. A Poincaré embedding trained on the similarity graph of each memory does not beat Dense at equal k on these benchmarks, but norm (distance from the origin) tracks how general a memory is, and a small generality prior on top of hyperbolic distance helps on LongMemEval questions that ask for the gist of a topic.
The first half is a prediction that Horizon will not win, made so that the result cannot be spun. If it does win, we will have been wrong in a useful way.
4. Design
- Embedding
- A Poincaré embedding in the style of Nickel and Kiela, fitted by Riemannian gradient descent to the similarity graph of one memory’s items (nearest neighbours under bge-small). It is trained per memory, from the memory’s own contents and no labels.
- Retrieval
- Rank by hyperbolic distance from the embedded query. A norm-based generality term is added only in the second analysis, and its weight is fitted on a split disjoint from the pilot sets.
- Equal k
- The same k as every method.
- Seeds
- Training is stochastic: five seeds, mean and range.
- The ablation
- The archived tanh projection, run on the same real questions at last. It shows what the untrained version scores on the data that matters.
5. How it is tested
Tier 0 on LongMemEval_S and tier 1 on FactConsolidation, against Dense at equal k. The one figure specific to Horizon is a plot of embedding norm against a measured generality (how many other items an item is a near neighbour of), which shows whether the geometry puts general things in the middle at all, separate from whether it helps retrieval.
The standing rules for every Engram experiment apply here unchanged.
- Fixed, seeded pilot sets: 100 LongMemEval_S questions (83 scored under the official skipping rules) and the 100 questions each of FactConsolidation single-hop and multi-hop at 6K. The seed is 20261008 and the sets are committed.
- Equal k. Every method, ours and the anchors alike, returns at most the same k ids, and the runner asserts it.
- No training or warm-up on scored data. Anything learned uses a seeded split disjoint from the pilot sets, and the code asserts the disjointness.
- Official metrics: recall_any@k and NDCG@k for retrieval, and the official SubEM with the official FactConsolidation prompt for answers.
- 95% confidence intervals on every number (Wilson for 0/1 metrics, bootstrap for NDCG), and paired tests against BM25 and Dense on the same questions (paired bootstrap and McNemar).
- Stochastic methods run on five seeds; the table reports the mean and the range.
6. Results
Horizon · results
| Method | LongMemEval-S recall@5 | LongMemEval-S NDCG@10 | MAB FC-MH 6K answer in top 10 | MAB FC-SH 262K SubEM | MAB FC-MH 262K SubEM |
|---|---|---|---|---|---|
| Horizon (trained Poincaré embedding, blended) | 0.952 | 0.888 | 0.35 | not run | not run |
| Horizon, hyperbolic distance only | 0.928 | 0.783 | 0.14 | not run | not run |
| Horizon, Euclidean control | 0.952 | 0.877 | 0.41 | not run | not run |
| BM25 (ours) | 0.928 | 0.875 | 0.27 | 75.0 | 0.0 |
| Dense bge-small (ours) | 0.952 | 0.885 | 0.39 | 72.0 | 0.0 |
| BM25 (published, MAB Table 3, GPT-4o-mini) | n/a | n/a | n/a | 48.0 | 3.0 |
| Text-Embed-3-Small (published, MAB Table 3, GPT-4o-mini) | n/a | n/a | n/a | 28.0 | 3.0 |
| RAPTOR (published, MAB Table 3, GPT-4o-mini) | n/a | n/a | n/a | 14.0 | 1.0 |
How to read thisHorizon’s rows are one run on the pilot sets with the new harness; no archived number is carried over. The LongMemEval columns are session retrieval (83 questions). The FC-MH 6K column is retrieval only: does any of the 10 returned facts hold the answer (100 questions; a 95% interval on a single cell is about ±0.1). The 262K SubEM columns are answer accuracy, which we ran for the BM25 and dense anchors only; the published rows sit at that context. “not run” means exactly that.
7. What would change our mind
- If the trained embedding scores below Dense and the tanh version scores below that, the geometry did not help anywhere, and the archive’s own April note was right.
- If norm does not track generality, the picture in the middle of this page is not what the embedding learned, whatever it scores.
- If the trained embedding matches Dense and the norm tracks generality, the geometry is real and not needed for these benchmarks. That is a finding, not a failure.
Sources
- Archived manifold tests · cosine against a tanh-projected Poincaré ball on synthetic clusters; the project’s own conclusion
- Archived theory note · memory as a dynamical field; reads and writes both perturb it
- Archived scorecard · the first round’s hand-written results table, with no raw logs behind it
- The Engram bench · one Memory interface, fixed pilot sets, official metrics and prompts
- Published rows · every number copied from the two papers’ tables, with the table named
- Rebaseline plan · the ten experiments and their improved designs
Experiment 6 of 10 in the Engram series. ← Bloodhound · Engram overview · Loom →
Horizon is one of ten experiments in Engram. Its rows come from one run on the pilot sets.