Skip to content

Competitive evidence and claim boundary

Updated: 2026-11-18 — Scope: WP7 evidence that can be reproduced without calling provider APIs

This page reports what HotMem's committed benchmark establishes and identifies what remains unmeasured. It does not claim HotMem beats Mem0 or Cognee.

Pinned references

Project Reference version Use in this report
HotMem 0.2.7 source; WP3 benchmark commit 735d78c The two local retrieval runs below
Mem0 Python SDK v2.2.1 Reference release only; not installed or benchmarked
Cognee v1.6.7 Reference release only; not installed or benchmarked

The Mem0 and Cognee tags pin the versions considered for a future comparison; they are not evidence that either system was run. The current importer work validates local file formats and does not exercise either product's extraction or retrieval pipeline.

Reproduce HotMem's results

The reports use the same deterministic synthetic corpus of 1,000 memories and 368 graded queries. The semantic run uses the pinned local model; neither run calls a hosted provider.

uv run python scripts/retrieval_eval.py \
  --output /tmp/hotmem-wp7-hash.json \
  --compare-to bench/retrieval/wp3-hash.json

hotmem model fetch
uv run python scripts/retrieval_eval.py \
  --embedder local-semantic \
  --output /tmp/hotmem-wp7-semantic.json \
  --compare-to bench/retrieval/wp3-semantic.json

For the semantic command, install the optional semantic dependency first as described in the retrieval benchmark guide. Each committed run is one-machine diagnostic evidence, not a service-level guarantee. MRR@5 and Recall@5 are scored over 322 queries with relevant results; false-positive rate is calculated over all 368 queries, including no-answer queries.

Results

HotMem embedder MRR@5 Recall@5 False-positive rate Search p50 / p95 Clone equivalence
hotmem-hash-v1 0.483 0.592 0.876 11.881 / 16.585 ms 1.000
local-m2v/potion-base-8M 0.864 0.929 0.801 32.063 / 50.976 ms 1.000

Sources: wp3-hash.json and wp3-semantic.json. On this fixture, the semantic embedder improves HotMem's retrieval metrics over its hash fallback, with higher local search latency. The semantic run still has a 0.270 near-duplicate slot rate and a 0.801 false-positive rate; it does not establish production behavior.

What the comparison supports

  • HotMem can run its retrieval evaluation locally, and its snapshot/hydration checks report clone equivalence of 1.000 on this fixture.
  • The local semantic model improves this fixture's retrieval scores over the hash fallback.
  • These results say nothing about Mem0 or Cognee retrieval quality, latency, memory use, or extraction quality. Their result cells are intentionally absent because no peer run was performed.
  • HotMem has a documented, verifiable JSONL interchange and import paths for Mem0 history and Cognee COGX exports. That establishes a migration path; it does not establish a head-to-head advantage.
  • HotMem core does not perform LLM extraction; see the WP4 extraction decision.

No provider calls were made for this report. A head-to-head result would need an approved spend limit and credentials for the competitor extraction steps, then a controlled run of these pinned releases on the same benchmark. Until that work is authorized and recorded, the supported claims are limited to HotMem's measured local behavior and documented portability.