CLS++ by the Numbers

The nightly scoreboard

Every night, 28 metrics run against the real engine (last run: 3,161.7 seconds of wall clock). Current state: 19 green, 8 yellow, 1 red. We publish all three colors.

Retrieval accuracy

Ask a fact back in the exact words you stored it: 100% at rank 1. Ask it in completely different words ("what do people call my pet?" for a stored dog's name): 80% at rank 1, 95% somewhere in the top 10. That 80% is our one red metric, and it's the current focus.

Truth maintenance

When you say "I moved to Mumbai," the Bangalore fact gets archived with lineage — not deleted, not returned as the answer. Zero stale-answer wins across the suite, with 100% precision and recall on contradiction detection.

Speed

Reads and writes are both under 1.5 ms at p95 — 6–33× inside their gates. No LLM call sits on the write path (more on why that matters below).

Durability and retention

A fact stored today is still retrievable at simulated ages of one, three, and seven years. Under a 100,000-write flood, 86.5% of memories survive in hot+dormant tiers — yellow, honestly reported, being worked. The extraneous-write rate (junk that shouldn't have been stored) dropped from an 87.5% baseline to 25%.

Under the hood, every write is acked only after an fsync to a write-ahead log with CRC32-framed records — Redis and Postgres replicate off the hot path. Both replicas can be down and nothing is lost.

Head-to-head: the Memory Integrity Benchmark

Ten contradiction/supersession scenarios, same runner for every system, reproducible from the repo:

The structural difference: OSS mem0 keeps both the old and the new fact and defers resolution to your downstream LLM — that's a 90% contradiction rate and 261 tokens per correct answer at 2.1 LLM calls per scenario. CLS++ resolves supersession in the memory layer, deterministically, with zero LLM calls and 19 tokens per correct answer — a 13.7× token reduction — while keeping the superseded fact recoverable (100% preservation for both systems).

Latency compounds this: OSS mem0 spends ~2.6 seconds per write on LLM fact-extraction. CLS++ writes are embedding-only and sub-second (1.48 ms p95 at the engine).

The scenario we lose: "vegetarian" → "eats chicken" (embedding cosine 0.35) needs world knowledge, not geometry. That's the 1-of-10 miss at τ=0.40, and the motivation for optional LLM adjudication — one call per detected conflict, not per write.

Failure injection: the silent-loss test

Reproducing Mem0 issue #5245 (silent memory loss when the embedder fails): we write 12 facts through the engine while the embedding backend fails on 100% of calls, then query every fact back with re-worded questions.

Result: 0.0% loss.Dense embeddings are a re-ranking signal in CLS++, not a storage dependency — retrieval falls back to the engine's own token index.

External data, not just our own

Our own scenarios are a fair thing to be skeptical of, so the same runner targets LongMemEval (ICLR 2025), specifically its knowledge-update category: 70 of 78 items converted cleanly (the 8 exclusions are printed, not silently dropped). On that third-party data, every toy heuristic — naive recall, BM25, even last-write-wins — collapses to 27% truth retention, because real people state updates indirectly. That's the bar the real engine has to clear.

The stack, in numbers

• ~75,600 lines of Python in the engine
• 3,832 test functions across the test suite
• 4 memory tiers — L0 working buffer → L1 index → L2 schema graph → L3 deep recess — plus a new L4 derivation tier that combines two stored facts into a new one
• 11 MCP tools — plug into Claude Code, Cursor, Windsurf, VS Code
• 4 framework adapters: CrewAI, LangGraph, LlamaIndex, AutoGen
• 2 SDKs, PyPI and npm, both named clsplusplus, method-for-method mirrors

The honest summary

Three numbers to remember: 0% stale answers, 0 LLM calls per write, 1.5 ms p95 writes. And two we're still earning: paraphrase recall at 80% against a 95% gate, and survival at 86.5% under a 100k-write flood. The scoreboard runs every night; the reds are published next to the greens.