Engineering · hands-on
Test your agent's memory in 10 minutes: a reliability benchmark you can run yourself
By Rajamohan Jabbala · July 1, 2026
We've written about the four ways agent memory silently loses data and the correctness contract a memory layer should hold. This one is hands-on: you can reproduce all four failure classes — and watch a memory engine survive them — in about ten minutes, on your own machine, with no signup.
Honest scope: these are CLS++-only reproductions. The benchmark does not install or run any competitor — it reproduces failure classes documented in public issues (mem0 #5245 silent loss, #4536 delete-on-conflict) and proves CLS++ does not exhibit them. It is not a head-to-head score; it's a durability proof you can run and read for yourself.
1Install the package
It's a lightweight Python package — just httpx and pydantic. No server, no database, no account needed to run the benchmark.
pip install clsplusplus2Run the benchmark
One command runs the reliability reproductions and prints a human-readable summary. Add --json if you want to pipe the result into CI or a report.
cls bench # human-readable summary
cls bench --json # machine-readable output3Read what it tested
Each check injects a real failure into the write path and then verifies the fact is still there. The four classes:
- Embedder outage. The embedding call fails or returns an empty vector. In a vector-first layer the write is discarded and the API still returns success. Here the fact persists on the durable path and the write is flagged degraded, not dropped.
- Contradiction handling.A fact changes ("ship Friday" → "ship Monday"). Instead of deleting the old value, it's archived with lineage — you keep the history of what changed and when.
- Embedding dimension switch. You change embedding models and new vectors no longer fit the index. Writes survive the switch rather than getting silently rejected by the store.
- Empty extraction.When the extraction step returns nothing usable, the raw item is still stored — a dropped write here is indistinguishable from "the user never said it."
What a passing run means
A recall benchmark tells you how often you get the right memory back. It says nothing about how often a memory shouldhave been there and silently wasn't. That's the gap this closes: under a fully-broken embedder, a dimension change, a contradiction, or an empty extraction, nothing is lost without being surfaced. For an agent that acts on memory — edits code, answers a customer, books travel — that difference is a wrong action versus a correct one.
If you run it against your own workload and something behaves differently than you expect, that's exactly the kind of report worth sending — open a ticket or email support@clsplusplus.com.
Prefer to see it before you install?
The live demo does the same thing in the browser: tell it something, change it, watch the old value get archived instead of destroyed.