← Blog

Benchmarks · agent memory research

What happens when an agent's memory contradicts itself?

By Rajamohan Jabbala · August 17, 2026

A public 300-case contract benchmark of Google Memory Bank's documented direct-create behavior and CLS++ lineage-aware belief revision.

The boundary of this comparison

These figures describe a local simulation of Google's documented direct-create contract. They are not hosted Google production measurements, a security report, or general product-accuracy scores. We have published the implementation, every fixture, the raw results, and the limits of the claim so the interpretation is reviewable—not implied.

The smallest useful example

Long-term agent memory is not only a storage problem. It is a truth-management problem. Consider two statements made by the same user at different times:

Earlier memory

“I love Chinese food.”

Later revision

“I now hate Chinese food.”

A useful memory system has to answer four separate questions: Which belief is current? Should the old belief remain available? Can ordinary retrieval avoid returning both as equally current? Can an auditor reconstruct the change?

The Google contract we evaluated

Google Memory Bank exposes more than one ingestion path. In the generated-memory path, Memory Bank extracts information and consolidates it against existing memories. Google says this process checks for duplication and contradiction and may create, update, or remove memories.

The direct CreateMemorypath is intentionally different. Google's quickstart cautions that directly created memories are not consolidated with existing memories at creation time and can produce duplicates in the same scope. They remain available for retrieval and can participate in a later generation-time consolidation.

Our baseline models that published distinction: direct creates append without immediate consolidation; retrieval requires an exact scope match; explicit deletion works. We do not invent undocumented Google lineage, currentness, or conflict-ranking behavior.

What the 300 cases test

The suite generates fixed synthetic cases in six categories. Every case names the scope, subject, relation, event order, expected current value, and expected historical state. A fixed generator makes the corpus and raw JSON results byte-for-byte reproducible.

CategoryCasesDocumented baseline simulationCLS++
Contradiction750/75 passes; 75/75 ambiguous75/75 passes; 0/75 ambiguous
Duplicate direct insertion50100 excess rows0 excess rows
Temporal revision5050/50 ambiguous50/50 latest value correct
Exact-scope isolation5050/50 passes; 0 leaks50/50 passes; 0 leaks
Explicit deletion2525/25 passes25/25 passes
Provenance500/50 lineage-contract passes50/50 lineage-contract passes

The aggregate count—75/300 baseline category-contract passes and 300/300 CLS++ passes—is deliberately not presented as general product accuracy. The categories were selected to examine CLS++'s revision contract. The category-level numerators and denominators are the defensible comparison.

What changed in the revision cases

Across the 175 contradiction, temporal-revision, and provenance cases, the CLS++ path selected the expected latest value in 175/175, preserved the prior value as history in 175/175, created a lineage link in 175/175, and incremented the revision version in 175/175.

Current belief

One non-superseded value

Archived belief

Prior value retained, not returned as current

Version

Revision advances from version 1 to version 2

Lineage

New belief links to the retained predecessor

The baseline simulation produced neither a version chain nor a current-versus-archived distinction because those properties are not promised for direct-created memories in the reviewed public documentation. This is a contract observation, not a claim that Google lacks internal audit data.

Where Google's contract passes

A useful comparison has to report the controls that work. Both paths passed all 50 exact-scope isolation cases with zero cross-scope leakage. Both passed all 25 explicit-deletion cases. The baseline is not constructed to fail every category; it implements the safe behaviors Google actually documents.

Google also supplies managed infrastructure that this benchmark does not evaluate: cloud IAM, regional and multi-regional operation, managed extraction, metadata filters, lifecycle configuration, long-running operations, quotas, and production service management. Nothing in this test establishes equivalent CLS++ operational scale or service-level guarantees.

What remains unverified

  • Google's hosted service was not called, measured, stressed, or attacked.
  • Generated-memory consolidation was not modeled as a failure; the comparison focuses on direct creation.
  • Google may maintain internal provenance signals that are not part of the public API contract reviewed here.
  • No hosted latency, cost, recall-quality, durability, or large-scale operational benchmark was run.
  • The public CLS++ package supplies the phase-memory engine; the small revision adapter makes the archive-with-lineage state transition explicit and inspectable.
  • The deletion compatibility path is modeled for package versions that predate a standalone engine forget method, and no CLS++ advantage is claimed for deletion.

Reproduce every number

The public repository contains both adapters, all 300 JSONL fixtures, the metric implementation, individual-case output, aggregate results, automated tests, a minimal contradiction demonstration, and Python and curl examples.

git clone https://github.com/rajamohan1950/clsplusplus-memory-bank-benchmark.git
cd clsplusplus-memory-bank-benchmark
python3 -m venv .benchmark-venv
.benchmark-venv/bin/python -m pip install -r requirements.txt

PYTHONPATH=examples/memory-bank-comparison-benchmark \
  .benchmark-venv/bin/python \
  examples/memory-bank-comparison-benchmark/benchmark.py

Three questions for Google

We invite the Memory Bank team to review the implementation, correct incomplete assumptions, and help replace the documented-contract simulation with an authorized hosted adapter.

  1. 1Does hosted retrieval apply additional, undocumented conflict resolution or currentness ranking to direct-created memories?
  2. 2Is revision lineage available through another API, audit surface, or upcoming Memory Bank capability?
  3. 3Can we jointly run these deterministic cases against an authorized Memory Bank instance and publish the hosted results alongside the simulation?

If Google supplies additional contract details or hosted evidence, we will update the report, benchmark, and this article transparently. The purpose is to make belief-revision guarantees testable—not to freeze an incomplete comparison in place.

Discuss the work in the Google Developer Agents forum .

The architectural question

Should agent memory overwrite history, leave conflicting facts side by side, or preserve an auditable evolution of belief?

Our position is that a correction should produce one current belief without destroying the evidence of how that belief changed. The public benchmark turns that position into a contract anyone can inspect, reproduce, challenge, and improve.