fix(core): compare storage and content checksums like with like; record the materialized object's storage checksum

[phernandez]

Core prerequisites C1 to C3 for basic-memory-cloud's docs/MATERIALIZATION_WEBHOOK_PLAN.md.

Problem

Every note the app saves is indexed in the request, then materialized to storage. When storage notifies us about that write:

• It always re-reads and fully re-indexes the note. Materialization never recorded the written object's storage checksum, so the index gate (IndexedChecksums.recognizes, which compares the object's storage checksum with entity.checksum) cannot recognize it. In production that is ~23k re-reads and full re-indexes a day.
• In cloud it also marks the note superseded. storage_object_checksum_for_index_match returned bm-file-checksum (a content sha256) as if it were a storage checksum, and the planners compared it with the indexed checksum. Cloud indexing records S3 ETags, so those never match. Every app-written note indexed through a notification was marked content_superseded, which skipped its embedding job and dropped its provenance.

Change

C1/C2: record the materialized object's storage checksum.

• RuntimeWrittenFileState gains storage_checksum: the PUT's ETag in cloud, the content sha256 on a local filesystem.
• The publisher records it as entity.checksum, in the same guarded update that clears sync_checksum.

C3: each check compares one kind of checksum.

• Supersession: the object's storage checksum against the indexed storage checksum. A different object means a newer write replaced the file after the job read it, whoever wrote it. That write's own notification indexes it.
• Provenance trust (docs(skills): note optional web search setup for memory-research #1589): bm-file-checksum against the content checksum of the markdown indexed.
  • FileIndexResult gains content_checksum.
  • CurrentMaterializedNoteEntity carries storage_checksum (entity.checksum) and content_checksum (note_content.file_checksum), loaded alongside the entity.
• Removed: the mixing helper, its source enum and the plan diagnostic fields nothing read.
• Types: file_checksum_from_object_metadata is now typed as a content checksum.

Behavior change to review

Supersession now fires for any writer whose newer object replaced the file mid-job. Before, it fired only when own-stack bm-* metadata was present, because an ETag never equalled a sha256. A superseded result skips embeddings and withholds the content checksum and version from live updates; the newer write's own notification job indexes the current content. Locally, storage checksums are sha256s, so local results are unchanged.

Tests

• Planner tests are rewritten in the cloud shape (ETag storage checksums, sha256 content checksums). They cover:
  • regression: an app-written note is neither superseded nor stripped of provenance;
  • a replaced object is superseded;
  • stale bm-* metadata on different bytes is not trusted;
  • an object written outside the app;
  • the current-path equivalents, including an unrecorded content lineage.
• New test-int/test_materialized_note_index_gate.py publishes a materialization against a real database and asserts the index gate recognizes the written object's ETag. Confirmed it fails with the publisher change removed.
• Locally: just fast-check passes, and 964 tests in tests/indexing, tests/index, tests/cloud and tests/test_runtime.py pass.

Follow-up in cloud

The cloud materialization writer must return the written ETag as storage_checksum. That's a required field, so it goes in the same cloud PR that bumps the pin.

🤖 Generated with Claude Code

https://claude.ai/code/session_01T8wjd6HrtSA2LN9ssC4NzF

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-07T03:21:51.743457Z	8bd3d8b	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.