Skip to content

[Core] Support caller snapshot properties in native commits - #1104

Merged
JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/native-snapshot-properties
Oct 11, 2026
Merged

JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/native-snapshot-properties

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Summary

PyPaimon already supports caller snapshot_properties in batch, stream and overwrite commits, but this forced the Python publication path. Rust could deserialize snapshot properties but did not write caller metadata during publication.

  • Add TableCommit.with_snapshot_properties, using the existing immutable committer configuration style. A configured clone retains the publication environment and retry configuration while owning the caller's property map.
  • Copy properties into the common snapshot construction path, covering append, empty checkpoints, overwrite and retries. Properties are not inherited from the previous snapshot, and empty maps are omitted.
  • Add an optional snapshot_properties argument to the batch and stream Python commit methods, and update their stubs. Each binding call creates its own configured committer so metadata cannot leak into later stream checkpoints.
  • Keep publication failure and prepared-file ownership behavior intact.

The semantics follow Java ManifestCommittable.properties, FileStoreCommitImpl.tryCommit and Java's normalization of empty property maps to absent snapshot metadata. This only accelerates an existing PyPaimon capability.

Verification

  • cargo test -p paimon --lib table::table_commit::tests::: 121 passed, including new properties isolation, overwrite recovery, CAS conflict and publication-response-loss cases.
  • Python binding test_table_commit.py: 58 passed, covering batch/overwrite metadata, stream checkpoint isolation and empty checkpoints.
  • Built the extension and ran related PyPaimon tests with Native plan/read/write/commit enabled: 117 passed; 68 Native plans, 56 reads, 54 writes and 73 commits exercised.
  • cargo +1.98.0 clippy -p paimon -p pypaimon_rust --all-targets -- -D warnings, formatting and diff checks passed.
  • Independent core/API and Python adapter reviews completed with no remaining findings.

Tests verify that retries preserve caller properties, recovered identities retain the originally published metadata, and a lost publication response preserves referenced data files.

Companion Python PR: apache/paimon#10505. It removes the snapshot-properties fallback and adds REST coverage for Append, PK and Data Evolution, including static/dynamic empty overwrite semantics and fallback/error boundaries.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 77f5848. The per-call immutable committer configuration keeps snapshot properties isolated across stream checkpoints, and the common snapshot construction path preserves metadata across append, empty commit, overwrite, retry, conflict, and uncertain publication cases. The focused table commit suite passes 121 tests. No blocking correctness issue found.

@JingsongLi
JingsongLi merged commit 1ee2a10 into apache:main Oct 11, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants