Repository navigation
[Core] Support caller snapshot properties in native commits - #1104
Merged
JingsongLi merged 1 commit intoOct 11, 2026
Merged
Conversation
leaves12138
approved these changes
Oct 11, 2026
leaves12138
left a comment
There was a problem hiding this comment.
Reviewed at 77f5848. The per-call immutable committer configuration keeps snapshot properties isolated across stream checkpoints, and the common snapshot construction path preserves metadata across append, empty commit, overwrite, retry, conflict, and uncertain publication cases. The focused table commit suite passes 121 tests. No blocking correctness issue found.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PyPaimon already supports caller
snapshot_propertiesin batch, stream and overwrite commits, but this forced the Python publication path. Rust could deserialize snapshot properties but did not write caller metadata during publication.TableCommit.with_snapshot_properties, using the existing immutable committer configuration style. A configured clone retains the publication environment and retry configuration while owning the caller's property map.snapshot_propertiesargument to the batch and stream Python commit methods, and update their stubs. Each binding call creates its own configured committer so metadata cannot leak into later stream checkpoints.The semantics follow Java
ManifestCommittable.properties,FileStoreCommitImpl.tryCommitand Java's normalization of empty property maps to absent snapshot metadata. This only accelerates an existing PyPaimon capability.Verification
cargo test -p paimon --lib table::table_commit::tests::: 121 passed, including new properties isolation, overwrite recovery, CAS conflict and publication-response-loss cases.test_table_commit.py: 58 passed, covering batch/overwrite metadata, stream checkpoint isolation and empty checkpoints.cargo +1.98.0 clippy -p paimon -p pypaimon_rust --all-targets -- -D warnings, formatting and diff checks passed.Tests verify that retries preserve caller properties, recovered identities retain the originally published metadata, and a lost publication response preserves referenced data files.
Companion Python PR: apache/paimon#10505. It removes the snapshot-properties fallback and adds REST coverage for Append, PK and Data Evolution, including static/dynamic empty overwrite semantics and fallback/error boundaries.