Skip to content

[Bug] Cloud BE coordinator address matching leaks prepared transactions and blocks schema changes with FQDN #68755

Closed
#68756
Closed
[Bug] Cloud BE coordinator address matching leaks prepared transactions and blocks schema changes with FQDN#68755
#68756

Description

@tuantran0910
tuantran0910
opened on Oct 7, 2026

Search before asking

  • I searched existing issues and PRs and found no duplicate for this coordinator-ID/FQDN mismatch. Related reports and fixes are listed below.

Version

Apache Doris 4.1.3

What's Wrong?

In cloud mode with FE FQDN addressing (enable_fqdn_mode), a stream-load transaction can remain in PREPARE after its coordinator BE is restarted/rescheduled. Schema-change and rollup jobs then remain in WAITING_TXN because the abandoned transaction is not recognized as belonging to the failed coordinator.

On the affected cluster, the jobs had to be unblocked by manually aborting leaked transactions. The default stream-load timeout is three days, so normal expiration can leave DDL blocked for hours or days.

There are two related identity checks:

  1. FE records a stream-load coordinator using the backend ID, the connection's client IP, and backend startup time. GlobalTransactionMgr.checkFailedTxnsByCoordinator() looks up the backend by ID but additionally requires be.getHost().equals(coordinator.ip) before checking restart/liveness. An FQDN does not equal the recorded pod IP. Finding the backend sets offline=false, so the address mismatch causes the method to return false despite a restart.
  2. Meta-service get_prepare_txn_by_coordinator() requires IP equality, and abort_txn_with_coordinator() requires both ID and IP equality. Heartbeat cleanup passes the BE host. A hostname/IP mismatch, or a changed pod IP after rescheduling, prevents matching old prepared transactions even when their backend ID is unchanged.

For example, the old transaction records (backend ID=123, IP=A, startup=T1). After rescheduling, the retained backend registration has (ID=123, hostname=be-0.example, startup=T2), with T2>T1 and a new pod IP B. The existing address checks miss the old transaction.

What You Expected?

For a retained BE registration, coordinator identity should be determined by its backend ID. A restarted or sufficiently long unavailable coordinator's abandoned transactions should be aborted automatically, allowing waiting schema-change and rollup jobs to progress.

Healthy transactions belonging to the new BE process must remain protected by startup-time filtering. Committed/final transactions must not be selected for abort, and concurrent cleanup must not cancel DDL jobs.

How to Reproduce?

Suggested controlled integration scenario (the new end-to-end scenario has not yet been run):

  1. Run a cloud deployment with FQDN addressing and both enable_abort_txn_by_checking_coordinator_be and enable_abort_txn_by_checking_conflict_txn enabled.
  2. Create a table and start a stream load. Hold it before commit so its transaction remains PREPARE; the existing StreamLoadExecutor.commit_txn.block debug point in test_abort_txn_by_be.groovy can help create this state.
  3. Start a heavyweight schema change or rollup on that table and verify WAITING_TXN.
  4. Restart/reschedule the coordinating BE while preserving its FQDN, heartbeat port, and FE backend registration/ID. A changed pod IP demonstrates the unstable-address case.
  5. Verify the BE startup time advances. Without the identity fix, the old prepared transaction is not selected by the address-based cleanup checks, and the DDL remains waiting until transaction expiration or manual abort.

A smaller deterministic meta-service reproduction is to create prepared BE transactions with ID 123 and address A, then call get_prepare_txn_by_coordinator with ID 123 and address B. Current upstream returns no matches; ID-based matching returns the old prepared transactions.

Anything Else?

Proposed fix: when both coordinator IDs are nonzero, match by backend ID; fall back to the address when an ID is unknown. In the cloud FE failed-coordinator check, remove the redundant host/IP guard after backend lookup by ID, retaining restart/liveness checks. Preserve non-cloud FE behavior.

The local patch adds a same-ID/different-IP meta-service regression case. Isolated compiled checks passed for 54 identity combinations and 33 FE failure-detection combinations. Full FE validation is running in Docker; no successful full-suite or new end-to-end result is claimed here.

Related upstream work:

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Activity

  1. Doris-Breakwater commented on Oct 7, 2026

    @Doris-Breakwater
    Doris-Breakwater
    on Oct 7, 2026
    Collaborator

    Breakwater-GitHub-Analysis-Slot: slot_ace030d334c0

    Assessment: The cited upstream code confirms a coordinator-selection defect for a retained, nonzero BE ID when the transaction address differs from the registered BE host. This mechanism can explain an abandoned PREPARE transaction blocking schema change or rollup. The reported production incident and the proposed end-to-end scenario have not been independently reproduced, so this is a code-backed diagnosis under the stated conditions, not a verified deployment-wide fix.

    The issue is currently open with no labels, assignee, or milestone. Suggested routing: cloud transaction management, with FE heartbeat/DDL and meta-service maintainers involved.

    Verified code path

    I inspected upstream master 7ac0eb2fdb426595f810e94ab77b7aa2eece12d5 from the local repository and fetched the corresponding files at branch-4.1 78aa9c996accd114a6e0f5943d7882ee1a34011a. Both contain the reported identity predicates.

    1. Transaction identity: Stream-load begin records the request's backend ID, connection client IP, and the backend startup time known to FE. TxnUtil persists these fields into the coordinator protobuf without normalizing the IP to the registered host. Consequently, hostname/IP inequality is possible without any change in backend identity. See transaction creation and coordinator serialization.
    2. DDL failure detection: checkFailedTxnsByCoordinator() finds the BE by ID and sets offline=false, but requires host/IP equality before evaluating restart or prolonged heartbeat loss. With the example (123, A, T1) versus (123, FQDN, T2), the address guard is false and the method returns false even when T2>T1. This is directly established by the FE predicate, also present in branch-4.1.
    3. Heartbeat cleanup: HeartbeatMgr supplies be.getId() and be.getHost(). In cloud mode, cleanup calls get_prepare_txn_by_coordinator(), then aborts the returned transaction IDs individually. The query requires IP equality even for matching nonzero IDs, so an address mismatch yields no candidates. The separate abort_txn_with_coordinator() RPC also requires ID and IP equality; it should receive consistent semantics, but it is not the heartbeat cleanup RPC used by this FE path. See heartbeat dispatch, cloud cleanup, and meta-service matching.
    4. Why DDL waits: Schema-change and rollup helpers retrieve unfinished transactions preceding their watershed and return false while that list is nonempty. An identity miss leaves the abandoned transaction in that list. Successful abort can require another polling round before DDL advances. The three-day stream-load default exists, but actual waiting time depends on the transaction's configured timeout and expiration processing. See schema-change helper, rollup helper, and timeout default.

    Candidate fix and safety checks

    The linked patch addresses both identity predicates and gates the FE behavior change on cloud mode. It is a focused change with no new configuration, RPC fields, or persisted format. Stable-ID matching should be scoped to the existing instance and retained BE registration; it does not recover an old identity after re-registration with a different ID.

    • Preserve BE source-type and PREPARED selection checks, strict coordinator.start_time < restart_start_time, and FE exclusion of COMMITTED/final transactions. The abort-by-ID handler re-reads transaction state and rejects COMMITTED/VISIBLE transactions, providing an additional protection when commit races cleanup. Preserve the existing DDL retry handling on abort failure. See abort state checks.
    • Explicitly document and test zero-ID compatibility. Falling back when either ID is zero broadens existing behavior: the query previously accepted a stored zero ID, but a request ID of zero did not match a stored nonzero ID. Missing ID fields must still be distinguished from explicitly supplied zero. Address fallback cannot resolve an address change when stable identity is unavailable.
    • Code-supported safety concern, not runtime-reproduced: Startup-time filtering protects restart cleanup, but BE-down cleanup calls the query with beStartTime=0, omitting the filter. Cleanup runs on a separate single-thread executor and does not recheck BE liveness at execution. If a down task waits in the queue, the BE recovers, and a new PREPARE load starts before that task queries meta-service, ID-only selection can include the new load despite its new address/startup time. Add a deterministic delayed-down-task/recovery test and assess liveness/epoch fencing before claiming that all new-process transactions are protected. This follows from the executor and dispatch and the cloud cleanup linked above; it is not evidence that this race caused the reported incident.

    Evidence still needed

    • Exact FE, BE, and meta-service build SHAs, relevant fork differences, and effective cloud/FQDN, cleanup-switch, heartbeat-loss threshold, and stream-load timeout settings.
    • One correlated timeline: transaction ID/status and persisted coordinator (source, ID, address, start_time); BE registration ID/host/startup time/liveness before and after rescheduling; DDL job ID, table, watershed transaction ID, and WAITING_TXN history. Redacted addresses should preserve equality/inequality relationships.
    • FE heartbeat/cleanup and DDL logs plus meta-service query request/status/candidate counts and abort results. Include any RPC/KV errors to distinguish an identity miss from scan or abort failure, and the actual before/after test output for the candidate patch.

    Recommended next steps

    1. Port the focused identity change onto the intended upstream revision and extend native FE/meta-service tests: same ID/different address must match; different nonzero IDs/same address must not; test zero/missing IDs, FE-source records, startup times below/equal/above the cutoff, and non-PREPARED statuses. Cover both coordinator RPCs and both query scan modes.
    2. Run the controlled cloud/FQDN integration scenario with heartbeat cleanup and DDL conflict cleanup enabled separately, then together. Exercise schema change and rollup, unchanged and changed pod IPs, and verify that the old transaction aborts while a new-process transaction survives and DDL finishes. Include delayed cleanup/recovery and concurrent commit/abort cases.
    3. Keep the existing abort-failure recovery and scan-reliability protections when integrating/backporting. #67661 and #67761 / #68059 address different parts of this flow; the inspected revisions retain the address mismatch.

    Validation here was read-only source and public patch inspection. No Doris code was modified, and no build, native test suite, or cloud integration test was run. The reported isolated check counts are author-provided evidence, not independently verified full-system validation.

  2. yiguolei
    closed this as completedin #68756on Oct 20, 2026
  3. yiguolei
    added a commit that references this issue on Oct 20, 2026

    [fix](cloud) Match transaction coordinators by stable backend ID (#68756

    83f6f9e
  4. github-actions
    added a commit that references this issue on Oct 20, 2026

    [fix](cloud) Match transaction coordinators by stable backend ID (#68756

    b7c6e8d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions