Skip to content

Add fenced native minion scheduling and offline execution - #19962

Draft
xiangfu0 wants to merge 1 commit into
xiangfu0/codex2/ratis-minion-task-foundationfrom
xiangfu0/codex2/ratis-minion-execution
Draft

xiangfu0 wants to merge 1 commit into
xiangfu0/codex2/ratis-minion-task-foundationfrom
xiangfu0/codex2/ratis-minion-execution

Conversation

@xiangfu0

Copy link
Copy Markdown
Contributor

The Ratis task ledger can now drive a normal minion starter through scheduling, execution, and fenced offline output publication. A controller commits each finite schedule occurrence and its watermark atomically; an authenticated minion claims the task, generates a real Pinot segment, and commits immutable content plus the publication manifest and SUCCEEDED state in one transaction.

Stacked on #19961. Helix/ZK remains the default, and native query listeners remain disabled.

  • Add bounded committed-round scheduling, retry/cancellation recovery, and admission backpressure that preserves capacity for claims and progress.
  • Add certificate-scoped minion work/command/outcome RPCs, independent heartbeats, durable exact-request retries, and normal StartMinion integration.
  • Fence publication by live worker incarnation, attempt, table incarnation, captured input versions, and immutable output identity. Expired or cancelled workers cannot publish.
  • Expose authorized experimental table-scoped scheduling/status/cancellation APIs and document the qualification limits.

This draft qualifies NativeSegmentGenerationTask for digest-pinned local JSON-lines input and an approved shared local output directory. General legacy task/plugin API parity, cron/timeouts, remote deep stores, automatic replica placement, and retention/compaction remain outstanding. Publication records committed content; it does not assert query availability.

Validation:

  • 70 focused tests passed across catalog/snapshot, membership, authenticated transport, controller scheduling, minion execution/outbox, legacy minion configuration/registry, and normal starters. The final affected-behavior rerun passed 27 tests.
  • The real three-voter, mTLS normal-starter test generated and published a segment, installed it through the existing native assignment protocol, preserved a single publication after minion restart, verified closed query ports, and exercised quorum-loss failure.
  • Affected-module formatting, Checkstyle, and license checks passed. Compilation used warning/deprecation flags; no warnings matched added Java lines. git diff --check passed.

@xiangfu0 xiangfu0 added feature New functionality federation Related to federated query support minion Related to Pinot Minion task framework labels Oct 20, 2026
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex2/ratis-minion-task-foundation branch from ee0084a to 32cf7d2 Compare October 22, 2026 10:14
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex2/ratis-minion-execution branch from 4d1ceb4 to 841edf3 Compare October 22, 2026 10:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature New functionality federation Related to federated query support minion Related to Pinot Minion task framework

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant