[Bug] BE ls_normal thread_pool_queue_size grows monotonically and saturates at capacity — phantom queue count in the time-sharing scan pool (4.1.2) #68771

Activity

1. Doris-Breakwater commented on Oct 8, 2026

   

   Doris-Breakwater

   on Oct 8, 2026

   Collaborator

   More actions

   Breakwater-GitHub-Analysis-Slot: slot_841d0d77defc

   Initial assessment: This strongly matches a known TimeSharingTaskExecutor queue-accounting defect. The defect is present in the upstream 4.1.2 source; the matching fix was merged as PR #63568, with the 4.1 backport in PR #64855. I verified that backport commit 1f197f6c931217f546463371865c4d756b1a6058 is absent from the 4.1.2 ancestry and present in 4.1.3. Confidence is high for the source-level defect and its match to these symptoms; confirming the exact running BE builds is still necessary.

   Verified code mechanism: I inspected upstream tag 4.1.2 (commit aec169d20256a788de448f46737b5fb53ee3b4e3) without changing the checkout.

   • With enable_task_executor_in_internal_table=true (the default), WorkloadGroup creates TaskExecutorSimplifiedScanScheduler for ls_<workload_group>, using doris_scanner_thread_pool_queue_size as its queue limit. See workload_group.cpp.
   • The metric hook reads get_queue_size(), which returns _total_queued_tasks, rather than the actual split-queue size. _do_submit() increments that count when offering a split; a worker decrements it when taking the split. See executor metric/submission code and queue-size getter.
   • ScannerContext::stop_scanners() and its destructor call remove_task(). That function closes the task handle and physically removes its queued splits with remove_all(), without decrementing _total_queued_tasks. Workers decide whether to sleep from the real split-queue size, so they can be idle while the reported count retains removed entries. See scanner cleanup, remove_task, and worker dispatch.

   A minimal sequence is therefore: enqueue a split (+1), remove its task before a worker takes it (real queue -1, accounting unchanged), repeat. This explains accumulated phantom entries and why continuing to execute unrelated scan tasks does not drain the leaked offset. It does not require token shutdown itself to omit a decrement.

   This can affect query execution, not just the dashboard. The capacity check uses:

   capacity_remaining = max_threads - active_threads + max_queue_size - _total_queued_tasks

   It rejects submissions and increments submit_failed when this is below 1. Thus the exact boundary depends on active workers; with 48 maximum workers and zero active workers, it is 102448, rather than exactly 102400. The projected exhaustion date remains an estimate.

   What remains unverified: The supplied samples support this mechanism but do not identify which queries removed queued splits. Equal execution rates do not imply equal cancellation/cleanup histories. The supposedly healthy BE's flat, nonzero count of 19030 could also be an older leaked offset. Also, active-thread and queue gauges are read under separate lock acquisitions, so one active=0 sample alone is not conclusive; sustained observations and source evidence are stronger.

   Please provide:

   1. Full BE version/build Git hashes for all five nodes, any local patches, and effective enable_task_executor_in_internal_table, scanner queue limit, and workload-group scan-thread settings.
   2. Several timestamped raw /metrics captures retaining all labels, including pool id and BE identity, plus the dashboard PromQL. This will exclude aggregation across different pool instances.
   3. Representative query IDs, sanitized SQL/reproduction steps, and BE/FE logs showing cancellation, timeout, early scan termination, or cleanup around count increases. Please also include any TooManyTasks / Thread pool ... is at capacity errors and workload/configuration changes near the reported onset. These events are candidates, not established triggers here.

   Recommended next steps: Use a tested maintenance build containing the 4.1 backport; upstream 4.1.3 contains this specific fix, but verify the packaged BE hash. The patch subtracts only entries actually removed from the executor queue, centralizes offer accounting, and handles the idle transition/notification. Subtracting the entire task's split list would be incorrect because it can include running or otherwise nonqueued splits.

   For validation, use the existing TimeSharingTaskExecutorTest.test_remove_task_clears_queued_task_count from the backport, then exercise scan cancellation/cleanup in staging and check that an idle pool returns to zero and accepts subsequent submissions. I inspected that test; I did not build or run the BE suite or reproduce this against your cluster. A BE restart recreates the counter and can temporarily restore headroom, but does not remove the defect from an unpatched binary. Increasing the queue limit only postpones rejection.

   If the deployed build already includes the fix and still exhibits growth, keep this issue open for further investigation: obtain a maintainer-assisted snapshot of the actual split-queue size and _total_queued_tasks under the same executor lock, plus worker thread stacks. That comparison distinguishes an accounting mismatch from a real queue that is failing to dispatch.
2. 15767714253

   closed this as completedon Oct 10, 2026
3. 15767714253

   reopened this on Oct 10, 2026
4. 15767714253 commented on Oct 10, 2026

   

   15767714253

   on Oct 10, 2026

   ContributorAuthor

   More actions

   Key paradox — the queue is a phantom and it grows while the pool is idle.

   Sampling / 采样（primary evidence）

   Consecutive scrapes of /metrics on 172.31.17.165 (ls_normal, workload_group=normal) — 10 samples, 10 s apart, 2026-10-10 14:16:29 → 14:17:59 (GMT+8):

   for i in $(seq 1 10); do
     curl -s http://172.31.17.165:8040/metrics \
       | grep -E 'thread_pool_(queue_size|active_threads|submit_failed|task_execution_count_total)\{[^}]*thread_pool_name="ls_normal"'
     sleep 10
   done

   time (GMT+8)	queue_size	active_threads	task_execution_count_total	submit_failed
   14:16:29	25,610	0	181,602,147	0
   14:16:39	25,610	0	181,606,709	0
   14:16:49	25,611	1	181,611,036	0
   14:16:59	25,611	1	181,615,106	0
   14:17:09	25,612	0	181,618,385	0
   14:17:19	25,613	0	181,622,243	0
   14:17:29	25,613	0	181,625,538	0
   14:17:39	25,614	0	181,630,293	0
   14:17:49	25,615	0	181,634,608	0
   14:17:59	25,615	0	181,639,119	0

   Derived:

   • queue_size: +5 / 90 s ⇒ ≈ 3.3 / min ⇒ ≈ 4,800 / day.
   • task_execution_count_total: +36,972 / 90 s ⇒ ≈ 411 splits/s actually consumed by the pool.
   • max_threads = 48; active_threads ∈ {0, 1} for the whole window; submit_failed = 0 throughout.

   Why this proves the counter is wrong (independent of the "two gauges are read under separate locks" caveat):

   The pool reports it is consuming ≈411 splits/s. If a 25,610-entry real queue were feeding those ≤48 workers, even a single active worker would drain it in far under a minute; with ≥47 workers idle at every sample, a real queue of 25,610 would be emptied in well under 65 s. Yet across the entire 90 s window the value does not fall by a single unit — it only rises. A real backlog cannot simultaneously be huge, almost-idle-drained (411/s), and monotonically growing. ⇒ The reported depth cannot correspond to real queued splits; the gauge is over-counted (phantom queue depth).

   This argument does not rely on the atomicity of reading active_threads and queue_size separately — it only relies on the fact that a real queue of that size cannot survive an almost-idle worker set that is draining the pool at ~411 splits/s.

   Earlier corroborating sample / 早期佐证采样（2026-10-08，间隔 4 分钟两次读数）

   BE	queue_size	task_execution_count_total	active_threads
   172.31.20.48	14,893 → 14,905 (+12)	102,235,465 → 102,275,040 (+39,575)	1 → 0
   172.31.17.165	14,750 → 14,764 (+14)	106,108,764 → 106,152,122 (+43,358)	0 → 0
   172.31.17.66 (healthy)	19,030 → 19,030 (+0)	3,361,977,363 → 3,362,017,498	0 → 0

   Rate ≈ 3–3.5 / min ≈ 4,700–5,100 / day. From the 2026-10-05 14:16 restart to the 2026-10-10 14:16 sample (5.00 days) this predicts ~25,600 — matching the observed 25,610 almost exactly. Extrapolating, the pool will hit the 102400 capacity again around 2026-10-25, at which point submit_failed should become non-zero.

   Note that a healthy BE processes the same volume of tasks (≈40,000 in 4 minutes) while its queue_size stays perfectly flat at the same ~19,000 floor — so this is a state-dependent accounting leak, not a throughput/backlog problem.
5. yiguolei commented on Oct 10, 2026

   

   yiguolei

   on Oct 10, 2026

   Contributor

   More actions

   You mean the queue size is large, but active thread number is very low?
6. 15767714253 commented on Oct 15, 2026

   

   15767714253

   on Oct 15, 2026

   ContributorAuthor

   More actions

   5. You mean the queue size is large, but active thread number is very low?

   YES