Live warm-up of a few thousand minute subscriptions is stopped by the 10-minute single-time-loop limit; backfill throughput on 2 workers varies 10-30x between deployments #9881

Summary

In live trading, a SetWarmUp over a few thousand minute-resolution equity subscriptions on QuantConnect's data provider is regularly stopped by the Isolator's 10-minute single-time-loop limit while the warm-up backfill is still running. The algorithm's own code is idle (no user frames); the whole step is spent inside the warm-up enumeration. The throughput of the backfill, measured as completed history calls per minute, varies 10-30x between deployments of the same build on the same node class, and even the deployments that complete warm-up spend 270-280 s on a single step, so they pass with less than 2x margin.

This is the follow-up to #9715 / #9718 (null custom-data source), #9721 (history provider failure escaping the try/catch) and #9722 (parallel live warm-up enumeration, which gave a large improvement when it shipped). The timeouts are back on current builds.

Reproduce

Observed on cloud live deployments, builds 2.5.0.0.18155, 18158 and 18171, nodes with Engine.Run(): Resource limits '4' CPUs. 8192 MB RAM., QuantConnect data provider (CloudDataPermissionManager(): live job using QuantConnect data provider: True), about 5,000 equities at Minute resolution (trade + quote), SetWarmUp(TimeSpan.FromDays(4)). Seven deployments between 2026-10-03 and 2026-10-08 ended the same way, inside warm-up:

2026-10-08T03:11:10.5332Z ERROR:: AlgorithmTimeLimitManager.IsWithinLimit(): the current time step has been executing for 3.0 minutes and the algorithm will be stopped if it exceeds 10 minutes
2026-10-08T03:18:11.6101Z ERROR:: Security.ExecuteWithTimeLimit(): Algorithm took longer than 10 minutes on a single time loop. CurrentTimeStepElapsed: 10.0 minutes
2026-10-08T03:18:11.6103Z ERROR:: Extensions.SetRuntimeError(): RuntimeError at 10/06/2026 19:10:00 UTC. Context: Engine Isolator System.TimeoutException: Algorithm took longer than 10 minutes on a single time loop.

Begin DataStream for that run: AlgorithmManager.Run(): Begin DataStream - Start: 10/7/2026 ... Time: 10/4/2026 11:07:00 PM Warmup: True, AlgorithmManager.Stream(): Subscriptions count before warm up: 1164, then SubscriptionChanges: ... Active subscriptions: 5008, WeightedWorkScheduler(): will use 2 workers.

Measured from the syslogs (completed calls = InfluxDbHistoryProvider.GetHistory(): db returned N entries using K requests in X ms lines; the bulk of the requests cover the stretch from the last session on the node's files to deploy time, so the whole previous session for every subscription is backfilled from the history provider):

In the killed runs only 48-117 of the 2,100-3,000 warm-up history enumerators reached the database before the kill. Memory is not the limit (Isolator.ExecuteWithTimeLimit(): Used: 1561, Sample: 2584 of 8,192 MB), process CPU is 10-31% throughout the stall, and every symbol/type/range is requested exactly once. Runs F and G are the same algorithm family deployed with the paper brokerage instead of Interactive Brokers, same build, same node class, same cluster, launched within an hour of the killed runs, with twice the subscriptions. The brokerage is not on the data path (the history provider and feed are QuantConnect's in all runs); what differs on the IB deployments is the IB Gateway process sharing the node and the brokerage's account-update callbacks (300-800 HandlePortfolioUpdates traces per 10 minutes).

On build 18009 (right after #9722) the same warm-up configuration peaked at about 9,800 completions/min; a measurement across builds on 2026-09-01 showed 18024 at 1,700-2,200, 18029 at 200 and 18041 at 13-211. The database calls themselves have stayed at tens to hundreds of milliseconds, so the step is mostly dead time outside the query.

Reproduction without any user code (QuantConnect cloud, our organization): a C# algorithm with a dollar-volume coarse universe of 5,000 equities at Minute resolution and SetWarmUp(TimeSpan.FromDays(7)), project 35719298 (24 cores / 128 GB node, build 18036: warm-up completed in 334 s with one 157 s step) and project 35962132 (L4-8, 8 GB, build 18043: 362 s total, worst step 97.6 s, 2,235 MB peak). Kills are the tail of the same distribution (every observed kill is a 540-600 s step; survivors span 50-382 s), so a long-lived deployment samples it once and repeated stop/create cycles are needed to hit it.

Where the code sits

• Engine/DataFeeds/LiveTradingDataFeed.cs GetWarmupEnumerator: per subscription, a ConcatEnumerator of GetFileBasedWarmupEnumerator and GetHistoryWarmupEnumerator, wrapped by SubscriptionUtils.ScheduleEnumerator (Parallelize live warmup enumeration #9722) so the blocking IO runs on the work scheduler.
• Engine/DataFeeds/WorkScheduling/BaseWorkScheduler.cs: WorkersCount = Config.GetInt("data-feed-workers-count", Globals.LiveMode ? 2 : Environment.ProcessorCount); WeightedWorkScheduler.cs: MaxWorkWeight = ... LiveMode ? 100 : 400. In live mode the backfill of 10,000 enumerators runs on 2 workers regardless of the node's core count.
• Engine/AlgorithmManager.cs: algorithm-manager-time-loop-maximum applies to every time step, including the warm-up steps, so a warm-up whose backfill cannot keep up is a runtime error rather than a slow start.

Proposed change / open questions

1. During warm-up, either exclude the backfill wait from the single-time-loop budget (the algorithm is not running user code; the Isolator's memory monitoring still protects the node), or make the limit for warm-up steps a job-packet control, so a documented, supported warm-up configuration cannot end as Runtime Error.
2. Size the live warm-up workers to the node (Environment.ProcessorCount, as in backtesting) or expose data-feed-workers-count through the job packet; the 2-worker default was chosen for small nodes and is the ceiling on 4-core nodes too.
3. Open question for anyone with access to the node: where the 1-8 s between completed history calls goes on the IB deployments (file-based warm-up reads, the scheduler, or the provider), given the same build and node class complete the same work 10-30x faster with the paper brokerage.

Support conversation: 215475520166104.