Skip to content

resource_group client: prevent NaN in calcAvg when elapsed duration is zero - #11023

Merged
ti-chi-bot[bot] merged 1 commit into
tikv:masterfrom
JmPotato:jmpotato/fix-calcavg-nan
Jul 20, 2026
Merged

ti-chi-bot[bot] merged 1 commit into
tikv:masterfrom
JmPotato:jmpotato/fix-calcavg-nan

Conversation

@JmPotato

@JmPotato JmPotato commented Jul 20, 2026 •

Copy link
Copy Markdown
Member

What problem does this PR solve?

Issue Number: Close #11022, ref #9455

What is changed and how does it work?

A freshly created `groupCostController` can be published to the controller
map between the `updateRunState` and `updateAvgRequestResourcePerSec` passes
of the same tick. In that case `gc.run.now` still equals the `avgLastTime`
set by `initRunState` (the same `time.Now()` value), so `calcAvg` computes
`(0-0)/0 = NaN`, permanently poisoning `avgRUPerSec` (the moving average
keeps NaN forever) and making every subsequent token-bucket request carry
`required-token=NaN` until the idle cleanup deletes the controller.

Skip the sample when the elapsed duration is non-positive, and update the
`burstable` flag before sampling so a skipped sample does not delay it.
A regression test reproduces the exact scenario and verifies `avgRUPerSec`
stays finite and sampling resumes once the run state advances.

Check List

Tests

  • Unit test

Related changes

  • Need to cherry-pick to the release branch

Release note

Fix the issue that the resource group client might permanently send `NaN` token requests when a newly created resource group controller races with the periodic state update.

Summary by CodeRabbit

  • Bug Fixes

    • Improved request-rate tracking when no time has elapsed between measurements.
    • Prevented invalid rate values from affecting resource usage calculations.
    • Ensured burst handling state is updated consistently during request-rate updates.
  • Tests

    • Added coverage for skipped measurements and resumed tracking after time advances.

…s zero

A freshly created groupCostController can be published between the
updateRunState and updateAvgRequestResourcePerSec passes of the same tick,
in which case gc.run.now still equals the avgLastTime set by initRunState
and calcAvg computes (0-0)/0 = NaN, permanently poisoning avgRUPerSec and
making every subsequent token-bucket request carry required-token=NaN
until the idle cleanup deletes the controller.

Skip the sample when the elapsed duration is non-positive, and update the
burstable flag before sampling so a skipped sample does not delay it.
Add a regression test covering the exact scenario.

Signed-off-by: JmPotato <github@ipotato.me>
@ti-chi-bot ti-chi-bot Bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. dco-signoff: yes Indicates the PR's author has signed the dco. do-not-merge/needs-triage-completed needs-cherry-pick-release-8.5 Should cherry pick this PR to release-8.5 branch. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jul 20, 2026
@coderabbitai

coderabbitai Bot commented Jul 20, 2026 •

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 705c4787-590e-4520-83e5-74680eb01bd5

📥 Commits

Reviewing files that changed from the base of the PR and between 0b37df9 and f2bcfc2.

📒 Files selected for processing (2)
  • client/resource_group/controller/group_controller.go
  • client/resource_group/controller/group_controller_test.go

📝 Walkthrough

Walkthrough

The resource group controller now records burstability before average calculation and skips RU-per-second sampling when runtime time has not advanced. A regression test verifies no NaN is produced, accounting state remains unchanged, and sampling resumes after time advances.

Changes

Resource average sampling

Layer / File(s) Summary
Average sampling guard and regression coverage
client/resource_group/controller/group_controller.go, client/resource_group/controller/group_controller_test.go
The controller updates burstability before calculating averages and returns early for non-positive durations. Tests verify preserved state, non-NaN output, and resumed sampling after a later tick.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested labels: type/development

Suggested reviewers: yuhaozhang00

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly describes the main fix: preventing NaN in calcAvg when elapsed duration is zero.
Description check ✅ Passed The description includes the issue number, change summary, unit test, related changes, and release note as required.
Linked Issues check ✅ Passed The changes address #11022 by guarding non-positive delta durations, preventing NaN, and adding a regression test for the race.
Out of Scope Changes check ✅ Passed The patch stays focused on the race fix and its regression test, with no明显 unrelated changes.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@ti-chi-bot

ti-chi-bot Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

@YuhaoZhang00: adding LGTM is restricted to approvers and reviewers in OWNERS files.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@JmPotato

Copy link
Copy Markdown
Member Author

/retest

@ti-chi-bot ti-chi-bot Bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Jul 20, 2026
@codecov

codecov Bot commented Jul 20, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.25%. Comparing base (0b37df9) to head (f2bcfc2).

Additional details and impacted files
@@            Coverage Diff             @@
##           master   #11023      +/-   ##
==========================================
+ Coverage   79.23%   79.25%   +0.01%     
==========================================
  Files         541      541              
  Lines       76035    76037       +2     
==========================================
+ Hits        60245    60261      +16     
+ Misses      11545    11523      -22     
- Partials     4245     4253       +8     
Flag Coverage Δ
unittests 79.25% <100.00%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ti-chi-bot ti-chi-bot Bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jul 20, 2026
@ti-chi-bot

ti-chi-bot Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

[LGTM Timeline notifier]

Timeline:

  • 2026-07-20 05:57:05.356377199 +0000 UTC m=+1211611.392472254: ☑️ agreed by lhy1024.
  • 2026-07-20 06:08:10.204419066 +0000 UTC m=+1212276.240514122: ☑️ agreed by bufferflies.

@ti-chi-bot

ti-chi-bot Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bufferflies, lhy1024, rleungx, YuhaoZhang00

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:
  • OWNERS [bufferflies,lhy1024,rleungx]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@JmPotato

Copy link
Copy Markdown
Member Author

/retest

@JmPotato

Copy link
Copy Markdown
Member Author

/test pull-unit-test-next-gen-3

2 similar comments
@JmPotato

Copy link
Copy Markdown
Member Author

/test pull-unit-test-next-gen-3

@JmPotato

Copy link
Copy Markdown
Member Author

/test pull-unit-test-next-gen-3

@ti-chi-bot
ti-chi-bot Bot merged commit 39b6220 into tikv:master Jul 20, 2026
40 of 43 checks passed
@ti-chi-bot

Copy link
Copy Markdown
Member

In response to a cherrypick label: new pull request created to branch release-8.5: #11028.

@JmPotato
JmPotato deleted the jmpotato/fix-calcavg-nan branch July 20, 2026 11:16
JmPotato added a commit to ti-chi-bot/pd that referenced this pull request Jul 20, 2026
Signed-off-by: JmPotato <github@ipotato.me>
ti-chi-bot Bot pushed a commit that referenced this pull request Jul 21, 2026
…s zero (#11023) (#11028)

ref #9455, close #11022\n\nA freshly created `groupCostController` can be published to the controller
map between the `updateRunState` and `updateAvgRequestResourcePerSec` passes
of the same tick. In that case `gc.run.now` still equals the `avgLastTime`
set by `initRunState` (the same `time.Now()` value), so `calcAvg` computes
`(0-0)/0 = NaN`, permanently poisoning `avgRUPerSec` (the moving average
keeps NaN forever) and making every subsequent token-bucket request carry
`required-token=NaN` until the idle cleanup deletes the controller.

Skip the sample when the elapsed duration is non-positive, and update the
`burstable` flag before sampling so a skipped sample does not delay it.
A regression test reproduces the exact scenario and verifies `avgRUPerSec`
stays finite and sampling resumes once the run state advances.\n\nSigned-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>\nSigned-off-by: JmPotato <github@ipotato.me>\n\nCo-authored-by: JmPotato <ghzpotato@gmail.com>\nCo-authored-by: JmPotato <github@ipotato.me>
ti-chi-bot Bot pushed a commit that referenced this pull request Jul 24, 2026
…11029)

ref #9455, ref #11022\n\nUpdate the burstable state before sampling so a zero-duration sample does not
delay it. Add a regression test covering the creation race, finite
moving-average values, resumed sampling, and the burstable state update.\n\nSigned-off-by: JmPotato <github@ipotato.me>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved dco-signoff: yes Indicates the PR's author has signed the dco. lgtm needs-cherry-pick-release-8.5 Should cherry pick this PR to release-8.5 branch. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

resource_control: creation race lets calcAvg compute 0/0, permanently poisoning avgRUPerSec with NaN (likely root cause of #9455)

6 participants