Skip to content

Region will not be synced if the channel buffer is full #9154

Description

@okJiang

Bug Report

pd/server/cluster/cluster.go

Lines 1293 to 1298 in e8050f7

if saveKV || needSync {
select {
case c.changedRegions <- region:
default:
}
}

What did you do?

Too large heartbeat qps leads to changedRegions full, and the region miss syncing.

What did you expect to see?

Keep syncing

What did you see instead?

Region miss in the PD follower.

What version of PD are you using (pd-server -V)?

Activity

  1. 2 remaining items

  2. okJiang commented on Mar 27, 2025

    @okJiang
    MemberAuthor

    Although #9159 can prevent sync tasks from being discarded, it may lead to a significant backlog of tasks. To address this issue more effectively, we need to greatly enhance the consumption capacity of the syncer to avoid task accumulation.

    For example, we can directly enable multiple streams for synchronization for each follower.

    All is in #9159

  3. added a commit that references this issue on Mar 27, 2025
    f3d41fa
  4. okJiang commented on Apr 21, 2025

    @okJiang
    MemberAuthor

    If pending tasks accumulate for too long, newly arrived tasks will be regarded as failures. Therefore, we need to set retained=true for the sync runner.

    Image
  5. added a commit that references this issue on Apr 22, 2025
    1f565a6
  6. added 2 commits that reference this issue on Jul 14, 2025
    0604aa3
    21ea44b
  7. added a commit that references this issue on Aug 28, 2025
  8. added 2 commits that reference this issue on Sep 22, 2025
    8e7f6c6
    2351720
  9. added a commit that references this issue on Sep 22, 2025
    0e92ca1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    affects-8.5This bug affects the 8.5.x(LTS) versions.severity/majortype/bugThe issue is confirmed as a bug.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions