TiKV Documentation Error: Peer vs Storage

The TiKV documentation contains a terminology inconsistency regarding PD scheduling. The statement that “PD ensures replicas of a Region are not placed on the same peer” is technically incorrect and should refer to “storage” instead of “peer” for accuracy. In TiKV architecture, peers are the individual replicas participating in Raft consensus, while stores refer to the physical TiKV nodes where these replicas are stored. PD’s scheduling logic prevents multiple replicas of the same region from being placed on the same physical store to ensure fault tolerance, not from being on the same peer (which would be redundant since each peer is already a separate replica).

---

Contents

• Understanding TiKV Architecture
• PD Scheduling Mechanics
• Terminology Analysis: Peer vs Storage
• Documentation Inconsistency
• Correct Scheduling Behavior
• Practical Implications

---

Understanding TiKV Architecture

TiKV organizes data into Regions, which are the basic units of data distribution and replication in the cluster. Each Region contains multiple peers that participate in the Raft consensus protocol for data consistency and availability. These peers are distributed across different physical TiKV nodes, which are referred to as stores in the TiKV ecosystem.

The Placement Driver (PD) acts as the cluster manager and scheduler, responsible for making decisions about where to place these Region replicas to ensure high availability and performance. PD’s scheduling component works by analyzing the cluster state and determining optimal replica placement according to various rules and constraints.

---

PD Scheduling Mechanics

PD implements sophisticated scheduling logic to maintain cluster health and performance. The scheduling system includes several checkers that monitor various aspects of the cluster state:

• RuleChecker: The most critical checker that verifies whether regions have proper replica counts and placement
• MergeChecker: Handles region merging operations
• JointStateChecker: Manages joint consensus states
• SplitChecker: Handles region splitting operations

According to the TiKV scheduling documentation, when a scheduling operator is generated, it’s sent to TiKV through the heartbeat mechanism of the region being scheduled. The RuleChecker specifically ensures that regions maintain their required replica count and follow placement rules.

---

Terminology Analysis: Peer vs Storage

The confusion in terminology stems from the distinction between peers and stores in TiKV:

• Peers: Individual replicas of a Region that participate in the Raft consensus protocol. Each peer is a separate entity that can vote and hold data.
• Stores: Physical or logical TiKV nodes that provide storage capacity. Each store can host multiple peers from different regions.

The scheduling architecture documentation clarifies this distinction: “Replicas of a Region must not be in one unit. So, we can configure labels for the TiKV peers, and set…”

This means that PD prevents multiple replicas (peers) of the same region from being placed on the same physical store (TiKV node). If multiple peers from the same region were on the same store, a failure of that store would result in the loss of multiple replicas, defeating the purpose of replication.

---

Documentation Inconsistency

The inconsistency appears in the TiDB scheduling documentation, which states: “Generally PD can only ensure that replicas of a Region are not at a same peer to avoid that the peer’s failure causes more than one replicas to become lost.”

This statement contains the terminology error. Logically, if two replicas are “at the same peer,” they would be the same replica, which is nonsensical. The intended meaning is that replicas should not be placed on the same storage node/store.

The PD configuration documentation further supports this interpretation when discussing store failures: “When PD fails to receive the heartbeat from a store after the specified period of time, it adds replicas at other nodes.”

---

Correct Scheduling Behavior

PD’s actual scheduling behavior focuses on store-level placement rather than peer-level placement. The scheduling logic ensures:

1. Anti-affinity: Replicas of the same region are distributed across different physical stores
2. Topology awareness: Placement considers the cluster’s physical topology using location labels
3. Load balancing: Replicas are distributed to balance the load across available stores
4. Failure recovery: When stores become unavailable, PD schedules replicas to healthy stores

The placement rules documentation emphasizes that “you must configure location-labels for PD and labels for TiKV at the same time” for proper topology-aware scheduling.

---

Practical Implications

Understanding the correct terminology is crucial for:

1. Troubleshooting: When replica placement issues occur, understanding that the constraint is at the store level helps diagnose problems correctly
2. Configuration: Proper configuration of location labels requires knowing that stores (not peers) are the primary scheduling targets
3. Monitoring: Monitoring tools need to track replica distribution across stores rather than peers
4. Documentation: Clear terminology prevents confusion when discussing scheduling behavior

The pd-ctl documentation provides practical commands for managing scheduling, such as setting maximum pending peer counts and configuring store-related parameters, all of which operate at the store level.

---

Conclusion

The TiKV documentation contains a terminology error where “peer” should be replaced with “storage” when describing PD scheduling behavior. PD ensures that replicas of a Region are not placed on the same storage node (store), not on the same peer (which would be illogical since each peer is already a separate replica). This distinction is crucial for understanding the actual fault tolerance mechanisms in TiKV and for proper cluster configuration and troubleshooting.

For accurate implementation and troubleshooting, always refer to store-level placement constraints and topology-aware scheduling rules that PD implements to maintain high availability across the cluster.

---

Sources

1. TiKV Scheduling Documentation
2. TiDB Scheduling Documentation
3. TiKV PD Configuration
4. TiKV Scheduling Introduction Wiki
5. Schedule Replicas by Topology Labels
6. TiKV PD Configuration File
7. TiKV pd-ctl Documentation