Leader election & consensus

Why one leader

Many systems need exactly one machine in charge: the database primary that accepts writes, the scheduler that runs a nightly job once, the node that assigns shards. If two machines both believe they're the leader (asplit brain), you get duplicate jobs or conflicting writes. Leader election makes the group agree on one.

How consensus works (Raft, in brief)

1. The leader sends regular heartbeats. Followers that stop hearing them wait a random timeout, then become candidates.
2. A candidate asks the others for votes in a new term. Each node votes once per term.
3. Whoever gets votes from a majority becomes leader. Two leaders can't both get a majority, which prevents split brain.
4. The leader replicates every change to the followers and treats it as committed once a majority has it.

Majorities are why clusters have odd sizes: 3 nodes tolerate 1 failure, 5 tolerate 2. Adding a 4th node to 3 adds no fault tolerance.

In practice

• Don't implement consensus yourself. Use etcd, ZooKeeper or Consul, or a database with built-in failover.
• Use leases with expiry and fencing tokens (an increasing number with every new leader) so an old leader that wakes up late can't do damage.