Commit e35cdb6
committed
refactor leader election around DB-issued terms
The old elector treated leadership as a lease held by one client ID and
renewed over time. That was simple, but it left too much implicit. One
leadership term was not clearly separated from the next, so reelection
and resignation were not scoped to a specific term. That made same-
client reacquisition harder to reason about, made it easy for stale work
or cleanup to target the wrong lease, and left the elector carrying more
responsibility for edge cases than it should have.
This change makes the database issue explicit leadership terms using the
columns we already have. `leader_id` remains the stable client ID, while
`(leader_id, elected_at)` identifies one specific term. Elect, reelect,
and resign now all operate on that exact term and return the leader row
from the database. The elector keeps a bounded local trust window for
its last successful confirmation, but that window is anchored to the
attempt that produced it, not to when the response happened to arrive.
That keeps slow successful reelections from stretching leadership past
its real lease budget while still avoiding direct app-vs-database clock
comparisons in the state machine.
The notification and test story is also clearer after the rewrite. Slow
subscribers now receive each leadership transition in order without
blocking the elector, resignation wakeups are coalesced safely, and the
poll-only coverage uses isolated fixtures so it can exercise real handoff
behavior without shared-schema flakiness. The shared driver suite now
covers term-scoped elect, reelect, and resign behavior across
PostgreSQL and both SQLite backends, including same-client term
replacement and stale-term rejection. The elector tests focus on the
observable behaviors that matter: gaining leadership, handing it off,
responding to resign requests, and stepping down cleanly when its trust
window expires.
This also rolls up the branch's earlier flake investigation and keeps
the original CI reference for the shared-schema failures that led to the
redesign:
https://github.com/riverqueue/river/actions/runs/244064651521 parent a678c97 commit e35cdb6
14 files changed
Lines changed: 1462 additions & 730 deletions
File tree
- internal/leadership
- riverdriver
- riverdatabasesql
- internal/dbsqlc
- riverdrivertest
- riverpgxv5
- internal/dbsqlc
- riversqlite
- internal/dbsqlc
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
10 | 14 | | |
11 | 15 | | |
12 | 16 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
0 commit comments