Skip to content

Split-Brain Safety

When the network splits and both nodes accept writes independently, hook-sync uses last-write-wins by timestamp to resolve conflicts on reconnect. This guarantees both nodes converge to the same state — no divergence, no silent data corruption.

Scenario During partition On reconnect Data loss?
INSERT (new rows) Both create different UUIDs Merge — both rows appear ❌ None
UPDATE same row Node A: value=100, Node B: value=200 Converge to higher updated_at Older update dropped
DELETE vs UPDATE Node A deletes, Node B updates UPDATE wins if newer than delete Delete intent dropped

Two nodes update the same row during partition. On reconnect, the receiver checks updated_at before applying:

Incoming change: { id: "X", value: 200, updated_at: T2 }
Existing row: { id: "X", value: 100, updated_at: T1 }
if existing.updated_at > incoming.updated_at:
skip (keep existing — it's newer)
else:
INSERT OR REPLACE (apply incoming — it's newer)

Both nodes converge to the row with the higher updated_at. The older update is silently dropped — no divergence.

Node A deletes a row, Node B updates the same row during partition. On reconnect:

Incoming DELETE: { op: "DELETE", old_id: "X", row: { updated_at: T_delete } }
Existing row: { id: "X", updated_at: T_update }
if existing.updated_at > delete.updated_at:
skip delete (keep the update — it's newer)
else:
DELETE FROM items WHERE id = X

If the row was updated after it was deleted (on the other node), the delete is skipped and the update wins. Both nodes converge.

INSERTs are always safe — UUID primary keys guarantee no collision. Two nodes creating rows during partition produce different UUIDs. On reconnect, both rows merge. No conflict resolution needed.

Last-write-wins means the older update is silently dropped. If two users edit the same field on different nodes during partition, one edit is lost. This is acceptable for:

  • Append-heavy workloads (event logs, telemetry)
  • Multi-tenant where tenants don’t share rows
  • Geographic distribution (each region owns its data)

For collaborative editing of shared rows, use CRDT-based sync (e.g., cr-sqlite).

LWW relies on wall clocks. If node clocks diverge, the node with the ahead clock wins conflicts even if its write is older in real-time. This is a fundamental limitation of LWW without a coordinator.

hook-sync detects clock skew but does not fix it. Fixing requires NTP at the OS level — same requirement as Cassandra, DynamoDB, and CockroachDB.

When applyChanges() receives remote changes, the library compares each change’s updated_at against the local wall clock. The maximum observed skew is tracked and exposed via health():

{
"ok": true,
"node_id": "nodeA",
"item_count": 42,
"pending_changes": 0,
"dead_letter": 0,
"peers": ["http://nodeB:9188"],
"clock_skew_ms": 5000,
"clock_ok": false
}
Field Type Description
clock_skew_ms int Max observed skew between local clock and remote updated_at. Resets on process restart.
clock_ok bool true if clock_skew_ms < 1000. false means NTP sync likely broken.
  1. Check NTP: chronyc tracking or ntpq -p on both nodes
  2. Restart NTP: sudo systemctl restart chronyd
  3. Alert: monitor clock_skew_ms in your health check endpoint
  4. Optional: refuse writes if skew exceeds your threshold (app-level decision)

36 checks across all 3 runtimes — 36/36 PASS:

Runtime Checks Passed Result
Go 12 12 ✅ PASS
Bun 12 12 ✅ PASS
Node 12 12 ✅ PASS
Total 36 36 ✅ ALL PASS
  1. Start both nodes, create shared item, verify sync
  2. Network partition (kill both nodes)
  3. Start nodes independently (no peer), update same item + create new items
  4. Reconnect (restart with peer)
  5. Verify convergence: same value, all items merged, 0 dead letter
  6. DELETE vs UPDATE conflict test

Run with: bash bench-splitbrain.sh (all runtimes) or bash bench-splitbrain.sh go (single runtime)

Traditional distributed systems use consensus (Raft, Paxos) to prevent split-brain: only one leader accepts writes. hook-sync takes a different approach:

  1. All nodes accept writes — no leader, no quorum
  2. UUID PKs — no ID collision across nodes
  3. Timestamp LWW — conflicts resolve deterministically on reconnect
  4. Idempotent apply — INSERT OR REPLACE makes re-sends safe

The tradeoff: last-write-wins drops the older update. The benefit: zero downtime, zero coordinator dependency, zero consensus overhead. Both nodes always converge.