Coordinating Write-Quorum Across 50 Agent Instances Without a Central Coordinator

When you scale an agent swarm beyond a handful of instances, writes become the bottleneck. Each agent needs to persist state—task results, shared memory, coordination logs—but if they all hammer a single database, contention spikes. Worse, a central coordinator becomes a single point of failure and a throughput ceiling.

For a swarm of 50 agents operating on a mesh of three Hetzner servers (RiNET's typical topology), a central coordinator is unacceptable. Instead, we need a write-quorum protocol that is fully decentralized, resilient to partitions, and doesn't require consensus on every write. Here's how to build one.

The Problem: Concurrent Writes Without a Leader

Imagine 50 agents, each running a fine-tuned Qwen model on vLLM, sharing a Postgres database for persistent state. If every agent writes directly to Postgres, you face:

• Lock contention on rows or tables.
• Serialization overhead from transactions.
• A single database endpoint that can be flooded.

A common pattern is to use a message queue (e.g., RabbitMQ, NATS) as a write buffer, but that still introduces a central coordinator (the queue broker). If the broker goes down, all writes stop. For a sovereign infrastructure, we want no single point of failure.

Decentralized Write-Quorum via Gossip

The core idea: each agent holds a local replica of the state it cares about. Writes are propagated via a gossip protocol, and a quorum of replicas must acknowledge before the write is considered committed. No central coordinator—agents discover each other through a WireGuard mesh and a membership list (e.g., SWIM or a simple heartbeat table in a shared key-value store like etcd or Consul, but we avoid those dependencies by using a lightweight gossip layer).

Vector Clocks for Causality

To handle concurrent writes, each state object carries a vector clock. Each agent has a unique ID (e.g., agent-1 through agent-50). The vector clock is a map of agent-ID -> logical counter. When an agent writes, it increments its own counter and includes the full vector in the write message. Other agents compare vectors to detect conflicts.

// Pseudocode for vector clock update
VectorClock {
    counters: Map<AgentID, u64>,
}

fn increment(clock: &mut VectorClock, id: AgentID) {
    let counter = clock.counters.entry(id).or_insert(0);
    *counter += 1;
}

fn happens_before(a: &VectorClock, b: &VectorClock) -> bool {
    // true if all entries in a are <= b, and at least one is <
    let mut less = false;
    for (id, count_a) in &a.counters {
        let count_b = b.counters.get(id).unwrap_or(&0);
        if count_a > count_b {
            return false;
        }
        if count_a < count_b {
            less = true;
        }
    }
    // also check if b has any extra entries
    for (id, count_b) in &b.counters {
        if !a.counters.contains_key(id) && *count_b > 0 {
            less = true;
        }
    }
    less
}

Quorum Size and Consistency

For a swarm of N agents, we choose a write quorum size W and a read quorum size R such that W + R > N. This ensures that any read quorum overlaps with any write quorum, giving you strong consistency for overlapping reads and writes. A typical choice: W = floor(N/2) + 1, R = floor(N/2) + 1. For N=50, W=26, R=26. That means a write must be acknowledged by 26 agents before it's considered committed.

But waiting for 26 acknowledgments per write is slow. Instead, we can relax consistency: use an 'eventual consistency' model where writes are propagated asynchronously, and conflicts are resolved lazily. The quorum becomes a 'hint'—we only need a few acknowledgments (say, W=3) to consider the write accepted, and then gossip it to the rest. This is similar to Amazon Dynamo's approach.

The Write Protocol (Decentralized)

1. Agent issues a write: It creates a write request with the key, value, and its current vector clock.
2. Select a random subset: The agent randomly picks W-1 other agents from its membership list (it already knows about 50 agents via gossip). It sends the write to them directly over the WireGuard mesh (TCP or QUIC).
3. Acknowledge: Each recipient checks the vector clock against its local state. If the write is causally newer (i.e., the sender's clock happens-before the recipient's clock? Actually, we check if the recipient's local clock is not newer), it applies the write locally and sends an acknowledgment. If there's a conflict (concurrent clocks), it stores both versions and acknowledges.
4. Quorum reached: Once the sender receives W-1 acknowledgments (plus itself), it considers the write committed locally. It then starts a background gossip to all other agents.
5. Gossip propagation: Periodically (e.g., every 100ms), each agent picks a random peer and exchanges their vector clocks and any missing writes. This ensures the write reaches all agents eventually.

Conflict Resolution

When an agent receives a write with a vector clock that is concurrent to its local version, it must store both. This is a 'sibling' state. On read, the application must resolve conflicts. A simple strategy: use last-write-wins (LWW) based on a wall clock timestamp, but that can lose data. Better: use a CRDT (Conflict-free Replicated Data Type) for the state, such as a grow-only set or an LWW-register with vector clocks. For a key-value store, you can store a list of conflicting values and let the agent swarm reconcile via a deterministic merge function (e.g., for a counter, sum; for a set, union).

Implementation Details on RiNET Stack

On our three Hetzner servers, we run 50 agent processes distributed across them (about 17 per server). Each agent has a local SQLite database for its replica (or a small Postgres schema per agent, but SQLite is simpler). The WireGuard mesh provides a flat network with low latency (~0.5ms between servers in the same datacenter).

We use a simple gossip membership: each agent maintains a list of all agents, refreshed via a heartbeat mechanism. Every second, each agent sends a heartbeat to a few random peers. If an agent is silent for 3 seconds, it's marked as suspect; after 10 seconds, it's removed. This is a variation of the SWIM protocol.

For vector clocks, we serialize them as a JSON map and attach to each write. The overhead is small: for 50 agents, the vector is at most 50 entries, each 8 bytes for the counter plus the agent ID string (e.g., "agent-17"). That's ~1KB per write, acceptable.

Handling Partitions

In a network partition, some agents cannot reach others. The write quorum may not be achievable. Our protocol handles this by allowing a 'partial quorum': if an agent cannot gather enough acknowledgments within a timeout (e.g., 500ms), it stores the write locally and retries later. When the partition heals, the gossip exchange will propagate the writes and resolve conflicts via vector clocks.

This trades consistency for availability (AP in CAP theorem). For agent swarms, this is acceptable: agents can operate with stale data temporarily, and eventual consistency catches up.

Comparison to Centralized Approaches

A central coordinator (e.g., a single Postgres with serializable isolation) is simpler to implement but limits throughput to what one node can handle. With 50 agents writing at, say, 10 writes per second each, that's 500 writes/s. Postgres can handle that easily, but the coordinator becomes a bottleneck for reads and a single point of failure. If the database goes down, the entire swarm stalls.

Decentralized quorum spreads the load: each agent only handles writes for its own replicas and a few others. The gossip overhead is O(N^2) in the worst case, but with 50 agents and periodic gossip (not per-write), it's manageable. Typical bandwidth: each agent sends ~1KB per gossip message, 10 messages per second, so 10KB/s per agent—negligible.

Putting It Together

1. Each agent runs a local state store (SQLite).
2. Membership via SWIM-like gossip over WireGuard.
3. Writes use a vector clock and a partial quorum (W=3).
4. Conflicts are stored as siblings; resolved on read via CRDT merge.
5. Background gossip ensures full propagation.

This design is battle-tested in distributed databases like Riak and Cassandra. For an agent swarm, it provides the resilience and scalability needed without a central coordinator. The trade-off is complexity in conflict resolution, but for many agent workloads (e.g., accumulating results, updating shared state), CRDTs work well.

Code Sketch: Gossip Write in Python

import random
import json
import time
from typing import Dict, List

class Agent:
    def __init__(self, agent_id: str, peers: List[str]):
        self.id = agent_id
        self.peers = peers  # list of other agent addresses
        self.vector_clock: Dict[str, int] = {self.id: 0}
        self.store: Dict[str, bytes] = {}  # key -> value
        self.pending_writes: List[dict] = []

    def write(self, key: str, value: bytes):
        self.vector_clock[self.id] += 1
        write_msg = {
            'key': key,
            'value': value,
            'vector_clock': self.vector_clock.copy(),
            'sender': self.id
        }
        # select W-1 random peers
        quorum_size = 3
        targets = random.sample(self.peers, min(quorum_size-1, len(self.peers)))
        acks = 1  # self-ack
        for target in targets:
            # send over TCP (simplified)
            ack = self.send_write(target, write_msg)
            if ack:
                acks += 1
        if acks >= quorum_size:
            self.apply_write(write_msg)
            # gossip asynchronously
            self.gossip(write_msg)
        else:
            self.pending_writes.append(write_msg)

    def send_write(self, target: str, msg: dict) -> bool:
        # placeholder: actually send via socket
        return True  # assume success

    def apply_write(self, msg: dict):
        # check vector clock for conflicts
        local_clock = self.vector_clock
        incoming_clock = msg['vector_clock']
        if self.happens_before(local_clock, incoming_clock):
            self.store[msg['key']] = msg['value']
            self.vector_clock = self.merge_clocks(local_clock, incoming_clock)
        elif self.happens_before(incoming_clock, local_clock):
            # incoming is older, ignore
            pass
        else:
            # concurrent: store as sibling
            # simplified: just overwrite with last-write-wins
            self.store[msg['key']] = msg['value']
            self.vector_clock = self.merge_clocks(local_clock, incoming_clock)

    def happens_before(self, a: Dict[str,int], b: Dict[str,int]) -> bool:
        # standard vector clock comparison
        less = False
        for key, val_a in a.items():
            val_b = b.get(key, 0)
            if val_a > val_b:
                return False
            if val_a < val_b:
                less = True
        for key, val_b in b.items():
            if key not in a and val_b > 0:
                less = True
        return less

    def merge_clocks(self, a: Dict[str,int], b: Dict[str,int]) -> Dict[str,int]:
        merged = a.copy()
        for key, val in b.items():
            merged[key] = max(merged.get(key, 0), val)
        return merged

    def gossip(self, msg: dict):
        # periodically send to random peer
        pass

This is a simplified sketch. In production, you'd add retries, timeouts, and persistent storage.

Final Thoughts

Decentralized write-quorum is not a silver bullet. For workloads requiring strong consistency, a centralized approach may be simpler. But for agent swarms that value availability and scalability over strict consistency, this pattern works well. On RiNET's infrastructure, with three servers and 50 agents, we've found that a gossip-based quorum with vector clocks provides the resilience needed for autonomous systems without a single point of failure.