Postgres Advisory Locks and the Serialized Agent Swarm

Agent swarms look concurrent on a whiteboard. N workers, each pulling a task, each calling tools, each writing results. In practice, a large fraction of those swarms funnel through one Postgres database, and a single advisory lock held across a slow operation can collapse the whole thing into a queue. The failure is quiet: no errors, no deadlocks, just throughput that tracks the slowest holder rather than the number of workers.

This is not a Postgres bug. It is a design pattern that emerges when teams reach for pg_advisory_lock as a cheap distributed mutex without accounting for what "session-scoped" actually means under a connection pool.

What advisory locks actually are

Postgres advisory locks are application-defined locks stored in shared memory. They do not protect rows or tables. They protect whatever the application decides they protect. There are two families:

• Session-level: pg_advisory_lock(key) and friends. Held until explicitly released or the session ends.
• Transaction-level: pg_advisory_xact_lock(key). Released automatically at commit or rollback.

The key can be a single bigint or a pair of int4. That is the entire API surface. There is no fairness guarantee, no timeout by default, and no visibility in pg_locks beyond the lock key and the holding PID.

-- session-level: survives COMMIT, must be released manually
SELECT pg_advisory_lock(42);

-- transaction-level: released at COMMIT/ROLLBACK
SELECT pg_advisory_xact_lock(42);

The distinction matters enormously in a pooled environment. A session-level lock is tied to the backend connection, not to your application request. If you acquire it inside a request and the connection returns to the pool without an explicit unlock, the lock stays held by an idle backend. The next request that needs it blocks. And the request after that. The pool slowly fills with waiters.

How a swarm serializes

A common pattern in agent systems is a "claim" step: a worker wants to process a task, so it takes a lock keyed on the task or on a shared resource, does the work, and releases. When the work is fast, this behaves like a mutex and everyone is happy.

When the work is slow, the lock becomes a serialization point. Consider a swarm where each agent needs to append to a shared log, or refresh a shared cache, or coordinate a rate-limited external API. A natural instinct is to guard that section with an advisory lock:

# Pseudocode: the shape of the problem, not a recommendation
def refresh_shared_state(conn, key):
    conn.execute("SELECT pg_advisory_lock(%s)", (key,))
    try:
        state = compute_expensive_state()
        write_state(conn, state)
    finally:
        conn.execute("SELECT pg_advisory_unlock(%s)", (key,))

If compute_expensive_state() takes hundreds of milliseconds, and the swarm has dozens of workers, the lock turns the parallel section into a single-lane road. Every worker that needs the refreshed state waits. The database is not the bottleneck; the lock is. And because the lock is held across application code, the hold time includes network round trips, Python execution, and any external calls inside that block.

The insidious part is that nothing looks wrong. CPU is low. The database reports few active queries. Latency graphs show a smooth increase that correlates with worker count, which teams often misread as "the database is getting loaded" rather than "the lock is serializing us."

Why the failure hides

Advisory locks do not appear in pg_stat_activity as anything meaningful. A waiting backend shows up as wait_event_type = 'Lock' with wait_event = 'advisory', which is easy to miss if you are watching query duration instead of wait events. The holder is often idle-in-transaction or running an unrelated query, because the lock is held across application logic, not inside a single statement.

-- Finding advisory lock waiters
SELECT pid, wait_event_type, wait_event, state, query
FROM pg_stat_activity
WHERE wait_event = 'advisory';

A second hiding place is the connection pool. If the pool is sized smaller than the worker count, workers queue for connections before they ever reach the lock. Teams then tune the pool upward, which increases the number of backends competing for the same lock, which makes the serialization worse. The feedback loop is counterintuitive: more connections, less throughput.

A third hiding place is retry logic. Workers that time out on the lock and retry create a thundering herd against the same key. Without jitter or backoff, the retries synchronize and the lock becomes a metronome.

The design question: what is the lock actually for?

Before reaching for advisory locks, it helps to classify the coordination problem. Advisory locks are a good fit for coarse, infrequent, correctness-critical mutual exclusion where the critical section is short and the cost of serialization is acceptable. They are a poor fit for high-frequency coordination, for anything that wraps external I/O, and for anything where the critical section can grow with load.

A useful test: if the lock is held across a network call, a model inference, or a loop over an unbounded collection, it is probably the wrong tool. The critical section should be a few statements, not a workflow.

Patterns that avoid the collapse

Move the lock inside the transaction

If the protected work is a single logical database operation, use pg_advisory_xact_lock. It releases automatically, which removes the entire class of "forgot to unlock" bugs and the idle-backend-holding-lock failure mode.

BEGIN;
SELECT pg_advisory_xact_lock(hashtext('cache:refresh'));
-- do the bounded work here
UPDATE cache SET value = $1 WHERE key = 'refresh';
COMMIT;

The lock is still a serialization point, but its hold time is now bounded by the transaction, not by application code.

Shard the key space

A single key serializes everything. A key derived from the resource being protected serializes only the relevant subset. If agents are coordinating per-tenant, per-region, or per-task, the lock key should reflect that granularity.

-- Instead of one global key, derive from the resource
SELECT pg_advisory_xact_lock(hashtext('tenant:' || $1));

This does not eliminate contention, but it converts a global bottleneck into a set of smaller ones. The tradeoff is that correctness now depends on the key derivation being stable and collision-resistant enough for the domain.

Replace the lock with a queue

Many advisory-lock patterns are really trying to express "one worker at a time should do X." That is a queue, not a lock. A dedicated table with SELECT ... FOR UPDATE SKIP LOCKED gives you work distribution without a global mutex, and it degrades gracefully as workers are added.

WITH claimed AS (
  SELECT id FROM jobs
  WHERE status = 'pending'
  ORDER BY created_at
  FOR UPDATE SKIP LOCKED
  LIMIT 1
)
UPDATE jobs SET status = 'running', claimed_at = now()
WHERE id IN (SELECT id FROM claimed)
RETURNING id;

SKIP LOCKED is the key detail. It lets workers bypass rows that are already claimed instead of blocking on them. The result is parallel claim throughput that scales with worker count rather than collapsing to one.

Use a lease with an expiry

For coordination that genuinely needs mutual exclusion but spans slow work, a lease table with an expiry timestamp is often more robust than an advisory lock. The lease is a row. It can be inspected, expired, and reclaimed. It survives connection churn. It does not depend on a backend staying alive.

UPDATE leases
SET holder = $1, expires_at = now() + interval '30 seconds'
WHERE name = 'shared-refresh'
  AND (holder IS NULL OR expires_at < now())
RETURNING name;

If the update returns a row, the caller holds the lease. If it returns nothing, someone else does. The expiry handles the case where a holder dies without releasing. The tradeoff is that leases require clock discipline and a reaper for expired rows, and they can allow two holders briefly if clocks drift.

Bound the critical section

If a lock must be held across slow work, the work should be decomposed. Compute outside the lock, then take the lock only to publish the result. This is the classic double-checked pattern: do the expensive thing optimistically, then verify under the lock that the result is still needed.

# Compute outside the lock
new_state = compute_expensive_state()

with transaction(conn):
    conn.execute("SELECT pg_advisory_xact_lock(%s)", (key,))
    current = read_state(conn, key)
    if current.version < new_state.version:
        write_state(conn, new_state)

The lock is now held for a comparison and a write, not for the computation. Multiple workers may compute redundantly, but they no longer serialize on the expensive part. Redundant computation is often cheaper than serialized computation, especially when the work is CPU-bound and the lock is the bottleneck.

Observability: making the invisible visible

Advisory lock contention is hard to see because it does not show up as slow queries. The instrumentation that helps:

• Track wait_event = 'advisory' over time, not just current waiters. A rising count of advisory waiters is the signal.
• Log lock acquisition and release with the key and the duration. If the duration distribution has a long tail, the critical section is too large.
• Alert on idle-in-transaction sessions that hold advisory locks. These are the leaks that fill the pool.
• Measure the ratio of workers to effective parallelism. If adding workers does not increase throughput, a serialization point exists somewhere, and advisory locks are a common candidate.

A simple query to find long-held session-level locks:

SELECT a.pid, a.state, a.query_start, l.objid
FROM pg_locks l
JOIN pg_stat_activity a ON a.pid = l.pid
WHERE l.locktype = 'advisory'
  AND l.granted
  AND a.state = 'idle in transaction';

The general lesson

Advisory locks are a sharp tool. They are cheap, they are correct for what they do, and they are easy to misuse in exactly the way agent swarms invite: a shared resource, a slow operation, and a desire for mutual exclusion. The failure mode is not a crash. It is a gradual loss of concurrency that looks like load and behaves like a queue.

The design move is to ask what the lock is protecting and whether that thing needs a lock at all. Often it needs a queue, a lease, or a sharded key. When a lock is genuinely required, keep the critical section bounded, prefer transaction-scoped variants, and instrument the wait events so that serialization is visible before it becomes the architecture.

Agent swarms are concurrent systems. The database is usually the coordination substrate. Treating that substrate as a set of independent, non-blocking primitives is what keeps the swarm parallel. Treating it as a place to park a global mutex is what turns it into a queue with extra steps.