Why We Ditched Kafka for Postgres LISTEN/NOTIFY

When we started building agent swarms that coordinate via events, the default choice felt obvious: Apache Kafka. It's the industry standard for event streaming, battle-tested at massive scale. But as we designed our orchestration layer, we kept hitting a wall: Kafka's operational complexity was disproportionate to our actual needs. We weren't processing millions of events per second; we were coordinating a few dozen agents that emit events at human-interaction speeds. So we took a step back and asked: what's the simplest thing that could possibly work?

The answer turned out to be Postgres LISTEN/NOTIFY. It's a feature that's been in Postgres for decades, often overlooked. It lets a client subscribe to a channel and receive notifications when another client sends a message. It's not a message queue in the traditional sense—there's no persistence, no replay, no consumer groups. But for our use case—agent event streaming—it turned out to be more than enough, and it eliminated an entire class of operational problems.

In this article, I'll walk through why we made the switch, the architecture we built, the tradeoffs we accepted, and the patterns that make LISTEN/NOTIFY viable for event-driven orchestration. This isn't a blanket recommendation to ditch Kafka everywhere; it's a lesson in matching infrastructure to actual requirements.

The Problem with Kafka for Agent Orchestration

Kafka is a distributed commit log. It's designed for high-throughput, durable, replayable event streaming. It shines when you have multiple consumers, need to reprocess events, or require exactly-once semantics across a fleet. But those features come with a cost: you need to run a ZooKeeper or KRaft cluster, manage partitions and replication, tune producer/consumer configurations, and monitor lag. That's a full-time job.

For our agent orchestration, we didn't need most of that. Our events are ephemeral: an agent finishes a task, emits a completion event, and other agents react. If an event is lost because a consumer crashes, the downstream agent will time out and retry or escalate. We don't need replay; we need timely delivery. We also don't have multiple independent consumer groups; we have a single orchestrator that decides which agent handles what.

Kafka's operational overhead was also a mismatch. We run a small infrastructure—three Hetzner servers in a WireGuard mesh. Adding Kafka meant dedicating resources to a cluster that would sit mostly idle. It also meant more attack surface and more backup complexity. The principle of least power applies: use the simplest tool that meets your needs.

Why Postgres LISTEN/NOTIFY?

Postgres was already our system of record. We use it for agent state, task metadata, and audit logs. Adding LISTEN/NOTIFY meant we didn't introduce a new technology; we just used an existing feature. The API is simple: LISTEN channel and NOTIFY channel, payload. The payload is a text string, typically a JSON blob.

What makes LISTEN/NOTIFY attractive for our use case is that it's transactional. You can NOTIFY within a transaction, and the notification is only sent if the transaction commits. This gives us atomicity: if an agent updates its state and emits an event, both happen together or not at all. That's a powerful guarantee that Kafka doesn't provide out of the box (you'd need to use a transactional outbox pattern).

Another benefit is that it's push-based. In Kafka, consumers poll for new messages. With LISTEN/NOTIFY, the server pushes notifications to connected clients. This reduces latency and simplifies client code: you just wait for a notification. It also provides natural backpressure: if a consumer is slow, the server doesn't queue messages for it; the consumer misses them. That forces us to design for eventual consistency, which is fine for our orchestration.

Architecture Overview

Our current setup looks like this:

• Three Hetzner servers in a WireGuard mesh, each running Postgres in a replication setup (primary and two replicas).
• Agents are long-running processes that connect to Postgres and issue LISTEN on specific channels, e.g., agent_tasks, agent_events.
• Orchestrator is a service that listens on those channels and dispatches work to agents via NOTIFY.
• State is stored in Postgres tables; events are also logged to an event table for audit and debugging.

Here's a simplified example of how we use NOTIFY in a transaction:

BEGIN;
UPDATE agent SET status = 'busy' WHERE id = 1;
NOTIFY agent_events, '{"type": "task_started", "agent_id": 1, "task_id": 42}';
COMMIT;

The NOTIFY payload is a JSON string. The listener receives it and parses it. Since the notification is sent only after commit, we avoid the race where a listener sees an event for a state that hasn't been committed yet.

On the client side, we use a simple Python script with psycopg2:

import psycopg2
import select

conn = psycopg2.connect("dbname=mydb")
conn.set_isolation_level(psycopg2.extensions.ISOLATION_LEVEL_AUTOCOMMIT)
cur = conn.cursor()
cur.execute("LISTEN agent_events")

while True:
    if select.select([conn], [], [], 5) == ([], [], []):
        print("No events, checking for new connections...")
    else:
        conn.poll()
        while conn.notifies:
            notify = conn.notifies.pop(0)
            print(f"Got notification: {notify.payload}")
            # Handle event

This is the core loop. It's simple, robust, and doesn't require any external libraries beyond the database driver.

Handling Backpressure and Reliability

One of the first concerns people raise with LISTEN/NOTIFY is reliability: if a client disconnects, it misses notifications. That's true. But we mitigate that by combining LISTEN/NOTIFY with a fallback polling mechanism. If an agent hasn't received a notification within a certain timeout, it checks a pending_events table for any missed events. This table is populated by the same transaction that sends the NOTIFY.

Here's how it works: instead of just NOTIFY, we also insert a row into an events table. The listener processes the notification and then deletes the row (or marks it as processed). If the notification is lost, the agent will eventually poll the table and pick up the event. This gives us at-least-once delivery with a simple retry mechanism.

INSERT INTO events (payload, created_at) VALUES ('{"type": "task_finished"}', NOW());
NOTIFY agent_events, '{"type": "task_finished"}';

The agent, upon receiving the notification, processes the event and then executes DELETE FROM events WHERE id = .... If it never receives the notification, it runs a periodic query: SELECT * FROM events WHERE created_at > NOW() - interval '5 minutes' and processes those.

This hybrid approach gives us the low latency of push notifications with the safety net of a durable queue. It's a common pattern in event-driven systems: use a fast path and a slow path for reconciliation.

Scaling Considerations

LISTEN/NOTIFY doesn't scale to thousands of concurrent listeners or millions of messages per second. The Postgres documentation warns that NOTIFY is not designed for high-throughput. But for our agent swarm, we're talking about a few dozen agents emitting a few events per second. That's well within Postgres's capabilities.

If we ever need to scale beyond that, we have options: we could use a more robust queue like RabbitMQ, or we could shard the channels by agent type. But the beauty of this design is that we can defer that decision until we actually need it. The interface between agents and the orchestrator is just a JSON payload; we can swap the transport without changing the agents.

Another consideration is that LISTEN/NOTIFY is not persistent across connections. If the orchestrator restarts, it misses any notifications that occurred during downtime. But again, the events table covers that. The orchestrator on startup can process any pending events.

Operational Simplicity

By removing Kafka, we eliminated an entire class of operational concerns. We don't have to monitor broker health, manage partitions, or worry about consumer lag. We already monitor Postgres, so we get observability for free. Our backup strategy for Postgres covers the event table, so we have a record of all events.

We also reduced our infrastructure footprint. Instead of running a Kafka cluster, we just use the Postgres instances we already have. This saves memory, CPU, and disk I/O. On our three Hetzner servers, every bit of resource counts.

When NOT to Use This Pattern

This pattern is not a silver bullet. If you need to replay events from the beginning of time, or if you have multiple independent consumer groups with different processing speeds, Kafka is the right tool. LISTEN/NOTIFY is also not suitable for high-throughput event streams (think telemetry from millions of devices).

It's also important to note that LISTEN/NOTIFY is not a queue. It's a signaling mechanism. If multiple listeners are on the same channel, they all receive the notification—there's no load balancing. In our architecture, we only have one orchestrator, so that's fine. If we had multiple orchestrators, we'd need to implement a leader election or use a different approach.

Lessons Learned

• Start with the simplest thing that works. We over-engineered initially by reaching for Kafka because it's the industry standard. But our requirements were modest, and Postgres LISTEN/NOTIFY was more than sufficient.
• Leverage what you already have. Postgres was already in our stack. Using its built-in features reduced operational overhead and cognitive load.
• Design for failure. Even with LISTEN/NOTIFY, we added a fallback polling mechanism. This gives us reliability without the complexity of a full message broker.
• Keep the interface decoupled. By using JSON payloads, we can change the transport later without affecting the agents.

Conclusion

We replaced Kafka with Postgres LISTEN/NOTIFY for our agent event streaming, and it's been a win for our infrastructure. We have fewer moving parts, lower latency, and simpler operations. It's not a solution for every scale, but for our agent swarm, it's the right fit.

If you're building a similar system, don't automatically reach for the heavy machinery. Evaluate your actual needs: how many events per second, how many consumers, do you need replay? If the answer is modest, consider Postgres LISTEN/NOTIFY. It might save you a lot of operational headache.

Remember, the best infrastructure is the one you don't have to think about.