Postgres listens, Qdrant indexes, Neo4j navigates: a three-store pattern for agent memory

Agent memory is the hardest part of building autonomous agents. Single-store solutions always leak abstraction: a vector store alone can't handle facts, and a relational store can't do similarity search. The three-store pattern solves this by assigning each database the job it does best.

The problem with single-store memory

Most agent frameworks default to a single vector store (e.g., Chroma, FAISS) for all memory. This works for simple RAG but falls apart when agents need to:

• Remember precise facts (e.g., "user's timezone is UTC+2")
• Navigate complex relationships (e.g., "this document was cited by that paper")
• Maintain conversation state across sessions

Vector stores are bad at exact match, relational stores are bad at similarity, and graph stores are bad at dense retrieval. Trying to force one store to do everything leads to hacks like storing JSON blobs in vector metadata or running graph traversals on relational joins. Don't.

The three stores

Postgres: the source of truth

Postgres holds all structured data: user profiles, conversation logs, entity attributes, and event streams. It's the authoritative store that other stores sync from. Use pgvector only when you need lightweight embeddings inline; for production agent memory, keep Postgres clean and push vector search to a dedicated store.

Key tables:

CREATE TABLE agents (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    name TEXT NOT NULL,
    created_at TIMESTAMPTZ DEFAULT now()
);

CREATE TABLE memory_events (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    agent_id UUID REFERENCES agents(id),
    event_type TEXT NOT NULL,  -- 'fact', 'observation', 'interaction'
    content JSONB NOT NULL,
    created_at TIMESTAMPTZ DEFAULT now()
);

CREATE INDEX idx_memory_events_agent_type ON memory_events(agent_id, event_type);

Qdrant: the similarity engine

Qdrant handles all vector search. It's fast, supports filtering, and has a clean gRPC API. Every memory event that needs semantic retrieval gets embedded (e.g., with BGE-M3) and stored in Qdrant with a pointer back to the Postgres row.

Example payload:

{
  "id": "<uuid>",
  "vector": [0.1, 0.2, ...],
  "payload": {
    "agent_id": "<uuid>",
    "memory_event_id": "<uuid>",
    "event_type": "observation",
    "timestamp": 1700000000
  }
}

Use Qdrant's filter to scope searches to a specific agent or time range. Prefer cosine distance for most use cases.

Neo4j: the relationship navigator

Neo4j stores the graph of entities and their connections. When an agent needs to answer "what documents are related to this project through person X?", Neo4j does it in milliseconds. Use it for knowledge graphs, entity linking, and provenance tracking.

Example schema:

CREATE CONSTRAINT FOR (e:Entity) REQUIRE e.id IS UNIQUE;

// Nodes
MERGE (:Entity {id: 'project-1', type: 'Project', name: 'Project Alpha'});
MERGE (:Entity {id: 'person-1', type: 'Person', name: 'Alice'});
MERGE (:Entity {id: 'doc-1', type: 'Document', title: 'Architecture v2'});

// Relationships
MATCH (p:Entity {id: 'person-1'}), (d:Entity {id: 'doc-1'})
MERGE (p)-[:AUTHORED]->(d);

MATCH (p:Entity {id: 'person-1'}), (pr:Entity {id: 'project-1'})
MERGE (p)-[:WORKS_ON]->(pr);

Wiring them together

Data flows in two directions: write-through and read-merge.

Write path

1. Agent produces a memory event (e.g., "Alice authored doc-1").
2. Write the event to Postgres memory_events.
3. Embed the event text (if needed) and upsert into Qdrant.
4. Upsert the entities and relationships into Neo4j.

Use a transactional outbox pattern for reliability: write to Postgres first, then push to a queue (e.g., NATS) for async indexing into Qdrant and Neo4j. If the async step fails, the data is safe in Postgres.

Read path

1. Agent asks "find documents related to Project Alpha that mention the architecture."
2. First, query Qdrant for semantic similarity on "architecture" filtered by relevant entities (optional).
3. Get the set of memory_event_ids from Qdrant results.
4. Fetch full events from Postgres.
5. Simultaneously, query Neo4j for graph traversal: "starting from Project Alpha, find documents via any path."
6. Merge the two result sets (intersection or union, depending on use case).

This merge step is where the pattern shines: Qdrant gives you semantic matches, Neo4j gives you relational matches, and Postgres gives you the raw data.

Code sketch

Here's a Python example using qdrant-client, psycopg2, and neo4j:

import psycopg2
from qdrant_client import QdrantClient
from neo4j import GraphDatabase

# Initialize clients
pg_conn = psycopg2.connect("dbname=agent_memory")
qdrant = QdrantClient("http://localhost:6333")
neo4j_driver = GraphDatabase.driver("bolt://localhost:7687")

def write_memory_event(agent_id, event_type, content):
    # 1. Postgres
    with pg_conn.cursor() as cur:
        cur.execute(
            "INSERT INTO memory_events (agent_id, event_type, content) VALUES (%s, %s, %s) RETURNING id",
            (agent_id, event_type, json.dumps(content))
        )
        event_id = cur.fetchone()[0]
    pg_conn.commit()

    # 2. Qdrant (embedding generation omitted for brevity)
    embedding = generate_embedding(content['text'])
    qdrant.upsert(
        collection_name="agent_memory",
        points=[
            models.PointStruct(
                id=event_id.int,
                vector=embedding,
                payload={"agent_id": agent_id, "event_type": event_type}
            )
        ]
    )

    # 3. Neo4j (if content contains entities/relationships)
    with neo4j_driver.session() as session:
        session.run(
            "MERGE (e:Entity {id: $eid}) SET e.type = $type, e.name = $name",
            eid=content['entity_id'], type=content['entity_type'], name=content['entity_name']
        )
    return event_id

When to use this pattern

This pattern is overkill for simple chatbots. Use it when:

• Your agent needs to recall specific facts with high precision.
• You need to trace provenance across multiple interactions.
• You're building multi-agent systems where agents share memory.
• You need to comply with data sovereignty (all stores can run on-prem).

Operational notes

• Run all three databases on the same Docker network for low latency.
• Use pgbouncer for Postgres connection pooling.
• Qdrant's optimizers config: set default_segment_number to 2 for small deployments.
• Neo4j: use heap memory tuning, not just page cache.
• Back up Postgres daily; Qdrant and Neo4j can be rebuilt from Postgres if needed.

The takeaway

Don't compromise. Postgres for facts, Qdrant for similarity, Neo4j for relationships. Three stores, one coherent memory system. Your agents will thank you.