Three-Headed Memory: Resolving Cross-Store Conflicts for Agent Recall

Every serious agent swarm eventually hits the same wall: a single memory store is not enough. Vector databases give you semantic similarity but no relationships. Graph databases give you relationships but weak similarity search. Relational databases give you consistency and transactions but no semantics. Teams building production agents typically end up running all three, and then the real problem starts: the same fact lives in three places, and the three copies disagree.

This article is about that disagreement. Not the theory of multi-store memory, but the concrete engineering of keeping Postgres, Qdrant, and Neo4j in rough alignment when your agents are writing to all of them asynchronously. I'll walk through the conflict taxonomy, a resolution pipeline, and the tradeoffs you accept when you choose eventual consistency over a distributed transaction.

Why Three Stores at All?

Before we resolve conflicts, let's justify the mess. A single store can't cover the three access patterns agents need.

Vector store (Qdrant, in our stack): Fast approximate nearest neighbor search over embeddings. This is how an agent recalls "things similar to this document" or "past conversations about X." The vector store has no notion of subject-object-predicate; it's a bag of vectors with payloads.

Graph store (Neo4j): Explicit nodes and edges. This is how an agent reasons about "who knows whom," "which task depends on which," or "what caused what." Graph queries are precise and relationship-aware, but they don't do fuzzy similarity.

Relational store (Postgres): The source of truth for structured facts, audit logs, and transactional state. Postgres gives you ACID, but you'd never run a similarity search on text columns at scale.

Each store is optimized for a different query shape. The problem is that they all describe the same underlying reality. A user's preference, a project's status, a tool's output — these are single facts that get duplicated across all three representations. And because agents write to them at different times and through different code paths, the copies drift.

The Conflict Taxonomy

Not all conflicts are the same. In practice, teams see three distinct flavors:

1. Temporal conflicts: The same fact was updated at different times in different stores. The vector embedding was generated from an old version of the document, while the relational row has the new text. The graph edge was created before the node properties were updated.

2. Semantic conflicts: The stores disagree on what the fact is. The relational table says the project status is "blocked," but the graph has an edge project -[:DEPENDS_ON]-> other_project that implies it's waiting. The vector payload contains a summary that contradicts the structured status.

3. Identity conflicts: The same real-world entity has different IDs in different stores. The Postgres row has user_id = 42, the graph node has userId: "42", and the vector payload has user_ref: "user-42". If any of those mappings break, recall returns the wrong entity.

You can't resolve conflicts until you can detect them. That requires a unified view of what a "fact" is.

The Entity-Attribute-Value (EAV) Canonical Layer

Our approach is to define a canonical, store-agnostic representation of every fact. We call it the fact envelope. It's a simple JSON structure that looks like this:

{
  "entity_type": "project",
  "entity_id": "proj-123",
  "attribute": "status",
  "value": "blocked",
  "timestamp": "2025-03-14T10:30:00Z",
  "source": "task-manager-agent",
  "version": 7
}

Every write to any store goes through a single service that first creates or updates this envelope in Postgres (the system of record), then asynchronously propagates the change to the vector and graph stores. The envelope is the source of truth for the fact; the stores are just indexes.

This is a classic CQRS/event-sourcing pattern. The write path is a command that updates the envelope and emits an event. The read path queries the appropriate index. The conflict resolution happens when the indexes lag behind the envelope.

The Write Path: Event Sourcing for Facts

Concretely, the write path looks like this:

# Pseudo-code for the write path
def write_fact(entity_type, entity_id, attribute, value):
    # 1. Validate and write to Postgres (source of truth)
    fact_id = postgres.insert_fact_envelope(...)

    # 2. Emit an event to a message queue (e.g., Redis Streams)
    event_bus.publish("fact.updated", {
        "fact_id": fact_id,
        "entity_type": entity_type,
        "entity_id": entity_id,
        "attribute": attribute,
        "value": value,
        "timestamp": now_utc(),
    })

    # 3. Return immediately; consumers update the indexes
    return fact_id

Consumers for Qdrant and Neo4j subscribe to the event bus and update their respective stores. This gives you eventual consistency by design. The window of inconsistency is typically milliseconds, but under load it can stretch to seconds.

The Read Path: Conflict Detection on the Fly

When an agent queries memory, it rarely needs just one store. A typical recall request is: "Find projects similar to this document, and show me their dependencies and current status." That's a vector search, a graph traversal, and a relational lookup — three queries, three results.

The agent needs a single, coherent answer. We build that with a reconciliation layer that merges results and flags conflicts.

def reconcile(query, vector_hits, graph_nodes, relational_rows):
    # Merge by entity_id (after normalizing IDs)
    merged = {}
    for hit in vector_hits:
        entity_id = normalize_id(hit.payload["entity_id"])
        merged[entity_id] = {"vector": hit, "graph": None, "relational": None}
    for node in graph_nodes:
        entity_id = normalize_id(node["entity_id"])
        if entity_id not in merged:
            merged[entity_id] = {"vector": None, "graph": node, "relational": None}
        else:
            merged[entity_id]["graph"] = node
    # ... same for relational rows ...

    # Now detect conflicts per entity
    conflicts = []
    for entity_id, sources in merged.items():
        # Compare attribute values across sources
        if sources["vector"] and sources["relational"]:
            if sources["vector"]["status"] != sources["relational"]["status"]:
                conflicts.append({
                    "entity_id": entity_id,
                    "attribute": "status",
                    "vector_value": sources["vector"]["status"],
                    "relational_value": sources["relational"]["status"]
                })
    return merged, conflicts

This is the detection step. It doesn't resolve anything; it just surfaces the disagreement. The agent can then decide whether to trust one source over another, or to trigger a repair.

Resolution Strategies: Last-Write-Wins, Version Vectors, and Semantic Reconcile

Once a conflict is detected, you need a policy. The simplest is last-write-wins (LWW). Each fact envelope has a timestamp, and the store with the most recent timestamp wins. This works for temporal conflicts, but it fails for semantic ones — the latest write might be wrong.

A more robust approach is version vectors. Each store keeps a version counter per entity. When a conflict is detected, you compare the version vectors to determine causality. If one version dominates (all counters >= the other), you can safely merge. If they're concurrent, you need a merge policy.

For semantic conflicts, you need domain logic. For example, if the relational status says "blocked" and the graph says "has no blockers," you might run a validation rule that checks the actual dependency edges. If no blocker edges exist, the "blocked" status is stale, and you repair the vector and relational stores.

In practice, teams use a hybrid: LWW for simple attribute updates, version vectors for entity-level changes, and a rule engine for semantic conflicts.

The Repair Loop: Asynchronous Backfill

Detection and resolution are useless without a repair loop. We run a background worker that periodically scans for inconsistencies and backfills the lagging store.

# Pseudo-code for repair worker
def repair_loop():
    while True:
        # Find entities with recent updates in Postgres
        updated_entities = postgres.get_recently_updated(within_minutes=5)
        for entity in updated_entities:
            # Check if vector store is stale
            if vector_store.get_version(entity.id) < entity.version:
                vector_store.upsert(entity.id, embed(entity.text), entity.version)
            # Check if graph store is stale
            if graph_store.get_version(entity.id) < entity.version:
                graph_store.sync_entity(entity)
        sleep(10)

This is a simple polling loop. In production, you'd use the event bus to trigger repairs more directly, but the principle is the same: eventually, all stores converge.

Handling Identity Conflicts

The nastiest conflicts are identity ones. If the vector store has user_id: "42" and the graph has userId: "user-42", no amount of timestamp comparison will fix it. You need a canonical ID mapping.

Our approach is to store a mapping table in Postgres:

CREATE TABLE entity_id_map (
    canonical_id TEXT PRIMARY KEY,
    store_name TEXT NOT NULL,
    store_id TEXT NOT NULL,
    UNIQUE(store_name, store_id)
);

Every store write must go through a translation layer that looks up the canonical ID from the store-specific ID. If a store returns an ID that isn't in the map, it's a new entity and you create a canonical ID.

This mapping is the backbone of the reconciliation layer. Without it, you're comparing apples and oranges.

Practical Tradeoffs: Why Not Distributed Transactions?

You might wonder: why not use a distributed transaction to write to all three stores atomically? The answer is that it's almost never worth it. Distributed transactions across heterogeneous systems (Postgres, Qdrant, Neo4j) are slow, fragile, and often unsupported. Qdrant and Neo4j don't participate in XA transactions. You'd end up building a saga anyway.

Eventual consistency is the pragmatic choice. The window of inconsistency is small, and the repair loop catches stragglers. The key is to design your agents to tolerate stale reads. For example, an agent should not make a critical decision based solely on a vector search result without checking the relational source of truth.

What This Looks Like in Our Stack

In our stack at RiNET, we run three Hetzner servers connected via WireGuard. The memory layer is Postgres for facts, Qdrant for embeddings, and Neo4j for relationships. We use vLLM to serve Qwen models, and BGE-M3 for embeddings. Every night, we run a LoRA fine-tuning job that might update the embedding model, which means we have to re-embed a lot of content — that's a whole other source of drift.

The conflict resolution pipeline sits in a small Python service that listens to a Redis Stream. It's not glamorous, but it works. The key lesson is that you can't bolt on conflict resolution after the fact; you have to design for it from day one.

Conclusion

Multi-store memory is unavoidable for serious agent swarms. The vector-graph-relational triad each covers a distinct access pattern, but they will disagree. The solution is not to eliminate conflicts — that's impossible — but to detect them, resolve them with a clear policy, and repair the stores asynchronously.

Start with a canonical fact envelope in your relational store. Build an event-driven write path. Implement a reconciliation layer that merges results and flags conflicts. Run a repair loop. And above all, normalize your IDs across stores. Do that, and your agents will recall the right thing, most of the time.

The three-headed memory is a feature, not a bug — as long as you teach the heads to talk to each other.