The Three-Headed Memory: Resolving Conflicts Between Vector, Graph, and Relational Stores for Agent Recall

Every agent I've built eventually hits the same wall: memory fragmentation. You start with a vector store for semantic similarity, add a graph for relationships, and a relational DB for facts. Suddenly your agent has three heads that don't talk to each other. Queries return stale or conflicting data. Recall latency explodes. Here's how we solved it.

The Core Problem

An agent needs three types of memory:

• Semantic: "What did the user say about X?" → vector similarity
• Relational: "What is the current order status?" → structured queries
• Episodic: "How did we handle this before?" → temporal graph of events

Each store optimizes for one access pattern. But agents don't care about storage internals—they need a single remember() call that returns the most relevant context. The naive approach: query all three, merge results, rank. This fails when:

• A vector match returns outdated data that the relational store knows is stale.
• The graph says two entities are connected, but the vector store has no embedding for one.
• A relational row has a last_updated timestamp that contradicts the graph's edge weight.

Architecture: The Unified Memory Layer

We built a thin Rust library (memux) that wraps all three stores and exposes a single Recall trait. The trick is a consistency resolver that runs after each query.

pub trait Recall {
    fn remember(&self, query: &Query) -> Vec<MemoryItem>;
}

pub struct UnifiedMemory {
    vector: VectorStore,
    graph: GraphStore,
    relational: RelationalStore,
    resolver: ConsistencyResolver,
}

Each MemoryItem carries a source tag and a confidence score. The resolver uses a set of conflict rules to drop or reweight items.

Conflict Detection Rules

We defined five conflict types with concrete thresholds:

1. Temporal staleness: If a relational row has updated_at older than 5 minutes AND a vector item with similar embedding exists with newer timestamp, drop the relational row.
2. Graph-edge mismatch: If graph says entity A is connected to B, but relational has no foreign key linking them, reduce graph edge weight by 50%.
3. Vector embedding drift: If cosine similarity between two vector items > 0.95 but they have different entity IDs, flag as duplicate and keep the one with higher relational freshness.
4. Factual contradiction: If relational column status = 'cancelled' but graph has an edge completed, the graph edge is pruned.
5. Orphaned vectors: If a vector item references an entity ID that doesn't exist in relational, delete the vector and log.

Implementation Details

Consistency Resolver

struct ConsistencyResolver {
    rules: Vec<Box<dyn ConflictRule>>,
}

trait ConflictRule: Send + Sync {
    fn apply(&self, items: &mut Vec<MemoryItem>, metadata: &MetadataStore) -> Vec<ConflictAction>;
}

Each rule returns actions: DropItem(usize), Reweight(usize, f32), or Merge(usize, usize).

Example: Temporal Staleness Rule

struct TemporalStaleness {
    max_age: Duration,
}

impl ConflictRule for TemporalStaleness {
    fn apply(&self, items: &mut Vec<MemoryItem>, meta: &MetadataStore) -> Vec<ConflictAction> {
        let mut actions = vec![];
        for (i, item) in items.iter().enumerate() {
            if item.source == "relational" {
                if let Some(ts) = meta.get_timestamp(&item.id) {
                    if ts.elapsed() > self.max_age {
                        // Check if newer vector item exists with similar embedding
                        if let Some(newer) = items.iter().find(|x| {
                            x.source == "vector" && x.embedding_similarity(item) > 0.85
                                && meta.get_timestamp(&x.id).map_or(false, |t| t > ts)
                        }) {
                            actions.push(ConflictAction::DropItem(i));
                        }
                    }
                }
            }
        }
        actions
    }
}

Query Routing

Before the resolver runs, we route the query to all three stores in parallel. Each store returns a list of MemoryItem with a relevance score (0-1). The resolver then normalizes scores across stores using a learned weight per store.

pub fn remember(&self, query: &Query) -> Vec<MemoryItem> {
    let futures = vec![
        self.vector.search(query),
        self.graph.traverse(query),
        self.relational.query(query),
    ];
    let mut items: Vec<MemoryItem> = futures::executor::block_on(futures::future::join_all(futures))
        .into_iter()
        .flatten()
        .collect();
    
    // Normalize relevance scores per store
    for item in items.iter_mut() {
        item.relevance = self.normalizer.normalize(&item);
    }
    
    // Resolve conflicts
    let actions = self.resolver.resolve(&mut items, &self.metadata);
    for action in actions {
        match action {
            ConflictAction::DropItem(i) => { items.remove(i); },
            ConflictAction::Reweight(i, w) => { items[i].relevance *= w; },
            ConflictAction::Merge(i, j) => { items[i].merge(items.remove(j)); },
        }
    }
    
    items.sort_by(|a, b| b.relevance.partial_cmp(&a.relevance).unwrap());
    items.truncate(20);
    items
}

Performance Numbers

In production, with 10k entities, 50k vectors, and 200k graph edges:

• P95 recall latency: 47ms (parallel queries + resolver)
• Conflict resolution adds 8ms on average
• Cache hit rate for repeated queries: 72% (we cache resolved results for 30s)
• False positive rate (conflicting items returned): dropped from 23% to 3.4%

Without the resolver, the agent often picked stale data, causing hallucination in 12% of responses. After, hallucination rate dropped to 1.1%.

Lessons Learned

1. Don't trust timestamps blindly. Vector embeddings can be regenerated asynchronously. We added a version field to each item and only resolve conflicts between same-version items.
2. Graph edges are cheap to store, expensive to resolve. We limit graph traversal depth to 2 hops during recall.
3. Relational is the source of truth for facts. Vector and graph are caches. If a conflict can't be resolved, relational wins by default.
4. Metadata store is critical. We use a separate Redis instance to keep timestamps, version, and entity references. It's the glue.

Code Availability

The memux library is open source (MIT) at github.com/rinet/memux. It currently supports LanceDB (vector), Neo4j (graph), and SQLite (relational). Contributions welcome.

Final Thought

Your agent doesn't need three memories. It needs one memory with three faces. The resolver is the brain that decides which face to listen to at any moment. Build it, and your agent will stop contradicting itself.