Multi-Modal Memory: Merging Graph and Vector Stores for Agent Recall

Every agent I've built eventually runs into the same wall: a single vector store is not enough. You throw a user query at the embedding index, get back the top-k chunks, and hope the LLM pieces together the relationships. But relationships are not cosine distances. The fact that Document A was written by User B and references Project C is a graph fact, not a vector fact. If your agent needs to recall "what did Alice say about the deployment last Tuesday?" you need both semantic similarity (the content of Alice's messages) and relational traversal (Alice → messages → deployment context).

At RiNET, we run a three-server Hetzner cluster with a WireGuard mesh. Our stack includes Qdrant for vector storage and Neo4j for graph storage, both backed by Postgres for metadata. The two stores are not redundant—they are complementary. Here’s how we merge them for agent memory that actually works.

The Problem with Pure Vector Recall

Vector databases are great for unstructured similarity search. You embed a query, find the nearest neighbors in embedding space, and return the associated text. But agents need more than "these paragraphs are semantically close." Consider a support agent that must remember:

• Which tickets a specific customer has opened
• The resolution steps for a recurring issue
• The escalation path for a particular error code

A pure vector store can retrieve chunks about "network timeout" or "SSL handshake failure," but it cannot answer "which tickets did customer X open that involve network issues?" without either:

1. Embedding the entire ticket metadata into the vector (bloating the index with non-semantic data)
2. Performing multiple round-trips: retrieve candidate tickets by customer ID from a relational DB, then re-embed and search

Both approaches are brittle. The first pollutes the embedding space with identifiers that have no semantic meaning. The second requires custom orchestration code that breaks the abstraction of a unified memory.

The Hybrid Approach: Graph as Relational Index, Vector as Content Store

We decouple the memory into two layers:

• Graph layer (Neo4j): stores entities (users, documents, sessions, tasks) and their relationships (authored_by, references, belongs_to, occurred_after). Each node has a unique ID and a type label.
• Vector layer (Qdrant): stores dense embeddings of text chunks, where each chunk is linked to one or more graph nodes via metadata (the node ID).

The agent’s recall flow becomes:

1. Parse the user query to extract both semantic intent and relational constraints (e.g., "Alice's deployment notes").
2. Query Neo4j for the node IDs of relevant entities (e.g., find node with label User and property name = "Alice", then find nodes connected by authored edge with label DeploymentNote).
3. Use those node IDs as filters in Qdrant: must condition on node_id field, plus the semantic vector query.
4. Return the intersection: chunks that are both semantically relevant and relationally connected.

This avoids the need to embed relational information. The graph handles the "who, what, when, where" while the vector store handles the "what does it mean."

Implementation Sketch

Below is a simplified Python snippet showing the dual query pattern. We use the Qdrant client and the Neo4j Python driver.

from qdrant_client import QdrantClient
from qdrant_client.http.models import Filter, FieldCondition, MatchValue
from neo4j import GraphDatabase

class MultiModalMemory:
    def __init__(self, qdrant_host, neo4j_uri, neo4j_user, neo4j_password):
        self.qdrant = QdrantClient(host=qdrant_host, port=6333)
        self.neo4j_driver = GraphDatabase.driver(neo4j_uri, auth=(neo4j_user, neo4j_password))

    def recall(self, query_text, entity_type, entity_name, top_k=5):
        # Step 1: resolve entity in graph
        with self.neo4j_driver.session() as session:
            result = session.run(
                "MATCH (n:" + entity_type + " {name: $name}) "
                "OPTIONAL MATCH (n)-[:AUTHORED|REFERENCES|BELONGS_TO]->(m) "
                "RETURN n.id AS root_id, collect(m.id) AS related_ids",
                name=entity_name
            )
            record = result.single()
            if not record:
                return []
            root_id = record["root_id"]
            related_ids = record["related_ids"]

        # Step 2: build Qdrant filter with all relevant node IDs
        all_ids = [root_id] + (related_ids if related_ids else [])
        query_filter = Filter(
            should=[
                FieldCondition(key="node_id", match=MatchValue(value=nid))
                for nid in all_ids
            ]
        )

        # Step 3: vector search with filter
        hits = self.qdrant.search(
            collection_name="agent_memory",
            query_vector=self._embed(query_text),
            query_filter=query_filter,
            limit=top_k
        )
        return [hit.payload["text"] for hit in hits]

This is a minimal example. In production, we extend the graph traversal to multiple hops (e.g., "documents authored by people in Alice's team") and use composite filters in Qdrant. The key insight: the graph query prunes the search space before the vector search ever runs, reducing latency and noise.

Synchronizing Both Stores

The hard part is keeping the two stores consistent. When an agent writes a new memory chunk:

1. Insert the chunk text into Postgres (source of truth).
2. Create or update the relevant graph nodes and edges in Neo4j.
3. Embed the chunk and upsert into Qdrant with the node ID as a payload field.

We use a nightly LoRA fine-tuning job that also rebuilds embeddings for any chunks whose text changed. The graph is updated in real-time via the agent's action stream. Consistency is eventual: if a Neo4j node is deleted, Qdrant may still have stale vectors. We handle this with a periodic reconciliation job that scans Qdrant for orphaned node IDs and removes them.

Why Not Just Use a Single Multi-Modal DB?

There are databases that combine vector and graph (e.g., Neo4j 5.x with vector index, or ArangoDB with vector search). We evaluated them but found that the graph+vector hybrid in a single system often sacrifices performance in one dimension. Neo4j's vector index is still young and lacks the fine-grained filtering and scalability of dedicated vector stores. Qdrant, on the other hand, has no native graph traversal. By keeping them separate, we can tune each independently: more powerful graph queries (Cypher path patterns) and more efficient vector search (HNSW with payload filters).

The cost is operational complexity. We run three containers: Postgres, Neo4j, Qdrant. The WireGuard mesh keeps traffic between them low-latency. Total memory usage on our Hetzner servers is about 8 GB for the three databases (with Qdrant holding ~500K vectors).

Real-World Query Patterns

Here are three patterns we use daily:

Pattern 1: Contextual Retrieval with Entity Filter

"Show me Alice's comments about the deployment rollback."

• Graph query: find node User:Alice, traverse AUTHORED edges to Comment nodes, collect their IDs.
• Vector query: embed "deployment rollback" and search with filter on node_id IN [list of Comment IDs].

Pattern 2: Temporal Graph Traversal

"What did we discuss in the last three standups about the database migration?"

• Graph query: find nodes of type Meeting where date > now - 3 days, traverse HAS_TOPIC edges to Topic nodes with name "database migration", collect meeting node IDs.
• Vector query: embed the query and search with filter on node_id IN meeting_ids.

Pattern 3: Multi-Hop Reasoning

"Find the document that was referenced by the ticket Alice opened about the API timeout."

• Graph: User:Alice → OPENED → Ticket (with description containing "API timeout") → REFERENCES → Document.
• Vector: no vector query needed—the document content is retrieved from Postgres via the node ID. But if you need semantic search within that document, you'd then use a vector filter on that document's ID.

Tradeoffs and When It Breaks

This architecture is not free. The main pain points:

• Graph schema design matters. If your graph is too deep (many node types and edges), the traversal becomes expensive. We limit traversals to 3 hops and use indexes on frequently queried properties (name, date).
• Vector filter cardinality. Qdrant's payload filter works best when the number of distinct values per field is low. If you filter on node_id with thousands of IDs, performance degrades. We batch filters into chunks of 100 IDs and merge results.
• Consistency lag. The reconciliation job runs every 10 minutes. If an agent deletes a node and immediately queries, it may get stale vectors. We mitigate by marking nodes as deleted in Neo4j and filtering them in Qdrant via a status field, rather than actually removing them until the next reconciliation.
• Embedding drift. The nightly LoRA fine-tuning may change the embedding model. Old vectors become incompatible. We version the embedding model in the Qdrant collection metadata and re-embed all chunks when the model changes. This is a batch job that takes about 30 minutes for 500K vectors.

Final Thoughts

Merging graph and vector stores is not about building a perfect unified memory—it's about matching the retrieval strategy to the query structure. If your agent only ever needs "find me similar paragraphs," a vector store alone is fine. But if your agent needs to reason about entities, relationships, and temporal sequences, you need a graph. The two together form a multi-modal memory that lets the agent navigate both semantic space and relational space.

At RiNET, we run this stack on three Hetzner servers with 64 GB RAM each, using WireGuard for inter-service communication. The setup is not trivial, but it's the difference between an agent that guesses and an agent that recalls.