Sub-5ms Recall: Hot RAM, Warm SSD, Cold Tape for Agent Memory

Semantic memory is the backbone of any agent that needs to recall facts, past decisions, or user preferences. But when your agent's memory grows past a few million vectors, a single in-memory vector index stops being a reasonable default. The classic answer is a memory hierarchy: hot data in RAM, warm data on SSD, cold data on tape. This isn't a new idea—databases have done it for decades—but applying it to vector search for agents introduces specific constraints and tradeoffs that are worth examining in detail.

In this post, I'll walk through a practical architecture for a hot-warm-cold semantic memory system. I'll focus on the engineering decisions: where to store what, how to move data between tiers, and how to keep queries fast without pretending that every query needs to hit RAM. I'll avoid marketing numbers; instead, I'll give you the reasoning and the code skeletons you need to build your own.

The Problem with Single-Tier Memory

Most vector databases default to keeping everything in RAM. That's great for latency—every query is a pure in-memory ANN search. But it's terrible for cost as your corpus grows. RAM is expensive, and for agent memories—which include raw text, embeddings, and metadata—you can easily hit tens or hundreds of gigabytes. At that point, you're paying for a lot of memory that's rarely touched.

Agents don't access all memories equally. Some are queried constantly (recent conversations, frequently referenced facts), while others are rarely looked up (archived logs, old project notes). A single-tier design forces you to either over-provision RAM for the hot set or accept that cold data slows down every query. Neither is good.

A memory hierarchy solves this by matching storage to access frequency. Hot data lives in RAM, warm data on SSD, cold data on tape (or object storage). The challenge is making the transitions seamless and the query routing correct.

The Hot-Warm-Cold Model for Vectors

For agent semantic memory, I define three tiers:

• Hot: A small, high-performance vector index (e.g., HNSW in RAM) that holds the most frequently accessed vectors. This is the tier that serves the majority of queries.
• Warm: A larger index on SSD, using a disk-based ANN index (e.g., HNSW with disk-backed graphs, or a product-quantized index). It's slower than RAM but still random-access, and it can hold millions of vectors at a fraction of the cost.
• Cold: Bulk storage on tape or object storage, holding the full history. No index—just raw vectors and metadata, retrievable on demand but with high latency (seconds to minutes).

The key insight is that you don't need to search cold data in real time. You only need to retrieve it when a query explicitly asks for it, or when a background process needs to refresh the warm tier.

Query Routing and Tier Selection

Every query starts at the hot tier. If the hot index returns results with sufficient confidence (e.g., top-1 similarity above a threshold), you're done. If not, you fall back to the warm tier. This is a classic cascade: fast path first, slower path only when needed.

How do you decide when to fall back? A common pattern is to use a threshold on the similarity score. If the best match in hot is below, say, 0.7 (cosine similarity), it's likely that the true match is in the warm tier. But thresholds are brittle; they depend on your embedding space. A better approach is to track query distribution and adapt the threshold dynamically, but that's a topic for another post.

Another pattern is to use a separate lightweight classifier to route queries: if the query contains keywords that match a known hot topic, go hot; otherwise, go warm. This is more deterministic but requires maintaining a mapping.

For cold data, you never query it directly in the request path. Instead, you have a separate retrieval API that is explicitly called when a user asks for "everything about X" or when a background job needs to rehydrate a memory. That API reads from tape, deserializes the vectors, and optionally re-embeds them if the embedding model changed.

Implementation Sketch: Hot Index in RAM

For the hot tier, you want an in-memory HNSW index. In Python, hnswlib is a solid choice—it's a C++ library with Python bindings, and it's fast. Here's a minimal setup:

import hnswlib

# Define index with cosine similarity
dim = 768  # e.g., BGE-M3 embedding size
index = hnswlib.Index(space='cosine', dim=dim)
index.init_index(max_elements=100000, ef_construction=200, M=16)

# Add vectors
index.add_items(vectors, ids)

# Query
labels, distances = index.knn_query(query_vector, k=10)

This gives you sub-millisecond queries on a few hundred thousand vectors—easily enough for a hot set. The ef parameter controls recall; you can tune it per query, but for a hot tier you want low latency, so keep ef small (e.g., 50) and rely on the warm fallback for accuracy.

Implementation Sketch: Warm Index on SSD

For the warm tier, you need a disk-based index. Options include FAISS with IndexIVFPQ (product quantization) or IndexHNSW with disk storage. FAISS is a good choice because it's battle-tested and supports both memory-mapped and on-disk indexes.

Here's a rough sketch using FAISS with IVF-PQ:

import faiss

# Build a coarse quantizer (e.g., 4096 centroids)
quantizer = faiss.IndexFlatIP(dim)
index = faiss.IndexIVFPQ(quantizer, dim, nlist=4096, M=32, nbits=8)

# Train on a sample of vectors
index.train(sample_vectors)

# Add vectors
index.add_with_ids(vectors, ids)

# Write to disk
faiss.write_index(index, 'warm.index')

# Later, load with memory mapping
index = faiss.read_index('warm.index', faiss.IO_FLAG_MMAP)

Memory-mapping the index means the OS loads pages on demand, so you don't need to load the whole thing into RAM. Queries will be slower than RAM—maybe a few milliseconds—but still acceptable for a fallback path.

One caveat: IVF-PQ introduces lossy compression. You need to balance M (number of subquantizers) and nbits to keep recall acceptable. A common practice is to use a separate reranking step: get a candidate set from the compressed index, then re-score with exact vectors loaded from disk. That adds complexity but improves accuracy.

Implementation Sketch: Cold Storage on Tape

Cold storage is the easiest part technically—it's just files. You can store vectors as binary blobs (e.g., numpy arrays) or in a columnar format like Parquet. Tape is the cheapest option, but it has high latency and is sequential. For most agent use cases, object storage (like S3) is a more practical "cold" tier because it's random-access and still cheap.

Here's a simple pattern: store each memory entry as a separate object keyed by ID, with a manifest file that maps IDs to file paths. When you need to retrieve a cold memory, you read the manifest, fetch the object, and deserialize.

import boto3
import numpy as np

s3 = boto3.client('s3')

def get_cold_vector(memory_id):
    # Fetch from S3 (or tape via a gateway)
    obj = s3.get_object(Bucket='agent-memories', Key=f'{memory_id}.npy')
    vector = np.frombuffer(obj['Body'].read(), dtype=np.float32)
    return vector

For tape, you'd use something like LTFS (Linear Tape File System) to make tapes appear as a file system, but the latency is still seconds to minutes. So cold retrieval is only for batch processes or explicit user requests.

Data Movement and Promotion/Demotion

A memory hierarchy is only useful if data moves between tiers based on access patterns. This is the classic cache management problem. You need a policy to decide when to promote a vector from warm to hot, and when to demote from hot to warm, and when to archive from warm to cold.

A simple LRU (Least Recently Used) policy works well for hot/warm promotion. You can track access timestamps in a separate table (e.g., in Postgres) and periodically scan for hot candidates. For demotion, you evict the least recently used vectors from the hot index. For archiving, you can use a TTL or a rule like "memories older than 6 months and not accessed in the last 30 days go to cold."

Here's a sketch of a promotion job:

import time

def promote_to_hot(candidate_ids):
    # Load vectors from warm index
    vectors = load_from_warm(candidate_ids)
    # Add to hot index
    hot_index.add_items(vectors, candidate_ids)
    # Update metadata in Postgres

def demote_from_hot():
    # Find LRU ids in hot
    lru_ids = get_lru_from_hot(limit=1000)
    # Remove from hot index
    hot_index.remove_ids(lru_ids)
    # They stay in warm, no need to write back

You also need to handle the case where a vector is updated (e.g., re-embedded with a new model). That means invalidating it in all tiers and re-inserting.

Consistency and Deletes

In a multi-tier system, consistency is a headache. If you delete a memory, you must remove it from all tiers. If you update a vector, you must update all copies. A common pattern is to use a single source of truth (e.g., Postgres) for metadata, and treat the indexes as caches. Every write goes through a service that updates Postgres and then asynchronously updates the relevant tiers.

For deletes, you can use a tombstone approach: mark the ID as deleted in Postgres, and have background jobs purge it from indexes. This avoids race conditions.

When Not to Bother

A hot-warm-cold hierarchy adds complexity. If your agent's memory is under a few hundred thousand vectors, just keep it all in RAM. The cost is manageable, and the simplicity is worth it. If you're building a small prototype, don't over-engineer. The hierarchy pays off when your corpus grows to millions of vectors and you need to balance latency against cost.

Also, consider whether you actually need cold storage. Many agent systems can get away with just hot and warm, using object storage as a backup. Tape is only relevant if you have regulatory requirements or truly massive archives.

Real-World Considerations

In practice, teams building agent memory systems often start with a single vector DB and then add caching layers. A common pattern is to use a vector database like Qdrant (which RiNET uses in its stack) as the warm tier, and a separate in-memory index for hot data. Qdrant supports disk-based indexes and payload filtering, which makes it a good warm candidate. For hot, you might use a simple hnswlib index in the application process.

The key is to measure your query distribution. If 90% of queries hit the hot tier, the fallback to warm is rare, and you can afford a slightly slower warm tier. If the distribution is flat, you might need a larger hot tier or a better routing strategy.

Conclusion

A hot-warm-cold memory hierarchy is a practical way to scale agent semantic memory without exploding costs. By keeping hot data in RAM, warm data on SSD, and cold data on tape or object storage, you get sub-5ms recall for common queries while retaining the ability to access the full history when needed. The implementation is straightforward if you use the right tools and design your query routing carefully.

The key takeaways are:

• Hot tier: in-memory HNSW, low latency, small size.
• Warm tier: disk-based ANN (FAISS or Qdrant), larger size, moderate latency.
• Cold tier: object storage or tape, no index, high latency, used for batch or explicit retrieval.
• Data movement: LRU for promotion, TTL for archiving, and a metadata store for consistency.

Start simple, measure your access patterns, and add complexity only where it pays off. Your agent's memory will thank you.