Vector Store Upserts: Stop Silent Metadata Overwrites

A full upsert in a vector store replaces the entire payload, not just the fields you send. That means any metadata added by another process—tags, timestamps, access counts—is silently erased. The fix is a partial update pattern that merges fields instead of overwriting the record.

Why does a full upsert erase metadata?

Most vector stores treat an upsert as a complete record replacement. When you send a vector and a payload, the store writes that payload as the new truth for that ID. Fields not included in the request are dropped. This is by design: upsert means "insert or replace," not "insert or merge."

Consider a document ingested with metadata: {"source": "contract.pdf", "page": 3, "tags": ["legal"]}. Later, a re-embedding job upserts the same ID with only {"source": "contract.pdf"}. The page and tags fields vanish. Downstream filters that rely on tags now miss the document entirely.

This behavior is common across vector databases. Teams typically discover it after a metadata-dependent query returns incomplete results—often days after the overwrite occurred.

What does a partial update look like in practice?

A partial update sends only the fields that changed and instructs the store to merge them. Not all vector stores support this natively. Some offer a set_payload or update operation that merges, while others require a read-modify-write cycle.

Here is a read-modify-write pattern in Python for a store that only supports full upserts:

from typing import Any, Dict

def partial_upsert(store, id: str, vector: list[float], payload_delta: Dict[str, Any]):
    existing = store.get(id)
    if existing is None:
        store.upsert(id=id, vector=vector, payload=payload_delta)
        return
    merged = {**existing.payload, **payload_delta}
    store.upsert(id=id, vector=vector, payload=merged)

This works but introduces a race condition: two concurrent partial updates can read the same base and overwrite each other's changes. For high-concurrency workloads, prefer stores with atomic merge operations.

Which vector stores support atomic merges?

Support varies. Some stores expose a dedicated update method that merges payloads atomically. Others treat upsert as a full replace and offer no merge primitive. Before designing your ingestion pipeline, check the store's documentation for:

• Update semantics: does update merge or replace?
• Atomicity guarantees: is the merge atomic under concurrent writes?
• Field-level operations: can you increment a counter or append to an array without reading first?

If the store lacks atomic merges, you must serialize writes per ID or accept eventual consistency with a read-modify-write loop.

How do you design payloads that survive overwrites?

Treat the payload as a set of independent fields owned by different processes. A common pattern is to namespace fields by owner:

{
  "ingest": {"source": "contract.pdf", "page": 3},
  "enrich": {"tags": ["legal"], "summary": "..."},
  "ops": {"last_accessed": "2025-01-01T00:00:00Z", "access_count": 42}
}

Each process updates only its namespace. A full upsert from the ingest job still overwrites the whole payload—so this alone does not solve the problem. But it makes the ownership boundary explicit and simplifies merging logic.

A stronger approach is to store metadata in a separate record or table keyed by the vector ID. The vector store holds only the embedding and a minimal payload for filtering. Metadata lives in a transactional database where partial updates are native. This adds a join at query time but eliminates silent overwrites.

What should you check before deploying an upsert pipeline?

Before shipping, verify the failure mode. Write a test that upserts a record with two fields, then upserts the same ID with one field, and asserts that the second field survives. If it does not, your pipeline has a silent data loss bug.

Also audit every code path that calls upsert. A re-embedding job, a backfill script, or a migration tool can each overwrite payloads written by another service. Document which fields each job owns and enforce it in code review.

The key decision: either use a store with atomic payload merges, or keep metadata outside the vector store. Mixing full upserts with shared metadata is a recipe for silent loss.

Next step: run the two-field upsert test against your current store and see what survives.

How do you migrate existing data without losing metadata?

If you have already experienced overwrites, recovery is manual. You need a source of truth for the lost fields—often an upstream database or a backup. Rebuild the payload by joining the vector store IDs with the source records, then re-upsert with the full merged payload.

To prevent recurrence, add a schema check to your ingestion code. Reject any upsert that omits fields marked as required. This turns a silent overwrite into a loud error.

REQUIRED_FIELDS = {"source", "page", "tags"}

def validate_payload(payload: dict):
    missing = REQUIRED_FIELDS - payload.keys()
    if missing:
        raise ValueError(f"Missing required fields: {missing}")

This is a blunt instrument but effective for pipelines where every writer should send the full payload. For partial updates, validate only the fields the writer owns.

Conclusion

Full upserts replace payloads wholesale; partial updates merge. The difference is silent data loss versus predictable metadata evolution. Choose a store with atomic merges, or keep metadata in a transactional store and join at query time. Test the overwrite behavior before it bites you in production.

What does your current vector store do when you upsert an existing ID with a smaller payload? Share your findings in the comments.