Schema Migrations Across Postgres, Qdrant, and Neo4j

A schema change in a polyglot persistence stack is rarely local. You add a field to a document, and suddenly the relational row that owns it, the vector payload that filters on it, and the graph property that connects it all need to agree. None of the three stores share a transaction manager. There is no two-phase commit you would actually want to run. So the migration becomes a distributed systems problem disguised as a DDL script.

The RiNET stack uses three Hetzner servers connected over a WireGuard mesh, with Postgres as the system of record, Qdrant for vector search, and Neo4j for relationship traversal. Embeddings come from BGE-M3, served alongside Qwen on vLLM. That layout is not special — it is the same shape most retrieval-augmented and agentic systems converge on. The problem is also the same shape.

The three stores fail in different ways

Postgres gives you transactional DDL. You can run ALTER TABLE inside a transaction and roll it back if a later step fails. Qdrant and Neo4j do not offer that in the same way, and even if they did, they live on different machines with independent failure domains. A network partition between the WireGuard peers is enough to leave you half-migrated.

The practical consequence: design every migration so that each store reaches a valid state on its own, and the system tolerates the intermediate states where they disagree.

-- Postgres: the system of record gets the new column first,
-- nullable, with no default rewrite on large tables.
ALTER TABLE documents
  ADD COLUMN tenant_id uuid;

CREATE INDEX CONCURRENTLY idx_documents_tenant
  ON documents (tenant_id);

CREATE INDEX CONCURRENTLY cannot run inside a transaction block. That is a feature, not an inconvenience — it forces you to think about what happens if the migration process dies between the column add and the index build.

Expand, migrate, contract

The pattern that survives contact with three stores is the same one that works for zero-downtime single-database migrations, stretched across services: expand, backfill, switch reads, contract.

1. Expand: add the new field everywhere, nullable or optional, without changing any reader.
2. Backfill: populate the new field from existing data, in batches, idempotently.
3. Switch: move reads to the new field, keep writing both for a window.
4. Contract: stop writing the old field, then drop it after a soak period.

Each step is independently resumable. If the backfill job dies at 40 percent, you restart it and it picks up where it left off because the query is keyed on a monotonic cursor, not an offset.

# Backfill cursor pattern. Store last_id in a small control table.
# Never use OFFSET — rows shift under concurrent writes.
last_id = load_cursor("tenant_backfill")
while True:
    rows = pg.execute(
        """SELECT id, tenant_id FROM documents
           WHERE id > %s AND tenant_id IS NULL
           ORDER BY id LIMIT 500""",
        (last_id,),
    ).fetchall()
    if not rows:
        break
    for row in rows:
        qdrant.set_payload(row.id, {"tenant_id": row.tenant_id})
        neo4j.run(
            "MATCH (d:Document {id: $id}) SET d.tenant_id = $t",
            id=str(row.id), tenant_id=str(row.tenant_id),
        )
    last_id = rows[-1].id
    save_cursor("tenant_backfill", last_id)

The order matters. Postgres is authoritative, so it is read first. Qdrant and Neo4j are derived, so they are written from Postgres values, never the reverse. If a derived store write fails, you log the id and retry; you do not fail the batch, because the cursor has already advanced past it. Keep a dead-letter table for ids that fail repeatedly.

Qdrant payloads are not schemaless in practice

Qdrant will accept arbitrary JSON payloads, which makes it tempting to treat as a dumping ground. The catch is filtering. If you filter on tenant_id and the payload index does not exist for that field, queries fall back to full scans or reject the filter depending on version and configuration. Payload indexes are the migration target, not the payload itself.

PUT /collections/documents/index
Content-Type: application/json

{
  "field_name": "tenant_id",
  "field_schema": "keyword"
}

Create the payload index before you backfill, not after. A backfill that writes a million payloads into an unindexed field, followed by a filter-heavy read path, is a self-inflicted outage. Index creation on a large collection is not instant, so schedule it as its own step with its own rollback story.

If the migration changes vector dimensionality or the embedding model, you cannot patch payloads. You build a new collection, dual-write during the transition, and flip the alias.

# New collection with the new dimension, then atomic alias swap.
PUT /collections/documents_v2

POST /collections/aliases
{
  "actions": [
    {"delete_alias": {"alias_name": "documents"}},
    {"create_alias": {"collection_name": "documents_v2", "alias_name": "documents"}}
  ]
}

Dual-writing embeddings means every ingest path now calls the embedding model twice or stores two vectors. With BGE-M3 served on vLLM, that is a throughput question, not a correctness one — but it is a real cost during the transition window. Plan the window to be short and the rollback to be a single alias swap back.

Neo4j wants property changes, not schema rewrites

Neo4j is closer to schema-optional than the other two. Adding a property to a node label is free. The expensive part is the constraint and index you attach to it, and the traversal patterns that assume it exists.

CREATE CONSTRAINT document_id IF NOT EXISTS
FOR (d:Document) REQUIRE d.id IS UNIQUE;

CREATE INDEX document_tenant IF NOT EXISTS
FOR (d:Document) ON (d.tenant_id);

IF NOT EXISTS makes these idempotent, which matters when your migration runner retries. Unlike Postgres, Neo4j index creation does not block writes in current versions, but it does consume resources. On a shared server that is also running other workloads, that shows up as latency in unrelated queries.

The harder problem is the graph shape itself. If the migration changes relationships — say, documents move from belonging to a user to belonging to a tenant — you cannot do that with a property update. You are rewriting edges, and that is a batched job with the same cursor discipline as the Postgres backfill.

// Batched edge rewrite. Run repeatedly until zero rows returned.
MATCH (d:Document)-[r:OWNED_BY]->(u:User)
WHERE d.tenant_id IS NOT NULL
  AND NOT (d)-[:BELONGS_TO]->(:Tenant {id: d.tenant_id})
WITH d, u LIMIT 1000
MATCH (t:Tenant {id: d.tenant_id})
MERGE (d)-[:BELONGS_TO]->(t)
RETURN count(d);

MERGE is not free. On a hot label it takes locks, and concurrent writers can deadlock against it. Run edge rewrites during low-traffic windows or serialize them against the ingest path with an application-level lock.

Reconciliation is the only real consistency guarantee

There is no point in pretending the three stores will stay consistent through a migration. They will not. The question is how fast you detect drift and how cheaply you repair it.

A reconciliation job that runs after the migration, and periodically forever after, compares a sample of ids across all three stores and reports mismatches. It does not need to be exhaustive to be useful — sampling catches systematic bugs, which are the ones that matter.

-- Postgres side of the reconciliation query.
SELECT d.id, d.tenant_id
FROM documents d
WHERE d.updated_at > now() - interval '1 hour'
ORDER BY random()
LIMIT 1000;

Feed those ids into Qdrant payload fetches and Neo4j property reads, compare, and write mismatches to a repair queue. The repair direction is always Postgres to derived store. Never let a reconciliation job "fix" Postgres from Qdrant — that inverts your source of truth and turns a bug into data loss.

What actually goes wrong

A few failure modes recur often enough to plan around explicitly.

Ordering. If you backfill Qdrant before the payload index exists, filters silently underperform. If you build the Neo4j index before the property exists on most nodes, the index is nearly empty and traversals fall back to scans. Sequence the steps so each store is in a usable state before the next begins.

Partial writes. A batch that writes 500 rows to Postgres, 480 to Qdrant, and 490 to Neo4j leaves 20 ids unaccounted for. The cursor pattern above hides this if you advance the cursor regardless of derived-store success. That is deliberate — the dead-letter table is where those ids live, and a separate repair job drains it. Do not silently drop them.

Rollback. Rolling back a Postgres column add is trivial. Rolling back a Qdrant collection swap is an alias flip. Rolling back a Neo4j edge rewrite is not possible without the original edges, so if the migration is destructive, snapshot first or write the old edges to a shadow relationship type before rewriting.

Read paths during transition. Application code must tolerate both old and new shapes for the duration of the migration. That means feature flags or version checks in the read path, and it means the migration is not done until the flags are removed. A migration that leaves permanent branching in the codebase is a migration that never finished.

A migration runner that assumes failure

The runner itself should be boring. Each step is a function with an idempotent implementation and a recorded completion state. The runner resumes from the last incomplete step. It never assumes the previous step finished cleanly, because on a three-server mesh with a WireGuard link in the middle, sometimes it did not.

STEPS = [
    "pg_add_column",
    "pg_create_index_concurrently",
    "qdrant_create_payload_index",
    "neo4j_create_index",
    "backfill_pg_to_qdrant",
    "backfill_pg_to_neo4j",
    "reconcile_sample",
    "switch_reads",
    "soak",
    "contract",
]

for step in STEPS:
    if is_done(step):
        continue
    run_step(step)          # must be safe to run twice
    mark_done(step)

The soak step is not a no-op. It is a deliberate delay measured in days, during which reconciliation runs and the old field is still written. Contracting too early is the single most common way these migrations cause incidents, because the rollback path was deleted before anyone confirmed the new path worked under production traffic.

The shape of the answer

Three stores, no shared transaction, and a migration that must survive partial failure. The answer is not a clever distributed commit protocol. It is a system of record, a one-directional data flow, idempotent steps, and a reconciliation loop that assumes drift will happen. Postgres leads. Qdrant and Neo4j follow. Every step is resumable. Nothing is contracted until it has soaked.

That is more work than a single ALTER TABLE. It is also the only version that survives a dropped packet halfway through the backfill.