systemd Timer Drift: Why Nightly Reindex Ran Twice

A nightly reindex that runs twice can corrupt a vector index. The usual cause is systemd timer drift combined with a non-idempotent job. Teams typically discover this after a duplicate write or a partially built index. The fix is two-part: make the job idempotent and make the timer self-healing.

Why does a systemd timer drift at all?

systemd timers are not cron. OnCalendar= schedules are evaluated against the real-time clock, but a timer that fires late does not shift the next fire time. If a job runs long or the host suspends, the next scheduled activation can land close to the previous one. With Persistent=true, systemd also runs a missed activation after boot. That combination can produce two runs in quick succession.

A common pattern is a nightly job at 02:00 with Persistent=true. The host reboots at 01:55. systemd runs the missed job at boot, then the regular timer fires at 02:00. Two reindexes start within minutes. If the job writes to the same index path, the second run can read a half-written index.

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true

Persistent=true is useful for laptops and hosts that sleep. On always-on servers it is often the source of duplicate runs. Decide per host whether missed runs matter more than duplicate risk.

What makes a reindex job non-idempotent?

Idempotency means running the job twice produces the same result as running it once. Most reindex scripts fail this test for three reasons:

• They write in place instead of building a new index and swapping it.
• They append or upsert without a stable document key.
• They assume exclusive access to the index directory.

The safe pattern is build-then-swap. Write the new index to a temporary path, validate it, then atomically rename or symlink it into place. If the job dies mid-build, the live index is untouched.

#!/usr/bin/env bash
set -euo pipefail

INDEX_ROOT=/var/lib/reindex
LIVE="$INDEX_ROOT/live"
BUILD="$INDEX_ROOT/build.$$"

trap 'rm -rf "$BUILD"' EXIT

reindex --output "$BUILD"
validate-index "$BUILD"

# Atomic swap: rename is atomic on the same filesystem.
rm -rf "$INDEX_ROOT/previous"
mv "$LIVE" "$INDEX_ROOT/previous" 2>/dev/null || true
mv "$BUILD" "$LIVE"
trap - EXIT

The mv swap is atomic only within one filesystem. Keep build and live under the same mount. If they are on different mounts, use a symlink swap instead and accept that readers must resolve the link once.

How do you stop two runs from overlapping?

systemd offers two mechanisms. Conflicts= prevents two units from running at once. A lock file inside the script is more portable and survives manual runs.

[Unit]
Description=Nightly vector reindex
Conflicts=reindex.service

Conflicts= is simple but coarse: a manual systemctl start will stop the timer-driven run. A flock lock is finer-grained and works for cron, CI, and manual invocations alike.

exec 9>/var/lock/reindex.lock
flock -n 9 || { echo "reindex already running"; exit 0; }

Exiting 0 on lock contention is deliberate. A skipped run is not a failure. If you exit non-zero, systemd may mark the unit failed and trigger OnFailure= handlers that add noise.

Where does self-healing fit in?

Self-healing does not mean retrying blindly. It means detecting a bad state and repairing it without human intervention. For a vector index, the cheapest self-healing check is a manifest file written at the end of a successful build. On startup, the job verifies the manifest matches the live index. If not, it rebuilds.

if ! verify-manifest "$LIVE"; then
  echo "live index missing or stale; rebuilding"
  rebuild-from-source
fi

A manifest should record the source snapshot ID, the build timestamp, and a checksum of the index files. Without a source snapshot ID, a rebuild may silently use newer data than the previous run, which breaks reproducibility.

The manifest is the contract between the timer and the index. If the timer fires twice, the second run sees a valid manifest and exits. If the first run died mid-build, the second run rebuilds from scratch. Either way, the live index is never half-written.

What should the timer unit actually look like?

Keep the timer dumb and the service defensive. The timer decides when; the service decides whether. Put the idempotency logic in the service, not in the schedule.

[Unit]
Description=Nightly vector reindex
After=network-online.target
OnFailure=reindex-alert@%n.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/reindex-safe
Nice=10
IOSchedulingClass=idle

Type=oneshot is correct for batch jobs. It tells systemd the service is done when the process exits, so Conflicts= and After= behave predictably. Avoid Type=simple for reindex jobs; it marks the unit started before the work is done.

Before you change anything, check the journal for duplicate activations: journalctl -u reindex.service --since yesterday. Look for two Starting lines within a short window. That confirms drift or a missed-run replay.

Which trade-offs should you accept?

Every choice here has a cost:

• Persistent=true buys missed-run recovery and costs duplicate-run risk.
• Conflicts= buys mutual exclusion and costs the ability to run manually during a scheduled window.
• Build-then-swap buys safety and costs disk space for two index copies.
• A manifest buys self-healing and costs one more file to keep consistent.

For a nightly job on an always-on host, drop Persistent=true and rely on the manifest to catch missed runs. For a laptop or an edge node, keep it and rely on the lock file to serialize duplicates. There is no single correct answer; the point is to choose deliberately.

What to do next

Open your reindex unit and check three things: whether Persistent=true is set, whether the script writes in place, and whether it takes a lock. Fix the cheapest one first. Then run the job twice by hand and confirm the second run is a no-op.

If you have a timer that fired twice in production, what did the journal show: a missed-run replay or a long-running job overlapping the next window? Reply with the journalctl excerpt and I will look at the schedule logic with you.