Simulating Daylight Savings Chaos: Stress-Testing Agent Swarms Against Temporal Edge Cases

Agent swarms are only as reliable as their weakest temporal link. A swarm that handles a million tasks perfectly during stable time will implode when the clock jumps forward or backward. Daylight saving time (DST) transitions are the most common—and most ignored—source of temporal edge cases. We built a test harness that systematically breaks agents by warping time, and here's exactly how you can do the same.

Why Temporal Edge Cases Matter

Agent swarms rely on time for:

• Scheduling: cron-like triggers for periodic tasks.
• Timeouts: RPC calls between agents that must complete within a window.
• Ordering: tasks sorted by created_at or scheduled_for.
• Heartbeats: leader election and failure detection.
• Retries: exponential backoff with jitter, usually anchored to now().

DST transitions introduce two distinct failure modes:

1. Spring forward (lose one hour): Tasks scheduled at 2:30 AM never run. Timeouts that fire after 1:59 AM but before 3:00 AM may be skipped or fire twice.
2. Fall back (gain one hour): The same 1:30 AM occurs twice. Cron jobs can double-fire. Timestamp comparisons can produce negative durations or duplicate keys.

The Test Harness Architecture

Our harness runs on a single bare-metal node (Ubuntu 22.04) with a modified systemd-timesyncd and a custom time-warp daemon. The core components:

• PostgreSQL 16 with pg_timetz extension for timezone-aware timestamps.
• pgbouncer 1.21 as connection pooler (important: pooler caches timezone settings).
• Agent swarm written in Rust with Tokio timers (tokio::time::sleep, tokio::time::interval).
• Time-warp script that uses timedatectl set-timezone and adjtimex to simulate DST jumps.

Time-Warp Script (simulate-dst.sh)

#!/bin/bash
# Simulate spring forward: jump from 01:59:59 to 03:00:00
# Requires CAP_SYS_TIME
timedatectl set-timezone UTC
# Fast forward 1 hour in 1 second
date -s "$(date -d '1 hour')"
sleep 1
timedatectl set-timezone America/New_York
# Now the system clock is 1 hour ahead, but timezone offset changed
# Actual wall clock jump: 2:00 AM -> 3:00 AM

We run this inside a systemd service that coordinates with the agent swarm via a shared PostgreSQL advisory lock.

PostgreSQL: The Control Plane

All temporal state lives in PostgreSQL. We use TIMESTAMPTZ (not TIMESTAMP) to ensure UTC storage. Critical queries:

Task scheduling with DST-safe window

SELECT id, scheduled_at
FROM tasks
WHERE scheduled_at >= NOW()
  AND scheduled_at < NOW() + INTERVAL '1 hour'
ORDER BY scheduled_at ASC
FOR UPDATE SKIP LOCKED;

This query is vulnerable during fall-back: NOW() returns the same UTC instant for both 1:30 AM occurrences. The ORDER BY will produce duplicate results if the interval spans the repeated hour.

Fix: Use epoch-based ordering

SELECT id, scheduled_at
FROM tasks
WHERE EXTRACT(EPOCH FROM scheduled_at) >= EXTRACT(EPOCH FROM NOW())
  AND EXTRACT(EPOCH FROM scheduled_at) < EXTRACT(EPOCH FROM NOW()) + 3600
ORDER BY EXTRACT(EPOCH FROM scheduled_at)
FOR UPDATE SKIP LOCKED;

This eliminates the ambiguity of timestamptz comparisons during repeated hours.

Agent-Side Mitigations

1. Use monotonic clocks for timeouts

Tokio's tokio::time::sleep uses Instant (monotonic) by default, which is immune to wall-clock jumps. But if you use SystemTime::now() for deadline calculations, you'll break. We enforce a lint rule: never use SystemTime for timeouts.

2. Heartbeat with epoch seconds

use std::time::{SystemTime, UNIX_EPOCH};

fn heartbeat() -> i64 {
    SystemTime::now()
        .duration_since(UNIX_EPOCH)
        .unwrap()
        .as_secs() as i64
}

Store this in PostgreSQL as BIGINT. Compare using integer arithmetic, not timestamptz.

3. Retry backoff with jitter

Avoid std::time::Duration::from_secs(now.elapsed()). Compute backoff from a monotonic base:

let base = Instant::now();
// ... after failure
let elapsed = base.elapsed();
let wait = Duration::from_secs(1).min(elapsed * 2);
tokio::time::sleep(wait + jitter).await;

Running the Stress Test

We define three phases:

1. Pre-warm: Run the swarm for 1 hour at stable time. Record baseline metrics: task latency, timeout rate, scheduling accuracy.
2. Spring forward: At the exact transition (simulated), inject the time warp. Monitor agent logs for tokio::time::error::Elapsed or task duplication.
3. Fall back: Repeat with the reverse jump. Watch for duplicate task execution.

Metrics to Watch

• Task completion rate: Should remain 100%.
• Duplicate tasks: Count tasks with same scheduled_at and agent_id.
• Timeout errors: Log entries containing timeout or deadline.
• PostgreSQL deadlocks: pg_stat_database.deadlocks.

Real-World Bugs We Caught

1. PgBouncer transaction pooling: The pooler cached the timezone from the first connection. After DST change, new connections got the wrong timezone, causing NOW() to return a shifted value. Fix: set timezone in pgbouncer.ini to UTC and use AT TIME ZONE in queries.

2. Cron-like scheduler: A Rust agent used tokio::time::interval with a fixed 1-hour period. During spring-forward, the interval fired at 2:00 AM, then at 3:00 AM (skipped 2:00-3:00). The task scheduled for 2:30 AM never ran. Fix: use cron crate with timezone-aware parsing.

3. Heartbeat timeout: Leader election used SystemTime::now() - last_heartbeat. During fall-back, the difference became negative, causing immediate leader re-election. Fix: use monotonic clock for heartbeats.

Automation with systemd

We run the test suite in a systemd service that restarts the swarm after each transition:

[Unit]
Description=DST stress test

[Service]
Type=oneshot
ExecStartPre=/usr/local/bin/simulate-dst.sh spring
ExecStart=/opt/swarm/bin/agent --duration 120
ExecStartPost=/usr/local/bin/collect-metrics.sh

[Install]
WantedBy=multi-user.target

Conclusion

Daylight savings chaos is a cheap, high-impact test. Our harness runs in under 10 minutes and has uncovered bugs in every agent swarm we've tested. The key takeaways:

• Store everything as epoch seconds in PostgreSQL.
• Use monotonic clocks for timeouts and heartbeats.
• Never trust NOW() during repeated hours—use EXTRACT(EPOCH FROM NOW()).
• Test both spring and fall transitions.

Your swarm will survive the next DST shift. Ours does.