Stale Telemetry Poisoned Our Agent Loop: The 24-Hour Freshness Gate

We run a small swarm of agents on three Hetzner servers, wired together with WireGuard. Each agent has a job: one watches logs, another curates context, a third proposes LoRA training updates. The loop is simple: agents collect telemetry, a proposer suggests a new LoRA, we validate, and if it passes, we train overnight. It sounded elegant. Then it started lying to us.

The problem wasn't the model. It was the data. Our agents were pulling lifetime telemetry — every event since the swarm booted — and using that to propose improvements. But the swarm's behavior changes daily. New tasks, new data, new failure modes. Lifetime telemetry is a history book, not a newspaper. The proposals were being written from a history book, and the newspaper was being ignored.

The Poison: Lifetime Telemetry

Our telemetry pipeline is straightforward. Each agent emits structured events: task_started, task_completed, embedding_miss, retry_exhausted, response_rejected. These land in Postgres, with a JSONB payload. We also keep a vector store (Qdrant) for semantic search over past interactions, and a Neo4j graph for relationships between tasks and outcomes.

The proposer agent — let's call it the critic — runs a nightly job. It pulls a sample of telemetry, computes statistics, and writes a proposal: "Increase the retry limit for embedding misses," or "Add a new system prompt for ambiguous queries." The proposal goes to a validator, which checks it against a set of rules, then we decide whether to train a new LoRA.

The critic was using a query like this:

SELECT event_type, count(*) FROM telemetry GROUP BY event_type;

No time filter. That's lifetime data. The critic saw that embedding_miss had a 5% rate over the last month, but in the last 24 hours it was 40%. The proposal said "no change needed," while the system was actively degrading.

The root cause is common in agent loops: we built the telemetry store for debugging, not for decision-making. We never considered that the data would be used to make real-time adjustments. The critic was essentially reading a diary when it needed a live feed.

The Fix: A Freshness Gate

The fix was to add a freshness gate to every telemetry query that feeds the critic. The gate is a simple time window: only consider events from the last 24 hours. But it's more than a WHERE clause. It's a policy enforced at the data access layer.

We created a view in Postgres that only exposes recent events:

CREATE VIEW telemetry_recent AS
SELECT * FROM telemetry
WHERE event_time > now() - interval '24 hours';

But that's not enough. The critic also queries Qdrant and Neo4j. For Qdrant, we filter by timestamp in the metadata. For Neo4j, we add a last_seen property and filter on that.

More importantly, we changed the critic's prompt to explicitly require fresh data. The prompt now says: "You are analyzing telemetry from the last 24 hours only. Do not use lifetime statistics." This is a guardrail against the model's tendency to reason from stale patterns.

We also added a freshness check in the validator. Before a proposal is accepted, the validator queries the same views and checks that the data used in the proposal is within the window. If the critic references a metric that's older than 24 hours, the proposal is rejected.

Why 24 Hours?

We chose 24 hours because it matches our nightly training cycle. The loop is: day of telemetry, night of training. The 24-hour window gives us a full day of data, but not so much that older patterns dominate.

But the window should be configurable. Different metrics have different decay rates. For example, retry_exhausted might be spiky, so a 6-hour window might be better. response_rejected might be more stable, so 48 hours is fine. We made the window a parameter in the critic's configuration, and we can adjust per metric.

Implementation Details

Here's how we implemented the gate in practice.

Postgres View

We created a view that filters by time, but we also added a trigger to log any query that accesses the raw table without the view. This is a warning sign for future agents that might bypass the gate.

CREATE OR REPLACE VIEW telemetry_recent AS
SELECT * FROM telemetry
WHERE event_time > now() - interval '24 hours';

Qdrant Filter

For Qdrant, we store events as points with a timestamp in the payload. The critic's retrieval query now includes a filter:

from qdrant_client import QdrantClient
from qdrant_client.models import Filter, FieldCondition, Range

client = QdrantClient(host='localhost', port=6333)

filter_ = Filter(
    must=[
        FieldCondition(
            key='timestamp',
            range=Range(
                gte=int(time.time()) - 86400  # 24h in seconds
            )
        )
    ]
)

results = client.search(
    collection_name='telemetry_events',
    query_vector=query_vector,
    query_filter=filter_,
    limit=10
)

Neo4j Query

In Neo4j, we store nodes with a last_seen property. The critic's Cypher query now includes:

MATCH (e:Event)
WHERE e.last_seen >= datetime() - duration('P1D')
RETURN e.type, count(e) ORDER BY count(e) DESC

Critic Prompt

We rewrote the critic's system prompt to be explicit:

You are a critic agent. Your job is to propose improvements to the swarm's behavior.
You will be given a set of telemetry events from the last 24 hours.
Use ONLY these events. Do not use lifetime statistics.
If you need more data, say so, but do not infer from older data.

Validator Rule

We added a rule to the validator that checks the freshness of any metric mentioned in the proposal. The validator has a list of known metrics and their windows. For each metric, it queries the database with the appropriate window and compares the value to the one in the proposal. If they differ by more than a threshold, the proposal is rejected.

What Changed

After deploying the freshness gate, the proposals became more relevant. The critic started catching issues that were happening now, not last month. For example, it noticed that embedding_miss spiked during a specific time of day, and proposed a targeted retry policy for that window.

The validator also became more useful. It rejected a proposal that suggested lowering the retry limit based on a month-long average, because the last 24 hours showed the opposite trend.

We also saw a reduction in the number of proposals that were obviously wrong. Before, the critic would sometimes propose changes that contradicted recent behavior. Now, with the gate, the proposals are more consistent with the current state.

Caveats and Trade-offs

A freshness gate is not a silver bullet. Here are the trade-offs we considered.

Data Loss

By filtering to 24 hours, we lose long-term trends. If a problem builds slowly over a week, the critic won't see it. To mitigate, we run a separate weekly analysis that uses lifetime data to identify slow drifts. That analysis is not part of the self-improvement loop; it's a human review tool.

Cold Start

If the swarm has been down for more than 24 hours, there's no data. The critic has nothing to work with. We handle this by falling back to a default behavior: if no events in the last 24 hours, the critic proposes no changes and logs a warning.

Window Size Tuning

24 hours works for our nightly cycle, but it's not universal. We made the window configurable per metric, and we're experimenting with adaptive windows based on event frequency. For example, if an event occurs once an hour, a 6-hour window gives 6 samples; if it occurs once a day, a 24-hour window gives 1 sample. We need enough samples for statistical significance, so we set a minimum sample count and expand the window until we reach it.

The Bigger Picture

This incident taught us a broader lesson about agent self-improvement loops: the data you feed the loop is as important as the model. A loop that uses stale data will make stale decisions, and the loop will reinforce its own staleness. The freshness gate is a simple mechanism to keep the loop honest.

We've since applied the same principle to other parts of the stack. Our context curator now uses a freshness score when retrieving from Qdrant, weighting recent events higher. Our anomaly detector uses a rolling window. The pattern is to always ask: "Is this data fresh enough for this decision?"

Conclusion

A freshness gate is a small piece of engineering that prevents a big failure mode. It's not glamorous, but it's the kind of thing that keeps an autonomous system from drifting into irrelevance. If you're building an agent loop that learns from its own telemetry, add a gate. Your future self will thank you.

This post is part of our ongoing series on building sovereign AI infrastructure. We're sharing our engineering journey with the RiNET stack: three Hetzner servers, WireGuard mesh, Qwen on vLLM, BGE-M3 for embeddings, and Postgres, Qdrant, and Neo4j for storage. All opinions are our own.