A 12-Hour Agent Run That Failed at Minute 11: Postmortem on a Silent Input Poison

The Setup

We have a long-running agent that processes a queue of database migration tasks. Each task involves reading schema, generating SQL, applying it, and verifying. The agent runs inside a systemd service, logs to journald, and uses a loop: fetch task → execute → store result → next. The expected runtime per task is 30–60 seconds; the whole queue is about 800 tasks, so roughly 12 hours.

We launched it at 22:00 and went to bed. At 08:00 we checked: the agent was still running, but the log showed only 3 tasks completed. The remaining 797 tasks were skipped with a cryptic error: ValueError: invalid literal for int() with base 10: ''.

The Timeline

• Minute 0–10: Tasks 1–3 completed normally.
• Minute 11: Task 4 failed. The agent entered an error loop: retry → fail → retry → fail. After 3 retries, the agent marked the task as failed and moved on. But the state machine got corrupted: every subsequent task also failed with the same error.
• Minute 12–720: The agent processed zero useful work, but kept running, logging failures, and consuming resources. No alert fired because the agent was still alive.

Root Cause Analysis

The Immediate Error

The error ValueError: invalid literal for int() with base 10: '' pointed to a line in our code that parses a tool output:

rows_affected = int(output.strip())

output was an empty string. But how? The tool (psql) should always return a number.

Tracing the Input

We added debug logging to capture the raw output before parsing. Replaying the task with the same inputs reproduced the failure. The output was:

That’s a single newline character. Not a number. Not empty string. A newline.

How Did psql Return a Newline?

The task was to run ANALYZE on a partition. The SQL was:

ANALYZE VERBOSE public.partition_2024;

psql with -t -A -q flags should output just the number of rows analyzed. But ANALYZE VERBOSE outputs verbose messages to stderr, not stdout. However, our agent was capturing stdout only. So stdout was empty. But why the newline?

We discovered that psql, when run with -t (tuples only), still prints a trailing newline after the last tuple. If there are zero tuples, it prints a blank line. So output was "\n", and strip() turned it into "", which int() choked on.

The Silent Poison

The real problem wasn’t the newline. It was that the agent’s state machine used the parsed integer to update an internal counter. When parsing failed, the counter was never updated. But the code that decided whether to retry or skip used a different condition: it checked if the task’s attempts < max_attempts. Since parsing failed before incrementing attempts, the agent retried infinitely until the max_attempts limit kicked in. After that, it marked the task as failed and moved on.

But here’s the poison: the state machine stored the last parsed integer in a global variable last_rows. When the next task started, it read last_rows to calculate a delta. last_rows was still the value from task 3 (say, 12345). The new task expected to parse its own output, but the parsing code was wrapped in a try-except that, on failure, left last_rows unchanged. So the delta calculation used stale data, producing a negative number that caused a downstream assertion failure. The agent then threw an unhandled exception, which the main loop caught and logged as ValueError: invalid literal for int(). But the real error was the stale state.

The Fixes

We deployed three changes:

1. Input Validation and Sanitization

We now validate every tool output before passing it to any parser. For numeric outputs, we use a regex to extract the first integer:

import re

def extract_int(output: str) -> int | None:
    match = re.search(r'\d+', output)
    return int(match.group()) if match else None

If extract_int returns None, we log the raw output and raise a clear error.

2. Structured Outputs

Instead of parsing raw text, we switched to JSON output from psql using -A -t --csv and then parsing with csv.DictReader. This eliminates ambiguity:

import csv, io

def parse_csv_output(output: str) -> list[dict]:
    reader = csv.DictReader(io.StringIO(output))
    return [row for row in reader]

For ANALYZE, we now use psql -c "EXPLAIN (ANALYZE, FORMAT JSON) ..." and parse the JSON directly.

3. State Machine with Immutable Snapshots

The global last_rows variable was replaced with a per-task state dict that is never mutated after creation. Each task receives a snapshot of the needed state at the start and returns its own results. The main loop only aggregates results after successful completion:

@dataclass
class TaskState:
    last_rows: int | None = None
    current_rows: int | None = None

def execute_task(task: dict, state: TaskState) -> TaskState:
    # state is read-only; create new state for output
    new_state = TaskState(last_rows=state.last_rows)
    output = run_psql(task['sql'])
    rows = extract_int(output)
    if rows is None:
        raise ValueError(f"Could not parse rows from output: {output!r}")
    new_state.current_rows = rows
    return new_state

Lessons Learned

• Silent corruption is worse than a crash. A crash stops the system; corruption lets it run in a broken state. Always validate and fail fast.
• Don’t trust tool output. Even well-known tools can produce unexpected output in edge cases. Use structured formats (JSON, CSV) when possible.
• State should be immutable. Shared mutable state is a bug magnet. Pass snapshots, return new state.
• Log raw inputs. Without logging the actual tool output, we would never have found the newline. Add debug logging for all external inputs.
• Test edge cases. We tested with normal data, but not with empty partitions. Add tests for empty results, error outputs, and malformed data.

The Aftermath

We re-ran the queue after the fixes. It completed in 11 hours and 47 minutes with zero errors. The agent now logs the raw output of every tool call at DEBUG level, and we have a Grafana dashboard that tracks parsing failures. We also added a systemd watchdog that kills the agent if it goes more than 10 minutes without completing a task.

That 12-hour run taught us that a single newline can poison an entire day’s work. Validate early, validate often, and never assume the tool will behave.

Tools used: PostgreSQL 16, psql, Python 3.12, systemd 255, Grafana 10.4. All fixes were deployed as a systemd service update with zero downtime.