The Sovereign Stack

A nightly reindex can run twice when systemd timers drift. Learn how to make vector index jobs idempotent and self-healing with concrete systemd patterns.

The OOM killer is a post-mortem. Admission control decides which agent requests get VRAM at all, so a swarm degrades predictably instead of dying mid-generation.

A UNIQUE constraint over a nullable column does not block duplicates when the value is NULL. Learn why deduplication fails silently and how to fix it.

Full upserts replace entire payloads, silently erasing metadata fields. Learn how to design partial updates that preserve annotations and avoid data loss.

A handler that issues an HTTP request back to its own service can deadlock the event loop under concurrency. This is how to recognize, reproduce, and fix it.

Inline comments after a value are not portable across dotenv and EnvironmentFile parsers. A trailing `#` can end up inside the secret, producing silent auth failures.

Croatia's act on legalising illegally built buildings has an Article 10 that governs how an application is filed. Its text has existed in three versions:

Tenderi is a search and alerting product for companies that bid on public contracts in Croatia. The index holds 410,121 notices published between May 2016 and today, of which 337,247 come from the EU's TED feed and 72,401 from the national portal, plus 25,382 decisions of the…

Masking a systemd unit tells systemd to refuse to start it. It does not tell a container orchestrator, config-management agent, or watchdog to stop trying. Here is why masked units come back.

RuntimeMaxSec is ignored for Type=oneshot units because the runtime clock only starts for units that enter the activating/running state. Here is what actually caps them.

Memory-mapping a vector index trades explicit I/O for page-fault latency and kernel reclaim policy. The failure mode is a swap storm, not a slow query.

Reactive thermal management causes latency spikes and hardware degradation. A predictive scheduler that throttles workloads before the GPU does preserves performance and sovereignty.

Agent systems that depend on Postgres need controlled fault injection to prove they can survive connection loss. Here is a methodology for safe, reversible chaos experiments.

Advisory locks are session-scoped, invisible to the planner, and easy to hold too long. In agent swarms they quietly convert concurrency into a queue.

Agent swarms need crash-consistent context. A common pattern treats Postgres WAL as the authoritative ledger for agent state, not just database durability.

HNSW segment merges in Qdrant are CPU and memory-bandwidth heavy. When they overlap with live inference, they can starve the same host resources your GPU pipeline depends on.

Paged attention lets one engine serve many tenants, but the block table is shared state. If a write path crosses tenants, one request can poison another's context window.

PCIe link degradation is invisible to most monitoring stacks until inference performance craters. Learn how to read link state from sysfs, set up proactive alerts, and design self-healing remediation.

A dual-socket host is two computers sharing RAM. NUMA-blind agent placement lets the scheduler scatter workers across sockets, and cross-socket memory access becomes the bottleneck.

When context exceeds VRAM, paging the KV cache to NVMe adds a measurable per-token latency tax. Here is how to think about it.

A migration touching Postgres, Qdrant, and Neo4j is not one transaction. Treating it as one is how vector indexes end up pointing at deleted rows.

Audit logs demand durability, but fsync on every turn is a throughput killer. Here's how to structure writes so a sudden power loss can't corrupt or lose the log, even without fsync per turn.

Embedding drift silently degrades RAG quality. Learn to build a statistical gate that flags stale vectors before inference, using only your existing stack.

Vector indexing at scale is a crash-prone pipeline. Here's how to treat Postgres as the durable backbone of agent memory, with checkpoints and idempotent recovery.

Hardware resets on 4-GPU nodes traced to NMI watchdog timeouts caused by IRQ storms from PCIe AER errors. A methodical debug path from dmesg to firmware settings.

Temperature 0 alone doesn't guarantee reproducible agent runs. Hidden state traps in sampling, kernels, and infrastructure can silently break determinism. Here's how to design for replay.

A 90-day crash-recovery benchmark on NVMe comparing ext4 and XFS for agent memory journals. Findings on corruption rates, recovery times, and practical implications for autonomous systems.

When an agent's state goes bad, you need more than a hunch. Learn to use Postgres xmin and LSN to trace every write back to the exact commit that corrupted it.

How to build token-level provenance chains connecting inference logs to training data lineage, and why the EU AI Act does not actually require it.

WireGuard meshes look simple until they aren't. Here are the failure modes we've hit—MTU mismatches, handshake storms, and the dreaded 3am timeout—and how to diagnose them.

Multi-GPU inference often stalls on NUMA mismatches. Learn how CPU-GPU affinity, PCIe topology, and memory placement shape latency—and how to diagnose it.

We replaced Kafka with Postgres LISTEN/NOTIFY for agent event streaming. Simpler ops, fewer moving parts, and enough throughput for our scale.

When your AI stack spans Postgres and a vector store, two-phase commit is a trap. Async write-through with idempotent retries gives you consistency without the latency hit.

Learn how to verify agent state transitions using Merkle trees in Postgres, enabling deterministic replay and auditability without blockchain.

A 4-day autonomous agent run killed a GPU at 92°C. Here's the thermal postmortem and what to change for long-running inference.

A write-ahead journal for token streams survives sudden power loss. Here's the architecture, the tradeoffs, and why fsync discipline beats buffering.

Agentic systems need fast semantic recall without burning cash. A hot-warm-cold memory hierarchy—RAM, SSD, tape—keeps latency low and costs sane.

A swarm of autonomous agents slowly leaked VRAM over 8 hours due to attention-mask fragmentation. Here's how we traced it and fixed it.

Raft gives you linearizability, but at the cost of a leader. For agent swarms, CRDTs offer an alternative—if you can prove they actually converge. Here's a property-based testing methodology that catches the bugs Jepsen

Moving agent state out of Redis into an in-process KV store with a write-ahead log simplified our stack and cut sync overhead. Here's the reasoning and a reference implementation.

A deep dive into a CUDA memory leak caused by attention masks in transformer inference, and how we traced it through agent crashes after eight hours.

Durable agent memory means fsync. But fsync on every turn can tank throughput. Here's how to reason about the tradeoff and design for power-loss safety without crippling your agent.

Model weights are your IP. Encrypting them at rest usually kills inference speed. Here's how to use hardware-backed keys and a decryption cache to get both.

Temperature 0 and a fixed seed don't guarantee identical outputs. Here's what actually breaks determinism in production LLM serving and how to work around it.

The EU AI Act demands logging for high-risk AI systems. But logging raw prompts is a privacy and security nightmare. Here's how we do it with hashes and why it's better.

Autonomous agents need more than one memory store. Here's how to reconcile vector, graph, and relational views without losing consistency.

Running GPUs without air conditioning is feasible if you accept high temperatures and manage thermal throttling with software. This article covers the practical engineering behind it.

Calibrating confidence scores lets LLM systems refuse low-confidence answers, reducing hallucinations. A practical guide to thresholds, calibration, and guardrails.

A single bit flip in GPU memory can corrupt weights and derail an entire agent swarm. Here's how ECC memory and vigilant monitoring saved our inference cluster from silent failure.

Central orchestrators create single points of failure and coordination bottlenecks. Here's how we used Raft to build a self-organizing agent swarm that survives node loss and scales horizontally.