The True TCO of Sovereign AI: Bare Metal vs. Spot Instances Over 6 Months

When you commit to sovereign AI—running your own models, on your own data, with no third-party inference APIs—the infrastructure decision is the first real bottleneck. The common reflex is to reach for spot instances: cheap, elastic, no upfront. But after six months of running a production stack (Qwen 2.5 72B on vLLM, BGE-M3 embeddings, Postgres+Qdrant+Neo4j for retrieval, nightly LoRA fine-tuning), the TCO picture shifts. This article walks through the real costs, not the marketing pitches.

The Workload Profile

A sovereign AI stack typically runs three classes of workload:

• Inference – latency-sensitive, steady-state traffic. Qwen 72B on vLLM needs ~140 GB VRAM for FP16, meaning 2× NVIDIA A100 80GB or 4× RTX 6000 Ada. The inference load is predictable: you know your daily query rate within 20% after a week.
• Embedding – BGE-M3 runs on a single GPU, usually a mid-range card (RTX 4090 or A10). Low VRAM, high throughput. This is the cheapest part.
• Fine-tuning – LoRA runs nightly, consuming the same GPU(s) as inference but with higher memory pressure (gradients, optimizer states). A typical Qwen 72B LoRA with rank 16 needs ~160 GB VRAM. This workload is batchable, can tolerate preemption, and runs for 2–4 hours.

Bare Metal: The Upfront Tax

Buying three dedicated servers (the RiNET pattern: one for inference, one for embedding + vector DB, one for orchestration + fine-tuning) costs roughly:

• Server A: 2× A100 80GB, 64-core EPYC, 512 GB RAM, 2× 3.84 TB NVMe. ~$30k–$40k depending on vendor.
• Server B: 1× RTX 6000 Ada, 32-core, 256 GB RAM, 1× 3.84 TB NVMe. ~$15k–$20k.
• Server C: 16-core, 128 GB RAM, 2× 1.92 TB NVMe (no GPU). ~$5k.
• Networking: 3× 10 GbE NICs, switch, cables, WireGuard mesh setup. ~$1k.
• One-time colocation: $500–$1k for rack space, power, cooling (if not self-hosted).

Total upfront: ~$55k. That’s a hard number. But it’s a capital expense that depreciates over 3–5 years. Monthly equivalent (3-year straight-line): ~$1,500.

Spot Instances: The Variable Drain

Spot instances from major clouds (AWS, GCP, Azure) for the same GPU capacity:

• p4d.24xlarge (8× A100 40GB): ~$3.91/hr spot, but you only need 2× A100 80GB. The smallest single-A100 spot instance is g5.48xlarge (1× A100 40GB) at ~$1.21/hr spot. For 2× A100 80GB you’d need 4× g5.48xlarge or a p4d.24xlarge. Let’s pick p4d.24xlarge at $3.91/hr spot.
• For embedding: g5.xlarge (1× A10G) at ~$0.25/hr spot.
• For orchestration: t3.medium at ~$0.01/hr spot.
• EBS volumes: 500 GB gp3 per instance: ~$0.08/GB-month = $40/month per instance.
• Data egress: inference outputs, embedding vectors, model downloads. At 1 TB/month egress: ~$90.

Monthly spot cost (6 months, no preemption):

• Inference: $3.91/hr × 730 hrs = $2,854
• Embedding: $0.25/hr × 730 = $182
• Orchestration: $0.01/hr × 730 = $7
• EBS: 3 instances × $40 = $120
• Egress: $90
• Total monthly: ~$3,253

Over 6 months: $19,518. Add one-time setup (AMI, networking, security groups): ~$200. Total: ~$19,718.

But that assumes zero preemption. In practice, spot A100 instances get reclaimed at 5–15% frequency per month. Each preemption costs: lost work (fine-tuning checkpoint restarts), re-queuing inference requests, and potential data loss if buffers aren’t drained. A common pattern is to run inference on a mix of spot and on-demand: 70% spot, 30% on-demand. On-demand p4d.24xlarge is $32.77/hr. That blend yields:

• 0.7 × $3.91 + 0.3 × $32.77 = $12.57/hr effective.
• Monthly: $12.57 × 730 = $9,176.
• Over 6 months: $55,056.

That’s already above bare metal’s 6-month TCO of $55k + $1,500/month colo × 6 = $64k. But wait—bare metal has residual value. After 6 months, you can sell the hardware for ~70% of purchase price (GPUs hold value). So net cost: $55k × 0.3 = $16,500 + colo $9k = $25,500. Spot (blended): $55k.

Operational Overhead

Bare metal:

• Hardware failure: you replace a PSU or fan yourself. Mean time between failures for enterprise gear is 2–3 years. Assume one incident in 6 months: 4 hours of downtime + $200 part.
• OS patching: monthly reboot, 10 minutes.
• Networking: WireGuard mesh is set-and-forget.
• Monitoring: Prometheus + Grafana on the orchestration node.

Spot instances:

• Instance health checks: auto-scaling groups, lifecycle hooks, spot termination notifications. You need a termination handler that saves state and drains connections. That’s code to write and maintain.
• Persistent storage: EBS snapshots, S3 for model weights. Object storage costs add up.
• Network latency: inter-instance traffic goes through the cloud backbone, not a local switch. For inference + vector DB, that adds 1–3 ms per call.

Data Sovereignty Costs

Spot instances run on shared hypervisors. If your threat model includes side-channel attacks (Spectre-class), you need dedicated hosts. Dedicated p4d: $40/hr. That kills the spot advantage.

Bare metal: physical isolation is inherent. No noisy neighbors, no hypervisor tax. PCIe passthrough to GPUs is direct.

The 6-Month Verdict

But this table is a trap. It assumes you can sell the GPUs at 70% after 6 months. If you keep them for 2 years, the bare metal cost drops to ~$12k/year. Spot stays at $110k/year.

When Spot Wins

• Short-lived experiments: 1–2 months of exploratory fine-tuning. No commitment.
• Bursty workloads: inference spikes from a product launch. Auto-scale spot, then tear down.
• Preemption-tolerant training: LoRA with frequent checkpointing (every 5 minutes). You lose at most 5 minutes of work.

When Bare Metal Wins

• Steady-state inference: 24/7 API serving. The cost per token is lower after 3–4 months.
• Data sovereignty: No cloud provider touches your weights or inference data.
• Fine-tuning at scale: Nightly LoRA runs that need deterministic GPU availability. No preemption risk.
• Latency-sensitive pipelines: Vector DB + inference in the same LAN. Sub-millisecond embedding retrieval.

The Hybrid Play

A pragmatic sovereign stack uses bare metal for the steady-state inference and embedding tier, and spot instances for fine-tuning bursts. The RiNET pattern achieves this with:

• Three bare-metal servers (inference, embedding, orchestration) connected via WireGuard.
• When nightly LoRA kicks off, it runs on the inference server’s GPUs during low-traffic hours (e.g., 2–6 AM). No spot needed.
• If you need extra capacity, spin up spot instances with a pre-configured vLLM image, point them at the same Qdrant and Postgres, then tear down.

This hybrid avoids the worst of both worlds: you don’t pay spot prices for 24/7 inference, and you don’t buy GPUs for peak load.

Practical Advice

1. Calculate your break-even point. Take the bare metal upfront cost (minus residual) and divide by the hourly spot rate for the equivalent GPU. That’s the number of hours you need to run to break even. For 2× A100 80GB vs. spot p4d.24xlarge blended: $16,500 / $12.57/hr = 1,312 hours ≈ 55 days. If you run inference 24/7, you break even in under two months.
2. Never use spot for stateful databases. Qdrant and Postgres on spot with EBS is a recipe for corruption on preemption. Use bare metal or on-demand for the persistence layer.
3. Automate spot termination handling. Even if you only use spot for fine-tuning, write a script that saves LoRA adapters to S3 every N steps and drains the GPU before shutdown.
4. Benchmark your own latency. The 1–3 ms inter-instance penalty in the cloud adds up. For RAG pipelines with multiple embedding calls per query, that can double end-to-end latency.

The Bottom Line

Sovereign AI isn’t just about model weights—it’s about infrastructure independence. Over 6 months, bare metal is cheaper if you have steady workloads. Spot instances are only cheaper if you treat them as ephemeral and accept the operational tax. The real TCO includes your time spent handling preemption, monitoring instance health, and managing cloud IAM. Put a dollar value on that, and bare metal often wins before you even plug in the power cord.