WireGuard Mesh Pitfalls: MTU, Handshake Storms, and the 3am Timeout

WireGuard is the quiet workhorse of modern VPNs. It's fast, auditable, and the config file fits in a tweet. But when you stitch three servers into a mesh, the simplicity evaporates. The protocol is fine—the network around it is not. This is a field guide to the failure modes we've seen (and fixed) running a small sovereign stack across three Hetzner hosts, with a WireGuard mesh at its core.

We run a modest setup: three Hetzner dedicated servers, each in a different DC, connected via a WireGuard mesh. On top of that mesh, we run Postgres, Qdrant, Neo4j, a vLLM serving Qwen, and a nightly LoRA fine-tune job. The mesh is the backbone, and when it breaks, everything breaks. The symptoms are rarely obvious: a query that times out at 3am, a handshake that never completes, a packet that vanishes in the middle. Here's what we've learned.

The Anatomy of a WireGuard Mesh

A mesh means every node talks to every other node directly. With three nodes, that's three point-to-point tunnels. Each tunnel has its own private key, public key, and endpoint. The config is simple:

[Interface]
PrivateKey = <node1_private>
Address = 10.0.0.1/24
ListenPort = 51820

[Peer]
PublicKey = <node2_public>
AllowedIPs = 10.0.0.2/32
Endpoint = node2.example.com:51820
PersistentKeepalive = 25

[Peer]
PublicKey = <node3_public>
AllowedIPs = 10.0.0.3/32
Endpoint = node3.example.com:51820
PersistentKeepalive = 25

That's the happy path. The problems start when reality intervenes: asymmetric routing, NAT, firewalls, and the dreaded MTU.

MTU: The Silent Killer

MTU (Maximum Transmission Unit) is the largest packet size a link can carry without fragmentation. On Ethernet, it's typically 1500 bytes. WireGuard wraps your packets in UDP, adding 60 bytes of overhead (IPv4) or 80 (IPv6). If your underlying interface has MTU 1500, your WireGuard interface must be smaller—typically 1420 (1500 minus 80) for IPv6, or 1440 for IPv4. Get this wrong, and you get silent drops.

We've seen two common mistakes:

1. Setting the WireGuard MTU too high. If you set it to 1500, packets that are exactly 1500 bytes will be fragmented at the IP layer, but WireGuard doesn't handle fragments gracefully. The result: intermittent timeouts, especially with large database queries or file transfers.

2. Forgetting that the path MTU might be lower. Even if your server's NIC is 1500, the path between your servers might traverse a tunnel or a network with a lower MTU. For example, if your servers are behind a cloud provider that uses VXLAN (overhead 50 bytes), the effective MTU is 1450. WireGuard's default of 1420 is safe, but if you set it to 1440 because you measured your local interface, you're on the edge.

How to diagnose: ping -M do -s 1472 to test a 1500-byte packet (1472 payload + 28 ICMP header). If it fails, lower the size until it works. That gives you the path MTU. Subtract 80 for IPv6, 60 for IPv4, and set that as your WireGuard MTU.

# On node1, test path MTU to node2's WireGuard IP
ping -M do -s 1472 10.0.0.2

If ping fails, try 1400, then 1300. Once you find the max, set it in the interface config:

[Interface]
MTU = 1420

And don't forget to restart the interface. wg-quick down and wg-quick up are your friends.

Handshake Storms: When Peers Get Chatty

WireGuard uses a simple handshake to establish a session. It's a one-time thing, but if the handshake fails, peers retry. Under normal conditions, that's fine. But when a peer is unreachable (e.g., a server is down or a firewall drops UDP), WireGuard will keep trying, and the retry logic can become a storm.

The default behavior: if a peer doesn't respond, WireGuard sends a handshake initiation every few seconds, with exponential backoff. That's fine for one peer. But if you have a mesh of three, and one node goes down, the other two will both try to handshake with it, and they'll also try to handshake with each other (if the mesh is full). The result: a flood of handshake packets on the network, which can saturate a small link or trigger rate-limiting on the firewall.

We've seen this happen during maintenance. We took down node3 for a kernel upgrade, and node1 and node2 started hammering each other with handshake initiations, even though they were already connected. Why? Because WireGuard's handshake is per-peer, and if a peer's endpoint changes (or is unreachable), the session is invalidated. In our case, node3's endpoint was unreachable, but node1 and node2 were also trying to re-establish their own session because they thought node3's failure meant the mesh was down.

The fix: use PersistentKeepalive wisely. This sends a keepalive packet every N seconds to maintain NAT bindings and detect dead peers. But it also triggers a handshake if the session is stale. We set it to 25 seconds, which is fine for most cases. But if you have a peer that's down, the keepalive will cause a handshake attempt every 25 seconds, which is manageable. The storm comes when you have multiple peers and a flapping network.

Another culprit: misconfigured AllowedIPs. If you set AllowedIPs = 0.0.0.0/0 on a peer, WireGuard will route all traffic to that peer, including handshake packets. That can cause a routing loop if the peer is unreachable. We keep AllowedIPs tight—only the specific /32 of the peer's WireGuard IP.

To diagnose a handshake storm, use wg show:

wg show

Look at the latest handshake column. If it's constantly updating, you have a storm. Check the transfer column—if it's growing rapidly, you're sending a lot of data (handshake packets are small, but many of them add up).

If you see a storm, the first step is to check if the peer is actually up. ping the endpoint IP. If it's down, you can either wait for it to come back or temporarily remove the peer from the config. We've also used iptables to rate-limit UDP on the WireGuard port, but that's a sledgehammer.

The 3am Timeout: When the Mesh Goes Silent

This is the classic: everything works during the day, but at 3am, a cron job that syncs data between nodes fails with a timeout. The logs show nothing—WireGuard says the handshake is fine, but packets are dropped.

The root cause is often NAT timeouts or firewall idle timeouts. If your servers are behind NAT (e.g., a home office or a cloud NAT gateway), the NAT mapping expires after a certain idle period. WireGuard's PersistentKeepalive is designed to prevent this, but if your keepalive interval is longer than the NAT timeout, the mapping expires, and incoming packets are dropped.

Another cause: TCP keepalive vs. WireGuard's keepalive. If you have a long-running TCP connection (like a Postgres connection) that goes idle, the OS's TCP keepalive might be disabled or set to a very long interval. When the NAT mapping expires, the next packet from the server to the client is dropped, and the client doesn't see it until it sends something. That's the 3am timeout.

We've seen this with our nightly LoRA training job. It runs at 3am, pulls data from the Postgres node, and if the connection has been idle for an hour, the NAT mapping is gone. The fix is twofold:

1. Set PersistentKeepalive to a value lower than the NAT timeout. We use 25 seconds, which is safe for most NATs (the default is 25, but some providers use 30 or 60). If you're behind a strict NAT, set it to 10 seconds.

2. Enable TCP keepalive on the application layer. For Postgres, set tcp_keepalives_idle, tcp_keepalives_interval, and tcp_keepalives_count in postgresql.conf. For other services, use the OS-level net.ipv4.tcp_keepalive_time.

# On each node, set TCP keepalive to 60 seconds
sysctl -w net.ipv4.tcp_keepalive_time=60
sysctl -w net.ipv4.tcp_keepalive_intvl=10
sysctl -w net.ipv4.tcp_keepalive_probes=5

But the deeper issue is that WireGuard is a UDP protocol, and UDP has no notion of connection state. The PersistentKeepalive is your only defense against NAT timeouts. If you have a service that needs to be reachable at any time, you might want to consider a TCP-based VPN like OpenVPN, but that adds overhead. We stick with WireGuard and tune the keepalive.

Another cause of the 3am timeout is asymmetric routing. In a mesh, each node has a direct tunnel to every other node. But if you have routes that send traffic through a different path (e.g., via a central gateway), the return packet might take a different route, and the NAT mapping on the intermediate device might not match. This is more common in complex networks, but we've seen it with our setup when we added a fourth node temporarily and had a routing table mess.

To diagnose, use tcpdump on the WireGuard interface:

tcpdump -i wg0 -n

Look for packets that are sent but not acknowledged. If you see a SYN sent and no SYN-ACK, the packet is being dropped. Check the firewall on both ends. We've had iptables rules that accidentally dropped UDP on the WireGuard port.

Practical Tips for a Robust Mesh

Here's what we've learned from running this mesh for a while:

• Keep your configs in version control. We have a Git repo with the WireGuard configs for all nodes. When we change something, we commit it. That makes it easy to rollback.
• Use a consistent MTU. We set all WireGuard interfaces to 1420, regardless of the underlying NIC. That avoids surprises.
• Monitor handshake times. We have a simple script that runs wg show and checks the latest handshake timestamp. If it's older than 5 minutes, we alert.
• Test your failover. If you have a mesh, you should be able to lose one node and still have connectivity between the others. We test this by shutting down a node and checking if the others can still communicate.
• Consider a central controller. For a mesh of three, it's manageable. But if you grow to ten nodes, you'll want something like wg-mesh or a configuration management tool to generate the configs. We use Ansible to push configs, but that's a separate topic.

The 3am Timeout, Revisited

We thought we had solved the 3am timeout by setting PersistentKeepalive to 25 and enabling TCP keepalive. But it came back. The culprit: a firewall on the Hetzner side that had a UDP idle timeout of 120 seconds. Our keepalive was 25, so that should have been fine. But the keepalive only sends a packet if there's no traffic. If you have a TCP connection that's sending data every 30 seconds, the NAT mapping stays alive. But if the TCP connection is idle, the keepalive is the only thing keeping it alive.

We had a service that only sent data every hour (the nightly job). The TCP keepalive was set to 60 seconds, but the service's TCP stack wasn't sending keepalives because it was a long-lived connection that didn't have the SO_KEEPALIVE option set. We fixed it by setting tcp_keepalives_idle in Postgres, but we also had to set it for the client (the script that runs the job). The lesson: keepalive is a stack-wide concern, not just a WireGuard one.

Conclusion

WireGuard is a great tool, but it's not magic. The pitfalls are real, but they're avoidable with careful configuration and monitoring. Our mesh has been stable for months now, but we still keep an eye on the handshake times and MTU settings. The 3am timeout is a rite of passage for any WireGuard user—once you've debugged it, you'll never forget it.

If you're building a similar stack, start with the basics: set your MTU correctly, use PersistentKeepalive, and test your failover. And when something breaks at 3am, remember: it's probably not the protocol, it's the network around it.