NMI Watchdog Resets on 4-GPU Bare Metal: Tracing IRQ Storms to PCIe AER Errors

When a 4-GPU bare-metal node resets itself under load, the first instinct is to blame thermals or power. But sometimes the real culprit is a silent flood of PCIe errors that never quite reach the console. This article walks through a real-world pattern: NMI watchdog timeouts triggered by an IRQ storm caused by PCIe AER (Advanced Error Reporting) errors. The goal is to give you a repeatable debugging path, from first symptoms to BIOS-level fixes, without relying on vendor magic.

Symptoms: Random Resets Under Load

The classic presentation: a 4-GPU server runs fine for hours or days, then suddenly reboots without a clean shutdown. The system logs show nothing immediately before the reset—just a hard power cycle. If you're lucky, the BMC logs an NMI event or a watchdog timeout. If you're not, you're left guessing.

In many cases, the resets correlate with GPU compute loads, but not always. Sometimes it's network traffic or even idle periods. The key is to gather as much data as possible before the next reset. Enable serial console logging, configure crash dumps, and set up remote logging to a separate machine. You need to catch the kernel's last words.

First Stop: dmesg and NMI Watchdog

When the system comes back up, the first place to look is dmesg. Look for messages like:

NMI watchdog: Watchdog detected hard LOCKUP on cpu 12

Or:

BUG: soft lockup - CPU#4 stuck for 23s! [kworker/4:0]

These indicate that the NMI watchdog—a hardware timer that fires a non-maskable interrupt if a CPU core doesn't schedule for a while—tripped. The CPU was stuck in an interrupt handler or a kernel path for too long.

But why would a CPU be stuck? Often it's because an interrupt storm is overwhelming a single core. The kernel's interrupt handling becomes the bottleneck, and the core can't make progress on other tasks.

Identifying the IRQ Storm

To see interrupt counts per CPU and per IRQ, use /proc/interrupts. Look for a line that is incrementing rapidly, especially on a single CPU. For example:

cat /proc/interrupts | grep -E 'CPU0|PCIe'

You might see something like:

           CPU0       CPU1       CPU2       CPU3
 26:  123456789   123456789   123456789   123456789   PCI-MSI 0000:00:01.0

If the count is in the hundreds of millions and climbing, you have a storm.

But what's generating the interrupts? The IRQ line name often points to a PCIe device. In the example above, 0000:00:01.0 is a PCIe root port. That's a clue.

PCIe AER Errors: The Hidden Flood

PCIe AER is a mechanism for reporting errors on the PCIe bus. These errors can be corrected (like a single-bit ECC error) or uncorrected (like a link failure). The kernel logs them via the aer driver. You might see messages like:

pcieport 0000:00:01.0: AER: Corrected error received: 0000:01:00.0

Or, if the error is uncorrected and non-fatal:

pcieport 0000:00:01.0: AER: Uncorrected (Non-Fatal) error received: 0000:01:00.0

These messages can be sporadic, but when they happen frequently, the kernel's AER handler runs on the CPU that owns the affected root port's interrupt. If the errors come in a burst, that CPU can be saturated.

The IRQ Storm Mechanism

Here's the chain of events:

1. A PCIe device (often a GPU or its bridge) starts generating errors—corrected or uncorrected.
2. The root port receives these errors and raises an interrupt.
3. The kernel's AER driver handles the interrupt, logs the error, and may attempt recovery.
4. If errors are frequent, the interrupt handler runs repeatedly, consuming CPU time on the assigned core.
5. Other interrupts on the same core (like the NMI watchdog) get delayed, causing a lockup.

The NMI watchdog is an NMI, so it should fire even if the core is stuck in a normal interrupt handler. But if the core is stuck with interrupts disabled, or in a critical section, the NMI handler might not be able to run either. The result is a watchdog timeout and a panic.

A Common Culprit: GPU and PCIe Link Errors

In multi-GPU systems, a frequent source of PCIe errors is the GPU's link to the CPU. This can be due to signal integrity issues, power delivery problems, or even a faulty riser card. The AER errors might be corrected (e.g., replay timers) but still cause a storm.

A known pattern is that the GPU's PCIe link goes through a PCIe switch or a PLX chip. If that switch is flaky, it can generate a flood of errors.

Debugging Steps: A Methodical Approach

Here's a step-by-step approach to confirm the diagnosis and find a fix.

1. Confirm AER Errors Are Occurring

Enable verbose AER logging and watch the kernel log for a while:

echo 1 > /sys/module/aer/parameters/log_root_device

Or use dmesg -w to watch live. If you see a steady stream of AER messages, you've found your storm.

2. Identify the Affected Device

The AER message includes a PCIe address like 0000:01:00.0. Use lspci -v to see what device that is:

lspci -v -s 01:00.0

It will likely be a GPU or a bridge.

3. Check Interrupt Affinity

If the storm is on a single CPU, you might be able to spread the load by changing the interrupt affinity. The IRQ number for the root port can be found in /proc/interrupts. Then set its affinity to multiple CPUs:

echo 3 > /proc/irq/32/smp_affinity

But this is a temporary workaround. The root cause is the errors themselves.

4. Reduce AER Noise

If the errors are corrected and non-fatal, you might be able to tell the kernel to ignore them. The kernel has a module parameter for the AER driver: pcie_aspm=off or pcie_ports=compat can change behavior, but these are blunt instruments.

A more targeted approach is to disable AER for a specific device using setpci or kernel boot parameters. For example, to disable AER on a bus:

pcie_ports=compat

But this disables all PCIe port services, including hotplug and power management. Not ideal.

5. Update Firmware and Drivers

Often, these issues are fixed by a BIOS update or a GPU firmware update. Check the vendor's release notes. Also ensure the NVIDIA driver (if using NVIDIA GPUs) is up to date, as it often handles AER errors internally.

6. Check Physical Connections

Reseat GPU cards, check power cables, and ensure proper cooling. Sometimes a loose connection causes signal integrity issues that lead to AER errors.

Kernel Parameters That Help

There are several kernel boot parameters that can mitigate the impact of AER errors:

• nmi_watchdog=0 disables the NMI watchdog entirely. This prevents the panic but hides the underlying problem. Not recommended for production.
• nowatchdog disables both NMI and soft lockup detectors.
• pci=noaer disables AER completely. This stops the error logging and might prevent the storm, but you lose the ability to detect serious errors.
• pcie_aspm=off disables Active State Power Management, which can sometimes cause link errors.

A better approach is to keep AER enabled but increase the watchdog timeout:

• watchdog_thresh=30 increases the threshold from 10 seconds to 30 seconds, giving the CPU more time to recover.

But these are band-aids. The real fix is to eliminate the source of the errors.

A Realistic Scenario: What Often Works

In many cases, after confirming the AER storm, the following steps have resolved the issue:

1. Update the BIOS to the latest version. This often includes microcode fixes for PCIe error handling.
2. Update the GPU driver and, if available, the GPU's VBIOS.
3. Check the PCIe link speed and width. Forcing a lower link speed (e.g., gen3 instead of gen4) can stabilize the link if signal integrity is marginal.
4. Ensure that the motherboard's PCIe slot is not sharing bandwidth with other devices in a way that causes contention.
5. If using a riser or switch, try plugging the GPU directly into the motherboard slot.

If the problem persists, it might be a hardware fault. Some teams have found that replacing the motherboard or the GPU resolves the issue.

A Cautionary Note: Don't Disable the Watchdog Too Quickly

It's tempting to set nmi_watchdog=0 to stop the resets, but that's like removing the smoke detector because it beeps. The watchdog is there to catch genuine lockups. If you disable it, you might miss a real kernel bug that leads to silent data corruption.

Instead, use the watchdog as a diagnostic tool. The fact that it fires tells you something is wrong. Dig deeper.

Conclusion: A Systematic Approach Wins

The path from random NMI resets to a PCIe AER-induced IRQ storm is not obvious. It requires careful observation, log analysis, and a willingness to question initial assumptions about thermals or power. The steps outlined here provide a framework:

1. Capture the kernel messages before the reset.
2. Identify the IRQ storm via /proc/interrupts.
3. Correlate with AER errors in the kernel log.
4. Apply targeted fixes: firmware updates, driver updates, hardware reseating, or kernel parameters.

By methodically ruling out causes, you can save days of frustration and keep your GPU nodes stable. Remember: the NMI watchdog is your friend—it's telling you that something is stealing CPU time. Listen to it.