The Kernel's Distress Signal
When a GPU encounters a fatal error, it does not simply crash silently. The NVIDIA kernel driver writes an Xid error to the Linux kernel ring buffer (visible via dmesg), tagged NVRM: Xid with a numeric code and the PCI address of the affected GPU. These are your primary signal that something has gone wrong at the hardware or driver level.
Common Xid codes and their meaning:
| Xid | Name | Meaning | Action |
|---|---|---|---|
| 13 | Graphics Exception | Illegal instruction executed by a shader | Check CUDA code for out-of-bounds access |
| 31 | GPU Memory Page Fault | GPU accessed an invalid virtual address | Bug in CUDA kernel — check pointer arithmetic |
| 43 | GPU Stopped Processing | Timeout waiting for GPU to respond | Often precedes Xid 79; check thermals and power |
| 48 | Double Bit ECC Error | Uncorrectable VRAM hardware error | GPU VRAM is physically failing — replace hardware |
| 63 | ECC Page Retirement | A VRAM page has been retired due to errors | Monitor; if count grows, replace GPU |
| 74 | NVLINK Error | NVLink data integrity error | Check NVLink bridges; may indicate physical damage |
| 79 | GPU Fallen Off Bus | GPU stopped responding to PCIe | Requires hard reboot; often hardware failure |
| 92 | High Single Bit ECC | Correctable ECC errors exceeding threshold | Warning sign; monitor for escalation to Xid 48 |
Operational strategy:
Xid 48 and Xid 79 are hardware emergencies. Xid 31 is almost always a software bug (your CUDA kernel has a pointer error). Xid 63 (page retirement) is a slow-burn warning — a few retired pages are normal, but a growing count means the GPU is degrading.
Because Xid errors are the earliest and most reliable signal of GPU hardware trouble, they belong in log monitoring — a Splunk or Datadog alert (or local journalctl tailing) on the NVRM: Xid string, paging out immediately rather than waiting for the training job itself to fail.
SRE interviews at ML infrastructure companies often include: "A training job crashes with no Python traceback. The process just disappears. How do you diagnose it?" The answer: check dmesg for Xid errors first. Xid 79 (GPU fallen off bus) kills the process without a Python exception because the GPU hardware itself has disconnected. The Python process gets a signal, not an exception. This is also why production training jobs should checkpoint frequently — an Xid 79 means you lose all in-memory state.