The "top" for GPUs
nvidia-smi (System Management Interface) is the primary CLI tool for inspecting GPU state. It ships with the NVIDIA driver and requires no additional installation. Think of it as top or htop, but for your GPU.
Basic invocation:
Running nvidia-smi with no arguments prints a status table for every GPU in the system: driver and CUDA version up top, then a per-GPU row of temperature, fan speed, performance state, power draw versus cap, memory used versus total, and utilization percentage.
Key fields and what they tell you:
| Field | What to Look For |
|---|---|
GPU-Util |
% of time GPU kernels were executing. 90–100% = healthy. <20% = GPU is starving (CPU bottleneck). |
Memory-Usage |
VRAM allocated. PyTorch's caching allocator pre-allocates aggressively, so 100% usage is normal. OOM errors appear in application logs, not here. |
Pwr:Usage/Cap |
Power draw vs. TDP. A 400W card drawing 50W is idle. Low power with low GPU-Util confirms the GPU is waiting for work. |
Temp |
GPU die temperature. Most GPUs throttle at 80–85°C (thermal throttling). Sustained throttling indicates inadequate cooling. |
Perf |
Performance state. P0 = maximum performance. P8 = idle. If you see P8 during a workload, the driver is not recognizing the load. |
Persistence-M |
Persistence mode. When On, the driver stays loaded between jobs, eliminating ~1–2s initialization overhead per job. |
Continuous monitoring:
The tool can refresh that table on an interval for live monitoring, or switch to a scriptable CSV output of just the fields you name — timestamp, utilization, memory, temperature, power draw, and so on. That CSV format is what feeds GPU metrics into monitoring systems like Prometheus (via nvidia_gpu_exporter) or into a file for post-hoc analysis.