The Connectivity Matrix
When you have multiple GPUs, the performance of collective operations (All-Reduce, All-Gather) depends entirely on how those GPUs are connected. nvidia-smi topo -m prints the full connectivity matrix: an N×N grid where each cell names the connection type between that pair of GPUs, alongside columns for each GPU's local CPU core range and NUMA node.
Connectivity codes:
| Code | Meaning | Relative Speed |
|---|---|---|
X |
Same GPU | — |
NV# |
NVLink (# = number of links) | Fastest |
PIX |
PCIe Switch (same switch, different slot) | Fast |
PHB |
PCIe Host Bridge (different PCIe domains, same CPU) | Slower |
NODE |
Different NUMA node, same system | Slow |
SYS |
Different NUMA nodes, different sockets | Slowest |
Practical use:
If you are running model parallelism where GPU 0 and GPU 4 exchange activations every forward pass, and the matrix shows SYS between them, you have a serious performance problem. Reconfigure your parallelism strategy to keep tightly-coupled GPU pairs within the same NVLink domain. Cross-referencing that same topology output against the system's NUMA layout tells you which CPU cores to pin the process to for optimal locality.