Hardware Acceleration: Why Integers Are Cheaper

The Silicon Argument

The speed gains from quantization are not just about moving less data — the hardware itself is physically more efficient at integer arithmetic than floating-point arithmetic.

A floating-point adder on a chip must handle exponents, mantissas, normalization, and rounding according to the IEEE 754 standard. An integer adder just adds bits. The integer unit is physically smaller, uses less power, and can be replicated more times on the same die area.

Modern hardware exploits this with specialized instruction sets:

• Intel CPUs: AVX-512 VNNI (Vector Neural Network Instructions) — processes INT8 dot products in a single instruction, achieving 4× the throughput of equivalent FP32 operations.
• NVIDIA GPUs: DP4A instruction and Tensor Cores — can perform INT8 matrix multiplications at 2–4× the throughput of FP16, and up to 8× versus FP32.
• ARM CPUs (mobile/edge): NEON SDOT instructions — similar INT8 acceleration for mobile inference.
• Dedicated AI accelerators (Google TPU, Apple Neural Engine, Qualcomm Hexagon): designed from the ground up for low-precision integer math, often running exclusively in INT8 or INT4.

The Memory Bandwidth Bottleneck

For large models, inference is often memory-bound, not compute-bound. This means the GPU's compute units sit idle waiting for weights to arrive from memory — the bottleneck is the speed of the memory bus, not the speed of the math.

Quantization directly attacks this bottleneck:

• A 4× smaller model means 4× less data to transfer from DRAM to compute units.
• This alone can produce a 2–4× speedup even on hardware that doesn't have specialized INT8 instructions.
• On edge devices (Raspberry Pi, mobile phones) where memory bandwidth is severely constrained, this effect is even more pronounced.

Practical Speedup Expectations

The speedup is multiplicative with model size reduction: a 4× smaller model that also runs 2× faster per operation delivers an 8× improvement in effective throughput.