Quantization maps high-precision weights to lower-precision formats using a scaling factor to preserve dynamic range

Converting a model weight from FP16 to FP8 naively clips values outside the FP8 dynamic range (e.g., FP8 E4M3 spans only ±448 versus FP16’s ±65504) and wastes bits on rarely-used extremes. The standard fix is absmax scaling: compute the absolute maximum of the tensor, derive a scale factor that maps that maximum to the edge of the FP8 range, divide the tensor by the scale before quantizing, and store the scale alongside the quantized data. At inference, dequantize by multiplying back by the scale. This produces a ‘derived datatype’ — the quantized values are meaningless without the scale. Per-tensor scaling is simple; finer-grained block scaling improves accuracy for non-uniform distributions.

Examples

scale = abs_max / 448.0; tensor_q = (tensor / scale).to(torch.float8_e4m3fn); tensor_dq = tensor_q.float() * scale.

Assessment

Explain why per-tensor absmax scaling loses accuracy for a weight tensor with a few very large outliers and most values near zero. What finer-grained approach addresses this?