Quantization

TL;DR

• Quantization reduces model weight precision from 32-bit floats to 8-bit or 4-bit values, often cutting weight memory by 4-8x; the quality and speed impact must be measured for the target task.
• A Llama 3 70B model in FP16 needs about 140GB for weights (before runtime overhead). An INT4 artifact can be around 35-40GB, but the actual deployment also needs KV-cache memory and supported kernels.
• INT8 and INT4 quality changes vary by model, calibration data, backend, and benchmark. Below INT4, the tradeoff is often harder to justify, but no bit width is universally safe.
• Three formats dominate: GGUF (CPU/local via llama.cpp and Ollama), GPTQ (GPU, layer-by-layer error minimization), AWQ (GPU, activation-aware, increasingly the default).
• QLoRA quantizes the base model to 4-bit and adds trainable LoRA adapters in FP16, making 70B fine-tuning possible on a single 40GB GPU.
• The engineering decision: quantization can be a practical path from a memory-constrained model to a deployable artifact, provided the target backend and task quality are validated.

---

30-second mental model

Quantization stores some model values with fewer bits. That reduces weight memory and often improves bandwidth, but it introduces approximation error and can add dequantization work. The useful question is not “what is the smallest file?”; it is “which precision and runtime meet this task's quality, latency, memory, and safety targets on the target hardware?”

5-minute explanation

Start from a validated FP16 or BF16 baseline. Weight-only quantization is usually the simplest deployment change; activation-aware or mixed-precision methods can protect sensitive layers when weight-only INT4 loses too much quality. The artifact format and kernel support matter as much as the bit width: GGUF, GPTQ, AWQ, and other formats target different runtimes and hardware.

Measure peak memory—including the KV cache—alongside throughput, tail latency, batch behavior, and task-specific quality. Calibration data should resemble production inputs. A quantized checkpoint is a deployment candidate, not a drop-in guarantee: keep a rollback path and validate the exact backend and model configuration.

---

The problem it solves

You've found the perfect open-weights model for your use case. Llama 3 70B crushes your evaluation benchmarks, handles your domain terminology, and your team is ready to deploy. Then you check the hardware requirements.

A 70B parameter model in FP16 (the standard training precision) needs roughly 140GB just to load the weights into memory. A single NVIDIA A100 has 80GB. So you need at least two A100s in a tensor-parallel configuration, which costs $5-8/hour on AWS or GCP. For a startup running 24/7 inference, that's $3,600-5,800/month, and that's before you account for KV cache memory, concurrent requests, or redundancy.

Teams facing this memory wall often either choose a smaller model or use a hosted API, trading away some combination of quality, latency control, cost predictability, or data locality. Quantization adds a third option, provided the resulting artifact meets the task's quality and operational requirements.

The root cause is simple: each model parameter is stored as a 16-bit floating-point number. A 70B model has 70 billion of these. 70 billion x 2 bytes = 140GB. The precision of those numbers is far higher than what inference actually needs.

The practical point is that model weights may carry more precision than a deployment needs, but the savings must be weighed against quality loss, runtime support, and the memory used by activations and KV cache.

Here's the scale of the problem across common model sizes:

---

What is it?

Quantization is the process of reducing the numerical precision of a model's weights from high-bit floating-point numbers (FP32 or FP16) to lower-bit integers (INT8 or INT4). Each parameter takes less storage, so the entire model fits in less VRAM and transfers faster through the memory bus.

Think of it like converting a high-resolution photo to a JPEG. The raw image might be 50MB, but a well-compressed JPEG at 90% quality is 5MB, and your eyes can barely tell the difference. Below 60% quality, things get noticeably worse. Quantization works the same way: there's a large compression range where quality loss is negligible, and a threshold below which things degrade fast.

A 70B model in FP16 takes 140GB. In INT8 that drops to 70GB. In INT4 it's roughly 35-40GB. That single transition (FP16 to INT4) is what makes a 70B model deployable on a single consumer-grade GPU instead of a multi-GPU cluster. That's the jump that changed the open-source LLM ecosystem.

The tradeoff is precision. You're mapping continuous floating-point values into a small set of discrete integers. Modern quantization methods (GPTQ, AWQ, GGUF variants) minimize this mapping error carefully, but some quality is always lost. The engineering question is: how much quality can you afford to lose?

---

How it works

Numeric precision: the precision ladder

Every number in a model has a bit-width that determines how precisely it can represent a value. Here's the precision ladder LLMs typically move down:

FP32 stores each weight as a 32-bit IEEE 754 float with full mantissa precision. FP16 halves that to 16 bits, and BF16 (Brain Float 16) uses the same 16 bits but allocates more to the exponent range, which works better for the value distributions in neural networks. FP16/BF16 are common training and serving baselines, but lower precision can still affect quality relative to FP32 or a particular workload.

The real compression starts at INT8: you're converting floats to 8-bit values. Below INT4, quality often degrades more sharply, especially on small or sensitive models. INT2 and 1-bit methods remain workload- and backend-dependent and are less common in general-purpose serving.

Here's the full quantization pipeline, from trained model to deployed endpoint:

Post-training quantization (PTQ)

PTQ is the most common approach: take an already-trained model and convert its weights to lower precision without any retraining. You're compressing after the fact.

The simplest version is uniform quantization: find the min and max of a weight tensor, divide that range into $2^n$ buckets (where $n$ is your target bit-width), assign each weight to its nearest bucket, and store the bucket index plus a scale factor. At inference, multiply the integer by the scale factor to recover an approximate float.

The problem with uniform quantization across an entire tensor is that weight distributions aren't uniform. Some layers have outliers 100x larger than the median. If you set your scale based on those outliers, you waste most of your integer range on values that never appear.

Modern PTQ methods fix this with per-group quantization: instead of one scale factor per tensor, compute separate scale factors for small groups of weights (typically 32 or 128). Each group gets its own min/max range, so outliers in one group don't waste precision in another.

The calibration step is crucial. PTQ methods run a small calibration dataset (typically 128-512 samples) through the model to measure activation patterns and weight distributions. This statistical profile guides the quantization decisions. Poor calibration data leads to higher-than-expected quality loss.

Quantization-aware training (QAT)

QAT takes a different approach: simulate quantization during training itself. The model learns to produce good outputs despite the reduced precision, effectively adapting its weights to be more "quantization-friendly."

During forward passes, weights are quantized (simulated) to the target precision. During backward passes, gradients flow through the quantization step using straight-through estimators (treating quantization as an identity function for gradient purposes). The model adjusts its weights so that the quantized versions still produce good outputs.

QAT can produce better quality at the same bit-width than PTQ, but the lift varies by model, data, and training recipe. The cost: you need to retrain (or continue-train) the model, which requires suitable training infrastructure.