Model Quantization Explained: Concepts, Methods, and Real-World Impact

Large language models keep growing, and the hardware needed to run them often cannot keep pace. Model quantization has become one of the most reliable solutions to this gap. It lets teams take a model that would otherwise demand a rack of GPUs and run it on far more modest infrastructure, usually without giving up much in the way of output quality. This article walks through what quantization actually does, how it works under the hood, the methods practitioners rely on, and why it has become important for improving LLM inference efficiency.

The Basic Idea Behind Quantization

At its core, an AI model is a massive collection of numbers called weights, learned during training and stored at high precision. Model quantization takes those numbers and represents them using fewer bits, for example moving from 32-bit floating point down to 8-bit or 4-bit integers. Since a model can contain billions of these values, even a modest cut in bits per number produces a large drop in total memory footprint.

A useful comparison is compressing a high-resolution photo into a smaller file. The image still looks nearly the same to the eye, yet it takes up a fraction of the space. Quantization works on a model in much the same way, trading a small amount of precision for a large gain in efficiency.

The numbers speak clearly here. According to NVIDIA’s technical breakdown, a Llama2 7B model stored in FP16 needs roughly 14 GB of memory, and quantizing it to FP8 cuts that figure to about 7 GB, halving the footprint in one step. Outcome School’s walkthrough puts similar numbers in even starker terms: a 7 billion parameter model needs about 28 GB in FP32, drops to roughly 7 GB in INT8, and shrinks further to around 3.5 GB in INT4. That is an eightfold reduction from the original size, enough to move a model from a data center requirement down to something that fits on a single consumer GPU or a capable laptop.

What Actually Gets Quantized

Three components of a model are candidates for quantization, and each behaves differently.

• Weights: These are static and fixed once training ends, which makes them the simplest and most common target for compression.
• Activations: These are the intermediate outputs produced as data flows through the model during inference. They are dynamic, meaning their range depends on the input, so they need to be observed and calibrated rather than quantized in isolation.
• KV cache: For decoder-based transformer models, this cache stores key and value pairs used to speed up autoregressive generation. With long context windows, it can add several gigabytes to total memory use, making it a worthwhile target as well.

Understanding which of these pieces a given method touches helps explain why some techniques deliver bigger speed gains than others, a point worth keeping in mind as we examine how quantization is computed.

The Mechanics: Scale and Zero Point

Every quantization scheme relies on two values to translate a wide range of decimal numbers into a small set of integers. The scale defines how much real value one integer step represents, and the zero point marks which integer stands in for the real value zero.

A number gets quantized with the formula quantized_int = round(real_value / scale) + zero_point, and it can be approximately recovered with real_value ≈ (quantized_int − zero_point) × scale. That word "approximately" matters. Converting a value down and back rarely returns the exact original number, and the resulting gap is called quantization error. It is small and mostly harmless at 8-bit precision, and it becomes more noticeable as bit width drops further.

There are two common ways to set up this mapping. Symmetric quantization centers the range around zero, which keeps the math simple and suits data such as weights that spread fairly evenly across positive and negative values. Asymmetric quantization shifts the zero point to fit data that leans to one side, such as activation outputs that never go negative. Teams also choose how finely to apply these parameters, whether across an entire tensor at once, per channel, or in smaller blocks, with finer granularity generally producing better accuracy at the cost of a bit more overhead.

Two Paths to a Quantized Model

Broadly, there are two moments at which quantization can happen.

Post-training quantization takes a model that has already finished training and converts its numbers afterward. It needs little more than a small calibration dataset to measure typical value ranges, which is why it has become the default choice for most teams working with open-weight models.

Quantization-Aware Training (QAT) instead simulates the effects of low-precision arithmetic while the model is still being trained, using a technique called the straight-through estimator to work around the fact that rounding operations are not differentiable. The model effectively learns to compensate for the errors quantization will later introduce. This approach costs more time and compute, but it tends to preserve more capability, particularly at very low bit widths where post training methods start to strain.

Recent releases show how this approach is being used in practical deployments. In June 2026, Google DeepMind released Quantization-Aware Training (QAT) versions of Gemma 4 designed for laptops, mobile devices, and consumer GPUs. The mobile-focused format reduces the memory footprint of the Gemma 4 E2B model to around 1 GB, while the QAT approach is designed to preserve model quality better than standard post-training quantization.

The Outlier Problem and How Modern Methods Handle It

Quantizing weights is relatively straightforward. Quantizing activations is harder, and the difficulty grows sharply once models pass a certain scale. Most activation values sit in a narrow, well-behaved range, but a small number of outliers can spike far outside it. Forcing a single scale to cover both the ordinary values and the extreme ones crushes the ordinary values together, wiping out detail that the model actually needs.

Several methods have been built specifically to work around this. GPTQ (Generative Pre-trained Transformer Quantization) quantizes weight matrices one row at a time, using second-order information to correct for errors as it goes, and it was the first approach to compress models down to 4-bit precision while keeping accuracy intact. AWQ (Activation-aware Weight Quantization) identifies the small set of weights most closely tied to large activation values and protects them from aggressive rounding. SmoothQuant takes a different route, mathematically shifting some of the outlier burden from activations onto weights so that both can be quantized to 8 bits together. For running models locally, tools such as bitsandbytes and the GGUF (GGML Unified Format), used by llama.cpp package these ideas into a form that lets a large model run on a personal laptop with a few lines of code.

Quantization is also moving beyond conventional 4-bit and 8-bit approaches. Google DeepMind’s Gemma 4 QAT work uses targeted 2-bit quantization for selected parts of the model while keeping more important reasoning layers at higher precision. This reflects a broader shift towards hardware-aware and selective quantization, where different parts of a model can use different precision levels depending on their role and the device running them.

Why This Work Matters in Practice

The value of AI model optimization through quantization is not abstract. It determines whether a promising model ships as a usable product or stays confined to expensive infrastructure. A model that would otherwise require several GPUs can often be served on one once quantized, and that single change lowers hosting costs, shortens response times, and makes it realistic to keep sensitive data processing on premises rather than routing it through external servers.

The production impact can be substantial. Oracle reports that FP8 quantization reduced inference latency by 10% for the multimodal Llama 3.2-90B model while allowing the deployment to use half the number of GPUs. For Llama 3.3-70B, the company achieved more than 99% model-quality recovery, a 30% reduction in latency, and a 50% increase in server throughput using the same number of GPUs. These results show why quantization is increasingly treated as a deployment requirement rather than simply an optimization technique.

Machine learning compression techniques like this also stack with other efficiency strategies. A model that has already been distilled into a smaller student version gains further savings once it is quantized on top of that, compounding the benefits rather than competing with them. The choice of precision level should still match the job at hand. A high-volume classification workload can usually tolerate aggressive compression without issue, while a task involving multi-step reasoning may call for a higher precision setting to avoid degraded output.

Conclusion

Quantization has moved from a niche academic technique into a standard part of deploying language models responsibly. It offers a workable answer to the tension between model capability and available hardware, letting organizations run capable systems on equipment they already own while keeping accuracy losses within an acceptable range. As models continue to scale up, the methods covered here are likely to remain one of the most dependable tools for keeping AI practical to run.