memory and model

LLM quantization: choosing Q4, Q8 or FP16

It is the same model, but in the download list it has several names such as Q4 and Q8. If I get a smaller file, will the answers be just as bad? Is it always a good choice if I receive a large file? Let's see what I can lose by reducing the capacity and how I will see the difference in my own work.

Reduce the precision of the weights, not the number of words

Model weights are stored as large numbers. Quantization changes the representation of this number into fewer bits, reducing file and memory burden. It is not a compression method that removes sentences to summarize, and it generally does not restore the original numbers exactly the same.

As a simple assumption, if we reduce the representation of a number from 16 bits to 4 bits, that part becomes 1/4. The actual file also contains auxiliary information and parts with higher precision, so they are not reduced at exactly the same rate. That's why you need to check the download size yourself.

Illustration of the quantization process, which reduces dense model weights to a smaller grid by grouping them into several levels of values.
Quantization packs weights into fewer steps, saving files and memory, but stronger compression means more loss potential.

Even within the same Q4 name, it is different

The results will vary depending on which weights you leave more precise and how you divide the bundle. The 4-bit notations of Q4_K_M, AWQ, GPTQ and MLX cannot be treated as one file. First, check the formats supported by the executable program.

In addition to the size of the model, you should also check whether the transform includes the required tokenizer, template, vision or MTP components. It is also necessary to distinguish between quantization or missing necessary functions that cause the file to become smaller.

An illustration in which the weight expression level is reduced from the fine original to the medium and coarse levels.
As the number of bits decreases, the storage size decreases, but even with the same 4 bits, the results vary depending on the group size and correction value method.

Crossing the memory-fit threshold can make a big difference

Models that were partially pushed to RAM at high precision may end up on the GPU at low precision. In this case, there is room for a greatly improved experience by reducing the transmission burden. There is also the benefit of reducing the amount of weights to be read.

Conversely, models that already fit comfortably may experience small or unexpected speed differences depending on the cost of processing the compressed representation and kernel efficiency. Instead of calculating that half the files require half the creation time, compare across the same app and device.

Illustration showing the balance between model size, memory savings, and preserving answer quality.
It is better to choose a quantization that will fit comfortably into your device's memory while maintaining quality for frequently used tasks rather than the smallest file.

I pick the mistakes I care about.

Even small changes in overall benchmark score can make a significant difference in my work. Prepare items that can confirm the correct answer, such as the result of executing the code, the amount or conditions in the document, or the format of the tool call. It is difficult to judge accuracy based on natural sentences alone.

Send the same question to Q4 and Q8 and see if the error repeats. Multiple cases are needed because a single difference in answer may be a variation in production. Whether the lower precision of a large model or the higher precision of a small model is better may vary from task to task.

Keeping higher precision has a cost

If you need to buy more memory to use the Q8, check first if you really need the difference. If the Q4 does what you often do well enough, there's less reason to upscale the machine for higher precision. Conversely, if critical mistakes are reduced, there is a reason for additional costs.

Model weights and KV cache precision are separate. Instead of changing both settings at once, leave the compared files and settings. The criteria for selection is to obtain results that I can trust, review, and write about, rather than having the largest file.