memory and model

LLM quantization: choosing Q4, Q8 or FP16

Quantization stores models more compactly and reduces memory requirements.

LLM¹ quantization² represents model weights using fewer bits. Converting the same original model to Q4 or Q8 can reduce storage and execution memory. Choose a file by considering the app, available memory, and answer quality on your tasks.

A download list containing Q4_K_M, Q8, and FP16³ can make the smallest file tempting. Smaller files help fit larger models on your machine, but the compression method can also affect answers and speed. We will first explain the labels, then narrow the choice with the same questions.

Reduce the bits used to represent weights.

A model uses numerical weights to process input and calculate the next token⁴. Quantization represents these values with fewer levels to store them compactly. Moving from 16 to 4 bits quarters the bits for the numbers themselves. Actual files also contain auxiliary values, metadata, and components kept at higher precision, so check the download size.

A model can answer the same PDF differently after its weight representation changes because quantization reduces the levels used to express the original values. Compare the benefit of fitting the model with whether the answer retains the information important to your work.

Labels are a starting point; check actual sizes in the distributed files.
Label	Weight representation	When to consider
Q4	Primarily 4-bit quantization	Reducing memory to run a model
Q8	Primarily 8-bit quantization	Comparing quality with Q4 when memory allows
FP16	16-bit floating point	Examining higher precision and its memory needs

Dense weight values grouped into fewer levels during quantization — Representing weights in fewer bits reduces storage.

Choose a file supported by your app first.

Start by narrowing candidates to formats your app can read: GGUF⁵ for llama.cpp-based runtimes, MLX⁶ conversions for MLX, and the appropriate releases for AWQ/GPTQ servers. Within that format, compare file sizes and answers at different precisions, such as Q4 and Q8. Different group sizes, correction values and mixed precision can change results even with the same 4-bit label.

Also check the original model, tokenizer, and chat template. Features needing additional components, such as an image encoder or MTP⁷, require them in the distribution. Checking both why the file is smaller and whether the needed features are supported reduces repeat downloads.

Weight representation changing from a fine grid to a coarse grid — Grouping and correction can differ even at the same bit count.

A smaller file can change the execution path.

Consider Q8 exceeding VRAM⁸ and placing some weights in system RAM⁹, while Q4 fits entirely on the GPU¹⁰. Less offloading¹¹ with Q4 reduces weight-transfer time between the two memories. This is one reason quantization can noticeably change performance on the same machine.

If both files fit on the GPU, data volume and the efficiency of the quantized kernels matter. Record prefill¹² and decode¹³ separately for the same input instead of using the file-size ratio. Keeping the original model, exact filenames, app version, memory placement, and cache settings makes the combination reproducible.

Longer contexts need more memory — Qwen3.8-27B · Q4_K_M · MTP off · site memory estimate — Calculated weights, KV cache and runtime overhead. Leave additional memory for the operating system and other apps.

A balance comparing model size, memory savings, and answer quality — Compare results and review effort on actual questions.

Compare answers and review time on important questions.

For contract renewal dates and automatic-renewal exceptions, first mark the correct answers and evidence in the source. Give Q4 and Q8 the same PDF and questions and compare dates, applicability, and exceptions. Keep the extracted input the same to examine weight-representation differences.

Where possible, match sampling and seed settings and repeat across several questions. Separate incorrect evidence, missing conditions, and unanswered items, and record human review time. Reliable Q4 answers can justify starting with the current setup; consistently fewer important errors with Q8 can justify additional memory.

Change weight quantization and KV-cache¹⁴ precision one at a time. Comparing a larger Q4 model with a smaller Q8 model also changes size and architecture, so compare the results and time to finish the task in each setup. The goal is reliable task completion rather than the smallest file or highest bit count.

Terminology notes

Large language model — A model trained on large text datasets to learn language patterns and process or generate text in context.
Back to the text
Quantization — A way to represent model weights or values with fewer bits, reducing storage and memory use. The coarser representation can also affect accuracy.
Back to the text
FP16 — A 16-bit floating-point format. It uses the same bit width as BF16 but allocates bits differently between exponent and significand.
Back to the text
Token — A unit into which a model divides input or output for processing.
Back to the text
GGUF — A file format for model data, widely used by llama.cpp-based tools.
Back to the text
MLX — Apple’s machine-learning framework, which runs models using memory shared by the CPU and GPU on Apple Silicon.
Back to the text
MTP — A training objective or model component for predicting multiple future tokens at once. In speculative decoding, those predictions can be proposed as candidate tokens.
Back to the text
VRAM — Memory used by a graphics card’s GPU to store model weights and intermediate values.
Back to the text
System RAM — System memory that temporarily holds data while programs run.
Back to the text
GPU — A processor designed to handle many calculations in parallel. It performs model computations during AI inference.
Back to the text
Offloading — Moving some model data from GPU memory to system RAM or storage when capacity is limited. This adds data transfer.
Back to the text
Prefill — The stage where an LLM reads the input prompt and computes representations for its tokens. Longer prompts contain more tokens to process.
Back to the text
Token decoding — The stage that generates successive output tokens using the input and the response produced so far.
Back to the text
KV cache — Memory that stores attention keys and values from earlier tokens for reuse during later token generation. Its size depends on context length and batch size.
Back to the text

Frequently asked questions

What should you check first in this article?: LLM quantization represents model weights using fewer bits. Converting the same original model to Q4 or Q8 can reduce storage and execution memory. Choose a file by considering the app, available memory, and answer quality on your tasks.
How much memory does it need?: Where possible, match sampling and seed settings and repeat across several questions. Separate incorrect evidence, missing conditions, and unanswered items, and record human review time. Reliable Q4 answers can justify starting with the current setup; consistently fewer important errors with Q8 can justify additional memory.