What should you check first in this article?

The RTX 3090 BF16 unlock raises matrix throughput, but single-request token generation is bound by memory bandwidth. Prefill of long documents, batched requests and LoRA fine-tuning carry more compute, so they may see an effect. No public 3090 measurement exists yet, so set an expectation and measure it yourself.

Executable programs and extensions

Does the 3090 unlock speed up local LLMs? Prefill, batching, decode

The difference can appear in prefill, concurrent requests and fine-tuning rather than single-request decode.

The RTX 3090 BF16¹ unlock raises matrix throughput², but single-request token³ generation is bound by memory bandwidth⁴. Prefill⁵ of long documents, batched requests and LoRA⁶ fine-tuning carry more compute, so they may see an effect. No public 3090 measurement exists yet, so set an expectation and measure it yourself.

Contents

Requirements and key details

Decode is bound by 936 GB/s of bandwidth, so higher BF16 throughput barely changes it.
Prefill, batching and LoRA carry more compute, so the unlock may show up there.
Fix the power limit, model and quantization, and compare before and after under the same conditions.

Decode is bound by bandwidth, not compute

Generating one token at a time means reading weights from memory at every step. The RTX 3090's 936 GB/s decides how long those reads take, not the BF16 matrix throughput. A higher compute rate therefore barely moves single-request tok/s.

The 3090 measurements published on this site point the same way: Qwen3.8 27B UD-Q4_K_XL with MTP⁷ n-max 3 ran at 68.9 tok/s at 350 W and 51.0 tok/s at 250 W. Lowering the power limit slows it down, which is a clock and power effect, not a compute one.

An RTX 3090 in a case, colored blocks moving along a conveyor to shelves, and cards printed with network diagrams on the work surface — Decode speed is tied to memory bandwidth, so a compute unlock alone changes it little.

Prefill is compute-bound and may change

Prefill reads a long document at once and runs matrix products over every input token, a compute-bound stage. A runtime⁸ that prefills in BF16 may see a shorter time to first token⁹ after the unlock. Paths that do not use BF16 matrix products, such as llama.cpp's quantized kernels, see little effect.

Which runtime prefills in BF16 depends on the model and kernel, and vLLM and llama.cpp do not take the same path for the same model. To estimate the prefill gain, start with the kernels your runtime uses.

Concurrency and batching widen the gap

Serving several requests at once reuses the same weights, reducing bandwidth pressure and raising the compute share. BF16 throughput can then set aggregate throughput, which is why a continuously batching server such as vLLM may show more benefit.

In a single-user, one-request-at-a-time setup, no batch forms and this advantage mostly disappears. Expectations differ between a server operator and a personal desktop user.

Fine-tuning and image generation are separate cases

LoRA fine-tuning uses many BF16 matrix products, so it is theoretically where the unlock helps most. The 3090's 24 GB of VRAM¹⁰ limits batch size¹¹ and sequence length, though, so memory becomes the constraint before compute does. Larger batches have more room to benefit.

In image generation, the compute and memory shares depend on model, resolution and step count. Compute-heavy FP16¹² sections may benefit, while high resolutions hit VRAM and bandwidth first. The before-and-after difference in image generation varies with the model, resolution and step count, so measuring your own settings answers it fastest.

Two graphics cards side by side, one short with a single fan and one long with two fans — Even 24 GB cards differ by generation and bandwidth, so compare them separately from the unlock.

Measure with power fixed, before and after

Record a baseline at the same model, quantization¹³, context length¹⁴ and power limit, then repeat after the unlock with a cold boot. Decode¹⁵ alone will show almost nothing, so log prefill time and concurrent throughput as well.

Bottleneck and expected effect by workload
Workload	Bottleneck	Expectation
Single-request decode	Memory bandwidth	Minimal
Long-document prefill	Matrix compute	Depends on runtime kernels
Concurrent requests	Matrix compute	Larger with bigger batches
LoRA fine-tuning	Matrix compute, VRAM	VRAM binds first

So should you buy one

The unlock is an experiment on compute headroom you already own. It is a weak reason to buy a card; if decode is the goal, look at other options such as a 5090 or a 48 GB mod. The 3090's strength is 24 GB of capacity and price, not compute rate.

The decision comes from measurement, so decide after running the same card before and after.

A bar chart comparing RTX 3090 decode speed of 68.9, 65.3, 62.3 and 51.0 tok/s against measured GPU power of 347.7, 298.7, 274 and 249.5 W at 350, 300, 275 and 250 W limits — Lowering the power limit reduces decode speed, so keep power fixed in the comparison.

Terminology notes

BF16 — A 16-bit floating-point format for model values. Its exponent has the same width as FP32, giving it a similarly wide value range.
Back to the text
Throughput — The amount of work processed or generated over time. Comparisons need the unit, such as tokens per second or requests per second.
Back to the text
Token — A unit into which a model divides input or output for processing.
Back to the text
Memory bandwidth — The amount of data transferable between memory and a processor per unit time, usually expressed in bytes per second (B/s).
Back to the text
Prefill — The stage where an LLM reads the input prompt and computes representations for its tokens. Longer prompts contain more tokens to process.
Back to the text
LoRA — A method that learns and applies small additional weights to adapt a base model. The adapter is used with the base model for purposes such as style adaptation or few-step generation.
Back to the text
MTP — A training objective or model component for predicting multiple future tokens at once. In speculative decoding, those predictions can be proposed as candidate tokens.
Back to the text
Runtime — The software environment that provides facilities needed while a program runs. In local AI, it can also refer to a model execution engine.
Back to the text
Time to First Token — The time from sending a request until its first output token arrives.
Back to the text
VRAM — Memory used by a graphics card’s GPU to store model weights and intermediate values.
Back to the text
Batch size — The number of inputs or requests processed together in one batch.
Back to the text
FP16 — A 16-bit floating-point format. It uses the same bit width as BF16 but allocates bits differently between exponent and significand.
Back to the text
Quantization — A way to represent model weights or values with fewer bits, reducing storage and memory use. The coarser representation can also affect accuracy.
Back to the text
Context window — The token span of input and generated content a model can handle in one request. The supported limit and memory use depend on the model and runtime settings.
Back to the text
Token decoding — The stage that generates successive output tokens using the input and the response produced so far.
Back to the text

Frequently asked questions

What should you check first in this article?: The RTX 3090 BF16 unlock raises matrix throughput, but single-request token generation is bound by memory bandwidth. Prefill of long documents, batched requests and LoRA fine-tuning carry more compute, so they may see an effect. No public 3090 measurement exists yet, so set an expectation and measure it yourself.
How much memory does it need?: Generating one token at a time means reading weights from memory at every step. The RTX 3090's 936 GB/s decides how long those reads take, not the BF16 matrix throughput. A higher compute rate therefore barely moves single-request tok/s.