How does my computer rank?

Measure local AI generation speed directly. The GPU name and logical processor count are displayed within the range accessible through the browser. Clicking on the chipset details will retrieve additional information from the graphics driver. CPU and RAM specifications that are not recognized can be entered manually, and specifications entered manually are stored only on the device and are not reflected in the rankings.

Standard measurement performs three measurements of 128-token decoding, excluding the first token, after prefill execution, using the same question, and records the median value. You may share your personal best records and results.

Initially, it displays the top % based on the experience-based virtual comparison group. This is not based on actual computer distribution measurements or estimations. If there are 30 or more records agreeing to submission under identical conditions, the ranking is converted to participant ranking, and the virtual comparison group and actual participant data are not mixed. This is not a ranking of overall computer performance or the speed ranking of large models.

What can you check?

It is not a simulation that only reorders pre-defined sentences. When a user directly inputs a question, the model file and computation run within the user's browser, and responses appear one token at a time. After completion, it simultaneously displays the time taken from the first token to appear and the actual token generation speed.

Things to know before execution

On the first run, download the Qwen 3.5 0.8B Q4 model file; from subsequent runs, use the browser cache. Execution requires approximately 1.6GB of GPU memory and a web browser supporting WebGPU, such as the latest Chrome, Edge, or Safari. Due to its small size, it is better suited for examining local execution processes and device-level perceived performance rather than for verifying the latest facts.

Where is the question processed?

The input question and generated answer are not sent as analysis data for this experience feature. The model is downloaded only after the button is pressed, and currently running tasks are terminated when the tab is closed.

Illustrative demo

Illustrative experience: no model is run and no recording or audio is uploaded. Displayed times use measured data or estimates under stated conditions and may differ from actual use.

Speech models covered

Qwen3-ASR 0.6B·1.7B, Whisper, faster-whisper, Parakeet v3, Qwen3-TTS, Kokoro, Chatterbox

Model guides

• Transcribe Korean speech live and save text with Nemotron 3.5 ASR
• Transcribe English speech in real time with Moonshine on CPU
• Align Korean subtitles and speaker labels with WhisperX
• Generate Korean speech on CPU with Supertonic 3
• Transcribe English audio locally with Granite Speech 5 TurboCTC
• Generate Korean speech with VoxCPM2's 30-language TTS
• Generate Korean speech with OmniVoice's 600+ language and dialect coverage
• Generate speech on a CPU with Pocket TTS in six languages
• Make a two-speaker dialogue WAV with Fish Audio Drama 3
• Transcribe Korean recordings on a Mac with WhisperKit
• Edit Audio Locally with AuK: Speech Revisions, Cleanup, and TTS
• Creating a local voice assistant: ASR·LLM·MCP·TTS connection guide
• Local AI speech recognition: transcribe on your own PC
• Qwen3-ASR vs Whisper: which local transcription model should you choose?
• Make speech on your own computer: a local TTS guide
• What is Qwen3-TTS? Turn text into speech on your computer
• Compare local TTS models: languages, hardware and speed evidence
• Transcribe Korean recordings on your own computer with whisper.cpp
• mlx-audio: TTS and STT from a single install
• Fish Audio Drama 3 int4: 41.5 GB down to 22 GB
• Piper in Korean: adding a ko_KR voice to Home Assistant
• Home Assistant voice announcements: speaking through the house with a Piper voice
• Splitting speech off with Wyoming: moving transcription to another machine

comparable items

• expected time to generate one result
• difference based on output resolution and video length
• GPU memory loading status and necessity of offloading
• Expected power consumption per generation

Tasks possible on local devices

• Generate multiple thumbnails and sians
• Create posters and product images with text chunks
• Transform and synthesize based on reference image
• Turn a single photo into a short video
• Creating short-form content with dialogue and sound effects

image generation model

• Z-Image-Turbo: Diffusers · 9 steps / 8 DiT forwards · CFG 0
• Qwen-Image 2.1 Turbo: Diffusers · checkpoint 8-step schedule · CFG 1
• Pruna Qwen-Image 2.1: Pruna · distilled checkpoint
• FLUX 3 Image Klein · distilled: Create images with references and bounding boxes
• Qwen-Image 2.1: Create images, replace backgrounds and make transparent assets with one model
• FLUX.2 [klein] 4Bfast sketch, thumbnail, repeated generation
• FLUX.2 [klein] 9B9B model for image generation
• Qwen-Image-2512poster, mixed Korean-English image, realistic representation
• FLUX.2 [dev] 32Bimage with multiple objects

video generation model

• MiniMax H3-Baseshort-form with dialogue and sound effects, control over first and last frames
• LTX-2.5 Audio Videoa local pipeline that generates audio and video at once
• HunyuanVideo-1.5a short video applied with motion to a single image
• Wan2.2 TI2V-5B24GB graphics card for 720p video generation

MiniMax H3 local scope

This comparison includes only the local execution of the public H3-Base at 768p. Context understanding and 2K regeneration stages are excluded from local generation time.

Recommendation criteria

If the lower end of the estimated range reaches 30 tok/s on your current hardware, we show that option first. Otherwise, we find a faster smaller model for the same device.

Four choices

• Spend nothing: a model that stays above 30 tok/s on your current device
• Balanced new system: the lowest upfront price among new hardware that stays above 30 tok/s
• Large-model headroom: the new system with the most fast memory left after loading the target model
• Top performance: the fastest single-response token generation among new systems for the same model

Core specifications

Available memory

13GB

Memory bandwidth

120GB/s

Estimated sustained local LLM load

about 42W

Korean price range (KRW)

165–169 × KRW 10,000

Who it fits

• Summary, translation, light coding
• Quiet, always-on operation
• Private Local LLM

Check before purchasing

The available memory is small, making it difficult to use models larger than 20B and long contexts together.

What tasks is it suitable for?

Configuration Selection Criteria

The first bottleneck encountered

The first limitation is the context length, as we need to fit the model and KV cache together in 13GB.

Recommended purchase setup

M4 · Integrated memory 16GB · SSD 512GB or more

Criteria for spending more money

It is enough if you only use 8B~12B. If you are thinking of 27B, 24GB or more is better from the beginning.

4K context · Q4 model-specific expected performance

Single-user expected range and may vary depending on runtime, cooling, and quantization files.

Frequently asked questions

What LLM can be run on the Mac mini M4 16GB?

Compare K2 Horizon 0.9B, Gemma 4 12B, Gemma 4 26B-A4B (MoE) with a 4K input and the recommended format for each hardware platform. Required memory varies by model and context length.

What is the local LLM power of the Mac mini M4 16GB?

Expected to draw approximately 42W under sustained load, with an actual range estimated between 28–58W.

Are the displayed token speeds ground truth?

These ranges normalize collected benchmark measurements to the same conditions. Only combinations without matching measurements are estimated from nearby measured values. Results can vary by runtime, cooling, model file, and context length.

Core specifications

Available memory

20GB

Memory bandwidth

120GB/s

Estimated sustained local LLM load

about 45W

Korean price range (KRW)

150–235 × KRW 10,000

Who it fits

• Short document summaries with 12B
• Questions about a function or code excerpt
• A compact desktop unit

Check before purchasing

If your main task is feeding long documents to a 27B Q4 model, check memory headroom at that input length before choosing this configuration. macOS and open apps share the same 24GB, and you cannot add RAM after purchase.

What tasks is it suitable for?

Configuration Selection Criteria

The first bottleneck encountered

Once the model fits, adding memory does not increase generation speed in proportion. Both the M4 24GB and 32GB use 120GB/s memory bandwidth. If waiting is the problem, compare speed with a Pro under the same conditions rather than looking only at capacity.

Recommended purchase setup

M4 · 10-core GPU · 24GB unified memory · 512GB SSD recommended

Criteria for spending more money

If the answers and wait times from a 12B model suit you, you can stop here. Consider paying for 32GB only after choosing a larger model or finding that other open apps leave too little memory.

4K context · Q4 model-specific expected performance

Single-user expected range and may vary depending on runtime, cooling, and quantization files.

Frequently asked questions

What LLM can be run on the Mac mini M4 24GB?

Compare K2 Horizon 0.9B, Gemma 4 12B, K2 Horizon MoVA 36B-A4B with a 4K input and the recommended format for each hardware platform. Required memory varies by model and context length.

What is the local LLM power of the Mac mini M4 24GB?

Expected to be around 45W under sustained load, with an actual range of 30–60W.

Are the displayed token speeds ground truth?

These ranges normalize collected benchmark measurements to the same conditions. Only combinations without matching measurements are estimated from nearby measured values. Results can vary by runtime, cooling, model file, and context length.

Core specifications

Available memory

27GB

Memory bandwidth

120GB/s

Estimated sustained local LLM load

about 48W

Korean price range (KRW)

180–240 × KRW 10,000

Who it fits

• Run 27B Q4 on one machine
• A personal server without PC assembly
• When 24GB runs out of memory

Check before purchasing

Do not choose this configuration if you need to move CUDA-only extensions over unchanged. Having 32GB of memory does not make them compatible with macOS. On the product listing, also check the M4 chip, 32GB unified memory, and SSD option separately.

What tasks is it suitable for?

Configuration Selection Criteria

The first bottleneck encountered

Generation-speed differences add up more over long answers than short ones. Apart from checking whether 27B Q4 fits, try an answer length you would actually be willing to wait for. Longer context also leaves less free memory.

Recommended purchase setup

M4 · 10-core GPU · 32GB unified memory · 512GB SSD recommended

Criteria for spending more money

If you only use 12B models and have memory to spare in 24GB, you do not need to move to 32GB. If you like the answers from 27B but not the wait, compare prefill and generation times with the M4 Pro 48GB rather than buying a larger SSD.

4K context · Q4 model-specific expected performance

Single-user expected range and may vary depending on runtime, cooling, and quantization files.

Frequently asked questions

What LLM can be run on the Mac mini M4 32GB?

Compare K2 Horizon 0.9B, Qwen3.8 27B, K2 Horizon MoVA 36B-A4B with a 4K input and the recommended format for each hardware platform. Required memory varies by model and context length.

What is the local LLM power of the Mac mini M4 32GB?

Expected to be around 48W under sustained load, with an actual range of 32–63W.

Are the displayed token speeds ground truth?

These ranges normalize collected benchmark measurements to the same conditions. Only combinations without matching measurements are estimated from nearby measured values. Results can vary by runtime, cooling, model file, and context length.

Core specifications

Available memory

41GB

Memory bandwidth

273GB/s

Estimated sustained local LLM load

about 92W

Korean price range (KRW)

263–344 × KRW 10,000

Who it fits

• Repeated document and coding tasks with 27B
• Run the model alongside work apps
• A Mac setup without PC assembly

Check before purchasing

If you need to keep using CUDA-only ComfyUI nodes unchanged, look for a supported GPU setup rather than a Mac. Do not choose this machine for 70B solely because it has 48GB either. Account for the model files, input cache, and other apps together.

What tasks is it suitable for?

Configuration Selection Criteria

The first bottleneck encountered

Reading a long input and generating an answer do not speed up by the same amount. Dividing the Pro's 273GB/s bandwidth by the M4's 120GB/s does not give the speedup for the whole task. Compare the two stages with the same model and input.

Recommended purchase setup

M4 Pro · 20-core GPU · 48GB unified memory · 1TB SSD recommended

Criteria for spending more money

If you ask short questions with a 12B model and the wait is acceptable, a base M4 may be enough. Judge the price difference only after checking whether the time saved on repeated 27B use or longer inputs matters to your work.

4K context · Q4 model-specific expected performance

Single-user expected range and may vary depending on runtime, cooling, and quantization files.

Frequently asked questions

What LLM can be run on the Mac mini M4 Pro 48GB?

Compare K2 Horizon 0.9B, Qwen3.8 27B, K2 Horizon MoVA 36B-A4B with a 4K input and the recommended format for each hardware platform. Required memory varies by model and context length.

What is the local LLM computing power of the Mac mini M4 Pro 48GB?

Expected to consume approximately 92W under sustained load, with an actual range estimated between 62–122W.

Are the displayed token speeds ground truth?

These ranges normalize collected benchmark measurements to the same conditions. Only combinations without matching measurements are estimated from nearby measured values. Results can vary by runtime, cooling, model file, and context length.

Core specifications

Available memory

27GB

Memory bandwidth

170GB/s

Estimated sustained local LLM load

about 55W

Korean price range (KRW)

235–320 × KRW 10,000

Who it fits

• Run a 27B Q4 model alongside work apps
• Start with short prompts and extend context
• A compact Mac without building a PC

Check before purchasing

If you only plan to use 12B-class models with short inputs, you do not need to start with 32GB. Check whether the model and input cache fit comfortably in 24GB. More unified memory will not add compatibility for CUDA-only apps or nodes. Check both SSD capacity and the final price of the current configuration.

What tasks is it suitable for?

Configuration Selection Criteria

The first bottleneck encountered

The model cannot use all 32GB: macOS, open apps, and prompt cache share the unified memory. Apple's specifications give the M6 24GB and 32GB the same 170GB/s bandwidth, so when a workload fits on both, the capacity difference alone will not increase generation speed proportionally.

Recommended purchase setup

M6 · 12-core GPU · 32GB unified memory · 1TB or larger SSD recommended (check available configurations)

Criteria for spending more money

If you already own an M4 32GB, memory capacity alone is not a reason to replace it with an M6. Compare prompt processing and long-answer generation using the same model, quantization, and input length. For a new purchase, first check whether the M6 24GB can do the job; pay for 32GB only if a 27B model plus your other apps leave too little room. If waiting time is the main problem, the M4 Pro 48GB is also worth comparing.

4K context · Q4 model-specific expected performance

Single-user expected range and may vary depending on runtime, cooling, and quantization files.

Frequently asked questions

What LLM can be run on the Mac mini M6 32GB?

Compare K2 Horizon 0.9B, Qwen3.8 27B, K2 Horizon MoVA 36B-A4B with a 4K input and the recommended format for each hardware platform. Required memory varies by model and context length.

What is the local LLM power of the Mac mini M6 32GB?

Expected to be around 55W under sustained load, with an actual range estimated between 35–80W.

Are the displayed token speeds ground truth?

These ranges normalize collected benchmark measurements to the same conditions. Only combinations without matching measurements are estimated from nearby measured values. Results can vary by runtime, cooling, model file, and context length.

Core specifications

Available memory

114GB

Memory bandwidth

546GB/s

Estimated sustained local LLM load

about 130W

Korean price range (KRW)

780–888 × KRW 10,000

Who it fits

• 70B class Q4
• Simultaneous storage of multiple models
• Large integrated memory

Check before purchasing

Memory is ample, but tasks that require CUDA-specific tools require separate confirmation.

What tasks is it suitable for?

Configuration Selection Criteria

The first bottleneck encountered

It is strong for loading large models, but has fewer options for tasks that require a CUDA-specific kernel and multi-GPU scalability.

Recommended purchase setup

M4 Max · Integrated memory 128GB · SSD 1TB or more

Criteria for spending more money

It is worth considering if you want to put a 70B class model in one vehicle. If you only use 35B or less, M4 Pro 48GB is economical.

4K context · Q4 model-specific expected performance

Single-user expected range and may vary depending on runtime, cooling, and quantization files.

Frequently asked questions

Which local LLM can be run on the Studio M4 Max 128GB?

Compare K2 Horizon 0.9B, Qwen3.8-Flash-Next with a 4K input and the recommended format for each hardware platform. Required memory varies by model and context length.

What is the local LLM computing power of the Studio M4 Max 128GB?

Expected to draw approximately 130W under sustained load, with an actual range estimated between 95–165W.

Are the displayed token speeds ground truth?

These ranges normalize collected benchmark measurements to the same conditions. Only combinations without matching measurements are estimated from nearby measured values. Results can vary by runtime, cooling, model file, and context length.

Core specifications

Available memory

22.5GB

Memory bandwidth

936GB/s

Estimated sustained local LLM load

about 420W

Korean price range (KRW)

239–239 × KRW 10,000

Who it fits

• Add 24GB VRAM to an existing PC
• Q4 workloads that fit in 24GB
• ComfyUI image generation

Check before purchasing

Avoid a used card if you cannot check its condition under load or the return terms if something goes wrong. Seeing it display an image once is not enough. Before buying, also check whether that manufacturer's model fits your PC in length, thickness, and power connections.

What tasks is it suitable for?

Configuration Selection Criteria

The first bottleneck encountered

A 70B model's raw 4-bit weights alone are roughly 35GB before metadata and context cache. Do not compare an offloaded run as though it were fully resident on one 24GB card. RAM offloading may enable execution without preserving fully resident speed.

Recommended purchase setup

RTX 3090 24GB card only · 64GB system RAM in this comparison · Check PSU, case fit, and warranty for the manufacturer's model

Criteria for spending more money

A 24GB RTX 4090 may improve speed but does not automatically solve a capacity problem. Before moving to 32GB, check that the target workload fits there. Tasks that fit 16GB can also be compared with smaller new configurations, including power, warranty and full-system cost.

4K context · Q4 model-specific expected performance

Single-user expected range and may vary depending on runtime, cooling, and quantization files.

Frequently asked questions

What LLM can be run on a local LLM using an RTX 3090 24GB?

Compare K2 Horizon 0.9B, Qwen3.8 27B, Qwen3.8-Flash-Next with a 4K input and the recommended format for each hardware platform. Required memory varies by model and context length.

What is the power consumption of a local LLM on an RTX 3090 24GB?

Expected to be around 420W under sustained load, with an actual range of 330–540W.

Are the displayed token speeds ground truth?

These ranges normalize collected benchmark measurements to the same conditions. Only combinations without matching measurements are estimated from nearby measured values. Results can vary by runtime, cooling, model file, and context length.

Core specifications

Available memory

22.5GB

Memory bandwidth

1008GB/s

Estimated sustained local LLM load

about 470W

Korean price range (KRW)

325–400 × KRW 10,000

Who it fits

• Fast 12B to 27B inference
• ComfyUI image generation
• CUDA development environment

Check before purchasing

Models with more than 24GB cannot be loaded even if they are fast, and their appeal diminishes if used prices are close to the 5090.

What tasks is it suitable for?

Configuration Selection Criteria

The first bottleneck encountered

The 24GB VRAM limit is reached before compute performance.

Recommended purchase setup

RTX 4090 24GB · System RAM 64GB · 1000W power · 4 slots available

Criteria for spending more money

It's worth it when the price difference over the 5090 is big enough and 24GB is enough. If you need 32GB, look no further than the 5090.

4K context · Q4 model-specific expected performance

Single-user expected range and may vary depending on runtime, cooling, and quantization files.

Frequently asked questions

What LLM can be run on an RTX 4090 24GB?

Compare K2 Horizon 0.9B, Qwen3.8 27B, Qwen3.8-Flash-Next with a 4K input and the recommended format for each hardware platform. Required memory varies by model and context length.

What is the power consumption of a local LLM on an RTX 4090 with 24GB?

Expected to consume approximately 470W under sustained load, with an actual range estimated between 350–600W.

Are the displayed token speeds ground truth?

These ranges normalize collected benchmark measurements to the same conditions. Only combinations without matching measurements are estimated from nearby measured values. Results can vary by runtime, cooling, model file, and context length.

Core specifications

Available memory

30.5GB

Memory bandwidth

1792GB/s

Estimated sustained local LLM load

about 620W

Korean price range (KRW)

899–929 × KRW 10,000

Who it fits

• Repeated 27B–35B Q4 runs
• Image and video tasks that fit in 32GB
• Tools with confirmed Blackwell support

Check before purchasing

Do not replace the card before checking Blackwell support in the tools you use. The driver, PyTorch, and runtime versions need to work together. Length, thickness, and power requirements also vary by manufacturer's model, so check your existing case and PSU separately.

What tasks is it suitable for?

Configuration Selection Criteria

The first bottleneck encountered

Moving a model and cache that exceed 32GB into RAM changes the conditions of the task. A video model's name alone is not enough either. Compare at the same resolution, frame count and offloading settings.

Recommended purchase setup

RTX 5090 32GB card only · 64GB system RAM in this comparison · check the board vendor's PSU, connector and clearance requirements

Criteria for spending more money

If your current 24GB card finishes the work you need and the wait is acceptable, there is no rush to replace it. Once the time saved on repeated tasks and the need for 32GB are clear, include the cost of other PC parts before deciding.

4K context · Q4 model-specific expected performance

Single-user expected range and may vary depending on runtime, cooling, and quantization files.

Frequently asked questions

What LLM can be run on an RTX 5090 with 32GB?

Compare K2 Horizon 0.9B, Qwen3.6 35B-A3B (MoE), Qwen3.8-Flash-Next with a 4K input and the recommended format for each hardware platform. Required memory varies by model and context length.

What is the power consumption of a local LLM on an RTX 5090 with 32GB?

Expected to be around 620W under sustained load, with an actual range of 480–760W.

Are the displayed token speeds ground truth?

These ranges normalize collected benchmark measurements to the same conditions. Only combinations without matching measurements are estimated from nearby measured values. Results can vary by runtime, cooling, model file, and context length.

Core specifications

Available memory

96GB

Memory bandwidth

256GB/s

Estimated sustained local LLM load

about 125W

Korean price range (KRW)

538–562 × KRW 10,000

Who it fits

• Loading large models
• Select Windows/Linux
• 128GB cost-effectiveness

Check before purchasing

Even if there is ample memory, the actual speed difference is large depending on the runtime and GPU acceleration support, so you should check compatibility for each model first.

What tasks is it suitable for?

Configuration Selection Criteria

The first bottleneck encountered

GPU acceleration support by runtime and kernel completeness determine actual usability rather than hardware capacity.

Recommended purchase setup

Ryzen AI Max+ 395 · System memory 128GB · NVMe 2TB recommended

Criteria for spending more money

It is advantageous if you want to put up with the settings and install a large capacity model cheaply. If you want reliable tools out of the box, you're better off with a Mac or NVIDIA finished product.

4K context · Q4 model-specific expected performance

Single-user expected range and may vary depending on runtime, cooling, and quantization files.

Frequently asked questions

Which local LLM can be run on Ryzen AI Max+ 395 128GB?

Compare K2 Horizon 0.9B, Qwen3.8-Flash-Next with a 4K input and the recommended format for each hardware platform. Required memory varies by model and context length.

How much power does the local LLM of Ryzen AI Max+ 395 with 128GB consume?

Expected power consumption under sustained load is approximately 125W, with an actual range anticipated between 80–170W.

Are the displayed token speeds ground truth?

These ranges normalize collected benchmark measurements to the same conditions. Only combinations without matching measurements are estimated from nearby measured values. Results can vary by runtime, cooling, model file, and context length.

Core specifications

Available memory

236GB

Memory bandwidth

1200GB/s

Estimated sustained local LLM load

about 300W

Korean price range (KRW)

1,850–2,000 × KRW 10,000

Who it fits

• Local agents with long context
• Always-on Flash-Next at 4 or 5 bit
• Multiple parallel requests on one machine

Check before purchasing

An RTX 5090 can still be faster for short requests when the whole model fits in 32GB. CUDA-only tools do not carry over unchanged. In the tested 256GB system, 6- and 8-bit Flash-Next needed their PLE embedding tables on SSD, so wait for the 512GB configuration if the goal is to keep those higher-precision builds entirely in memory.

What tasks is it suitable for?

Configuration Selection Criteria

The first bottleneck encountered

Single-request generation is largely constrained by memory bandwidth, while first-token latency depends heavily on GPU compute and prompt length. Three concurrent requests raise aggregate throughput, but each request becomes slower.

Recommended purchase setup

M5 Ultra · 36-core CPU · 80-core GPU · 256GB unified memory · 2TB or larger SSD recommended for model storage

Criteria for spending more money

Published runs used about 155GB for Flash-Next oQ4e and about 179GB for oQ5e, so both can operate within 256GB. Unless you plan to keep oQ6e or oQ8e entirely in RAM, or keep several large models resident at once, check the 256GB system's actual latency and concurrent throughput before paying for 512GB.

4K context · Q4 model-specific expected performance

Single-user expected range and may vary depending on runtime, cooling, and quantization files.

Frequently asked questions

Which local LLM can be run on the Studio M5 Ultra 256GB?

Compare K2 Horizon 0.9B, Qwen3.8-Flash-Next, GLM-5.3-Flash (MoE) with a 4K input and the recommended format for each hardware platform. Required memory varies by model and context length.

What is the local LLM power of the Studio M5 Ultra 256GB?

Expected to consume approximately 300W under sustained load, with an actual range estimated between 220–430W.

Are the displayed token speeds ground truth?

These ranges normalize collected benchmark measurements to the same conditions. Only combinations without matching measurements are estimated from nearby measured values. Results can vary by runtime, cooling, model file, and context length.

Core specifications

Available memory

480GB

Memory bandwidth

1200GB/s

Estimated sustained local LLM load

about 330W

Korean price range (KRW)

3,805–3,890 × KRW 10,000

Who it fits

• Model files and prompt caches that will not fit in 256GB
• Long contexts and multiple simultaneous workloads
• Large unified-memory capacity in one Mac

Check before purchasing

512GB costs ₩19.55 million more than 256GB. If the same model already fits comfortably in 256GB, that money does not make generation faster. If your budget is fixed, start from 256GB or 128GB. CUDA-only tools and extensions stay out of reach no matter how much unified memory you add.

What tasks is it suitable for?

Configuration Selection Criteria

The first bottleneck encountered

512GB is the total of the unified memory. macOS, open apps, model weights and the KV cache all share this space. The 480GB shown here is headroom used for the calculation, not a figure Apple guarantees. If the same model fits comfortably in 256GB, 512GB does not raise generation speed in proportion.

Recommended purchase setup

M5 Ultra · 36-core CPU · 80-core GPU · 512GB unified memory · 1TB SSD ₩38,050,000 (2TB ₩38,900,000)

Criteria for spending more money

Published runs used about 155GB for Flash-Next oQ4e and about 179GB for oQ5e, so both work on 256GB. Choosing 512GB becomes clear mainly when you need 6- or 8-bit without SSD offload, or several large models and long caches resident at once. With a 1TB SSD, 512GB is ₩38.05 million and 256GB is ₩18.5 million, so judge whether ₩19.55 million is worth that workload.

4K context · Q4 model-specific expected performance

Single-user expected range and may vary depending on runtime, cooling, and quantization files.

Frequently asked questions

Is the M5 Ultra 512GB twice as fast as the 256GB configuration?

No. With the same 36-core CPU, 80-core GPU, and 1.2TB/s memory bandwidth, the main benefit of 512GB is capacity. The practical difference can become substantial only when a workload runs out of memory or has to offload on the 256GB configuration.

Which local LLM can be run on the Studio M5 Ultra 512GB?

Compare K2 Horizon 0.9B, Qwen3.8-Flash-Next, GLM-5.3-Flash (MoE) with a 4K input and the recommended format for each hardware platform. Required memory varies by model and context length.

What is the local LLM power of the Studio M5 Ultra 512GB?

Expected to consume approximately 330W under sustained load, with an actual range estimated between 240–460W.

Are the displayed token speeds ground truth?

These ranges normalize collected benchmark measurements to the same conditions. Only combinations without matching measurements are estimated from nearby measured values. Results can vary by runtime, cooling, model file, and context length.

Core specifications

Available memory

480GB

Memory bandwidth

819GB/s

Estimated sustained local LLM load

about 230W

Korean price range (KRW)

2,400–2,600 × KRW 10,000

Who it fits

• Model and cache combinations that do not fit in 256GB
• Keep several large models loaded on one computer
• Large-memory workloads using macOS and MLX

Check before purchasing

If your serving tools require CUDA, 512GB of unified memory will not solve the compatibility problem. A listed price for an Apple Certified Refurbished 512GB unit does not mean it is in stock. Check availability, SSD capacity, warranty, and the 80-core GPU configuration before ordering.

What tasks is it suitable for?

Configuration Selection Criteria

The first bottleneck encountered

The 512GB is total unified memory shared by macOS, the model, and the prompt cache. This site's 480GB usable-memory figure is a planning allowance, not GPU-only memory guaranteed by Apple. If a model already fits comfortably on a smaller configuration, adding capacity alone will not increase generation speed proportionally.

Recommended purchase setup

M3 Ultra · 32-core CPU · 80-core GPU · 512GB unified memory · 2TB or larger SSD · Verify current stock

Criteria for spending more money

If one Qwen3.8-Flash-Next Q4 model is your goal, first check memory headroom and waiting time on a 128GB or 256GB machine. Paying for 512GB makes sense only when the actual model files and cache will not fit in a 256GB configuration. A displayed price for an out-of-stock refurbished M3 Ultra is not a price you can buy at. A new M5 Ultra 512GB is ₩38.05 million with a 1TB SSD. Compare that total with the refurbished M3 Ultra 512GB listing at the same storage capacity.

4K context · Q4 model-specific expected performance

Single-user expected range and may vary depending on runtime, cooling, and quantization files.

Frequently asked questions

Which local LLM can be run on the Studio M3 Ultra 512GB?

Compare K2 Horizon 0.9B, Qwen3.8-Flash-Next, DeepSeek-V4.1-Flash with a 4K input and the recommended format for each hardware platform. Required memory varies by model and context length.

What is the local LLM computing power of the Studio M3 Ultra 512GB?

Expected to be around 230W under sustained load, with an actual range estimated between 180–270W.

Are the displayed token speeds ground truth?

These ranges normalize collected benchmark measurements to the same conditions. Only combinations without matching measurements are estimated from nearby measured values. Results can vary by runtime, cooling, model file, and context length.

Core specifications

Available memory

116GB

Memory bandwidth

273GB/s

Estimated sustained local LLM load

about 190W

Korean price range (KRW)

952–1,030 × KRW 10,000

Who it fits

• Load models that need more than 32GB
• CUDA and ARM64 serving development
• Run on one machine without multiple GPUs

Check before purchasing

Do not buy a Spark first if the tools you need are x86-only with no ARM64 execution path. Check the containers and extensions you will use on DGX OS. Connecting two units with a cable does not simply pool their memory; you need a setup that splits the model across them.

What tasks is it suitable for?

Configuration Selection Criteria

The first bottleneck encountered

128GB is the total memory shared by the CPU and GPU. The OS and input cache also need space, and more capacity does not give it the same model-reading bandwidth as GeForce. Compare single-request speed separately from the number of requests handled at once.

Recommended purchase setup

GB10 · 128GB unified memory · One complete system · Check SSD capacity and warranty for the listed configuration

Criteria for spending more money

If you only want one 27B chat session, try your existing PC or a single GPU first. If you need plenty of memory but not necessarily CUDA, the Ryzen AI Max+ 395 128GB is also worth comparing. Consider two Sparks after identifying the capacity or throughput that one unit cannot provide.

4K context · Q4 model-specific expected performance

Single-user expected range and may vary depending on runtime, cooling, and quantization files.

Frequently asked questions

Which local LLM can be run on DGX Spark 128GB?

Compare K2 Horizon 0.9B, Qwen3.8-Flash-Next, DeepSeek-V4.1-Flash with a 4K input and the recommended format for each hardware platform. Required memory varies by model and context length.

How much local LLM computing power does DGX Spark 128GB have?

Expected to be around 190W under sustained load, with an actual range of 160–233W.

Are the displayed token speeds ground truth?

These ranges normalize collected benchmark measurements to the same conditions. Only combinations without matching measurements are estimated from nearby measured values. Results can vary by runtime, cooling, model file, and context length.

Model specifications

Estimated TERNARY_PQ2 memory

About 9GB

total parameters

27.36B

active parameter

27.36B

native context

262,144 tokens

A ternary-weight model derived from Qwen3.8-27B. The PQ2_0 language weights are 7.21 GB and require a Hadamard-aware runtime rather than stock llama.cpp. The speed experience prioritizes PrismML's published PQ2_0 measurements.

Main equipment

• RTX 5060 8GB — available 7.25GB, 448GB/s · Feel the speed · Price, power, and sales composition
• 2× RTX 5090 64GB (PCIe) — available 61GB, 3584GB/s Feel the speed · Price, power, and sales composition
• MBA M4 16GB — available 13GB, 120GB/s · Feel the speed · Price, power, and sales composition

Model specifications

Q2_K model file

264.5GB · memory-mapped paging

total parameters

552B

active parameter

16B

native context

1,048,576 tokens

An MoE model with a 552B backbone and separate Engram tables. Published runs on Spark, RTX 5090, and high-memory Macs are separated by hardware and format, including experimental paths that require SSD streaming and dedicated runtimes.

Main equipment

• DGX Spark 128GB — available 116GB, 273GB/s · Feel the speed · Price, power, and sales composition
• Studio M3 Ultra 512GB — available 480GB, 819GB/s · Feel the speed · Price, power, and sales composition

Model specifications

Estimated MXFP4 memory

About 71GB

total parameters

124.848460496B

active parameter

5.1B

native context

131,072 Token

An image/video-input MoE model with about 124.85B total and 5.1B active parameters. Native context is 128K. Only official DGX Spark INT4/MXFP4 measurements anchor the speed experience; rates are not transferred to other hardware.

Main equipment

• DGX Spark 128GB — available 116GB, 273GB/s · Feel the speed · Price, power, and sales composition

Model specifications

Estimated Q4_K_M memory

About 3GB

total parameters

2.51675648B

active parameter

2.51675648B

native context

131,072 Token

A dense open-weight model with 2.51675648B parameters and a native 131,072-token context. Displayed memory and generation speeds are estimates based on quantization and hardware specifications, not measured results.

Main equipment

• RTX 5060 8GB — available 7.25GB, 448GB/s · Feel the speed · Price, power, and sales composition
• 2× RTX 5090 64GB (PCIe) — available 61GB, 3584GB/s Feel the speed · Price, power, and sales composition
• MBA M4 16GB — available 13GB, 120GB/s · Feel the speed · Price, power, and sales composition

Speed tuning for each device

When selecting equipment, you can view quantization, safe context, GPU offloading, Flash Attention, KV cache, and batch start values together.

A safe local starting point for MiniCPM5-2B Q4_K_M

Verify prefill and token generation speed of this model

Model specifications

Estimated Q4_K_M memory

about 18GB

total parameters

27B

active parameter

27B

native context

262,144 tokens

27B dense open weight model with built-in multi-stage multi-token prediction (MTP) learning head. Supports native 262,144 (256K) token contexts.

Main equipment

• RTX 5060 8GB — available 7.25GB, 448GB/s · Feel the speed · Price, power, and sales composition
• 2× RTX 5090 64GB (PCIe) — available 61GB, 3584GB/s Feel the speed · Price, power, and sales composition
• MBA M4 24GB — available 20GB, 120GB/s · Feel the speed · Price, power, and sales composition

Speed tuning for each device

When selecting equipment, you can view quantization, safe context, GPU offloading, Flash Attention, KV cache, and batch start values together.

Qwen3.8-27B setup by hardware: Mac oMLX and RTX llama.cpp

Verify prefill and token generation speed of this model

Model specifications

Estimated Q4_K_M memory

about 108GB

total parameters

125B

active parameter

6B

native context

262,144 tokens

Approximately 180B resident weight MoE model consisting of 125B core LMs (6B active) with 51B n-gram/PLE embeddings and 4B MTPs built into the architecture. The built-in 51B PLE embedding table is RAM offloadable and is independent of the Prompt Lookup speculative decoding that looks for repeated prompt phrases.

Main equipment

• MBP M3 Max 128GB — available 114GB, 400GB/s · Feel the speed · Price, power, and sales composition
• RTX PRO 6000 96GB — available 93GB, 1792GB/s Feel the speed · Price, power, and sales composition

Speed tuning for each device

When selecting equipment, you can view quantization, safe context, GPU offloading, Flash Attention, KV cache, and batch start values together.

Qwen3.8 Flash-Next on DGX Spark: SGLang, PLE, and MTP recipe

Verify prefill and token generation speed of this model

Model specifications

Estimated Q4_K_M memory

about 8GB

total parameters

11.95B

active parameter

11.95B

native context

262,144 tokens

A single 11.95B parameter dense model supports 256K (262,144) native contexts. High-speed standalone operation is possible in 16GB~24GB GPU and Mac environments.

Main equipment

• RTX 5060 8GB — available 7.25GB, 448GB/s · Feel the speed · Price, power, and sales composition
• 2× RTX 5090 64GB (PCIe) — available 61GB, 3584GB/s Feel the speed · Price, power, and sales composition
• MBA M4 16GB — available 13GB, 120GB/s · Feel the speed · Price, power, and sales composition

Speed tuning for each device

When selecting equipment, you can view quantization, safe context, GPU offloading, Flash Attention, KV cache, and batch start values together.

Gemma 4 12B serving: Q4 on desktop and BF16 on a server

Verify prefill and token generation speed of this model

Model specifications

Estimated Q4_K_M memory

about 16GB

total parameters

25.2B

active parameter

3.8B

native context

262,144 tokens

This is an MoE model in which 3.8B of the total 25.2B parameters are activated per token. Supports 256K (262,144) native contexts and provides fast decoding per load.

Main equipment

• RTX 5060 8GB — available 7.25GB, 448GB/s · Feel the speed · Price, power, and sales composition
• 2× RTX 5090 64GB (PCIe) — available 61GB, 3584GB/s Feel the speed · Price, power, and sales composition
• MBA M4 16GB — available 13GB, 120GB/s · Feel the speed · Price, power, and sales composition

Speed tuning for each device

When selecting equipment, you can view quantization, safe context, GPU offloading, Flash Attention, KV cache, and batch start values together.

Gemma 4 26B-A4B MTP: How Speed Changes Between One and Many Requests

Verify prefill and token generation speed of this model

Model specifications

Estimated Q4_K_M memory

about 22GB

total parameters

35B

active parameter

3B

native context

262,144 tokens

This is an MoE model in which approximately 3B of the total 35B parameters are activated per token. Memory loading requires 35B capacity, and decoding bandwidth is consumed based on 3B active parameters.

Main equipment

• RTX 5060 8GB — available 7.25GB, 448GB/s · Feel the speed · Price, power, and sales composition
• 2× RTX 5090 64GB (PCIe) — available 61GB, 3584GB/s Feel the speed · Price, power, and sales composition
• MBA M4 24GB — available 20GB, 120GB/s · Feel the speed · Price, power, and sales composition

Speed tuning for each device

When selecting equipment, you can view quantization, safe context, GPU offloading, Flash Attention, KV cache, and batch start values together.

Qwen3.6 35B-A3B setup: NVFP4 on Mac, RTX, and DGX Spark

Verify prefill and token generation speed of this model

Model specifications

Estimated Q4_K_M memory

about 191GB

total parameters

320B

active parameter

18B

native context

1,048,576 tokens

Hybrid sparse+linear attention MoE model with a total of 320B parameters (18B active). Supports the official 1,048,576 (1M) token context (text_config.max_position_embeddings) and loads in large memory environments of 256GB or more.

Main equipment

• 2× A40 96GB (NVLink) — available 92GB, 1392GB/s · Feel the speed · Price, power, and sales composition
• Studio M3 Ultra 512GB — available 480GB, 819GB/s · Feel the speed · Price, power, and sales composition

Speed tuning for each device

When selecting equipment, you can view quantization, safe context, GPU offloading, Flash Attention, KV cache, and batch start values together.

GLM-5.3-Flash Serving Guide: A 320B Server-Scale MoE Recipe

Verify prefill and token generation speed of this model

Model specifications

Estimated Q4_K_M memory

about 181GB

total parameters

304B

active parameter

13B

native context

1,048,576 tokens

Large MoE model with a total of 304B parameters (13B active) supporting 1,048,576 (1M) token contexts (max_position_embeddings). The official example uses 4×GB300 nodes, and the community compressed version must be checked separately for actual size and runtime compatibility for each artifact.

Main equipment

• 2× A40 96GB (NVLink) — available 92GB, 1392GB/s · Feel the speed · Price, power, and sales composition
• Studio M5 Ultra 256GB — available 236GB, 1200GB/s Feel the speed · Price, power, and sales composition
• Studio M3 Ultra 512GB — available 480GB, 819GB/s · Feel the speed · Price, power, and sales composition

Speed tuning for each device

When selecting equipment, you can view quantization, safe context, GPU offloading, Flash Attention, KV cache, and batch start values together.

DeepSeek-V4-Flash: distributed serving for a 304B model

Verify prefill and token generation speed of this model

Model specifications

Estimated Q4_K_M memory

about 18GB

total parameters

27B

active parameter

27B

native context

262,144 tokens

It is a 27B dense computer usage model that handles web, desktop, and mobile screen manipulation as well as code and tool calls. Automation is not complete with models alone; you need an executor with screen capture, input tools, and acceptance boundaries.

Main equipment

• RTX 5060 8GB — available 7.25GB, 448GB/s · Feel the speed · Price, power, and sales composition
• 2× RTX 5090 64GB (PCIe) — available 61GB, 3584GB/s Feel the speed · Price, power, and sales composition
• MBA M4 24GB — available 20GB, 120GB/s · Feel the speed · Price, power, and sales composition

Model specifications

Estimated Q4_K_M memory

about 22GB

total parameters

35B

active parameter

3B

native context

262,144 tokens

A computer-enabled MoE model that activates approximately 3B per token out of a total of 35B. The activation parameter describes the amount of computation, and local loading requires full weights and vision/KV cache headroom.

Main equipment

• RTX 5060 8GB — available 7.25GB, 448GB/s · Feel the speed · Price, power, and sales composition
• 2× RTX 5090 64GB (PCIe) — available 61GB, 3584GB/s Feel the speed · Price, power, and sales composition
• MBA M4 24GB — available 20GB, 120GB/s · Feel the speed · Price, power, and sales composition

Model specifications

Estimated Q4_K_M memory

About 3GB

total parameters

3B

active parameter

3B

native context

32,768 tokens

It is a roughly 3B vision language model designed to quickly process single, short requests such as OCR, document layout, and image description. Using a separate DSpark draft model speeds up decode on supported devices.

Main equipment

• RTX 5060 8GB — available 7.25GB, 448GB/s · Feel the speed · Price, power, and sales composition
• 2× RTX 5090 64GB (PCIe) — available 61GB, 3584GB/s Feel the speed · Price, power, and sales composition
• MBA M4 16GB — available 13GB, 120GB/s · Feel the speed · Price, power, and sales composition

Model specifications

Estimated Q4_K_M memory

About 2GB

total parameters

0.9B

active parameter

0.9B

native context

131,072 Token

This is the smallest compact model in the K2 Horizon family. It is a first candidate for lightweight classification/summarization/local automation, and a distinction must be made between long context limits and the context length to be maintained on actual devices.

Main equipment

• RTX 5060 8GB — available 7.25GB, 448GB/s · Feel the speed · Price, power, and sales composition
• 2× RTX 5090 64GB (PCIe) — available 61GB, 3584GB/s Feel the speed · Price, power, and sales composition
• MBA M4 16GB — available 13GB, 120GB/s · Feel the speed · Price, power, and sales composition

Model specifications

Estimated Q4_K_M memory

About 3GB

total parameters

3.7B

active parameter

3.7B

native context

524,288 Token

A compact dense model that provides 512K native contexts. It's easy to start with small memories, but longer inputs require larger KV cache and prefill times, so you'll need to start with shorter baselines and increase them.

Main equipment

• RTX 5060 8GB — available 7.25GB, 448GB/s · Feel the speed · Price, power, and sales composition
• 2× RTX 5090 64GB (PCIe) — available 61GB, 3584GB/s Feel the speed · Price, power, and sales composition
• MBA M4 16GB — available 13GB, 120GB/s · Feel the speed · Price, power, and sales composition

Model specifications

Estimated Q4_K_M memory

About 6GB

total parameters

7B

active parameter

7B

native context

524,288 Token

A 7B dense model that sits between quality and local execution burden. We support the official 512K contexts, but on desktops it is better for responsiveness and memory management to limit this to the length required for the task.

Main equipment

• RTX 5060 8GB — available 7.25GB, 448GB/s · Feel the speed · Price, power, and sales composition
• 2× RTX 5090 64GB (PCIe) — available 61GB, 3584GB/s Feel the speed · Price, power, and sales composition
• MBA M4 16GB — available 13GB, 120GB/s · Feel the speed · Price, power, and sales composition

Model specifications

Estimated Q4_K_M memory

About 21GB

total parameters

32B

active parameter

32B

native context

524,288 Token

This is a 32B dense model of K2 Horizon. On 24GB devices, check Q4 and short context first, and if you want to use long inputs and simultaneous requests, you should also check the available memory of 32GB or more.

Main equipment

• RTX 5060 8GB — available 7.25GB, 448GB/s · Feel the speed · Price, power, and sales composition
• 2× RTX 5090 64GB (PCIe) — available 61GB, 3584GB/s Feel the speed · Price, power, and sales composition
• MBA M4 24GB — available 20GB, 120GB/s · Feel the speed · Price, power, and sales composition

Model specifications

Estimated Q4_K_M memory

about 22GB

total parameters

36B

active parameter

4B

native context

524,288 Token

This is a MoE model with a MoVA structure that activates approximately 4B per token out of a total of 36B. The decode computation is lightweight, but the full weights must be accessible in memory, and the official server example is the multi-GPU route.

Main equipment

• RTX 5060 8GB — available 7.25GB, 448GB/s · Feel the speed · Price, power, and sales composition
• 2× RTX 5090 64GB (PCIe) — available 61GB, 3584GB/s Feel the speed · Price, power, and sales composition
• MBA M4 24GB — available 20GB, 120GB/s · Feel the speed · Price, power, and sales composition

core difference

Which device suits the task?

Local pricing

The Mac mini M4 Pro 64GB is a complete system, while the RTX 5090 32GB is a GPU-only unit. The GPU-only price should be compared after adding the base system cost.

Each device detail

• Mac mini M4 Pro 64GB local LLM speed, memory, electricity bill
• RTX 5090 32GB local LLM/image/video generation performance

core difference

Which device suits the task?

Local pricing

Mac mini M4 32GB is about ₩940,000 cheaper at the midpoint. Check perceived speed and the task-by-task verdicts together.

Each device detail

• Mac Mini M4 32GB Local LLM 27B Running Performance
• Mac Mini M4 Pro 48GB Local LLM 27B·35B Performance

core difference

Which device suits the task?

Local pricing

RTX 3090 24GB is about ₩6,750,000 cheaper at the midpoint. Check perceived speed and the task-by-task verdicts together.

Each device detail

• RTX 3090 for local AI: keep it or buy used?
• RTX 5090 32GB local LLM/image/video generation performance

core difference

Which device suits the task?

Local pricing

2× RTX 3090 48GB (NVLink) consists of 2 GPUs, while RTX 5090 32GB is a single GPU price. When comparing single GPU prices, the base cost of the GPU itself must be added.

Each device detail

• 2× NVIDIA GeForce RTX 3090 NVLink 48GB local LLM speed·memory·electricity cost
• RTX 5090 32GB local LLM/image/video generation performance

core difference

Which device suits the task?

Local pricing

RTX 4090 24GB is about ₩5,510,000 cheaper at the midpoint. Check perceived speed and the task-by-task verdicts together.

Each device detail

• RTX 4090 24GB local LLM and image creation performance
• RTX 5090 32GB local LLM/image/video generation performance

core difference

Which device suits the task?

Local pricing

Studio M4 Max 128GB is about ₩1,570,000 cheaper at the midpoint. Check perceived speed and the task-by-task verdicts together.

Each device detail

• NVIDIA DGX Spark 128GB Local LLM Selection Criteria
• Mac Studio M4 Max 128GB local LLM large model performance

core difference

Which device suits the task?

Local pricing

DGX Spark 128GB is about ₩9,910,000 cheaper at the midpoint. Check perceived speed and the task-by-task verdicts together.

Each device detail

• NVIDIA DGX Spark 128GB Local LLM Selection Criteria
• 2× NVIDIA DGX Spark cluster 256GB local LLM speed·memory·electricity cost

core difference

Which device suits the task?

Local pricing

2× DGX Spark 256GB is a complete system, 2× RTX 5090 64GB (PCIe) is the price for 2 GPU units. When comparing GPU units, the base unit cost must be added.

Each device detail

• 2× NVIDIA DGX Spark cluster 256GB local LLM speed·memory·electricity cost
• 2× NVIDIA GeForce RTX 5090 PCIe 64GB local LLM speed·memory·electricity cost

core difference

Which device suits the task?

Local pricing

Ryzen AI Max+ 395 128GB is about ₩2,840,000 cheaper at the midpoint. Check perceived speed and the task-by-task verdicts together.

Each device detail

• Ryzen AI Max+ 395 128GB local LLM cost-effectiveness
• Mac Studio M4 Max 128GB local LLM large model performance

core difference

Which device suits the task?

Local pricing

NVIDIA A40 48GB is a single GPU unit, while 2× RTX 3090 48GB (with NVLink) consists of two GPUs at a single price. When comparing, the cost of the GPU unit must include the base unit cost.

Each device detail

• NVIDIA A40 48GB local LLM speed, memory, electricity rate
• 2× NVIDIA GeForce RTX 3090 NVLink 48GB local LLM speed·memory·electricity cost

core difference

Which device suits the task?

Local pricing

DGX Spark 128GB is the price of finished product, RTX 5090 32GB is the price of GPU separately. For individual GPUs, you need to add the cost of the main body before comparing.

Each device detail

• NVIDIA DGX Spark 128GB Local LLM Selection Criteria
• RTX 5090 32GB local LLM/image/video generation performance

core difference

Which device suits the task?

Local pricing

RTX 5090 32GB is about ₩16,850,000 cheaper at the midpoint. Check perceived speed and the task-by-task verdicts together.

Each device detail

• NVIDIA RTX PRO 6000 Blackwell 96GB local LLM speed/memory/electricity rate
• RTX 5090 32GB local LLM/image/video generation performance

Start with the workload that needs 96GB

The buying decision differs between a model such as Qwen3.8-27B with compact quantized files and models with weights spanning many tens of GB. The PRO 6000 has a clear purpose when your chosen file and KV cache exceed one 5090. Check capacity across models, then compare first response and generation for a model that fits both cards.

Match the fast runtime to its model file

Both cards are based on Blackwell, but the path depends on the runtime and model format. Check the configuration actually selected by the comparison, including Strata, NVFP4 or MTP, and prepare its matching file. Follow the runtime's guide rather than putting a GGUF file with the same model name into a different engine.

Evaluate the premium against larger models

Before paying the PRO 6000 premium, list what it would make possible: reading a long report in one request, keeping a larger model resident or serving several requests. If a 5090 already delivers the answers you need, compare the effect of context and runtime tuning before replacing the card.

core difference

Which device suits the task?

Local pricing

RTX 5080 16GB is about ₩6,790,000 cheaper at the midpoint. Check perceived speed and the task-by-task verdicts together.

Each device detail

• NVIDIA GeForce RTX 5080 16GB local LLM speed, memory and powers
• RTX 5090 32GB local LLM/image/video generation performance

Exceeding 16GB changes the execution path

If the 5080 runs out of memory, options include a smaller quantization or moving part of the model to system RAM. File choice, CPU and RAM then matter together. Check whether the same file fits in the 5090's VRAM to judge how much its price premium reduces offloading.

Compare small and large models separately

Start with a model that runs on both cards and compare short-answer generation. Switch to the larger model you actually want and inspect placement and first response. Finally increase context to check conversation headroom after loading. This separates a speed upgrade from a capacity upgrade.

Include costs outside the graphics card

Even a card-only upgrade requires checking PSU capacity, cables and case space. Include a planned RAM upgrade too. Distinguish card prices below from the cost of a finished PC, and compare sustained power if the machine will stay on for long periods.

core difference

Which device suits the task?

Local pricing

RTX 5070 Ti 16GB is about ₩530,000 cheaper at the midpoint. Check perceived speed and the task-by-task verdicts together.

Each device detail

• NVIDIA GeForce RTX 5070 Ti 16GB local LLM speed, memory and powers
• NVIDIA GeForce RTX 5080 16GB local LLM speed, memory and powers

Choose the model you will use within 16GB

If a model and its KV cache exceed the 5070 Ti's capacity, the 5080 keeps the same total VRAM. You still need a smaller quantization or shorter context. If the model fits comfortably, compare prefill and generation next. Choosing the model first keeps a higher product tier from becoming the only reason to spend more.

Separate first response from completion

For document summaries, use full mode with a long input. For a short prompt that generates long code, decode mode is a more direct comparison. Measure the wait saved by the 5080 at your workload length, then consider the current price ranges.

Tune an existing 5070 Ti before replacing it

If you own a 5070 Ti, first match the recommended engine and quantized file shown here. Keeping an old setup can miss improvements from newer runtimes on the same GPU. If repeated waits remain disruptive after tuning, compare a 5080 or a card with more memory.

core difference

Which device suits the task?

Local pricing

RTX 5070 12GB is about ₩600,000 cheaper at the midpoint. Check perceived speed and the task-by-task verdicts together.

Each device detail

• NVIDIA GeForce RTX 5070 12GB local LLM speed, memory and powers
• NVIDIA GeForce RTX 5070 Ti 16GB local LLM speed, memory and powers

The extra 4GB can also serve conversation headroom

VRAM holds both weights and the conversation's KV cache. A model that barely fits in 12GB may run out of room with longer input. Select the same model and increase context in steps to see whether 16GB expands model choice or enables longer conversations with the same model.

Use the same question for heavily compressed larger models

A smaller file can make a larger model runnable. Alongside capacity, check whether its answers preserve the requirements and code your usual questions need. Try both a fast smaller model and a more heavily compressed larger model to choose the combination that suits you.

Consider the next upgrade too

If you plan to keep moving to larger models, compare stopping at 16GB with moving to 24GB or more. Upgrading from a 5070 to a 5070 Ti and quickly hitting capacity again means paying for two changes. Check the model you need now and the one you expect to use soon.

core difference

Which device suits the task?

Local pricing

RTX 5060 Ti 16GB is about ₩190,000 cheaper at the midpoint. Check perceived speed and the task-by-task verdicts together.

Each device detail

• NVIDIA GeForce RTX 5060 Ti 16GB local LLM speed/memory/electricity rate
• NVIDIA GeForce RTX 5070 12GB local LLM speed, memory and powers

A higher-tier card can have less memory

The 5070 has higher bandwidth but 12GB of VRAM. A file that fits the 5060 Ti 16GB may need partial offloading on the 5070. In that case, model placement matters before GPU compute. Choose the required file, then check the loading status on both machines.

Test short answers and long documents

For conversation with a small model that fits both cards, compare generation speed. For long documents, increase input to check KV cache headroom and first-token time. Model and input choices can change the reason to choose a card, so compare two of your usual workloads separately.

Check that the product is the 16GB version

The 5060 Ti comes in both 8GB and 16GB versions. This page compares the 16GB configuration. Check the VRAM option as well as the chip name on a listing. An apparently much cheaper 8GB product has different model capacity from this comparison.

core difference

Which device suits the task?

Local pricing

A price comparison is deferred because a device has no confirmed price yet.

Each device detail

• NVIDIA GeForce RTX 4060 Ti 16GB local LLM speed, memory and powers
• NVIDIA GeForce RTX 5060 Ti 16GB local LLM speed/memory/electricity rate

Separate lack of capacity from slow answers

If the model loads and completes conversations on your 4060 Ti, locate the slow stage. A delayed first character after long input points to prefill; slowly continuing answers point to decode. If execution stops for lack of memory, another 16GB card still requires adjusting the file and context first.

Use the runtime path suited to the newer card

The 5060 Ti is Blackwell and the 4060 Ti is Ada. Distinguish keeping the old file and engine after upgrading from adopting the recommended setup. The comparison selects supported configurations per device; use its runtime and quantization labels when preparing your installation.

Keep the existing card if it meets your needs

For occasional short questions, a new card may offer little value relative to its price. Repeated code generation or document summaries accumulate the time saved. Set your usual input and answer lengths, run both, and decide based on the reduction in waiting.

core difference

Which device suits the task?

Local pricing

RTX 4080 SUPER 16GB is about ₩450,000 cheaper at the midpoint. Check perceived speed and the task-by-task verdicts together.

Each device detail

• NVIDIA GeForce RTX 4080 SUPER 16GB local LLM speed, memory and powers
• NVIDIA GeForce RTX 5080 16GB local LLM speed, memory and powers

Does more bandwidth help your conversations?

Generating an answer token by token makes fast weight reads important. Check bandwidth, then run decode mode. If you often process long input, compare full mode too. Looking at both stages gives a better account of workload time than judging from one metric.

Distinguish Blackwell files from your existing ones

Fast quantized paths on the 5080 require the runtime's matching files and packages. Your 4080 SUPER's GGUF setup may differ from the new recommendation. Update the existing card's supported path too, so you can distinguish a hardware upgrade from a software improvement.

Use a larger-memory comparison for workloads beyond 16GB

If loading still fails after reducing context, or your model requires a smaller quantization, compare increasing capacity. Moving from a 4080 SUPER to a 5080 solves a different problem from moving to a 5090. Related comparisons below include larger-VRAM options.

core difference

Which device suits the task?

Local pricing

A price comparison is deferred because a device has no confirmed price yet.

Each device detail

• NVIDIA GeForce RTX 4070 Ti SUPER 16GB local LLM speed, memory and powers
• NVIDIA GeForce RTX 5070 Ti 16GB local LLM speed, memory and powers

Start with the same model capacity

Changing chip generation here does not increase total VRAM. Start with a model file and conversation that fit within 16GB. Compare a short question on your usual model, then use a long document to inspect the first-response difference.

Account for switching runtimes

A new recommended path may require another model file and a server-address change if it differs from your app. Check the runtime label and guide to see how your chat app connects. Keeping the workflow you use most makes a speed improvement easier to carry into daily work.

Separate buying new from replacing an owned card

For a first PC, compare current prices alongside power and cooling requirements. If you own a 4070 Ti SUPER, weigh a new card's cost against the wait it saves now. The same performance difference has different value to a first-time buyer and an existing owner.

core difference

Which device suits the task?

Local pricing

RTX 5080 16GB is about ₩40,000 cheaper at the midpoint. Check perceived speed and the task-by-task verdicts together.

Each device detail

• RTX 3090 for local AI: keep it or buy used?
• NVIDIA GeForce RTX 5080 16GB local LLM speed, memory and powers

Can the new card load your current file?

A 27B quantized file that works on the 3090 may leave too little conversation room on the 5080. Compare loading with the same model and input before replacing it. If you plan a smaller quantization, check answer quality on your usual questions too. Preserving the workload is the first criterion when capacity falls.

Check new runtimes and the existing card's setup

Use each card's supported path rather than applying the 5080's recommendation directly to the 3090. First check whether updating the 3090's runtime reduces waiting. Changing the model here also updates runtime and placement recommendations for both sides; use these labels when choosing files.

If you need 24GB, compare keeping, adding and replacing

If 16GB is regularly insufficient, keeping the 3090 or adding another GPU are options. Splitting a model needs runtime support plus slots, power and cooling. For a single-card replacement, compare the 32GB 5090 too. This helps find a faster path without losing workloads the old card could handle.

core difference

Which device suits the task?

Local pricing

RTX 5080 16GB is about ₩1,280,000 cheaper at the midpoint. Check perceived speed and the task-by-task verdicts together.

Each device detail

• RTX 4090 24GB local LLM and image creation performance
• NVIDIA GeForce RTX 5080 16GB local LLM speed, memory and powers

Compare the file and context as a pair

A file can load in 16GB yet leave too little room for long input. Compare using the document length and conversation depth you use on the 4090. Even a promising short-question speed difference can be offset by more offloading in real document work.

Check new runtime benefits on supported models

Engines using Blackwell need matching model formats and kernels. Check the recommendation here and follow the model's guide. If you intend to keep an existing CUDA tool, compare with a file that tool supports too, so the comparison reflects the workflow after replacement.

Name the problem the replacement should solve

If your 4090 is hot or noisy, power limits and cooling adjustments are options to test first. For larger models, expand the comparison to higher-memory configurations. For repeated waiting on the same model, use generation with the recommended setup. Identifying the problem narrows replacement candidates.

core difference

Which device suits the task?

Local pricing

RTX 5090 32GB is the price of GPU separately, 2× RTX 5090 64GB (PCIe) is the price of 2 GPUs. For individual GPUs, you need to add the cost of the main body before comparing.

Each device detail

• RTX 5090 32GB local LLM/image/video generation performance
• 2× NVIDIA GeForce RTX 5090 PCIe 64GB local LLM speed·memory·electricity cost

Choose between splitting a model and dividing requests

Splitting one large model makes GPUs exchange data during computation. Keeping a small model on each card lets you route requests separately. The first serves capacity, the second concurrent work. After the single-answer comparison here, test the concurrency your actual server needs.

Check PCIe slots and physical spacing first

Two 5090s require checking actual motherboard lane wiring, card thickness and cooling clearance. The PSU and cables must accommodate both cards. Add any component replacements to the second GPU's cost and compare the complete build. Confirm that the case and system can house both before buying.

Check waiting time after moving to a larger file

If the goal of a second card is a larger model, comparing only the old model's speed is insufficient. Select the larger model and context you will actually use, then inspect first response and completion. Consider both whether it gives the required answer and whether the wait fits your workflow.

core difference

Which device suits the task?

Local pricing

2× RTX 5090 64GB (PCIe) is the price of 2 GPUs, RTX PRO 6000 96GB is the price of GPU separately. For individual GPUs, you need to add the cost of the main body before comparing.

Each device detail

• 2× NVIDIA GeForce RTX 5090 PCIe 64GB local LLM speed·memory·electricity cost
• NVIDIA RTX PRO 6000 Blackwell 96GB local LLM speed/memory/electricity rate

One 96GB card and two 32GB cards need different setups

The PRO 6000 fits a broader range on one GPU. Two 5090s require splitting the model or assigning requests separately. For one model's response, check the splitting runtime and inter-card connection. For a multi-user server, independent models on each card are another option.

Budget beyond the card prices

Two 5090s may require a new PSU, case or motherboard in addition to the cards. The PRO 6000 also needs suitable power and cooling. Use the card prices and power figures below, then add the parts your existing PC actually needs to compare total cost.

Test the large model and your everyday model

Even if a large model fits, long waits on the smaller everyday model can disrupt work. Compare a model that fits both and one that fits only one because of capacity. This clarifies whether you are paying for the PRO 6000's memory or dual-5090 execution performance.

core difference

Which device suits the task?

Local pricing

Mac mini M4 Pro 48GB is about ₩2,130,000 cheaper at the midpoint. Check perceived speed and the task-by-task verdicts together.

Each device detail

• Mac Mini M4 Pro 48GB Local LLM 27B·35B Performance
• Mac mini M4 Pro 64GB local LLM speed, memory, electricity bill

Reserve memory for your other apps

Unified memory serves macOS, the browser and editor as well as the model. Check headroom after opening documents and coding tools with the model loaded. If you keep many apps open, the difference between 48GB and 64GB can matter more in daily work than in an isolated benchmark.

Compare the same model before increasing file size

Compare generation with the same model and input first. Then try a larger quantization or model to examine capacity. Rather than expecting twice the speed for the same small model, check how comfortably the larger-memory configuration accommodates your intended file and context.

Account for memory that cannot be added later

Mac unified memory is chosen as part of the purchased configuration. Even if you use a small model now, compare against the larger file you plan to use soon. Instead of choosing maximum capacity without a plan, name your current and next models to evaluate the price difference.

core difference

Which device suits the task?

Local pricing

Studio M4 Max 64GB is about ₩3,250,000 cheaper at the midpoint. Check perceived speed and the task-by-task verdicts together.

Each device detail

• Mac Studio M4 Max 64GB local LLM speed, memory and power
• Mac Studio M4 Max 128GB local LLM large model performance

Will you keep frequently used models resident?

If every model switch means waiting for a load, switching time matters alongside answer speed. Decide whether to keep one large model or several smaller ones resident. The 128GB configuration leaves more room for that workflow. oMLX's pinning and unload settings can also reveal waiting you can reduce with current capacity.

Include workloads with growing conversation history

A short first question may not reveal the capacity difference. Compare long documents and follow-up questions that require a larger KV cache. Check that requests finish while other apps are open to choose memory for the work you actually do.

The extra cost buys capacity on the same chip

This pair compares memory options on M4 Max. If the goal is a substantial speed improvement on the same model, open comparisons with a higher-tier chip or another device. Allocate the extra budget after deciding whether the required improvement is capacity or faster computation.

core difference

Which device suits the task?

Local pricing

Studio M5 Max 128GB is about ₩9,390,000 cheaper at the midpoint. Check perceived speed and the task-by-task verdicts together.

Each device detail

• Mac Studio M5 Max 128GB local LLM speed, memory and power
• Mac Studio M5 Ultra 256GB: Real-World Local LLM and Agent Performance

Separate large-model capacity from small-model speed

There are two main reasons to choose Ultra: fitting a model beyond Max's capacity or reducing waiting on a model that fits both. Run the everyday model on both, then select one that was difficult on Max because of memory. The two results separate the need for capacity from speed.

Match oMLX and the model format

The recommended Mac path depends on both engine version and model-specific acceleration. Check the oMLX, quantization and MTP labels, then prepare the corresponding file. Update an existing Max's runtime before comparing to distinguish waits requiring new hardware from those reduced through settings.

Weigh the price difference against repeated waits

A few short questions per day and continuous code or report processing justify cost differently. Compare first response and completion at your usual context length, then consider how often that benefit matters. If memory currently prevents a workload, include the value of making that work possible.

core difference

Which device suits the task?

Local pricing

Studio M5 Ultra 256GB is about ₩19,230,000 cheaper at the midpoint. Check perceived speed and the task-by-task verdicts together.

Each device detail

• Mac Studio M5 Ultra 256GB: Real-World Local LLM and Agent Performance
• Mac Studio M5 Ultra 512GB: ₩38.05M Price and 256GB Comparison

Decide what you will keep in 512GB together

Capacity depends on whether you run one large model or keep coding, document and embedding models resident together. Budget files plus workspace and KV caches. Pinning only frequently used models is another option. Define that usage plan first to make the reason for 512GB concrete.

Capacity is the key difference for the same model

This pair uses the same M5 Ultra, so compare capacity instead of interpreting memory size as an answer-speed multiplier. Compare one model and input, then select a larger file to see if only one configuration fits it. Check separately whether the larger file improves answers for your workload.

Prepare storage for the model files too

A large memory option still needs SSD space to download and store models. Trying several quantizations can leave multiple copies of one model. Plan the files to retain, cache locations and how to remove unneeded copies so the machine's capacity is used efficiently.

core difference

Which device suits the task?

Local pricing

Studio M3 Ultra 512GB is about ₩13,480,000 cheaper at the midpoint. Check perceived speed and the task-by-task verdicts together.

Each device detail

• Mac Studio M3 Ultra 512GB for Local LLMs: Is 256GB Enough?
• Mac Studio M5 Ultra 512GB: ₩38.05M Price and 256GB Comparison

Update the existing Mac's runtime first

An older engine on M3 Ultra may leave room for shorter waits without new hardware. Match the recommended engine, quantization and acceleration before comparing. This separates gains from a new chip from those achieved by updating software.

Run long prefill and long answers separately

For frequent reports, full mode with long input matters. For code or long drafts, compare continuing answer speed. Decide which stage must improve to keep work flowing, then weigh it against the new machine's cost.

Distinguish SSD loading from inference waits

Initial model loading and waiting on every question need different improvements. For slow initial loading, check residency and storage location. For repeated documents, examine cache settings. Keep these waiting stages separate when comparing the new Mac's SSD and chip.

core difference

Which device suits the task?

Local pricing

Studio M5 Max 128GB is about ₩1,550,000 cheaper at the midpoint. Check perceived speed and the task-by-task verdicts together.

Each device detail

• MacBook Pro M5 Max 128GB local LLM speed, memory and power
• Mac Studio M5 Max 128GB local LLM speed, memory and power

Choose local laptop use or connecting to a server

A MacBook Pro can run files directly while travelling. A Mac Studio server can accept requests from apps on other devices, but remote access and authentication need preparation. Decide first whether you need offline use away from home or access from several devices at home.

Check cooling and power with long-running work

A few short questions and keeping a model active for hours impose different demands. Consider placement, ambient temperature and concurrent work. Compare response speed and sustained power here; check long-session noise and temperature on the actual configuration.

Include display and input-device costs

The MacBook Pro includes a display and keyboard. A new Mac Studio setup may need a monitor, keyboard and mouse. Costs differ for someone with peripherals and someone building from scratch. List what you actually need rather than comparing machine prices alone.

core difference

Which device suits the task?

Local pricing

DGX Spark 128GB is about ₩9,340,000 cheaper at the midpoint. Check perceived speed and the task-by-task verdicts together.

Each device detail

• NVIDIA DGX Spark 128GB Local LLM Selection Criteria
• Mac Studio M5 Ultra 256GB: Real-World Local LLM and Agent Performance

Check the environment required by your tools

Prepare DGX Spark around NVIDIA software and CUDA, and Mac around MLX or Metal paths. The same model can require different files and server setup. Check your app's server connection, packages and CPU-architecture support first to avoid software obstacles after buying.

Check 128GB versus 256GB with the actual file

Large models need workspace and conversation cache beyond weights. Choose your quantized file and long input, then compare headroom. Check speed on a file that fits Spark first, then consider why you would use a larger file on Mac. This shows whether the extra spending is for capacity.

Separate model serving from everyday computing

Decide between editing and inference on one Mac or using a separate Spark server. Separation frees memory on the work PC but adds connections and management. One machine is simpler but shares memory with other apps. Include that operating model alongside hardware prices.

core difference

Which device suits the task?

Local pricing

Ryzen AI Max+ 395 128GB is about ₩4,410,000 cheaper at the midpoint. Check perceived speed and the task-by-task verdicts together.

Each device detail

• Ryzen AI Max+ 395 128GB local LLM cost-effectiveness
• NVIDIA DGX Spark 128GB Local LLM Selection Criteria

Start with the memory available to the GPU

On a Ryzen AI Max+ system, check both GPU allocation and memory left for the OS. DGX Spark also needs operating-system and runtime headroom. Compare loading using the usable-memory figures below and check whether the actual machine's BIOS and OS can use that allocation.

Choose between AMD runtimes and CUDA tools

To keep CUDA-only packages, examine the Spark environment. On AMD, select the ROCm, Vulkan or HIP path actually supported by the system and tool. If you have a specific tool in mind, its OS, architecture and package support table matters before the GPU name.

Keep prices tied to the actual chassis

Ryzen AI Max+ 395 machines vary in memory, SSD and cooling. GB10-family products also vary by vendor. Match the memory option and actual chassis to this comparison when checking prices. A different chassis also calls for checking power and sustained execution conditions.

core difference

Which device suits the task?

Local pricing

Mac mini M4 Pro 64GB is the price of finished product, RTX 5060 Ti 16GB is the price of GPU separately. For individual GPUs, you need to add the cost of the main body before comparing.

Each device detail

• Mac mini M4 Pro 64GB local LLM speed, memory, electricity bill
• NVIDIA GeForce RTX 5060 Ti 16GB local LLM speed/memory/electricity rate

Choose between smaller VRAM and larger shared memory

The 5060 Ti uses 16GB VRAM separately from system RAM; Mac mini shares 64GB across CPU, GPU and apps. Separate the room for a large file in unified memory from the speed of a smaller model fitting VRAM. Offloading between system RAM and GPU also depends on the PC configuration.

Will you also use image-generation tools?

If you also want image or video tools, check their CUDA and Mac support. One LLM's generation speed does not cover every workload. Compare text responses here, then inspect the image and video experience linked from device details for the models you will use.

An existing PC changes the extra cost

If the PC already has suitable power, slots and cooling, adding a GPU can be the starting cost. A new PC also needs CPU, board, RAM, SSD, PSU and case. For Mac mini, add a display and input devices if required. Separate owned parts from new purchases for a fair budget.

core difference

Which device suits the task?

Local pricing

RTX 5090 32GB is the price of GPU separately, Studio M5 Max 128GB is the price of finished product. For individual GPUs, you need to add the cost of the main body before comparing.

Each device detail

• RTX 5090 32GB local LLM/image/video generation performance
• Mac Studio M5 Max 128GB local LLM speed, memory and power

Compare answers from a fast small model and a larger one

Choose a few everyday questions before selecting hardware. Compare answers from a model that runs quickly on the 5090 with a larger file on Mac. Seeing which delivers the coding, summary or document answers you need helps prioritize speed versus capacity.

Include software and model files in the choice

NVIDIA paths need CUDA and the runtime's quantized files; Mac paths need MLX or Metal-compatible files. Check whether your chat or image app can connect to the new server. Follow the recommended configuration to its guide to connect the hardware choice with practical preparation.

Compare a GPU and a complete machine on total cost

A 5090 price does not include the whole PC. Reusing a chassis or building a new one changes cost substantially. Add required PC parts and peripherals on the same basis when comparing with Mac Studio's machine price. Check sustained power below for always-on use too.

Key takeaways

1. Distinguish consultation on the final rules from the pre-export registration already in effect.
2. API access, overseas self-hosting, and already public technology have different treatment.
3. Prepare a local model and its runtime settings for a task you use regularly.

What has started and what is still being decided

On October 16, 2026, China's commerce and science ministries proposed adding AI, biotechnology, and solar technologies to its catalog of prohibited or restricted technology exports. The AI categories cover large models, agents, computing clusters, and information processing. The controls focus on which technology is transferred overseas and how that transfer happens.

The consultation runs through November 15. Until the revised catalog formally takes effect, exporters must register covered technology transfers in advance. Keep the registration already underway separate from the final rules still being consulted on. For a model you use now, its delivery method and the service's own notices are the practical points to check.

• Ministry of Commerce and Ministry of Science and Technology Announcement No. 48 of 2026
• Where can you use open-weight AI for free? Five API services
• Choosing local LLM hardware: memory, speed and power
• What is a local LLM? From installation and PC specifications to model selection and speed checks

Receiving an API answer versus transferring a model overseas

When you send a question to OpenRouter, a connected provider runs the model on its servers and returns an answer. If an overseas operator receives the model files and installs them on its own servers, the model technology itself is transferred. The official Q&A treats these cases differently.

Based on the Chinese Ministry of Commerce announcement and Q&A dated October 16, 2026.

• Official Ministry of Commerce Q&A

What could change on OpenRouter?

OpenRouter lets you use multiple providers' inference APIs through one connection. A model may have several providers with different speeds, prices, and data-processing locations. Even when you select the same model on OpenRouter, different providers' servers may handle your requests.

A provider acquiring restricted technology to operate it on overseas servers may be affected by technology-transfer procedures. Current preregistration and the management rules still to be finalized could affect when an overseas provider adds a new endpoint. This is a scenario based on the distinction between connecting an existing API and newly hosting a model overseas.

This announcement contains no decision to stop model availability on OpenRouter or block all Qwen or DeepSeek use. If a model you use changes, check its model page for the provider and available regions, then read that provider's notice to identify affected endpoints.

• How providers connect to OpenRouter

What a local LLM lets you keep doing

A local LLM runs downloaded model files on your PC or Mac. For a task such as finding a bug in public code and suggesting a fix, you can send the same code to a local application. Your computer performs the inference, letting you continue that task without relying on a hosted inference API's availability or pricing policy.

Once the model, application, and required runtime are downloaded, LM Studio can chat and run a local server without internet access. Finding and downloading additional models requires connectivity. Tasks that use web search or external tools also need those connections, so a self-contained job such as reviewing a code snippet is a straightforward starting point.

Keeping a model locally preserves access to that version's capabilities. If you also need future larger models or a hosted service's search features, divide tasks between the existing API and your local model. Keep the license and usage terms with the downloaded files.

• LM Studio's offline capabilities

What to prepare before buying another GPU

Start with the [hardware finder](/?tab=finder), selecting your current PC or Mac and a model to narrow down feasible configurations. Leave room for the model files plus the input and conversation state. If your usual work is short exchanges, you can begin with a smaller local model rather than matching the size of the hosted model.

Load the model in your app and enter a piece of code that is safe to share. Ask for the cause of an error and a proposed fix, then follow up about the same function. If you can complete the review and follow-up at a pace that suits your work, keep that setup as a fallback. Separately record whether the answer covers the needed cause and fix, and how long you waited for the first response, to identify reasons for changing models or hardware.

To keep using your existing app, follow the [guide to switching from OpenRouter to a local LLM](/guides/openrouter-to-local-llm-fallback-guide). Preserve a working configuration with the [model backup and restore guide](/guides/local-llm-model-backup-restore-guide). The goal is to have a task you can continue if a service changes, rather than to buy new hardware first.

What to check when something changes next

Watch the final Chinese announcement for the settled scope and exceptions, and your provider's notices for changes to models, regions, or delivery methods. If a model's provider changes, check the data-processing location and privacy terms as well as price. Adjust the connection for the affected task when there is a concrete change.

If your current API works well, keep using it. If a change in provider or terms makes it hard to continue your task, you can switch to the local setup you tested earlier.

Frequently asked questions

What should you check first in this article?

China has proposed additional controls on transferring certain AI technologies overseas. As of October 17, 2026, pre-export registration has begun for covered transfers, while the final scope remains under consultation. Receiving an answer through an API and deploying a model on an overseas server receive different treatment. Start with how you access the model to decide what to keep using and what to prepare.

How much memory does it need?

Start with the [hardware finder](/?tab=finder), selecting your current PC or Mac and a model to narrow down feasible configurations. Leave room for the model files plus the input and conversation state. If your usual work is short exchanges, you can begin with a smaller local model rather than matching the size of the hosted model.

Where can you check the conditions and sources?

Keeping a model locally preserves access to that version's capabilities. If you also need future larger models or a hosted service's search features, divide tasks between the existing API and your local model. Keep the license and usage terms with the downloaded files.

Key takeaways

1. Download a model file supported by an LM Studio runtime and load it into memory. Complete any runtime installation prompt on the first run.
2. Check the server address in the Developer tab and the `/v1/models` response; enter a model ID shown in that list in your app.
3. Keep a separate local profile and new chat instead of overwriting the OpenRouter profile. If server authentication is enabled, enter only its local token; do not send an external service key.

A local API server receives app requests and returns answers

LM Studio's API server receives code and questions from the app, runs them through the local model, and sends the answer back. First check whether your app's settings offer OpenAI-compatible or Custom provider. This is where you will enter LM Studio's server address (Base URL) and model ID. [LM Studio server guide](https://lmstudio.ai/docs/developer/core/server) · [Compatible API and Base URL](https://lmstudio.ai/docs/developer/openai-compat)

The function below calculates what percentage `part` represents of `total`. Send it to the local model and ask what happens when `total` is zero and how to fix it.

def percentage(part, total):
    return part / total * 100

• GGUF vs MLX: which model format should you download?
• Qwen3.8 27B in 7.89 GB? Install Saluki and Plan for an 8 GB GPU
• Gemma 4 12B serving: Q4 on desktop and BF16 on a server
• How much VRAM or unified memory does a local LLM need?
• Building an ESPHome voice satellite: an ESP32-S3 microphone and wake word

Prepare the model and runtime while you have a connection

Install LM Studio and find a model in the Discover tab. Check its model card for a format supported by an LM Studio runtime, such as GGUF or MLX, and choose a quantized file that fits your memory. If your task needs tool calls, check that the selected model supports them. [Model download guide](https://lmstudio.ai/docs/app/basics/download-model)

After downloading, select the model in the Chat tab and load it into memory. Complete any runtime installation prompt on the first run. You can import an external GGUF into LM Studio's model folder with `lms import --copy`. If only OpenRouter is unavailable but your internet connection works, you can still prepare the model. To prepare for losing internet access too, download the model and runtime in advance. Model search, downloads, runtime installation, and updates require a connection. [Offline operation](https://lmstudio.ai/docs/app/offline) · [Import models](https://lmstudio.ai/docs/app/advanced/import-model)

Check the localhost server and actual model ID

Open the Developer tab in LM Studio and turn on Start server. The default example port is 1234, and Server Settings can change it, so use the actual API address shown in the app. Leave Serve on Local Network off and keep the server on this computer's localhost. [Start the server](https://lmstudio.ai/docs/developer/core/server) · [Server settings](https://lmstudio.ai/docs/developer/core/server/settings)

Open `http://localhost:1234/v1/models` in a browser or terminal and copy an `id` from its `data` array into the app. If you changed the port, replace 1234 with the actual number. With JIT loading enabled, the list may include models stored on disk but not yet loaded into memory. Load the model first from the Chat tab's model loader, or enable JIT in Server Settings. [List models API](https://lmstudio.ai/docs/developer/openai-compat/models) · [Server settings](https://lmstudio.ai/docs/developer/core/server/settings)

With the default authentication setting, this model-list request needs no external API key. If you enabled Require Authentication in LM Studio, use a local API token created through Manage Tokens in Server Settings, and enter it in the app only if its authentication field requires one. Never put an OpenRouter or other cloud key in a localhost request. [LM Studio authentication](https://lmstudio.ai/docs/developer/core/authentication)

curl http://localhost:1234/v1/models

Create a separate local profile and open a new chat

Create a local profile by choosing OpenAI-compatible or Custom provider in your app's model-provider settings. Do not duplicate the OpenRouter profile because its provider type or automatic routing may remain. Set Base URL to `http://localhost:1234/v1` and use the ID copied from `/v1/models`. If the app offers automatic provider fallback or a model router, turn it off so failed requests are not sent to an external service. [LM Studio Base URL format](https://lmstudio.ai/docs/developer/openai-compat)

After saving the profile, select the local profile and open a new chat. Review the messages and attachments the app includes, then enter the code and question you want reviewed. Bring over any code or conditions from the previous cloud conversation that are needed for this review.

Check the connection with a short question, then review the code

Ask `What is 1 + 1?` in the app. If an answer returns, the basic request-and-response connection works. If no answer arrives, use the error table in the next section. To also check that the request actually reached LM Studio, use the optional log check below.

To inspect request logs, run `lms --help` in a terminal and check that the command list appears. If the command is not found, follow the [LM Studio CLI setup guide](https://lmstudio.ai/docs/cli). Then run the log command below and leave the terminal open.

lms log stream --source server

If you started the log stream, repeat the same short question and check that the request appears. [Log command guide](https://lmstudio.ai/docs/cli/serve/log-stream) Once the connection is checked, send the example function with `Find the bug and suggest a minimal fix. What happens when total is 0?` After the model identifies `ZeroDivisionError` when `total == 0` and proposes a fix, compare how the code handles zero and what it returns.

A local model may have a different context limit or tool-calling support from the cloud model. For a long request, keep the code and question but shorten the input; for a task that needs tools, compare results with the app's tool integration off and on. For this example, check that the model finds the division-by-zero issue and that its proposed handling produces the intended return value.

Check the address, model ID, and authentication when an error occurs

When an error occurs, note the first status code or message in the app and LM Studio logs. Find the symptom in the table, change one item at a time, and repeat the same short question. Check whether a request appears in the server log to help distinguish an app connection problem from a model response problem. If task results differ after a valid response, also check context length, tool support, and output limits. [Server status guide](https://lmstudio.ai/docs/cli/serve/server-start)

Record the connection details and resume later

Note the server port, model ID, and runtime so you can continue the same check in a new chat later. For preserving model files with runtime settings, see the [local LLM backup and restore guide](/guides/local-llm-model-backup-restore-guide); for the scope of related policy, see the [AI export controls and local models guide](/guides/china-ai-export-controls-openrouter-local-llm-guide).

Hardware-specific speed tuning and serving recipe summary

Provides weights, KV cache, prefill batch, and concurrent request with acceleration functionality, separated for balanced performance, speed, and long-context profiling.

The detail page shows each profile's command, settings, stop conditions, and measurement steps. Commands appear only for combinations with a confirmed model file and runtime.

Frequently asked questions

What should you check first in this article?

LM Studio runs models on your PC or Mac and can answer questions from other apps. If your app supports a custom OpenAI-compatible connection, create a separate local profile that connects to LM Studio. Prepare the model in advance, then test the connection and response with a short code review.

How much memory does it need?

Install LM Studio and find a model in the Discover tab. Check its model card for a format supported by an LM Studio runtime, such as GGUF or MLX, and choose a quantized file that fits your memory. If your task needs tool calls, check that the selected model supports them. [Model download guide](https://lmstudio.ai/docs/app/basics/download-model)

Where can you check the conditions and sources?

Install LM Studio and find a model in the Discover tab. Check its model card for a format supported by an LM Studio runtime, such as GGUF or MLX, and choose a quantized file that fits your memory. If your task needs tool calls, check that the selected model supports them. [Model download guide](https://lmstudio.ai/docs/app/basics/download-model)

Key takeaways

1. Record the model repository ID, fixed commit ID, original file names, and format.
2. A GGUF may be one file, but a split model needs every shard; MLX and safetensors models should be backed up with their related files.
3. Keep the original folder and test a copy at another path so you can continue using the model you know if the test fails.

Choose your everyday model and a test document first

Start with one model you regularly use for document summaries. Before copying it, enter ‘Service will be unavailable for 10 minutes next Tuesday at 3 p.m. for maintenance,’ and ask, ‘Summarize the schedule and expected impact in one sentence.’ Keep both the notice and prompt so you can repeat the test after loading the copy. Leave your original model folder in place until the restore test is complete.

To retrieve the same files, record the fixed commit rather than the repository's `main` branch or a movable tag. Redownloading from a private or gated repository requires an authorized account and access. Restore model files you already have locally; check the license terms before sharing or distributing the model.

• Hugging Face file downloads and revisions
• GGUF vs MLX: which model format should you download?
• Building Fully Offline Local AI: Air-Gap Installation and Update Procedures
• Run local AI on a Mac with oMLX: what is new in 0.7.1rc1?
• Which local LLM app or serving engine should you start with?

Check what to keep based on the model format

First, find the folder the app actually loads in its model manager. Remembering only the model name can lead you to download a similarly named quantization or a newer revision. Record the repository owner and ID, exact commit revision, original file names, format and quantization, app and runtime versions, and any settings you changed, such as the chat template or context length.

A GGUF file contains weights and metadata describing the model, and some models run from a single file. Check the model card and runtime's file requirements. If the model is split across multiple files, keep every shard together in one folder. Models that handle images or audio may need a separate projector file, so follow the file names and layout specified in the model card.

MLX and safetensors models can use configuration, tokenizer, index, and shard files to load their weights. Copy the full repository folder your app currently reads and include the files named in the model card. Keep a copy of the model card and license in the folder so you can identify the repository and its terms later.

• GGUF metadata and format
• safetensors shards and index files
• Hugging Face cache and snapshot structure

Copy the whole model folder and compare its files

Open the model folder you identified in the app using File Explorer or Finder, then copy the entire folder to your backup drive. Keep any weight shards, configuration files, and tokenizer files together inside it. Once copying finishes, compare file lists and sizes with the original and keep your recorded versions and settings, model card, and license with the backup. Leave the original folder intact until the restore test is complete.

Pay attention to symbolic links when copying a folder from the default Hub cache. If entries in `snapshots` link to data in `blobs`, either follow those links to copy the actual files or preserve the links and copy both `snapshots` and `blobs`. Moving only a snapshot can break links that pointed to the original computer's cache.

Compare the copy's file list and sizes with the original, then use a SHA-256 checksum to check whether important large files have identical bytes. On macOS, open Terminal and replace the two paths below with the actual paths to the source and copied GGUF files. If the checksum strings before the file names match, the files at those paths have identical contents.

shasum -a 256 "$HOME/Models/Publisher/Model/model.gguf" "/Volumes/Model-Backup/Publisher/Model/model.gguf"

• Hugging Face cache management

Import a GGUF backup into a separate LM Studio folder and run it

Open the LM Studio app once, then run `lms --help` in a terminal and check that the command list appears. If it is not found, follow the [CLI setup guide](https://lmstudio.ai/docs/cli). The command below imports one copied text-only GGUF into a test model folder named `backup-check/restored-model`. `--copy` also leaves the imported source file intact, and `--user-repo` selects a folder separate from your everyday model. If that test folder is already in use, choose a new name rather than overwrite existing files.

lms import "/Volumes/Model-Backup/Publisher/Model/model.gguf" --copy --user-repo "backup-check/restored-model"
lms ls

After importing, select the test model under `backup-check/restored-model` in `lms ls` or the app's My Models screen. The test uses this model imported from your backup drive. Match the recorded app and runtime versions and settings, then send the opening notice and summary prompt again. Check that the answer includes Tuesday at 3 p.m. and the ten-minute interruption.

To check offline use, disconnect the network once the model and runtime are ready and repeat the same summary. LM Studio can chat with downloaded LLMs and run a local server offline. Model search, new file and runtime downloads, and update checks require a connection.

If the result differs from your usual one, compare the model path, any missing shards, the loaded model ID, template, and context settings with your record. A load error can point to runtime compatibility or missing companion files; a response difference can follow from changed settings or runtime versions. Check where the process diverged before downloading files again so you can distinguish a difference in the original from one in the copy.

• Import a local GGUF into LM Studio
• Set up the LM Studio CLI and basic commands
• LM Studio offline operation

Use a fixed commit when downloading files again

When downloading a model from Hugging Face, copy the full fixed commit ID from the repository's commit history. Pass it to `hf download` with `--revision` to select the same version; add `--dry-run` first to see the file list and total size before downloading. Add `--local-dir` when you want the full repository saved as regular files in a folder you choose. Downloading a gated model requires authentication and access approval for that account.

If you already backed up the model folder, restore from those files. For a fresh download, `--local-dir` saves regular files in the repository structure under the folder you choose. After downloading, follow the earlier copying, file comparison, and restore test steps to check your backup.

• Hugging Face CLI downloads, revisions, and dry runs

A model backup and a full app backup cover different things

The restore command in this guide is for a single text-only GGUF used with LM Studio. For MLX or safetensors, back up the folder and companion files as described earlier, then follow your original app's model registration procedure. Register the copy in the folder structure that app requires and test it with the recorded settings and the same summary prompt.

For a personal model backup, record the model files and the app and runtime versions and settings used to load them. If the new computer must start without an internet connection, prepare the operating system, app installer, and required runtime separately. First test the copy with the app and runtime you normally use; if you change versions, run the representative document summary again and check for differences.

When one person is moving one model, keep the model folder used by the current app and its settings record. For deployment across multiple devices or bringing a validated installation bundle into a secure zone, see the organizational process in the [offline and air-gapped local AI deployment guide](/guides/offline-airgapped-local-ai-deployment). Personal recovery is complete when you repeat your usual summary with the copy while keeping the original.

To connect the restored model to your usual app, continue with the local profile and server address settings in the [OpenRouter-to-local LLM guide](/guides/openrouter-to-local-llm-fallback-guide).

Frequently asked questions

What should you check first in this article?

A local LLM backup keeps your working model folder, version, and runtime settings on another drive. To restore it, register the backup files in your app and test your usual task. Start by copying the complete folder, then follow a restore example that imports one text-only GGUF into LM Studio and summarizes a notice.

Where can you check the conditions and sources?

To retrieve the same files, record the fixed commit rather than the repository's `main` branch or a movable tag. Redownloading from a private or gated repository requires an authorized account and access. Restore model files you already have locally; check the license terms before sharing or distributing the model.

Key takeaways

1. 74B is the total parameter count, A9B the active parameter scale per token; each GGUF file has its own listed size.
2. Expert weights use IQ2_XXS, Mamba linear layers use NVFP4, and selected components use Q8_0.
3. Memory planning must include context state, runtime buffers, and other applications in addition to model weights.

Compressing expert weights in the 74B model to 2 bits

NVIDIA Nemotron 3.5 Bolt is a 74B model that takes prompts such as questions or code and generates a response. For the 24 GB configuration, most expert weights use IQ2_XXS at 2.0625 bits per element; NVIDIA applies quantization-aware distillation (QAD) to recover quality at the lower precision.

The 24 GB configuration budgets memory for batch size 1 and a 128K context. NVIDIA recommends an actual runtime context of either 65,536 or 262,144 tokens, depending on GPU memory.

Precision differs by component. Mamba linear layers use NVFP4, while the lm_head, token embeddings, attention, and Latent MoE weights use Q8_0. The GGUF file list in the model card lets you add up storage for the main model, image projector, and drafter you plan to use.

For each token, the router selects some experts and computation also passes through Mamba and Transformer blocks. A9B refers to the parameter scale activated per token. When comparing speed, record memory bandwidth, runtime kernels, GPU offload, input length, and concurrent requests.

• RTX 5090 32GB memory, speed and price
• Context length, RoPE and sliding-window attention
• MoE vs dense: why low active parameters still need memory
• GGUF vs MLX: which model format should you download?
• What Is LiLiCorr? Speculative decoding for Bolt and Super VL
• AstaBrief 8B: Create a Cited Research Report Locally

What to check first on a 24 GB PC

Plan memory separately for GPU and CPU placement. Compare the weights, KV cache, and work buffers placed on the GPU with available VRAM. Check CPU-resident model parts alongside the system RAM used by the OS, browser, and editor. Changing GPU-layer offload changes the balance between both memory pools.

Before running it, check the exact GGUF filenames and download sizes in the model card, including any drafter files. Your llama.cpp build must support Mamba and the relevant quantization formats. An older runtime can fail to load the model for reasons unrelated to memory, so compare the model-card instructions with your installed runtime release. NVIDIA's published Bolt test path used llama.cpp on Windows 11 ARM x64.

On an RTX desktop, the runtime may place all layers in GPU memory or offload some to system memory. Partial offloading can make execution possible, but it changes the memory path and may substantially change throughput. Start with a short context and one request. Watch dedicated GPU memory, system memory, and the runtime log; once the model answers reliably, increase input length to isolate the next limit.

Get a first answer with llama.cpp

On Windows, install llama.cpp from PowerShell with `winget install llama.cpp`. Run the quick-start command to download the model and open its local chat UI at `http://127.0.0.1:8080`. Send a short prompt, confirm a response, and stop the server with Ctrl+C. For an RTX 5090, launch the official profile below separately. [NVIDIA GGUF model card](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF)

During the first run, the terminal shows the model name and download progress. After loading, the server prints a ready message and address. Open it in a browser on the same PC and send: `Summarize this sentence in one sentence: The model receives a prompt and generates an answer.` A relevant summary confirms text inference is working. For speed tests, record file loading, time to first token, and generation token/s separately.

The RTX 5090's 32 GB VRAM fits NVIDIA's 32 GB+ discrete-GPU tier. Download the main GGUF, mmproj, and LiLiCorr drafter with the model-card filenames, set their PowerShell paths, and start the profile with a 65,536 context, one sequence, up to six DFlash draft tokens, and Q8_0 KV cache. Quick start downloads automatically; this profile loads the three files directly. Stop the quick-start server with Ctrl+C before starting this one. [NVIDIA Windows recipe](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF)

If the first run reports unsupported tensor, compare your llama.cpp version with the formats supported by the model card. For a CUDA allocation error, reduce context and batch size first, then check dedicated and shared GPU memory in Windows Task Manager. If the server starts but the browser cannot connect, confirm the terminal's port is 8080 and open 127.0.0.1 on that PC.

winget install llama.cpp
llama serve -hf nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF --host 127.0.0.1 --port 8080

python -m pip install --upgrade huggingface_hub
$env:MODEL_DIR = Join-Path $PWD.Path "models\nemotron-bolt"
hf download nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF `
  NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf `
  mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf `
  --local-dir "$env:MODEL_DIR"
hf download nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr-GGUF `
  NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr.gguf `
  --local-dir "$env:MODEL_DIR"
$env:MODEL = Join-Path $env:MODEL_DIR "NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf"
$env:MMPROJ = Join-Path $env:MODEL_DIR "mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf"
$env:SPEC_DRAFT = Join-Path $env:MODEL_DIR "NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr.gguf"

$env:MODEL_DIR = Join-Path $PWD.Path "models\nemotron-bolt"
$env:MODEL = Join-Path $env:MODEL_DIR "NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf"
$env:MMPROJ = Join-Path $env:MODEL_DIR "mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf"
$env:SPEC_DRAFT = Join-Path $env:MODEL_DIR "NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr.gguf"
llama-server.exe `
  --model "$env:MODEL" `
  --mmproj "$env:MMPROJ" `
  --batch-size 2048 `
  --ubatch-size 512 `
  --ctx-size 65536 `
  --parallel 1 `
  -ngl 999 `
  --spec-type draft-dflash `
  --spec-draft-n-max 6 `
  --spec-draft-model "$env:SPEC_DRAFT" `
  --spec-draft-ngl 999 `
  --cache-type-k q8_0 `
  --cache-type-v q8_0 `
  --temp 1.0 `
  --top-p 0.95 `
  --flash-attn on `
  --jinja `
  --host 127.0.0.1 `
  --port 8080

Published generation and agent-task speeds on RTX Spark

NVIDIA compared models on an RTX Spark in a 128 GB Surface Laptop Ultra, using Windows 11 ARM x64 and llama.cpp. AIPerf SPEED-Bench-Coding measures generation speed; AA-AgentPerf Local replays multi-step tasks. The model results and conditions are collected in the table in the companion guide, ‘Nemotron Bolt vs Qwen3.8-27B.’ Here, distinguish what each metric represents.

TPS is token throughput for generating against a fixed input; replay duration is the completion time for a defined set of agent tasks that includes tool use. For short code completion, look first at TPS and time to first token. For workflows chaining several tools, start with end-to-end replay time.

Rather than applying the published figures to your PC, choose one task you actually plan to run and use the same prompt, document, and tools. First record answer quality and time to first token with a short input. Then add your usual document and repeated tool calls and compare total time, VRAM, and failures to see which setting needs to change.

• See the full Bolt and Qwen3.8 results table

When it belongs on your shortlist

Bolt is worth evaluating if you want a model with a 70B-class total size for agent tasks on RTX-class hardware or DGX Spark instead of a data-center server. If your use case involves terminal actions, web browsing, or multi-step planning like NVIDIA's evaluation tasks, first match the model card's prompt format. Quality and tool-call reliability depend on your own instructions, tools, and safety settings, so validate each task in a sandbox before connecting it to real automation.

To test Bolt on a 24 GB card, first check whether the CPU/RAM-first command returns an answer, then add GPU layers to the working setup. Change one setting at a time to distinguish model-load failures from GPU-memory limits. The 3090 guide contains the concrete starting command.

Frequently asked questions

What should you check first in this article?

Nemotron 3.5 Bolt 74B-A9B is a compressed hybrid language and vision model. This guide walks through downloading it and running NVIDIA's llama.cpp recipes for different GPU memory tiers.

How much memory does it need?

The 24 GB configuration budgets memory for batch size 1 and a 128K context. NVIDIA recommends an actual runtime context of either 65,536 or 262,144 tokens, depending on GPU memory.

Where can you check the conditions and sources?

Precision differs by component. Mamba linear layers use NVFP4, while the lm_head, token embeddings, attention, and Latent MoE weights use Q8_0. The GGUF file list in the model card lets you add up storage for the main model, image projector, and drafter you plan to use.

Key takeaways

1. The 3090's 24 GB is dedicated VRAM shared by weights, KV cache, and runtime buffers during execution.
2. Start a 5090 with the official 28 GB dGPU recipe and a 3090 with the CPU/RAM-first path.
3. On a 3090, increase `-ngl` from a working setup in steps and record VRAM and system RAM.

Weights and context both use runtime memory

The RTX 3090 has 24 GB of dedicated VRAM. During inference, model weights share it with the context KV cache, compute buffers, display output, and driver. After launching, check VRAM in Task Manager and find a stable configuration starting with a short `-c 4096` context.

The RTX 5090's 32 GB VRAM fits NVIDIA's 32 GB+ dGPU profile. Confirm that your current llama.cpp build supports the Mamba and Transformer operations and IQ2_XXS, load the model, then adjust context length and concurrent requests one at a time while recording memory use.

• RTX 5090 32GB memory, speed and price
• Context length, RoPE and sliding-window attention
• GGUF vs MLX: which model format should you download?
• LLM memory bandwidth vs compute: how to find the speed bottleneck
• RTX Spark laptops: how do they differ from DGX Spark and MacBook Pro?

Use the official profile on a 5090; start on CPU with a 3090

The RTX 5090's 32 GB meets NVIDIA's official 28 GB discrete-GPU recipe tier. Download the main GGUF, mmproj, and separate LiLiCorr drafter; set their paths in PowerShell environment variables, then run the model-card command as written. It starts the server with a 65,536 context, one sequence, up to six DFlash draft tokens, and Q8_0 KV cache. Keep the mmproj file required by the recipe even when your first test is text-only.

For a CPU-first test on an RTX 3090, download the roughly 22 GB main GGUF from the [official model card](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF) and place it in `models\nemotron-bolt`. Reserve additional system RAM for the OS and runtime buffers. As an experimental first diagnostic, run `-c 4096 -ngl 0` and record the response and RAM use. If it responds, choose the -ngl 8 and 16 commands below one at a time and compare VRAM and output.

Run only one server at a time. Record its response and memory use, then press Ctrl+C in that terminal before moving to the next setting.

python -m pip install --upgrade huggingface_hub
$env:MODEL_DIR = Join-Path $PWD.Path "models\nemotron-bolt"
hf download nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF `
  NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf `
  mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf `
  --local-dir "$env:MODEL_DIR"
hf download nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr-GGUF `
  NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr.gguf `
  --local-dir "$env:MODEL_DIR"
$env:MODEL = Join-Path $env:MODEL_DIR "NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf"
$env:MMPROJ = Join-Path $env:MODEL_DIR "mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf"
$env:SPEC_DRAFT = Join-Path $env:MODEL_DIR "NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr.gguf"

$env:MODEL_DIR = Join-Path $PWD.Path "models\nemotron-bolt"
$env:MODEL = Join-Path $env:MODEL_DIR "NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf"
$env:MMPROJ = Join-Path $env:MODEL_DIR "mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf"
$env:SPEC_DRAFT = Join-Path $env:MODEL_DIR "NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr.gguf"
llama-server.exe `
  --model "$env:MODEL" `
  --mmproj "$env:MMPROJ" `
  --batch-size 2048 `
  --ubatch-size 512 `
  --ctx-size 65536 `
  --parallel 1 `
  -ngl 999 `
  --spec-type draft-dflash `
  --spec-draft-n-max 6 `
  --spec-draft-model "$env:SPEC_DRAFT" `
  --spec-draft-ngl 999 `
  --cache-type-k q8_0 `
  --cache-type-v q8_0 `
  --temp 1.0 `
  --top-p 0.95 `
  --flash-attn on `
  --jinja `
  --host 127.0.0.1 `
  --port 8080

$env:MODEL_DIR = Join-Path $PWD.Path "models\nemotron-bolt"
$env:MODEL = Join-Path $env:MODEL_DIR "NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf"
llama-server.exe --model "$env:MODEL" --ctx-size 4096 --parallel 1 -ngl 0 --host 127.0.0.1 --port 8080

$env:MODEL_DIR = Join-Path $PWD.Path "models\nemotron-bolt"
$env:MODEL = Join-Path $env:MODEL_DIR "NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf"
llama-server.exe --model "$env:MODEL" --ctx-size 4096 --parallel 1 -ngl 8 --host 127.0.0.1 --port 8080

$env:MODEL_DIR = Join-Path $PWD.Path "models\nemotron-bolt"
$env:MODEL = Join-Path $env:MODEL_DIR "NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf"
llama-server.exe --model "$env:MODEL" --ctx-size 4096 --parallel 1 -ngl 16 --host 127.0.0.1 --port 8080

What to record when comparing the 3090 and 5090

To compare the cards, hold the PC, driver, llama.cpp build, model files, context, and prompt constant. Record initial load time, time to first token, generation token/s, VRAM use, and answer accuracy separately. TPS on a short-prompt benchmark can differ from completion time on a real document, so measure both if both matter to you. Use the same number of runs and compare medians alongside averages to reduce the influence of one background task.

NVIDIA's published test used the RTX Spark in a 128 GB Surface Laptop Ultra. Its results cannot be scaled to desktop RTX 3090 or 5090 by GPU generation alone. VRAM, memory bandwidth, CPU/GPU offloading, and runtime kernels differ, so test each card separately with the same model, context, and prompt.

Scale up from a short test to your real workload

After the first response arrives, test a short coding or web-research task as a single prompt and check answer quality. Then enable tool calls and run the same request in a sandbox; inspect changed files and sources. Record total task time, including time spent reviewing tool results.

If CPU/RAM execution fails, check the model path, llama.cpp support for the format, available system RAM, and error log. If a memory error appears after adding GPU layers, return to the last `-ngl` setting that answered. Increase context length in steps after confirming a normal response.

Hardware-specific speed tuning and serving recipe summary

Provides weights, KV cache, prefill batch, and concurrent request with acceleration functionality, separated for balanced performance, speed, and long-context profiling.

The detail page shows each profile's command, settings, stop conditions, and measurement steps. Commands appear only for combinations with a confirmed model file and runtime.

Frequently asked questions

What should you check first in this article?

NVIDIA's official GGUF recipes use different context and batch settings for discrete GPUs with 32 GB+ and 48 GB+ memory. This guide explains which starting point to use on RTX 3090/5090 and what to measure.

How much memory does it need?

The RTX 3090 has 24 GB of dedicated VRAM. During inference, model weights share it with the context KV cache, compute buffers, display output, and driver. After launching, check VRAM in Task Manager and find a stable configuration starting with a short `-c 4096` context.

Where can you check the conditions and sources?

For a CPU-first test on an RTX 3090, download the roughly 22 GB main GGUF from the [official model card](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF) and place it in `models\nemotron-bolt`. Reserve additional system RAM for the OS and runtime buffers. As an experimental first diagnostic, run `-c 4096 -ngl 0` and record the response and RAM use. If it responds, choose the -ngl 8 and 16 commands below one at a time and compare VRAM and output.

Key takeaways

1. Bolt had higher TPS and shorter replay duration than Qwen3.8-27B, under one published setup.
2. TPS is generation throughput; replay duration is the completion time of a fixed task set.
3. A comparison using your prompts, tools, and context length is more directly useful.

Two metrics describe different kinds of waiting

TPS on AIPerf SPEED-Bench-Coding is the token throughput while generating against a controlled benchmark input. It shows how quickly text is produced, but not the full cycle of selecting and running tools or revising a plan after reading results. Perceived wait also depends on time to first token and total output length.

AA-AgentPerf Local replay duration is the time to replay a set of heterogeneous agent trajectories. Because it includes prompts and tool-use steps, it is closer to end-to-end automation time than coding-benchmark TPS, but remains tied to fixed scenarios and a specific runtime. Your agent may rank models differently if it uses other tools or longer documents.

• GGUF vs MLX: which model format should you download?
• What is Lightning MTP? Faster answers with oMLX 0.7.1rc1
• SpecPrefill: reducing the wait before the first token
• Prefill vs decode: should you look at TTFT or tok/s?
• Can you run the 74B Nemotron 3.5 Bolt model in 24 GB?

Where did Bolt beat Qwen3.8 in the published test?

The test used the same RTX Spark system in a 128 GB Surface Laptop Ultra, Windows 11 ARM x64, and llama.cpp. TPS is from AIPerf SPEED-Bench-Coding; replay time is from a fixed AA-AgentPerf Local task set.

Build a small comparison around your workflow

In the table, Bolt had about three times Qwen3.8's generation TPS and finished the fixed agent replay in about 42% less time. Qwen3.6 and Gemma4 had higher TPS and shorter replay durations than Bolt. For continuous token generation, look first at TPS; for multi-step tool automation, focus on replay duration.

Download each model from its official source and match the Qwen3.8-27B quantization and DFlash setup named in the table. Specify whether your Bolt setup includes IQ2_XXS and LiLiCorr. Record file sizes, runtime versions, and memory use for reproducibility. If drafters differ, you are comparing the full acceleration setups rather than only the base models.

Choose one representative task and give both models the same document, question, tools, and output limit. Start with a short exchange to verify operation and answer accuracy, then add real document length and repeated tool calls. Record time to first token, generation speed, end-to-end time, peak VRAM, and failures. If answers differ, score them with the same rubric to compare speed and quality together.

Which one to choose

For tasks with many steps and decisions based on tool results, prioritize reliable tool calls in your tool format and end-to-end replay time. Bolt's shorter AA-AgentPerf Local result than Qwen3.8-27B is useful for shortlisting, but may not apply if your tools differ. Use a sandbox for automation that edits files or retrieves web content.

For repeated short code completions or fixed-input generation, measure TPS and time to first token separately. Qwen3.6 and Gemma4 had higher TPS than Bolt in the published table. Repeat the test with the same input length and GPU setup to see which result suits your workload.

Frequently asked questions

What should you check first in this article?

This guide compares Nemotron Bolt and Qwen3.8-27B on token throughput and agent-task replay time in NVIDIA's RTX Spark test, then shows how to test them with your own prompts and tools.

How much memory does it need?

Download each model from its official source and match the Qwen3.8-27B quantization and DFlash setup named in the table. Specify whether your Bolt setup includes IQ2_XXS and LiLiCorr. Record file sizes, runtime versions, and memory use for reproducibility. If drafters differ, you are comparing the full acceleration setups rather than only the base models.

Key takeaways

1. Bolt uses llama.cpp with IQ2_XXS GGUF; Super VL uses vLLM with NVFP4 on DGX Spark.
2. Published acceptance length and tokens/s are different metrics with different conditions.

LiLiCorr selects candidates that fit together

A parallel draft model proposes next-token candidates at several positions at once. LiLiCorr scores the connections between candidates to choose a path for the target model to verify. [LiLiCorr method](https://research.nvidia.com/labs/nemotron/lilicorr/)

Speed gains vary with the drafter and target pairing, input style, and number of concurrent requests. Bolt runs with llama.cpp and Super VL with vLLM, so use the settings for each path.

• DGX Spark 128GB local LLM baseline
• Context length, RoPE and sliding-window attention
• GGUF vs MLX: which model format should you download?
• How to run Nemotron 3.5 Super VL on DGX Spark or RTX PRO 6000
• What is ReplaySSM? Faster decoding for Nemotron and Qwen servers

Bolt: connect the GGUF drafter in llama.cpp

For Bolt 74B-A9B, prepare both the target GGUF and the LiLiCorr drafter GGUF. Connect the llama.cpp server with `--spec-type draft-dflash` and `--spec-draft-model` to use the two files.

NVIDIA lists a 32 GB recipe for a 128 GB RTX Spark or a GeForce RTX with at least 48 GB of VRAM. It uses a 262,144 context, one concurrent sequence, and up to 7 draft tokens.

For a 64 GB RTX Spark or a GPU with at least 32 GB of VRAM, NVIDIA lists a 28 GB recipe with a 65,536 context, one sequence, up to 6 draft tokens, and a Q8_0 KV cache. [Bolt GGUF model card and recipes](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF/blob/main/README.md)

Download the target GGUF and LiLiCorr drafter GGUF. Replace the paths in the first code block with the actual file locations and run it in your shell. Once both variables are set, use the next command to pair the files. Apply the context, cache, and draft-token options from the model-card recipe for your GPU.

export BOLT_GGUF="/absolute/path/to/target.gguf"
export LILICORR_GGUF="/absolute/path/to/lilicorr-drafter.gguf"

llama-server -m "$BOLT_GGUF" \
  --spec-type draft-dflash \
  --spec-draft-model "$LILICORR_GGUF" \
  --host 127.0.0.1

Super VL: use vLLM LiLiCorr on DGX Spark

On DGX Spark (GB10), use NVIDIA's Super VL LiLiCorr profile. Run the NVFP4 target and a separate `...NVFP4-LiLiCorr` drafter inside the vLLM container.

The NVIDIA settings use one concurrent sequence, 7 draft tokens, 0.75 GPU memory utilization, and 8,192 batched tokens. The vLLM recipes list this DGX Spark LiLiCorr path separately from the native MTP path for RTX PRO 6000. [Super VL LiLiCorr model card](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4)

The target also has a native MTP path. MTP uses prediction heads inside the target, while LiLiCorr requires a separate drafter checkpoint; keep their speculative method and model arguments distinct. Start with a short text request and confirm the API model name, then add image input and your usual concurrency while recording memory use and completion time.

• Run Bolt 74B-A9B in llama.cpp

Read the workload alongside each metric

NVIDIA's study reports that LiLiCorr increased acceptance length by 9–19% and greedy decoding throughput by 4–13% over vanilla DFlash trained on the same data. The comparison tested models including Qwen3 4B and 8B on H100 GPUs.

The Super VL LiLiCorr card's SPEED-Bench reports acceptance lengths of 4.74 (temperature 0) and 4.36 (temperature 1), averaged across three seeds with draft length 7, 880 prompts, and 64 concurrent requests.

The 64 is the benchmark load. The DGX Spark serving recipe sets `--max-num-seqs 1`. Acceptance length means the average number of accepted tokens.

[Research results](https://research.nvidia.com/labs/nemotron/lilicorr/) · [Model card measurements](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4-LiLiCorr)

A separate published Bolt measurement used a 128GB Surface Laptop Ultra with RTX Spark, Windows 11 ARM, and llama.cpp. Bolt IQ2_XXS with LiLiCorr recorded 83.99 TPS on AIPerf SPEED-Bench-Coding and 600.80 seconds for AA-AgentPerf Local replay. These measurements apply to that Bolt setup and workload. [Bolt measurement conditions](https://research.nvidia.com/labs/nemotron/Nemotron-3.5-Bolt-24GB/)

Frequently asked questions

What should you check first in this article?

LiLiCorr is a speculative-decoding drafter that scores connections among candidate next tokens and selects a draft path for a target model. The required drafter files and settings vary by target and runtime. Bolt uses a llama.cpp GGUF pair; Super VL uses a DGX Spark vLLM pair, so this guide separates their files, commands, and published metrics.

How much memory does it need?

NVIDIA lists a 32 GB recipe for a 128 GB RTX Spark or a GeForce RTX with at least 48 GB of VRAM. It uses a 262,144 context, one concurrent sequence, and up to 7 draft tokens.

When is it better not to choose this?

The target also has a native MTP path. MTP uses prediction heads inside the target, while LiLiCorr requires a separate drafter checkpoint; keep their speculative method and model arguments distinct. Start with a short text request and confirm the API model name, then add image input and your usual concurrency while recording memory use and completion time.

Where can you check the conditions and sources?

For a 64 GB RTX Spark or a GPU with at least 32 GB of VRAM, NVIDIA lists a 28 GB recipe with a 65,536 context, one sequence, up to 6 draft tokens, and a Q8_0 KV cache. [Bolt GGUF model card and recipes](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF/blob/main/README.md)

Key takeaways

1. DGX Spark LiLiCorr and RTX PRO 6000 MTP are separate command paths.
2. The NVFP4 vLLM context allocations differ: 114,688 on Spark and 16,384 on PRO 6000.

What can you send and receive?

Super VL is a vision-language model that accepts text, images, and frames sampled from video. You can send a document table or screen image and receive a text response. The model card supports MP4 video up to 2 minutes. Image and video requests use context for visual tokens and frames, so start with a small input.

The model combines Mamba-2, MoE, and attention layers with a C-RADIO encoder and vision-language connector. The NVFP4 weight index is 75.01 GB; the vLLM recipe calculates a 91 GB minimum with a 20% allowance. That is a weight-loading baseline: context, batch size, and media-frame settings use additional memory. [Hardware-specific vLLM recipes](https://recipes.vllm.ai/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4)

• DGX Spark 128GB local LLM baseline
• MoE vs dense: why low active parameters still need memory
• Running local LLMs across two or more GPUs
• What Is LiLiCorr? Speculative decoding for Bolt and Super VL
• RTX Spark laptops: how do they differ from DGX Spark and MacBook Pro?

DGX Spark path: run with LiLiCorr drafting

To use NVIDIA's vLLM quick start on DGX Spark (GB10), first run the official `vllm/vllm-openai:v0.32.0` container or the listed nightly container. Enter the `vllm serve` command below in a terminal inside the container. It runs the NVFP4 target with a separate LiLiCorr drafter.

The settings are GPU utilization 0.75, 8,192 batched tokens, one concurrent sequence, and 7 draft tokens. [NVIDIA Super VL model card](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4) · [Official vLLM container image](https://hub.docker.com/r/vllm/vllm-openai/)

Start the server in the container's first shell and wait for model loading and kernel compilation. Leave the server running, open another shell in the same container, and run the check below. The API is ready when the model list includes `nemotron-3.5-super-vl`. Follow the model-card quick-start for the container and its detailed options.

export MODEL=nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4
export DRAFT_MODEL=nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4-LiLiCorr
vllm serve "$MODEL" \
  --host 127.0.0.1 \
  --served-model-name nemotron-3.5-super-vl \
  --max-num-seqs 1 \
  --trust-remote-code \
  --gpu-memory-utilization 0.75 \
  --max-num-batched-tokens 8192 \
  --speculative_config.model "$DRAFT_MODEL" \
  --speculative_config.num_speculative_tokens 7 \
  --speculative_config.method dflash \
  --mamba-backend flashinfer \
  --mamba-ssm-cache-dtype float16 \
  --enable-mamba-cache-stochastic-rounding \
  --mamba-cache-philox-rounds 5 \
  --enable-mamba-shared-prefix-checkpoint \
  --video-pruning-rate 0.5 \
  --media-io-kwargs '{"video": {"fps": 2, "num_frames": 256}}' \
  --block-size 64 \
  --prefix-match-unit 64 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser nemotron_v3

curl --fail http://127.0.0.1:8000/v1/models

RTX PRO 6000 path: NVFP4 with native MTP 5

The vLLM cookbook provides an NVFP4 profile for one RTX PRO 6000 Blackwell. Its settings are tensor parallel 1, pipeline parallel 1, data parallel 1, max model length 16,384, and up to 128 concurrent sequences.

This profile uses a target-native MTP draft of 5. It is a different path from DGX Spark, which specifies a separate LiLiCorr drafter. [Single RTX PRO 6000 profile and command](https://recipes.vllm.ai/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4)

The RTX PRO 6000 profile on the recipe page is labeled for a text workload. For image or video input, check for a profile that explicitly matches the GPU and media mode; a text-only command does not establish multimodal support. Begin with the profile’s sequence and context settings, then test with your own inputs.

vllm serve nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4 \
  --host 127.0.0.1 \
  --served-model-name nemotron-3.5-super \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --pipeline-parallel-size 1 \
  --data-parallel-size 1 \
  --max-model-len 16384 \
  --max-num-seqs 128 \
  --speculative_config.method mtp \
  --speculative_config.num_speculative_tokens 5 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser nemotron_v3

Match input type and limits when comparing

The DGX Spark LiLiCorr and RTX PRO 6000 native-MTP profiles here are separate recipes for different hardware, so comparing them also includes differences in device and context allocation. To isolate LiLiCorr versus MTP, measure both on the same hardware and runtime while holding target, input, sampling, context, and concurrency constant.

The 1M figure is the model’s maximum context. Published quick starts and hardware profiles use smaller allocations. Verify an image request with one small PNG, then add document pages. For video, start from the card’s 2 FPS and up-to-256-frame settings, adjusting clip length and resolution. [Super VL API and media examples](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4)

The table’s context and speculative settings are the values listed for each hardware profile in the vLLM cookbook.

Frequently asked questions

What should you check first in this article?

Nemotron 3.5 Super VL is a multimodal MoE model with 120B total and 12B active parameters. It accepts text, images, and video frames, then returns text. The published vLLM paths differ: DGX Spark uses LiLiCorr, while one RTX PRO 6000 recipe uses NVFP4 and native MTP with 5 draft tokens. This guide distinguishes their context allocations and input paths.

How much memory does it need?

The model combines Mamba-2, MoE, and attention layers with a C-RADIO encoder and vision-language connector. The NVFP4 weight index is 75.01 GB; the vLLM recipe calculates a 91 GB minimum with a 20% allowance. That is a weight-loading baseline: context, batch size, and media-frame settings use additional memory. [Hardware-specific vLLM recipes](https://recipes.vllm.ai/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4)

When is it better not to choose this?

The RTX PRO 6000 profile on the recipe page is labeled for a text workload. For image or video input, check for a profile that explicitly matches the GPU and media mode; a text-only command does not establish multimodal support. Begin with the profile’s sequence and context settings, then test with your own inputs.

Where can you check the conditions and sources?

The settings are GPU utilization 0.75, 8,192 batched tokens, one concurrent sequence, and 7 draft tokens. [NVIDIA Super VL model card](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4) · [Official vLLM container image](https://hub.docker.com/r/vllm/vllm-openai/)

Key takeaways

1. RTX Spark uses Windows on Arm and CUDA, DGX Spark uses DGX OS/Linux, and MacBook Pro uses macOS.
2. 128GB is the Surface top-SKU figure, not a guarantee the GPU can use all of it.

RTX Spark laptops and their top specifications

RTX Spark laptops are a platform with Windows on Arm, a Grace CPU, a Blackwell RTX GPU, and unified memory. Microsoft's maximum Surface Laptop Ultra configuration has a 15-inch display, up to 20 Grace CPU cores, up to 6,144 Blackwell GPU cores, and up to 128 GB of unified memory.

The maximum AI performance figure is a theoretical value using FP4 and sparsity. Measure actual processing time with the runtime, model, and input you plan to use. [Surface announcement and SKU specifications](https://blogs.windows.com/devices/2026/10/07/pre-order-our-most-powerful-surface-devices-ever/)

The GPU can use less than the full 128 GB of system unified memory. Windows, the browser, IDE, and image or video frames share the same memory pool with model weights and the KV cache.

Distinguish the total memory in a product SKU from GPU-addressable memory, and check the model recipe's minimum memory and context allocation as well.

• DGX Spark 128GB local LLM baseline
• Context length, RoPE and sliding-window attention
• GGUF vs MLX: which model format should you download?
• How much VRAM or unified memory does a local LLM need?

Compare DGX Spark and MacBook Pro

For these three platforms, OS and runtime compatibility often matter more than the shared ability to run local inference. RTX Spark is a candidate when you want CUDA and Windows creative apps on a laptop. DGX Spark suits a desktop system centered on Linux and NVIDIA serving recipes. If your workflow uses macOS apps and MLX or Core ML, start with MacBook Pro configurations.

MacBook Pro memory options vary by chip configuration. The NVIDIA and Apple figures describe each manufacturer’s platform, not comparative performance.

Apple lists MacBook Pro unified memory by chip and GPU SKU. Its 2026 M5 Pro/M5 Max specifications list up to 128GB for an M5 Max with a 40-core GPU and up to 64GB for M5 Pro configurations. For Mac, check the model formats supported by apps such as MLX or Core ML; for RTX Spark, check the Windows on Arm and CUDA path. [Apple MacBook Pro specifications](https://www.apple.com/macbook-pro/specs/)

Check Windows on Arm and app versions

Windows has Arm64 apps and x86 apps that run through Prism. NVIDIA and Microsoft announced CUDA and developer tool support for RTX Spark. Check each product's documentation to see whether your packages, plugins, and containers support the Windows on Arm CUDA path.

An x86 command-line app running does not establish that CUDA extensions or drivers will work the same way. [Windows development environment on RTX Spark](https://blogs.windows.com/windowsexperience/2026/05/31/introducing-a-powerful-new-chapter-for-windows-pcs-accelerated-by-nvidia-rtx-spark/)

RTX Spark laptop availability on October 16, 2026, may vary by region and SKU. Use the announced manufacturer list to identify candidates, then make a final choice from the exact product page’s memory, storage, display and thermal design, and Arm app compatibility. [NVIDIA COMPUTEX platform announcement](https://www.nvidia.com/en-us/geforce/news/computex-2026-nvidia-geforce-rtx-announcements/)

Before purchase: check the exact SKU and app support

Before buying, note the checkpoint size, required context, and whether you need image or video input, then compare them with GPU-addressable memory on the candidate SKU. Check Windows Arm64 and CUDA support in the model and app documentation. Test a real request after receiving the device, or earlier if you can use a demo unit.

The published Surface Laptop Ultra specifications are an example of the highest confirmed configuration among announced RTX Spark laptops. Other RTX Spark models may use the same platform but differ in memory options, display, battery, and cooling.

Compare memory and AI workload performance using the exact order SKU and manufacturer specifications of the product you plan to buy.

• Set up Super VL on DGX Spark or RTX PRO 6000
• Bolt memory tiers and GPU recipes

After purchase: test with your model and workload

NVIDIA tested Bolt IQ2_XXS and LiLiCorr on a 128 GB Surface Laptop Ultra using Windows 11 ARM and llama.cpp. The AIPerf SPEED-Bench-Coding result was 83.99 TPS, and AA-AgentPerf Local replay time in the same test was 600.80 seconds.

This is an example of an actual model and runtime combination among published RTX Spark laptop results. Compare it separately from measurements on DGX Spark or MacBook Pro. [Bolt test conditions](https://research.nvidia.com/labs/nemotron/Nemotron-3.5-Bolt-24GB/)

After receiving the device, prepare the checkpoint you plan to use and repeat a short request to separate first-load time from subsequent responses. Then open your usual apps and increase context, image resolution, or video frames while recording GPU memory, completion time, fan noise, and power state. For a useful MacBook Pro comparison, match model, quantization, context, and request.

Frequently asked questions

What should you check first in this article?

RTX Spark laptops are a new Windows on Arm platform with CUDA-capable Blackwell GPUs and shared CPU/GPU memory. DGX Spark is a Linux desktop AI system, while MacBook Pro uses macOS and the Apple Silicon ecosystem, including MLX and Core ML. Compare them by GPU-addressable memory on the exact SKU and by the apps and model recipes you need.

How much memory does it need?

RTX Spark laptops are a platform with Windows on Arm, a Grace CPU, a Blackwell RTX GPU, and unified memory. Microsoft's maximum Surface Laptop Ultra configuration has a 15-inch display, up to 20 Grace CPU cores, up to 6,144 Blackwell GPU cores, and up to 128 GB of unified memory.

Key takeaways

1. It activates 25B of its 560B parameters per token.
2. Official SGLang examples use 8×H200 or 4×GB300 servers.

What Ling 3.1 Flash does

Ling 3.1 Flash generates answers from text prompts for coding and agent tasks. It has 560B total parameters, activates 25B per token, and supports up to a 1M-token context. The official FP8 checkpoint file listing totals about 574 GB. [Model card and files](https://huggingface.co/inclusionAI/Ling-3.1-flash/tree/main)

Context length is the maximum input a model can process at once. Longer documents also require more memory for the KV cache and inference engine. Check the available GPU memory and document length, then set the context to fit the task.

• Context length, RoPE and sliding-window attention
• MoE vs dense: why low active parameters still need memory
• Running local LLMs across two or more GPUs
• A 125B model on a 12GB GPU? How Strata runs it

Plan around the official server recipes and 574 GB checkpoint

The official SGLang quick start uses TP8 and EP8 across eight H200 GPUs, or TP4 and EP4 across four GB300 GPUs. The model files total about 574 GB; inference also needs memory for the KV cache and runtime, depending on context length. Check storage and GPU capacity, then choose a context length for your requests. [SGLang setup](https://huggingface.co/inclusionAI/Ling-3.1-flash)

Tensor parallelism (TP) splits computation across GPUs, while expert parallelism (EP) distributes MoE experts. Use the table to match these settings to the official server recipes, then allow for KV-cache and runtime memory at your actual context length.

Start the server and send a first request

First, pull the SGLang container as described in the model card. The command below targets 8×H200 and uses `--host 127.0.0.1` instead of the official example's `0.0.0.0`, keeping access on the same machine. Add network controls and authentication before allowing other computers to connect. Mount the Hugging Face cache on host storage so the roughly 574 GB checkpoint does not need to be downloaded again after each container removal. [Cache locations](https://huggingface.co/docs/huggingface_hub/en/package_reference/environment_variables)

docker pull lmsysorg/sglang:dev-ling-3.1-flash
mkdir -p "$HOME/.cache/huggingface"

docker run --gpus all --network host --ipc=host --rm \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  lmsysorg/sglang:dev-ling-3.1-flash \
  python3 -m sglang.launch_server \
    --model-path inclusionAI/Ling-3.1-flash \
    --tp 8 --ep 8 \
    --speculative-algorithm NEXTN \
    --host 127.0.0.1 --port 30000

When the server is ready, send a request to `http://127.0.0.1:30000/v1/chat/completions`. The OpenAI-compatible API uses the model ID `Ling-3.1-flash`; refer to the model card examples for `enable_thinking`, temperature, top-k, and top-p.

First send a short text prompt and confirm that a response arrives, then test your document length and tool calls. If there is no response, inspect the container logs for model loading errors or insufficient GPU memory. [Official setup](https://huggingface.co/inclusionAI/Ling-3.1-flash)

curl -s http://127.0.0.1:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Ling-3.1-flash","messages":[{"role":"user","content":"짧은 요약을 한 문장으로 써 주세요."}],"chat_template_kwargs":{"enable_thinking":true},"temperature":1.0,"top_k":20,"top_p":0.95,"stream":true}'

• Local AI serving engine comparison

What changed in the October 19 checkpoint

A checkpoint dated October 19 (Ling-3.1-flash-1019) is up. The architecture and window are unchanged and the post-training differs; 560B total with 25B active, a 1M window and the 8×H200 recipe all stay as they were.

The gains look like this: the Artificial Analysis intelligence index moves from 41 to 44, AA-Briefcase from about 1400 to 1524 Elo, AutomationBench from 62 to 68 percent, CritPt from 18 to 26 and SciCode from 54 to 57.

Dated checkpoints get mixed up with their predecessor: check which revision --model-path resolves to and which revision sits in your cache.

Decide whether it fits your workload

To check whether you need long context or coding and agent features, choose a representative task and repeat it with the same document and output length. Record time to first token and total completion time to see whether 1M context or reasoning options help your work. Published benchmark throughput applies to its test hardware and conditions.

Before providing the API to teammates, decide who can access it and how long request and server logs will be retained. For internal documents, also check the permitted scope of use.

Frequently asked questions

What should you check first in this article?

Ling 3.1 Flash is an MoE language model with 560B total parameters and 25B active per token. Official SGLang recipes run its 574 GB FP8 checkpoint on 8×H200 or 4×GB300 servers.

How much memory does it need?

Context length is the maximum input a model can process at once. Longer documents also require more memory for the KV cache and inference engine. Check the available GPU memory and document length, then set the context to fit the task.

Where can you check the conditions and sources?

Ling 3.1 Flash generates answers from text prompts for coding and agent tasks. It has 560B total parameters, activates 25B per token, and supports up to a 1M-token context. The official FP8 checkpoint file listing totals about 574 GB. [Model card and files](https://huggingface.co/inclusionAI/Ling-3.1-flash/tree/main)

Key takeaways

1. It accepts text, image, video, and audio and returns text.
2. Apple's measurements used two or four M5 Ultra systems at 8,192-token context.

What does this model take in and return?

Muse Spark Open is a multimodal language model that accepts text, images, video, and audio and returns text. It has 816B total parameters, 42B active per token, and supports up to a 1M-token context at inference. It targets coding and agent work, but local deployment starts with checking whether you have enough memory for the weights and runtime headroom. [Meta model card](https://huggingface.co/meta-models/Muse-Spark-Open-816B-A42B)

Checkpoint size varies by precision and quantization. Meta's model card lists at least 700 GB VRAM for NVFP4 GPU deployment and 2 TB for BF16, with additional memory for KV cache and activations. These figures describe large server deployment requirements, not consumer-GPU compatibility.

• Mac Studio M5 Ultra 256GB baseline
• Context length, RoPE and sliding-window attention
• Prefill vs decode: should you look at TTFT or tok/s?
• GGUF vs MLX: which model format should you download?
• Run local AI on a Mac with oMLX: what is new in 0.7.1rc1?

Choose between one 512 GB Mac and multiple M5 Ultra systems

The MLX distributed run described by Meta and Apple uses two M5 Ultra Mac Studios, each with 512 GB of unified memory. Separately, oMLX 0.7.1rc1 reports running a 370 GB oQ3e checkpoint on a single M3 Ultra Mac with 512 GB of unified memory.

The quantization and inference engines differ, so do not compare the performance figures directly. [Apple's MLX setup](https://machinelearning.apple.com/research/mlx-distributed-inference) · [oMLX 0.7.1rc1](https://github.com/jundot/omlx/releases/tag/v0.7.1rc1)

Distributed inference requires macOS 26.2 or later and direct Thunderbolt 5 RDMA links between the Macs. First list each Mac, its Thunderbolt IP, and RDMA device in the hostfile. Meta's mlx.distributed_config instructions explain how to create hostfile.json; the existing JACCL guide also covers connection setup and validation. Run the server command below on the primary Mac where that file is saved.

mlx.launch --hostfile hostfile.json --backend jaccl \
  --env MLX_METAL_FAST_SYNCH=1 -- \
  mlx_lm.server \
    --model meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \
    --port 8080

The table shows Apple's distributed NVFP4 MLX results on M5 Ultra systems at an 8,192-token context. The single-Mac oQ3e path uses a different model format and runtime, so do not compare the figures directly; test each path with your own inputs.

• JACCL connection and hostfile setup
• Meta Muse Spark Mac setup and hostfile

Start with oMLX on one 512 GB Mac

For one Mac, use the `mlx-community/Muse-Spark-Open-816B-A42B-oQ3e` variant in oMLX 0.7.1rc1, separate from Meta’s distributed NVFP4 path. The model files total 370 GB, and the release notes report loading this checkpoint on an M3 Ultra Mac with 512 GB. oMLX is a release candidate, not the stable release. [oMLX release notes](https://github.com/jundot/omlx/releases/tag/v0.7.1rc1)

In the oMLX app, open Models > Downloader and select Muse Spark Open oQ3e from the Recommended tab. Check that the disk has room for the 370 GB files before downloading, then load the model from the list. Start with a short text prompt and modest context; confirm it answers without a memory warning before adding images or longer context. [Model recommendations](https://github.com/jundot/omlx/releases/tag/v0.7.1rc1)

For this M3 Ultra 512 GB oQ3e run, oMLX 0.7.1rc1 reports 225 tok/s prefill and 21.3 tok/s decode. The release notes do not include the full prompt and output conditions, so do not compare these directly with Apple’s 8,192-token M5 Ultra table. Repeat your own documents and record model loading, prefill, and decode separately to evaluate your workload.

If the model is missing or fails to load, confirm that you selected the `oQ3e` variant and that its download completed. If the files are present but a memory warning appears, unload other models and retry with a shorter context. Check the first load error in oMLX logs and verify that your version lists support for this model.

• Muse support and measurements in oMLX 0.7.1rc1

Check modality support and license terms

Meta’s model card lists text, image, video, and audio inputs with text output. Image, video, and audio request formats vary by MLX adapter or runtime, so follow the examples in the documentation for your chosen setup.

The weights are released under the Muse Model License. Before using them in a service or redistributing the model, check the applicable terms in the [full license](https://huggingface.co/meta-models/Muse-Spark-Open-816B-A42B/blob/main/LICENSE).

Frequently asked questions

What should you check first in this article?

Muse Spark Open is a multimodal model with 816B total and 42B active parameters. Meta and Apple's NVFP4 MLX path uses two M5 Ultra Mac Studios; oMLX 0.7.1rc1 reports a 370 GB oQ3e run on one 512 GB M3 Ultra Mac.

How much memory does it need?

Checkpoint size varies by precision and quantization. Meta's model card lists at least 700 GB VRAM for NVFP4 GPU deployment and 2 TB for BF16, with additional memory for KV cache and activations. These figures describe large server deployment requirements, not consumer-GPU compatibility.

When is it better not to choose this?

The quantization and inference engines differ, so do not compare the performance figures directly. [Apple's MLX setup](https://machinelearning.apple.com/research/mlx-distributed-inference) · [oMLX 0.7.1rc1](https://github.com/jundot/omlx/releases/tag/v0.7.1rc1)

Where can you check the conditions and sources?

Muse Spark Open is a multimodal language model that accepts text, images, video, and audio and returns text. It has 816B total parameters, 42B active per token, and supports up to a 1M-token context at inference. It targets coding and agent work, but local deployment starts with checking whether you have enough memory for the weights and runtime headroom. [Meta model card](https://huggingface.co/meta-models/Muse-Spark-Open-816B-A42B)

Key takeaways

1. Compare images, video, audio, and text in one embedding space.
2. Use SearchQuery for queries and Document for corpus items.

Searching beyond text documents

EmbeddingGemma 2 turns text, code, images, video, and audio into numerical embeddings for search. It retrieves documents, photos, or recordings related to a query; a separate LLM can summarize the results. All inputs share a 768-dimensional embedding space, so text can retrieve photos and descriptions can be compared with audio. [Google model card](https://huggingface.co/google/embeddinggemma-2)

An embedding is an array of numbers representing the meaning of the original content. The model is trained so that semantically similar queries and items sit near one another. For example, a question about refund timing can retrieve a policy saying refunds take up to five business days. Korean retrieval quality depends on wording and domain, so evaluate it on your own materials.

• Run local AI on a Mac with oMLX: what is new in 0.7.1rc1?
• Jev-Omni: multimodal decisions on local GPUs
• Which local LLM app or serving engine should you start with?
• DeepSeek Search and Fetch API: giving a local LLM web results
• When local RAG answer is wrong: Search, chunk, model diagnosis order
• Local LLM·RAG quality evaluation: Verifying changed settings with DeepEval

Split documents and start with text retrieval

Split content into readable units, such as one FAQ entry or one manual section, instead of embedding an entire document as a single item. Store the document ID, title, text, and source location so a result can be opened and checked. Choose chunk boundaries that do not separate tables or code from their meaning.

The installation commands below are for bash/zsh on macOS or Linux. In Windows PowerShell, create a virtual environment and activate it with `.\.venv\Scripts\Activate.ps1`. Install the packages and run the example in that environment.

For text-only search, you can disable the image and audio encoders and load the 270M text configuration. Use the `SearchQuery` prompt for queries and `Document` for documents to be searched. Format a titled document as `title: title | text: content`. See the [official developer guide](https://developers.googleblog.com/embeddinggemma-2-the-developer-guide/).

python -m venv .venv
source .venv/bin/activate
python -m pip install -U sentence-transformers transformers

from sentence_transformers import SentenceTransformer

model = SentenceTransformer(
    "google/embeddinggemma-2",
    config_kwargs={"vision_config": None, "audio_config": None},
)
documents = [
    "title: 환불 처리 | text: 결제 취소 후 영업일 기준 5일 이내에 환불됩니다.",
    "title: 배송 변경 | text: 주문 준비가 시작되기 전까지 배송지를 바꿀 수 있습니다.",
]
query = "환불은 며칠 안에 되나요?"
doc_vectors = model.encode(documents, prompt_name="Document", normalize_embeddings=True)
query_vector = model.encode(query, prompt_name="SearchQuery", normalize_embeddings=True)
scores = model.similarity(query_vector, doc_vectors)[0]
for index in scores.argsort(descending=True).tolist():
    print(f"{scores[index]:.3f}\t{documents[index]}")

Save the code as `search.py` and run it in the same virtual environment. The first search works when it prints both documents and ranks the refund policy first. Check whether expected items appear near the top using query-document pairs from your own data, then adjust chunking and titles.

python search.py

Add multimodal queries to a text index

An existing index can be reused for image or audio queries only when vectors are compatible with the same EmbeddingGemma 2 checkpoint, task setup, and dimensions. This does not mean vectors from another model such as Qwen or BGE can be mixed into it.

For photos or recordings, keep source paths, captions, and recording dates as metadata. Vectors do not replace source files; retain identifiers that let reviewers open the result.

Search photos and audio with text queries

First, place `photos/lake.jpg`, `photos/street.jpg`, and `recordings/meeting.wav` in the working folder, or change the paths below to your files. Install the multimodal encoders and media-processing libraries. Pass photos to `encode()` with the `image` key and recordings with the `audio` key. The example compares a text description with two image embeddings and ranks them by similarity.

python -m pip install -U sentence-transformers transformers pillow soundfile torchcodec

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("google/embeddinggemma-2")
query = model.encode("눈 덮인 산과 호수가 보이는 풍경", prompt_name="SearchQuery")
photos = [{"image": "photos/lake.jpg"}, {"image": "photos/street.jpg"}]
photo_vectors = model.encode(photos)
scores = model.similarity(query, photo_vectors)[0]
print(scores.argsort(descending=True).tolist())

For audio, pass the file path as `model.encode({"audio": "recordings/meeting.wav"})` and compare it with a text query. The example below ranks two questions by similarity to the meeting recording. Audio is resampled to 16 kHz internally. Video defaults to one sampled frame per second; adjust the frame count for denser scene retrieval. [Google multimodal examples](https://ai.google.dev/gemma/docs/embeddinggemma/multimodal-embeddinggemma-with-sentence-transformers)

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("google/embeddinggemma-2")
audio = model.encode({"audio": "recordings/meeting.wav"})
queries = [
    "회의 일정을 언제로 정했나요?",
    "환불은 며칠 안에 되나요?",
]
query_vectors = model.encode(queries, prompt_name="SearchQuery")
scores = model.similarity(audio, query_vectors)[0]
for index in scores.argsort(descending=True).tolist():
    print(f"{scores[index]:.3f}\t{queries[index]}")

Distinguish text-only and full multimodal loading

The official card lists a 270M text-only selective configuration and a 740M configuration with text, vision, and audio encoders. Enable only the modalities you need, and record the library options and actual memory use.

Do not treat the full-model specification from the earlier introduction as the memory requirement for a text-only path. The first run downloads weights; use the documented text-only configuration for text retrieval comparisons.

Choose an embedding dimension and review the license

EmbeddingGemma 2 returns 768-dimensional vectors by default. Sentence Transformers can truncate them to 512, 256, or 128 dimensions to reduce index storage, but retrieval quality still needs checking. Google reports quality close to the default down to 256 dimensions and recommends 128 dimensions for text-only workloads. Encode queries and documents at the same dimension, and normalize after truncation.

When changing dimensions, re-embed the stored documents and rebuild the index. Queries and corpus items must use the same dimension. The model uses the Apache 2.0 license, which permits use, modification, and redistribution subject to its terms; retain the license and copyright notices when redistributing. [Model card and license](https://huggingface.co/google/embeddinggemma-2)

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("google/embeddinggemma-2")
query = "환불은 며칠 안에 되나요?"
document = "title: 환불 처리 | text: 결제 취소 후 영업일 기준 5일 이내에 환불됩니다."
query_vector = model.encode(
    query, prompt_name="SearchQuery", truncate_dim=256, normalize_embeddings=True
)
document_vector = model.encode(
    document, prompt_name="Document", truncate_dim=256, normalize_embeddings=True
)
print(model.similarity(query_vector, document_vector))

Match query instructions and document formatting

For text search, prefix queries with `task: search result | query: ...` and format documents as `title: ... | text: ...`. Do not add text prefixes to image, video, or audio inputs. Use `title: none` when no document title exists.

The official Sentence Transformers example uses `SearchQuery` and `Document` task names to apply the corresponding prompts. Choose either the library prompt or manual prefixes; do not apply both and duplicate the instruction.

Troubleshoot runtime and memory errors

If you see `ModuleNotFoundError`, check that the Python running `search.py` is the same virtual environment where you installed the packages. Run `python -m pip show sentence-transformers` to confirm; activate the environment and install again if needed.

If an image or audio file cannot be opened, check its relative path from the current working directory. Try the absolute file path to separate a path issue from a model issue. If memory runs out, use the text-only configuration or close other apps and models before retrying.

Update a folder index incrementally and measure its size

Store file IDs, hashes, and modification times so only changed documents and media are re-embedded, and remove vectors for deleted files. For photo or audio folders, link captions to source files so the index points back to the right item.

Truncating to 128, 256, 512, or 768 dimensions can reduce raw vector storage but may affect quality. Compare ranks on the same queries and base storage decisions on the full database index and metadata, not raw arrays alone.

Evaluate each modality and verify sources separately

Create separate query sets for the directions you will use, such as text-to-image, image-to-text, or audio-to-text. Record whether the gold file ID appears in the top k. Success in one direction does not establish performance for every modality pair.

Open the retrieved documents, photos, and recordings to confirm they contain the evidence. Separate retrieval misses from parsing or caption errors. This guide reports no run on reader data; measure accuracy, latency, and memory on your own files.

Frequently asked questions

What should you check first in this article?

EmbeddingGemma 2 maps text, code, images, video, and audio into a shared 768-dimensional embedding space. It creates vectors for retrieval; a separate LLM can summarize the results.

How much memory does it need?

The official card lists a 270M text-only selective configuration and a 740M configuration with text, vision, and audio encoders. Enable only the modalities you need, and record the library options and actual memory use.

Where can you check the conditions and sources?

EmbeddingGemma 2 turns text, code, images, video, and audio into numerical embeddings for search. It retrieves documents, photos, or recordings related to a query; a separate LLM can summarize the results. All inputs share a 768-dimensional embedding space, so text can retrieve photos and descriptions can be compared with audio. [Google model card](https://huggingface.co/google/embeddinggemma-2)

Key takeaways

1. Accumulate recent inputs and periodically fold them into the checkpoint to reduce state writes.
2. Retain accepted draft inputs to handle speculative-decoding rollback.
3. Kernel and full-step gains differed in large-batch standard decoding.

Buffer recent inputs instead of writing the state every step

An SSM layer incorporates new information into its previous state when processing a token. With concurrent requests, reading and writing a state for every conversation creates more memory traffic. ReplaySSM keeps a checkpoint and buffers recent inputs. It computes outputs from both, then folds the buffered inputs into the checkpoint when the buffer fills.

Hybrid Nemotron and Qwen models combine SSM and attention layers. ReplaySSM reduces the cost of reading and writing the SSM state. It can coexist with weight quantization, while the file-size setting is selected separately.

• ReplaySSM paper and algorithm
• GGUF vs MLX: which model format should you download?
• Running local LLMs across two or more GPUs
• Qwen3.8-Flash-Next PLE Offload: RAM and SSD Requirements and Setup
• Prefill vs decode: should you look at TTFT or tok/s?

Rollback after verifying MTP drafts becomes cheaper

When MTP or a separate drafter proposes several tokens, the target model accepts a prefix. After discarding the rest, its conversation state must match the accepted tokens. ReplaySSM adjusts the end of the recent-input buffer to retain that prefix, reducing the need to save a large state snapshot for every draft token.

This optimization changes state computation and storage inside the engine. LiLiCorr instead improves the proposals likely to be accepted. Recording draft acceptance length separately from state-verification cost helps identify which server change was useful.

• Understand LiLiCorr drafters

Results differ between one request and large batches

NVIDIA and Princeton researchers compared four Mamba-2 and GDN models in vLLM. The table covers standard decoding without speculative acceleration. Separating kernel speedup from speedup of the entire decode step makes the server-level effect clearer.

4B models: one H100; 120B-class: one B300 · no tensor parallelism · FP32 SSM state · window L=16 · tuned vLLM baseline

Published tests showed no standard-decoding speedup at batch size 1 and gains at batch size 256. On a personal PC, measure one request’s TTFT and generation speed first. For a team server, reproduce actual concurrency and compare per-request latency and total throughput.

• Models and hardware in NVIDIA’s tests

What to record before adopting it on a server

ReplaySSM is integrated into TensorRT-LLM, vLLM and FlashInfer. Check that your version supports the model’s SSM path and start on a test server. Fix the checkpoint, input and output lengths, state precision and drafter setting, then record the current engine’s baseline. After changing the engine, compare TTFT, completion time, peak memory and failures using the same requests.

If your goal is a faster personal PC, first compare supported MTP, DFlash or more efficient quantization for the model. For a server shared by several agents, add ReplaySSM to the candidates and evaluate request completion during busy periods before switching.

• ReplaySSM implementation and reproduction materials
• Lightning MTP on Mac

Frequently asked questions

What should you check first in this article?

ReplaySSM is an inference optimization that changes how SSM layers retain past input. It stores a checkpoint separately from recent inputs, reducing full-state writes at each token. It is relevant to hybrid-model servers handling concurrent requests.

How much memory does it need?

ReplaySSM is integrated into TensorRT-LLM, vLLM and FlashInfer. Check that your version supports the model’s SSM path and start on a test server. Fix the checkpoint, input and output lengths, state precision and drafter setting, then record the current engine’s baseline. After changing the engine, compare TTFT, completion time, peak memory and failures using the same requests.

Key takeaways

1. 24B total and 2.45B active: large weights, light per-token compute.
2. Both an official GGUF (Q8_0) and an MLX 4-bit build are published.
3. The 184 tok/s on the model card was measured on an Apple M5 Max.

The file is 24B; the computation is 2.45B

LFM2.5-24B-A2B is a text reasoning model that takes a question and produces an answer. The 24B in the name is the total parameter count the model carries; A2B means roughly 2B parameters are active for each token. The model card states 2.45B active parameters. Of its 40 layers, 30 use short convolution blocks and 10 use standard attention, and each MoE block selects 4 experts from 64.

What makes this practical is that memory and speed move separately. Loading the model needs space for 24B of weights, but token throughput stays close to the compute of 2.45B active parameters. Compared with a dense 24B model it should produce more tokens from the same memory, while a 7B-sized budget cannot hold it at all. That is why both numbers matter.

If you are looking for a local LLM that supports Korean, separate the language list from actual Korean quality. This model lists Korean among 16 languages, and it also defines the scope of free use.

• MoE vs dense: why low active parameters still need memory
• GGUF vs MLX: which model format should you download?
• KRAFTON Raon-1 mini local LLM: a Korean 30B MoE on one RTX 5090 32GB
• Flint local LLM: can an open 40B coding agent run on a personal PC?

It supports 16 languages, including Korean

The model card lists English, Arabic, Chinese, French, German, Italian, Japanese, Korean, Portuguese, Spanish, Vietnamese, Thai, Indonesian, Hindi, Russian and Polish. Korean appearing in that list matters for Korean users. The list means the language was in training data, though, not that Korean quality matches a 24B model built specifically for Korean.

Choose the weights that match your hardware

The official repositories publish the same model in several formats: the native checkpoint for Transformers, vLLM and SGLang, block-wise FP8, GGUF, ONNX, and a 4-bit MLX build for Apple Silicon. Choose GGUF to run it in llama.cpp or Ollama, and the 4-bit MLX file when you use MLX-based tools on an M-series Mac. The two cannot be swapped.

File sizes differ sharply by format. The 4-bit variants are smaller than the original weights, and quantization changes both answer quality and speed. Note which file you used for the first run so you can keep conditions equal when comparing formats later.

Separate the vendor's measurement from your own speed

The model card states it runs up to 1.7 times faster than comparable MoEs, reaching 184 tok/s on an Apple M5 Max and up to 21K output tok/s on a single NVIDIA B300. Those figures come from conditions the vendor chose. To read them against your own machine, first match the hardware, the quantization file, the context length, and whether thinking tokens counted toward the output.

Because this is a reasoning model, thinking tokens appear before any answer. To compare perceived latency, time the arrival of the first answer and the completion separately. Whether thinking tokens count in the output changes the second number substantially.

Built for agents, not for knowledge questions

The model card recommends it for agents that reason and call tools, not for knowledge-heavy tasks on its own, and suggests pairing it with retrieval or web search. Work that already has source documents, such as meeting summaries or internal document search, plays to its strengths when combined with RAG.

Public benchmarks appear in both the vendor blog and the model card. When comparing against rival scores in the same table, check that the evaluation conditions match. Tool-use benchmarks shift with prompts and runtime, so trying your own work sentences once is more reliable.

Check the licence and runtime conditions

The licence is not a standard open-source one but Liquid's own LFM Open License v1.0. Personal and research use is unrestricted, while commercial use is conditional. Commercial use is permitted while annual revenue stays below 10 million US dollars, and above that threshold you need separate permission. The licence defines this as the Threshold and conditions the commercial grant on not exceeding it.

• Liquid AI model card
• LFM Open License v1.0
• MLX 4-bit files

Frequently asked questions

What should you check first in this article?

LFM2.5-24B-A2B is a reasoning model Liquid AI released on 12 October 2026. Its mixture-of-experts structure routes each token through 4 of 64 experts, so the weights occupy 24B while a single token costs roughly 2.45B of computation. It supports 16 languages including Korean, and the official GGUF and MLX 4-bit files mean both Macs and NVIDIA PCs can run it. The 184 tok/s in the model card was measured on an Apple M5 Max, so do not read it as your machine's speed.

How much memory does it need?

What makes this practical is that memory and speed move separately. Loading the model needs space for 24B of weights, but token throughput stays close to the compute of 2.45B active parameters. Compared with a dense 24B model it should produce more tokens from the same memory, while a 7B-sized budget cannot hold it at all. That is why both numbers matter.

Where can you check the conditions and sources?

The model card states it runs up to 1.7 times faster than comparable MoEs, reaching 184 tok/s on an Apple M5 Max and up to 21K output tok/s on a single NVIDIA B300. Those figures come from conditions the vendor chose. To read them against your own machine, first match the hardware, the quantization file, the context length, and whether thinking tokens counted toward the output.

Key takeaways

1. 39.8B total, about 2.5B active, and a 262,144-token context.
2. The official runtime assumes one B300 or H200 GPU.
3. GGUF builds are separate and do not match server performance.

An agent model that inspects a codebase itself

Flint is a coding-agent model that reads a repository, fixes bugs, implements features and writes tests. It handles text input and output, reasoning, and multi-turn tool use. Its architecture is a decoder-only MoE Transformer: of 49 layers, 2 are dense and 47 are MoE layers. Each MoE layer selects 6 of 256 experts, plus one shared expert that always runs.

Attention is split: 35 layers use a short 513-token window and 14 layers use full attention. The short-window layers hold a smaller KV cache, which reduces the memory cost of long conversations. The 262,144-token context counts prompt plus generated output.

To use an open coding agent on a personal PC, check the runtime before the model scores. This article follows that order.

• Context length, RoPE and sliding-window attention
• MoE vs dense: why low active parameters still need memory
• GGUF vs MLX: which model format should you download?
• Run Xing4.0 Locally: Install the Coding Model and Plan GPU Memory
• LFM2.5-24B-A2B local LLM: a Korean-capable 24B MoE on your own PC

The official runtime is a server GPU

The vendor's documented path is SGLang, not GGUF or llama.cpp. You download about 80 GB of weights, build the SGLang image, and start a server on one B300 or H200 GPU. The address binds to 127.0.0.1, and /health_generate reports readiness. The model card states that Transformers modelling code is not yet included.

These instructions were not written with a personal PC in mind. B300 and H200 are data-centre GPUs whose memory configurations differ sharply from what an individual buys. On personal hardware you must use the separately published quantized GGUF, which runs on a different engine from the one the vendor measured.

Read the measurement conditions behind the scores

The vendor reports 54.9% on Terminal-Bench 2.1, 89.7% on SWE-bench Pro v2 and 74.2% on SWE-bench Verified. The model card also states the evaluation conditions: temperature 1.0, top-p 0.95 and top-k 20, with Terminus 2 for Terminal-Bench and the mini-swe-agent harness for SWE-bench. Agentic scores average pass@1 over at least five attempts.

The model card also states the limits of comparison. Evaluation setups vary by source, and among the tabulated scores the ones Recursive measured itself are marked. Flint's Terminal-Bench score applies an adjustment that counts solutions obtained through web retrieval as failures, while developer-reported scores may omit that adjustment. Keep that difference in mind when reading two numbers side by side.

On personal hardware, check support first

Two things come first on a personal PC. One is whether your runtime supports this architecture. Flint uses sliding-window attention across 35 layers, learned attention sinks and per-head RMSNorm, which differ from a typical llama-family model. Unless the runtime's documentation confirms these are implemented, the file may load poorly or not at all.

The other is memory. Quantizing 39.8B of weights to 4 bits still needs roughly a quarter of the original size. Using the full 262K context adds a KV cache even though only the selected experts compute. Start with short code reviews and increase context gradually to see how much headroom remains.

Consider the agent harness and permissions together

A coding agent's score does not come from the model alone. Which harness reads files, runs commands and retries failures is part of the result, and the vendor states it used the mini-swe-agent harness. To expect similar scores in your setup, match both the harness and the tool permissions.

An agent can run shell commands. Connected to a repository without a sandbox or allowlist, it can delete or transmit files by accident. Even in a personal repository, check for sensitive keys first and narrow the execution permissions.

Check the licence and redistribution terms

Flint is released under the Apache 2.0 licence. Modification, redistribution and commercial use are permitted with no revenue threshold, but Apache 2.0 also requires keeping copyright notices, marking changes and honouring the patent clause. When embedding it in a team tool or redistributing it, keep the NOTICE and licence files with it.

• Recursive Flint model card
• Flint GGUF quantized files
• Recursive launch post

Frequently asked questions

What should you check first in this article?

Flint is a mixture-of-experts model for coding agents published by Recursive in October 2026. It has 39.8B total parameters, about 2.5B active per token, and a 262,144-token context. Recursive reports 54.9% on Terminal-Bench 2.1, 89.7% on SWE-bench Pro v2 and 74.2% on SWE-bench Verified. Its official instructions assume a single B300 or H200 GPU, and Transformers modelling code is not included yet. Running it on a personal PC needs separately published GGUF weights and a llama.cpp-class runtime, whose performance may differ from the vendor figures.

How much memory does it need?

These instructions were not written with a personal PC in mind. B300 and H200 are data-centre GPUs whose memory configurations differ sharply from what an individual buys. On personal hardware you must use the separately published quantized GGUF, which runs on a different engine from the one the vendor measured.

Where can you check the conditions and sources?

The vendor reports 54.9% on Terminal-Bench 2.1, 89.7% on SWE-bench Pro v2 and 74.2% on SWE-bench Verified. The model card also states the evaluation conditions: temperature 1.0, top-p 0.95 and top-k 20, with Terminus 2 for Terminal-Bench and the mini-swe-agent harness for SWE-bench. Agentic scores average pass@1 over at least five attempts.

Key takeaways

1. 30B total, 4B active, with a 262,144-token context.
2. The vendor states NVFP4 runs on a single RTX 5090 with 32GB.
3. Hugging Face access approval is required, and this is a preview, not the final model.

Trained from scratch with Korean in mind

Raon-1 mini is a 30B-A4B MoE language model that KRAFTON AI designed and trained from scratch. The 30B is the total parameter count and A4B means about 4B active per token. The model card describes strong performance in Korean conversation and agentic tasks. It is trained in-house rather than fine-tuned from another open model.

Its training languages are Korean, English, Japanese, Chinese, Spanish, French and Thai. The model card describes strong Korean and solid English. For Korean users, Korean appearing among the training languages matters by itself. How much training weight each language received is not published, so verify Korean quality directly.

This model matters for Korean users because it is a Korean-built model with Korean in its training data. If you are looking for a Korean local LLM, a single RTX 5090 with 32GB is enough to try it first.

• RTX 5090 32GB memory, speed and price
• Context length, RoPE and sliding-window attention
• MoE vs dense: why low active parameters still need memory
• GGUF vs MLX: which model format should you download?
• LFM2.5-24B-A2B local LLM: a Korean-capable 24B MoE on your own PC

A changed attention layout reduces the cost of long context

Of its 25 layers, 18 use Raon Delta Attention and 7 use MLA. The linear-attention RDA layers hold a fixed-size state, so the KV cache does not keep growing as input lengthens. Only the 7 MLA layers keep a cache, and the model card states roughly 8 KiB per token in BF16.

For long-context work this makes memory use easier to predict. It only helps if the runtime supports this attention layout, though. Check when llama.cpp, Ollama or vLLM added support before choosing a file.

Request access first, then choose a file

This model is not downloadable straight from the Hugging Face repository. You fill in your name, affiliation, job title, country and intended use, and agree to the licence. After approval you can fetch the model files and the GGUF or MLX conversions. Approval is not automatic, so allow time for the first attempt.

Once you have access, pick the format that matches your use. Target a single RTX 5090 and the NVFP4 file is the one the model card points to; prefer compatibility and simpler tooling and GGUF is the choice. MXFP8 is also offered. Each format needs different memory and runtime support, so the same model does not mean the same running conditions.

Treat preview status as part of how you read results

The model card states that this release is not final and that specifications and results may change before the finished version. The benchmark tables are preview figures too. If a final release under the same name appears in a few weeks, scores and recommended hardware may differ, so record the version and measurement date before putting it into work.

The comparison table places Raon-1 mini beside Qwen3.6-35B-A3B, Gemma 4 26B-A4B, Qwen3.8-27B and Muse Glimmer 30B. Sharing a table does not prove the measurement environments match, so verify that separately. Some knowledge benchmarks are lower than rivals, so read them alongside the reasoning, instruction-following and agentic results the release highlights.

For Korean document work, it is worth comparing early

What makes this model interesting for Korean users is that a model with Korean in its training data can be tried on RTX 5090-class hardware. For work where Korean input is the point, such as meeting summaries, internal document triage or Korean customer replies, it is worth comparing against the Qwen or Gemma models you already use on identical sentences.

Conversely, if your work is only English code writing or long English documents, there is little reason to pick this model first. Switch only when a comparison on the same memory shows an advantage over a model you have already validated. Keep a preview model as a candidate for testing rather than a work baseline.

Check the Apache 2.0 terms

Raon-1 mini is released under the Apache 2.0 licence. Commercial use, modification and redistribution are allowed with no revenue threshold. The access gate is a distribution method, not a narrower licence. Apache 2.0 does require keeping copyright notices and marking changes, so retain the licence and NOTICE when embedding it in a product.

• KRAFTON AI Raon-1 mini model card
• Raon-1 mini official playground

Frequently asked questions

What should you check first in this article?

Raon-1 mini is a 30B-A4B MoE model that KRAFTON AI trained from scratch for Korean and agentic work. Of 30B total parameters, 4B are active per token, and the context reaches 262,144 tokens. It was trained on Korean, English, Japanese, Chinese, Spanish, French and Thai. The model card states that NVFP4 quantization lets it run on a single RTX 5090 with 32GB. It is a preview release rather than the final model, and you must request access on Hugging Face before downloading.

How much memory does it need?

For long-context work this makes memory use easier to predict. It only helps if the runtime supports this attention layout, though. Check when llama.cpp, Ollama or vLLM added support before choosing a file.

Where can you check the conditions and sources?

Its training languages are Korean, English, Japanese, Chinese, Spanish, French and Thai. The model card describes strong Korean and solid English. For Korean users, Korean appearing among the training languages matters by itself. How much training weight each language received is not published, so verify Korean quality directly.

Key takeaways

1. 1017 is a retrained checkpoint with the same architecture; the `deepseek-flash` API name and price are unchanged.
2. All 48 shards changed, including the two Engram shards and the vision tower; mixing them with September files silently runs a mismatched model.
3. The mainline llama.cpp pull request for V4.1 is not merged; vLLM and SGLang on server GPUs, plus dedicated MLX conversions, come first.

1017 is a checkpoint that fills the same slot, not a new model

Released on October 17, 2026, V4.1-Flash-1017 is a retrained checkpoint that follows the V4.1 Flash from September 10. DeepSeek calls it a minor update, yet its own benchmark table shows Terminal-Bench 4.0 up 11.1 points and ProgramBench up 8.8, with shorter outputs for the same task. The web, app and API model names stay as they were, and API users keep calling `deepseek-flash`.

The architecture is unchanged. A 552B backbone, 196B of Engram, the DSpark draft and the vision tower together appear as 763B parameters on the model card. The 1,048,576-token context, 384K maximum output and MIT license also match September. For local users, the useful question is therefore not what was added, but which files in the same slots were replaced.

• DeepSeek 1017 release note
• DGX Spark 128GB local LLM baseline
• Context length, RoPE and sliding-window attention
• GGUF vs MLX: which model format should you download?
• DeepSeek 1017 vs Opus 5 vs GLM-5.3: scores and cost together

The file names and sizes are identical; the contents are not

All 48 official safetensors shards were rewritten for 1017. Their names and individual sizes match September, but the two Engram n-gram tables and the vision tower carry new values. The problem is that serving engines usually read file names and tensor shapes, not the version behind them. Point a 1017 model at September's Engram tables and loading still succeeds; the replies still look plausible. They come from a mismatched combination, not from 1017.

If you already have the September checkpoint, download 1017 into a new folder instead of overwriting the old one. Deleting or renaming the previous folder reduces the chance that an engine picks up stale files. After downloading, compare the shard names and sizes with the repository file list; for a converted build, also follow the checksum file that repository provides.

hf download deepseek-ai/DeepSeek-V4.1-Flash-1017 \
  --revision 1bae0e1088993b9b82de08f59de5be5a34e42170 \
  --local-dir ds41-1017
du -sh ds41-1017

• 1017 repository file list

How much you re-download depends on the format you run

The official checkpoint totals 510.3 GB across 48 files, or 475.25 GiB. Converting it needs room for both the source and the result, and at run time the KV cache and runtime memory come on top of the weights. Planning for 1 TB or more of free space is sensible, but that is arithmetic for planning, not an official minimum.

Two routes exist for personal hardware. A VQ GGUF aimed at a single 128 GB DGX Spark keeps a 113.56 GB base, a 0.97 GB vision sidecar and 203 GB of Engram tables separately, reading Engram rows from disk; the download is about 318 GB. An MLX 3-bit conversion stores 223.5 GiB of converted shards plus 189.1 GiB of original Engram, for 412.6 GiB. Neither has published 1017-specific serving speed yet.

• VQ GGUF conversion for DGX Spark
• MLX 3-bit conversion

The runtime that reads 1017 today is not mainline llama.cpp

The llama.cpp pull request adding DeepSeek V4.1 support opened on September 10, 2026 and remained unmerged and conflicting on October 18. A claim that it runs in llama.cpp therefore usually refers to a fork or a separate conversion. The same command on a mainline build can fail because the architecture is not recognized.

On server-class GPUs, the vLLM and SGLang recipes can run this architecture, and NVIDIA's Dynamo documentation covers V4.1 Flash deployment. In vLLM, thinking is on by default at effort 50, so a small max_tokens can spend the whole budget on the trace and return empty content with finish_reason=length. The model is not broken; turn thinking off or give the output budget room.

Prompt formatting also needs care. The 1017 repository ships no Jinja chat template; the Python reference in the encoding folder and deepseek-recipe provide the same format. Treating the DSML tool-call tags, which carry a leading space, or the 1–100 numeric effort like an ordinary template can misalign the conversation.

• llama.cpp V4.1 support pull request
• vLLM DeepSeek V4.1 recipe

Read the benchmark table for cost per task, not just scores

The model card reports 1017 ahead of the September checkpoint on most rows, and above Opus 5 on one coding benchmark. Terminal-Bench 4.0 rises from 31.2 to 42.3, DeepSWE v1.1 from 74.2 to 77.0 and ProgramBench from 20.3 to 29.1. HLE improves from 36.8 to 42.6 but stays below Opus 5 at 56.3.

These are results under the vendor's published harness conditions. Terminal-Bench used the Claude Code harness and DeepSWE used Mini-SWE-V2; they are not token rates or first-token latency on your machine. Because 1017's practical gain comes from finishing the same task with shorter output, the realistic local comparison is how many tokens the same request consumes against the September checkpoint.

API pricing is unchanged, but search tools start charging in November

On the API, both `deepseek-flash` and the legacy alias `deepseek-v4-flash` route to 1017, and the price list matches September. Cached input costs $0.003 per million tokens off-peak and $0.006 at peak; uncached input costs $0.15 and $0.3; output costs $0.6 and $1.2. Peak hours are 01:00–04:00 and 06:00–10:00 UTC on weekdays.

The newly opened Search and Fetch APIs charge $0.75 per 1,000 successful requests each. Calls made as server-side tools from DeepSeek Harness, Claude Code or Codex are free until 00:00 Beijing time on November 1, 2026, and billed after that. Even with a local model, adding search creates this separate cost.

• DeepSeek models and pricing
• Search and Fetch API guide

The order to check before you re-download

First decide whether the model runs locally or through the API. With the API, the name stays as it is. For local use, verify that your engine actually loads the deepseek_v41 architecture at the 1017 revision, then budget storage and conversion space, download into a separate folder and re-run one representative task under the same conditions.

Record the repository revision 1bae0e1, the conversion and runtime commit you used, the context length, output token count and time to first token. If a setup that was stable on the September checkpoint does not behave the same way on 1017, resolve that difference before chasing the new benchmark scores.

Frequently asked questions

What should you check first in this article?

DeepSeek-V4.1-Flash-1017 is a retrained checkpoint of the September V4.1 Flash. The architecture, 1M-token context, MIT license and the `deepseek-flash` API name are unchanged, but all 48 weight shards were replaced. If you already run V4.1 Flash locally, do not overwrite the old folder: bring the Engram tables and vision files to the 1017 revision and re-run your usual task under the same conditions.

How much memory does it need?

The official checkpoint totals 510.3 GB across 48 files, or 475.25 GiB. Converting it needs room for both the source and the result, and at run time the KV cache and runtime memory come on top of the weights. Planning for 1 TB or more of free space is sensible, but that is arithmetic for planning, not an official minimum.

Key takeaways

1. Search lives at `/v1/tools/search` and Fetch at `/v1/tools/fetch`; both use bearer authentication.
2. Each costs $0.75 per 1,000 successful requests; calls through Harness, Claude Code or Codex are free until November 1.
3. For a local model, putting search summaries in the prompt is more reliable than relying on tool calls.

Search and page fetching are separate APIs

The Search API returns a list of web pages matching a query, and the Fetch API returns the body of one URL as Markdown. Both sit under `https://api.deepseek.com` at `/v1/tools/search` and `/v1/tools/fetch`, and both use a DeepSeek API key as a bearer token. Search alone does not give you the full page, so decide first whether a summary is enough or you need the body.

Search accepts a query of 1 to 128 characters and returns 1 to 50 results, 10 by default. Setting `need_content` includes page bodies in the response, which makes it much larger. `ori_question` takes the original question, up to 128 characters, to improve ranking.

• How can you search your local photos and recordings with EmbeddingGemma 2?
• Run Jev-Style Local Decision Models on Mac, RTX 3090, and Servers
• Which local LLM app or serving engine should you start with?

Read the ranking score and summary first

Each result carries `rerank_score` (0 to 3), `authority` (0 to 4), `date`, `summary` (usually 600 to 800 characters) and `content`. A score of 1.5 or above is described as high quality, and an authority of 3 or above marks a highly authoritative source. Keep the summary, URL and date together in the prompt so citations stay easy.

• Search and Fetch API guide

Feed the local model with search results, not tool calls

Handing tool calls to a local model makes schema mistakes and repeated calls more likely as the model gets smaller. A simpler pattern is for your app to search first, put the top three to five summaries into the prompt with citation numbers, and ask the local model to answer only from that material and say when it does not know.

For questions that need the source text, fetch only the URLs you selected and trim the body. Keep the paragraphs that matter to the question, and record each URL and fetch time. The retrieval step then stays reusable when you change models.

curl https://api.deepseek.com/v1/tools/search \
  -H "Authorization: Bearer $DEEPSEEK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query":"DeepSeek V4.1 Flash 1017","count":5}'

Billing counts successful requests

Search and Fetch each cost $0.75 per 1,000 successful requests. Forty searches and twenty fetches a day come to 1,200 searches and 600 fetches a month, about $1.35. Failed requests are not billed, but a retry that succeeds does count.

Calls made as server-side tools from DeepSeek Harness, Claude Code or Codex are free until 00:00 Beijing time on November 1, 2026. After that the same calls are billed per request, so recording your volume during the free window makes the post-change cost easier to estimate.

Handle Fetch failures by cause

Fetch can return 403 when the target site blocks it, 404 when no usable body is available and 500 when the address cannot be resolved. An empty balance returns 402. Retrying the same URL rarely changes a 403 or 404, so switch sources or fall back to the search summary.

A URL cannot carry a username or password, and the API must not be used to get around a site's access policy. Retry only 5xx responses, briefly, and cache successful URLs with a body hash to avoid paying repeatedly for the same page.

What to log before production

Log the query, the URLs you used, the fetch time, the model and the prompt version. Search results change over time, so you need the material as it looked when the question was asked. Before putting personal or internal content into a query, decide what is allowed to leave your network.

When storing response bodies, check copyright and site terms and keep only the length you need. With a local model plus this API, the model runs on your machine but queries and URLs still leave it; tell users that plainly.

Frequently asked questions

What should you check first in this article?

DeepSeek's Search API returns a list of web pages for a query, and the Fetch API returns one page's body as Markdown. Both use the same API key and cost $0.75 per 1,000 successful requests each. Rather than forcing a local model to call tools, let your app search first and put the top summaries and URLs into the prompt; it is simpler and easier to reproduce.

Key takeaways

1. `dsh` is a harness for tools, sessions and plugins rather than a model, and it is MIT licensed.
2. `npx @deepseek-ai/dsh web` starts it at `127.0.0.1:3080`, and models are connected under Settings → Models.
3. 1017 support and the Search and Fetch tools arrived in 0.2.1-rc or later, and the preview warns of compatibility changes.

The harness and the model do different jobs

DeepSeek Harness (`dsh`) is an agent harness released by DeepSeek AI under the MIT license. It is not the model: it wraps conversation sessions, file access, shell execution, search and plugins so a model can carry a task through. Its everything-is-a-plugin design lets you enable only the parts you need.

It is in developer preview, and compatibility-breaking changes are expected. For work use, pin a version and read the release notes before updating.

• DeepSeek Harness repository
• What Is a Local AI Agent? When It Makes Sense to Run One
• Granite 4.2 at home: choose between 3B, 8B and 30B
• What is a local LLM? From installation and PC specifications to model selection and speed checks

Install and first run take one command

You need Node.js 22.19 or later, or 24 and later. Running `npx @deepseek-ai/dsh web` starts the Web UI at `http://127.0.0.1:3080` and opens the browser for a local launch. Add `--no-open` to run the server alone.

To run from source, clone the repository and run `pnpm install`, `pnpm run build`, then `pnpm dsh web`. Settings and credentials live under `$DSH_HOME` after the first run, so base backups and deletions on that folder.

npx @deepseek-ai/dsh web

Connect models in settings

Under Settings → Models, enter an API key in the DeepSeek card to use `deepseek-flash`. The key cannot be read back; it is stored in `$DSH_HOME/.credentials.yaml` and settings keep only a reference.

Add other providers or an internal gateway through Custom model API under Add model provider. Supply a provider ID, base URL, one API protocol (OpenAI Chat Completions, OpenAI Responses or Anthropic Messages) and at least one model ID; Fetch available models can list them. The provider ID cannot be renamed later, so choose it carefully.

Match three values when you attach a local server

A local server that exposes an OpenAI-compatible API, such as vLLM or llama.cpp's server, can also be added as a Custom model API. Set the base URL, model ID, context window and maximum output tokens to match the server. If the harness exceeds the real limit while compacting a long conversation or inserting tool output, the request fails.

Tool-calling reliability on a local model varies with the model and runtime. A small model may break the tool schema or repeat a call, so narrow file and shell permissions and start with a short task. Attaching 1017 locally needs a server path such as vLLM; mainline llama.cpp still does not read this architecture.

1017 and the search tools arrive with 0.2.1-rc

DeepSeek says 1017 is tuned for dsh 0.2.1-rc.1 or later. Version 0.2.1-rc.2 adds a switch for the Search and Fetch APIs in web search settings. Those tools are free until November 1, 2026, and billed per successful request after that.

Note that a session keeps the model recorded in its own log. A session that has already sent a request does not switch to a new default, so open a new session to compare.

• dsh release notes
• Model configuration guide

What to check before updating

A preview release can change the settings file and plugin API. Back up `$DSH_HOME/profiles/web/cordis.patch.yml`, record the plugins and model settings you use, then update. Pinning a tag reproduces better than automatic updates.

Do not leave shell and file permissions at their defaults; narrow the working folder and allowed commands. A harness may take broad permissions for convenience, so check the sandbox and approval flow first on important repositories.

Frequently asked questions

What should you check first in this article?

DeepSeek Harness (`dsh`) is an agent harness released by DeepSeek AI under the MIT license. It is not the model: it wraps file access, shell execution, search and plugins so a model can carry a task through. Start the local Web UI with `npx @deepseek-ai/dsh web`, then connect a DeepSeek key or a custom API in settings. It is a developer preview, so pin the version you use.

Key takeaways

1. 1017 reaches 42.3 on Terminal-Bench and 77.0 on DeepSWE, well above September and past Opus 5 on DeepSWE.
2. HLE is 42.6 against Opus 5 at 56.3, so broad reasoning still shows a gap.
3. DeepSeek's API is far cheaper on both input and output, and only 1017 and GLM-5.3-Flash offer local weights here.

The comparison table comes from fixed harness conditions

DeepSeek's model card compares 1017, September V4.1, K3, GLM-5.3, GPT-5.6 Sol and Opus 5 on the same rows. Terminal-Bench 4.0 was measured with the Claude Code harness and DeepSWE v1.1 with Mini-SWE-V2. Different harnesses give different scores for the same model, so the table does not transfer directly to your tooling.

These are also not local speed figures. The scores measure whether a task was completed; tokens per second and first-token latency need separate measurement.

• DeepSeek-V4.1-Flash-1017: what to re-download and what runs it
• MiMo V2.6 Pro and Flash launched together: how strong are they?
• What is a local LLM? From installation and PC specifications to model selection and speed checks

1017 leads on coding and agent tasks

On Terminal-Bench 4.0, 1017 scores 42.3, which is 11.1 points above September's 31.2 but below Opus 5 at 51.8. DeepSWE v1.1 reaches 77.0, above Opus 5 at 74.0, while ProgramBench rises from 20.3 to 29.1 but stays under Opus 5 at 37.0.

On agent rows, Automation-Bench reaches 60.5 and Agents' Last Exam 35.4, above both September and Opus 5. These are the model card's published values, not a promise that your repository sees the same result.

Broad reasoning still favors Opus 5

HLE without tools rises from 36.8 to 42.6, still under Opus 5 at 56.3 and GPT-5.6 Sol at 44.5. NL2Repo-Bench is 68.2 against Opus 5 at 75.3. On visual and tool-assisted rows such as Chartography and BabyVision, however, it has come close to Opus 5.

A claim that 1017 beat Opus 5 is true on one coding row, not as a conclusion about overall capability. The choice depends on the tasks you actually run.

The price gap is larger than the score gap

On the DeepSeek API, `deepseek-flash` charges $0.3 at peak for uncached input and $1.2 for output. Opus 5 is listed at $5 input and $25 output. For the same tokens that is about 17 times more on input and 21 times on output; DeepSeek's off-peak rates halve again, widening the gap.

Agent work tends to produce long output. A task using 20,000 input and 10,000 output tokens costs about $0.018 on DeepSeek at peak and about $0.35 on Opus 5. Shorter outputs in 1017 can widen that per-task gap further, but retries for failed tasks belong in the calculation.

• DeepSeek pricing
• Claude Opus 5 documentation

The local options are 1017 and GLM-5.3-Flash

1017 publishes 475.25 GiB of official files under the MIT license, and GLM-5.3-Flash is an open-weight model with 320B total and 18B active parameters. Opus 5 has no public weights and is API-only. If local execution is a requirement, the choice narrows to 1017 and GLM-5.3-Flash.

With a 1M context and a 552B backbone, 1017 on personal hardware needs compressed conversions and SSD streaming. GLM-5.3-Flash is also 320B and needs server-class memory. Open weights do not mean it runs on your laptop.

Three checks that decide it for your work

First, pick three tasks you run often and compare them with the same prompts. Second, record the tokens and retries each one needed. Third, confirm whether the data may leave your network. These three checks matter more than the score table.

Rather than chase a one-point benchmark difference, choose the option with a lower failure rate and predictable cost on your tasks. In short: 1017 for coding, agents and local execution; Opus 5 for broad reasoning and a managed API; GLM-5.3-Flash as another open-weight option.

Frequently asked questions

What should you check first in this article?

On DeepSeek's model card, 1017 leads the September model by a wide margin on coding and agent benchmarks such as Terminal-Bench and DeepSWE, and passes Opus 5 on some of them. Opus 5 still leads broad reasoning rows such as HLE and NL2Repo. Add API price differences and whether the weights can run locally, and no single score decides the choice.

How much memory does it need?

With a 1M context and a 552B backbone, 1017 on personal hardware needs compressed conversions and SSD streaming. GLM-5.3-Flash is also 320B and needs server-class memory. Open weights do not mean it runs on your laptop.

Key takeaways

1. gpu-unlock is not a driver patch but a kernel module that writes the SS0 selector before the driver binds.
2. On the 3090 Ti, BF16 rose from 87.50 to 157.41 TFLOPS, but there is no published 3090 table yet.
3. VRAM, bandwidth and token generation do not change, and the verified environment is one card on one distribution.

A kernel module published on October 18

gpu-unlock is a Linux kernel module that opens the blocked BF16 throughput on the RTX 3090. It is neither a UEFI app nor a patched NVIDIA driver: it writes a speed selector before the driver binds the card. The module is GPL-2.0, and the generated firmware derives from CMP unlock reference material.

The repository says it writes `0x88888888` to SS0 to select the full BF16 rate, and also writes the 4-bit SS1 value `0x8`, whose effect is untested. The card must be cold-booted; warm reboots are not supported.

• gpu-unlock repository
• RTX 3090 24GB memory, speed and price
• Prefill vs decode: should you look at TTFT or tok/s?
• LLM memory bandwidth vs compute: how to find the speed bottleneck
• Verifying a 3090 unlock: boot logs, GEMM and llama-bench

What opens is BF16 and FP16 matrix throughput

The maintainer answers that BF16 and F16 operations get about 40% faster. The published figures come from a separate fork tested on a 3090 Ti: at 450 W, dense 8192³ GEMM with FP32 accumulation and five-sample medians, BF16 rose from 87.50 to 157.41 TFLOPS, FP16 from 87.46 to 118.39 and TF32 from 42.58 to 82.61.

The table is for the 3090 Ti, and that test used Ubuntu kernel 7.0.0-38 with a modified 610.43.02 driver and a separate initramfs hook. A 3090 was also unlocked and passed CUDA checks, but no 3090 GEMM table has been published.

VRAM and bandwidth do not change

The unlock touches only the compute path. The RTX 3090 keeps its 24 GB of GDDR6X, 936 GB/s bandwidth and power limit. It is unrelated to the 48 GB memory mod, and it does not let you hold larger models.

Single-token decode is bound by how fast weights are read from memory. A higher BF16 rate therefore does not raise one request's tok/s by the same ratio.

Only one machine has been verified

The repository verified one RTX 3090 at PCI address `0000:21:00.0` on CachyOS kernel 7.2.8-1 with NVIDIA 615.71.09. Another card in the same machine served as an unmodified control. The module rejects other GA102 device IDs, so the targets are the 3090 and, through that separate fork, the 3090 Ti.

On kernels at or below 6.15, the power state does not come up before the driver loads, so the unlock fails; a patch is posted as an issue. The 3090 Ti is supported only by a separate fork, and other GA102 device IDs are rejected.

Cold boot and rollback are part of the deal

The module runs only when the kernel command line carries `gpu_unlock.targets=`, for example `gpu_unlock.targets=0000:21:00.0` or `all`. It must load from initramfs before `nvidia`, and the machine must be powered off completely and started again.

To roll back, remove the option from the kernel command line and uninstall the packages. If firmware never stops, the module keeps DMA buffers until the next power-off, so power off before retrying. If the display stays blank, delete the option in the bootloader edit screen and boot.

Who should consider testing it

It matters to people running prefill-heavy, concurrent or LoRA fine-tuning workloads on a 3090. For short single-request answers the expected gain is small. Try it on a test PC rather than a work machine, after recording a baseline.

When testing, fix the power limit and the model, and record prefill time alongside concurrent throughput. Use a GEMM to isolate matrix compute, and llama-bench for the felt result.

Frequently asked questions

What should you check first in this article?

gpu-unlock, published on October 18, 2026, is a Linux kernel module that writes a speed selector before the NVIDIA driver binds the card, opening BF16 throughput on the RTX 3090. The repository verified it on one 3090, and the 3090 Ti numbers came from a separate fork. It does not add VRAM or speed up decode, and it requires a cold boot with a rollback path.

How much memory does it need?

The unlock touches only the compute path. The RTX 3090 keeps its 24 GB of GDDR6X, 936 GB/s bandwidth and power limit. It is unrelated to the 48 GB memory mod, and it does not let you hold larger models.

Key takeaways

1. Decode is bound by 936 GB/s of bandwidth, so higher BF16 throughput barely changes it.
2. Prefill, batching and LoRA carry more compute, so the unlock may show up there.
3. Fix the power limit, model and quantization, and compare before and after under the same conditions.

Decode is bound by bandwidth, not compute

Generating one token at a time means reading weights from memory at every step. The RTX 3090's 936 GB/s decides how long those reads take, not the BF16 matrix throughput. A higher compute rate therefore barely moves single-request tok/s.

The 3090 measurements published on this site point the same way: Qwen3.8 27B UD-Q4_K_XL with MTP n-max 3 ran at 68.9 tok/s at 350 W and 51.0 tok/s at 250 W. Lowering the power limit slows it down, which is a clock and power effect, not a compute one.

• RTX 3090 24GB memory, speed and price
• Prefill vs decode: should you look at TTFT or tok/s?
• GGUF vs MLX: which model format should you download?
• LLM memory bandwidth vs compute: how to find the speed bottleneck
• The RTX 3090 48 GB mod: what to verify before you accept the card
• Two RTX 3090s with NVLink: how much it actually changes inference

Prefill is compute-bound and may change

Prefill reads a long document at once and runs matrix products over every input token, a compute-bound stage. A runtime that prefills in BF16 may see a shorter time to first token after the unlock. Paths that do not use BF16 matrix products, such as llama.cpp's quantized kernels, see little effect.

Which runtime prefills in BF16 depends on the model and kernel, and vLLM and llama.cpp do not take the same path for the same model. To estimate the prefill gain, start with the kernels your runtime uses.

Concurrency and batching widen the gap

Serving several requests at once reuses the same weights, reducing bandwidth pressure and raising the compute share. BF16 throughput can then set aggregate throughput, which is why a continuously batching server such as vLLM may show more benefit.

In a single-user, one-request-at-a-time setup, no batch forms and this advantage mostly disappears. Expectations differ between a server operator and a personal desktop user.

Fine-tuning and image generation are separate cases

LoRA fine-tuning uses many BF16 matrix products, so it is theoretically where the unlock helps most. The 3090's 24 GB of VRAM limits batch size and sequence length, though, so memory becomes the constraint before compute does. Larger batches have more room to benefit.

In image generation, the compute and memory shares depend on model, resolution and step count. Compute-heavy FP16 sections may benefit, while high resolutions hit VRAM and bandwidth first. The before-and-after difference in image generation varies with the model, resolution and step count, so measuring your own settings answers it fastest.

Measure with power fixed, before and after

Record a baseline at the same model, quantization, context length and power limit, then repeat after the unlock with a cold boot. Decode alone will show almost nothing, so log prefill time and concurrent throughput as well.

So should you buy one

The unlock is an experiment on compute headroom you already own. It is a weak reason to buy a card; if decode is the goal, look at other options such as a 5090 or a 48 GB mod. The 3090's strength is 24 GB of capacity and price, not compute rate.

The decision comes from measurement, so decide after running the same card before and after.

Frequently asked questions

What should you check first in this article?

The RTX 3090 BF16 unlock raises matrix throughput, but single-request token generation is bound by memory bandwidth. Prefill of long documents, batched requests and LoRA fine-tuning carry more compute, so they may see an effect. No public 3090 measurement exists yet, so set an expectation and measure it yourself.

How much memory does it need?

Generating one token at a time means reading weights from memory at every step. The RTX 3090's 936 GB/s decides how long those reads take, not the BF16 matrix throughput. A higher compute rate therefore barely moves single-request tok/s.

Key takeaways

1. Record the kernel, driver, VBIOS and power limit for the comparison to hold.
2. GEMM isolates matrix compute; llama-bench reports prefill and decode separately.
3. Check the cold-boot logs, and roll back by removing the command-line option if it fails.

Record the baseline first

Before measuring, record the kernel version, NVIDIA driver, VBIOS, power limit and current clocks. `uname -r`, `nvidia-smi` and `nvidia-smi -q` show them. The same card gives different results under a different driver or power setting.

Keep the power limit fixed for the whole experiment. The 3090 defaults to 350 W, and lowering it to 300 or 280 W slows decode slightly. Changing power between runs mixes the compute effect with the power effect.

uname -r
nvidia-smi --query-gpu=name,driver_version,vbios_version,power.limit,clocks.max.sm,memory.total --format=csv
nvidia-smi -q | grep -A2 "GPU Power Limit"

• Prefill vs decode: should you look at TTFT or tok/s?
• GGUF vs MLX: which model format should you download?
• RTX 3090 BF16 unlock: what opens and what stays the same
• llama.cpp: finding which settings are slowing you down

Measure matrix compute with GEMM

The published comparison used a dense 8192³ GEMM with FP32 accumulation and five-sample medians. Measuring FP16, BF16 and TF32 under the same setup isolates the unlock's effect on matrix compute. Discard warm-up runs and use the median.

In PyTorch, TF32 is used only when `torch.backends.cuda.matmul.allow_tf32` is enabled, and BF16 requires `torch.bfloat16` inputs. Write the script yourself, following the conditions in the repository issue.

import torch, time, statistics
assert torch.cuda.is_available()
torch.backends.cuda.matmul.allow_tf32 = True
n = 8192
for dtype in (torch.float16, torch.bfloat16):
    a = torch.randn(n, n, device='cuda', dtype=dtype)
    b = torch.randn(n, n, device='cuda', dtype=dtype)
    for _ in range(3): a @ b
    torch.cuda.synchronize()
    times = []
    for _ in range(5):
        start = time.perf_counter(); a @ b; torch.cuda.synchronize()
        times.append(time.perf_counter() - start)
    med = statistics.median(times)
    print(dtype, round(2 * n**3 / med / 1e12, 2), 'TFLOPS')

Measure prefill and decode with llama-bench

`llama-bench` reports prefill and decode separately for the same model. Fix context and repetitions with something like `-p 4096 -n 128 -r 5`, and keep the model, quantization and power limit identical before and after. Use a GGUF already on the card.

Many of llama.cpp's quantized kernels do not use BF16 matrix products. A large GEMM difference can therefore leave llama-bench decode unchanged. You need to explain that gap to interpret the results.

llama-bench -m /path/to/model.gguf -p 4096 -n 128 -r 5

Check logs after a cold boot

The unlock applies after a full power-off and restart. After boot, check the module log with `journalctl -k -b | grep gpu_unlock` and the driver binding with `lspci -nnk -s <PCI address>`, and confirm the card appears normally in `nvidia-smi`.

Stop the experiment if you see Xid errors or a driver reset. A card already used by a driver in the same power cycle is refused, so power off before retrying.

journalctl -k -b | grep gpu_unlock
lspci -nnk -s 0000:21:00.0
nvidia-smi

Rollback and failure recovery

If something goes wrong, remove the `gpu_unlock.targets=` entry from the kernel command line and reboot. Uninstalling the packages also removes the module and initramfs configuration. If the display stays blank, delete the option temporarily in the bootloader edit screen and boot.

If the log says firmware never stopped, DMA buffers remain until the next power-off. Power off before retrying, and test on a spare card when possible.

Record sheet

Fill in these fields for both runs to make the comparison reproducible. If a value differs, explain it first; if it cannot be explained, do not count it as the unlock's effect.

Frequently asked questions

What should you check first in this article?

Comparing before and after starts with fixing conditions: record the kernel, driver, VBIOS, power limit and context length, measure matrix throughput with a torch GEMM and prefill and decode with llama-bench. Check the logs after a cold boot, and roll back by removing the kernel command-line option if it fails.

Key takeaways

1. A modified card is not a normal GPU; booting and displaying output does not prove it passes.
2. The checks are a 49152 MiB report, compute capability 8.6, 92 to 95 percent memory allocation, and driver, PCIe and Xid logs.
3. Bandwidth stays at 936 GB/s, so decode does not speed up; the gain is resident capacity for larger models.

The 48 GB mod replaces memory chips

The 48 GB mod is not a software setting but a physical job that replaces the card's GDDR6X chips. When it succeeds, the driver reports `49152MiB` and frameworks such as PyTorch see the same size. It is a different kind of work from a kernel-module unlock that opens compute rate.

Memory bandwidth stays at 936 GB/s, the same as the 24 GB model. Single-token generation is set by that bandwidth, so the mod does not speed up decode. What grows is the model size you can keep resident on one card.

• RTX 3090 24GB memory, speed and price
• LLM memory bandwidth vs compute: how to find the speed bottleneck
• Running local LLMs across two or more GPUs
• Does the 3090 unlock speed up local LLMs? Prefill, batching, decode

Booting is not proof of acceptance

A modified card can boot and display output and still fail at handover. The team behind the acceptance tool lists the failure modes: the driver reports the wrong memory size, CUDA sees less VRAM than expected, high-memory allocation crashes, or the OS log collects NVIDIA, WHEA, Xid or PCIe errors.

The acceptance run therefore has two stages: a short check and a long stress run. The short check reads the baseline, the stress run holds a large allocation for a long time, and then you reboot and check again.

The acceptance checklist

The tool checks that the GPU name contains `3090`, that VRAM reports about `49152MiB`, that PyTorch CUDA sees the same size, that compute capability is `8.6` as expected for a 3090, and that writes reach 92 percent of the target memory by default. It also records the PCIe link, VBIOS, driver, power, temperature and utilization.

OS log review is part of the checklist: NVIDIA, Xid and PCIe kernel errors on Linux, and NVIDIA, Display, WHEA and PCIe system events on Windows. Checks that could not run for lack of log permissions are not treated as passes but flagged for human review.

Running the test and reading the verdict

In a workshop you run the short check first and use a longer stress run for handover. The tool takes a duration and allocation fraction such as `--stress-seconds 1800 --vram-fraction 0.95`, and you can adjust the expected size with `--expected-memory-mib 49152 --memory-tolerance-mib 1024`. Reports are saved under `reports/` as Markdown and JSON.

There are three verdicts: `ACCEPT` when no check fails or warns, `REVIEW` when nothing hard-fails but a human should look, and `REJECT` when any required check fails. Do not accept on `REJECT`; for `REVIEW`, resolve the PCIe warning, temperature or permission issue before deciding.

chmod +x install.sh gpu-validator.sh
./install.sh
./gpu-validator.sh --quick
./gpu-validator.sh --stress-seconds 1800 --vram-fraction 0.95
# 재부팅 후 같은 카드에서 다시 확인
./gpu-validator.sh --quick

Capacity grows, speed does not

The mod has one benefit: a model that does not fit in 24 GB can live on one card. Unlike splitting across two 24 GB cards with tensor parallelism, no inter-card communication or synchronization cost is involved. Power and heat, however, stay concentrated on one card.

Expecting speed here leads to disappointment. In the 3090 measurements published on this site, decode was 68.9 tok/s at 350 W and 51.0 tok/s at 250 W, following the power limit, while memory capacity did not change speed. A modified card sits on the same bandwidth.

Risks that remain with a modified card

A modified card is not covered by the manufacturer's warranty. A driver or kernel update can change what memory is reported, and then you need to repeat the same checks. Check whether the seller's warranty language points to a test result, such as a passing 49152 MiB report and stress run, rather than a vague promise that it works.

The same report matters when you sell the card on. Keeping the test record lets both buyer and seller verify the same conditions, so run the same check before selling as well.

Frequently asked questions

What should you check first in this article?

The 48 GB mod physically replaces memory chips, and the first gate is whether the driver reports 49152 MiB. A card can boot and display output yet fail under high-memory allocation or a long stress run, so run the acceptance test and keep the report before accepting it. Bandwidth matches the 24 GB model, so token generation does not get faster; the gain is fitting a larger model on one card.

How much memory does it need?

The 48 GB mod is not a software setting but a physical job that replaces the card's GDDR6X chips. When it succeeds, the driver reports `49152MiB` and frameworks such as PyTorch see the same size. It is a different kind of work from a kernel-module unlock that opens compute rate.

When is it better not to choose this?

The mod has one benefit: a model that does not fit in 24 GB can live on one card. Unlike splitting across two 24 GB cards with tensor parallelism, no inter-card communication or synchronization cost is involved. Power and heat, however, stay concentrated on one card.

Key takeaways

1. In the community benchmark, NVLink raised decode throughput about 5 percent and cut p95 latency.
2. Tensor parallelism works over PCIe without a bridge, and the default setup leaves P2P off.
3. Whether P2P is really on has to be checked at three layers: driver, NCCL and vLLM.

What an NVLink bridge does

The RTX 3090 supports NVLink 3.0 on GA102, so a pair of cards can carry four links of 112.5 Gbit/s each for 112.5 GB/s bidirectional in total, 56.25 GB/s per direction. PCIe 4.0 x16 provides about 32 GB/s per direction, so the dedicated path between cards is much wider.

Having an NVLink connector on the 3090 is not the same as having a bridge installed. The reference notes state that a bridge has to be physically installed and that most consumer setups do not have one. Without it, multi-card setups run over PCIe.

• RTX 3090 24GB memory, speed and price
• Context length, RoPE and sliding-window attention
• Prefill vs decode: should you look at TTFT or tok/s?
• LLM memory bandwidth vs compute: how to find the speed bottleneck

Measured: NVLink on and off at TP=2

A community benchmark ran the same model on two 3090s with vLLM tensor parallelism (TP=2). With Devstral Small 2505 Q8_0, 50 code generation prompts and responses averaging about 1650 tokens, NVLink gave 85.0 tok/s against 81.1 tok/s over PCIe alone, roughly a 5 percent difference.

The tail latency stands out more: the 95th percentile response time was 239.1 seconds with NVLink against 277.6 seconds without, so NVLink was steadier. If variance on long requests matters more than average throughput, that difference counts. The author concluded that for inference alone the money is better spent on VRAM or a card than on a bridge.

The default is PCIe with P2P off

The 2x3090 operating notes describe the default dual and multi-card path as PCIe-only with P2P off: `NCCL_P2P_DISABLE=1` and custom all-reduce disabled, chosen so it works out of the box on any consumer rig without tuning.

The same notes say enabling P2P over PCIe can add a workload-dependent few percent up to about 20 percent. The emphasis is on order: confirm the default PCIe path works, then read the topology and enable P2P to compare, which makes problems easier to isolate.

Three layers to check whether P2P is on

P2P is not one question but three stacked ones: whether the driver grants direct GPU-to-GPU access, whether NCCL uses that path for transfers, and whether vLLM's extra custom all-reduce kernel is active. Conflating them makes it impossible to explain why P2P looks enabled yet does not speed anything up.

vLLM's gate checks for NVLink by name, but that check is a cheap pre-filter ahead of an expensive one. As a result, rigs with three or more cards over PCIe keep custom all-reduce off by design, because they lack a fully connected NVLink mesh. That is intended behaviour rather than misconfiguration, and P2P benefits can still flow through the NCCL layer.

What TP=2 costs and requires

In vLLM, tensor parallelism shards the weights across the cards. The vLLM documentation notes that this leaves more KV cache space per card but can add synchronization overhead, so context length and concurrency have to be tuned alongside it.

Enabling chunked prefill splits compute-bound prefill and bandwidth-bound decode so both resources get used at once. Power limits apply per card, so two cards mean watching the combined draw and case heat. As the chart above shows, lowering a 3090's power limit lowers decode speed with it.

nvidia-smi topo -m
nvidia-smi topo -p2p rw

CUDA_VISIBLE_DEVICES=0,2 CUDA_DEVICE_ORDER=PCI_BUS_ID \
vllm serve /models/model.gguf \
  --tensor-parallel-size 2 \
  --max-num-seqs 4 --max-model-len 64000 \
  --gpu-memory-utilization 0.95 \
  --max-num-batched-tokens 16384 --enable-chunked-prefill \
  --quantization gguf

So should you buy a bridge

For inference, a bridge is the last item to consider. The measured gain was about 5 percent throughput and a better tail, while larger differences come from whether a model can be split across two 24 GB cards or must live on one larger card. With two cards, confirm the default PCIe path works first.

Work that keeps card-to-card traffic running, such as training or fine-tuning, is a different story. The common assessment is that NVLink's bandwidth advantage shows up more in training than in inference, and the benchmark above reached the same conclusion for inference: about 5 percent. Decide whether your use is inference or training before committing.

Frequently asked questions

What should you check first in this article?

The RTX 3090 has an NVLink 3.0 connector, but a bridge must be physically installed and most consumer setups do not have one. In a 2x RTX 3090 vLLM tensor-parallel community benchmark, NVLink led at 85.0 vs 81.1 tok/s, about 5 percent, with a steadier p95 of 239.1 vs 277.6 seconds. The default multi-GPU path runs on PCIe with P2P off.

Key takeaways

1. Temporary offers, daily free quotas and developer trials have different terms.
2. Fix the model ID and account plan, and keep auto-recharge off.
3. Start with code and documents that are safe to share.

Try the answers before downloading the model

With a free open-weight model API, you can send questions or code to a publicly released model and receive answers generated on the provider's servers. Before preparing a large GPU or downloading model files, you can review public code or summarize a short document. This guide helps you choose a service and understand its free-access terms.

For example, OpenCode offers access to the MIT-licensed DeepSeek V4.1 Flash, while Groq hosts GPT-OSS, whose weights use Apache 2.0. Downloaded weights are governed by their model license; using an API also brings the service's usage and data policies into scope.

• DeepSeek model release
• GPT-OSS model release
• China's AI Export Controls: What Happens to the Models You Use on OpenRouter?
• Local RAG Setup Guide: Build an AI That Answers from Your Documents
• What is a local LLM? From installation and PC specifications to model selection and speed checks

Choose by the type of free access

Start with OpenCode for a coding-tool workflow, or OpenRouter to try multiple models through one API format. Groq publishes request and token limits per model, NVIDIA offers developer free endpoints for new Nemotron models, and Mistral enables limited API use in account Free mode.

As of October 16, 2026. Check the service's model list and your account limits when availability or free-access policies change.

Hugging Face Inference Providers currently includes no monthly credits for ordinary Free accounts, so it is excluded from this free API list. Downloading publicly released model files remains a separate option.

• Hugging Face pricing and included credits

Token limits can run out before request limits

OpenRouter permits up to 20 free-model requests per minute. Accounts with less than $10 in total purchases have a daily limit of 50 requests; accounts that have purchased at least $10 get 1,000 per day. Start with the default free allowance rather than purchasing credits just to raise it.

Groq's public Free Plan table lists both GPT-OSS models at 30 requests per minute, 1,000 per day, 8,000 tokens per minute and 200,000 tokens per day. A single long document can exceed the minute token limit even while requests remain. Check your actual per-model limits in the account's Limits page and adjust input length and request spacing.

An agent can exchange multiple requests while reading files, checking results and revising one task. Start with one short code review and inspect call counts and token usage in the usage history. If you receive `429`, stop automatic repetition and follow the response's retry guidance.

• OpenRouter free limits
• Groq per-model limits

Turn off automatic switching for a free-only test

Pin the model ID to a free entry and check whether your application switches models after an error. To compare open-weight models on OpenRouter, specify the chosen model's `:free` ID. The free router selects a model for each request, so use an individual ID when comparing the same open-weight model consistently. Choose `deepseek-v4.1-flash-free` on OpenCode. Leaving paid models out of fallback lists helps prevent a failed free request from triggering an unexpected paid call.

Keep account auto-recharge off as well. Coding tools such as OpenCode can use another model for auxiliary tasks such as session titles. Inspect model IDs and costs in the full usage history; use the direct API examples in each guide to isolate a single request.

• OpenCode auto-recharge settings
• Free models and auxiliary requests

Start with an example that fits the data policy

Begin with a public example stripped of secrets and personal data. OpenCode's free DeepSeek provider states zero retention and no training use, while NVIDIA free endpoints are for development and testing and may log session information for security and product improvement. NVIDIA instructs users not to submit personal or confidential information.

Inputs and outputs in Mistral Free mode may be used for training. Check Anonymous improvement data under Admin → Privacy and disable it before use if you do not want that training use. On OpenRouter, terms vary with the provider actually handling the request, so also read the data policy on the model page.

• Mistral training-data policy
• Mistral training-use settings
• Data policies by free provider

Decide on local hosting after testing your task

For comparison, send the same code and question, then record whether each model identifies the issue and whether its proposed fix passes your tests. The Python examples in these guides print the answer after receiving it in full, so compare total response time. Measuring time to the first token requires a separate streaming configuration. Free API response times also include server queues and network latency. Compare your GPU's speed separately by running the same model locally.

If you work with public material occasionally, first see whether the free allowance is enough. If quotas repeatedly interrupt a growing workload, or the material must stay on your machine, follow the [local LLM agent starter guide](/guides/local-ai-agent-first-guide). Once you choose a model, use the [hardware finder](/?tab=finder) and [speed comparison](/compare/hardware-speed-comparison) to explore local configurations.

Frequently asked questions

What should you check first in this article?

Try publicly released models through OpenCode, OpenRouter, Groq, NVIDIA and Mistral to assess them for your work.

When is it better not to choose this?

OpenRouter permits up to 20 free-model requests per minute. Accounts with less than $10 in total purchases have a daily limit of 50 requests; accounts that have purchased at least $10 get 1,000 per day. Start with the default free allowance rather than purchasing credits just to raise it.

Where can you check the conditions and sources?

For example, OpenCode offers access to the MIT-licensed DeepSeek V4.1 Flash, while Groq hosts GPT-OSS, whose weights use Apache 2.0. Downloaded weights are governed by their model license; using an API also brings the service's usage and data policies into scope.

Key takeaways

1. The exact free model ID is deepseek-v4.1-flash-free.
2. The offer is temporary, with no published end date or guaranteed quota.
3. Start with the direct API example to isolate the cost of a single request.

Send code through OpenCode for a DeepSeek review

OpenCode is an AI coding tool that reads project files and helps explain or edit code. Connect **DeepSeek V4.1 Flash Free** to request reviews without downloading its model files. OpenCode prepares the request and the hosted model generates the answer.

DeepSeek V4.1 Flash has publicly released weights under the MIT license. If hosting the large model is a significant commitment, its free API offer provides a way to assess its answers first. Here we start by reviewing a short Python function that is safe to share.

• DeepSeek model and license
• Try GPT-OSS with Groq's Free API: From Account Setup to Your First Response
• Connect a Free Open-Weight Model on OpenRouter for Code Review
• How much VRAM or unified memory does a local LLM need?

Choose the entry ending in Free

As of October 16, 2026, OpenCode lists input, output and cache reads as free for DeepSeek V4.1 Flash Free. Check for the Free suffix in the model picker. The ordinary DeepSeek V4.1 Flash entry is a separate paid option.

This is a limited-time promotion. No end date or guaranteed per-minute or daily quota is published, so start with model evaluation rather than an unattended long-running job.

• Model list and free-offer policy

Connect and pin the model in OpenCode

For a first installation, run `node --version` and `npm --version` in a terminal to check that Node.js and npm are available. Once both commands show a version, install OpenCode V2 with the command below. Existing OpenCode users should read the V2 installation and migration instructions first. Start in a separate folder containing a public sample rather than a private repository.

npm install -g @opencode/cli

Run `opencode` in the sample folder, then enter `/connect`. Choose the OpenCode pay-as-you-go provider and connect an API key issued in Console. Next, use `/models` to select DeepSeek V4.1 Flash Free. If the account screen requests payment or a top-up, check the free-access conditions before proceeding; do not purchase credits just to follow this example.

To start with the same model, set the `model` value below in the project's `opencode.jsonc`. Merge this field into existing configuration instead of overwriting the file. Check that the selected model still includes Free in a new session.

{
  "$schema": "https://opencode.ai/config.json",
  "model": "opencode/deepseek-v4.1-flash-free"
}

• V2 installation
• Model selection and configuration

Ask for an explanation before allowing edits

Save the function below as `average.py` in the sample folder. Before sending a question, select Plan through `/agents` to start by exploring without editing normal project files.

def average(values):
    return sum(values) / len(values)

Type `@` in OpenCode to attach `average.py`, then ask: ‘Explain what happens with an empty list and how to fix it. Do not edit files or run commands.’ Reject shell-command requests for this exercise. In the response, look for the connection between an empty list having length zero and the resulting division by zero.

If the review is useful, request a proposed fix next. To let OpenCode apply it, switch to an agent that permits file edits and approve the change request. In a Git repository, inspect the changes with `git diff`; in a plain sample folder, compare the files before and after editing. Run tests with both empty and non-empty lists to check the result.

• File attachment and TUI controls
• Plan agent

Manage free requests and auto-recharge separately

Check Auto-recharge rules under Console Settings → Billing and keep them off on an account used only for free evaluation. OpenCode may use other low-cost models to generate session titles. Review the model IDs and costs of both the main answer and auxiliary requests in usage history.

To isolate the request, call only the free model API directly as shown below. The example uses Python 3's standard library and prompts for the key without echoing it. Save it as `opencode-free-check.py` and run `python3 opencode-free-check.py`. It makes one request without automatic retries or switching to a paid model.

import getpass
import json
import urllib.request

key = getpass.getpass('OpenCode API key: ')
payload = {
    'model': 'deepseek-v4.1-flash-free',
    'messages': [{'role': 'user', 'content':
        'Review this Python function for empty-list handling:\n'
        'def average(values):\n    return sum(values) / len(values)'}],
    'max_tokens': 2048,
}
request = urllib.request.Request(
    'https://opencode.ai/inference/openai/v1/chat/completions',
    data=json.dumps(payload).encode(),
    headers={'Authorization': 'Bearer ' + key,
             'Content-Type': 'application/json'},
)
with urllib.request.urlopen(request, timeout=120) as response:
    result = json.load(response)
choice = result['choices'][0]
print(choice['message'].get('content') or '(empty content)')
print('finish_reason:', choice.get('finish_reason'))

In a normal response, review text is returned in `choices[0].message.content`. If `finish_reason` is `length` or the answer is empty, reasoning may have exhausted the output budget; try a shorter input. For authentication errors, check the key and account. For limit errors, check the free entry's availability and retry guidance.

• Auto-recharge and billing settings
• Budget controls

Decide which code to send and which to keep local

The provider behind the free DeepSeek entry states a zero-retention policy and no use of request data for model training. The code is still sent to a hosted service, so begin with public samples containing no secrets or customer data. Other free entries, including MiMo and NVIDIA models, have different data terms; review those when switching models.

For occasional reviews of public projects, this free route is a useful place to start. If private material must stay on your machine or you want to run on your own hardware without a provider quota, continue with the [local agent starter guide](/guides/local-ai-agent-first-guide). The [free open-weight service comparison](/guides/free-open-weight-ai-services-guide) brings together other providers' limits and connection methods.

• Per-model data policies

Hardware-specific speed tuning and serving recipe summary

Provides weights, KV cache, prefill batch, and concurrent request with acceleration functionality, separated for balanced performance, speed, and long-context profiling.

The detail page shows each profile's command, settings, stop conditions, and measurement steps. Commands appear only for combinations with a confirmed model file and runtime.

Frequently asked questions

What should you check first in this article?

Select DeepSeek V4.1 Flash Free in OpenCode to have a hosted model review code.

Key takeaways

1. A model ID ending in `:free` is a free endpoint; `openrouter/free` is a dynamic router that selects a free model for the request.
2. The default OpenRouter free tier allows 20 requests per minute and 50 per day; a cumulative credit purchase of at least $10 raises the daily cap to 1,000.
3. OpenRouter says it does not train on your inputs or outputs, but each request is forwarded to a model provider. Use public code for this review example and check the provider's policy.

Send a public code sample and receive a review

OpenRouter connects inference APIs from multiple model providers through one request format. Send a public code sample and review instructions to receive the model's comments in the response. This guide builds an API request that pins the free variant of Llama 3.3 70B Instruct to review a short Python snippet.

Llama 3.3 publishes its weights under the Meta Llama 3.3 Community License. OpenRouter's `:free` label means hosted API token pricing is zero. [Meta Llama 3.3 model card](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct) · [OpenRouter free endpoint](https://openrouter.ai/meta-llama/llama-3.3-70b-instruct:free)

• Try GPT-OSS with Groq's Free API: From Account Setup to Your First Response
• How to use DeepSeek V4.1 Flash Free with OpenCode
• How much VRAM or unified memory does a local LLM need?
• Retrieve English documents with UltraBERT-Embed-150M

Choose the model ID and free route

A model ID ending in `:free` is a free hosted variant of that model. `openrouter/free` is a router that selects a suitable free model for each request, so the actual model can change between calls. For comparable or reproducible code reviews, use a specific free model ID as this example does. Choose `openrouter/free` when automatic selection matters, while checking weight availability separately. The example's review instruction is written in English.

Send the first request with Python's standard library

You need Python 3, an internet connection, and an API key created in your OpenRouter account. This script sends a request with Python's standard `urllib` library, so no SDK installation is required. `getpass` hides the key while you type, keeping it out of command history and source files. This example sends only a short Python function that is safe to make public.

Save the code as `review.py` and run `python3 review.py` in a terminal. When prompted, enter your API key and press Enter.

Do not save the key in a shared document or repository. If it has already been exposed, revoke it in the OpenRouter dashboard and create a new key.

import getpass
import json
import urllib.request

api_key = getpass.getpass("OpenRouter API key: ")
payload = {
    "model": "meta-llama/llama-3.3-70b-instruct:free",
    "messages": [
        {
            "role": "user",
            "content": (
                "Review this public Python example for correctness. State one concrete issue, "
                "if any, and suggest a minimal fix:\n\n"
                "def percentage(part, total):\n"
                "    return part / total * 100\n"
            ),
        }
    ],
    "max_tokens": 200,
}
request = urllib.request.Request(
    "https://openrouter.ai/api/v1/chat/completions",
    data=json.dumps(payload).encode("utf-8"),
    headers={
        "Authorization": f"Bearer {api_key}",
        "Content-Type": "application/json",
    },
    method="POST",
)
with urllib.request.urlopen(request, timeout=60) as response:
    result = json.load(response)
print(result["choices"][0]["message"]["content"])

Recognize a response and diagnose the first error

On success, the terminal prints the code review. The response JSON stores it at `choices[0].message.content` and may include token details in `usage`, but the current script prints only the review.

To display token details too, add `print(result.get("usage"))` on the line after `print(result["choices"][0]["message"]["content"])` at the end of the script.

The model may point out division by zero when `total` is 0 and suggest a guard, but verify every suggestion against the code and tests.

For `HTTP Error 401`, check that the key is correct and active, and that a space follows `Bearer`. `HTTP Error 429` can mean you reached the 20-per-minute or daily request limit. Stop retry loops and reduce calls until the free allowance resets. For `No endpoints found` or a model error, check whether that `:free` endpoint is currently available and whether your requested features and provider filters match; then choose an available free model ID.

Keep costs at zero within the daily free limit

As of October 16, 2026, OpenRouter's pricing page lists 20 free-model requests per minute and 50 per day for a default account. Failed requests count toward the daily limit. Buying at least $10 in credits raises the free-model cap to 1,000 requests per day. If you are testing within the free tier, leaving credits unpurchased makes costs easier to control.

Keep the model string `meta-llama/llama-3.3-70b-instruct:free` in this example, and check the listed price whenever choosing another model. An ID such as `meta-llama/llama-3.3-70b-instruct` without `:free` may be a paid route. The OpenRouter Free plan lists no budget or spend-control feature, so do not add credits or enable auto top-up while testing; check Activity for the model used and its cost. Do not add a paid model as an automatic fallback.

Control what code and personal data reach model providers

OpenRouter says it does not use inputs or outputs to train its own models, but it forwards requests to the selected model provider. Provider retention and training policies can differ, so read the provider details on the Llama 3.3 page and send only public code. [OpenRouter privacy policy](https://openrouter.ai/privacy/) · [Llama 3.3 free endpoint](https://openrouter.ai/meta-llama/llama-3.3-70b-instruct:free)

To apply a provider filter, add a `provider` setting to the request body. `data_collection: "deny"` excludes providers that collect data, while `zdr: true` selects only providers with a Zero Data Retention policy.

Stop the request if no free provider meets the filter, and do not configure a fallback to a paid model. In the Python example, add the code after creating `payload` and immediately before the `request = urllib.request.Request(` line. [Provider data collection controls](https://openrouter.ai/docs/guides/features/sovereign-ai)

payload["provider"] = {
    "data_collection": "deny",
    "zdr": True,
}

Choose what to test with the free API and when to use another route

The free API suits requests within the daily 50-call allowance, such as asking for possible bugs in public code or refining a prompt and output format.

If your request volume grows or you need consistent latency, provider selection, and retention controls, compare the current model pages and paid-plan controls. For internal code that must stay within your repository without inference charges, a local model is a better fit. Free models and providers can change, so review the [free-model collection](https://openrouter.ai/collections/free-models/) and [pricing and limits](https://openrouter.ai/pricing) before adopting a route.

Hardware-specific speed tuning and serving recipe summary

Provides weights, KV cache, prefill batch, and concurrent request with acceleration functionality, separated for balanced performance, speed, and long-context profiling.

The detail page shows each profile's command, settings, stop conditions, and measurement steps. Commands appear only for combinations with a confirmed model file and runtime.

Frequently asked questions

What should you check first in this article?

OpenRouter connects inference APIs from multiple model providers through one request format. This recipe pins Meta Llama 3.3 70B Instruct's `:free` variant to review public Python code and check the request body, free-tier limits, and provider privacy policies.

When is it better not to choose this?

Do not save the key in a shared document or repository. If it has already been exposed, revoke it in the OpenRouter dashboard and create a new key.

Where can you check the conditions and sources?

Llama 3.3 publishes its weights under the Meta Llama 3.3 Community License. OpenRouter's `:free` label means hosted API token pricing is zero. [Meta Llama 3.3 model card](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct) · [OpenRouter free endpoint](https://openrouter.ai/meta-llama/llama-3.3-70b-instruct:free)

Key takeaways

1. The current public Free Plan table lists the same request and token limits for `openai/gpt-oss-20b` and `openai/gpt-oss-120b`.
2. The Python SDK can read the key through hidden input; inspect `finish_reason` alongside the final response.

Send a request to an open-weight model through the Groq API

The Groq API accepts conversation messages and a model ID over HTTPS, then returns a response generated by a model hosted on GroqCloud. Here, a short Python request asks `openai/gpt-oss-20b` to review code, and reads both the response text and its completion status.

GPT-OSS 20B and 120B are available as Apache 2.0 open-weight models that you can also run on your own computer. The GroqCloud API uses the model IDs `openai/gpt-oss-20b` and `openai/gpt-oss-120b` for hosted inference. See [OpenAI's open-weight model overview](https://help.openai.com/en/articles/11870455-openai-open-weight-models) and the [GPT-OSS 20B model card](https://huggingface.co/openai/gpt-oss-20b).

• Groq supported models and API IDs
• Connect a Free Open-Weight Model on OpenRouter for Code Review
• How to use DeepSeek V4.1 Flash Free with OpenCode
• How much VRAM or unified memory does a local LLM need?
• Try NVIDIA Nemotron's Free API: Send a First Request to the Developer Endpoint

Choose a model within the Free Plan limits

In Groq's published Free Plan summary, the two GPT-OSS models share the same request and token limits. RPM means requests per minute, RPD requests per day, TPM tokens per minute, and TPD tokens per day.

This free table is a public organization-level summary. Check the current values for your account in the [Groq Console Limits page](https://console.groq.com/dashboard/limits). Input and generated tokens share the 8K-per-minute budget, so start with the relevant function and error description instead of pasting a long file.

• Groq rate limits by plan

Prepare an account and API key

Check that your GroqCloud account shows the Free Plan, then create a key under API Keys. Groq says a payment method is required to upgrade to the paid Developer plan. To stay within the free tier, check the plan shown in your account and review any payment or upgrade screen before proceeding. [Groq billing guide](https://console.groq.com/docs/billing-faqs)

The installation commands below are for a macOS or Linux shell. Enter the API key through a hidden prompt at runtime instead of writing it into the code, so it stays out of shell history and the source file. After saving the example in the next section, run `python groq_review.py` in the same terminal with the virtual environment active. Another Python environment may not find the installed `groq` package.

python3 -m venv .venv
source .venv/bin/activate
python -m pip install groq

Send a short code-review request

Save the following as `groq_review.py` and run it. `getpass` reads the API key without echoing it to the screen, then the SDK sends a JSON request to the chat completions endpoint.

from getpass import getpass
from groq import Groq

client = Groq(api_key=getpass("Groq API key: "))
completion = client.chat.completions.create(
    model="openai/gpt-oss-20b",
    messages=[
        {
            "role": "user",
            "content": (
                "Review this function for the empty-list edge case and suggest a fix:\n\n"
                "def average(values):\n"
                "    return sum(values) / len(values)"
            ),
        }
    ],
    reasoning_effort="low",
    include_reasoning=False,
    max_completion_tokens=2048,
)

choice = completion.choices[0]
print(choice.message.content or "(No final text was returned)")
print(f"finish_reason={choice.finish_reason}")
print(f"completion_tokens={completion.usage.completion_tokens}")
if choice.finish_reason == "length" and not choice.message.content:
    print("The token limit may have been used before final text; shorten the request or raise max_completion_tokens.")

python groq_review.py

• Groq Python SDK quickstart

Read the response text and finish status together

A successful response places the code-review text in `choices[0].message.content`; `finish_reason=stop` means the model completed its reply. The output is model-generated text, so read it against the actual input and function behavior before applying a change.

`finish_reason=length` means the configured generation-token limit was reached. GPT-OSS also spends the response budget on reasoning, so if the final text is empty, shorten the request or raise `max_completion_tokens` and try again. Keep the 8K TPM limit in mind and size the output cap for the answer you need.

• GPT-OSS reasoning options and token settings

Distinguish authentication, model, and limit errors

For a `401`, re-enter the key and check in API Keys that it is still active. If a `400` reports a model error, compare the exact spelling `openai/gpt-oss-20b` with the [current supported-model list](https://console.groq.com/docs/models).

A `429` means the request or token limit per minute or per day has been reached. Check the `retry-after`, `x-ratelimit-remaining-requests`, and `x-ratelimit-remaining-tokens` headers, then wait before retrying or reduce input and output length. The [Rate Limits documentation](https://console.groq.com/docs/rate-limits) and Console Limits page show the limit names and current remaining amounts.

Choose where the request belongs

Groq says customer inputs and outputs from ordinary inference requests are not retained by default. It may temporarily log them for up to 30 days when needed for reliability work or abuse investigations, and it collects usage metadata separately. Zero Data Retention can be configured in Data Controls; the [Groq data guide](https://console.groq.com/docs/your-data) explains how this affects features that require storage.

For sensitive company code or customer information, check your organization's transfer and processing rules first and send only the relevant excerpt. If you need to process data on your own computer without an internet connection, download the open weights and use a local runtime; to try a hosted API without handling model files, use GroqCloud. Compare other providers in the [free open-weight AI services guide](/guides/free-open-weight-ai-services-guide).

Hardware-specific speed tuning and serving recipe summary

Provides weights, KV cache, prefill batch, and concurrent request with acceleration functionality, separated for balanced performance, speed, and long-context profiling.

The detail page shows each profile's command, settings, stop conditions, and measurement steps. Commands appear only for combinations with a confirmed model file and runtime.

Frequently asked questions

What should you check first in this article?

GroqCloud hosts open-weight models and returns responses through an API. This guide covers a Python request to GPT-OSS 20B, the published free limits, secure key entry, response endings, and data-retention conditions.

Where can you check the conditions and sources?

GPT-OSS 20B and 120B are available as Apache 2.0 open-weight models that you can also run on your own computer. The GroqCloud API uses the model IDs `openai/gpt-oss-20b` and `openai/gpt-oss-120b` for hosted inference. See [OpenAI's open-weight model overview](https://help.openai.com/en/articles/11870455-openai-open-weight-models) and the [GPT-OSS 20B model card](https://huggingface.co/openai/gpt-oss-20b).

Key takeaways

1. The current catalog lists Nemotron 3.5 Lightning as both a Free Endpoint and Download Available.
2. Check the hosted API trial terms separately from the OpenMDW license for downloaded weights.

Send a text request and the hosted model returns an answer

NVIDIA API Catalog is a developer service for trying models hosted on NVIDIA servers through an API. Browse models and examples in the catalog, then send a question from Python using an API key and model ID. Here, we'll ask Nemotron 3.5 Lightning a short question and inspect its answer and `finish_reason`.

• NVIDIA Nemotron API Catalog and code sample
• Try GPT-OSS with Groq's Free API: From Account Setup to Your First Response
• Start with Mistral Studio’s Free API: Send a Request to Small 4
• How much VRAM or unified memory does a local LLM need?

Check the free endpoint and model terms

The current catalog page for `nvidia/nemotron-3.5-lightning-30b-a3b` lists both Free Endpoint and Download Available. Use `nvidia/nemotron-3.5-lightning-30b-a3b` as the hosted API model ID; the public weights are also available on Hugging Face. The API Trial Terms govern use of the hosted endpoint, while OpenMDW License 1.1 applies to the model materials.

Nemotron 3.5 Lightning is a text model with 3B active parameters out of 30B total. Its model card explicitly lists English and Japanese among its supported languages, but not Korean. If your task requires Korean output, first check quality with a short, non-sensitive prompt.

• Nemotron 3.5 Lightning model card
• Public NVFP4 weights and OpenMDW license