How much memory does it need?

The 24 GB configuration budgets memory for batch size 1 and a 128K context. NVIDIA recommends an actual runtime context of either 65,536 or 262,144 tokens, depending on GPU memory.

Where can you check the conditions and sources?

Precision differs by component. Mamba linear layers use NVFP4, while the lm_head, token embeddings, attention, and Latent MoE weights use Q8_0. The GGUF file list in the model card lets you add up storage for the main model, image projector, and drafter you plan to use.

new model · 2026.10.16

Can you run the 74B Nemotron 3.5 Bolt model in 24 GB?

Follow the model setup and official profile, verify a first response, and check memory use on your system.

Nemotron 3.5 Bolt 74B-A9B is a compressed hybrid language and vision model. This guide walks through downloading it and running NVIDIA's llama.cpp recipes for different GPU¹ memory tiers.

Nemotron 3.5 Bolt 74B-A9B is a local language model designed to answer prompts² and tool results and handle multi-step agent tasks. 74B is its total parameter count; A9B describes the approximate number activated for a token³. Rather than training a small model from scratch, NVIDIA compressed the structure of Nemotron 3.5 Super VL and heavily quantized its weights for smaller systems. NVIDIA model overview

Download the official GGUF⁴ and run it with llama.cpp, then check the recommended context lengths by GPU memory tier and verify a first response. The release separates the main model, image projector, and DFlash draft model, so choose the components required for your task. Official GGUF model card

Contents

Requirements and key details

74B is the total parameter count, A9B the active parameter scale per token; each GGUF file has its own listed size.
Expert weights use IQ2_XXS, Mamba linear layers use NVFP4, and selected components use Q8_0.
Memory planning must include context state, runtime buffers, and other applications in addition to model weights.

Compressing expert weights in the 74B model to 2 bits

NVIDIA Nemotron 3.5 Bolt is a 74B model that takes prompts such as questions or code and generates a response. For the 24 GB configuration, most expert weights use IQ2_XXS at 2.0625 bits per element; NVIDIA applies quantization-aware distillation (QAD) to recover quality at the lower precision.

The 24 GB configuration budgets memory for batch size⁵ 1 and a 128K context. NVIDIA recommends an actual runtime⁶ context of either 65,536 or 262,144 tokens, depending on GPU memory.

Precision differs by component. Mamba linear layers use NVFP4⁷, while the lm_head, token embeddings, attention⁸, and Latent MoE⁹ weights use Q8_0. The GGUF file list in the model card lets you add up storage for the main model, image projector, and drafter you plan to use.

For each token, the router selects some experts and computation also passes through Mamba and Transformer¹⁰ blocks. A9B refers to the parameter scale activated per token. When comparing speed, record memory bandwidth¹¹, runtime kernels, GPU offload¹², input length, and concurrent requests.

A monitor, compact computer, and desktop tower on a workstation — Load the model on your PC and verify its first response.

What to check first on a 24 GB PC

Plan memory separately for GPU and CPU¹³ placement. Compare the weights, KV cache¹⁴, and work buffers placed on the GPU with available VRAM¹⁵. Check CPU-resident model parts alongside the system RAM¹⁶ used by the OS, browser, and editor. Changing GPU-layer offload changes the balance between both memory pools.

Before running it, check the exact GGUF filenames and download sizes in the model card, including any drafter files. Your llama.cpp build must support Mamba and the relevant quantization¹⁷ formats. An older runtime can fail to load the model for reasons unrelated to memory, so compare the model-card instructions with your installed runtime release. NVIDIA's published Bolt test path used llama.cpp on Windows 11 ARM x64.

On an RTX desktop, the runtime may place all layers in GPU memory or offload some to system memory. Partial offloading can make execution possible, but it changes the memory path and may substantially change throughput¹⁸. Start with a short context and one request. Watch dedicated GPU memory, system memory, and the runtime log; once the model answers reliably, increase input length to isolate the next limit.

Get a first answer with llama.cpp

On Windows, install llama.cpp from PowerShell with winget install llama.cpp. Run the quick-start command to download the model and open its local chat UI at http://127.0.0.1:8080. Send a short prompt, confirm a response, and stop the server with Ctrl+C. For an RTX 5090, launch the official profile below separately. NVIDIA GGUF model card

During the first run, the terminal shows the model name and download progress. After loading, the server prints a ready message and address. Open it in a browser on the same PC and send: Summarize this sentence in one sentence: The model receives a prompt and generates an answer. A relevant summary confirms text inference¹⁹ is working. For speed tests, record file loading, time to first token²⁰, and generation token/s separately.

The RTX 5090's 32 GB VRAM fits NVIDIA's 32 GB+ discrete-GPU tier. Download the main GGUF, mmproj, and LiLiCorr drafter with the model-card filenames, set their PowerShell paths, and start the profile with a 65,536 context, one sequence, up to six DFlash draft tokens, and Q8_0 KV cache. Quick start downloads automatically; this profile loads the three files directly. Stop the quick-start server with Ctrl+C before starting this one. NVIDIA Windows recipe

If the first run reports unsupported tensor, compare your llama.cpp version with the formats supported by the model card. For a CUDA²¹ allocation error, reduce context and batch size first, then check dedicated and shared GPU memory in Windows Task Manager. If the server starts but the browser cannot connect, confirm the terminal's port is 8080 and open 127.0.0.1 on that PC.

Run the llama.cpp Windows quick start

winget install llama.cpp
llama serve -hf nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF --host 127.0.0.1 --port 8080

Let the model download, test a short prompt at `http://127.0.0.1:8080`, then press Ctrl+C in the terminal to stop the server.

Download the GGUF files for the RTX 5090 profile

python -m pip install --upgrade huggingface_hub
$env:MODEL_DIR = Join-Path $PWD.Path "models\nemotron-bolt"
hf download nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF `
  NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf `
  mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf `
  --local-dir "$env:MODEL_DIR"
hf download nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr-GGUF `
  NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr.gguf `
  --local-dir "$env:MODEL_DIR"
$env:MODEL = Join-Path $env:MODEL_DIR "NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf"
$env:MMPROJ = Join-Path $env:MODEL_DIR "mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf"
$env:SPEC_DRAFT = Join-Path $env:MODEL_DIR "NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr.gguf"

This block downloads the profile files and sets their paths. After downloading, run the next server command in the same PowerShell session.

NVIDIA's official 28 GB server profile for an RTX 5090

$env:MODEL_DIR = Join-Path $PWD.Path "models\nemotron-bolt"
$env:MODEL = Join-Path $env:MODEL_DIR "NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf"
$env:MMPROJ = Join-Path $env:MODEL_DIR "mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf"
$env:SPEC_DRAFT = Join-Path $env:MODEL_DIR "NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr.gguf"
llama-server.exe `
  --model "$env:MODEL" `
  --mmproj "$env:MMPROJ" `
  --batch-size 2048 `
  --ubatch-size 512 `
  --ctx-size 65536 `
  --parallel 1 `
  -ngl 999 `
  --spec-type draft-dflash `
  --spec-draft-n-max 6 `
  --spec-draft-model "$env:SPEC_DRAFT" `
  --spec-draft-ngl 999 `
  --cache-type-k q8_0 `
  --cache-type-v q8_0 `
  --temp 1.0 `
  --top-p 0.95 `
  --flash-attn on `
  --jinja `
  --host 127.0.0.1 `
  --port 8080

Place the exact GGUF files in `models\nemotron-bolt` and run only this server. Press Ctrl+C after testing to stop it.

An open rack-server chassis showing its internal modules — Plan runtime memory for both weights and context state.

Published generation and agent-task speeds on RTX Spark

NVIDIA compared models on an RTX Spark in a 128 GB Surface Laptop Ultra, using Windows 11 ARM x64 and llama.cpp. AIPerf SPEED-Bench-Coding measures generation speed; AA-AgentPerf Local replays multi-step tasks. The model results and conditions are collected in the table in the companion guide, ‘Nemotron Bolt vs Qwen3.8-27B.’ Here, distinguish what each metric represents.

TPS is token throughput for generating against a fixed input; replay duration is the completion time for a defined set of agent tasks that includes tool use. For short code completion, look first at TPS and time to first token. For workflows chaining several tools, start with end-to-end replay time.

Rather than applying the published figures to your PC, choose one task you actually plan to run and use the same prompt, document, and tools. First record answer quality and time to first token with a short input. Then add your usual document and repeated tool calls²² and compare total time, VRAM, and failures to see which setting needs to change.

See the full Bolt and Qwen3.8 results table

When it belongs on your shortlist

Bolt is worth evaluating if you want a model with a 70B-class total size for agent tasks on RTX-class hardware or DGX Spark instead of a data-center server. If your use case involves terminal actions, web browsing, or multi-step planning like NVIDIA's evaluation tasks, first match the model card's prompt format. Quality and tool-call reliability depend on your own instructions, tools, and safety settings, so validate each task in a sandbox²³ before connecting it to real automation.

To test Bolt on a 24 GB card, first check whether the CPU/RAM-first command returns an answer, then add GPU layers to the working setup. Change one setting at a time to distinguish model-load failures from GPU-memory limits. The 3090 guide contains the concrete starting command.

A small computer on a desk with server racks behind it — Test context and request count on the PC you plan to use.

Terminology notes

GPU — A processor designed to handle many calculations in parallel. It performs model computations during AI inference.
Back to the text
Prompt — The input text sent to a model. For generative models it describes the desired result and strongly shapes the output.
Back to the text
Token — A unit into which a model divides input or output for processing.
Back to the text
GGUF — A file format for model data, widely used by llama.cpp-based tools.
Back to the text
Batch size — The number of inputs or requests processed together in one batch.
Back to the text
Runtime — The software environment that provides facilities needed while a program runs. In local AI, it can also refer to a model execution engine.
Back to the text
NVFP4 — A 4-bit floating-point data format defined by NVIDIA. Support depends on GPU generation, model, and software implementation.
Back to the text
Attention — A neural-network operation that calculates relationships between input tokens and combines information from each position into the current representation.
Back to the text
MoE — A model architecture that selects some of several expert subnetworks for each input. Total parameters can differ from the number activated for one token.
Back to the text
Transformer — A neural-network architecture centered on attention, which computes relationships within an input. It is used in language as well as image, audio, and video models.
Back to the text
Memory bandwidth — The amount of data transferable between memory and a processor per unit time, usually expressed in bytes per second (B/s).
Back to the text
Offloading — Moving some model data from GPU memory to system RAM or storage when capacity is limited. This adds data transfer.
Back to the text
CPU — The central processor that runs the operating system and general-purpose instructions; in AI workloads it also handles tasks such as preprocessing and data movement.
Back to the text
KV cache — Memory that stores attention keys and values from earlier tokens for reuse during later token generation. Its size depends on context length and batch size.
Back to the text
VRAM — Memory used by a graphics card’s GPU to store model weights and intermediate values.
Back to the text
System RAM — System memory that temporarily holds data while programs run.
Back to the text
Quantization — A way to represent model weights or values with fewer bits, reducing storage and memory use. The coarser representation can also affect accuracy.
Back to the text
Throughput — The amount of work processed or generated over time. Comparisons need the unit, such as tokens per second or requests per second.
Back to the text
Inference — The process of using a trained model to compute an output for an input. Here, local inference means running the model on the user’s device.
Back to the text
Time to First Token — The time from sending a request until its first output token arrives.
Back to the text
CUDA — A software platform for general-purpose computing on NVIDIA GPUs.
Back to the text
Tool call — A structured request from a model for an external function such as reading a file, searching, or running a command. The agent runtime and its permission settings decide whether the request is actually executed.
Back to the text
Sandbox — An isolated environment that restricts a program’s access to files, networks, and host capabilities. Its protection depends on mount, network, and execution-permission settings.
Back to the text

Frequently asked questions

What should you check first in this article?: Nemotron 3.5 Bolt 74B-A9B is a compressed hybrid language and vision model. This guide walks through downloading it and running NVIDIA's llama.cpp recipes for different GPU memory tiers.
How much memory does it need?: The 24 GB configuration budgets memory for batch size 1 and a 128K context. NVIDIA recommends an actual runtime context of either 65,536 or 262,144 tokens, depending on GPU memory.
Where can you check the conditions and sources?: Precision differs by component. Mamba linear layers use NVFP4, while the lm_head, token embeddings, attention, and Latent MoE weights use Q8_0. The GGUF file list in the model card lets you add up storage for the main model, image projector, and drafter you plan to use.