memory and model
Last updated
Strata PC memory guide: choosing a model with 32, 48, 64, or 96 GB RAM
RAM must hold more than model weights; leave room for the operating system and context too.
Strata usually keeps all experts in RAM1 and caches frequently used copies on the GPU2. Resident low-RAM mode copies non-GPU experts into RAM; mmap mode maps the file and reads experts when needed. At 32 GB RAM, candidates are Coder or low-RAM Q2_0/IQ2_XS depending on GPU capacity; at 48 GB, IQ2_XS or Q2_0; at 64 GB, IQ2_XS and larger sizes; and at 96 GB or more, IQ3_S or UD-IQ4_XS. Budget for the operating system, apps, context KV cache3, and work handled by the CPU4 and NVMe5 drive.
VRAM holds expert cache; RAM supports CPU work
Strata runs Qwen3.8-Flash-Next by placing some weights in GPU memory (VRAM6) and distributing expert work across system RAM and the CPU. In the default configuration, RAM holds all experts while the GPU caches copies of frequently used ones. For each token7, the GPU computes experts in its cache; the CPU computes experts in RAM that are not on the GPU, alongside GPU work.
A larger GPU can cache more experts in VRAM and reduce the CPU's share. System RAM is also needed by the operating system, apps such as browsers, the context KV cache, and model loading. Strata's model guide adds about 10 GB beyond expert requirements when estimating RAM. Leave room for other processes instead of comparing only the model file size with RAM.
Resident and mmap handle low RAM differently
Low-RAM resident mode copies the experts not held by the GPU into RAM at startup. It does not read expert weights from disk while answering. The engine cannot use this variant if RAM cannot hold the non-GPU experts plus about 4 GB of headroom. Strata documents Q2_0, IQ2_XS, and Coder in resident mode on a 32 GB RAM, 24 GB VRAM PC.
Mmap mode maps the expert file into memory addresses and reads parts as needed. The operating system can reclaim cached file data when RAM is short, but if a frequently needed expert is absent from cache, it must be read from the SSD8 again. With 12–16 GB VRAM and 32 GB RAM, Coder may fit resident while a size such as IQ3_XXS that exceeds available RAM may use mmap. If heavy SSD reads coincide with slow replies, choose a smaller model or close other apps.
Narrow model choices by RAM capacity
Strata's RAM guide combines model size with GPU capacity. This table summarizes a starting model and its expert-weight capacity for 32, 48, 64, and 96 GB RAM systems.
| System RAM | Starting model | Configuration notes |
|---|---|---|
| 32 GB | Coder | Starting point for a typical GPU setup; with 24 GB VRAM, low-RAM mode can also run Q2_0 or IQ2_XS |
| 48 GB | IQ2_XS or Q2_0 | IQ2_XS is balanced; Q2_0 is faster |
| 64 GB | IQ2_XS; IQ3_XXS or IQ3_S also fit | Keep other apps to a minimum for IQ3_S |
| 96 GB+ | IQ3_S or UD-IQ4_XS | UD-IQ4_XS can keep all experts in RAM from about 80 GB RAM |
The RAM tiers are starting points. Allow about 10 GB for the OS and other programs; low-RAM choices on a 32 GB PC depend on GPU capacity. Check the remaining headroom in Monitor with your usual context and apps.
Context cache and image input also use memory
Context length9 limits the prompt10 and output the model can handle in one conversation. The KV cache grows as the conversation gets longer. At 64K and above, settings can stream part of the cache through RAM and keep only frequently read portions in GPU memory, changing RAM and VRAM use. If memory is short, reduce context first and choose a length based on whether you need short prompts or long documents.
Enabling image input adds a vision encoder that uses VRAM and can reduce the space left for expert cache. If you do not need images, turn the option off during setup; if you do, close other GPU apps to free space. If replies stop after loading on a low-RAM system, lower context and image settings one at a time to see which allocation is short.
CPU and NVMe storage handle work beyond GPU memory
The CPU computes experts that are not in the GPU cache. A recent desktop CPU with AVX2 is the baseline; some CPUs with AVX-512 can help with CPU-side expert computation. CPU capability and RAM bandwidth affect the speed of expert work outside the GPU, so check RAM channel configuration and actual memory speed in system tools.
Strata stores a per-layer embedding table on the SSD and reads needed rows for each token. This is separate from resident copies of expert files in RAM. NVMe storage speeds initial model loading and PLE11 row reads. In mmap low-RAM mode, frequent expert reads from disk also depend on SSD performance and the RAM file cache. If disk activity stays high while output slows, consider a smaller model or a GPU and RAM combination that supports resident mode.
Terminology notes
System RAM — System memory that temporarily holds data while programs run.
Back to the textGPU — A processor designed to handle many calculations in parallel. It performs model computations during AI inference.
Back to the textKV cache — Memory that stores attention keys and values from earlier tokens for reuse during later token generation. Its size depends on context length and batch size.
Back to the textCPU — The central processor that runs the operating system and general-purpose instructions; in AI workloads it also handles tasks such as preprocessing and data movement.
Back to the textNVMe — A storage protocol used by SSDs communicating over PCI Express.
Back to the textVRAM — Memory used by a graphics card’s GPU to store model weights and intermediate values.
Back to the textToken — A unit into which a model divides input or output for processing.
Back to the textSSD — A data storage device that uses flash memory and retains data without power.
Back to the textContext window — The token span of input and generated content a model can handle in one request. The supported limit and memory use depend on the model and runtime settings.
Back to the textPrompt — The input text sent to a model. For generative models it describes the desired result and strongly shapes the output.
Back to the textPLE embedding table — Short for Per-Layer Embedding, a structure that adds embedding information at particular model layers. The Qwen Flash-Next family uses n-gram tables; supported runtimes can offload these tables to RAM or SSD.
Back to the text
Frequently asked questions
- What should you check first in this article?
- Strata usually keeps all experts in RAM and caches frequently used copies on the GPU. Resident low-RAM mode copies non-GPU experts into RAM; mmap mode maps the file and reads experts when needed. At 32 GB RAM, candidates are Coder or low-RAM Q2_0/IQ2_XS depending on GPU capacity; at 48 GB, IQ2_XS or Q2_0; at 64 GB, IQ2_XS and larger sizes; and at 96 GB or more, IQ3_S or UD-IQ4_XS. Budget for the operating system, apps, context KV cache, and work handled by the CPU and NVMe drive.
- How much memory does it need?
- Strata runs Qwen3.8-Flash-Next by placing some weights in GPU memory (VRAM) and distributing expert work across system RAM and the CPU. In the default configuration, RAM holds all experts while the GPU caches copies of frequently used ones. For each token, the GPU computes experts in its cache; the CPU computes experts in RAM that are not on the GPU, alongside GPU work.