Local AI technology/purchase guide

Where to start

Start by choosing what you want to do. You can start conversations and document work in the LLM equipment guide, and start with the image and video creation guide for pictures and short videos, then continue with the memory and model articles.

new model

• Qwen3.8-27B: can you run it on a 32GB computer?

  We summarize the 27B dense structure, Q4 memory requirement, 256K context, image/video input, and MTP of the Qwen3.8-27B open weight model from the perspective of local equipment selection.

• Qwen3.8-Flash-Next: why 6B active still needs 180B resident

  Distinguish between Qwen3.8-Flash-Next's 180B resident weights, 6B active MoE, 51B PLE table and 4B MTP, and illustrate the feasibility of DGX Spark and 256GB equipment.

• Gemma 4: choosing 12B or 26B-A4B for your hardware

  Focusing on the 12B dense type and 26B-A4B MoE of the Gemma 4 open weight model, we explain multimodal input, 256K context, Q4 memory, and 16GB·24GB equipment selection.

• GLM-5.3 and DeepSeek-V4: open weights, but for a personal PC?

  GLM-5.3-Flash 320B and DeepSeek-V4-Flash-0731 304B explain the differences between openweight and private local execution, activation parameters, distributed loading, and compressed version conditions.

read first

• Choosing local LLM hardware: memory, speed and power

  We explain in what order you should look at VRAM and integrated memory, prefill, token generation speed, electricity costs and TCO when choosing local LLM equipment.

• Prefill, TTFT and decode: three different measures of speed

  This is a method to read the actual performance by distinguishing between prefill throughput, first token time (TTFT), and decode token generation speed in the local LLM benchmark.

Model setup recipes

• Qwen3.8-27B setup: oMLX on Mac and llama.cpp on RTX

  When running Qwen3.8-27B on Mac and RTX, the Q4 weight, KV cache, prefill placement, MTP, context length, and oMLX·llama.cpp execution command are provided for each device.

• Qwen3.8-Flash-Next: a single DGX Spark SGLang recipe

  Qwen3.8-Flash-Next SGLang setup for running NVFP4 on one DGX Spark. Distinguish between NVMe PLE mmap, prefill chunks, memory ratio, NEXTN MTP and 262K long context configuration.

• Gemma 4 12B: LM Studio and vLLM on 16GB hardware

  We'll walk you through how to serve 16GB to 24GB single GPU and Mac desktops on the Gemma 4 12B model, Q4 GGUF and official google/gemma-4-12B-it weight settings, and 8K safety context and API validation.

• Gemma 4 26B-A4B: a serving recipe for 24–32GB hardware

  We describe Gemma 4 26B-A4B with a total of 25.2B parameters (3.8B active), distinguishing between the Q4 conversion path running on 24GB to 32GB devices and the official google/gemma-4-26B-A4B-it server path.

• Qwen3.6 35B-A3B: vLLM setup and the qwen3 parser

  Total 35B (3B active) Qwen3.6 Cleans up 35B-A3B MoE model's 24GB/32GB+ memory condition, vLLM >= 0.19.0 reasoning parser settings, MTP advanced options and 127.0.0.1 local serving recipe.

• GLM-5.3-Flash: serving a server-scale 320B MoE

  320B total (18B active) GLM-5.3-Flash Official checkpoints are differentiated between vLLM·SGLang settings serving multiple accelerators and memory requirements when reviewing compressed renditions.

• DeepSeek-V4-Flash: distributed serving for a 304B model

  Total 304B (13B active) DeepSeek-V4-Flash-0731 organizes the official 4×GB300 vLLM paths and SGLang paths, 16K startup context, and local API resolution procedures.

Image / video

• Local image generation: what can your computer create?

  We explain how to edit drafts, product images, posters, and reference images by creating local images, and the criteria for selecting equipment based on resolution, longevity, and memory.

• FLUX or Qwen Image: choose a model for the job

  FLUX.2 Klein·Dev and Qwen-Image-2512 are explained in terms of quick drafting, text expression, reference image editing, and precise results.

• Local video generation: from text or an image to a clip

  We explain how to create short videos from text or still images using the latest local video models such as Wan 2.2 and MiniMax, and equipment selection based on resolution, frame, and sound.

• Wan 2.2 and MiniMax: choosing a local video workflow

  Describes the features, camera motion, simultaneous audio creation, and required memory standards of Wan 2.2 TI2V-5B·T2V-14B and MiniMax H3-Base.

• How much VRAM do image and video generation need?

  We explain how the model body, text encoder, VAE, and intermediate workspace use VRAM in image and video generation, and the performance range from 12GB to 64GB+.

Response speed and acceleration

• Local AI speed report — September 2026

  As of September 2026, we compare the prefill and token generation speeds of major Macs, NVIDIA GPU, DGX Spark, and AMD integrated memory equipment under the same conditions.

• Long prompts can reverse a local LLM speed ranking

  We explain why the decode ranking of short chats and the ranking of prefill/total completion times for long documents are different from the perspective of actual equipment selection.

• MTP: how multi-token prediction works and when to use it

  We explain how Multi-Token Prediction (MTP) proposes and verifies multiple future tokens, why you need a supporting model and runtime, and when it gets faster.

• Lightning MTP: understanding acceleration in oMLX

  Explains how Lightning MTP utilizes the built-in MTP head of supported models in oMLX for Apple Silicon, compatibility conditions, differences from regular MTP, and precautions.

• Speculative decoding: drafting tokens and verifying them

  We explain how Speculative Decoding accelerates creation by creating candidate tokens with a small draft model or built-in prediction head, and then bundling and verifying them with the main model.

• Prompt Lookup and n-gram speculative decoding explained

  Describes how Prompt Lookup finds and validates subsequent tokens in repeated phrases (n-grams) of the prompt and creation history without an additional draft model, and how it differs from model weight offloading.

• Qwen3.8-Flash-Next: offloading PLE tables to RAM or SSD

  We summarize the principle of offloading 51B PLE embedding table of Qwen3.8-Flash-Next's 180B resident weight to RAM and NVMe SSD, compatibility conditions, page cache impact, and inspection checklist.

• SpecPrefill: reducing the wait before the first token

  We explain how SpecPrefill reduces time to first token (TTFT) on long inputs, the Mac (oMLX)/NVIDIA experiment path, answer quality variability, and rough estimate criteria.

• LLM memory bandwidth vs compute performance

  We describe how memory bandwidth and GPU compute performance affect prefill, single-user decode, and batch processing, respectively, in local LLM.

memory and model

• Why a 35B MoE can be faster than a smaller dense model

  By distinguishing between total and active parameters, we describe the combinations and memory conditions under which large MoE models are generated faster than small Dense models.

• How much VRAM or unified memory does a local LLM need?

  This is a method to determine the VRAM and integrated memory required for 12B, 27B, 35B, and 70B level local LLM, including quantization, runtime margin, KV cache, and context length.

• KV cache: the memory your conversation needs

  We explain how the KV cache saves the attention key and value of the previous token, reducing repetitive calculations, and the relationship between context length, memory usage, and quantization.

• Prompt caching: which part of the wait does it remove?

  We describe the conditions and limits of how repeating system prompts and prompt caches and prefix caches that reuse the computational state of common documents reduce TTFT.

• LLM quantization: choosing Q4, Q8 or FP16

  We explain how memory, speed, and quality change when reducing the weights of a local LLM to low bits such as Q4 and Q8, and the criteria for selecting a quantization format.

• MoE vs dense LLMs: total and active parameters

  Explains why the total parameters of the Mixture of Experts model and the parameters activated per token are different and how they affect memory and token speed on local devices.

• Context length, RoPE and sliding-window attention

  We explain what a model's maximum context length means and how long contexts affect prefill time, KV cache, RoPE scaling, sliding window, and answer quality.

Executable programs and extensions

• What actually improves when you connect two DGX Sparks?

  Distinguish between model load, single request creation rate, and multiple request throughput for one and two DGX Spark configurations.

• GGUF vs MLX: which model format should you download?

  Local LLM explains which runtime ecosystem model files for GGUF and MLX fit into and how to check quantization notation and transformation compatibility.

• Running local LLMs across two or more GPUs

  We explain layer partitioning using two GPU VRAMs for local LLM, the difference between tensor parallelism and pipeline parallelism, and PCIe·NVLink communication bottlenecks.

• FlashAttention vs PagedAttention: what each one changes

  We explain how FlashAttention and PagedAttention, which have similar names but solve different problems, affect attention calculation, KV cache memory management, and server throughput.

• Continuous batching: throughput is not single-user speed

  We explain how continuous batching increases server throughput by putting LLM requests of different lengths in and out of running batches, and how it differs from single-user latency.

Purchases and Costs

• Local LLM ownership cost over one or two years

  This method calculates the actual total cost of ownership by adding up the local AI hardware purchase price, depreciation, estimated sale price, electricity bills, maintenance, and cloud subscription savings.