Local AI Guide

Local AI technology/purchase guide

Explore local LLMs and image, video and music generation: how to start, choose a model, and understand memory and hardware costs.

Start with your question

What is a local LLM? From installation and PC specifications to model selection and speed checks

Learn what a local LLM does, choose Ollama or LM Studio, check RAM and VRAM, and pick a model. Try a real browser chat and compare hardware response times.

Local AI music generation: make songs and background music on your computer

Choose a local music workflow for your PC or Mac: YuE2, ACE-Step and HeartMuLa, lyrics and prompts, memory requirements, generation time and refining your first track.

ComfyUI first image: install, load a model and fix common errors

Install Comfy Desktop, locate the SD 1.5 checkpoint and open a downloadable 512px workflow, with checks for missing models and memory errors.

Mac mini 24GB or 32GB: is more memory worth it for local AI?

A buying guide to Mac mini memory that weighs document length, the apps you keep open and replacement costs—not model size alone.

More starting points

I Have an RTX 3090—Should I Upgrade My Graphics Card for Local AI?

Reduce memory shortages and waiting when running Qwen3.8-27B on an RTX 3090. Compare the cost and response speed of keeping, adding, or replacing a GPU.

Local AI for PDFs: check the document before buying more memory

Find whether PDF problems come from extraction, OCR, context length or retrieval before changing hardware.

A computer for local coding AI: autocomplete and agents need different tests

Choose local coding hardware by separating autocomplete, function edits and repository-level tasks, then compare through passing tests.

Local LLM slow on its first run? Five checks before blaming the hardware

Use the app’s status and a short prompt to distinguish downloading, model loading, and slow response generation.

Before buying a GPU for ComfyUI, decide on one image you need to make

A way to choose local image-generation hardware by separating base generation, partial edits and upscaling, then checking time and memory across the whole task.

Follow new articles via RSSAdd as a preferred source on Google

Where to start

Start with the task: conversation and documents, images and video, or music. Then check the memory and software requirements of the model you choose.

new model

Can you run the 74B Nemotron 3.5 Bolt model in 24 GB?

A guide to the IQ2_XXS Nemotron 3.5 Bolt 74B-A9B build, local runtime paths, and memory planning for the files and context you load.

LFM2.5-24B-A2B local LLM: a Korean-capable 24B MoE on your own PC

Liquid AI's LFM2.5-24B-A2B is a Korean-capable MoE model with 24B total and 2.45B active parameters. This guide covers the official GGUF and MLX 4-bit files, the 131K context, the vendor's Apple M5 Max 184 tok/s claim, and what to verify on your own machine.

Flint local LLM: can an open 40B coding agent run on a personal PC?

Recursive's Flint is a coding-agent MoE model with 39.8B total and 2.5B active parameters. Its official runtime is SGLang on B300 or H200 GPUs, with quantized GGUF builds published separately. This covers the vendor's SWE-bench conditions and what to check on personal hardware.

KRAFTON Raon-1 mini local LLM: a Korean 30B MoE on one RTX 5090 32GB

Raon-1 mini is a Korean local LLM trained from scratch by KRAFTON AI: a 30B-A4B MoE with a 262K-token context. This covers what NVFP4 needs on a single RTX 5090 with 32GB, plus the gated access, Apache 2.0 terms and preview limitations.

DeepSeek-V4.1-Flash-1017: what to re-download and what runs it

The 1017 update keeps the API name and price while replacing all 48 weight shards. Mixing it with September files loads without an error but runs a different model, so check what to re-download and which runtimes read it.

DeepSeek 1017 vs Opus 5 vs GLM-5.3: scores and cost together

Where 1017, Opus 5 and GLM-5.3 lead on the same benchmarks, and how API pricing and local hardware requirements change the choice.

AstaBrief 8B: Create a Cited Research Report Locally

Learn how to give AstaBrief 8B a research question and retrieved paper excerpts, create a cited report, trace citations to their sources, and assess published scores and local memory needs.

Run Xing4.0 Locally: Install the Coding Model and Plan GPU Memory

Learn how to use Xing4.0-29B-A4B for coding work. Compare its 4B active experts with 29B total parameters, review official agent evaluations, and plan the GPU, context length, and runtime.

Holo4 for Local Computer Use: Model, Runner, and Permission Boundaries

Distinguish Holo4-27B and Holo4-35B-A3B architectures, licenses, and runtime paths, then test a computer-use agent safely.

K2 Horizon 0.9B to 36B-A4B: choose the right model for your hardware

Compare context length, Q4 memory, and Mac/NVIDIA device selection and speed experience of K2 Horizon 0.9B·3.7B·7B·32B·MoVA 36B-A4B.

MiMo V2.6 Pro and Flash launched together: how strong are they?

A practical reading of MiMo V2.6 Pro and Flash: their 1.02T and 309B sparse MoE designs, published coding and agent results, 1M context, multimodal input and real serving scale.

Bonsai 2 27B: what to check before the 6 GB headline

A practical look at Bonsai 2 27B's ternary weights, the 5.95 GB and 7.21 GB packs, its custom runtime, and published RTX 4090, 5090 and M5 Pro performance.

Running DeepSeek V4.1 Flash locally: the numbers beyond 552B

A buyer-focused guide to DeepSeek V4.1 Flash's 552B backbone, 196B Engram, 475.2 GiB official checkpoint and early local-runtime support.

Running Ling-3.0-flash-VL locally: what the 70.87 GiB INT4 build means

A practical guide to Ling-3.0-flash-VL's 124.85B total and 5.1B active structure, 256K context, multimodal scope, official INT4 and FP4 files, and early server-runtime requirements.

Qwen3.8-27B release: Can it run on a 32 GB system?

An overview of the open-weight Qwen3.8-27B model's 27B dense architecture, Q4 memory needs, 256K context, image and video input, and MTP from the perspective of choosing local hardware.

Qwen3.8-Flash-Next: why 6B active still needs 180B resident

Distinguish between Qwen3.8-Flash-Next's 180B resident weights, 6B active MoE, 51B PLE table and 4B MTP, and illustrate the feasibility of DGX Spark and 256GB equipment.

Gemma 4: choosing 12B or 26B-A4B for your hardware

Focusing on the 12B dense type and 26B-A4B MoE of the Gemma 4 open weight model, we explain multimodal input, 256K context, Q4 memory, and 16GB·24GB equipment selection.

GLM-5.3 and DeepSeek-V4: open weights, but for a personal PC?

Using GLM-5.3-Flash 320B and DeepSeek-V4-Flash-0731 304B, this guide explains the difference between open weights and personal local inference, active parameters, distributed loading, and the conditions for running compressed conversions.

read first

China's AI Export Controls: What Happens to the Models You Use on OpenRouter?

Understand how China's October 2026 AI technology export proposal treats API access, overseas hosting, and public models, then prepare a local LLM option on the PC or Mac you already own.

RTX Spark laptops: how do they differ from DGX Spark and MacBook Pro?

Compare RTX Spark laptops with DGX Spark and MacBook Pro by operating system, memory, and software ecosystem.

DeepSeek Harness: the agent harness that runs 1017

Installing and first-running dsh, connecting a DeepSeek key or a custom API, the values a local OpenAI-compatible server must match, and 1017 support in the 0.2.1-rc releases.

Where can you use open-weight AI for free? Five API services

Compare five services for trying open-weight AI for free, including DeepSeek, GPT-OSS and Nemotron. Review quotas, setup and data policies before choosing a guide.

Can a Mac Neural Engine run local AI? GPU, MLX, and ANE roles

What Apple Neural Engine does in speech recognition and LLMs, Core ML and MLX execution paths, and the role of M5 GPU Neural Accelerators.

Local RAG Setup Guide: Build an AI That Answers from Your Documents

Follow an example that finds and answers a question from a vacation-policy PDF, then learn the local RAG workflow from text extraction and retrieval to answer checks and hardware selection.

What Is a Local AI Agent? When It Makes Sense to Run One

A local AI agent is not the same thing as a local language model. Separate the model, agent runtime, tools, and data routes to decide whether local operation fits your recurring work and privacy needs.

Hardware for Local AI Agents: Choose by Workload, Context, and Uptime

Agent frameworks and model inference place different demands on a computer. Evaluate RAM, VRAM or unified memory, real context, tool round trips, and always-on operation before choosing a setup.

What is a local LLM? From installation and PC specifications to model selection and speed checks

Learn what a local LLM does, choose Ollama or LM Studio, check RAM and VRAM, and pick a model. Try a real browser chat and compare hardware response times.

Local AI for PDFs: check the document before buying more memory

Find whether PDF problems come from extraction, OCR, context length or retrieval before changing hardware.

A computer for local coding AI: autocomplete and agents need different tests

Choose local coding hardware by separating autocomplete, function edits and repository-level tasks, then compare through passing tests.

Transcribe Korean recordings on your own computer with whisper.cpp

A practical path from one Korean recording to a reviewed local transcript, then to a sensible hardware decision.

Granite 4.2 at home: choose between 3B, 8B and 30B

Choose among IBM Granite 4.2's three sizes by task, usable memory and the cost of the whole setup.

Choosing local LLM hardware: memory, speed and power

We explain in what order you should look at VRAM and integrated memory, prefill, token generation speed, electricity costs and TCO when choosing local LLM equipment.

Prefill vs decode: should you look at TTFT or tok/s?

For long-document summaries, start with prefill and TTFT; for long outputs, start with decode tok/s. This guide explains how the three metrics divide the wait.

llmfit: find the models your Mac can actually run

llmfit detects CPU, memory and GPU, then ranks the models and quantizations that device can actually run. It shows the assumptions behind each estimate and measures real values with bench.

LFM3-1.7B: a 1.1 GB agent model on a Mac

LFM3-1.7B is a 1.1 GB text model built for local agents. Its 256K window, 1.6 GB of KV cache at 128K, and the 64 percent task success measured on macOS, with conditions attached.

Model setup recipes

When OpenRouter Is Unavailable, How Do You Switch Your App to a Local LLM?

Prepare a compatible model in LM Studio, start its local API server, then create a separate local profile in your app and continue a code review in a new chat.

How to run Nemotron Bolt on an RTX 3090 or 5090

Use NVIDIA's official 32 GB+ profile on an RTX 5090, start with CPU and system RAM on an RTX 3090, and track memory use.

How to use DeepSeek V4.1 Flash Free with OpenCode

Connect DeepSeek V4.1 Flash Free to OpenCode for code review. Learn the exact free model ID, temporary offer, billing controls and data policy.

Connect a Free Open-Weight Model on OpenRouter for Code Review

Connect OpenRouter's free Llama 3.3 endpoint to review a public code sample. Follow the request limits, cost safeguards, provider privacy checks, and first-error fixes.

Try GPT-OSS with Groq's Free API: From Account Setup to Your First Response

Learn how to call the open-weight GPT-OSS models through the GroqCloud free API, from account and API-key setup to Free Plan limits, a Python request, response handling, and troubleshooting.

Try NVIDIA Nemotron's Free API: Send a First Request to the Developer Endpoint

Call NVIDIA API Catalog’s free Nemotron endpoint for development and testing. Prepare an API key, send a Python request, inspect the response, and review the access and input-data terms.

Start with Mistral Studio’s Free API: Send a Request to Small 4

Try the Small 4 API for a published open-weight model through Mistral Studio Free mode. Create a key, call the Python SDK, review usage limits, and find the input-data training setting.

Create a 3D asset from one photo with TRELLIS.2

Use the official TRELLIS.2 example to turn an input image into a GLB model.

Which embedding model should you use for Korean document search? Qwen3, BGE, and EmbeddingGemma 2

Compare Qwen3-Embedding, BGE-M3, and EmbeddingGemma 2 on the same PDF and questions, including retrieval quality, indexing time, vector size, and reranker latency.

Search text, photos, video, and audio in one index with Jina v5 Omni

Use the shared space of Jina v5 text-small and omni-small to keep text documents indexed while adding image, video, and audio queries. Review licensing and storage needs.

Search English documents with a small model: UIBK-Embedding-v1

Learn how UIBK-Embedding-v1 combines dense retrieval and late interaction for English documents, and track quality and storage as your corpus grows.

Find PDF pages without OCR: a ColPali and ColQwen guide

Learn how ColPali-style models retrieve PDF page images, how that differs from OCR and text extraction, and how to evaluate them on real documents.

Parse PDFs into structured output with PaddleOCR-VL 1.6

Walk through PaddleOCR-VL 1.6 PDF parsing from installation to output, and distinguish structured extraction from page retrieval.

Compare Julia, Laya, and Kev decision models in llama.cpp

Apply Julia-1, Laya, and Kev-4B to the same ticket set and compare structured decisions through /v1/systemone.

Extract fields from English text with GLiNER2.5

Run the English base checkpoint of GLiNER2.5 locally and save product, error-code, and issue fields from a support message with its source text as JSON.

Retrieve English documents with UltraBERT-Embed-150M

Review the English-only UltraBERT embedding checkpoint's context, license, and evaluation workflow, while distinguishing it from the backbone models.

Build a local AI workflow with n8n and Ollama

n8n connects file reading, PDF text extraction, local LLM summarization, and output storage. This guide links Docker-based n8n to Ollama running natively on a Mac, tests the workflow manually, and adds a schedule after checking the results.

Save and resume a local LLM workflow with LangGraph

Connect a LangGraph checkpointer to an Ollama model and save a document-question workflow's state. Reuse the same thread_id to continue the conversation, and use a SQLite file to load checkpoints after restarting the process.

A Korean LoRA SFT walkthrough: data, training, and evaluation

Prepare Korean prompts and target answers as JSONL, save an adapter with TRL SFTTrainer and PEFT LoRA, then compare base and adapted outputs on a prompt excluded from training. Qwen2.5-0.5B in the code is a small library-wiring fixture, not a current model recommendation; the guide explains how to select a current model separately.

What is GraphRAG? Cross-document relationships and a Neo4j example

What connects facts split between a contract and an acquisition notice? Learn the retrieval idea behind GraphRAG and build a small relationship graph using three fictional companies.

Add embeddings and a reranker to Korean PDF search

Retrieve top candidates with BGE-M3 embeddings, then rerank them with BGE reranker v2-m3 using the same documents and queries. Compare relevant-document rank and latency before and after reranking to decide whether the models help with your PDFs.

A task-harness design example for a local coding agent

Connect a local coding agent's edit scope and tests with a small JavaScript example. Combine project instructions, permissions, a failing test, and the fixed result, then review the final diff yourself.

How do you start unlocking CMP 170HX memory on Linux?

Follow the cmpunlocker maintainer README for Linux x86-64 prerequisites, 8GB/10GB profile selection, post-reboot memory and PCIe checks, and removal or rollback.

How Can You Run Qwen3.8-27B Fastest on a CMP 170HX?

Compare measured vLLM DFlash2 and open-jet megakernel results on one CMP 170HX, with pinned Qwen3.8-27B serving references and tested context limits.

Qwen3.8 27B in 7.89 GB? Install Saluki and Plan for an 8 GB GPU

Underdog Saluki packs Qwen3.8-27B into a 7.89 GB GGUF. Set it up on Mac or NVIDIA, send a first prompt, plan memory for an 8 GB GPU, and explore tool calls and image input.

Install Strata on Windows or Linux and send a first prompt

Check GPU drivers and disk space before installing Strata 0.1.43, prepare the model with START-HERE.bat on Windows or setup.sh on Linux, and verify a first prompt. Includes first checks for download, port, and memory errors.

Run Jev in Ollama: get classifications and probabilities instead of prose

Send choice, true/false and score questions to Nimble or Tev through Ollama 0.35.0's /v1/systemone, then use the returned probabilities to route support tickets.

Run a local LLM across Macs: MLX, JACCL and performance

Distribute a model across Macs with MLX, MLX-LM and JACCL, then check connectivity and speed. Covers Thunderbolt RDMA preparation and configuration-specific Muse Spark Open measurements.

Qwen3-Omni Local Setup Conditions: Testing Audio and Video Paths

Review Qwen3-Omni thinker/talker roles and audio/video paths, then run an official Transformers audio example on supported Linux CUDA hardware.

Run a llama.cpp MTP Server on DGX Spark: Qwen3.6 Agent Requests

Build llama.cpp with CUDA on DGX Spark and connect the Qwen3.6-35B-A3B MTP GGUF to an OpenAI-compatible endpoint. Configure conversation history to preserve thinking, verify a normal response, and troubleshoot memory and connectivity.

LFM2.5-VL-3B DSpark: more than 2× faster visual responses on Mac

We summarize the OCR·image understanding purpose of LFM2.5-VL-3B, M5 Max·M3 Ultra official DSpark speed, and MLX-VLM·llama.cpp·SGLang execution conditions.

Run a 3B Model with MLX-LM and Read Its Speed: First Load, Prefill, Decode, and KV Cache

Run the official MLX-LM default 3B 4-bit model in a separate Python environment, then examine model loading, prompt processing, token generation, and memory pressure separately. No speed ranking is asserted without measurements.

Install PyTorch ROCm on AMD Radeon and verify GPU operation

Using an RX 9070 XT and Ubuntu as an example, follow AMD's PyTorch installation path, understand torch.cuda versus HIP, and verify GPU discovery and a real image workload.

Qwen-Image 2.1 GGUF download: choosing Q4_K_M, Q8_0 and the VAE

This guide covers Q4 and Q8 file sizes for Unsloth Qwen-Image 2.1 GGUF, its encoder and dedicated VAE, and the stable-diffusion.cpp command. It also explains memory considerations for 12–16 GB systems and notes for running on Mac.

A safe local starting point for MiniCPM5-2B Q4_K_M

Download the official GGUF, start a local server with a 4,096-token context, and keep the MLX and DSpark paths distinct.

Qwen3.8-27B setup by hardware: Mac oMLX and RTX llama.cpp

Run Qwen3.8-27B with oMLX on Apple Silicon Macs and llama.cpp on RTX, then compare input, memory, MTP, and INT8 prefill settings under their respective conditions.

Qwen3.8 Flash-Next on DGX Spark: SGLang, PLE, and MTP recipe

A reproducible sequence for running Qwen3.8 Flash-Next on one DGX Spark with SGLang, PLE offload, and MTP.

Gemma 4 12B serving: Q4 on desktop and BF16 on a server

How to serve Gemma 4 12B on a single 16GB–24GB GPU or a Mac desktop, with separate Q4 GGUF and official google/gemma-4-12B-it setups, a conservative 8K starting context and API checks.

Gemma 4 26B-A4B MTP: How Speed Changes Between One and Many Requests

Learn how to distinguish Gemma 4 26B-A4B weights from its MTP assistant, then compare single-request and concurrent workloads with MLX-VLM.

Qwen3.6 35B-A3B setup: NVFP4 on Mac, RTX, and DGX Spark

Load Qwen3.6-35B-A3B NVFP4 in the official DGX Spark vLLM container and verify an OpenAI-compatible endpoint. Distinguish the single-Spark baseline from multi-Spark paths and memory settings.

GLM-5.3-Flash Serving Guide: A 320B Server-Scale MoE Recipe

This guide distinguishes vLLM and SGLang settings for serving the official 320B (18B active) GLM-5.3-Flash checkpoint across accelerators from memory considerations for quantized conversions.

DeepSeek-V4-Flash: distributed serving for a 304B model

Serving DeepSeek-V4-Flash-0731, with 304B total and 13B active parameters: the official 4×GB300 vLLM setup, a 16K starting context and local API checks.

Building a Korean RAG index with Kakao's Kanana-2 3B embedding

2,560 dimensions, 4,096 tokens, and MRL down to 128 dimensions. We compare the Korean retrieval scores, calculate vector storage, and index documents on a Mac.

Local CCTV person detection with Frigate: N100, Coral or a Mac?

A network video recorder that keeps camera footage at home. We cover the H.264 substreams cameras need, Coral, OpenVINO and Apple Silicon detectors, and how far an N100 mini PC goes.

Building an ESPHome voice satellite: an ESP32-S3 microphone and wake word

Turning a microphone-equipped ESP32 board into a Home Assistant voice satellite: board choice, mic and speaker wiring, on-device wake word and resource limits.

Frigate detections as Home Assistant notifications: from MQTT to conditions

Get a phone alert when a person is detected: enable MQTT in Frigate, install the integration, then narrow alerts by zone and object type.

Image and video generation

Make your first image locally with FLUX.3 Klein

Set up the FLUX.3 Klein distilled INT8 model in ComfyUI and save your first PNG.

Does FLUX.3 Klein run in 8 GB? 5090, 5070 and Spark generation times

Compare memory and generation time across INT8, resolution, and step settings.

Edit reference photos and guide layout with FLUX.3 Klein

Use ComfyUI Image Edit, multiple references, and bounding boxes to guide edits and placement.

Generate an image locally with Z-Image-Turbo

Set up the official Diffusers path and save a 1024×1024 image.

Install LTX-2.5 locally and generate your first video

Use the official Python pipeline and 66 GiB of model files to save a short video.

LTX Retake and Dub-It guide: choose interval editing or dialogue changes

Separate Retake from Dub-It in LTX’s official pipelines and prepare a focused edit.

Add sound effects to a video with MMAudio

Generate synchronized audio from a video and scene prompt, then inspect the FLAC and MP4 outputs.

What Is a World Model? How AI Generates the Next Scene Locally

A world model predicts the next image or state from the current scene and an action. Compare the inputs, outputs, and local paths for Matrix-Game, DIAMOND, LingBot-World, Cosmos, GameCraft, and Oasis.

Run the Matrix-Game 4.0 Local Browser Demo on an RTX 5090

Install Matrix-Game 4.0 on Linux with an RTX 5090, fetch the Ancient Egypt model and game assets, then launch the local browser demo with PyTorch CUDA and MediaMTX.

Run Oasis 500M Locally: Generate Game Video from an Image and Action File

Set up Oasis 500M and ViT-VAE weights on Linux with NVIDIA CUDA, then create `video.mp4` from an opening image and action file. Includes access terms, sample inputs, and CUDA checks.

Run DIAMOND on a Mac: Play the CSGO world model with MPS

Run DIAMOND's CSGO branch on an Apple Silicon Mac and control its generated game view. Follow the setup commands, Atari ROM licensing note, and first-run checks.

Run LingBot-World V2 on a Mac: Turn an Image and Camera Path into Video

Install PandacatAI's MPS fork, provide an image, camera path, and prompt, and save an MP4. Assemble the checkpoints, run a 480×832 generation command, and locate the output file.

Turn an image into a short video with Cosmos 3 Edge on DGX Spark

Use Cosmos 3 Edge on DGX Spark to turn an image and a JSON prompt into an MP4. Set up the ARM64 container, call the local API, and read published generation times in context.

Generate an action-controlled clip with Hunyuan-GameCraft on a 24 GB GPU

Set up Hunyuan-GameCraft on Linux and use its official 24 GB single-GPU CPU-offload recipe to turn a reference image and camera actions into a video clip.

Qwen-Image-2.1-Turbo: How Do You Generate and Edit Images in 8 Steps?

Install the supported Diffusers path for Qwen-Image-2.1-Turbo, then generate an image or edit a reference image with its saved 8-step schedule.

Edit Reference Images with FLUX.2 Klein on Mac

Install FLUX.2 Klein through MFLUX's MLX path and run reference-image editing. This guide covers model sizes, licenses, input images, expected results, and first-line troubleshooting.

Restore old footage with an RTX 5090: a local SwiftVR guide

Set up and run SwiftVR video restoration on an RTX 5090. Learn how to distinguish its reported 1080p frame rate from H100 results and LightX2V figures, then inspect restored frames and full-file processing time.

Run Pruna-Qwen-Image-2.1 locally: choosing its 5- or 8-step adapter

Run the few-step LoRA adapters for Qwen-Image-2.1 with CUDA and Diffusers. Compare the 5-step speed option with the 8-step quality option, and check the H100 benchmark conditions, code and license.

Qwen-Image 2.1: local image generation and editing

Explore Qwen-Image 2.1 generation and editing on Mac or NVIDIA hardware, including LightX2V’s September 20, 2026 day-zero support, RTX 5090 measurement conditions, transparency, reference editing, and its research license.

ComfyUI first image: install, load a model and fix common errors

Install Comfy Desktop, locate the SD 1.5 checkpoint and open a downloadable 512px workflow, with checks for missing models and memory errors.

Before buying a GPU for ComfyUI, decide on one image you need to make

A way to choose local image-generation hardware by separating base generation, partial edits and upscaling, then checking time and memory across the whole task.

Make and edit images locally with LLaDA-Image: choosing Base or Turbo

A practical path through LLaDA-Image Base and Turbo for generation and editing, including what to test before buying hardware.

Animate a photo locally with Wan-Animate-2: what to check first

Understand Wan-Animate-2's reference-image and driving-video workflow, runtime choices and hardware checks.

Local image generation: software, hardware and your first image

A practical route from ComfyUI installation and model selection to checking VRAM, generating a first image and comparing hardware.

FLUX or Qwen Image: choose a model for the job

Compare FLUX.2 Klein and Dev with Qwen-Image-2512 for fast drafts, text rendering, reference-image editing, and detailed results.

Local video generation: workflows and PC requirements

Choose text-to-video or image-to-video, then match model, memory, frame count and duration. Start with a short clip before comparing hardware.

Wan 2.2 and MiniMax: choosing a local video workflow

Describes the features, camera motion, simultaneous audio creation, and required memory standards of Wan 2.2 TI2V-5B·T2V-14B and MiniMax H3-Base.

ComfyUI requirements: choosing 8, 12, 16 or 24GB VRAM

Compare starting workloads across VRAM capacities and understand resolution, batching and offloading without confusing weight size with peak memory.

Local music generation

Turn your Mac into a live AI instrument with Magenta RealTime 2

Install Magenta RealTime 2 on an Apple Silicon Mac and make live accompaniment from MIDI input. This guide covers DAW routing, 48 kHz setup, device support by model size, and the difference between control latency and full-song generation.

Create and revise ACE-Step music in a DAW: a local VST3 guide

Build the independent C++ implementation acestep.vst3, connect its local server to a DAW, and create an 8-step Turbo track. Covers model files, host routing, setup errors, and the scope of its VAE timing results.

Local AI music generation: make songs and background music on your computer

Choose a local music workflow for your PC or Mac: YuE2, ACE-Step and HeartMuLa, lyrics and prompts, memory requirements, generation time and refining your first track.

YuE2 local music generation: make vocal songs and arrangements from lyrics

Install YuE2, the current default version in the YuE repository, on Linux with an NVIDIA GPU. Create vocal songs from lyrics and style prompts, then inspect the editable score and saved generation artifacts.

Run ACE-Step 1.5 locally: setup through your first song

A practical guide to running ACE-Step 1.5 on your computer and shaping lyrics, style tags and duration. It distinguishes ACE Studio, explains Mac and NVIDIA memory requirements, and puts published speed figures in context.

Compare local music models for songs, instrumentals and sound effects

ACE-Step, HeartMuLa, YuE2, Stable Audio Open and MusicGen do different jobs. Compare them by whether you need a lyric-led song, instrumental passage or sound effect, along with hardware and license terms.

Local music generation speed and hardware: how to read RTF correctly

Separate official speed evidence from supported runtime paths for YuE, ACE-Step, HeartMuLa, Stable Audio Open and MusicGen, then measure music generation fairly on your own hardware.

Local speech recognition & synthesis

Transcribe Korean speech live and save text with Nemotron 3.5 ASR

Use Nemotron 3.5 ASR and NeMo-Speech.cpp to transcribe Korean microphone input live and save the final text.

Transcribe English speech in real time with Moonshine on CPU

Install Moonshine Voice for CPU microphone transcription, test a short English phrase, and check the separate streaming and license status for Korean.

Align Korean subtitles and speaker labels with WhisperX

Transcribe Korean speech with WhisperX, align word-level timings, then save an SRT file. Includes model access for diarization and a review against the source audio.

Generate Korean speech on CPU with Supertonic 3

Run Supertonic 3's public ONNX model on a CPU and save Korean speech as a WAV. Prepare the pinned model assets, choose a preset voice, and synthesize a short phrase.

Transcribe English audio locally with Granite Speech 5 TurboCTC

Understand Granite Speech 5.0 TurboCTC's input and output, then use the official model-card example as a starting point for local transcription.

Generate Korean speech with VoxCPM2's 30-language TTS

Check VoxCPM2's Korean TTS, 48 kHz output, and CUDA-oriented setup before using the official model-card example to create a first WAV.

Generate Korean speech with OmniVoice's 600+ language and dialect coverage

Review OmniVoice text input, audio output, language and speaker controls, and the difference between Apache 2.0 code and non-commercial weights.

Generate expressive speech with Fish Audio S2 Pro

Review Fish Audio S2 Pro's tag-based expressive controls, 83-language support, and research license.

Generate speech on a CPU with Pocket TTS in six languages

Generate WAV audio with Pocket TTS on a CPU and check its current language coverage, including the absence of Korean and Japanese.

Separate voices and sounds in recordings with SAM Audio

Use text, time-span, or video cues to isolate a sound from a recording with SAM Audio, and review its access requirements.

Make a two-speaker dialogue WAV with Fish Audio Drama 3

Use a speaker-tagged script, reference audio, and matching transcripts to create a multi-speaker WAV with Fish Audio Drama 3.

Transcribe Korean recordings on a Mac with WhisperKit

Transcribe recordings locally with a multilingual WhisperKit Core ML model. Covers installation, a first transcription, long recordings, and Neural Engine inspection.

What is FluidAudio? Local transcription, speakers, and speech on Mac

Start local transcription, speaker diarization, and Kokoro speech with FluidAudio Core ML models. Covers language selection and CPU, GPU, and ANE execution paths.

Label Speakers and Inspect Ambient Sounds in Meeting Recordings with LocalAI 4.11

Use LocalAI 4.11 Studio to transcribe a meeting, explicitly save speaker voices for later recordings, and inspect sound events such as coughing or applause. This guide also explains restart and shared-server limits.

Edit Audio Locally with AuK: Speech Revisions, Cleanup, and TTS

Use AuK with an instruction and optional audio to synthesize or edit speech, enhance recordings, and separate sources. Check the required Qwen2.5-Omni-3B encoder’s separate license and the conditions behind reported memory figures.

Orukeet local transcription: transcribe English and European-language audio

Install Orukeet ASR on Mac, CPU, or CUDA and transcribe audio files to text. The guide clarifies that Korean is not among its 25 supported languages and distinguishes NeMo, ONNX, and native-weight licenses and the official FLEURS evaluation conditions.

Phonon 2: A Local English Transcription Guide

Install Phonon 2 and the Fermion CLI to turn English recordings into text with optional word timestamps. This guide covers the first model download, JSON output, audio formats, and common setup errors.

Creating a local voice assistant: ASR·LLM·MCP·TTS connection guide

We explain the pipeline that converts speech into text, makes a decision by a local LLM, runs the MCP tool, and then responds back to text, as well as delay and security standards.

Local AI speech recognition: transcribe on your own PC

What should you install to transcribe recordings without uploading them? Learn how Whisper, faster-whisper, whisper.cpp, and Qwen3-ASR handle languages, Mac/NVIDIA/CPU runtimes, and performance measurement.

Qwen3-ASR vs Whisper: which local transcription model should you choose?

Compare local ASR for Korean, English, Japanese, and languages around the world. Separate language coverage, runtimes, licenses, and speed metrics for Qwen3-ASR, Whisper Turbo, and Nemotron 3.5.

Create local meeting notes with transcripts, timestamps, and speakers

Learn a local workflow for transcribing meetings and organizing speakers and timestamps. Check total time, first output, diarization, subtitles, and hardware needs for long recordings.

Make speech on your own computer: a local TTS guide

Start with the text and languages you need, then choose a Kokoro, Qwen3-TTS or Chatterbox workflow that fits your task and hardware.

What is Qwen3-TTS? Turn text into speech on your computer

Learn how Qwen3-TTS turns text into speech, choose CustomVoice, VoiceDesign or Base, compare RTF and streaming, and run it on Mac or NVIDIA hardware.

Compare local TTS models: languages, hardware and speed evidence

Compare Kokoro-82M, Qwen3-TTS and Chatterbox Multilingual by workflow and documented runtime, separating published timing claims from missing conditions.

Nemotron 3 Diarization: who said what in your meeting?

Add speaker labels to local meeting transcripts with Nemotron 3 Diarization. Compare Mac, Windows and Linux paths, latency settings and benchmark meaning.

mlx-audio: TTS and STT from a single install

mlx-audio bundles TTS, STT, speech-to-speech and music generation on Apple MLX. One CLI and one server, with only the model name changing between runs, and six of its models list Korean.

Fish Audio Drama 3 int4: 41.5 GB down to 22 GB

The 22.1 GB int4 quantization of Fish Audio Drama 3: what shrank from 41.5 GB, the published WER and latency figures, and the SGLang Omni command that serves it.

LFM3-Audio-2B: a 1 GB model that listens and speaks

LFM3-Audio-2B is a 2B model that takes 16 kHz speech in and answers in 24 kHz speech. Its 1.05 GB GGUF, a 50 ms first token and 18 languages including Korean, with conditions.

Nemotron 3.5 Transcribe: transcribe an hour-long meeting in one pass

Nemotron 3.5 Transcribe is a 30B-A4B speech LLM that transcribes an hour of audio in one pass and attaches speaker tags and word timestamps. Korean FLEURS CER is 2.78%, and the 4-bit GGUF starts at 19.68 GB.

A real-time voice assistant at home on an AMD Radeon: Ming Voice and its 20 GB VRAM budget

A local voice assistant that starts answering 0.23 s after you stop talking. We cover running it on a 16 GB card with int8 experts and why it does not run on a Mac.

A Home Assistant voice satellite that listens and answers without leaving the house

An Assist pipeline that keeps wake word, transcription and synthesis inside the house. We compare Speech-to-Phrase with Whisper, and cover Piper and satellite placement.

Piper in Korean: adding a ko_KR voice to Home Assistant

A local synthesizer that generates 1.6 s of speech per second even on a Pi 4. We cover the Korean ko_KR-kss-medium model, install, pipeline wiring and pronunciation checks.

Home Assistant voice announcements: speaking through the house with a Piper voice

Sending a sentence to a speaker with tts.speak: which speaker answers, how quiet it should be at night, and what changes when the sentence gets long.

Response speed and acceleration

Nemotron Bolt vs Qwen3.8-27B: which is faster for agent work?

Read NVIDIA's Bolt and Qwen3.8-27B results from the same RTX Spark test and choose the metric that fits your workload: TPS or agent-task completion time.

What is ReplaySSM? Faster decoding for Nemotron and Qwen servers

How ReplaySSM changes hybrid LLM state storage and draft verification. Explains NVIDIA and Princeton measurements and what they mean for a personal PC versus a concurrent server.

Is the Neural Engine actually working? Inspect Core ML execution

Use Xcode Performance reports to inspect CPU, GPU, and Neural Engine placement, loading and prediction time, then compare configurations for the same model.

Reading the public Arc Pro B70 benchmark: XeStrata and SYCL

This guide compares XeStrata xe0.1.43.1's Arc Pro B70 32 GB IQ3_XXS results: 1,511 tok/s prefill for a 4K prompt and 81.1 tok/s decode, against the upstream SYCL path. It also explains the measurement types and how to reproduce them under matching conditions.

Can You Lower CMP 170HX Power Without Losing Speed?

Compare measured GLM throughput and power draw at different CMP 170HX power limits. Learn the Linux commands to adjust the cap, check memory temperature, and restore the original setting.

Will Two CMP 170HX Cards Double the Speed?

Learn how model splitting, PCIe paths, and P2P affect two CMP 170HX cards, and what the public Qwen TP2 result does and does not show.

Qwen-Image 2.1 speed on RTX 5090 and Mac

Compare the RTX 5090 LightX2V 5.93-second result, FP4 and NF4 runs and Mac GGUF timings, keeping resolution, steps and runtime conditions separate.

Local AI speed report — September 2026

As of September 2026, we compare the prefill and token generation speeds of major Macs, NVIDIA GPU, DGX Spark, and AMD integrated memory equipment under the same conditions.

Long prompts can reverse a local LLM speed ranking

This guide explains, from a hardware-selection perspective, why decode rankings in short chats can differ from prefill and end-to-end completion rankings for long documents.

MTP: how multi-token prediction works and when to use it

We explain how Multi-Token Prediction (MTP) proposes and verifies multiple future tokens, why you need a supporting model and runtime, and when it gets faster.

What is Lightning MTP? Faster answers with oMLX 0.7.1rc1

How Lightning MTP proposes tokens for target verification and how to configure it. Separates oMLX 0.7.1rc1 Qwen3.8 and TurboQuant measurements from earlier development-release results.

Run DFlash and DSpark with llama.cpp: A Qwen3-4B Walkthrough

Learn how to convert and run DFlash and DSpark drafts paired with a Qwen3-4B target in llama.cpp, verify the baseline server and responses, and compare performance.

Prompt Lookup and n-gram speculative decoding explained

Describes how Prompt Lookup finds and validates subsequent tokens in repeated phrases (n-grams) of the prompt and creation history without an additional draft model, and how it differs from model weight offloading.

Qwen3.8-Flash-Next PLE Offload: RAM and SSD Requirements and Setup

Learn how Qwen3.8-Flash-Next PLE data can reside in RAM or on SSD, what storage and memory conditions matter, and how to interpret oMLX and DGX Spark measurements in context.

SpecPrefill: reducing the wait before the first token

How SpecPrefill selects input, model-specific oMLX requirements, paths that combine it with MTP, and checks for answer quality.

LLM memory bandwidth vs compute: how to find the speed bottleneck

We describe how memory bandwidth and GPU compute performance affect prefill, single-user decode, and batch processing, respectively, in local LLM.

memory and model

TensorFold 1.0.8 cuts Qwen3.8-27B Mac memory by 17.6 GiB

TensorFold 1.0.8 removes duplicate Qwen3.8-27B weight loading on Mac, saving 17.6 GiB when preparing the model with DFlash2. At the 70K-context check, M5 Max went from 47.2 to 29.6 GiB and M3 Ultra from 42.2 to 24.6 GiB.

What should you check before buying a CMP 170HX?

A pre-purchase checklist for CMP 170HX card rank and 32/40/64GB profiles, Linux drivers and Secure Boot, PCIe links, cooling and power setup.

CMP 170HX ECC Is Now Unlocked. What Actually Changes?

Separate the ECC behavior changed by cmpunlocker v0.5 from reporting counters, then review prerequisites and a practical stability check.

How Much Long-Context Text Fits on a 64GB CMP 170HX?

Separate 64GB memory capacity from usable context length, with the conditions behind open-jet's 18,976-token run and vLLM's 9,472-token failure in October 2026 measurements.

Strata PC memory guide: choosing a model with 32, 48, 64, or 96 GB RAM

Learn how VRAM and RAM divide expert placement in Strata, how resident and mmap low-RAM modes differ, and which sizes suit 32–96 GB RAM. Account for CPU work, NVMe storage, and context cache to avoid memory pressure.

M5 Ultra SSD: faster loading or faster local AI?

Separate the M5 Ultra SSD's effects on model loading, prefill and token generation. Use a documented offload example to decide where to spend on memory and storage.

Why a 35B MoE can be faster than a smaller dense model

By distinguishing between total and active parameters, we describe the combinations and memory conditions under which large MoE models are generated faster than small Dense models.

How much VRAM or unified memory does a local LLM need?

This is a method to determine the VRAM and integrated memory required for 12B, 27B, 35B, and 70B level local LLM, including quantization, runtime margin, KV cache, and context length.

Reduce KV cache memory: vLLM FP8 setup and response checks

Learn to distinguish vLLM FP8 KV cache from weight quantization, then check a baseline response, cache settings, and accuracy one condition at a time on a supported backend.

MLX-LM Server Cache: Reusing Shared Prompts and Testing Concurrency

A hands-on guide to shared system-prompt reuse and the effects of cache, latency, and memory under concurrent requests in MLX-LM server.

LLM quantization: choosing Q4, Q8 or FP16

We explain how memory, speed, and quality change when reducing the weights of a local LLM to low bits such as Q4 and Q8, and the criteria for selecting a quantization format.

MoE vs dense: why low active parameters still need memory

An MoE model may compute with only some experts, but it still has to store all weights. Compare total parameters, active parameters, and actual memory requirements with dense models.

Context length, RoPE and sliding-window attention

We explain what a model's maximum context length means and how long contexts affect prefill time, KV cache, RoPE scaling, sliding window, and answer quality.

turbovec: shrink the RAM a RAG index needs

turbovec stores embeddings at 2 or 4 bits per dimension and searches the compressed form. Ten million 768-dimensional vectors drop from 31 GB as float32 to 4 GB, and 4-bit search runs 3.8 to 5.1 times faster than FAISS.

Executable programs and extensions

How to Back Up and Restore a Local LLM You Already Use

Keep the model folder, version, and runtime settings for a local LLM you use, then repeat the same task with a copy while preserving the original.

What Is LiLiCorr? Speculative decoding for Bolt and Super VL

Learn how LiLiCorr links draft candidates, how its Bolt and Super VL recipes differ, and what the published measurements mean.

How to run Nemotron 3.5 Super VL on DGX Spark or RTX PRO 6000

Learn what Super VL accepts and how its vLLM recipes differ for DGX Spark and RTX PRO 6000.

What server do you need to run Ling 3.1 Flash?

Review Ling 3.1 Flash's size and official SGLang serving recipe, then check the server memory and access path you need.

Can you run Muse Spark Open on one Mac?

Separate Muse Spark Open's memory scale, Apple MLX distributed setup, and published M5 Ultra measurements.

How can you search your local photos and recordings with EmbeddingGemma 2?

Turn text, photos, and recordings into search vectors with runnable Python examples. Compare them in EmbeddingGemma 2's shared embedding space, evaluate retrieval on your own data, and troubleshoot environment, file-path, and memory errors.

DeepSeek Search and Fetch API: giving a local LLM web results

The request and response shape of the Search and Fetch APIs, the $0.75 per successful request price, the free window until November 1, and a two-step way to feed results to a local model.

RTX 3090 BF16 unlock: what opens and what stays the same

The gpu-unlock kernel module that opens BF16 and FP16 matrix throughput on the RTX 3090: its published scope, the measured 3090 Ti numbers, the single verified machine, and the cold-boot and rollback requirements.

Does the 3090 unlock speed up local LLMs? Prefill, batching, decode

Decode is bound by memory bandwidth, while prefill and batching are bound by compute. Which stages the 3090 BF16 unlock changes, and which it leaves alone.

Verifying a 3090 unlock: boot logs, GEMM and llama-bench

To compare before and after an unlock, fix the kernel, driver and power limit, then measure GEMM and llama-bench the same way. A record sheet and rollback procedure.

The RTX 3090 48 GB mod: what to verify before you accept the card

A 48 GB modified card is not accepted just because it boots and shows a picture. The checks that matter: a 49152 MiB report, compute capability 8.6, 92 to 95 percent memory allocation, driver, PCIe and Xid log review, and an ACCEPT/REVIEW/REJECT verdict.

Two RTX 3090s with NVLink: how much it actually changes inference

In a community benchmark of 2x RTX 3090 vLLM tensor parallelism, NVLink gave 85.0 vs 81.1 tok/s and a p95 of 239.1 vs 277.6 seconds. Why the default path is PCIe with P2P off, and how to check all three layers.

How to evaluate JEV-35B NVFP4 inference on B200

Distinguish the merged BF16 JEV-35B from its NVFP4 LoRA variant, then plan an evaluation around the NVFP4 card's memory and tested hardware scope.

What is Apple Minions? Sharing inference jobs across idle Macs

How Apple Minions distributes embedding and LLM jobs across Macs, how that differs from model sharding, and when it helps document-search workloads.

What can DiffusionGemma do on a Mac? Documents, images, and voice workflows

DiffusionGemma text, image, and video inputs, block generation, its Fast-dLLM-mlx path on Mac, and integration with transcription tools.

What is XeStrata? Running Qwen3.8 on Intel, AMD, or NVIDIA

XeStrata runs Qwen3.8-Flash-Next on Linux PCs with Intel Arc, AMD Radeon, or NVIDIA GPUs. This guide covers supported distributions and memory, then walks through downloading the model and getting a first local response.

Using XeStrata on Radeon: choosing a distro for RX 7000 and 9000

A Radeon model name alone does not determine whether XeStrata will run. This guide matches the gfx code of RX 7000 and 9000 cards with rocBLAS support on Ubuntu 26.04 or Fedora 44, then checks the first launch.

A 35B model on 4 GB VRAM? XeStrata Ornith and llm-jp requirements

See what 4 GB VRAM means for XeStrata's beta small-MoE paths, including system RAM and file sizes. Choose between Ornith-1.5 and llm-jp-4.1, then check the first run.

Using Intel Arc and RTX together with XeStrata

Learn how XeStrata's multi-GPU mode places model layers and expert caches across different cards. This guide covers setup commands, split selection, and what to verify in a real run.

What is TensorFold? A local model engine for Mac, RTX, and DGX Spark

TensorFold 1.0.8 serves qualified models on Macs and NVIDIA GPUs through an OpenAI-compatible API. This guide shows how to download Qwen3.8-27B on a Mac, start it with DFlash2, and review the supported model and hardware combinations.

GLM-5.3 744B on two M5 Ultras: memory and link requirements

TensorFold 1.0.8's GLM-5.3 EXLR8 home profile splits the model across two M5 Ultras. This guide covers why each needs 256 GB, split checkpoint downloads, memory at 8K context, and the conditions behind MTP speed.

One or two DGX Sparks for Qwen3.8 Flash Next?

The one-Spark recipe emphasizes concurrent throughput on TensorFold's Zig engine; the two-Spark recipe targets a 1M context and higher single-request decode. Compare conditions without conflating single decode, concurrent throughput, and prefill.

Run and compare GLM-5.3 Flash recipes on two, three, and four Sparks

Compare single-request speed and concurrent throughput in the GLM-5.3-Flash TensorFold recipes for two, three, and four Sparks. The three-Spark setup uses patches; the four-Spark result is experimental on a separate tp4 branch.

GLM-5.3 Flash on two RTX PRO 6000s: speed and power limits

Compare GLM-5.3-Flash TensorFold results on two and four RTX PRO 6000 Blackwell GPUs. Separate the 300 W and 250 W configurations, single-request decode, and concurrent aggregates, then follow the two-GPU setup.

Qwen3.8-27B on a 16 GB GPU: choosing ExLlamaV3 and EXL3

The Windows and Linux serving kit matches EXL3 bits per weight, context length, and image input to available VRAM. Compare profiles for 12, 16, 24, and 32 GB, then serve the selected file through an API.

How to measure local LLM speed with sparkDash

Set up sparkDash's decode and prefill benchmarks and read request count, prompt length, and TTFT. The guide follows the current sparkDash 2.40.0 setup and remote API configuration.

TensorFold Living Weights and RAG store knowledge differently

Living Weights updates weights in a supported model folder, while RAG stores source documents separately and retrieves them. This guide covers TensorFold 1.0.8's documented Nemotron scope, making a copy, and rollback.

Build a local AI chat interface with Open WebUI

Open WebUI connects to a local model server and lets you manage chats and document questions in a browser. This guide installs it with Docker, creates the first admin account, connects Ollama, and tests saved conversations and PDF questions.

Find slow steps in a local agent with Langfuse

Run Langfuse locally and record agent steps with observations from the Python SDK v4. Separating model calls, tool waits, and retries helps distinguish GPU token generation from end-to-end request latency.

What is Darwin-27B-ZTC? Probability-based decisions with 54 GB BF16

Darwin-27B-ZTC returns probabilities for yes/no, choice, and rubric questions instead of generating answer text. Review its input structure, a local Python call, and the BF16 memory requirement.

Parse documents, audio, and video into JSON with Youtu-Parsing-Omni

Select a task for a document image, chart, audio clip, or video and receive structured JSON. This practical workflow covers the pinned vLLM plugin, local file path restrictions, and output checks.

Extract scanned PDFs and tables locally with LightOnOCR-3

A practical workflow for saving a PDF as page-level Markdown and grounded blocks, then checking extracted tables against the source. Connect a vLLM server to the LightOnOCR CLI and inspect grounding-mode files.

Can an 8GB GPU run Qwen and Gemma faster? Install PolyStrata and use two GPUs

Run Qwen3.6 and Gemma 4 on an 8GB GPU with PolyStrata. Compare RAM requirements and generation speeds, then follow Linux AMD beta support, dual-NVIDIA setup, and the developer's backup download route.

Basalt at 665 tok/s? Qwen3.8 Flash-Next Setup and GPU Requirements

Learn what Basalt accepts and returns, which hardware it supports, how its .basalt packs use memory, and how to start the local API.

Can a CMP 170HX mining card run a 64GB local LLM?

Explore the CMP 170HX for local LLMs: its 32GB, 40GB and 64GB memory variants, Linux unlock requirements, model runtime options, cooling and PCIe limits.

Run a Local LLM in Your Browser with Transformers.js

Learn how to run an ONNX model in the browser with Transformers.js 4.3.0 and WebGPU. Follow a working example and understand the first model download, CPU-versus-GPU choices, and browser support limits.

Backburner: Accelerate Mac LLM Prefill with an iPhone

Backburner is a pre-release project that uses a USB-C-connected iPhone to extend Qwen3.8-27B prefill and context on a Mac. Review supported devices, published benchmarks, setup steps, and security conditions.

What Is Superfluid? Run Multiple Agents on One Local LLM

Serve GGUF and MLX models through a local API with Superfluid. Follow a Qwen3.8-27B example, prioritize interactive chat, configure concurrency and memory, and interpret the published benchmarks.

Set up Qwen3.8-27B prefill and caching with Splash 1.3.0

Run Qwen3.8-27B with Splash 1.3.0 on Apple Silicon, then check Neural Engine prefill and cached prompt reuse.

Run Qwen3.8 Locally with NInfer: RTX 5090, MTP, and DFlash2

Serve low-bit Qwen3.8 and Underdog Saluki on an NVIDIA GPU with NInfer-all. Follow the model download, Docker server, first API request, and MTP or DFlash2 setup, then read the RTX 5090 community results in context.

A 125B model on a 12GB GPU? How Strata runs it

Learn how Strata runs the Qwen3.8-Flash-Next 125B MoE model on Windows and Linux PCs, how the GPU, RAM, CPU, and SSD share the work, and how to choose a model and start it.

Choosing Strata quantization: Q2_0, IQ2_XS, IQ3_S, and Coder

Compare the memory and speed tradeoffs of Q2_0, IQ2_XS, and IQ3_S, plus Coder with half the experts and Swift 1.5's shorter reasoning. Choose for your RAM, Korean or Japanese work, and model-license requirements.

How to read Strata speed benchmarks: short chats and long prompts

Separate prompt-processing throughput from answer-generation speed in Strata benchmarks, and review RTX 5070 and Radeon conditions, TTFT, output length, and cache effects.

Strata AMD HIP guide: Run Radeon on Windows and Linux

Learn how to prepare and install Strata's HIP backend for AMD Radeon on Windows and Linux, verify a first response, and interpret the 0.1.43 measurements.

Strata Multi-GPU Layer Splitting: Setup and Results

Learn how Strata splits model layers across GPUs, configure automatic or manual splits, decide when another card helps, and interpret RTX 5080/3090 prompt and decode results.

Strata troubleshooting: downloads, first launch, and slow responses

When Strata 0.1.43 setup stalls or the first response is slow, check download resumption, model loading, RAM and SSD access, GPU detection, and port conflicts by stage.

Run Strata on an RTX 5070 12 GB: Model and RAM choices

See the RAM and model choices for running Strata’s Qwen3.8-Flash-Next on a 12 GB RTX 5070, with separate short-chat and long-context speeds.

Run Strata on an RTX 3090 24 GB: VRAM, RAM, and long context

Choose Strata model sizes for a 24 GB RTX 3090, compare 32/64/96 GB RAM setups, and understand CPU, PCIe, and long-context KV cache effects.

Run Strata on an RTX 5090 32 GB: RAM choices and published results

Plan IQ2_XS or IQ3_S with 64 or 96 GB RAM on an RTX 5090, and read the conditions behind long-prompt and community results.

Connect Strata to coding agents: OpenAI, Anthropic, and Codex APIs

Run Strata on localhost and send a real request through OpenAI Chat Completions, Anthropic Messages, or the Responses API used by Codex.

Strata vs llama.cpp: Which Should You Run on Your PC?

Compare Strata for Qwen3.8-Flash-Next with llama.cpp for GGUF models. Choose by GPU and RAM placement, Mac support, installation, first-answer latency, and your actual workload.

When local RAG answer is wrong: Search, chunk, model diagnosis order

Using a vacation policy PDF as an example, this guide shows how to locate an error in source extraction, retrieved evidence, answer generation, or index updates.

Local LLM·RAG quality evaluation: Verifying changed settings with DeepEval

Compare settings before and after a change using the same vacation-policy questions and retrieved evidence. This guide explains how to connect DeepEval to a local Ollama judge and check retrieval misses, unsupported answers, and latency changes.

Building Fully Offline Local AI: Air-Gap Installation and Update Procedures

We explain the air-gap local AI procedure that moves models, runtimes, and packages to an environment without the Internet, and operates verification, updates, and rollback.

Run GGUF directly in Transformers: how it differs from llama.cpp and MLX

We explain with practical examples how Transformers on Apple Silicon execute GGUF in compression, memory conditions, and selection criteria for llama.cpp·MLX.

vLLM Metal 0.30: turn one Mac into a multi-user local AI server

Describes the Apple Silicon installation conditions for vLLM Metal 0.30, PagedAttention, integrated KV storage, simultaneous request settings, and precautions when distributing multiple Macs.

Connect OpenClaw to a Local Model with Ollama or Managed llama.cpp

Separate the OpenClaw Gateway from the local inference server, choose Ollama or OpenClaw-managed llama.cpp, and verify chat and real tool calls step by step.

Install Hermes Agent and Connect a Local Model: From Your First Task to Permission Checks

Distinguish Hermes Agent from Hermes models, then verify local or OpenAI-compatible endpoints, tool calls, memory, MCP, and permissions through one first task.

OpenClaw, Hermes Agent, and NemoClaw Compared: Which Fits Your Tasks and Operating Style?

Compare OpenClaw, Hermes Agent, and NemoClaw by role and operating model, including channels, local models, memory, tool verification, sandboxing, and network boundaries.

Local AI Agent Security: Check Permissions, Network Paths, and Secrets

A local model does not automatically make tool execution or external connections safe. Review tool allowlists, sandbox limits, API binding, messenger and search data paths, key handling, and incident response.

Do you need an NVIDIA GPU and CUDA for local AI?

Understand the different roles of local AI apps, frameworks, the CUDA Toolkit and NVIDIA drivers, and find out when you need an NVIDIA GPU or another compute path.

When PyTorch seems to use the CPU despite an NVIDIA GPU

Diagnose why a GPU is missing or a model runs on CPU by separating nvidia-smi, the PyTorch CUDA build, torch.version.cuda and nvcc.

Will two GPUs make a local LLM twice as fast?

Learn how multi-GPU memory placement, model parallelism and replica throughput differ, and decide when a second GPU is useful.

Run local AI with MLX on Apple Silicon: frameworks and model tools explained

MLX, MLX-LM, Metal, PyTorch MPS and GUI apps play different roles. Check Apple Silicon versus Intel Mac, unified-memory limits and model-specific support before choosing a runtime.

Local AI with ROCm and Radeon: Reading Support Combinations and Runtime Paths

Learn how ROCm/HIP differs from CUDA, how to check GPU, OS, driver and framework combinations in AMD’s support matrices, and how to distinguish Windows, WSL and Linux paths.

If you port CUDA code to AMD, will it run just as fast?

Source conversion, correctness, and long-session stability must each be verified separately. Learn how to check HIP/HIPIFY and official AMD paths, then assess a Radeon purchase using the same image task.

Jev-Omni: multimodal decisions on local GPUs

Understand Jev-Omni's Gemma 4 12B classifier, CUDA and memory requirements, Python examples and published H200 results, and how it differs from other local JEV-style projects.

What is Crane? Run LLM, OCR and audio models behind a local Rust API

Starting from the hassle of separate Python environments for chat, OCR and audio, learn how Crane's Candle-based Rust project builds, runs Qwen3.5, connects through a localhost API and handles Bonsai 2.

Husky on Mac: Woof, MLX and the 730 tok/s result

Understand Husky's Woof-specific inference, Flash drafting and MLX comparison, how to try it on Mac, and why the result does not transfer to every model.

What is Jev? A System One model that decides instead of writing

How Jev differs from a general LLM, why it returns typed choices and probabilities, and what local Jev-like projects actually reproduce.

Local Jev-like decisions with Qwen: SemIf, Simple Jev and openjev-sglang

Compare direct-logit Qwen projects, a small CPU implementation, an SGLang server and trained Jev-like models without mistaking them for official Jev.

Run Jev-Style Local Decision Models on Mac, RTX 3090, and Servers

Learn how Jev-style models read a state and return probabilities for choices, then choose among CPU, Apple silicon, RTX 3090, and B200 server setups for your workload.

Local LLM slow on its first run? Five checks before blaming the hardware

Use the app’s status and a short prompt to distinguish downloading, model loading, and slow response generation.

What actually improves when you connect two DGX Sparks?

Distinguish between model load, single request creation rate, and multiple request throughput for one and two DGX Spark configurations.

GGUF vs MLX: which model format should you download?

Local LLM explains which runtime ecosystem model files for GGUF and MLX fit into and how to check quantization notation and transformation compatibility.

Running local LLMs across two or more GPUs

We explain layer partitioning using two GPU VRAMs for local LLM, the difference between tensor parallelism and pipeline parallelism, and PCIe·NVLink communication bottlenecks.

FlashAttention vs PagedAttention: what each one changes

We explain how FlashAttention and PagedAttention, which have similar names but solve different problems, affect attention calculation, KV cache memory management, and server throughput.

Continuous batching: throughput is not single-user speed

We explain how continuous batching increases server throughput by putting LLM requests of different lengths in and out of running batches, and how it differs from single-user latency.

Which local LLM app or serving engine should you start with?

Choose between Ollama, LM Studio, llama.cpp, oMLX, MTPLX, vLLM and SGLang for your hardware and workflow. Understand model files, single-response speed and concurrent requests.

MTPLX: will MTP make answers faster on your Mac?

Install MTPLX on Apple Silicon, choose supported MLX models, tune MTP and connect a local API. Learn how MLX files differ from GGUF, and how to check memory shortages and cache effects.

Ollama: connecting a model on your computer to other apps

Check local models and app connections in Ollama, then test the prerelease MLX automatic-selection path on Apple Silicon. Separate connection problems from backend and speed issues.

LM Studio guide: download a model and start chatting

Follow model search, download, loading and chat in LM Studio, with GGUF/MLX choices, memory checks, troubleshooting and API integration.

llama.cpp: finding which settings are slowing you down

Choose a GGUF file and GPU backend for llama.cpp, start a local llama-server, then check memory, prefill and decode separately.

Run local AI on a Mac with oMLX: what is new in 0.7.1rc1?

Run models on a Mac with oMLX and connect them to an API client. Covers 0.7.1rc1, Splash integration, model settings, Muse Spark Open support and choosing a release.

Set Up a Local vLLM Server: From a Small Model to Adaptive Verification in 0.30

This guide uses the DSpark example to examine DFlash and DSpark drafts and adaptive verification in vLLM 0.30.0, then explains how to compare results across CUDA graphs, model support, and request load.

SGLang HiCache: Spreading reusable KV across GPU, CPU and storage

Use a code question to examine SGLang HiCache tiers across GPU, host memory and storage, and prefix reuse in hybrid-attention models. Configure and distinguish cache hits, eviction, first requests and repeats.

Running several coding agents at once on a Mac with Locals

Locals is a local server that computes requests from several AI agents at once on a Mac. This covers installing it, connecting OpenCode, the throughput it gains under concurrent requests, and the cases where the 3x figure does not hold.

Slotstream: running a 125B model on a 16 GB Mac

Slotstream runs a 125B model on a 16 GB Mac by reading only the experts it needs from the SSD instead of holding the whole model in memory. It measured 20.91–23.69 tokens per second on a 48 GB M5 Pro, and 1–6 tokens per second on a 16 GB Mac.

agent-memory: making a coding agent remember across sessions

agent-memory writes what an agent learned in a session to Markdown files and lets the next session find and read them. On a bounded LongMemEval-S run it scored 52.9% against 5.8% with no memory, and it verified that three hosts share one store.

Laya-MLX: a decision model that answers in 13 ms on a Mac

Laya-MLX runs a decision model that answers choice, score and true/false questions with probabilities on Apple Silicon. One short question took 13.42 ms on an M3 Max, 7.39 ms with the multilingual checkpoint, with zero output tokens.

Unsloth Browser Use: hand a browser to your local model

Browser Use in Unsloth Desktop lets a local model open pages, press buttons and fill forms in a pane beside the chat. Sign-ins, payments and downloads wait for approval, and permissions are set per site.

Splitting speech off with Wyoming: moving transcription to another machine

Taking Whisper off the machine that runs Home Assistant: why Wyoming has no authentication, how to attach by address and port, and what to move.

Agents and automation

Can you run a Jev-like decision model on your PC? A guide to d1-3B and d1-omni

d1-3B and the experimental d1-omni-600M return choices and probabilities for predefined questions instead of long replies. This guide shows how to start a llama.cpp server, send local requests, use text, image, and audio input, and interpret latency measurements.

Can you run Cloudflare Clef locally? Choose 27B or 9B and send a SystemOne request

Clef and Clef-Flash take a state and question schema, then return probabilities for each choice. Learn to download a model with oMLX on Apple Silicon and send a first request with an API key, with an H200 server path as an alternative.

Let an AI answer your phone on a Mac: installing ai-phone and choosing 4B or 9B

A Mac-only agent that answers SIP calls and talks with a local model. We cover the model choice at 8, 16, and 32 GB, the install steps, and the measured call latency conditions.

Running local AI for a smart home: splitting voice, conversation and video across your own machines

Pair Home Assistant with an Ollama conversation agent, put speech on Wyoming and cameras on Frigate. We cover which machine should run what, and the latency and power trade-offs.

Adding an Ollama conversation agent to Home Assistant: exposed entities and instructions

How Assist hands questions to a local model, and what to limit before you let it control devices. We cover the model, the instructions and the aliases.

A morning brief from Home Assistant: turning sensor data into sentences with a local LLM

A model does not know the state of your house, so the values have to travel inside the sentence. This covers handing sensor and calendar values to conversation.process and getting the answer as a notification.

Can a local LLM run the house? Exposure scope and safety boundaries

A conversation model can only touch devices on the exposed list. We cover which devices to leave out, starting with reversible actions, and defaults for failure.

Raising local LLM voice command accuracy: aliases, exposure and instructions

Say the same thing twenty times, split the failures into three kinds, and fix them with aliases, the exposed list and instructions — before changing the model.