Executable programs and extensions
Running Jev-like local AI on a Mac, RTX 3090 or server
Buying a GPU first repeats the usual mistake of sizing hardware around a model name. Jev-like workloads generate little or no prose, so decode speed matters less than shared-context prefill and question branching. A laptop demonstration, a repeatable 24 GB GPU experiment and a multi-user server are different systems. Start on the hardware you own and scale only when a measured bottleneck appears.
Day one: complete one request on CPU
The Simple Jev path with Qwen3.5-0.8B can reveal the request and response structure without a dedicated GPU. Use one paragraph, two questions and two or three options, then inspect the JSON. The goal is not speed or production accuracy. It is to understand where state, questions and candidates are assembled. Start with examples whose answers you know, check probabilities and selected values, then change the wording slightly and record instability. That list defines what a larger model or faster machine would need to fix.
On a Mac, begin with 4B Q4 and MLX
SemIf's MLX backend can score Qwen3.5-4B directly on Apple Silicon. The Q4 browser artifact is around 3 GB, but runtime buffers, caches, macOS and other applications need additional space. A small experiment may fit on a 16 GB Mac, yet a setup that requires closing everything else is a poor baseline for repeated work. At 24 GB and above, increase the question count while keeping headroom. Compare fresh reads with shared-prefix reuse and record both latency and answer agreement.
A 24 GB GPU is useful for repeatable validation
Four billion BF16 weights are roughly 8 GB before overhead, leaving a 24 GB card room for the model, cache and batches of questions. SemIf's published RTX 3090 work shows why this class of GPU is a convenient bench for direct scoring, prefix reuse and parallel suffix experiments. The GPU name alone will not reproduce the numbers. Pin the driver, PyTorch and model revision, hold input tokens and question count constant, and exclude cold-start runs. If throughput is not yet a problem, validate decision quality on the smaller machine you already own.
The 35B SGLang path is a serving problem
Qwen3.6-35B-A3B activates only part of its parameters per token, but all weights still need storage. The default NVFP4 checkpoint in openjev-sglang is described at roughly 23 GB, while the published reference environment is B200. That does not promise the same kernels or memory headroom on a 24 GB consumer card. At this scale, design shifts from single-request latency to radix-cache reuse, concurrent branches and overload rejection. If the goal is one private decision service, stabilizing a 4B experiment is more useful than copying the server stack.
The purchase threshold needs both accuracy and latency
Build thirty representative states with question sets from your work. Label the answers yourself, then record choices, probabilities and latency for both direct and shared paths in one table. If a small model is accurate enough and already fast, there is no hardware problem. If accuracy is acceptable but large batches are slow, a GPU or prefix reuse may help. If a larger model still misunderstands the rules, change the option descriptions, data and human-review path instead of buying a more expensive machine. For Jev-like automation, the fallback path matters more than a tok/s ranking.
