Run Jev-Style Local Decision Models on Mac, RTX 3090, and Servers

A standard chat model reads a request and generates a response. A Jev-style decision model takes a request or event record as state and evaluates predefined questions and choices. choice returns probabilities for each option and the top choice; score returns a score calculated from probabilities across levels; noul returns the probability that a yes/no statement is true. Results are usually JSON, so they can feed steps such as ticket routing, safety checks, or selecting a next action without generating a prose answer.

For example, send an order inquiry as the state and receive a choice among billing, shipping, and technical, with a probability for each option. These probabilities differ from task accuracy, so compare them with human-labeled answers, set a threshold, and route low-probability or borderline cases for review.

oMLX 0.7.1.dev1 and llama.cpp v0.6.0 support the TypeSafe-compatible POST /v1/systemone endpoint. A regular /v1/chat/completions request generates tokens³ and returns prose; /v1/systemone returns structured results for choice, score, or noul questions. Confirm that the selected model is a supported decision model before calling this endpoint.

Simple Jev reads logits from a regular Qwen model and builds the choice result as JSON. Here, run Qwen3.5-0.8B on a CPU to route a duplicate-charge inquiry. The later oMLX section uses separately trained OpenJev or Clef checkpoints⁴.

Prepare Git and Python 3.12 or newer. The installation commands below use a macOS or Linux terminal with Python 3.13. Clone the repository, create a virtual environment⁵ in that folder, and install the server package.

git clone https://github.com/featherless-ai/simple-jev.git
cd simple-jev
python3.13 -m venv .venv
source .venv/bin/activate
python -m pip install -e './hf-server'

Start the CPU server in the same virtual environment. The first run downloads the model from Hugging Face, so wait for download and loading to finish. Use port 8001 to keep it separate from the oMLX server later in this guide.

python hf-server/hf_server.py --model Qwen/Qwen3.5-0.8B --device cpu --dtype float32 --classifier-prompt-policy baseline --max-model-len 4096 --max-batch-size 4 --max-batch-tokens 4096 --port 8001

Keep the server terminal open and query health and models from a new terminal. A ready status and the Qwen3.5-0.8B model ID mean you can send a short classification request.

curl http://127.0.0.1:8001/health
curl http://127.0.0.1:8001/v1/models

The request below sends a duplicate-charge inquiry as state and asks for a team among billing, shipping, and technical. Start with an example containing no sensitive data, and remove personal information before testing work samples.

curl http://127.0.0.1:8001/v1/classifier \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen/Qwen3.5-0.8B","state":"Customer message: I was charged twice for my order last week.","questions":{"route":{"type":"choice","instructions":"Which team should handle this?","criteria":{"billing":"Payments, invoices, and refunds","shipping":"Delivery or tracking issues","technical":"Product or account errors"}}}}'

Read the selected team and each candidate's probabilities in answers.route. Compare with a human-labeled team, then add rephrased inquiries and cases with missing information. Use the errors to adjust choice descriptions and human-review rules.

If the connection fails, check model download/loading in the server terminal and port 8001. For HTTP 422, read the error details and compare state, questions, and criteria with the example. If package installation fails, check the Python version and virtual environment first. When finished, stop the server with Ctrl+C in its terminal.

On Mac, the oMLX API⁶ section below provides the OpenJev/Clef execution path. Use the SemIf MLX⁷ experiment here to compare memory capacity and shared-input reuse. The SemIf-OpenJev MLX path supports experiments that score Qwen3.5-4B directly on Apple silicon. The public browser demo's Q4 weight file is about 3 GB. macOS, the runtime⁸, per-question workspace, and caches also use memory, so check Memory Pressure in Activity Monitor and leave headroom for repeated requests after the model loads.

A small request may run on a 16 GB unified-memory Mac, but a browser and development tools can reduce headroom. Process one item with your usual apps open and check memory pressure and completion time. If multiple questions share a long state, compare a shared path that reads the common state once with a fresh path that reads it for each question.

With 24 GB or more unified memory⁹, you may have room to test more questions and larger inputs. Run the same task samples individually and concurrently, recording choices, probabilities, processing time, and memory in each case.

Memory needs for a 35B model vary by checkpoint precision and serving runtime. For example, the JEV-35B NVFP4¹⁰ card lists 25.6 GB (23.3 GiB) of weights and says it was tested on B200. Check the card’s runtime requirements against the intended device and workload.

The published RTX PRO 6000 single-question medians are Julia-1 3 ms, Laya 5 ms, Kev-4B 12 ms, lev 36 ms, and OpenJev 43 ms. Keep each model name with its size and language in the table, separate from service latency at other input lengths or concurrency.

Raw BF16¹¹ weights for Qwen3.5-4B take about 8 GB: 4 billion parameters × 16 bits ÷ 8 bits per byte. A simple subtraction from the RTX 3090's 24 GB VRAM¹² leaves about 16 GB, but that space is not all available for question caches. The runtime, CUDA¹³ driver, temporary buffers, and KV cache¹⁴ also use VRAM. Q4 files reduce download size and weight memory but use a different numeric representation from BF16.

The SemIf repository provides examples comparing direct, prefix-reuse, and parallel-suffix execution on an RTX 3090. When several questions branch from one shared state, reuse can reduce repeated prefill¹⁵ work. Follow the linked repository README for SemIf installation and execution. Once that environment is ready, record three direct runs with the same checkpoint and question set, then compare prefix reuse and parallel suffix one at a time.

For each path, save the model revision, quantization¹⁶, input-token count, number of questions, GPU use, and result. Record the first run separately because it may include model loading. If choices differ, review the original state and each option's probability, then compare both paths against human-labeled examples.

OpenJev 27B is a Qwen3.8-27B-based decision model that also accepts image input. Support in llama.cpp and oMLX varies by checkpoint and project version. First confirm that the model appears as a decision type and that a /v1/systemone request works. Its published language list is English, German, French, Hindi, Chinese, and Japanese. Before using it for Korean work, evaluate choices and errors on a sample of Korean inquiries.

oMLX 0.7.1.dev1 supports Clef, Clef-Flash, and OpenJev decision models. The release says Clef-family models accept text and images, while OpenJev 27B accepts text and one image. These models return probabilities for choices instead of generating a prose response, so a regular chat API may not return the structure you expect.

If several users share a 27B server, plan for concurrent requests, long inputs, image use, and resident model memory, not just average single-request time. First check requests and results with a Q4 4B workload on an RTX 3090, then estimate the capacity needed for a 27B server from labeled samples and expected workload.

On Apple unified memory, reserve space for the operating system and apps. On a GPU, budget not only weights but also KV cache, activations, and concurrency. Weight size alone does not guarantee stable execution.

Errors can be costly if ticket routing skips an agent or approves a refund incorrectly. Route low-confidence outputs to review and shortlist hardware only when latency and review rates meet your target.

Collect recurring work inquiries and have a person label the correct answer. Thirty cases can serve as a small initial check. To assess accuracy or impact across the workflow, add more clear and borderline cases, and record each model's choices, probabilities, processing time, and errors using the same criteria. Add examples when new error types appear.

Set probability thresholds according to the cost of errors in your task. First define when low-probability or close-choice requests should go to a person. OpenJev probabilities use the temperature configuration in the model file. Check the threshold on separate validation samples before automating the evaluated cases.

For one-at-a-time workflows, measure per-request latency; for batches of inquiries, measure throughput¹⁷ and completion time. If answers are correct but requests queue up, consider GPU headroom or more parallel questions. If answers are often wrong, first improve the state, choices, or language support. Compare upgrades when repeated workloads and input sizes show that more capacity is needed.

Repeat each device with the same query set and model revision; log p50, p95, peak memory, and errors. Separate model loading, first-request, and warmed-request times so startup delay is not mixed with steady-state latency.

Increase concurrency gradually before scaling a small-batch configuration to service load. Consider a setup only when quality, latency, and memory limits all fit together.

Download an OpenJev or Clef decision checkpoint in oMLX or a supported server and start it. Replace YOUR_OMLX_API_KEY in both requests below with the server key. If the server uses no authentication, omit the Authorization header from both requests. First query the model list.

curl http://127.0.0.1:8000/v1/models -H 'Authorization: Bearer YOUR_OMLX_API_KEY'

Copy the decision model ID from data[].id in the response into YOUR_DECISION_MODEL_ID. Then send the choice request asking which team should handle the duplicate charge.

curl http://127.0.0.1:8000/v1/systemone \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_OMLX_API_KEY' \
  -d '{"model":"YOUR_DECISION_MODEL_ID","state":"Customer message: I was charged twice for my order last week.","questions":{"route":{"type":"choice","instructions":"Which team should handle this?","criteria":{"billing":"The customer reports a duplicate charge or refund issue","shipping":"The customer reports a delivery or tracking issue","technical":"The customer reports a product or account technical issue"}}}}'

If answers contains the route question, selected choice, and probabilities, the request was processed. Evaluate it against human-labeled examples from the actual workflow. HTTP 501 can indicate an unsupported model type or input format; HTTP 400 can indicate invalid request JSON. For a load error, check the first server-log error and available memory.