Local AI Guide

See the speed before choosing hardware

Executable programs and extensions

How to evaluate JEV-35B NVFP4 inference on B200

Start with the 25.6 GB weights and the card's B200-tested scope.

JEV-35B takes a situation, a question, and a set of choices, then returns a probability for each choice and a decision. The NVFP41 deployment compresses the expert weights to reduce the download size.

Contents
Requirements and key details
  • B200 is the hardware tested on the NVFP4 card.
  • The NVFP4 LoRA and merged BF16 checkpoints are distinct releases.

Identify the release variant first

JEV-35B takes a situation, a question, and a set of choices, then returns probabilities for the choices and a decision. For example, it can take a customer inquiry and choose billing, shipping, or technical support. The model is based on Qwen3.5-35B-A3B with a decision head and adapter added.

The NVFP4 deployment compresses the expert weights, reducing the download to 25.6GB; the weights occupied 23.3GiB in vLLM. The public run record lists a B200 as the test device. The card lists only kernel paths for the RTX 5090, RTX PRO 6000, and DGX Spark, so compatibility on those systems still needs to be checked.

A system chooses among fixed options for a request on a desk
Inspect the decision output separately from chat responses.

Evaluate decisions on the same requests

The example sends one ticket to the NVFP4 server to inspect its response. For accuracy, label at least 100 tickets and aggregate separately; compare BF162 by running each checkpoint3 independently and keeping results apart. The request below covers one NVFP4 call only.

The published NVFP4 run reports single-request medians of 227 ms for text and 371 ms for images on one B200. It used a sequential workload of 387 text and 200 image requests. On your server, repeat the same tickets and record decision accuracy plus p50/p95 latency.

A view comparing candidate scores and selecting the highest
Compare on the same ticket set as other decision models.

Keep the two recipes separate

This procedure is for NVFP4 only. hf download requires the Hugging Face CLI4, and the card says serve.sh was tested with a September 2026 vLLM development build. The README does not specify an exact vLLM package version or commit, so that runtime5 must be prepared separately.

Pin the NVFP4 card revision and start its documented server
python -m pip install -U huggingface_hub
hf download autotrust/JEV-35B-NVFP4 --revision 10c2c62610e315e31a75baca849ad79d526e4658 --local-dir JEV-35B-NVFP4
# Run this in the documented September 2026 vLLM development environment.
bash JEV-35B-NVFP4/serve.sh
Prepare the card’s vLLM development runtime separately, then run this in the server terminal.

Run the server in terminal 1 and wait until it is ready. Leave it running, then send the curl request from terminal 2. This asks it to choose a department for one inquiry. The default port is 8000.

Send a System 1 choice request
curl http://127.0.0.1:8000/v1/decide \
  -H 'Content-Type: application/json' \
  -d '{"kind":"choice","state":"Customer message: my card was charged twice for the same order.","question":"Which team should handle this ticket?","options":["billing","shipping","technical support","account security"]}'

The request’s options define the allowed choices. A successful response returns per-option probabilities, the selected choice, its choice_index, and usage. Compare the choice with the labeled action and set a human-review cutoff for low-confidence tickets using your own labeled set.

This release uses Apache 2.0. Keep the model revision, runtime and gold labels with evaluation results so the next version can be tested on the same requests.

A side-by-side view of chat answers and option-based decisions
Do not confuse the decision API with chat generation.

Terminology notes

  1. NVFP4 — A 4-bit floating-point data format defined by NVIDIA. Support depends on GPU generation, model, and software implementation.

    Back to the text

  2. BF16 — A 16-bit floating-point format for model values. Its exponent has the same width as FP32, giving it a similarly wide value range.

    Back to the text

  3. Checkpoint — A file containing saved model weights and related state. Versions or tasks in one model family may use different checkpoints.

    Back to the text

  4. CLI — Short for Command-Line Interface: operating a program by entering commands in a terminal.

    Back to the text

  5. Runtime — The software environment that provides facilities needed while a program runs. In local AI, it can also refer to a model execution engine.

    Back to the text

Frequently asked questions

What should you check first in this article?
JEV-35B takes a situation, a question, and a set of choices, then returns a probability for each choice and a decision. The NVFP4 deployment compresses the expert weights to reduce the download size.
Where can you check the conditions and sources?
The NVFP4 deployment compresses the expert weights, reducing the download to 25.6GB; the weights occupied 23.3GiB in vLLM. The public run record lists a B200 as the test device. The card lists only kernel paths for the RTX 5090, RTX PRO 6000, and DGX Spark, so compatibility on those systems still needs to be checked.