Executable programs and extensions

Ollama: connecting a model on your computer to other apps

The model answers in a chat, but stops responding when connected to your code editor. Before downloading a larger model or replacing your hardware, check three things: where the model runs, where the app sends requests, and which features the app expects.

When you want a simple connection setup

Ollama combines model management with a server that accepts requests and returns answers. It is useful when you want a small script or editor to call the same model. You can work mainly with a model name and server address instead of maintaining a detailed launch command each time.

OpenAI-compatible does not mean that every feature of every app works. Ordinary chat, tool calls and image inputs need separate checks. Look at the API and model features your app requires, confirm that a short text request works, then add the features you need.

A model name does not tell you where it runs

For local use, select a downloaded model. Ollama also offers cloud models, so the app name alone does not mean every request runs on your computer. The example below disables cloud features and starts a server that accepts connections only from this computer.

Apple Silicon can use Metal acceleration. On NVIDIA or AMD PCs, check GPU and driver support as well. Different tags and quantization formats can change the memory required by a model with the same name. Record more than the model name when comparing someone else's results.

Check for an existing server first

If the Ollama app or system service is already running, a second server cannot use the same port. The command below is for starting it from a terminal after stopping the existing server. If you keep using the app or service, apply settings to that environment and restart it instead.

The context is explicitly set to 4,096 tokens for a short connection check. This is not a claim about every version's default, or a sufficient context for coding agents. Once the server is running, use ollama ls in another terminal to check the names and tags of your downloaded models.

Start a local-only server

OLLAMA_NO_CLOUD=1 OLLAMA_HOST=127.0.0.1:11434 OLLAMA_CONTEXT_LENGTH=4096 ollama serve

This example uses a macOS or Linux terminal after installing Ollama. On Windows, set the same environment variables and restart the app. This command does not download a model.

Separate connection checks from speed measurements

Replace YOUR_LOCAL_MODEL with the name you just checked. This request only verifies that a short answer returns. With streaming disabled, the response appears after completion; that wait must not be recorded as time to first token.

After receiving an answer, check PROCESSOR in ollama ps. A model partly placed on the CPU is not directly comparable with one running entirely on the GPU. Separate initial model loading from repeated runs, too. For a speed record, warm up first, then record at least three runs with matching input/output lengths and settings.

Send a short request to a local model

curl -sS http://127.0.0.1:11434/api/chat -H "Content-Type: application/json" -d '{"model":"YOUR_LOCAL_MODEL","messages":[{"role":"user","content":"Reply with one short sentence."}],"stream":false,"options":{"num_ctx":4096,"num_predict":128}}'

YOUR_LOCAL_MODEL is a placeholder. Replace it with the exact tag of an installed model. This is a connection check, not a performance benchmark.

Before replacing everything because it feels slow

If short questions work well but documents cause long waits, examine input processing and memory first. Larger contexts and concurrent requests need more space. If switching to a smaller model file makes it faster, do not attribute that result solely to an engine setting.

Consider another engine when your current setup lacks support for the model or acceleration method you need. Continue to llama.cpp for direct control over GGUF settings, or the oMLX and MTPLX guides for MLX paths on a Mac. Keeping a familiar setup is also reasonable if it already handles your work.

Change log

These entries record changes to the site's guidance. They do not automatically check your installed engine or model version.

  1. Added Ollama local-serving and status-check guidance

    Added examples for starting a local-model server and sending a short API request. It covers checking where the model runs and its context settings, and explains why a request's completion time is not a first-token measurement.