Local AI Guide

See the speed before choosing hardware

Model setup recipes

Try GPT-OSS with Groq's Free API: From Account Setup to Your First Response

Use a small code-review request to check your Groq API connection and free limits.

GroqCloud hosts open-weight models and returns responses through an API1. This guide covers a Python request to GPT-OSS 20B, the published free limits, secure key entry, response endings, and data-retention conditions.

Contents
Requirements and key details
  • The current public Free Plan table lists the same request and token limits for openai/gpt-oss-20b and openai/gpt-oss-120b.
  • The Python SDK can read the key through hidden input; inspect finish_reason alongside the final response.

Send a request to an open-weight model through the Groq API

The Groq API accepts conversation messages and a model ID over HTTPS, then returns a response generated by a model hosted on GroqCloud. Here, a short Python request asks openai/gpt-oss-20b to review code, and reads both the response text and its completion status.

GPT-OSS 20B and 120B are available as Apache 2.0 open-weight models that you can also run on your own computer. The GroqCloud API uses the model IDs openai/gpt-oss-20b and openai/gpt-oss-120b for hosted inference2. See OpenAI's open-weight model overview and the GPT-OSS 20B model card.

A person at a laptop is surrounded by connected document, spreadsheet, folder, image, settings, and robot panels
The API receives the function and review instruction written in the code, then returns the model's answer.

Choose a model within the Free Plan limits

In Groq's published Free Plan summary, the two GPT-OSS models share the same request and token3 limits. RPM means requests per minute, RPD requests per day, TPM tokens per minute, and TPD tokens per day.

This free table is a public organization-level summary. Check the current values for your account in the Groq Console Limits page. Input and generated tokens share the 8K-per-minute budget, so start with the relevant function and error description instead of pasting a long file.

GPT-OSS request and token limits in Groq's published Free Plan
Model IDRequest limitsToken limits
openai/gpt-oss-20b30 RPM / 1K RPD8K TPM / 200K TPD
openai/gpt-oss-120b30 RPM / 1K RPD8K TPM / 200K TPD

Prepare an account and API key

Check that your GroqCloud account shows the Free Plan, then create a key under API Keys. Groq says a payment method is required to upgrade to the paid Developer plan. To stay within the free tier, check the plan shown in your account and review any payment or upgrade screen before proceeding. Groq billing guide

The installation commands below are for a macOS or Linux shell. Enter the API key through a hidden prompt4 at runtime5 instead of writing it into the code, so it stays out of shell history and the source file. After saving the example in the next section, run python groq_review.py in the same terminal with the virtual environment6 active. Another Python environment may not find the installed groq package.

Set up a virtual environment and the Groq SDK
python3 -m venv .venv
source .venv/bin/activate
python -m pip install groq
Activate the virtual environment in a macOS or Linux shell, then install the SDK.

Send a short code-review request

Save the following as groq_review.py and run it. getpass reads the API key without echoing it to the screen, then the SDK sends a JSON request to the chat completions endpoint.

Ask GPT-OSS 20B to review a code snippet
from getpass import getpass
from groq import Groq

client = Groq(api_key=getpass("Groq API key: "))
completion = client.chat.completions.create(
    model="openai/gpt-oss-20b",
    messages=[
        {
            "role": "user",
            "content": (
                "Review this function for the empty-list edge case and suggest a fix:\n\n"
                "def average(values):\n"
                "    return sum(values) / len(values)"
            ),
        }
    ],
    reasoning_effort="low",
    include_reasoning=False,
    max_completion_tokens=2048,
)

choice = completion.choices[0]
print(choice.message.content or "(No final text was returned)")
print(f"finish_reason={choice.finish_reason}")
print(f"completion_tokens={completion.usage.completion_tokens}")
if choice.finish_reason == "length" and not choice.message.content:
    print("The token limit may have been used before final text; shorten the request or raise max_completion_tokens.")
Run the saved file with `python groq_review.py` and read the returned text and finish reason.
Run the script
python groq_review.py
The script prompts for the key and then prints the model response.
Lines connect a compact computer to a browser, calendar, and chat window
Input and generated output share the 8K-token-per-minute limit. Start with one function rather than a long file.

Read the response text and finish status together

A successful response places the code-review text in choices[0].message.content; finish_reason=stop means the model completed its reply. The output is model-generated text, so read it against the actual input and function behavior before applying a change.

finish_reason=length means the configured generation-token limit was reached. GPT-OSS also spends the response budget on reasoning, so if the final text is empty, shorten the request or raise max_completion_tokens and try again. Keep the 8K TPM limit in mind and size the output cap for the answer you need.

Distinguish authentication, model, and limit errors

For a 401, re-enter the key and check in API Keys that it is still active. If a 400 reports a model error, compare the exact spelling openai/gpt-oss-20b with the current supported-model list.

A 429 means the request or token limit per minute or per day has been reached. Check the retry-after, x-ratelimit-remaining-requests, and x-ratelimit-remaining-tokens headers, then wait before retrying or reduce input and output length. The Rate Limits documentation and Console Limits page show the limit names and current remaining amounts.

Choose where the request belongs

Groq says customer inputs and outputs from ordinary inference requests are not retained by default. It may temporarily log them for up to 30 days when needed for reliability work or abuse investigations, and it collects usage metadata separately. Zero Data Retention can be configured in Data Controls; the Groq data guide explains how this affects features that require storage.

For sensitive company code or customer information, check your organization's transfer and processing rules first and send only the relevant excerpt. If you need to process data on your own computer without an internet connection, download the open weights and use a local runtime; to try a hosted API without handling model files, use GroqCloud. Compare other providers in the free open-weight AI services guide.

A user looks at progress indicators and a document-processing window on a desktop computer
Reading the response body and finish status together shows whether a request completed.

Terminology notes

  1. API — A defined interface that lets other code call a program’s functions.

    Back to the text

  2. Inference — The process of using a trained model to compute an output for an input. Here, local inference means running the model on the user’s device.

    Back to the text

  3. Token — A unit into which a model divides input or output for processing.

    Back to the text

  4. Prompt — The input text sent to a model. For generative models it describes the desired result and strongly shapes the output.

    Back to the text

  5. Runtime — The software environment that provides facilities needed while a program runs. In local AI, it can also refer to a model execution engine.

    Back to the text

  6. Python virtual environment — An isolated space for installing Python packages per project, helping reduce package-version conflicts.

    Back to the text

Frequently asked questions

What should you check first in this article?
GroqCloud hosts open-weight models and returns responses through an API. This guide covers a Python request to GPT-OSS 20B, the published free limits, secure key entry, response endings, and data-retention conditions.
Where can you check the conditions and sources?
GPT-OSS 20B and 120B are available as Apache 2.0 open-weight models that you can also run on your own computer. The GroqCloud API uses the model IDs `openai/gpt-oss-20b` and `openai/gpt-oss-120b` for hosted inference. See [OpenAI's open-weight model overview](https://help.openai.com/en/articles/11870455-openai-open-weight-models) and the [GPT-OSS 20B model card](https://huggingface.co/openai/gpt-oss-20b).