Model setup recipes

Retrieve English documents with UltraBERT-Embed-150M

Evaluate the retrieval-ready Embed-150M checkpoint, not the backbone.

A 150M embedding model for English document retrieval. The model-card example requires trust_remote_code=True; review remote code before loading and evaluate on an English search set.

Choose the retrieval checkpoint

UltraBERT is published as encoder backbones as well as a retrieval embedding checkpoint¹. Use dmis-lab/UltraBERT-Embed-150M for document search; do not treat UltraBERT-150M or UltraBERT-380M as interchangeable retrieval embedders.

The official card lists English-only use, 150M parameters, 768 dimensions, a 32K-token context, and Apache-2.0. Published MTEB results are benchmark figures, not measurements of your domain's retrieval accuracy or hardware speed.

Documents are split into search passages, with selected passages sent to a local computer — Build a fixed set of English passages, queries, and gold IDs.

Retrieve on the same English evaluation set

The code embeds three English passages and prints their similarity rank for one query. It does not aggregate Hit@5 or Recall@10; use a larger labeled set to choose a model.

You can include multilingual models such as BGE-M3. For this experiment, keep every input and gold label in English and compare on the same corpus as UltraBERT-Embed-150M. Evaluate Korean separately with a Korean test set.

UltraBERT-Embed-150M — three passages, one query

from sentence_transformers import SentenceTransformer

model = SentenceTransformer(
    "dmis-lab/UltraBERT-Embed-150M",
    revision="367d5343b5a0164d57b810cedf5de4b8983d5984",
    trust_remote_code=True,
)
documents = [
    "The battery health report is available in Settings under Power.",
    "Refunds are processed within five business days after cancellation.",
    "The delivery address can be changed before the order is prepared.",
]
query = "Where can I view the battery health report?"
doc_vectors = model.encode(documents, normalize_embeddings=True)
query_vector = model.encode(
    [query],
    prompt_name="query",
    normalize_embeddings=True,
)
scores = model.similarity(query_vector, doc_vectors)[0]
for index in scores.argsort(descending=True).tolist():
    print(f"{float(scores[index]):.4f}\t{documents[index]}")

Install the UltraBERT retrieval environment

python -m venv .venv
source .venv/bin/activate
python -m pip install -U "sentence-transformers>=6.0" "transformers>=5"

Cards organize the expected evidence for each query — Track document length alongside retrieval rank.

Review remote code and results

The example uses trust_remote_code=True and pins model revision 367d5343b5a0164d57b810cedf5de4b8983d5984. Review the repository code at that revision before running.

Keep English benchmark scores, your retrieval metrics, and local runtime² in separate columns. If gold pages are missed or latency exceeds your target, adjust chunking or candidate count and measure again, even when the published score is strong.

If you need to search Korean documents, do not pick this model first. Compare it for finding the right passage inside English documents, and evaluate other languages with embedding models that name them.

Two computers compare document indexing and retrieval results — Compare retrieval quality with latency and index size.

Terminology notes

Checkpoint — A file containing saved model weights and related state. Versions or tasks in one model family may use different checkpoints.
Back to the text
Runtime — The software environment that provides facilities needed while a program runs. In local AI, it can also refer to a model execution engine.
Back to the text

Frequently asked questions

What should you check first in this article?: A 150M embedding model for English document retrieval. The model-card example requires `trust_remote_code=True`; review remote code before loading and evaluate on an English search set.
When is it better not to choose this?: If you need to search Korean documents, do not pick this model first. Compare it for finding the right passage inside English documents, and evaluate other languages with embedding models that name them.