What should you check first in this article?

Model setup recipes

Which embedding model should you use for Korean document search? Qwen3, BGE, and EmbeddingGemma 2

Check whether each model finds the right passage for the same questions.

Use a leave-policy PDF as a shared test corpus, record retrieval and indexing results, and run one model at a time on CPU¹ with a Hit@5 baseline.

Compare on the same leave-policy PDF

Qwen3-Embedding, BGE-M3, and EmbeddingGemma 2 are embedding models that turn queries and documents into numeric vectors to find relevant passages. Compare the three models' search results using the same vacation-policy PDF and queries. A reranker compares retrieved candidates with the query again and adjusts their order; a separate LLM² writes the final answer.

We will search a leave-policy PDF for the passage answering whether an employee at six months of service can take summer leave. Compare extracted text with the PDF and assign an expected passage ID to each query.

Documents are split into search passages, with selected passages sent to a local computer — Split documents the same way to compare models fairly.

Match checkpoints and input formats

Qwen3-Embedding-0.6B supports a 32K input and 1,024-dimensional vectors in its official table, and recommends a task instruction on the query. BGE-M3 supports 8,192 tokens³ and dense, sparse, and multi-vector paths; its dense example does not add a query instruction. This comparison starts with dense retrieval.

EmbeddingGemma 2 can load a 270M text-only configuration and supports 128, 256, 512, or 768 dimensions. Use its text task prefixes and start at the native dimension. Parameter count and vector dimension do not equal on-disk index size, so measure that separately.

Text retrieval settings listed in the model cards
Model	Default vector	Query input
Qwen3-Embedding-0.6B	1,024 dimensions · 32K	English retrieval instruction
BGE-M3	1,024 dimensions · 8,192	No extra instruction in dense example
EmbeddingGemma 2	768 dimensions · 8,192	SearchQuery prefix

Measure one model at a time on CPU

Compare the three models one at a time with the same corpus.json and questions.json; the script stops if either file is missing. For answerable queries with one gold passage, Hit@5 is the fraction whose gold ID appears in the top five. Queries with gold_ids: [] are reported separately.

It prints model-load, document-embedding, and query-search times. .npy size covers only the numeric vector array, not a vector-database index or metadata; this code reports the array only.

Save the two JSON arrays below as files with those exact names, and save the Python code as qwen_retrieval_baseline.py. The two-passage sample only demonstrates the format; use a larger work corpus for Hit@5 comparison.

corpus.json

[
  {"id":"p-001","title":"여름휴가 자격","text":"입사 6개월 이상인 직원은 여름휴가를 신청할 수 있습니다."},
  {"id":"p-002","title":"신청 기한","text":"사용 희망일 14일 전까지 신청해야 합니다."}
]

Add work passages using the same fields.

questions.json

[
  {"id":"q-001","text":"입사 6개월 차도 여름휴가를 신청할 수 있나요?","gold_ids":["p-001"]},
  {"id":"q-002","text":"퇴직 후 미사용 휴가를 현금으로 받나요?","gold_ids":[]}
]

Use an empty `gold_ids` array for no-answer queries.

Create the environment

python -m venv .venv
source .venv/bin/activate
python -m pip install -U pip
python -m pip install 'sentence-transformers>=5.0' numpy FlagEmbedding
python qwen_retrieval_baseline.py --model qwen
python qwen_retrieval_baseline.py --model bge
python qwen_retrieval_baseline.py --model gemma

A minimal CPU setup for running one model at a time. The first run downloads model weights.

qwen_retrieval_baseline.py — run a fixed corpus

import argparse, json, time
from pathlib import Path
import numpy as np
MODELS = {"qwen":"Qwen/Qwen3-Embedding-0.6B", "bge":"BAAI/bge-m3", "gemma":"google/embeddinggemma-2"}
INSTRUCTION = "Instruct: Given a Korean policy question, retrieve passages that answer the question\nQuery: "
p = argparse.ArgumentParser(); p.add_argument("--model", choices=MODELS, required=True); args=p.parse_args()
for name in ("corpus.json", "questions.json"):
    if not Path(name).is_file(): raise SystemExit(f"Missing {name}; save the JSON example first.")
docs_data=json.loads(Path("corpus.json").read_text(encoding="utf-8")); questions=json.loads(Path("questions.json").read_text(encoding="utf-8"))
if not docs_data or not questions: raise SystemExit("Both JSON files need records.")
start=time.perf_counter()
if args.model=="bge":
    from FlagEmbedding import BGEM3FlagModel
    model=BGEM3FlagModel(MODELS[args.model], use_fp16=False, devices=["cpu"])
else:
    from sentence_transformers import SentenceTransformer
    kw={"device":"cpu"}
    if args.model=="gemma": kw["config_kwargs"]={"vision_config":None,"audio_config":None}
    model=SentenceTransformer(MODELS[args.model], **kw)
load_s=time.perf_counter()-start
docs=[f"title: {d.get('title','none')} | text: {d['text']}" for d in docs_data]
start=time.perf_counter()
if args.model=="bge": vectors=model.encode(docs,batch_size=2,max_length=8192)["dense_vecs"]
else: vectors=model.encode(docs,batch_size=2,normalize_embeddings=True,convert_to_numpy=True)
vectors=np.asarray(vectors,dtype=np.float32); index_s=time.perf_counter()-start
out=Path(f"{args.model}-vectors.npy"); np.save(out,vectors)
hits=answerable=0; no_answer=[]; rows=[]; start=time.perf_counter()
for q in questions:
    gold=q.get("gold_ids",[])
    if not gold: no_answer.append(q["id"]); continue
    if len(gold)!=1: raise ValueError(f"Exactly one gold ID required: {q['id']}")
    text=q["text"]
    if args.model=="qwen": text=INSTRUCTION+text
    if args.model=="gemma": text="task: search result | query: "+text
    if args.model=="bge": qv=model.encode([text],batch_size=1,max_length=8192)["dense_vecs"][0]
    else: qv=model.encode([text],normalize_embeddings=True,convert_to_numpy=True)[0]
    top=np.argsort(vectors@np.asarray(qv,dtype=np.float32))[-5:][::-1]; ids=[docs_data[i]["id"] for i in top]
    hit=gold[0] in ids; hits+=hit; answerable+=1; rows.append({"query_id":q["id"],"hit_at_5":bool(hit),"top5_ids":ids})
search_s=time.perf_counter()-start
print(json.dumps({"model":args.model,"model_load_s":round(load_s,3),"document_embedding_s":round(index_s,3),"query_search_s":round(search_s,3),"answerable_queries":answerable,"hit_at_5":hits/answerable if answerable else None,"no_answer_query_ids":no_answer,"raw_vector_array_bytes_in_ram":int(vectors.nbytes),"raw_npy_file_bytes_only":out.stat().st_size,"top5":rows},ensure_ascii=False,indent=2))

Save as `qwen_retrieval_baseline.py`, then run the commands below.

Cards organize the expected evidence for each query — Marking the expected passage shows whether retrieval missed a condition.

Compare rerankers on the same candidates

This code covers dense first-stage retrieval only. To test reranking, save candidate IDs returned for the same queries and send the shared candidate set separately to BAAI/bge-reranker-v2-m3 or Qwen/Qwen3-Reranker-0.6B.

Measure candidate generation and reranking separately, then compare gold rank and Hit@5 before and after reranking. The current code does not run a reranker.

Two computers compare document indexing and retrieval results — Record quality and time together to see how each model affects your work.

Terminology notes

CPU — The central processor that runs the operating system and general-purpose instructions; in AI workloads it also handles tasks such as preprocessing and data movement.
Back to the text
Large language model — A model trained on large text datasets to learn language patterns and process or generate text in context.
Back to the text
Token — A unit into which a model divides input or output for processing.
Back to the text

Frequently asked questions

What should you check first in this article?: Use a leave-policy PDF as a shared test corpus, record retrieval and indexing results, and run one model at a time on CPU with a Hit@5 baseline.