Add embeddings and a reranker to Korean PDF search

An embedding model converts the query and each passage into vectors and retrieves similar passages quickly. A reranker reads each query-passage pair together and scores relevance to reorder retrieved candidates. So this comparison does not use the reranker to search every PDF passage; it reranks the top K from embedding retrieval.

The BAAI/bge-m3 model card lists multilingual embeddings and an MIT license. The BAAI/bge-reranker-v2-m3 card describes a multilingual cross-encoder and lists Apache-2.0. Both can be evaluated on Korean, but a multilingual label does not establish retrieval accuracy for a specific domain.

Check each model card for repository license, language support, input length, and usage. Some embedding models require a query prefix such as query: ..., while the BGE-M3 example uses the same encode interface. On first load, model files download to the local cache.

First determine whether a PDF is scanned or text-based, then choose a parser that preserves tables, headers, and page numbers. Use the same PDFs, queries, and human-verified gold pages so embedding comparisons are not confounded by OCR¹ differences.

Keep source filenames and page numbers on passage IDs. When a malformed table looks like the gold passage, fix extraction before changing the retrieval model.

Read page text from sample.pdf with pypdf, split it into 700-character chunks, and attach source, page, and chunk_id. If the PDF contains only scanned images, extraction yields no text and the script raises an OCR-needed error. This search example uses real PDF passages but has no relevance labels, so it is not a quality evaluation. Install Sentence Transformers and pypdf; model weights download on first use.

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -U sentence-transformers pypdf

For the comparison, encode and retain document vectors once. Time baseline retrieval as query encoding plus cosine ranking, then pass the same top candidates to CrossEncoder. The code prints baseline and reranked ranks, retrieval time, reranker-only time, and total query time.

from pathlib import Path
from pypdf import PdfReader
from sentence_transformers import SentenceTransformer, CrossEncoder
from sentence_transformers.util import cos_sim
from time import perf_counter

pdf_path = Path("sample.pdf")
reader = PdfReader(pdf_path)
passages = []
for page_number, page in enumerate(reader.pages, start=1):
    text = " ".join((page.extract_text() or "").split())
    for start_char in range(0, len(text), 700):
        chunk = text[start_char:start_char + 700].strip()
        if chunk:
            passages.append({"chunk_id": f"p{page_number}-{start_char}", "text": chunk, "source": pdf_path.name, "page": page_number})
if not passages:
    raise ValueError("No selectable PDF text found; use a text PDF or run OCR first.")

query = "환불은 며칠 안에 신청해야 하나요?"
embedder = SentenceTransformer("BAAI/bge-m3")
reranker = CrossEncoder("BAAI/bge-reranker-v2-m3")
# One-time indexing is outside both per-query timers.
doc_vectors = embedder.encode([p["text"] for p in passages], normalize_embeddings=True)
start = perf_counter()
query_vector = embedder.encode([query], normalize_embeddings=True)
candidate_scores = cos_sim(query_vector, doc_vectors)[0].tolist()
candidates = sorted(zip(passages, candidate_scores), key=lambda x: x[1], reverse=True)[:3]
baseline_ms = (perf_counter() - start) * 1000
print("baseline ranks:")
for rank, (p, score) in enumerate(candidates, 1):
    print(rank, p["source"], p["page"], p["chunk_id"], round(float(score), 4))

start = perf_counter()
r_scores = reranker.predict([(query, p["text"]) for p, _ in candidates])
rerank_ms = (perf_counter() - start) * 1000
reranked = sorted(zip([p for p, _ in candidates], r_scores), key=lambda x: float(x[1]), reverse=True)
print("reranked ranks:")
for rank, (p, score) in enumerate(reranked, 1):
    print(rank, p["source"], p["page"], p["chunk_id"], round(float(score), 4))
print("baseline_query_ms", round(baseline_ms, 2))
print("reranker_only_ms", round(rerank_ms, 2))
print("query_total_ms", round(baseline_ms + rerank_ms, 2))

python rag_compare.py

This code has no relevance labels, so it does not calculate Recall@5 or MRR@10. Next, label relevant passages for a Korean query set and compare baseline and reranked positions for the same questions. Ten to twenty queries are an initial check. For p50/p95 latency, repeat queries after warm-up and exclude first model loading and document indexing.

Compare the expected document's rank before and after reranking. Calculate Recall@5 and MRR@10 over 10–20 queries and record p50/p95 latency to weigh retrieval improvement against added response time. Keep batch size², hardware, and extracted PDF text constant between runs.

Local inference³ still uses the network for the first weight and library downloads. Check each model repository's license. If model version or chunking changes, rebuild reusable document embeddings. A reranker cannot recover passages missing from candidate retrieval, so improve chunking or embedding search first when Recall@K is low.

BGE-M3 supports dense, sparse, and multi-vector paths; hold the first comparison to dense retrieval. Apply the BGE reranker only after producing the same top-20 candidates to separate first-stage misses from reranking gains.

For each query, record Hit@5 and gold rank; aggregate Hit@5 and MRR@10 over the full set. Log reranking time per query, while index-folder size belongs to one model/chunk/index configuration, not to a query. Also calculate the fraction of gold passages present in first-stage candidates.