Model setup recipes

A Korean LoRA SFT walkthrough: data, training, and evaluation

The input is Korean prompt/completion JSONL and the output is a saved adapter. Compare the base and adapter on the same question excluded from training.

The Qwen model card lists Qwen2.5-0.5B-Instruct as 0.49B parameters, supports 29+ languages including Korean, uses Apache-2.0, and provides a Transformers example. This recipe uses TRL SFTTrainer and PEFT LoraConfig targeting q_proj, k_proj, v_proj, and o_proj. The sample rows demonstrate format only; replace them with task examples and split train/held-out data by source document.

You may want support requests labeled consistently, for example as returns or refunds. LoRA¹ fine-tuning teaches that task behavior from input examples and desired answers. Create Korean examples, train a small model, save the adapter, and compare the same question before and after training.

Decide whether retrieval is a better fit than training

Use RAG² when answering requires looking up facts or documents that change. Fine-tuning is a candidate for repeated response formats or consistent Korean task behavior. If uncertain, evaluate a retrieval baseline and a training candidate on the same questions.

A set of picture cards with check tokens and a card organizer — Set a criterion for whether to test retrieval or an adapter before training.

Choose the Qwen model and prepare training data

Collect varied Korean examples from the task, but remove or anonymize personal and confidential documents. Deduplicate boilerplate and record input, expected output, and failure criteria for each example. Split by user or source document so duplicates and templates do not leak between training and evaluation.

One training row

{"prompt":"<|im_start|>user\n문의 유형을 분류해 주세요: 주문한 상품을 반품하고 싶어요.<|im_end|>\n<|im_start|>assistant\n","completion":"환불·반품<|im_end|>"}
{"prompt":"<|im_start|>user\n다음 문장에서 날짜와 할 일을 JSON으로 정리해 주세요: 10월 20일까지 견적서를 보내 주세요.<|im_end|>\n<|im_start|>assistant\n","completion":"{\"date\":\"10월 20일\",\"task\":\"견적서 보내기\"}<|im_end|>"}

Save the two lines as `train.jsonl`. They are fixtures for practicing the data format.

Use the small Qwen2.5-0.5B-Instruct for this exercise: 0.49B parameters, 29+ languages including Korean, Apache-2.0, and Transformers support make it a setup fixture for data files and adapter saving. For production model selection, check a current card's languages, license, and PEFT target modules. Qwen3.5-0.8B is a multilingual hybrid vision-language model, so verify its training path before transferring this text-LoRA configuration.

Install the training environment

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -U 'trl[peft]' datasets accelerate

The first run downloads packages and model weights. Use the Windows-specific virtual-environment activation command on Windows.

A person taking notes while inspecting reports on a desk with a magnifying glass — Keep source-document duplicates out of the evaluation set.

Start with a small TRL configuration and evaluate

train_lora.py first prints base responses to held-out questions that are not in the training file, then starts TRL training with the same tokenizer chat template. SFTConfig sets the output path, maximum length 512, batch size³ 1, gradient accumulation 8, 2 epochs, and LoRA learning rate 2e-4.

After training, the adapter and tokenizer are saved in output/adapter; generate responses to the held-out questions again and compare them. Record the training loss and actual memory use, then review the held-out answer quality.

train_lora.py: baseline, LoRA SFT, save, and compare

from datasets import load_dataset
from peft import LoraConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import SFTConfig, SFTTrainer

MODEL = "Qwen/Qwen2.5-0.5B-Instruct"
QUESTION = "문의 유형을 분류해 주세요: 결제한 물건을 돌려보내고 결제 취소를 요청하고 싶습니다."  # held-out wording; expected label: 환불·반품
tokenizer = AutoTokenizer.from_pretrained(MODEL)

def answer(model, question):
    messages = [{"role": "user", "content": question}]
    inputs = tokenizer.apply_chat_template(
        messages, add_generation_prompt=True, tokenize=True,
        return_dict=True, return_tensors="pt"
    ).to(model.device)
    output = model.generate(**inputs, max_new_tokens=80, do_sample=False)
    return tokenizer.decode(output[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)

base = AutoModelForCausalLM.from_pretrained(MODEL, device_map="auto", torch_dtype="auto")
print("base:", answer(base, QUESTION))
del base

dataset = load_dataset("json", data_files="train.jsonl", split="train")
trainer = SFTTrainer(
    model=MODEL,
    args=SFTConfig(
        output_dir="output/checkpoints",
        max_length=512,
        num_train_epochs=2,
        per_device_train_batch_size=1,
        gradient_accumulation_steps=8,
        learning_rate=2e-4,
        logging_steps=1,
        save_strategy="epoch",
        report_to="none",
        eos_token=tokenizer.eos_token,
    ),
    train_dataset=dataset,
    processing_class=tokenizer,
    peft_config=LoraConfig(
        r=8, lora_alpha=16, lora_dropout=0.05, bias="none",
        task_type="CAUSAL_LM",
        target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    ),
)
trainer.train()
trainer.save_model("output/adapter")
tokenizer.save_pretrained("output/adapter")
print("adapter:", answer(trainer.model, QUESTION))

Save as `train_lora.py`, then run the next command. For real training, collect varied examples and separate training from evaluation data.

Start training after saving both files

python train_lora.py

The console prints generated responses after base: and adapter:. On success, output/adapter/adapter_config.json and adapter weights should exist.

If CUDA⁴ runs out of memory, reduce max_length while keeping batch size 1, or use a larger GPU⁵. If model loading fails, check that Transformers is current; the Qwen2.5 card documents KeyError: 'qwen2' with versions below 4.37.0.

Review held-out accuracy, format, and Korean fluency in a score sheet, and compare against RAG when the task needs changing facts.

Two compact devices print document pages beside stacks of output sheets — Record held-out quality alongside VRAM and latency.

Terminology notes

LoRA — A method that learns and applies small additional weights to adapt a base model. The adapter is used with the base model for purposes such as style adaptation or few-step generation.
Back to the text
Retrieval-Augmented Generation — A method that retrieves material relevant to a query, adds it to the model input, and generates an answer from that context.
Back to the text
Batch size — The number of inputs or requests processed together in one batch.
Back to the text
CUDA — A software platform for general-purpose computing on NVIDIA GPUs.
Back to the text
GPU — A processor designed to handle many calculations in parallel. It performs model computations during AI inference.
Back to the text

Frequently asked questions

What should you check first in this article?: The Qwen model card lists Qwen2.5-0.5B-Instruct as 0.49B parameters, supports 29+ languages including Korean, uses Apache-2.0, and provides a Transformers example. This recipe uses TRL `SFTTrainer` and PEFT `LoraConfig` targeting `q_proj`, `k_proj`, `v_proj`, and `o_proj`. The sample rows demonstrate format only; replace them with task examples and split train/held-out data by source document.
How much memory does it need?: After training, the adapter and tokenizer are saved in `output/adapter`; generate responses to the held-out questions again and compare them. Record the training loss and actual memory use, then review the held-out answer quality.
Where can you check the conditions and sources?: Use the small Qwen2.5-0.5B-Instruct for this exercise: 0.49B parameters, 29+ languages including Korean, Apache-2.0, and Transformers support make it a setup fixture for data files and adapter saving. For production model selection, check a current card's languages, license, and PEFT target modules. Qwen3.5-0.8B is a multilingual hybrid vision-language model, so verify its training path before transferring this text-LoRA configuration.