In this tutorial we fine-tune a 7B–8B model on our own instruction dataset using QLoRA. The code is built on the Hugging Face libraries (transformers, peft, trl, and bitsandbytes). These libraries' APIs evolve quickly, so if a parameter name differs in your version, check the official documentation for the version you have installed.

1. Setting up the environment

bash
python -m venv .venv && source .venv/bin/activate
pip install -U torch transformers peft trl bitsandbytes datasets accelerate

2. Data format

We use a JSONL file where each line is a conversation in the messages format. The TRL library prefers this format and applies the model's own chat template automatically.

json
{"messages": [
  {"role": "system", "content": "You are a technical support assistant for Mit AI."},
  {"role": "user", "content": "How do I book a consultation?"},
  {"role": "assistant", "content": "Click the \"Book a free consultation\" button and fill in the form, and we will get in touch."}
]}

3. Loading the model in 4-bit

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

MODEL_ID = "Qwen/Qwen2.5-7B-Instruct"  # or any suitable open model

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    quantization_config=bnb_config,
    device_map="auto",
)
  • nf4: a quantization type designed for normally distributed weights, and the recommended choice in QLoRA.
  • double_quant: also quantizes the quantization constants themselves to save extra memory.
  • Use float16 instead of bfloat16 if your GPU does not support bf16.

4. Configuring LoRA

python
from peft import LoraConfig

peft_config = LoraConfig(
    r=16,                # matrix rank: higher = more capacity and more memory
    lora_alpha=32,       # scaling factor, usually 1–2x r
    lora_dropout=0.05,
    target_modules="all-linear",
    task_type="CAUSAL_LM",
)

5. Training

python
from datasets import load_dataset
from trl import SFTConfig, SFTTrainer

dataset = load_dataset("json", data_files={"train": "train.jsonl", "validation": "val.jsonl"})

training_args = SFTConfig(
    output_dir="out/mit-qlora",
    num_train_epochs=2,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,   # effective batch size = 16
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    bf16=True,
    gradient_checkpointing=True,     # saves memory at the cost of longer run time
    logging_steps=10,
    eval_strategy="steps",
    eval_steps=100,
    save_steps=100,
    max_length=2048,
)

trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset["train"],
    eval_dataset=dataset["validation"],
    peft_config=peft_config,
    processing_class=tokenizer,
)
trainer.train()
trainer.save_model("out/mit-qlora/adapter")

6. Quick test of the result

python
messages = [{"role": "user", "content": "How do I book a consultation?"}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(trainer.model.device)

output = trainer.model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

7. Merging the adapter and exporting the model

For deployment, we load the base model at full precision (fp16/bf16) and then merge the adapter into it. The result is a standalone model that does not need peft at runtime.

python
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(MODEL_ID, torch_dtype=torch.bfloat16, device_map="auto")
merged = PeftModel.from_pretrained(base, "out/mit-qlora/adapter").merge_and_unload()
merged.save_pretrained("out/mit-merged")
tokenizer.save_pretrained("out/mit-merged")

To run it with Ollama or llama.cpp, convert the merged model to GGUF using the convert_hf_to_gguf.py script from the llama.cpp repository, then quantize it (for example Q4_K_M). Details in the comparison of local runtimes.

Tips for better quality

  • Start with r=16 and 1–3 epochs; too many epochs often leads to memorizing the examples.
  • Keep a validation set the model has never seen and watch eval_loss: if it rises while training loss falls, you are overfitting.
  • Train on assistant responses only where possible (assistant-only loss) so the model does not learn to imitate user questions.
  • Always evaluate with a direct side-by-side comparison against the base model on the same questions.

FAQ

How long does QLoRA fine-tuning take for a 7B model?

It depends on dataset size, context length, and the GPU. For a few thousand examples on a single GPU, it usually takes between one hour and a few hours.

Can I use QLoRA with Arabic models?

Yes. The method is language-independent; quality depends on the base model and the quality of your Arabic training data. Models like Qwen support Arabic well.

Want the training result without the trial and error?

Our team trains the model on your business data, tests it, and launches it, so you get a ready assistant without buying hardware or losing weeks to experiments.

Get an assistant trained for your business