Training a language model on your data, on your own machine or servers, is no longer the preserve of the big labs. Thanks to open-weight models and memory-efficient training techniques, a small team can customize a model with 7 to 14 billion parameters on a single GPU. This report explains the available approaches, when to use each, and what you actually need.
First: do you actually need to train a model?
Before any hardware investment, ask yourself: what problem am I trying to solve? Many cases are solved more cheaply than by training.
| Need | Best solution | Reason |
|---|---|---|
| Answers from changing company documents | RAG | Knowledge stays outside the model and is updated without retraining |
| A specific writing style or tone | Fine-tuning (SFT) | The behavior is injected into the weights |
| Fixed JSON output or a narrow repeated task | Fine-tuning | A small trained model can match a large one at lower cost |
| Complete privacy or an offline environment | Local model + RAG or SFT | Data never leaves your infrastructure |
| Mastering a language or domain no model knows | Continued pre-training | Requires massive data and far more resources |
Second: the four training approaches
1. Pre-training from scratch
Building a model from random weights on trillions of tokens. It requires hundreds or thousands of GPUs for weeks, which is unrealistic for most companies. It is only used when you need a very small model for a highly specialized domain, or for research.
2. Continued Pre-training
Continuing to train an existing model on large amounts of raw text from your domain (law, medicine, a particular dialect). It improves the model's "fluency" in the domain, but needs a lot of data and risks what is called Catastrophic Forgetting if not mixed with general data.
3. Full Fine-tuning
Updating all of the model's weights on instruction-and-response pairs. It offers the highest potential quality, but needs memory equal to several times the model size because of optimizer states and gradients.
4. Parameter-Efficient Fine-Tuning (PEFT): LoRA and QLoRA
These freeze the original model weights and add small trainable low-rank matrices (adapters). The result: fewer than 1% of the parameters are trained, and the adapter file is only tens of megabytes. QLoRA loads the base model in 4-bit and trains the adapters on top of it, which makes training a 7B model possible on a GPU with 12–16 GB of memory.
Third: hardware requirements
The following figures are estimates and vary with context length, batch size, and the tool used, but they give you a starting point for planning.
| Model size | 4-bit inference | QLoRA (approx.) | Full fine-tuning (fp16) |
|---|---|---|---|
| 3B – 4B | ~3 GB | 8 GB | ~40 GB+ |
| 7B – 8B | ~5 GB | 12 – 16 GB | ~100 GB+ |
| 13B – 14B | ~9 GB | 20 – 24 GB | Multiple GPUs |
| 30B – 34B | ~20 GB | 40 – 48 GB | Multiple GPUs |
| 70B | ~40 GB | 80 GB or two GPUs | GPU cluster |
- Video memory (VRAM) is the deciding factor, not processing power.
- NVIDIA GPUs have the broadest support (CUDA). Apple Silicon machines are fine for experiments via MLX, and AMD works through ROCm with varying support.
- If you don't own hardware, rent a GPU by the hour for training, then deploy the resulting model locally. Training happens once; inference continues.
Fourth: the complete workflow
- Define the task and the success metric: what counts as a good answer? Prepare 50–200 evaluation examples before any training.
- Choose the base model: Llama, Qwen, Mistral, Gemma. For Arabic, try Qwen first; it is among the strongest open models in its support. Check the model's license.
- Prepare the data: the quality of 1,000 clean examples beats 50,000 noisy ones. See the training data preparation guide.
- Train with QLoRA or LoRA. Code details are in the QLoRA step-by-step tutorial.
- Evaluate: compare against the base model on examples it did not see during training, and review manually.
- Merge and convert: merge the adapter into the model or convert it to GGUF for local running.
- Deploy and monitor: see the comparison of local runtimes and monitor quality on real data.
Fifth: the most commonly used tools
| Tool | When to choose it |
|---|---|
| Hugging Face TRL + PEFT | Full flexibility and clear code; the foundation other tools are built on |
| Unsloth | Faster training and lower memory use on a single GPU |
| Axolotl | Ready-made YAML configs and repeatable experiments instead of writing code |
| LLaMA-Factory | A web UI and configurations for dozens of models |
| MLX-LM | Training on Apple Silicon machines |
Sixth: common mistakes
- Overfitting: the model memorizes the examples and loses its general ability. Watch the validation loss and stop early.
- Wrong chat template: every model has a Chat Template. Using a template different from the model's silently destroys quality.
- Evaluation data leaking into the training data, making results look better than reality.
- Training to inject changing information: facts that change belong in RAG, not in the weights.
- Neglecting security and licensing: make sure you have the right to use the data and that personal information has been removed.
Conclusion
The realistic path for most organizations: an open model of 7–14B, clean data in the thousands rather than millions, QLoRA training taking a day or less, rigorous evaluation, then local deployment. If you want us to shorten the road, our team offers a complete LLM Fine-Tuning service, from data preparation through deployment.
FAQ
How much data do I need to train an LLM?
To customize behavior or style with SFT, a few hundred to a few thousand high-quality examples are usually enough. Quality and diversity matter more than quantity.
Can an LLM be trained on a single GPU?
Yes. With QLoRA you can fine-tune a 7B to 8B model on a GPU with 12 to 16 GB of memory, as long as you set the context length and batch size carefully.
What is the difference between LoRA and QLoRA?
LoRA trains small matrices on top of a frozen model at its original precision, while QLoRA loads the frozen model in 4-bit to save memory and then trains the same matrices.
Prefer to leave it to the experts?
We build, train, and run the right model for your business, whether on your servers or in an isolated environment, with full confidentiality for your data. Start with a free consultation.