Definition
Local model training means taking an open-weight model and fine-tuning it on your own examples, so it learns your terms, formats or task. Methods such as LoRA and QLoRA change only a small part of the model, which keeps training within reach of a single GPU.
The trained model is then compressed (quantised) and runs on your own servers or computers. Your data and the model stay in-house.
Fine-tuning is not always the first step. Good instructions or a RAG setup often solve the problem with less effort. Training pays off when a task repeats at volume, needs a fixed format or must run without sending data outside.
Applications
- Teaching a model your domain terms and writing style.
- Classifying or extracting fields from private documents.
- Assistants that must work offline or on-premises.
- A small, fast model for one task instead of a large general one.
Well-known tools
Tools that are widely used for this kind of work, given as examples from the field. The right choice depends on your data, your systems and where the solution has to run.
DeepSeek
Open models
Qwen
Open-weight LLMs
Hugging Face
Model hub
Unsloth
LLM fine-tuning
llama.cpp
LLM inference
Ollama
Local LLMs
- PEFT: a Hugging Face library for LoRA and other light fine-tuning methods.
- TRL: a Hugging Face library for supervised fine-tuning and preference training.
- Axolotl: a configuration-driven fine-tuning tool.
- vLLM: a fast engine for serving models.
Real-world problems
Data that cannot leave
Privacy rules or contracts may forbid sending data to a cloud model. A local model keeps everything inside.
General models miss your terms
Off-the-shelf models do not know internal names, codes or formats. Fine-tuning on real examples teaches them.
Cloud cost at volume
Paying for every cloud call adds up for high-volume, repetitive tasks. A small local model can do one job cheaply.
No internet
Some sites and devices work offline. A local model keeps working without a connection.