LoRA stands for Low-Rank Adaptation, commonly referred to in Chinese as '低級適' (低級配). It is a highly efficient method for fine-tuning parameters: instead of directly changing all parameters of the large model, smaller trainable matrices are added next to certain layers, allowing the model to learn new tasks or styles.
Why it saves costs
Full fine-tuning is like retraining an entire complex machine, with high costs in memory, data, and engineering. LoRA is more like adding a set of replaceable modules to a machine, training only the new parameters for this part. This significantly reduces memory requirements and makes switching between different tasks more convenient.
LoRA is suitable for
It is suitable for allowing open-source models to learn specific writing styles, industry terminology, customer service scripts, structured output formats, image styles, or vertical tasks. Many teams use LoRA to build custom models that are "more stable than prompts but lighter than full training."
It doesn't fit for anything
LoRA is not a magic patch. If the base model doesn't have a certain capability and the data is small and messy, LoRA will find it hard to create strong capabilities out of thin air. It also cannot replace RAG: if the task requires answering frequently changing factual data, retrieving knowledge bases is usually better maintained than fine-tuning data into models.
and QLoRA and full fine-tuning
QLoRA can be understood as performing LoRA on top of quantization, further reducing training resources; Full fine-tuning modifies more original parameters, offering stronger control but higher costs. The practical judgment is: first solve with prompts and RAG; Still need stable style or fixed task performance before considering LoRA; Only when data, budget, and evaluation are mature will heavier training routes be considered.
