Quantized low-rank adaptation fine tuning
Quantized low-rank adaptation (QLoRA) fine tuning is a variant of LoRA that incorporates quantization to further reduce the memory footprint and computational resources that are required during tuning.
How QLoRA works
The QLoRA is a parameter-efficient fine-tuning (PEFT) method that enables you to apply the LoRA tuning method to quantized foundation models. For more information about how low-rank adaptation works, first see Low-rank adaptation fine tuning.
Foundation model quantization is a method by which the resouces required to store a foundation model are reduced by converting the data type of the model weights to a less precise data type. The weights are then returned to their original precision for inferencing.
Models that support quantization can be quantized during the tuning process also. For example, parameter weights of a foundation model that use a 16-bit float format are converted to a 4-bit integer format when the weights are stored and adjusted during a tuning experiment. The tuned model parameter weights are then returned to their original data type for inferencing.
You can use the QLoRA tuning method in watsonx.ai to fine tune only quantized foundation models.