Choosing a foundation model to tune

Foundation models for LoRA and QLoRA fine tuning

A set of IBM-provided base foundation models are available for you to tune in watsonx.ai. The available models differ based on the parameter-efficient fine tuning method.

The provided base foundation models are not available for inferencing from watsonx.ai. The following table shows the models that you can inference to get an idea of the capabilities of a model before you tune the base version of the model.

Low-rank adaptation (LoRA)

The following foundation models are supported:

• llama-3-1-8b

Base foundation models for LoRA

Model to use for experimental inferencing	Model to tune
llama-3-1-8b-instruct	llama-3-1-8b

Quantized low-rank adaptation (QLoRA)

The following foundation model is supported:

• llama-3-1-70b-gptq

Base foundation model for QLoRA

Model to use for experimental inferencing	Model to tune
llama-3-1-70b-instruct	llama-3-1-70b-gptq

Supported model architectures for fine tuning models

Fine tuning to customize a foundation model for a task involves trial and error. Follow these guidelines when you choose a foundation model to fine tune:

• The model has not been extensively fine tuned; base models are easier to adapt with fine tuning.

• You can apply parameter-efficient training fine-tuning (PEFT) techniques like low-rank adaptation (LoRA) and quantized low-rank adaptation (QLoRA), on foundation models with the following architectures:

  • GraniteForCausalLM
  • LlamawithCausalLM

Choosing a foundation model for tuning

To help you choose the best foundation model to tune, follow these steps:

1. Consider whether any measures were taken to curate the data that was used to train the foundation model to improve the quality of the foundation model output.

2. Review other general considerations for choosing a model.

   For more information, see Choosing a foundation model.

3. Consider the costs that are associated with the foundation model, both at inference time and at tuning time. A smaller model, such as a 3 billion parameter model, costs less to tune and is a good place to start.

   Tuning incurs compute resource consumption costs that are measured in capacity unit hours (CUH). The larger the model, the longer it takes to tune the model. A foundation model that is four times the size takes four times as long to tune.

   For more information about CUH costs, see watsonx.ai Runtime plans and compute usage.

4. Experiment with the models in the Prompt Lab.

   Use the largest version (meaning the version with the most parameters) of the model in the same model family for testing purposes. By testing with a larger, more powerful model you can establish the best prompt pattern for getting the output you want.

   Then, you can tune a smaller version of the same model type to save costs. A tuned version of a smaller model can generate similar, if not better results and costs less to inference.

   Craft and try prompts until you find the input pattern that generates the best results from the large foundation model. For details, see Prompt Lab.