Building a generative AI task in a service flow

You can author a generative AI task to implement generative AI logic in your service flow. Generative AI tasks enable you to send a prompt to a large language model (LLM) and to receive the generated output.

Before you begin

To use the generative AI tooling in the service flow editor, you must first configure the server with a connection to the watsonx.ai™ provider. If you navigate to the Gen AI tab in the service flow editor and the server is not configured, a page displays an error message and a link to the configuration documentation.

To learn how to configure generative AI tasks, see Enabling generative AI External link opens a new window or tab.

To learn how to use generative AI to process documents, see Processing documents with generative AI External link opens a new window or tab.

After the server is configured, a new AI entry (AI icon) for the generative AI task is added to the Activity section of the Service Flow editor palette.

Procedure

  1. Open the process application in the designer, and open an existing service flow or create a new one from the navigation tree on the left side.
  2. In the Diagram view of the service flow, click the arrow besides the Activity icon in the component palette to open the sub-palette, and drag the AI icon (AI icon) onto the diagram canvas.
  3. To configure the generative AI task, click that task in the diagram, go to the Implementation tab, and click Open Gen AI, or click the Gen AI tab at the top of the editor, and click the task name in the left-hand panel.
    An editor opens in the Gen AI tab.
  4. Select the large language model (LLM) that you want to use for your prompt.
    Each entry includes a short description of the LLM.
    The list of LLMs depends on how generative AI is enabled.

    For more information about the available LLMs from watsonx.ai, see the Foundation model library External link opens a new window or tab .

    Important: Some LLMs in the list use the deprecated watsonx.ai text/generation API and are tagged as Deprecated API. Do not use these models for new generative AI tasks. This API will be deprecated and removed in March 2027. See the watsonx announcement External link opens a new window or tab.
    • For new generative AI tasks: Select only models that are not tagged as Deprecated API to ensure long-term compatibility.
    • For existing generative AI tasks: If your task uses a model tagged as Deprecated API from a previous release, update the task to use a non-deprecated model to avoid service disruption after the API removal.
    By default, models that use a deprecated API are hidden from the list. To view and select these legacy models (for updating existing tasks only), select Allow deprecated models in the Large language model section.
  5. Create a prompt that will be sent to the LLM for processing.
    The prompt consists of two parts, Context and Input. How these fields are used depends on the type of LLM that you selected.
    • For standard models (recommended):
      • Context field: Contains the system instructions that define the AI's role, behavior, and guidelines. This is sent as system message in the chat request.
      • Input field: Contains the user's query or data to be processed. This is sent as user message in the chat request.
      • Both fields are required for standard models to function properly.
    • For models tagged as Deprecated API (legacy only):
      • Context field: Contains instructions and background information.
      • Input field: Optional data provided to context. The input may not be required for every context.
      Important: Models tagged as Deprecated API use the deprecated text/generation watsonx.ai API and should only be used for maintaining existing generative AI tasks from previous releases. Migrate these tasks to standard models before March 2027 to avoid service disruption when the deprecated API is removed.
    To create your prompt:
    1. In the Context field, enter your instructions:
      • For standard models, define the AI's role and behavior. For example, You are a helpful assistant that summarizes business documents in 3 bullet points.
      • For deprecated API models, provide an imperative statement with instructions.
    2. In the Input field:
      • For standard models, enter the user query or data to process. For example, Summarize the following contract or What are the key terms in this agreement?.

      • For deprecated API models, optionally, enter the text to which you want the model to respond.

    For examples on how to create effective prompts, see Sample prompts and the references listed in What steps are required to implement a generative AI task?.
  6. Optional: Add variables to make the context and input more reusable.
    1. Create variables in the service flow, if needed:
      • Go to the Variables tab and, for each input and output variable required in the service flow, click the + (plus) sign next to the Variables section. For Name, replace the default name with a more descriptive name. For Variable type, select String System Data.
      • Optionally, you can set a default variable value by selecting Has default and entering the default value in the corresponding field. Note that the default value that you set here will be used at run time only if the variable is not set by another value, and will not appear in the Test data section of the Gen AI tab.
    2. Add variables to your prompt:
      • Add variables from the service flow into the Context or Input fields by clicking the + (plus) sign or by pressing Ctrl+Space. The variable name then displays in the related field enclosed in double curly brackets (for example, {{topic}}). Only variables of type String that are not a List are supported and can be selected for addition. Alternatively, you can manually enter the variable structure directly in the Context or Input fields (for example, {{attribute}}), even if the variable has not yet been created in the Variables tab.
      • When you move focus away from the text area, any variables referenced in your prompt that are not yet listed in the Test data section are automatically added, regardless of whether they exist in the Variables tab. Conversely, if a variable is no longer referenced in the Context or Input fields, it is automatically removed from the Test data table.
    3. Test your prompt with different values:
      • To see how different inputs affect the LLM’s output, enter a test value for each variable in the Value field within the Test data section. These values are used for testing only and are not applied at run time.
      • If you reference a variable that has not yet been created in the Variables tab, you might see warnings; however, you can still test your prompt using values in the Test data section. Once your prompt is finalized, be sure to create any missing variables in the Variables tab.
        Warning: If a variable is deleted from the Variables tab but is still referenced in your prompt’s Context or Input, you can continue to use and test it in the Test data section of the Gen AI tab. However, this will result in a runtime error when the task executes. To avoid this, ensure that all variables referenced in your prompt exist in the Variables tab before deployment.
    4. Optional: Add document variables for document processing:
      If you want to process documents in your prompt, you can add a document variable of type ECMDocumentInfo or ECMDocument by clicking the attachment icon beside the + (plus) icon in the Input area. These variables must already exist in the Variables tab. You can then select the server file to use for testing in the Test data section.
  7. Optional: Configure generation parameters to control the LLM output.
    You can configure parameters that control how the LLM generates its response from the Parameters pane. The available parameters depend on the type of model you selected:
    • For standard models:
      • Maximum generated tokens: Specify the maximum number of tokens that the LLM can generate. The default value is 50, and the minimum value is 1.
      • (Optional) Temperature: Control the randomness of the generated output. Values range from 0 to 2, where:
        • Lower values (closer to 0) make the output more focused and deterministic.
        • Higher values (closer to 2) make the output more creative and varied.
        • This field can be left empty to use the default temperature setting that is set by the AI provider.
    • For models with the Deprecated API tag:
      1. Minimum generated tokens: Specify the minimum number of tokens that the LLM must generate. The default value is 1, and the minimum value is 1.
      2. Maximum generated tokens: Specify the maximum number of tokens that the LLM can generate. The default value is 50, and the minimum value is 1.
      Important considerations
      • What is a token? A token is a sequence of characters that holds semantic meaning for a model. Before LLM processing, the words in your prompt text are converted into tokens. Similarly, the raw output from the model is also in the form of tokens. These output tokens are then converted back into words to be displayed as the final result.
      • Token limits: Each LLM has a maximum limit on the total number of tokens, which includes both the tokens in the input prompt and the tokens in the generated output. When provided by the AI provider, you can view those limits in the LLM descriptions in the Large language model drop-down menu. For more information about tokens, see Tokens and tokenization External link opens a new window or tab.
      • Cost optimization: The cost of calling the LLM depends on the number of tokens it consumes and produces. To ensure that the response is relevant without being overly verbose, it is recommended to limit the result. For example, if a "yes" or "no" response is expected, you can set the maximum number of tokens to 1.
  8. Generate and review the output: click Generate to send your prompt to the LLM and generate output.
    Tip: Click RAW to view the information that is sent to the LLM, including the context, input, and any training examples.
    Review the result in the Generated Output pane. To view the prompts used to generate the output, click the Raw prompt icon.
  9. Optional: Add training examples to improve precision and relevance of the generative AI task. You can add training examples that show the LLM the type of output you expect for specific inputs.
    1. Create a training example from generated output:
      After generating satisfactory output, click Save as example. The prompt input and generated output are saved as a training example and listed in the Training examples section. To refine the example, you can modify its name, input, and output.
    2. Manage training examples:
    • Edit examples: You can edit the input text and expected output text of any training example in the Training examples section.
    • Rename examples: Click the example name to rename it for better organization.
    • Document examples: For examples that include document variables, you cannot update the document directly in the example. To use a different document:
      1. Delete the example.
      2. Change the document in the Test data section.
      3. Click Generate again.
      4. Click Save as example to create a new example with the different document.
    Important: Providing the LLM with appropriate training examples will result in better-matched outputs. However, be sure to not exceed the maximum token input limit set by the LLM and the overall maximum token limit for both input and generated output tokens. Each training example adds to the total token count sent to the LLM. To further improve your prompt results, you can:
    • Add multiple training examples that cover different scenarios.
    • Iterate by generating output, reviewing results, and saving successful examples.
    • Edit existing examples to fine-tune the expected outputs.
  10. Map the generated output to the variables in your service flow by going to the Data mapping section in the properties pane of the Diagram tab.
    The output of the task is mapped to the variables in the service flow so that the output can be used by other downstream tasks or calling processes or service tasks. There is no input mapping to fill in, as it is implicitly done when adding variables in the Gen AI tab. Four outputs from the task are available to map:
    • Generated text: The LLM output.
    • Generated token count: The number of tokens that are generated by the LLM for the Generated text.
    • Input token count: The number of tokens that are consumed by the LLM for the given input.
    • Stop reason: The reason why the call stopped. It can be:
      • not_finished: Possibly more tokens to be streamed.
      • max_tokens: Maximum requested tokens reached.
      • eos_token: End of sequence token encountered.
      • canceled: Request canceled by the client.
      • time_limit: Time limit reached.
      • stop_sequence: Stop sequence encountered.
      • token_limit: Token limit reached.
      • error: Error encountered.

Results

The generative AI prompt you configured for a generative AI task in a service flow can now be called from within your process application.
Note: Error handling for generative AI tasks is done in the same way as for all tasks in a service flow, by using boundary events. The generative AI task may raise an error event with error code GenAIError and error data type String. The error data value is a JSON string for an object that contains details on the LLM error: properties errorCode and errorMessage will be included when they are available from the LLM. For more information, see Handling errors in services.