Generating unstructured synthetic data

With the Synthetic Data Generator service, you can create large, high-quality unstructured text datasets tailored to your needs. You can use the unstructured synthetic datasets that are generated to tune and evaluate foundation models for your specific use case. You can access the service both through the watsonx.ai user interface and the watsonx.ai API.

Note: You can use the Synthetic Data Generator service to generate unstructured synthetic data only in the Mumbai, Sydney, and Toronto regions.
Cloud platforms
Required permissions for API
Admin or Editor role in the project
A IBM Cloud Identity and Access Management (IAM) token
A task credential
Data types
YAML files saved as project assets for seed data
PDF and Markdown files as a knowledge base for some data builders
JSONL for synthetic data generated
Data size
Any

Authentication requirements for using the API

To generate unstructured synthetic data programmatically, you must first complete the following setup:

  • Create a project and have the Admin or Editor role in the project. Your project must have an associated watsonx.ai Runtime service instance.

  • Associate the IBM watsonx.ai Runtime service with the project if the service is not already associated. For details, see Adding associated services to a project.

  • Create an IBM Cloud user API key and use it to get an IBM Cloud Identity and Access Management (IAM) token. For details, see Credentials for programmatic access. The IAM token is passed through the Authorization header in the request body.

  • Create a task credential.

    A task credential is an API key that is used to authenticate long-running jobs that are started by steps that you will take during the synthetic data generation procedure. You do not need to pass the task credential in the API request. The Synthetic Data Generator service gets the task credentials automatically. For more information about task credentials, see Creating task credentials.

Capabilities

Large language models (LLMs) need to train on large datasets to generate output that is customized for your organization. These models require a large amount of high-quality and accurate training data. A small or low-quality dataset is insufficient to successfully train models to generate output that is relevant to your specific use case.

Use the the Synthetic Data Generator service to create large unstructured text datasets by using data builders and data validators that are optimized for generating data for tuning and evaluating foundation models.

Data builders

A data builder leverages the reference documents that you provide to generate synthetic data that mimics the sample seed data. Based on your use case, you can choose from the following data builders:

Tool calling
The tool calling data builder creates training datasets that can be used to train AI models to interact with external tools, application programming interfaces (APIs), or systems to enhance their capabilities.
Text to SQL
The text to SQL data builder generates synthetic SQL data that contains a natural language statement describing a database operation and an equivalent SQL statement to perform the database operation.
Knowledge
The knowledge data builder generates question and answer (QnA) pairs based on examples in documents that are specific to a business domain.

For more information about seed data formats and choosing a data builder, see Data builders and seed data formats.

Ways to work

You can generate unstructured synthetic data in the following ways:

Graphical user interface
Use the graphical user interface to create and run jobs that generate unstructured synthetic data. For more information, see Creating jobs to generate unstructured synthetic data.
Programatically
Create and run jobs by using the synthetic data generation (SDG) API. For more information, see Creating jobs programmatically to generate unstructured synthetic data .

The synthetic data is generated with foundation models that are provided in watsonx.ai. The format of the generated data is based on sample seed data you provide and the data builder you use. After the foundation model generates the dataset, the data is validated against the data builder's quality requirements and stored in your project asset.

Attention: You incur charges for tokens that the foundation model generates. For details, see Supported foundation models.

For more information about the API endpoint, see the watsonx.ai API reference documentation.

Workflow

The following diagram shows how REST API generates unstructured synthetic data.

watsonx.ai unstructured synthetic data generation API workflow

Billing for token generation

When you use Synthetic Data Generator to generate unstructured synthetic data, costs are not calculated through capacity unit hours. Instead, costs are determined by the underlying usage of tokens by the LLM whenever it processes input data and generates synthetic data.

For more information about pricing, see IBM foundation models and Third-party foundation models.

Learn more