Generating unstructured synthetic data
With the Synthetic Data Generator service, you can create large, high-quality unstructured text datasets tailored to your needs. You can use the unstructured synthetic datasets that are generated to tune and evaluate foundation models for your specific use case. You can access the service both through the watsonx.ai user interface and the watsonx.ai API.
- Required permissions for API
- Admin or Editor role in the project
- A IBM Cloud Identity and Access Management (IAM) token
- A task credential
- Data types
- YAML files saved as project assets for seed data
- PDF and Markdown files as a knowledge base for some data builders
- JSONL for synthetic data generated
- Data size
- Any
Authentication requirements for using the API
To generate unstructured synthetic data programmatically, you must first complete the following setup:
-
Create a project and have the Admin or Editor role in the project. Your project must have an associated Watson Machine Learning service instance.
You can use a GitHub repository to store project assets such as seed data and outputs. For more information, see Accessing a Git repository and Projects with default Git integration.
-
Create an IBM Cloud user API key and use it to get an IBM Cloud Identity and Access Management (IAM) token. For details, see Credentials for programmatic access. The IAM token is passed through the
Authorizationheader in the request body.
Capabilities
Large language models (LLMs) need to train on large datasets to generate output that is customized for your organization. These models require a large amount of high-quality and accurate training data. A small or low-quality dataset is insufficient to successfully train models to generate output that is relevant to your specific use case.
Use the the Synthetic Data Generator service to create large unstructured text datasets by using data builders and data validators that are optimized for generating data for tuning and evaluating foundation models.
Data builders
A data builder leverages the reference documents that you provide to generate synthetic data that mimics the sample seed data. Based on your use case, you can choose from the following data builders:
- Tool calling
- The tool calling data builder creates training datasets that can be used to train AI models to interact with external tools, application programming interfaces (APIs), or systems to enhance their capabilities.
- Text to SQL
- The text to SQL data builder generates synthetic SQL data that contains a natural language statement describing a database operation and an equivalent SQL statement to perform the database operation.
- Knowledge
- The knowledge data builder generates question and answer (QnA) pairs based on examples in documents that are specific to a business domain.
For more information about seed data formats and choosing a data builder, see Data builders and seed data formats.
Ways to work
You can generate unstructured synthetic data in the following ways:
- Graphical user interface
- Use the graphical user interface to create and run jobs that generate unstructured synthetic data. For more information, see Creating jobs to generate unstructured synthetic data.
- Programatically
- Create and run jobs by using the synthetic data generation (SDG) API. For more information, see Creating jobs programmatically to generate unstructured synthetic data .
The synthetic data is generated with foundation models that are provided in watsonx.ai. The format of the generated data is based on sample seed data you provide and the data builder you use. After the foundation model generates the dataset, the data is validated against the data builder's quality requirements and stored in your project asset.
For more information about the API endpoint, see the watsonx.ai API reference documentation.
Workflow
The following diagram shows how REST API generates unstructured synthetic data.
