Extracting text from documents

Extract text from a complex, highly-structured document to a simpler text-based file format that you can easily incorporate into your RAG solution.

You can use the text extraction REST API to extract text from an input file that is stored in your project as an asset. Text extraction is an asynchronous process that converts one file at a time. You can make parallel method requests to extract text from a set of documents. The text extraction results are stored in your project in one or several files depending on the output formats you specify in your request.

Required permissions: To extract text from documents, you must have the Admin or Editor role in a project.
Required credentials: You must generate credentials to authenticate with watsonx.ai APIs. For details, see Generating a bearer token.

Before you begin

Your administrator must install a set of machine learning models that process documents. The ID for the set of required models is wdu.
Prepare your documents as follows before you add them to your project:
- Remove any password protection from your document.
- If your PDF document is digitally certified, convert your document to another file format, like DOC or DOCX.
Decide the parameters to include in your text extraction request to fit your specific use case. For details, see Text extraction parameters.

Important:

Do not modify or delete the wdu_project_models_dnd project. This project is automatically created during installation and is required to access watsonx.ai LLM services. Deleting the project causes LLM-based functionality, including text KVP extraction, schema generation, and text classification, to fail. Common errors include "Invalid project GUID encountered in request path", "Failed to load model 'semantickvp'", and "Cannot set Project or Space".

Procedure

The following diagram shows the workflow you use to extract structural information about a business document with the text extraction API.

watsonx.ai document text extraction API workflow

Follow these high-level steps to extract text from a business document by using the REST API:

Add the file from which you want to extract text to a storage asset. You can also specify the storage asset in which to store the results of the text extraction process.

You can use the following storage types:
Connected storage asset

Store your document or text extraction results as a connection asset in your project, and then reference the files in the API by a connection ID. Only connection assets that use the Access key and Secret key pair for credentials are supported. For details, see Adding files to reference from the API.

Container in a project or deployment space
To store your input documents and text extraction results in a container in a deployment space or project, upload your input file directly with the asset files API. For details, see Data & AI Common Core Software API documentation.

The input file and extraction results are referenced in the API by the file path as follows:
```
"document_reference": {
  "type": "container",
  "location": {
    "path": "dummy_path/ibm-annual-report-2024-pt1_1-20-1.pdf"
  }
},
"results_reference": {
  "type": "container",
  "location": {
    "path": "dummy_path/results/"
  }
}
```
Use the Start a text extraction request API method to start the extraction process. For API request details, see the REST API request example.

Note the ID that is returned in the metadata.id field. You use this ID to check the status of your request in the next step.
Use the Get the results of the request API method to check the status of your request.

Checking the status is the only way to find out whether the process failed for any reason.

When the status is completed, the extracted text file is available in the specified connected storage asset.
Download the generated file. For results generated in a container in a deployment space or project, you can use the asset files API to download the results of your text extraction request.

REST API request example

The following command submits a request to extract text from the retail_guidebook.pdf file stored in a Cloud Object Storage bucket in the project and saves the extracted results in the results_data folder.

curl -X POST \
  'https://cpd-<namespace-name>.apps.<OCP-domain>/ml/v1/text/extractions?version=2024-10-18' \
  --header 'Accept: application/json' \
  --header 'Content-Type: application/json' \
  --header 'Authorization: Bearer ${TOKEN}'

The request body is as follows:

{
    "project_id": "e40e5895-ce4d-42a3-b699-8ac764b89a09",
    "document_reference": {
      "type": "connection_asset",
      "connection": {
        "id": "5c0cefce-da57-408b-b47d-58f7785de3ee"
      },
      "location": {
        "bucket":"my-cloud-object-storage-bucket",
        "file_name": "retail_guidebook.pdf"
      }
    },
    "results_reference": {
      "type": "connection_asset",
      "connection": {
        "id": "5c0cefce-da57-408b-b47d-58f7785de3ee"
      },
      "location": {
        "bucket":"my-cloud-object-storage-bucket",
        "file_name": "results_data"
      }
    },
    "parameters": {
      "requested_outputs": [
        "assembly",
        "md",
        "html",
        "plain_text",
        "page_images",
      ],
      "languages": [
        "en"
      ],
      "mode": "standard",
      "ocr_mode": "enabled",
      "create_embedded_images": "disabled"
    }
  }

From the response, copy the metadata.id, such as 64162e0a-b05d-4ba6-a688-422893f58663. Specify this ID in the endpoint that you use to check the status of the extraction process.

curl -X GET \
  'https://cpd-<namespace-name>.apps.<OCP-domain>/ml/v1/text/extractions/64162e0a-b05d-4ba6-a688-422893f58663?project_id=e40e5895-ce4d-42a3-b699-8ac764b89a09&version=2024-09-23' \
  --header 'Accept: application/json' \
  --header 'Authorization: Bearer ${TOKEN}'

You can find the location of the extracted results in the response from the results attribute:

"results": {
  "completed_at": "2025-04-28T09:05:42.880Z",
  "location": ["/results_data/assembly.html", "/results_data/assembly.json",
               "/results_data/assembly.md",
               "/results_data/embedded_images_assembly/*.png",
               "/results_data/page_images/*.png"],
  "number_pages_processed": 1,
  "running_at": "2025-04-28T09:05:27.345Z",
  "status": "completed"
}

What to do next

You are ready to use the results of the text extraction as grounding data in your RAG solution. For details, see Adding extracted text to a RAG solution.