Ingesting data

Ingesting data includes creating a root folder, uploading the files, and embedding the files on to the server. You can ingest data into your search database and enhance the AI search capability to fetch data from the documents you upload apart from its capability to search relevant data from other factual sources such as IBM Documentation. Ingesting data enables the AI Assistant to provide answers to any questions related to the content in the files you uploaded.

1. Change the directory to the root folder.

   cd <root folder>

2. (Optional) perform the following steps to add and configure the certificate:
   1. Copy the certificate for zassist to verify the client ingestion server by using the following command:

      oc -n wxa4z-byos get secret wxa4z-client-ingestion-tls-secret -o jsonpath="{.data.ca\.crt}" | base64 -d > ca.crt

   2. Configure the certificate path for zassist by using the following command:

      export WXA4Z_CA_PATH=ca.crt

      Note: If you want to skip the server certificate verification, use the command export WXA4Z_TLS_VERIFY=false.
3. Create a proper file structure with one root folder to upload your documents.
   A root folder can have any number of subfolders and files. Currently, only PDF, HTML, and DOCX file formats are supported.

   root-folder/
   |-- product-docs/
   |   `-- index.html
   `-- redbooks/
       |-- doc1.pdf
       `-- redp5704.pdf

4. Initialize your root folder by using the following command:

   zassist init

   When you initialize the root folder, a config file is automatically created within the root folder.
5. Retrieve the URL for the client ingestion server by using the following command:

   echo https://$(oc -n wxa4z-byos get route wxa4z-client-ingestion -o jsonpath="{.spec.host}")

6. Retrieve the client-ingestion-authkey key by using the following command.

   oc -n wxa4z-byos get secret client-ingestion-authkey -o jsonpath="{.data.authkey}" | base64 -d

7. Log in to the server by using the following command:

   zassist login <url of the server>

   Alternatively, pipe the key into the login command by entering cat key.txt | zassist login <url>.
   Note: Ingesting data by using port forwarding causes issues when accessing data sources through the AI Assistant. For more information, see Known issues and limitations.
8. Ingest files on to the server by using the following command:

   zassist ingest [<folders, files, or both within the root directory>]

9. Upload the files in to zRAG by using the following command:

   zassist load [<folders and files within the root directory>]

   Important: Ensure that your computer does not hibernate or sleep during the process. So, configure your computer settings or use a CLI tool to prevent your computer from hibernating or sleeping.
   You can perform the following tasks after you ingest and load the files:

   • Check the status of both ingested and loaded files:

     zassist status

   • Delete files from the server:

     zassist delete ingested <folders, filesm or both within the root directory>

   • Delete files that are loaded on to the search database:

     zassist delete loaded <folders, files or both within the root directory>

   • Clean up the locally deleted files from the server:

     zassist trim ingested <root folder>

   • Clean up the loaded files that were ingested earlier.

     zassist trim loaded <root folder>