Configuring pagination for the REST API connector in Data Virtualization

You can enable pagination for the REST API connector by downloading and using the Autonomous REST Composer (ARC) tool from Data Virtualization.

About this feature

For your REST API queries, you can use the Autonomous REST Composer (ARC) tool to apply pagination and break high-volume queries into smaller chunks.

The ARC tool lets you leverage these pagination features:

  • Row offset paging: Request a specific number of rows starting from a given row.
  • Page number paging: Request a page by its number.
  • Next page token: Request the next page of results.

To start using the ARC tool, you must first download the tool, and then configure the pagination parameters. After you finish configuring the parameters, download the model file from the ARC tool window and then use it to connect to your REST API from Data Virtualization.

Downloading the ARC tool

To download the ARC tool from the Data Virtualization web client, navigate to the Data sources page, then select Add connections > Download ARC tool.

Configuring the pagination parameters

To use the ARC tool pagination features, you must specify the following parameters. You can also specify optional parameter that can be used by all pagination types.
Row offset paging
Ensure you specify these parameters:
  • PageSizeParameter: Specifies the URI parameter that contains the page size.
  • FirstRowNumber: Specifies the number of the first row. The default value is 0, but other systems might start at 1.
  • RowOffsetParameter: Specifies the URI parameter that contains the starting row number for the set of rows.
Page number paging
Ensure you specify these parameters:
  • PageNumberParameter: When the tool is requesting a page of rows, this is the URI parameter used to contain the page number.
  • PageSizeParameter: Specifies the URI parameter that contains the page size.
  • FirstPageNumber: Specifies the number of the first page. The default value is 0, but other systems might start at 1.
Next page token
Ensure you specify these parameters:
  • NextPageElement: Specifies the element in the current response that contains the token that must be passed in the URI to request the next page.
    • For nested elements, provide a path to the element.
    • If the API returns a URL, a query parameter value, or an HTTP header value that used paging in the body of the response, then set this parameter to specify the element in the response containing the applicable value.
    • If your API uses a starting after paging mode, then set this parameter to the field that contains the data value to pass in the next request. Ensure the value follows this format: <object_name>/<field_name>.
  • NextPageParameter: Specifies the URI parameter that holds the token used to fetch the next set of results. This parameter uses the value found on the current page at the location that is specified by the parameter NextPageElement.
    • In a starting-after scenario, specify the query parameter that the driver needs to use to tell the API which data value to continue from.
  • PageSizeParameter: Specifies the URI parameter that indicates how many results to return per page.
  • HasMoreElement: Specifies the element in the response that indicates whether another page exists.
    • For nested elements, provide a value that includes a slash (/) separated path.
    • If the service omits this element, or returns False, then this means there are no additional pages.
  • NextPageRequestHeader: Specifies the HTTP request header that must be included in the request to retrieve the next page of results.
  • NextPageResponseHeader: Specifies the HTTP response header that contains the value used to request the next page.
    • This value can be a URL, a query parameter value, or an HTTP header value that is passed in the next request to return the next page.
Optional parameters applicable to all pagination types
You can specify any of these parameters:
  • HasMoreElement: Specifies the element in the response that indicates whether there is another page.
    • For nested fields, provide a value that includes a slash (/) separated path.
    • If this field is missing or has a value of False, then there are no additional pages.
  • FieldListParameter: Specifies the URI parameter used to request a comma-separated list of fields from the service. For example, if you issue the query SELECT * FROM ORDERS with the FieldListParameter=fields, then the request is issued for an entry containing the id and success fields: https://www.example.com/ORDERS?fields=id%2Csuccess.
  • MaximumPageSize: Specifies the maximum number of rows that can be requested per page.
  • PageSizeElement: Specifies the element in the response that contains the number of rows that must be passed in the URI to retrieve the next page of results.
    • For nested elements, use a slash (/) separated path.
  • TotalPagesElement: Specifies the element in the response that contains the total number of pages contained in the results.
    • For nested elements, provide a value that includes a slash (/) separated path.
  • TotalRowsElement: Specifies the element that contains the total number of rows contained in the results.
    • For nested elements, provide a value that includes a slash (/) separated path.

What to do next

After you finish configuring the parameters, download the model file from the ARC tool window and then use it to connect to your REST API from Data Virtualization. See REST API.