Configuring a DataPower API Gateway proxy collector for API discovery

How to add a DataPower® API Gateway proxy data source collector to your API discovery capability.

Before you begin

Before an API discovery DataPower API Gateway proxy collector can collect data from a provider organization, you must first enable sharing of your provider organization analytics data; see Sharing analytics data for API discovery.

One of the following provider organization roles is required to be able to manage sources:
  • Organization Administrator
  • Owner
  • Custom role with the Settings: Manage permission.

About this task

API discovery is an add-on to IBM® API Connect that can be used to discover and add APIs to your API development process. Before you can discover any APIs, you must configure one or more data source collectors. These collectors then get automatically added to the Sources tab in the API Manager UI when the collector sends the first OpenAPI documents to your provider organization.

To configure an API discovery DataPower API Gateway proxy collector, you can either create a proxy API definition, or update an existing proxy API, in your provider organization in API Connect. This new or existing proxy API will contain the endpoint of an existing REST service that you want to use to discover OpenAPI documents from. The discovered OpenAPI documents can then be copied to draft APIs as required, to enable full lifecycle management in API Manager. If you're updating an existing proxy API definition, the minimum fields that are required for collection to occur, are the inclusion of the invoke policy log segment, and both of the set-variable segments, that are shown in the example YAML in Step 1.

Procedure

To configure a DataPower API Gateway proxy collector by using the API Manager UI, complete the following steps. You can also use the toolkit CLI to create your proxy API definition. For general information about how to use the toolkit CLI, see Overview of the command-line tool.
  1. Create a YAML file for your proxy API definition, like the following example:
    swagger: '2.0'
    info:
      version: version
      title: title
      x-ibm-name: name
      contact:
        name: contact_name
    basePath: /
    x-ibm-configuration:
      properties:
        target-url:
          value: target_url
          description: url_description
          encoded: false
      cors:
        enabled: true
      gateway: datapower-api-gateway
      type: rest
      phase: realized
      enforced: true
      testable: true
      assembly:
        execute:
          - invoke:
              title: invoke
              version: 2.0.0
              verb: keep
              target-url: $(target-url)$(request.path)$(request.search)
              follow-redirects: false
              timeout: 60
              parameter-control:
                type: allowlist
                values: []
              header-control:
                type: blocklist
                values: []
              inject-proxy-headers: true
              persistent-connection: true
          - log:
              version: 2.1.0
              title: log
              log-level: default
              mode: gather-only
          - set-variable:
              version: 2.0.0
              title: set-variable
              actions:
                - set: log.custom_data.discoveryServiceCollection
                  value: 'true'
                  type: string
              description: Setting specific collector key
          - set-variable:
              version: 2.0.0
              title: set-variable
              actions:
                - set: log.custom_data.apiTargetURL
                  value: $(target-url)
                  type: string
              description: Setting specific collector target URL
        catch: []
        finally: []
      activity-log:
        enabled: true
        success-content: payload
        error-content: payload
      catalogs: {}
      buffering: true
    paths:
      /{+pathparam}:
        get:
          responses:
            '200':
              description: success
              schema:
                type: string
          consumes: []
          produces: []
        post:
          responses:
            '201':
              description: success
              schema:
                type: string
          consumes: []
          produces: []
        put:
          responses:
            '201':
              description: success
              schema:
                type: string
        patch:
          responses:
            '200':
              description: success
              schema:
                type: string
          consumes: []
          produces: []
        delete:
          responses:
            '204':
              description: success
              schema:
                type: string
          consumes: []
          produces: []
        parameters:
          - name: +pathparam
            in: path
            required: true
            type: string
    schemes:
      - https
    
    Where:
    • version is the version number of the API.
    • title is the title of the API. For example, My proxy API.
    • name is the name of the API. The value of the name field must be a single string that can be used to identify the API in commands, for example, my-proxy-api.
    • contact_name is the name of a contact for the API.
    • target_url is the URL endpoint of the existing REST service that you want to expose. For example, https://myorg.com/api/.
    • url_description is a description of the target URL.
    • $(target-url)$(request.path)$(request.search) is an invoke policy that allows wildcard paths and parameters to be passed after the target_url endpoint of the existing REST service. Suffixing the $(target-url) with $(request.path)$(request.search), means that any number (1-n) of request paths and any number (1-n) of path parameters that are invoked by the users of the API, are captured and processed by the discovery collector, regardless of them not being explicitly defined in the proxy OpenAPI schema definition.
    • log is a log policy that enables a custom log to been added.
    • set-variable are set-variable policies that set a specific collector key called discoveryServiceCollection, and a specific collector target URL called apiTargetURL. The collector uses the discoveryServiceCollection key to filter analytics data for events that are associated with the proxy API definition, and the apiTargetURL to filter the duplicate invokes from different data sources.
    • activity-log is an activity-log policy that decides how complete the discovered OpenAPI reconstruction is. Setting both success-content and error-content fields to payload gives the collector enough data to be able to ensure that the request and response schemas can be created correctly. You can also set these fields to activity or header data, which in both cases means that less of the OpenAPI structure is discoverable. Note that the collector does not look at the payload values, and these values are redacted before being sent from the collector to the discovery capability.
    • paths is set to a wild card value, for example {+pathparam}, to ensure that there is the maximum range of discoverable paths available on the proxy API. The paths.parameters section should be defined to the same value. Under the paths:/{+pathparam} property, stub methods for get, post, patch, put, and delete can be added to give maximum reach through the proxy for all of the operations that can be discovered.
  2. Log in to your provider organization in the API Manager UI.
  3. In the navigation pane, click Develop icon in the API UI navigation pane Develop, then click Add > API.
    The Select API type screen is displayed.
  4. In the Import section, select Existing OpenAPI, then click Next.
  5. Choose one of the following methods to select your YAML file:
    • Drag and drop the file.
    • Browse for the file.
    • Specify the URL of the file.
    The wizard checks the validity of the YAML, and displays a message indicating successful validation.
  6. Click Next to import the selected file.
    A summary of the import confirms that an API definition is created.
  7. Click Edit API.
    Your API definition opens in the Design tab.
  8. Click the Test tab to start making your API discoveries.
    You use the Request field to find your API traffic by passing the GET, POST, PATCH, PUT, and DELETE methods through the proxy API. For example:
    GET https://myorg.com:9443/test/sandbox/users
    Returns a list of users.
    After API traffic is found, the DataPower API Gateway proxy collector becomes available in the API discovery capability.
  9. In the API Manager UI, click the Discover icon The discover icon is the outline of a pair of binoculars in white on a black background., then click the Sources tab to view your DataPower API Gateway proxy collector.
    Sources can have the following status:
    • Enabled - the source is enabled, and synchronizes with the collector according to its configuration.
    • Disabled - the source is disabled, and API Connect won't accept any files from the collector. This status is set by the discovery service if a discovery operation fails.
    • Unhealthy - the source collector is unavailable.
    • Paused - the source is paused, and API Connect won't accept any files from the collector.
  10. Optional: You can click the Options icon The options icon is a set of three vertical black dots on a white background. next to a source to change its status.
    • Pause collector - to pause the source.
    • Delete collector - to delete the source from the API Manager UI. The external source files are not deleted.
    • Resume collector - to restart a paused or disabled source.
  11. Optional: You can click the Manage columns icon The manage columns icon is an outline of a wheel in black on a white background. on the table header to change the order and view of the columns.

Results

Your DataPower API Gateway proxy data source collector is now ready for API traffic discovery. The collector posts updates to API Connect when a manual request for testing is made, or when the proxy API is provided to users, in order to interact with the applications that are accessible on the paths that are available behind the target_url of the proxy. Every time an API definition is discovered and processed, the Last run field of the data source is changed to reflect the activity.

What to do next

You can click the APIs tab in the Discover section of the API Manager UI, and review the API traffic in your data source collector. For more information, see Reviewing your discovered APIs.