Configuring a GitHub collector for API discovery

How to add a GitHub data source collector to your API discovery capability.

Before you begin

Before you can configure an API discovery GitHub collector, the following prerequisites are required:

You must have a GitHub repository that contains one or more OpenAPI reference documents.

One of the following provider organization roles is required to be able to manage sources:

Organization Administrator
Owner
Custom role with the Settings: Manage permission.

About this task

API discovery is an optional add-on to IBM® API Connect that can be used to discover and add APIs to your API development process. Before you can discover any APIs, you must configure one or more data source collectors. These collectors then get automatically added to the Sources tab in the API Manager UI when the collector sends the first OpenAPI documents to your provider organization.

To configure an API discovery GitHub collector, you create a GitHub Actions workflow in your GitHub repository. This workflow enables your OpenAPI reference documents to be automatically sent to, and kept in sync with, API Connect. These OpenAPI documents can then be copied to draft APIs as required, to enable full lifecycle management in API Manager.

Note: When an API with a large payload is sent (for example, a complex API that's more than 1 MB in size), there might be some delay in creating the discovered API. It is recommended that you don't run another workflow simultaneously, until you see that an API has been created in the API discovery service.

Procedure

To configure a GitHub collector, complete the following steps.

Create a .github/workflows directory in your repository on GitHub (if that directory doesn't already exist).
In the .github/workflows directory, create a file for your workflow action, for example discover-api.yml.

In the workflow action file, copy the following workflow job.

The following YAML contains the workflow for the run-discovery job. On a push commit to the GitHub repository, this job checks to see if there are any changes to the files or folders specified and, if there are, the job sends the git difference of the commits in the files to API Connect. API_FILES can be changed to API_FOLDERS when one or more entire folders need to be discovered.

name: Sync Discovered API with APIConnect

on: [pull_request, workflow_dispatch, push]

env:
  API_HOST: <host-name>
  PLATFORM_API_PREFIX: <platform-api>
  PROVIDER_ORG: <porg-name>
  API_FILES: <file/files name>

jobs:
  run-discovery:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - name: Difference
      id: difference-output
      run: |
        echo "action_updates=$(git diff --name-only --diff-filter=ACMRT ${{ github.event.before }} ${{ github.sha }} | xargs)" >> $GITHUB_OUTPUT
    - uses: ibm-apiconnect/apic-discovery-action@main
      id: discover-apis
      with:
        api_host: ${{ env.API_HOST }}
        platform_api_prefix: ${{ env.PLATFORM_API_PREFIX }}
        provider_org: ${{ env.PROVIDER_ORG }}
        api_key: ${{ secrets.apicApikey }}
        if: env.API_FILES
        api_files: ${{ env.API_FILES }}
        else if: env.API_FOLDERS
        api_folders: ${{ env.API_FOLDERS }}
        resync_check: ${{ true }}
        git_diff: ${{ steps.difference-output.outputs.action_updates }}
    - name: Display the action-result
      run: |
        echo "Result of the action: ${{ steps.discover-apis.outputs.action-result }}"
        echo "End"

Note that you can optionally also use the following check_changes_job workflow file. On a push commit to the GitHub repository, this job checks to see if there are any changes to the files or folders specified, or a change to the discover-api.yml file, and if there are, the job sends the files to API Connect. API_FILES can be changed to API_FOLDERS when one or more entire folders need to be discovered.

name: Sync Discovered API with APIConnect

on: [pull_request, workflow_dispatch, push]

env:
  API_HOST: <host-name>
  PLATFORM_API_PREFIX: <platform-api>
  PROVIDER_ORG: <porg-name>
  API_FILES: <file/files name>

jobs:
  check_changes_job:
    runs-on: 'ubuntu-20.04'
    # Declare outputs for next jobs
    outputs:
      action_changed: ${{ steps.check_workflow_changed.outputs.action_updates }}
      changed_filename: ${{ steps.changed_filename.outputs.api_file }}
      apifiles_env: ${{ steps.changed_filename.outputs.apifiles_env }}
      folder_changed: ${{ steps.check_apifolders_changed.outputs.folder_updates }}
    steps:
    - uses: actions/checkout@v4
      with:
        fetch-depth: 2
    - name: Check Workflow changed
      id: check_workflow_changed
      run: |
        echo "action_updates=$(git diff --name-only --diff-filter=ACMRT ${{ github.event.before }} ${{ github.sha }} | grep discover-api.yml | xargs)" >> $GITHUB_OUTPUT
    - name: Changed API File Name
      id: changed_filename
      run: |
        echo "api_file=$(git diff --name-only --diff-filter=ACMRT ${{ github.event.before }} ${{ github.sha }} | xargs)" >> $GITHUB_OUTPUT
        echo "apifiles_env=$(echo $API_FILES)" >> $GITHUB_OUTPUT
    - name: Check API Folders changed
      id: check_apifolders_changed
      run: |
        echo "folder_updates=$(git diff --name-only --diff-filter=ACMRT ${{ github.event.before }} ${{ github.sha }} | grep $API_FOLDERS | xargs)" >> $GITHUB_OUTPUT
  run-discovery:
    runs-on: ubuntu-latest
    needs: [ check_changes_job ]
    if: ${{ (contains(needs.check_changes_job.outputs.apifiles_env,needs.check_changes_job.outputs.changed_filename)) || (needs.check_changes_job.outputs.action_changed) || (needs.check_changes_job.outputs.folder_changed) }}
    steps:
    - uses: actions/checkout@v4
    - uses: ibm-apiconnect/apic-discovery-action@main
      id: discover-apis
      with:
        api_host: ${{ env.API_HOST }}
        platform_api_prefix: ${{ env.PLATFORM_API_PREFIX }}
        provider_org: ${{ env.PROVIDER_ORG }}
        api_key: ${{ secrets.apicApikey }}
        if: env.API_FILES
        api_files: ${{ env.API_FILES }}
        else if: env.API_FOLDERS
        api_folders: ${{ env.API_FOLDERS }}
        resync_check: ${{ needs.check_changes_job.outputs.action_changed && true || false }}
    - name: Display the action-result
      run: |
        echo "Result of the action: ${{ steps.discover-apis.outputs.action-result }}"

Where:

name is the name of your GitHub Action workflow.
API_HOST is the domain name of the API Connect instance where the discovered APIs will be sent. For example, dw-rosa-01.owe.p1.openshiftapps.com.
PLATFORM_API_PREFIX has a default value of platform-api. You can change it to match your system configuration, if different.
INSECURE_SKIP_TLS_VERIFY if set to true, this action skips the validity check for the server certificate. This action might be required if your API Connect system is using self-signed certificates. Note that setting this action to true makes your HTTPS connections insecure.
PROVIDER_ORG is the name of the API Connect provider organization.
API_FILES or API_FOLDERS specifies whether you want single or multiple API files or folders to be considered for the workflow action. Separate multiple files or folders by using a comma, for example mail-api.yaml,my-api.json,APIfolder/pet-exp.json. The API files can be in json or yaml format.
resync_check indicates whether changes to the action, such as at initial creation, should trigger an api-file synchronization.
git_diff checks the difference between the current and the previous commit. If multiple files have changed in the commit, the workflow only runs if one or more of the files supplied in API_FILES or API_FOLDERS has changed.
api_key are the credentials that are required for pushing API files to the API Manager. You can get the API key from the user who has access to the API Manager for posting APIs. For information about how to obtain an API key, see Managing platform REST API keys. You must then store the API key in a GitHub secret, that can then be used by this GitHub Action workflow. In the YAML in this step the api_key is called apicApikey, so if you want to reuse this YAML ensure that you create your secret with the same name. For information about how to store API keys as secrets in GitHub Actions, see Using secrets in GitHub Actions in the GitHub Docs.

To understand more about the apic-discovery-action, and how to use it, see apic-discovery-action and apic-discovery-test on github.com. These API Connect resources include working examples with a test repository.

Commit the workflow file to a branch in your GitHub repository.
Committing the workflow file triggers a push event and runs the sync workflow with API Connect.
In the API Manager UI, click the Discover icon , then click the Sources tab to view your GitHub collector.
Sources can have the following status:
- Enabled - the source is enabled, and synchronizes with the collector according to its configuration.
- Disabled - the source is disabled, and API Connect won't accept any files from the collector. This status is set by the discovery service if a discovery operation fails.
- Unhealthy - the source collector is unavailable.
- Paused - the source is paused, and API Connect won't accept any files from the collector.
Optional: You can click the Options icon next to a source to change its status.
- Pause collector - to pause the source.
- Delete collector - to delete the source from the API Manager UI. The external source files are not deleted.
- Resume collector - to restart a paused or disabled source.
Optional: You can click the Manage columns icon on the table header to change the order and view of the columns.

Results

Your GitHub data source collector is now ready to synchronize with API Connect. When the collector sends the first OpenAPI documents to your provider organization, the collector becomes listed in the Sources tab in the Discover section of API Manager. A GitHub collector posts updates to API Connect only when a difference is detected due to a commit or PR event in the original data source.

What to do next

You can click the APIs tab in the Discover section of the API Manager UI, and review the API traffic in your GitHub source. For more information, see Reviewing your discovered APIs.