Examples

See examples for using the Concert toolkit utilities to streaming data ingestion.

Preparing the environment

To prepare the environment, export the toolkit data directory to the environment variable TOOLKIT_DATA using the export command. Additionally, export the source code directory to the environment variable SRC_CODE_DIR.

export HOST_DIR_SRC_CODE="/home/user/git/sample-app"
export HOST_DIR_TOOLKIT_DATA="/home/user/toolkit-data"

Next, create the toolkit data directory on the host machine and grant read/write/execute permissions for all users.

mkdir -p "${HOST_DIR_TOOLKIT_DATA}"
chmod 777 "${HOST_DIR_TOOLKIT_DATA}"

Then, use the export command to export the toolkit prefix command to an environment variable.

export TOOLKIT_PREFIX_CMD="docker run \
  -v ${HOST_DIR_SRC_CODE}:/data/src \
  -v ${HOST_DIR_TOOLKIT_DATA}:/toolkit-data \
  ${TOOLKIT_IMAGE} \
  /bin/bash -c"

-v ${SRC_CODE_DIR}:/data/src is used to mount the source code directory from the host into the container's /data/src directory. This is only necessary for the code-scan utility to scan the source code.
-v ${TOOLKIT_DIR}:/toolkit-data is used to mount the toolkit data input/output directory from the host into the container. This is necessary to provide the toolkit with the input yaml files, and to access the generated files on the host machine after running the commands.
--rm is used to remove the container after it exits. This is not required, but generally considered a best practice to clean up space.
${TOOLKIT_IMAGE} is the image name and version tag of the toolkit container in the public IBM® Cloud Container Registry.
/bin/bash -c runs the designated command in the container using bash.

Finally, use the ${TOOLKIT_PREFIX_CMD} environment variable to run individual utilities.

Scanning a source code repository (code-scan)

This example generates the CycloneDX package SBOM file codescan-cyclonedx-sbom.json by scanning the code repository located at /data/src. The generated package SBOM file codescan-cyclonedx-sbom.json is produced in the host machine's /home/user/toolkit-data directory.

${TOOLKIT_PREFIX_CMD} "code-scan --src /data/src"

Running command: 
cdxgen -r -o /toolkit-data/codescan-cyclonedx-sbom.json --validate
SBOM has been generated successfully at /toolkit-data/codescan-cyclonedx-sbom.json

Scanning a container image (image-scan)

This example scans the image public.ecr.aws/ubuntu/ubuntu:latest and generate a CycloneDX package SBOM file, by passing the image name as input to the image-scan utility. The generated package SBOM file public.ecr.aws_ubuntu_ubuntu:latest-imagescan-cyclonedx-sbom.json is produced in the /home/user/toolkit-data directory on the host machine.

${TOOLKIT_PREFIX_CMD} "image-scan --images public.ecr.aws/ubuntu/ubuntu:latest"

Scanning image: public.ecr.aws/ubuntu/ubuntu:latest
 ✔ Parsed image                                                                                                                             sha256:6a3f33c07c2bbb03c5582be3510f560431d8498ba1a157c499372612edbc8916
 ✔ Cataloged contents                                                                                                                              fdd0d58847114c72fe1db36565ccbceba55d9dd529c6ad36abb8d6fd6b46c97f
   ├── ✔ Packages                        [91 packages]  
   ├── ✔ File digests                    [2,038 files]  
   ├── ✔ File metadata                   [2,038 locations]  
   └── ✔ Executables                     [722 executables]
SBOM has been generated successfully at /toolkit-data/public.ecr.aws_ubuntu_ubuntu:latest-imagescan-cyclonedx-sbom.json

Generating an application SBOM (app-sbom)

This example generates a Concert-defined (ConcertDef) deploy SBOM file using the app-sbom utility. Using the text editor vim, first create the app configuration file app-config.yaml on our host machine's ${TOOLKIT_DIR} directory.

vim "${TOOLKIT_DIR}/app-config.yaml"

Then, run the app-sbom command. Since the ${TOOLKIT_DIR} directory is mounted to the container's /toolkit-data directory, pass the value /toolkit-data/app-config.yaml to the --app-config option of the app-sbom utility. The resulting output file my-application.json is produced in the /home/user/toolkit-data directory on the host machine.

${TOOLKIT_PREFIX_CMD} "app-sbom --app-config /toolkit-data/app-config.yaml"

2024-09-24 23:59:14,191 - INFO - Reading configuration file: /toolkit-data/app-config.yaml
2024-09-24 23:59:14,193 - INFO - Processing image...
2024-09-24 23:59:14,193 - INFO - Processing repository...
2024-09-24 23:59:14,193 - INFO - Writing SBOM to output file: /toolkit-data/concert-sample-app-sbom.json

Generating a build SBOM (build-sbom)

This example generates a Concert-defined (ConcertDef) build SBOM file using the build-sbom utility. Using the text editor vim, first create the build configuration file build-config.yaml on our host machine's ${TOOLKIT_DIR} directory.

vim "${TOOLKIT_DIR}/build-config.yaml"

Then, run the build-sbom command. Since the ${TOOLKIT_DIR} directory is mounted to the container's /toolkit-data directory, pass the value /toolkit-data/build-sbom-values.yaml to the --build-config option of the build-sbom utility. The resulting output files abc-frontend.json and abc-backend.json are produced in the /home/user/toolkit-data directory on the host machine.

${TOOLKIT_PREFIX_CMD} "build-sbom --build-config /toolkit-data/build-config.yaml"

2024-09-25 00:00:13,180 - INFO - Reading configuration file: /toolkit-data/build-config.yaml
2024-09-25 00:00:13,182 - INFO - Processing component "concert-frontend"
2024-09-25 00:00:13,182 - INFO - Processing image...
2024-09-25 00:00:13,182 - INFO - Processing CycloneDX BOM link...
2024-09-25 00:00:13,182 - INFO - Processing repository...
2024-09-25 00:00:13,182 - INFO - Processing CycloneDX BOM link...
2024-09-25 00:00:13,183 - INFO - Writing SBOM to output file: /toolkit-data/concert-frontend-build-sbom.json

Generating a deploy SBOM (deploy-sbom)

This example generates a ConcertDef deploy SBOM file using the deploy-sbom utility. Using the text editor vim, first create the deploy configuration file deploy-config.yaml on our host machine's ${TOOLKIT_DIR} directory.

vim "${TOOLKIT_DIR}/deploy-config.yaml"

Then, run the deploy-sbom command. Since the ${TOOLKIT_DIR} directory is mounted to the container's /toolkit-data directory, pass the value /toolkit-data/deploy-config.yaml to the --deploy-config option of the deploy-sbom utility. The resulting output file abc-frontend.json is produced in the /home/user/toolkit-data directory on the host machine.

${TOOLKIT_PREFIX_CMD} "deploy-sbom --deploy-config /toolkit-data/deploy-config.yaml"

2024-09-25 00:01:16,899 - INFO - Reading configuration file: /toolkit-data/deploy-config.yaml
2024-09-25 00:01:16,902 - INFO - Processing component #1 in deployments
2024-09-25 00:01:16,902 - INFO - Processing repository...
2024-09-25 00:01:16,902 - INFO - Processing runtime component: ocp-cluster-1 of type kubernetes
2024-09-25 00:01:16,902 - INFO - Processing image...
2024-09-25 00:01:16,902 - INFO - Processing runtime component: rhel-vm-1 of type vm
2024-09-25 00:01:16,902 - INFO - Processing image...
2024-09-25 00:01:16,902 - INFO - Processing runtime component: mydb of type db
2024-09-25 00:01:16,903 - INFO - Writing SBOM to output file: /toolkit-data/concert-deploy-sbom.json

Validating an SBOM file (validate-sbom)

This example validate the SBOM files generated by the code-scan and image-scan utilities. The validate-sbom command receives the two SBOM files codescan-cyclonedx-sbom.json and public.ecr.aws_ubuntu_ubuntu:latest-imagescan-cyclonedx-sbom.json as input.

${TOOLKIT_PREFIX_CMD} "validate-sbom --files /toolkit-data/public.ecr.aws_ubuntu_ubuntu:latest-imagescan-cyclonedx-sbom.json,/toolkit-data/codescan-cyclonedx-sbom.json"

Validating SBOM file: /toolkit-data/public.ecr.aws_ubuntu_ubuntu:latest-imagescan-cyclonedx-sbom.json
Welcome to the sbom-utility! Version `v0.16.0` (sbom-utility) (linux/amd64)
===========================================================================
[INFO] Loading (embedded) default schema config file: `config.json`...
[INFO] Loading (embedded) default license policy file: `license.json`...
[INFO] Attempting to load and unmarshal data from: `/toolkit-data/public.ecr.aws_ubuntu_ubuntu:latest-imagescan-cyclonedx-sbom.json`...
[INFO] Successfully unmarshalled data from: `/toolkit-data/public.ecr.aws_ubuntu_ubuntu:latest-imagescan-cyclonedx-sbom.json`
[INFO] Determining file's BOM format and version...
[INFO] Determined BOM format, version (variant): `CycloneDX`, `1.6` (latest)
[INFO] Matching BOM schema (for validation): schema/cyclonedx/1.6/bom-1.6.schema.json
[INFO] Loading schema `schema/cyclonedx/1.6/bom-1.6.schema.json`...
[INFO] Schema `schema/cyclonedx/1.6/bom-1.6.schema.json` loaded.
[INFO] Validating `/toolkit-data/public.ecr.aws_ubuntu_ubuntu:latest-imagescan-cyclonedx-sbom.json`...
[INFO] BOM valid against JSON schema: `true`
Validation successful for /toolkit-data/public.ecr.aws_ubuntu_ubuntu:latest-imagescan-cyclonedx-sbom.json
Validating SBOM file: /toolkit-data/codescan-cyclonedx-sbom.json
Welcome to the sbom-utility! Version `v0.16.0` (sbom-utility) (linux/amd64)
===========================================================================
[INFO] Loading (embedded) default schema config file: `config.json`...
[INFO] Loading (embedded) default license policy file: `license.json`...
[INFO] Attempting to load and unmarshal data from: `/toolkit-data/codescan-cyclonedx-sbom.json`...
[INFO] Successfully unmarshalled data from: `/toolkit-data/codescan-cyclonedx-sbom.json`
[INFO] Determining file's BOM format and version...
[INFO] Determined BOM format, version (variant): `CycloneDX`, `1.5` (latest)
[INFO] Matching BOM schema (for validation): schema/cyclonedx/1.5/bom-1.5.schema.json
[INFO] Loading schema `schema/cyclonedx/1.5/bom-1.5.schema.json`...
[INFO] Schema `schema/cyclonedx/1.5/bom-1.5.schema.json` loaded.
[INFO] Validating `/toolkit-data/codescan-cyclonedx-sbom.json`...
[INFO] BOM valid against JSON schema: `true`
Validation successful for /toolkit-data/codescan-cyclonedx-sbom.json

Validating a ConcertDef SBOM file (validate-concertdef-sbom)

This example validates the SBOM files defined in the validate configuration file (validate-config.yaml) against the custom Concert-defined (ConcertDef) schema by calling the Concert API. Using the text editor vim, first create the validate configuration file validate-config.yaml in our host machine's ${TOOLKIT_DIR} directory.

vim "${TOOLKIT_DIR}/validate-config.yaml"

Then, run the validate-concertdef-sbom command. Since the ${TOOLKIT_DIR} directory is mounted to the container's /toolkit-data directory, pass the value /toolkit-data/validate-config.yaml to the --validate-config option of the validate-concertdef-sbom utility.

${TOOLKIT_PREFIX_CMD} "validate-concertdef-sbom --validate-config /toolkit-data/validate-config.yaml"
2024-12-06 15:02:45,886 - INFO - Read configuration file: /toolkit-data/validate-config.yaml
2024-12-06 15:02:47,575 - INFO - /toolkit-data/concert-deploy-sbom.json is a valid ConcertDef SBOM.
2024-12-06 15:02:47,575 - INFO - Validate call response /toolkit-data/concert-deploy-sbom.json: {'status': 'Success', 'status_message': 'Sbom validated sucessfully.No errors found'}

Generating a certificate inventory file (cert-inventory)

This example generates a ConcertDef certificate inventory file using the cert-inventory utility. Using the text editor vim, first create the certificate configuration file cert-config.yaml on our host machine's ${TOOLKIT_DIR} directory.

vim "${TOOLKIT_DIR}/cert-config.yaml"

Then, run the cert-inventory command. Since the ${TOOLKIT_DIR} directory is mounted to the container's /toolkit-data directory, pass the value /toolkit-data/cert-config.yaml to the --cert-config option of the cert-inventory utility. The resulting output file cert-inventory.json is produced in the /home/user/toolkit-data directory on the host machine.

${TOOLKIT_PREFIX_CMD} "cert-inventory --cert-config /toolkit-data/cert-config.yaml"

2024-09-07 01:48:40,077 - INFO - Reading configuration file: /app/samples/cert-config-sample.yaml
2024-09-07 01:48:40,080 - INFO - Fetching server certificate from: https://www.google.com
2024-09-07 01:48:40,203 - INFO - Parsing certificate file: /app/samples/certificates/cert.pem
2024-09-07 01:48:40,203 - INFO - Loaded file /app/samples/certificates/cert.pem as PEM certificate.
2024-09-07 01:48:40,203 - INFO - Parsing certificate file: /app/samples/certificates/cert.der
2024-09-07 01:48:40,204 - INFO - Loaded file /app/samples/certificates/cert.der as DER certificate.
2024-09-07 01:48:40,204 - INFO - Parsing certificate file: /app/samples/certificates/cert.pfx
2024-09-07 01:48:40,254 - INFO - Loaded file /app/samples/certificates/cert.pfx as PKCS#12 (PFX/P12) certificate.
2024-09-07 01:48:40,254 - INFO - Constructing certificate from properties entry with serial number: 416730047297653149520437355272553080019035
2024-09-07 01:48:40,255 - INFO - Certificates have been successfully written to /toolkit-data/certificates.json

Converting a CSV file to an OSCAL catalog JSON file (compliance-catalog)

This example generates an OSCAL catalog JSON file using the compliance-catalog utility. Using the text editor vim, first create the configuration file config.yaml on our host machine's ${TOOLKIT_DIR} directory.

vim "${TOOLKIT_DIR}/catalog-config.yaml"

Input file format

The input files must be in CSV or Excel (XLSX) format. The file must contain columns that match the YAML configuration.

Table 1. Example input file (CSV or Excel)
Family	Control ID	Control Name	Control Description	Related Controls
Family 1	AC-1	Access Control 1	Description of AC-1	AC-2, AC-3
Family 1	AC-2	Access Control 2	Description of AC-2	AC-1
Family 2	IA-1	Identification 1	Description of IA-1

Required Columns:

Family: The column that represents the control family, as mapped by columns.control_family in the configuration.
Control ID: The unique identifier for each control (for example, AC-1), mapped by columns.control_id.
Control Name: The name of the control, mapped by columns.control_name.
Control Description: A description of the control, mapped by columns.control_description.

Optional Column:

Related Controls: A list of related controls, mapped by columns.related_controls in the YAML configuration. If present, this column should contain a comma-separated list of control IDs.

Then, run the compliance-catalog command. Since the ${TOOLKIT_DIR} directory is mounted to the container's /toolkit-data directory, pass the value /toolkit-data/config.yaml to the --config-file option of the compliance-catalog utility, and specify the path to the input file.

${TOOLKIT_PREFIX_CMD} "compliance-catalog --input-file /toolkit-data/input-file.csv --config-file /toolkit-data/config.yaml"
2024-09-09 17:13:35,590 - INFO - File saved to /toolkit-data/catalogs/Custom Catalog Test/csv/0.1/Custom_Catalog_Test_0.1.csv
2024-09-09 17:13:35,595 - INFO - Config file saved to /toolkit-data/catalogs/Custom Catalog Test/config/catalog-config.yaml
2024-09-09 17:13:35,628 - INFO - Processing group: Access Control
2024-09-09 17:13:35,667 - INFO - Processing group: Assessment, Authorization, and Monitoring
2024-09-09 17:13:35,683 - INFO - Processing group: Audit and Accountability
2024-09-09 17:13:35,702 - INFO - Processing group: Awareness and Training
2024-09-09 17:13:35,711 - INFO - Processing group: Configuration Management
2024-09-09 17:13:35,747 - INFO - Processing group: Contingency Planning
2024-09-09 17:13:35,774 - INFO - Processing group: Identification and Authentication
2024-09-09 17:13:35,795 - INFO - Processing group: Incident Response
2024-09-09 17:13:35,810 - INFO - Processing group: Maintenance
2024-09-09 17:13:35,822 - INFO - Processing group: Media Protection
2024-09-09 17:13:35,831 - INFO - Processing group: Personally Identifiable Information Processing and Transparency
2024-09-09 17:13:35,843 - INFO - Processing group: Personnel Security
2024-09-09 17:13:35,853 - INFO - Processing group: Physical and Environmental Protection
2024-09-09 17:13:35,872 - INFO - Processing group: Planning
2024-09-09 17:13:35,882 - INFO - Processing group: Program Management
2024-09-09 17:13:35,916 - INFO - Processing group: Risk Assessment
2024-09-09 17:13:35,929 - INFO - Processing group: Supply Chain Risk Management
2024-09-09 17:13:35,943 - INFO - Processing group: System and Communications Protection
2024-09-09 17:13:35,977 - INFO - Processing group: System and Information Integrity
2024-09-09 17:13:36,047 - INFO - Processing group: System and Services Acquisition
2024-09-09 17:13:36,391 - INFO - Catalog successfully generated at /toolkit-data/catalogs/Custom Catalog Test/oscal/json/0.1/Custom_Catalog_Test_0.1.json

Uploading files to Concert (upload-concert)

This example uploads the generated SBOM files to Concert using the upload-concert utility. Using the text editor vim, first create the upload configuration file upload-config.yaml on our host machine's ${TOOLKIT_DIR} directory.

vim "${TOOLKIT_DIR}/upload-config.yaml"

Then, run the upload-concert command. Since the ${TOOLKIT_DIR} directory is mounted to the container's /toolkit-data directory, pass the value /toolkit-data/upload-config.yaml to the --upload-config option of the upload-concert utility.

${TOOLKIT_PREFIX_CMD} "upload-concert --upload-config /toolkit-data/upload-config.yaml"

2024-07-30 17:24:30,853 - INFO - Read configuration file: /toolkit-data/upload-config.yaml
2024-07-30 17:24:31,338 - INFO - Upload response for /toolkit-data/abc-frontend.json: {'record_paths': ['0000-0000-0000-0000/roja/application_sbom/2024/7/30/87104272_abc-frontend.json'], 'job_data': {'message': '200 OK'}}
2024-07-30 17:24:32,065 - INFO - Upload response for /toolkit-data/codescan-cyclonedx-sbom.json: {'record_paths': ['0000-0000-0000-0000/roja/package_sbom/2024/7/30/422159672_codescan-cyclonedx-sbom.json'], 'job_data': {'message': '200 OK'}}