Storage API (Open Data for Industries)

Store data and manage the metadata lifecycle on the Open Data for Industries storage layer.

Storage API controls the data and metadata information that is generated by any industry application and stores it in the Open Data for Industries data lake. The API methods manage the metadata lifecycle, which consists of:
  • Ingestion
  • Modification
  • Deletion
  • Versioning
  • Data schema

To use the API, see the Storage API reference.

Learn more

Accessing the Storage API

Storage API relies on Entitlement API to provide access control on the ingested metadata (or the record).

When the Storage API caller is a federated user, no additional configuration is necessary.

However, if the API caller is a service account, a mandatory configuration in the Entitlement API is needed.
skipdupes
The skipdupes parameter is only related to update operations. You are calling the Storage API with the record ID, which is already present in the data ecosystem.

  • If the skipdupes is set to true, the API checks the payload.

    • If the payload is the same, the API does not update the record.

      If the payload is different, a new version of the record is created.

  • If skipdupes is set to false, the API does not check whether the payload is the same. It creates a new version, even though it is identical to the previous.

skipedRecordIds

On the response side, the skipedRecordIds parameter represents the record IDs that weren't updated because skipdupes was set to true and the payload is the same.

In the PUT response, the record IDs have no more replication. They are specified in either the recordIds or skippedRecordIds parameters.

Record structure

The metadata that is ingested is called record.

The following example shows a sample record.

{
   "id": "common:hello:123456",
   "kind": "common:test:hello:1.0.0",
   "acl": {
     "viewers": ["data.default.viewers@common.[odi.ibm.com]"],
     "owners": ["data.default.owners@common.[odi.ibm.com]"]
   },
   "legal": {
     "legaltags": ["common-sample-legaltag"],
     "otherRelevantDataCountries": ["FR","US","CA"]
   },
   "data": {
     "msg": "Hello World, data ecosystem!"
   }
}
Use the following table to understand more about the record parameters.
Table 1. Record structure
Parameters Importance Description
id Optional A unique identifier in the data ecosystem.
  • When this parameter is not provided, the service creates and assigns an id to the record.
  • The parameter value must follow the naming convention: {Data-Partition-Id}:{object-type}:{uuid}.
kind Mandatory Parameter that shows what kind of data is ingested.
  • The parameter value must follow the naming convention: {Data-Partition-Id}:{dataset-name}:{record-type}:{version}.
acl Mandatory Group of users who have access to the record.
acl.viewers
List of valid groups, which have view or read privileges over the record.
acl.owners
List of valid groups, which have write privileges over the record.
The naming convention is that data groups begin with data.
legal Mandatory Attributes for the legal constraints that are associated with a record.
legal.legaltags
List of legal tag names with the record
legal.otherRelevantDataCountries
List of other relevant data countries.
Must have at least two values:
  1. Where the data was ingested from.
  2. Where the data ecosystem stores the data.
data Mandatory Record payload that is represented as a list of key-value pairs.

Using Storage API

To use the Storage API, you need to:
  1. Choose a partition.

    The Open Data for Industries data lake stores data in different tenants.

    A user might belong to many accounts. For example, a user might belong to both the own account and a customers account.

    When you log in to the industry applications, you choose which account is active.

    To use the Storage API, specify the active account by using the data-partition-id parameter.

  2. Create data groups and assign users to these groups.

    Data groups use the prefix data.

    For data access authorization purposes, the following groups must be previously created by using the Entitlements API. For more information, see the Entitlements API.

    • data.default.viewers@instance.odi.ibm.com
    • data.default.owners@instance.odi.ibm.com
  3. Use the API methods to store data.
    Table 2. Storage API classification
    API endpoint Description API reference
    GET /api/storage/v2/query/kinds The query endpoint to get all kinds. Get kinds of storage
    GET /api/storage/v2/query/records This API returns all the records of a particular kind that is optionally bounded by the limit and the cursor. Get all records
    POST /api/storage/v2/query/records This API returns record details for all the record identifiers as part of the request body. Get records
    POST /api/storage/v2/query/records:batch The API fetches multiple records (maximum 20) from the Storage API at once. It allows user to request the data to be converted to a common standard by using customized header {frame-of-reference}. Common standards are units in SI, crs in wgs84, elevation in msl, azimuth in true north, dates in Coordinated Universal Time. Currently only none and units=SI; crs=wgs84; elevation=ms; azimuth=true north; dates=utc; are valid values for the header {frame-of-reference}. Fetch records
    PUT /api/storage/v2/records The API performs an upsert on the record payload. Create or update records
    PATCH /api/storage/v2/records The API performs a bulk update to the records in the body of the request. Update records metadata
    GET /api/storage/v2/records/{id}/{version} The API retrieves the specific version of the record. Get specific record version
    GET /api/storage/v2/records/versions/{id} The API returns a list, which contains all versions for the record ID. Get record version
    GET /api/storage/v2/records/{id} This API returns the latest version of the record. Get latest record version
    DELETE /api/storage/v2/records/{id} The API performs a physical deletion of the record. This operation cannot be reverted later. Purge record
    POST /api/storage/v2/records/{id}:delete The API performs a logical deletion of the record. This operation can be reverted later. Delete record
    POST /api/storage/v2/records/delete The API performs a logical deletion of list of records. This operation can be reverted later. Bulk Delete Records

Storage API endpoint permissions

Table 3. API method permissions
Endpoint URL Method Minimum permissions required
/api/storage/v2/query/kinds GET service.storage.creator
/api/storage/v2/query/records:batch POST service.storage.viewer
/api/storage/v2/records/{id}/{version} GET service.storage.viewer
/api/storage/v2/records/versions/{id} GET service.storage.viewer
/api/storage/v2/records/{id} GET service.storage.viewer
/api/storage/v2/records/{id}:delete POST service.storage.creator
/api/storage/v2/records PUT service.storage.creator
/api/storage/v2/records/{id} DELETE service.storage.admin
/api/storage/v2/records/delete POST service.storage.creator
/api/storage/v2/query/records GET service.storage.admin
/api/storage/v2/query/records POST service.storage.viewer
/api/storage/v2/records PATCH service.storage.creator