Storage API (Open Data for Industries)

Store data on the storage layer.

The Storage API is used to store data and metadata information that is generated by any industry applications into the Open Data for Industries data layer. The Storage API provides a set of API methods to manage the entire metadata lifecycle such as ingestion (persistence), modification, deletion, versioning, and data schema.

To use the API, see the Storage API reference.

Learn more

Accessing the Storage API

The Storage API relies on the access authorization mechanisms of the Entitlement API to provide access control on the records. When the Storage API caller is a federated user, no additional configuration is necessary. However, if the API caller is a service account, you need to do a mandatory configuration in the Entitlement service.
skipdupes

The skipdupes parameter is used to update operations. You are calling the API with record IDs, which are already present into the data ecosystem.

If the skipdupes parameter is set to true, the API does not update the record if the payload is duplicated. If the payload is different, a new version of the record is created.

If the skipdupes parameter is set to false, the API does not check whether the payload is the same or not and it creates a new version, even if this new version is identical to the previous one.

On the response side, the skipedRecordIds parameter represents the record IDs that are skipped because skipdupes was set to true and the payload is the same.

In the PUT response, the record IDs have no more replication. They are specified in either the recordIds or skippedRecordIds parameters.

Record structure

From the Storage API perspective, the metadata to be ingested is called record. The following example shows a record with a brief explanation of each field:

{
   "id": "common:hello:123456",
   "kind": "common:test:hello:1.0.0",
   "acl": {
     "viewers": ["data.default.viewers@common.[odi.ibm.com]"],
     "owners": ["data.default.owners@common.[odi.ibm.com]"]
   },
   "legal": {
     "legaltags": ["common-sample-legaltag"],
     "otherRelevantDataCountries": ["FR","US","CA"]
   },
   "data": {
     "msg": "Hello World, data ecosystem!"
   }
}
Table 1. Record structure description
Parameters Description
id Unique identifier in the data ecosystem. When this parameter is not provided, the service creates and assigns an id to the record. The parameter value must follow the naming convention: {Data-Partition-Id}:{object-type}:{uuid}.

This parameter is optional.
kind The kind of data, which is ingested. Must follow the naming convention: {Data-Partition-Id}:{dataset-name}:{record-type}:{version}.

This parameter is mandatory.
acl Group of users who have access to the record.
acl.viewers
List of valid groups, which have view or read privileges over the record. The naming convention is that data groups begin with "data".
acl.owners
List of valid groups, which have write privileges over the record. The naming convention is that data groups begin with "data".

This parameter is mandatory.
legal Attributes, which represent the legal constraints that are associated with the record.
legal.legaltags
List of legal tag names with the record.
legal.otherRelevantDataCountries
List of other relevant data countries. Must have at least two values: where the data was ingested from and where data ecosystem stores the data.

This parameter is mandatory.
data Record payload that is represented as a list of key-value pairs.

This parameter is mandatory.

Using Storage API

To use the Storage API, follow these steps:
  1. Choose a partition:

    The Open Data for Industries data lake stores data in different tenants, depending on the different accounts on the system. A user might belong to many accounts. For example, a user might belong to both the own account and a customer's account. When you log in to the industry applications, you choose which account is active. To use the Storage API, specify the active account by using the data-partition-id parameter.

    The correct values can be obtained from FileNet Content Federation Services services.

  2. Create data groups:

    Create data groups and assign users to these groups. Data groups use the prefix data. For data access authorization purposes, in this example, the following groups must be previously created by using the Entitlements API. For more information, see the Entitlements API.

    • data.default.viewers@instance.odi.ibm.com
    • data.default.owners@instance.odi.ibm.com
  3. Use the API methods to store data on the storage layer of Open Data for Industries.
    Table 2. Storage API classification
    API endpoint Description API reference
    GET /api/storage/v2/query/kinds The query endpoint to get all kinds. Get kinds of storage
    GET /api/storage/v2/query/records This API returns all the records of a particular kind optionally bounded by the limit and the cursor. Get all records
    POST /api/storage/v2/query/records This API returns record details for all the record identifiers as part of the request body. Get records
    POST /api/storage/v2/query/records:batch The API fetches multiple records (maximum 20) from the Storage API at once. It allows user to request the data to be converted to a common standard by using customized header {frame-of-reference}. Common standards are units in SI, crs in wgs84, elevation in msl, azimuth in true north, dates in Coordinated Universal Time. Currently only none and units=SI; crs=wgs84; elevation=ms; azimuth=true north; dates=utc; are valid values for the header {frame-of-reference}. Fetch records
    PUT /api/storage/v2/records The API performs a upsert on the record payload. Create or update records
    PATCH /api/storage/v2/records The API performs a bulk update to the records in the body of the request. Update records metadata
    GET /api/storage/v2/records/{id}/{version} The API retrieves the specific version of the record. Get specific record version
    GET /api/storage/v2/records/versions/{id} The API returns a list, which contains all versions for the record ID. Get record version
    GET /api/storage/v2/records/{id} This API returns the latest version of the record. Get latest record version
    DELETE /api/storage/v2/records/{id} The API performs a physical deletion of the record. This operation can not be reverted later. Purge record
    POST /api/storage/v2/records/{id}:delete The API performs a logical deletion of the record. This operation can be reverted later. Delete record
    POST /api/storage/v2/records/delete The API performs a logical deletion of list of records. This operation can be reverted later. Bulk Delete Records

Storage API endpoint permissions

Table 3. API method permissions
Endpoint URL Method Minimum permissions required
/api/storage/v2/query/kinds GET service.storage.creator
/api/storage/v2/query/records:batch POST service.storage.viewer
/api/storage/v2/records/{id}/{version} GET service.storage.viewer
/api/storage/v2/records/versions/{id} GET service.storage.viewer
/api/storage/v2/records/{id} GET service.storage.viewer
/api/storage/v2/records/{id}:delete POST service.storage.creator
/api/storage/v2/records PUT service.storage.creator
/api/storage/v2/records/{id} DELETE service.storage.admin
/api/storage/v2/records/delete POST service.storage.creator
/api/storage/v2/query/records GET service.storage.admin
/api/storage/v2/query/records POST service.storage.viewer
/api/storage/v2/records PATCH service.storage.creator