Divulge storage blobs as files

Divulging blobs refers to saving blobs outside of AS4 Microservice for your own records and analysis. Saving blobs as files can save storage space and help you prepare for archiving.

Divulging is optional and can occur either at blob creation time or blob purge time. You can use settings in AS4 Microservice to control when divulging occurs. The files that are saved by the divulge process are not affected by the operations of AS4 Microservice.

The divulge process includes the following benefits:

  • The data from the blobs can be kept after the blobs are purged from the storage system. This process saves space and improves performance in the storage component.
  • You can write customized scripts to extract the blob data for reporting, analysis, and archiving. For example, you might archive for nonrepudiation purposes.
    CAUTION:
    Saving the blobs as files is not an archiving process. You cannot search for particular files in the directory where you saved them, and you cannot use these files to restore blobs. You can archive files only after you saved the blobs as files outside of AS4 Microservice.

This process is called divulging because the files created from the blobs divulge information that was formerly hidden by the storage system. These files are saved as clear text. Encrypted blobs are automatically decrypted when they are saved as files.

Restriction: Although you can see the file information that was created from the blobs, you cannot modify that information.

You can control the following parts of the divulge process when you provision storage:

  • The root directory where the divulged file is stored.
    The files are saved in the directory divulge_dir\divulge_date\last_two_characters_of_blob_ID. The directory path includes the following items:
    • The divulge_dir is specified when you provision the storage bucket variant.
    • The divulge_date is specified by the system when the blob is divulged. The divulge date uses the format yyyyMMdd and corresponds to the date in the Coordinated Universal Time (UTC) time zone in which the blob was divulged.
    • The last 2 characters of the blob ID are used in the path to spread the divulged files for a single date across multiple directories.

    IBM® recommends that you create your divulge directory in the same file system as the base file path used for the storage system. This setup reduces the chances for errors when blobs in the storage directory are saved as files in the divulge directory. The directories are in the same file system when the source and destination directories are on the same disk.

    Restriction: If your divulge directory and your storage directory are in the same file system, one directory cannot be a subdirectory of the other directory.
  • When divulging occurs (never, when the blob is created, or when the blob is purged).
    • The divulge process is set at the variant level. If you want to change the divulge setting on a variant, you must retire that variant and create a new variant with the same bucket name.
    • Partially uploaded blobs are never divulged. They are only purged.
    • If divulging is scheduled to occur when the blob is created, but does not occur, then the creation of the blob also fails.

When a blob is saved as a file, the following is true:

  • The file is saved in a directory that you can access with archiving and other data extraction tools.
  • The file is not controlled by the processes of AS4 Microservice, including the purge process. If the blob was divulged to a file when the blob was created, the blob can still be purged, but the divulged file cannot be purged.

Planning for divulged files

When you divulge files, you must specify the following items:

  • When you divulge the files (at blob creation time or at blob purge time).
  • The divulge directory. The divulge directory and the base file path directory cannot be in the same subdirectory path.

    For example, you can have the directory paths root_dir\base_file_path and root_dir\divulge_dir, but you cannot have the directory path root_dir\base_file_path\divulge_dir.

  • The bucket that contains the blobs that you are divulging, if you are using more than one bucket. After you create the second bucket, the name of the bucket must be assigned to the NON_REPUDIATION_STORAGE_BUCKET property in the system properties file.
  • If your base file path directory is a CIFS shared folder in Windows 2008, you must turn off caching by using the following procedure:
    Restriction: This procedure applies only to installations of two or more nodes.
    1. Right-click on the folder that is used for your base file path directory.
    2. Select Properties > Sharing > Advanced Sharing > Caching.
    3. Click No files or programs from the shared folder are available offline.
    4. Click OK twice to apply the change and close the dialog boxes.

Maintaining divulged files

The contents of the divulged files are controlled by the creator of the blob. You have the option of accessing blobs through the storage component in AS4 Microservice or the corresponding divulged files by directly accessing the file system directory or folder.

If you do not delete files regularly, the contents of the directory grow in number until any operation that uses the directory (like creating files, deleting files, or listing files) would slow down greatly. However, do not delete files until you are sure that they are no longer needed. Deleting divulged files does not affect the operations of the storage component.

Important: You need to promptly and continuously delete the divulged files to manage the directory space and the file information that you are extracting for your permanent records.

For performance, IBM recommends that you do not delete the divulge date\last two characters of blob ID subdirectory until at least two days after the date when the subdirectory was created. This policy prevents the subdirectory from having to be created again. It also ensures that external archiving programs have access to the subdirectory until it stops getting new files added to it.