Content

There are no explicit limits in the P8 code that prevent an object store from storing billions of documents and custom objects. However, each organization is likely to have practical concerns that limit the size to which an object store should grow. These concerns include, but are not limited to

• Time required to perform database backups and restores
• Time required to perform re-indexing, add new indexes, run statistics, and reorg the database
• Search performance
• Ingest performance

This paper describes methods that can be used to

• Limit the size of an object store
• Divide up content to replace a single monolithic object store with a series of smaller, more manageable object stores

Planning Ahead

The best course of action to prevent running into issues with an object store that has grown too large, is to consider how to split up data ahead of time. Both the Content Platform Engine APIs and the out of the box web interface, Content Navigator support the use of cross-repository searching. So, splitting content across multiple object stores does not need to make the information harder for end users to find. Strategies for using multiple object stores include

• Providing each department or segment of the organization with their own object store. This option has the advantage of making it easier to tailor an object store for a particular set of use cases.
• Identifying how long content must be kept, setting appropriate retention schedules, and running regular disposal sweeps to delete content that is no longer needed.

  Content Platform Engine supports fixed, variable, and permanent retention. Default settings can be put on document classes so that content is automatically assigned an appropriate retention setting when added to the repository.

  If retention depends on an event occurring, for instance, an employee leaving a company, a mortgage being paid off, or a project completing, use variable retention to ensure that the content cannot be deleted too soon. And then use the Content Platform Engine event mechanism and sweeps to update the retention to a specific period when the appropriate event occurs.

• If content actions need to be audited, ensure that only the required information is audited as auditing can impact performance. And, regularly export and then prune the audit log. If needed, add the exported audit logs to a separate object store.
• If workflows are being used, prune the workflow logs regularly.
• Rolling to a new object store regularly. The frequency of rolling over and when to roll over, depends on the rate of content being added to the repository and when logical break points occur. For instance, if users typically look for content related to a fiscal year, then rolling to a new object store each fiscal year or every five fiscal years might be a good strategy. When using this option, it is important to
  1. Use a tool like FileNet Deployment Manager to keep the schemas of the object stores in sync.
  2. Ensure that any custom applications can easily accommodate the regular generation of a new object store.
  3. Update any ingestion tools to use the new object store.
  4. If you are creating links between items in the object store, ensure that you can also use the same method to link objects in different object stores.

Remediation Strategies

If issues arise because of the size of an object store, consider taking some or all the following actions:

• Review the options provided by the database vendor for backups. Some offer flash copies, snapshots, point in time backups, and other options that could reduce the time it takes to perform a backup and a restore when needed. As a strategy, consider also using incremental backups nightly, and a full backup weekly.
• Review the disk and network used for the backups to see if speeds can be improved. Consider, for instance, using SSD disks.
• Work with the object store users to identify content that can be removed. And, develop an on-going strategy for identifying and deleting unneeded content.
• Prune any audit logs (content and process). Prior to pruning, export the audit logs if the information needs to be retained for regulatory or other purposes.
• Divide up the object store into a series of smaller object stores. FileNet Deployment Manager (FDM) can be used to export/import appropriate portions of the object store schema. FDM can also be used to export/import content, but also consider using IBM Expert Services or an IBM partner to move large amounts of content.
• Roll to a new object store. Use FDM to set up the new object store using schema exports from the existing object store. Then, update any
  1. Ingest applications to point to the new object store.
  2. Applications to use the new object store.
  3. Update any searches to point to the new object store, and if appropriate, build cross-repository searches to make it easier for end users to work with content after the new object store is available.

Other Considerations

Database Compression

Database compression can be used to minimize the size of an object store database. For DB2 databases, review the following information on compressing indexes and database tables: https://www.ibm.com/docs/en/filenet-p8-platform/5.5.11?topic=components-compressing-db2-database-tables-indexes

For information on compressing other database types, refer to the vendor documentation.

Partitioning

In general, we do not recommend using database partitioning as it can have a negative impact on performance.