Database and file storage areas

Database and file storage areas are the easiest types of storage areas to set up.

Database storage areas are useful when the size of your object store is not large in terms of number of documents and the sizes of those documents. Do not store documents over 100 MB in a database storage area. File storage areas are useful for large numbers of files with high ingestion rates.

Database storage areas

Content can be stored in the same database as used for the object store. That is, Content Platform Engine stores both the objects and the content for those objects in the same database.

A database storage area converts document content into Binary Large Objects (BLObs) for storage in the database specified as the object store database.

When you create an object store, the wizard prompts you to specify the default content storage: either a database storage area or a file storage area. Choosing the file storage area does not eliminate the database storage area; this selection simply determines which content store serves as your default store when you add documents to the object store. A database storage area (contained by a table named "Content") is automatically created when you create an object store and specify the default storage area for document content is a database.

Note that each object store has only one database storage area.

When you choose either file storage or database storage for a document class, consider all the factors. Because of Java™ Database Connectivity (JDBC) driver limitations, consider storing large content in file storage.

Determine the maximum size of the content elements your users store. This affects setting up database storage areas or file storage areas. When you create an object store, a database storage area is provided by default, allowing you to store content as database BLObs. You can also create one or more file storage areas to store content on local or remote file systems. If your users store large individual documents or other content elements, and you create the associated object stores on Microsoft SQL Server or IBM® Db2® databases, use only file storage areas. Otherwise, users can encounter memory-related errors when retrieving or indexing the large content.

Note: Controlled tests with limited concurrency exhibited errors when run with files that were 300 MB or larger. Factors affecting this file-size limitation include driver and application server memory demands, other activity such as concurrent retrieval or indexing of large content, and JVM memory allocations.
Note: This is not an issue with Oracle where, during commit actions, content is streamed, rather than stored, in memory.

See Document content and Object stores for more information.

File storage areas

A file storage area is an area that contains document content in a directory tree on a local or shared network drive. The disk drive can be a Windows NTFS volume, a UNIX file system, or an IBM General Parallel File System (GPFS).

Restriction: You cannot create a file storage area on an encrypted NTFS volume.

A many-to-many relationship exists between Content Platform Engine servers and file storage areas: many servers can manage one file storage area, and a single server can manage multiple file storage areas.

Content element model

A document can have zero or more content elements. An element is either a content reference element or a content transfer element. A content transfer element has an associated content file in the file storage area. This file is uniquely identified by the combination of the document object identifier (GUID) and an integer sequence number. Within the context of a particular document, a content file is uniquely identified by the sequence number.

When you check in a document, the document content set becomes immutable. Although no content element for a checked-in document can be added or deleted, the entire document can still be deleted. When you delete a document, the content element files for the document are deleted.

File storage area structure

The following diagram shows the directory structure of a file storage area and the relationship between multiple file storage areas:

The hierarchical relationship between the Base, Root, and Content directories.

  • Base directory:

    The base directory is the user-named parent directory for one or more file storage areas.

  • Root directory:

    The root directory is the user-named top-level directory for a specific file storage area.

  • Content directory tree:

    The directories at the lowest level of the content directory tree store the committed content element files. You can configure the number of content directories by making the directory tree structure either large or small. A small structure can be ideal for a development environment.

Prerequisites for creating a file storage area

  • Plan file storage areas for logical and convenient grouping of documents.

    Create separate file storage areas to ensure efficient document management. For example, you can create a file storage area to group documents with the same deletion or backup requirements.

  • Create and name the base and root directories and assign proper security privileges to the directories.

    For information about the privileges to assign, see Storage area security.

  • Make the file storage device accessible to every Content Platform Engine server that will use the device.

    Making devices accessible depends on the operating systems and locations of your file servers. For more information, see Preparing storage areas for object stores.

    Optionally, create distributed file system (DFS) links to shared directories to provide a single, logical representation of a file storage location. A link allows the file location to be referenced without the user knowing the location. (In particular, a DFS link might represent a shared directory on a remote server. During file storage area creation, you can select such a DFS link as a local directory.)
    Restriction: You cannot use DFS to replicate a file storage area.
    Important: If you use an NFS mount point as the staging area, you must configure the mount point to prevent loss of data when your disk volume is at full or near full capacity. For information, see Configuring the NFS mount point for use as a staging area.
  • Install remote file systems with a backup power supply.

    Install any remotely connected file system with an uninterrupted power supply (UPS) backup system. Failure to gracefully shut down a server on which a remote file storage area resides can result in data loss or corruption.

For File Storage Area, you can define the following storage area deletion methods:

  • Clear:

    Directly deletes the content file. However, the content of the deleted file might be recoverable by a file system utility such as Norton Utilities. The content is recoverable when other files have not overwritten the disk space that is occupied by the deleted file.

  • Destroy:

    Overwrites document content with zeros before deletion. Provides the deletion level that is recommended for clearing and sanitizing media that is less than top-secret.

  • Purge:

    Overwrites file content three times before deleting the document. Provides the highest level of deletion that is recommended for purging top-secret documents.

Note:

Destructive and Purge deletion carry signification performance cost and require complex file system locking operations, you should only use these 2 deletion methods when it is truly necessary.