Enabling an external content repository for source control

You can use the Administration page in the web client to enable an external content repository for source control. When enabled, source control content larger than the configured threshold is stored in an external content repository instead of in the application's database.

Before you begin

You must be logged in to the Administration page of a Change and Configuration Management (CCM) application that is registered with the Jazz® Team Server. You must be a member of the JazzAdmins repository permissions group to have administrative access to enable and modify the external content store settings.

About this task

Jazz Source Control can store content externally, either on a server that supports the WebDAV protocol, or on the application server's local filesystem. If you are configuring an external WebDAV server, you must supply credentials that the CCM application can use to access that server.

Procedure

  1. On the Administration page for the Change and Configuration Management application (not for the Jazz Team Server), click the Application tab.
    Important:
    • The way your files are stored in the external file store depend upon the nature of that file storage utility. The files could be stored in a database or could be encrypted depending on how the store is set up, such as whether it is a filesystem, a WebDAV server, or other server.
    • After you move your content from the database to an external store, a database administrator might be able to reclaim space in the database, depending on the database itself.
    • The external content store and the database should be backed up at the same time so that references from the database to the external content store are consistent.
    • When an object is stored externally and the external server becomes unavailable, you receive error messages when you try to access the large SCM objects. The errors are similar to the scenario where the server loses its connection to the database. When the connection is reestablished and there is no disintegration, operation should be back to normal.
  2. In the left pane, under Configuration, click Advanced Properties.
  3. On the Advanced Properties page, in the SCM area, set the following properties:
    1. Versioned Content External Repository - Write Enabled - true or false. When set to true, new content is written to the external content store. When set to false, new content is written to the Change and Configuration Management (CCM) database, but content that was previously stored to an external content store is still retrieved from that content store.
    2. Versioned Content External Content - Artifact Minimum Size - The minimum size of content (in bytes) that should be stored in the external content store (if configured).
    3. Versioned Content External Repository - Type - One of NONE, FILE, or WEBDAV.
    4. Versioned Content External Repository - URI - A URI pointing to the root of the external content store.
    5. Versioned Content External Repository - User ID - The user ID for the connection to the external content store (if required).
    6. Versioned Content External Repository - Password - The password for the connection to the external content store (if required).

Changing the configured external content repository

You can change the configuration values for the external content repository. Any changes affect where new content is stored; it does not affect how existing content is stored or retrieved.

Procedure

  1. Set the Versioned Content External Repository – Write Enabled property to false if you want all new content to be stored in the CCM server database regardless of its size. Content that was previously stored in an external content repository continues to be retrieved from that external content repository.
  2. Change the Versioned Content External Repository – URI property if you want new large content to be saved to the new server. Previously saved content remains on and is retrieved from the old server.
    Note: There is no limit to the number of external content repository servers that can host previously stored content. However, only one external content repository can be configured to store new content at any given time.

Changing WebDAV access credentials

IBM® Engineering Workflow Management stores the access credentials for each Versioned Content External Repository – URI value that you enter. If you change the Versioned Content External Repository – URI value and its associated credentials, the old URI and credentials are stored so that Engineering Workflow Management can continue to retrieve content on that server while using the new credentials to access the currently configured external content repository.

Procedure

  1. In the left pane, under Configuration, click Advanced Properties.
  2. Click Versioned Content External Repository – User ID to change the user name.
  3. Click Versioned Content External Repository – Password to change the password.
  4. To change the credentials for previously configured WebDAV servers, you must edit the external content repository credentials file directly:
    1. Open the JSON credentials file called externalContentRepositories.conf, located in the CCM directory, typically in ./<server>/conf/ccm, where <server> is the CCM server installation directory. The following externalContentRepositories.conf file contains two external content repositories with their credential data. Notice that the passwords are encrypted:
      [
          {
              "PASSWORD": "[qoSGR4gvSUY=]",
              "URI": "https:\/\/example.com\/artifactory\/content\/",
              "USER": "repoUser1"
          },
          {
              "PASSWORD": "[DZCEyaFThdWaWOsOL2l2Cg==]",
              "URI": "https:\/\/example.com\/artifactory\/content2\/",
              "USER": "repoUser2"
          }
      ]
    2. Use a text editor to change the USER and PASSWORD fields for the appropriate external content repositories. Edit the characters between the quotation marks for the USER and PASSWORD values. New passwords are entered in plain text. Save the file and Engineering Workflow Management automatically detects the changes, updates the password fields with their encrypted values, and begins using the new credentials. Here is the updated externalContentRepositories.conf file:
      [
          {
              "PASSWORD": "newPassword1",
              "URI": "https:\/\/example.com\/artifactory\/content\/",
              "USER": "repoUser1"
          },
          {
              "PASSWORD": "newPassword2",
              "URI": "https:\/\/example.com\/artifactory\/content2\/",
              "USER": "newRepoUser2"
          }
      ]

      The externalContentRepositories.conf file includes entries for file-based external content repositories; however, credentials are currently only used for WebDAV repositories, and are not supported for file-system-based content repositories.

Moving content

You can use the repository tools moveContent command to move previously stored content in three ways.

About this task

You can use the repository tools moveContent command to move previously stored content in these ways:

  • From the CCM server database to an external content repository
  • From an external content repository to the CCM server database
  • From an old external content repository to the currently configured one

The moveContent command has arguments to specify source and destination content repositories, as well as the minimum or maximum size of content to move. For details about how to run the command with various arguments, see the repository tools command-line reference documentation.

Deleting content

About this task

When content stored in an external content repository is deleted from Engineering Workflow Management, the server also attempts to delete the content from the external content repository. The server attempts the deletion, but it is not guaranteed. For more information, see: Deleting Content From Source Control

If a problem occurs with deleting content from the external repository, the problem is logged on the server but does not cause the delete operation to fail.

Reporting using JMX MBeans

In Engineering Workflow Management, you can monitor content stored in SCM by using Java Management Extensions (JMX) managed beans (MBeans).

About this task

An administrator can enable reporting of MBeans for SCM-versioned content by using the Advanced Properties page in the web client for the CCM server.

Procedure

  1. Open the Server Administration page of your CCM server, and click Manage Server.
  2. In the left navigation, click Advanced Properties. Scroll down to the Serviceability Component section, locate the com.ibm.team.scm.service.internal.content.metrics.VersionedContentMetricsTask section and enable the beans that you want to monitor.
    Table 1. Versioned Content Metrics Task
    Property name Default value Description
    Delay between invocations 604800 The time in seconds between invocations of the Versioned Content Metrics background task, which collects data to publish as MBeans.
    Note: After you enable MBeans and restart the server, the beans are collected after the "Delay between invocations" expires.
    Enable Versioned Content Repository Metrics MBean false When set to true, publishes information for all content repositories in which content has been stored.
    Enable Versioned Content Component Metrics MBean false When set to true, for each component, publishes an MBean that contains information about the number of files, and the size and storage location of content claimed by files in each component.
    Enable Top File Sizes MBean false When set to true, publishes information about the largest files stored in SCM.
    File Sizes MBean – Number of Files to report 20 The number of files to report on for the Top File Sizes MBean.
    File Sizes MBean – size threshold 1000000 The minimum size in bytes for files to be considered by the Top File Sizes MBean.

Content Repository Metrics MBean

When the Content Repository Metrics MBean is enabled, an MBean is published that contains a list of the content repositories in which content has been stored. The CCM database is represented as a repository with DATABASE as its URI.

About this task

The MBean name has the form com.ibm.team.scm.content:name="<app-context>",type=repositoryMetrics. The bean contains a list of repositories. For each content repository, the following information is published:

Table 2. Content Repository Metrics MBean
Property name Description
externalURI The external content repository URI for the content repository, or DATABASE for the default database repository.
totalNoOfContent The number of unique content objects stored in the content repository.
compressedDBSize The total number of bytes stored in the database for this content repository. For the default DATABASE repository, this is the compressed size of the content. External content repositories store metadata in the database.
totalSize The total uncompressed size of the content stored in the content repository.

Component Metrics MBean

When the Component Metrics MBean is enabled, an MBean is published for each SCM component.

About this task

The MBean names have the form com.ibm.team.scm.content:name="<app-context>",type=componentContent,componentNameAndId="<componentName>_<componentUUID>". For each content repository, the following information is published:

Table 3. Component Metrics MBean
Property name Description
componentName The name of the component.
componentId The UUID of the component.
ownerName The name of the component owner. This is the name of the containing project area if the component is owned by a project or team area, or the name of the owning contributor.
ownerId The UUID of the component owner. This is the UUID of the containing project area if the component is owned by a project or team area, or the UUID of the owning contributor.
totalNoOfContent The number of unique content objects claimed by files in this component.
dbSize The total number of bytes stored in the database for content claimed by files in this component.
totalSize The total uncompressed size of content claimed by files in this component.
externalStores A list of external content repositories in which content is stored for this component. This value will be empty if all content is stored in the CCM database. For each external content repository, see the table below:
Table 4. External Store Content
Property name Description
externalURI The URI of the external content repository.
totalSize The total number of bytes stored in the external content repository.
totalNoOfContent The number of unique content objects claimed by files in this component stored in this external content repository.

Engineering Workflow Management uses a hash-based storage mechanism to eliminate storing multiple copies of identical content. If identical content is checked in multiple times, only a single copy of that content is physically stored and each project, component, or file associated with that content points to the single persisted copy. The transparency to users applies to both content stored in the CCM server database and content stored in external content repositories.

The combined total of all the component metrics beans might be larger than the actual number of bytes physically stored in the database or external content repositories.

Top File Sizes MBean

The number of files reported by the Top File Sizes MBean is controlled by the CCM server advanced property File Sizes MBean – Number of Files to report under VersionedContentMetricsTask. When the Top File Sizes MBean is enabled, a bean is published for each of the largest files whose size is greater than that specified by the File Sizes MBean – size threshold (in bytes) property.

About this task

The MBean names have the form com.ibm.team.scm.content:name="<app-context>",type=topContent,rank="<N>". For each of the largest files, the following information is published:

Table 5. Top File Sizes MBean
Property name Description
contentSize The compressed size of this content if stored in the database; otherwise, the full content size when stored externally.
externalURI The URI for the external content repository where this content is stored, or DATABASE if stored in the CCM server database.
contentClaimer For each external content repository, see the following table.
componentName The name of the containing component.
itemId The item UUID for the claiming file.
stateId The state UUID for the claiming file.
approximatePath The approximate or most popular path for the file. The path for a file is determined by the directory structure of each repository workspace or stream in which it appears and, in general, cannot by computed without specifying a workspace or stream. The file might not actually exist at the approximatePath in any specific workspace, particularly if each workspace or stream that contains the file has it at a different path.

Engineering Workflow Management clustering considerations

You might use Engineering Workflow Management in a clustered environment.

About this task

If you use Engineering Workflow Management in a clustered environment, consider the following items:
  • Your external content repository must be accessible to all nodes in the cluster. This is true for both file-based and WebDAV-based content repositories.
  • If you have content in multiple WebDAV repositories, and need to edit the credential files manually, you must update the credentials file on each node in the cluster.

Backup

When you back up a Engineering Workflow Management configuration that uses external content repositories, it is important to back up the CCM database first, and then back up the external repositories. When data is restored, the server reflects the state immediately after the database backup regardless of when the external content repositories are backed up.

About this task

If you roll back the server, you only have to roll back the CCM database. You do not have to roll back the external content repositories. In this type of scenario, the external content repositories retain any orphaned content that was added before the database roll back, and the data integrity is not affected. The orphaned content is automatically reclaimed if the content is checked in to the server again.

The backup procedure for an external content repository depends on the type of repository. For a WebDAV content store, consult that product documentation for specific instructions. For a file system content store, any industry-standard file system backup procedure is appropriate.

During the time between the completion of the database backup and the completion of the external repository backups, do not use the delete content feature or the repository tools moveContent command. The delete content feature is a rare action performed by an administrator that removes content bytes from Jazz Team Server: it does not refer to the common action of checking in file deletions. Most users never use the delete content feature.

If any content deletions are performed between the two backups, after data is restored, the content delete must be reapplied to return the database to a consistent state for those items. Content deletes, if originally performed, must also be reapplied if the CCM database is rolled back and the external content store is not.

Best practices

Note the following best practices for configuring external content repositories:

About this task

WebDav
  • To use the external content repository feature with certain strictly compliant WebDAV servers such as Microsoft IIS, ensure that you install Rational® Team Concert® version 6.0.6 iFix003 or later.
  • Because various WebDAV servers might have specific configuration requirements, ensure that you read the server documentation before configuring the server.
  • To confirm that the WebDAV server is working properly, run the curl PUT and curl GET commands.
  • Before you configure the WebDAV server, know the following information:
    • Maximum file upload size limit imposed by the WebDAV server
    • File type and content type configuration of the server
    • Whether the basic authentication is installed and enabled on the server
  • Configure your WebDAV server to use HTTPS. Engineering Workflow Management uses basic authentication to send WebDAV login credentials. Configuring your WebDAV server to use HTTPS ensures that these credentials are encrypted and not passed in plain text.
  • Do not use very small values for the Versioned Content External Content – Artifact Minimum Size property. The added overhead of making HTTP requests to a WebDAV server negatively affects performance if very large numbers of small files need to be accessed. The external content repository feature was designed to move large files out of the CCM server database and into external content repositories: it is not intended to move all SCM files out of the CCM server database.

If you use a Microsoft IIS server, you might encounter the following issues:

Table 6. Microsoft IIS server issues
Issue Solution
Basic authentication for IIS is not installed or enabled by default. Install basic authentication for IIS by using the Windows Control Panel and enable the authentication in IIS Manager. For details, see the Microsoft IIS help on basic authentication.
IIS fails to return content that does not have file extensions.

Add a MIME type mapping for files with no file extension.

You can configure the MIME type at the virtual directory level for the virtual directory that contains the external content.

Configure the MIME type in IIS Manager. For details, see Microsoft IIS help for configuring MIME types in IIS 7.

The default maximum file upload size for IIS is only 30 MB. You can increase the maximum file upload size at the website level in IIS Manager. For details, see Microsoft IIS help for configuring request filtering in IIS.
IIS has a maximum upload size of 4 GB.

If you must handle larger files, you must select a different server such as Artifactory or use the File content store type.

If you configure a maximum file upload size on your WebDAV server, you must set the Engineering Workflow Management Advanced Server Property "Versioned Content Maximum Size" to the same value. If you set a value and check in a file that is larger than the set value, the Engineering Workflow Management client displays an error message instead of attempting to upload the file to the WebDAV server first.

Artifactory
  • Turn off anonymous access in Artifactory to prevent non-authenticated users from retrieving content from the Artifactory server.
  • If you use the content delete feature in Engineering Workflow Management to delete sensitive SCM content, you must empty your Artifactory trash folder, or configure Artifactory not to use a trash folder.
File
  • UNC paths are not supported. If you want to use network storage for a FILE external content repository, it must be mounted on the local file system on Linux or mapped as a network drive in Windows.
  • Use canonical paths when you configure the content repository URI. Paths that contain spaces should be appropriately escaped (for example, file:////C:/Content%20Repository).