Estimating storage requirements

Estimate the persistent storage requirements for your analytics subsystem.

This information applies only to data that is stored in your API Connect analytics subsystem, and not to data that is offloaded to a third-party system. If you plan to disable local storage, you can skip this topic. The formula and guidelines are based on how the API Connect analytics subsystem stores data, and cannot be directly applied to any other storage system.

Warning: Increasing the analytics storage space after installation is not a simple operation, and it can require reinstallation of the analytics subsystem. Decreasing storage space always requires reinstallation. If you do not have the information that is needed to make an estimate, then set your storage to at least 500Gi and monitor your usage closely. For an installation where you expect to store little or no analytics data, you can set the storage to 50Gi.

After installation, if you find you do not have enough space, then reduce the data retention period to keep less analytics data and stay within your persistent storage limit. See Running out of storage space for other steps that you can take if you run out of space.

To understand the concepts that are required for estimating your storage requirements, first review the following pages:

Use the following guidelines to provide an estimate of the amount of persistent storage you need for your API Connect analytics subsystem.

Factors that determine persistent storage requirement

Number of copies of the analytics data

The number of copies of the analytics data you store depends on your deployment profile:

Three replica profiles store 3 copies of your analytics data (one OpenSearch shard replica per analytics replica).
One replica profiles store 1 copy of your analytics data.

Number of days that API data is retained

The default number of days your API event data is retained depends on your deployment profile. For example, the n1xc4.m32 profile retains data for 30 days. To see the retention period for each analytics deployment profile, see retention and rollover.

API summary data is retained for 366 days by default on all deployment profiles.

If you customized your retention and rollover settings, then use the updated retention periods in your calculations.

API event logging policy and customization

The logging policy that you set for each API determines the size of the event records for that API. Typical sizes of API event records for each activity log policy are:

Activity logging: 1.5 Kb. This is the default log policy for successful API calls.
Header logging: 4 Kb.
Payload logging: 4 Kb plus size of expected request and response payloads. This is the default log policy for failed API calls.

You can also Customize your analytics data to add, redact, or remove fields. If you add fields to your API events, calculate the average size of the new fields and add that number to your event size. If you remove fields, subtract this size only if you are also removing headers or payloads.

Percentage of each type of data

Estimate the percentage of each type of log policy (activity, header, payload) for your published APIs. Use these percentages and your estimates of API event size to calculate you average API event size.

Remember: API logging policy can be set differently for success and failure responses of the same API. The default is activity logging for successful API calls, and payload logging for failed calls. Consider the percentages of successful and failed calls that you expect in your calculations.

API calls per day

Estimate the total number of API calls that you expect per day.

Backup configuration

If you enable backups and configure type=all, then double your calculated storage requirement.

If you enable backups and configure type=default (which only backs up summary data and shared queries), then increase your calculated storage requirement by 25%.

For more information about analytics backups, see Backup, restore, and disaster recovery.

Calculating your persistent storage requirements

Calculate the expected size of your API event data with the following formula:

Storage required for API events = (average bytes per API event) * (API events per day) * (API event retention period) * (copies of data)

Calculate the expected size of your API summary data with the following formula:

Storage required for API summary data = (expected size of all API events with header logging) * (API events per day) * (summary data retention period) * (copies of data) * 0.1

Note: Ignore the percentages of different logging types you calculated for your APIs and assume that all your APIs are set to header logging. API summary data uses approximately 10% of the space of header-logged API event data per day. Remember that header logging includes everything that is logged by activity logging plus the API call headers.

Your analytics data must not use more than 70% of your persistent storage space. Add some buffer space in case your analytics data is larger than you expect. It is recommended that the persistent storage space you provide for your analytics subsystem is double the space that you calculated your data requires.

Example calculation, based on the following information:

Copies of data = 3
API event data retention = 45 days
API summary data retention = 366 days
Activity log bytes per call = 1500 bytes
Header log bytes per call = 15000 bytes
Payload log bytes per call = 30500 bytes
% of Activity log = 70%
% of Header log = 20%
% of Payload log = 10%
Calls per day = 100000

Calculate the average API event size:

( (70*1500) + (20*15000) + (10*30500) ) / 100 = 7100 bytes

Calculate the space needed for your API event data:
```
7100 * 100000 * 45 * 3 = 89.3 Gi (95.6 Gb)
```
Calculate the space needed for your summary data:
```
15000 * 100000 * 366 * 3 * 0.1 = 153 Gi (165 Gb)
```
Add API event data and summary data totals:
```
153 Gi + 89.3 Gi = 242.3 Gi
```
Double the storage calculated in step 4 to get the storage size to allocate to your analytics subsystem:
```
242.3 Gi * 2 = 484.6 Gi
```
If you plan to enable backups, then add extra space as follows:
- If you plan to set backups type=all, then add the total that you calculated in step 4.
```
242.3 Gi + 484.6 Gi = 726.9 Gi
```
- If you plan to set backups type=default, then multiply the total that you calculated in step 4 by 0.25 and add this to your total storage requirement:
```
484.6 Gi + (242.3 Gi * 0.25) = 545.2 Gi
```