Configuring S3-compatible external storage

Edit online

Configure your external object storage or Amazon Simple Storage Service (S3) external object storage for log data instead of using the default provided object store. The configuration process varies depending on your cluster type.

About this task

External object storage allows you to use your existing storage infrastructure instead of the default object store. This provides flexibility to manage costs, leverage existing storage, and integrate with your organization storage policies. The configuration process differs between MSP clusters and tenant clusters (Account or Platform type).

Procedure

Determine your cluster type and follow the appropriate procedure:

For MSP clusters, follow the procedure in Configuring external storage for MSP clusters.
For tenant clusters (Account or Platform type), follow the procedure in Configuring external storage for tenant clusters.

Configuring external storage for MSP clusters

Edit online

Use this procedure to configure external S3-compatible storage for MSP clusters.

Before you begin

Before you configure external object storage, ensure you have:

Access to an S3-compatible object storage system
Storage credentials (access key and secret key)
Storage bucket or container created and accessible
Network connectivity between the logging system and object storage
Administrative access to the LokiStack configuration

Procedure

Review the supported object storage types and requirements.
The system supports S3-compatible object storage systems, including:
- Amazon S3
- IBM Cloud Object Storage
- MinIO
- Other S3-compatible storage systems
Gather the required connection information for your object storage:
1. Note the storage endpoint URL.
2. Obtain the access key ID and secret access key.
3. Identify the bucket or container name for log storage.
4. Determine the region (if applicable).

Create a Kubernetes secret to store the storage credentials.

Use the following namespace and secret name for each component. This keeps credentials secure and separate from configuration files.

Table 1. Namespace and secret name for storage credentials (MSP clusters)
Component	Namespace	Secret name
Loki	`loki`	`loki-s3`

Update the LokiStack configuration to use external storage.
Modify the configuration files according to the documentation, specifying:
- Storage type (S3)
- Endpoint URL
- Bucket name
- Region
- Secret reference for credentials
Configure storage retention and lifecycle policies.
Set up policies in your object storage system to manage log lifecycle, including archival and deletion based on age.
Apply the configuration changes to the LokiStack deployment.
Verify that logs are being written to the external storage.
Check the object storage bucket to confirm that log data is being stored correctly.
Test log retrieval and querying.
Perform test queries to ensure that logs can be retrieved from external storage and displayed in dashboards.

Results

External S3-compatible object storage is configured for your MSP cluster, and logs are stored in your specified storage system instead of the default object store.

What to do next

Monitor storage usage and costs in your object storage system. Set up billing alerts if available to track storage expenses.

Configuring external storage for tenant clusters

Edit online

Use this procedure to configure external S3-compatible storage for tenant clusters (Account or Platform type).

Before you begin

Before you configure external object storage for tenant clusters, ensure you have:

Access to an S3-compatible object storage system (such as IBM Fusion Storage Array)
Storage credentials (access key and secret key)
Separate storage buckets created for Loki and Tempo
Network connectivity between the cluster and object storage
Administrative access to the cluster

Important: Do not enable the observability flag while launching the cluster. The observability flag should be enabled only after configuring external storage.

Procedure

Launch the cluster without enabling the observability flag.
Wait for the cluster to be up and running, then log in to the cluster.
Create the observability-init namespace:
```
oc new-project observability-init
```
Create the Loki external storage secret:
```
oc create secret generic loki-external-s3-config \
  --from-literal=bucketnames=LOKI_BUCKET_NAME \
  --from-literal=endpoint=ENDPOINT \
  --from-literal=access_key_id=ACCESS_KEY \
  --from-literal=access_key_secret=SECRET \
  --from-literal=region=us-east-1 \
  -n observability-init
```
Replace the placeholder values with your actual storage configuration:
- LOKI_BUCKET_NAME: Name of the bucket for Loki logs
- ENDPOINT: S3-compatible storage endpoint URL
- ACCESS_KEY: Access key ID for authentication
- SECRET: Secret access key for authentication
Create the Tempo external storage secret:
```
oc create secret generic tempo-external-s3-config \
  --from-literal=bucket=TEMPO_BUCKET_NAME \
  --from-literal=endpoint=ENDPOINT \
  --from-literal=access_key_id=ACCESS_KEY \
  --from-literal=access_key_secret=SECRET \
  -n observability-init
```
Replace the placeholder values with your actual storage configuration:
- TEMPO_BUCKET_NAME: Name of the bucket for Tempo traces
- ENDPOINT: S3-compatible storage endpoint URL
- ACCESS_KEY: Access key ID for authentication
- SECRET: Secret access key for authentication
Enable the observability flag to use the external storage.
Follow your cluster management process to enable observability, which will now use the configured external storage.
(Optional) If using self-signed certificates with Fusion Storage Array, patch the certificate configuration:
1. For Loki, patch the lokistack-config configmap in the observability namespace:
```
oc patch configmap lokistack-config -n observability \
  -p '{"data":{"lokiExternalStorageCaName":"CONFIGMAP_NAME"}}'
```
  Replace CONFIGMAP_NAME with the name of the configmap that contains the certificate in the key service-ca.crt.
2. For Tempo, patch the tempo-config configmap in the tempo namespace:
```
oc patch configmap tempo-config -n tempo \
  -p '{"data":{"externalStorageCaName":"CONFIGMAP_NAME"}}'
```
  Replace CONFIGMAP_NAME with the name of the configmap that contains the certificate in the key service-ca.crt.
Verify that logs and traces are being written to the external storage.
Check the object storage buckets to confirm that data is being stored correctly.

Results

External S3-compatible object storage is configured for your tenant cluster, and observability data (logs and traces) is stored in your specified storage system.

What to do next

Troubleshooting TLS issues:

If you are using a trusted CA (not self-signed certificates) and encounter TLS issues, you can remove the TLS configuration from the custom resources:

Delete spec.storage.tls from the LokiStack CR in the observability operator
Delete spec.storage.tls from the TempoStack CR in the tempo operator

Monitor storage usage and costs in your object storage system. Set up billing alerts if available to track storage expenses.