Configuring persistent storage for logging

An Azure Storage Account is a core resource in Microsoft Azure that provides a secure, scalable, and highly available solution for storing data in the cloud. It serves as a container for all storage services and defines how data is accessed and managed. Configuring the Azure Storage account and the corresponding Kubernetes secret is a mandatory step to enable persistent logging for watsonx.data system applications. This ensures that logs can be retained and shared with the support team for troubleshooting when needed.

Before you begin

Ensure that the following prerequisites are met:

Azure CLI installed and configured
```
az --version
```
kubectl installed and configured to access your cluster
```
kubectl version --client
```

Azure authentication completed

az login
az account set --subscription <subscription-id>

Resource group exists in the same Azure account as your cluster
```
az group show --name <resource-group-name>
```

About this task

The observability stack uses Fluent Bit, a lightweight and high-performance log forwarder to collect logs from Kubernetes applications and deliver them to configured destinations. In this deployment, Fluent Bit forwards logs to an Azure Storage Account, which acts as the persistent storage location for logs that are generated by watsonx.data system applications running in your cluster.

Storing logs in Azure ensures that they remain available for diagnostics and can be shared with the support team for troubleshooting when needed. The following steps outline how to create and configure the Azure Storage Account and set up the corresponding Kubernetes secret so Fluent Bit can authenticate and forward logs successfully.

Procedure

To configure the Azure Storage Account and the corresponding Kubernetes secret, complete the following steps:

Create Azure Storage Account

Set environment variables for your configuration:

export RESOURCE_GROUP="<your-resource-group>"
export STORAGE_ACCOUNT_NAME="<your-storage-account-name>"  # Choose a unique name (3-24 chars, lowercase letters and numbers only) or use an existing account
export LOCATION="<azure-region>"  # e.g., canadacentral, eastus

Note: If you're using an existing storage account, skip to Step 2. If you are creating a new account, ensure that the name is globally unique and follows Azure naming conventions.

Create the storage account (skip this step if you are using an existing account):

az storage account create \
  --name "$STORAGE_ACCOUNT_NAME" \
  --resource-group "$RESOURCE_GROUP" \
  --location "$LOCATION" \
  --sku Standard_LRS \
  --kind StorageV2 \
  --allow-blob-public-access false \
  --min-tls-version TLS1_2

Verify that the storage account was created

az storage account show \
  --name "$STORAGE_ACCOUNT_NAME" \
  --resource-group "$RESOURCE_GROUP"

Create Storage Container

Create the container named wxd-system-logs:
```
az storage container create \
  --name wxd-system-logs \
  --account-name "$STORAGE_ACCOUNT_NAME" \
  --auth-mode login
```
Important: The container must be named exactly wxd-system-logs as this name is what FluentBit is pre-configured to use.

Verify that the container was created:

az storage container show \
  --name wxd-system-logs \
  --account-name "$STORAGE_ACCOUNT_NAME" \
  --auth-mode login

Create a retention policy for the storage account:

Create a local file named wxd-log-storage-retention-policy.json with the following content:

{
  "rules": [
    {
      "enabled": true,
      "name": "Watsonx.Data Logs Retention Policy",
      "type": "Lifecycle",
      "definition": {
        "actions": {
          "baseBlob": {
            "delete": {
              "daysAfterModificationGreaterThan": 30
            }
          }
        },
        "filters": {
          "blobTypes": [
            "blockBlob",
            "appendBlob"
          ],
          "prefixMatch": [
            "wxd-system-logs/"
          ]
        }
      }
    }
  ]
}

Note: This policy is set to delete logs after 30 days by default. You can adjust the daysAfterModificationGreaterThan value to your preferred retention period, with a minimum of 7 days.

Apply the retention policy to the storage account:

az storage account management-policy create \
  --account-name "$STORAGE_ACCOUNT_NAME" \
  --policy ./wxd-log-storage-retention-policy.json \
  --resource-group "$RESOURCE_GROUP"

Retrieve Storage Account Key

Get the storage account key:

STORAGE_KEY=$(az storage account keys list \
  --resource-group "$RESOURCE_GROUP" \
  --account-name "$STORAGE_ACCOUNT_NAME" \
  --query "[0].value" \
  --output tsv)

Verify that the key was retrieved
```
echo $STORAGE_KEY
```

Create Kubernetes Secret

Verify the argus-control namespace exists:

kubectl get namespace argus-control

If it doesn't exist, create it:

kubectl create namespace argus-control

Create the Kubernetes secret with the storage account key and name:
```
kubectl create secret generic az-blob-secret \
  --from-literal=storage_account_key="$STORAGE_KEY" \
  --from-literal=storage_account_name="$STORAGE_ACCOUNT_NAME" \
  -n argus-control
```
Important: The secret must be named az-blob-secret and must contain both storage_account_key and storage_account_name keys to match the FluentBit configuration.

Verify that the secret was created:

kubectl get secret az-blob-secret -n argus-control

(Optional) Verify that the secret contains the correct keys:

# Verify storage account key
kubectl get secret az-blob-secret -n argus-control -o jsonpath='{.data.storage_account_key}' | base64 -d
echo ""

# Verify storage account name
kubectl get secret az-blob-secret -n argus-control -o jsonpath='{.data.storage_account_name}' | base64 -d
echo ""

Verify FluentBit Container
1. Check for FluentBit pods in the argus-control namespace:
```
kubectl get pods -n argus-control
```
2. Verify the FluentBit pods are running (look for pods with "fluent-bit" in the name):
```
kubectl get pods -n argus-control -w
```
3. Check the FluentBit logs to ensure it's connecting to Azure Storage successfully (replace <pod-name> with the actual FluentBit pod name):
```
kubectl logs -n argus-control <pod-name> --tail=50
```
4. If there are any issues with the FluentBit container, try restarting the pod (replace <pod-name> with the actual FluentBit pod name):
```
kubectl delete pod -n argus-control <pod-name>
```

Verify the setup.

Check Storage Account

az storage account show \
  --name "$STORAGE_ACCOUNT_NAME" \
  --resource-group "$RESOURCE_GROUP" \
  --query "{name:name,location:location,provisioningState:provisioningState}"

Check Storage Container

az storage container show \
  --name wxd-system-logs \
  --account-name "$STORAGE_ACCOUNT_NAME" \
  --auth-mode key

Check Kubernetes Secret

kubectl describe secret az-blob-secret -n argus-control

Check FluentBit Status

kubectl get pods -n argus-control
kubectl logs -n argus-control <fluent-bit-pod-name> --tail=100

Verify Storage Account Activity.

After FluentBit runs for a few minutes, check if blobs are created in the container:

az storage blob list \
  --container-name wxd-system-logs \
  --account-name "$STORAGE_ACCOUNT_NAME" \
  --account-key "$STORAGE_KEY"

Note: If you don't have the storage key in your environment, retrieve it first:

STORAGE_KEY=$(az storage account keys list \
  --resource-group "$RESOURCE_GROUP" \
  --account-name "$STORAGE_ACCOUNT_NAME" \
  --query "[0].value" \
  --output tsv)