Troubleshooting
Problem
Planning Analytics Workspace is working correctly until the Windows Server is rebooted.
Each time the Windows Server is restarted, some of the containers fail to start ("Exit" state instead of "Up" state).
Some temporary containers might also appear, like:
bf16f3c7faa2_couchdb
32784fd3b06d_share-app
78fd50cfc238_pa-gateway
Each time it happens, a manual operation is necessary to get all the containers up again, which is not acceptable on a production environment.
Cause
Timing and resource issues due to the workload on Windows Server during startup.
Or antivirus intensively scanning the docker processes and folders.
Or insufficient number of CPU cores.
Resolving The Problem
First, in order to allow users to work again, fix the immediate issue by restarting the containers. Run this block of commands in Powershell:
cd <your_PAW_folder>
./scripts/paw.ps1 down
# if there are temporary containers, then uncomment and run this additional command too:
# docker rm $(docker ps -a -q)
stop-service docker
start-service docker
./scripts/paw.ps1
After about 2 or 3 minutes, verify all containers are "Up" again (except for bss-init and couchdb-init)
Now here are some recommendations in order to prevent this issue from happening again:
1) Give Windows Server some more time to initialize itself before docker is started:
-From Windows start menu, run "services.msc"
-Open the properties of "docker" service or "docker engine" service
-Change the "Startup Type" property of the service from "Automatic" to "Automatic (Delayed start)".
2) If there is an antivirus, then exclude these folders (and subfolders) and applications from being scanned:
-Folders:
<Your_PAW_folder>
C:\ProgramData\Microsoft\Windows\HNS
C:\ProgramData\docker [or the path defined by the "data-root" parameter in C:\ProgramData\docker\config\daemon.json]
C:\Program Files\docker
-Applications:
C:\Program Files\Docker\docker.exe
C:\Program Files\Docker\dockerd.exe
C:\Program Files\Docker\docker-compose.exe
-Folders:
<Your_PAW_folder>
C:\ProgramData\Microsoft\Windows\HNS
C:\ProgramData\docker [or the path defined by the "data-root" parameter in C:\ProgramData\docker\config\daemon.json]
C:\Program Files\docker
-Applications:
C:\Program Files\Docker\docker.exe
C:\Program Files\Docker\dockerd.exe
C:\Program Files\Docker\docker-compose.exe
3) Verify Data Execution Prevention (DEP) is set to "Turn on DEP for essential Windows programs and services only" option.
The option can be modified from Control Panel, System and Security, System, Advanced system settings, Advanced tab, click the Settings button of the Performance section, click Data Execution Prevention tab, and change the option.
4) Verify the specifications of the server are sufficient. For this specific issue, since it might be caused by the workload during startup, the number of CPU cores is the key. For example, if the server is running Planning Analytics Workspace, Cognos Analytics, Planning Analytics Spreadsheet services, and some TM1 servers, then 4 CPUs might not be sufficient.
5) If the problem is still happening after all the previous points were applied, then create a Powershell script that contains these commands:
cd <The_path_to_your_PAW>
./scripts/paw.ps1
Name it RestartPAW.ps1.
Once done, create a scheduled task to run this script a certain time after reboot:
- Go to the Windows Administration Tools
- Open the "Task Scheduler"
- Click "Create Task"
- Let's name it "Restart PAW"
- Select "Run whether user is logged on or not"
- Select "Configure for: Windows Server 2016"
- In the "Triggers" tab, click "New"
- Set "Begin the task: At startup"
- Select "Delay task for: 15 minutes" (for example)
- In "Actions" tab, click "New"
- Select "Start a program"
- Type "Powershell.exe" in the "Program/script" field and add the path to RestartPAW.ps1 in "Add arguments (optional):" field.
- Go to the Windows Administration Tools
- Open the "Task Scheduler"
- Click "Create Task"
- Let's name it "Restart PAW"
- Select "Run whether user is logged on or not"
- Select "Configure for: Windows Server 2016"
- In the "Triggers" tab, click "New"
- Set "Begin the task: At startup"
- Select "Delay task for: 15 minutes" (for example)
- In "Actions" tab, click "New"
- Select "Start a program"
- Type "Powershell.exe" in the "Program/script" field and add the path to RestartPAW.ps1 in "Add arguments (optional):" field.
At next reboot, wait 15 minutes and verify all containers are correctly started up.
Document Location
Worldwide
[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSD29G","label":"IBM Planning Analytics"},"Component":"Planning Analytics Workspace","Platform":[{"code":"PF033","label":"Windows"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB10","label":"Data and AI"}}]
Was this topic helpful?
Document Information
Modified date:
19 January 2023
UID
ibm11072356