Troubleshooting
Problem
I am having trouble upgrading my Application Stack. It either seems to fail outright or seem to be successful but is not upon closer examination. How can I examine progress, verify completion, or resolve any issues?
Symptom
Appstack Upgrade appears to hang, have failed, or partially successful
Cause
Appstack issues are usually caused by some specific step being done out of order or incorrectly, or services not running correctly before upgrade, or services not brought up in the correct order during the upgrade.
Environment
StoredIQ Application Stack Upgrade
Diagnosing The Problem
Appstack Upgrade Procedure
BEFORE EVERY UPGRADE, TAKE A SNAPSHOT OF YOUR STOREDIQ IMAGES
1) Shutdown the StoredIQ VMs in order Appstack > All Dataservers > Gateway
2) Take a snapshot of every StoredIQ Image
3) Power on StoredIQ VMs Gateway > All Dataservers > Appstack
4) Verify all services are running
ON THE GATEWAY
____________________________________________________________
VERIFYING UPGRADE WENT WELL
ON APPSTACK:
____________________________________________________________
IF UPGRADE SEEMED TO HAVE FAILED AND REMAIN HUNG, try the below
For additional troubleshooting please contact StoredIQ Support -- please note in ticket which steps you tried and the output you received
BEFORE EVERY UPGRADE, TAKE A SNAPSHOT OF YOUR STOREDIQ IMAGES
1) Shutdown the StoredIQ VMs in order Appstack > All Dataservers > Gateway
2) Take a snapshot of every StoredIQ Image
3) Power on StoredIQ VMs Gateway > All Dataservers > Appstack
4) Verify all services are running
ON THE GATEWAY
- 1) Make sure all services are running -- all services must be running prior to upgrade
service deepfiler status
2) Disables IP Tables Services with the command -- this step is turning off the firewall -- similar in opening port 80 but gets around any complications in opening a single port only
service iptables stop
3) Verify you can browse to the Gateway Products page through your internet browser -- this step is necessary because the Appstack needs to access the Gateway to download the upgrade package
In a browser, go to http://gatewayIP/products. You can see different folders with the application stack versions.
- 4) Make sure all services are running as all services must be running prior to upgrade
/siq/bin/monit summary (for older versions)
service appstack status (for newer versions)
5) Use the SSH tool to log in to the application stack and run these commands.- First setup the repository with the command below -- REPLACE GATEWAYIP WITH THE IP ADDRESS IF YOUR GATEWAY
bootstrap product.set appstack repository "pkgsource=\"http://gatewayIP/products\""
example:
bootstrap product.set appstack repository "pkgsource=\"http://9.0.23.24/products\""
6) Then kick off the upgrade by entering the command below- bootstrap product.upgrade appstack HEAD
7) Tail the appstack.log file while it is upgrading by entering- tail -f /var/bootstrap/appstack.log
When you are notified that the production mode is activated the upgrade is complete.
IMPORTANT: This should be AFTER all the packages are applied and services are restarted automatically in the script, should see 70+ packages installed.
ctrl+C to escape
8) After application stack completes, we need to close ports again on the Gateway and restart services - First setup the repository with the command below -- REPLACE GATEWAYIP WITH THE IP ADDRESS IF YOUR GATEWAY
- 9) service iptables start
10) service deepfiler restart
____________________________________________________________
VERIFYING UPGRADE WENT WELL
ON APPSTACK:
- 1) If upgrade is complete, the following file should NOT exist
/var/bootstrap/bootstrap.lock
STEPS- cd /var/bootstrap/
ls bootstrap*
If bootstrap.lock file does not exist, that is good. If it does exist, it means the upgrade is either still running, or it is hung.
Verify the status of progress by running command:
tail -f /var/bootstrap/appstack.log
Ctrl+C to escape
2) Verify services are all running- /siq/bin/monit summary (for older versions 7.6.0.6 and older)
service appstack status (for newer versions 7.6.0.7 and later)
All services must be up and running after the upgrade is complete.
If services are not up and running, check the /var/bootstrap/appstack.log and see if services failed to come up. If services failed to come up, note at which step it failed and contact Support.
3) cat /var/bootstrap/appstack.history- Should show history of the upgrade to the current version with the appropriate date.
Example (below shows fresh install -- NONE should be replaced by 'current version' for other upgrades):
2016-09-27 16:02:05 upgraded NONE to 7.6.0.9+4
If above does NOT show the upgrade line, the upgrade is either still in the process of upgrading, or failed.
4) tail -f /var/bootstrap/appstack.log- Should show the line 'production mode is activated' at the end AND should show all services being started successfully.
Ctrl+C to escape
5) cat /var/bootstrap/server-mode- Should show as production (this should be the same as typing bootstrap server.mode I believe)
If server-mode is not in production, services either were not restarted yet or failed to restart correctly.
6) cat /var/bootstrap/products/appstack/status- Should NOT show as 'upgrading', but as 'valid'
If it still shows as upgrading, the upgrade failed or is still in the process of upgrading.
7) /var/bootstrap/products/appstack/version- Should show as the NEW current version.
If it still shows as previous, then upgrade failed.
- cd /var/bootstrap/
____________________________________________________________
IF UPGRADE SEEMED TO HAVE FAILED AND REMAIN HUNG, try the below
- 1) Stop services via bootstrap server.mode idle
2) Confirm servics are stopped
3) kill -9 the PID listed in 'cat /var/bootstrap/bootstrap.lock'- ex: kill -9 123 where 123 is the PID
4) Delete bootstrap.lock- rm /var/bootstrap/bootstrap.lock
5) Run pidof each service to make sure no rogue processes are running
example:- pidof uwsgi
pidof tomcat
pidof postgresql
pidof platoon
pidof nginx
Make sure each command returns 0 lines
6) Change /var/bootstrap/products/appstack/status from 'upgrading' to
'valid'- 1. vim /var/bootstrap/products/appstack/status
2. Hit 'INSERT' to go into editing mode and change 'upgrading' to 'valid'
3. Hit 'ESC' to get out of editing mode
4. Hit ':' to get into command mode
5. Write ':wq' and hit 'Enter' to write and quit (save) the file
7) Bring up services via bootstrap server.mode production OR service appstack start on newer versions
8) Make sure /var/bootstrap/bootstrap.lock is gone- cd /var/bootstrap/
ls bootstrap*
9) Rerun product set appstack repository and product upgrade - REPEAT STEPS 8-12 IN UPGRADE GUIDE ON TOP
For additional troubleshooting please contact StoredIQ Support -- please note in ticket which steps you tried and the output you received
[{"Product":{"code":"SSSHEC","label":"StoredIQ"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"--","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.6","Edition":"All Editions","Line of Business":{"code":"LOB45","label":"Automation"}}]
Was this topic helpful?
Document Information
Modified date:
29 March 2019
UID
swg21999886