IBM Support

When Using HMC GUI you see message "Unable to connect to the Database Error occurred "

How To


Summary

When using HMC graphic user interface (GUI) to manage virtual networks or virtual storage on a managed system you might encounter a problem where you see a message, "Unable to connect to the Database Error occurred." The error is describing a message the HMC is getting back from one or more Virtual I/O Server (VIOS) lpars when its trying to access While you cannot proceed past this type of error in the HMC GUI you will have to take action on the VIOS lpars to fix the problem as the issue deals with VIOS not being able to parse data out of its Configuration Management Database (CMDB). Most of the time the issue can be resolved by properly configuring name resolution on VIOS and in cases where that has already been done you might have to take additional measures to deal with CMDB corruption.

Objective

The issues that need to be dealt with on VIOS include the following.
-- Configuring name resolution.
-- Stopping and starting VIO demons that are used to communicate with HMC.

Environment

IBM Power systems with VIOS v2.2.5 and higher managed by HMC running v8 or v9.

Steps

1. The first step to resolving the "Unable to connect to database" error that the HMC is getting back from VIOS is to access VIOS as padmin then use oem_setup_env to become root and verify some basic configuration for name resolution.

a. Check both the /etc/hosts file as well as /etc/resolv.conf to determine how hostnames are configured and if DNS is setup.   
(1) Make sure /etc/resolv.conf file contains the proper name server and domain search entries if the file exists.
(2) Make sure /etc/hosts entries for loopback and that the VIOS hostname has following format.
IP FQDN alias
EXAMPLE: vios2 has IP 9.19.176.117 and domain is dfw.ibm.com
127.0.0.1               loopback localhost      # loopback (lo0) name/address
9.19.176.117    vios2.dfw.ibm.com   vios2 

(3) If using IPv4 addresses (common) then edit /etc/netsvc.conf and add following string at end of file.
hosts=local,bind4
b. Test resolution of names and IPs using nslookup if a name server is used in /etc/resolv.conf. 
(1) Test should be for both VIOS’s IP and hostname. 
(2) The return from name server should match what is configured on the /etc/hosts file.
(3) Fully qualified domain name (FQDN) is critical if using DNS so make sure  /etc/hosts file has FQDN since that will also be required on DNS servers.
(4) Use the “host” command on VIOS to check local resolver to see what its returning for hostname and IP if there is no /etc/resolv.conf file being used to designate a name server.
(5) Syntax for using nslookup and host command:
nslookup 1.2.3.4
nslookup myhostname.mydomain
host 1.2.3.4
host myhostname.mydomain
EXAMPLE: vios2 has IP 9.19.176.117 and domain is dfw.ibm.com
-- Run a reverse name lookup (query for IP in DNS database)
# nslookup 9.19.176.117
Server:         9.0.130.50
Address:        9.0.130.50#53
Non-authoritative answer:
117.176.19.9.in-addr.arpa       name = vios2.dfw.ibm.com.
-- Run a hostname lookup using FQDN
# nslookup vios2.dfw.ibm.com
Server:         9.0.130.50
Address:        9.0.130.50#53
Non-authoritative answer:
Name:   vios2.dfw.ibm.com
Address: 9.19.176.117
NOTE: both the hostname lookup and reverse name lookup returned the same information and this is what you need to see if DNS is configured (/etc/resolv.conf file exists and is configured properly)
-- Also check local name resolution using the host command
# host 9.19.176.117
vios2.dfw.ibm.com is 9.19.176.117
# host vios2.dfw.ibm.com
vios2.dfw.ibm.com is 9.19.176.117
NOTE: both queries returned same information.
c. Correct /etc/hosts and if needed get entries in DNS fixed till all checks return the same information when you lookup VIOS's IP and hostname.  
2. Stop and start RSCT and VIO daemons
a. Using a script to stop and start VIOS as well as refresh CMdb is recommended
b. Use following FTP site to obtain the script:
ftp://ftp.software.ibm.com/systems/virtualization/vio/ztools/CMDB/
(1) Login with your IBM ID or as guest and download the script.  The current script as time of this printing is named "cleanup_cmdb_with_logging.sh"
(2) The script is updated periodically so if you see a different script name than the one above you can use it instead. 
(3) Since it is a script you can inspect it first.   If there is a readme document then you can also review that.
c. Download the script and copy it to /home/padmin on VIOS
d. Become root on VIOS (oem_setup_env)
e. Set permissions on file so it will execute (chmod +x inetd_cmdb_cleanup_refresh.sh)
f. Execute the script (./cleanup_cmdb_with_logging.sh).

g. Wait a few minutes then try using HMC GUI again to manage virtual storage or virtual network resources.
h.  If everything is working now you are done.
3.  If you are still getting errors then collect debug data and open a case with IBM Support (if not already open) to get additional help.
a. Collect screen shots of the task being used and error received that show the navigation path to the task and time references.  Any information that can be captured as text output and provided to the case as text will expedite resolution so don't just rely on screen shots.
b. Collect PE debug data from HMC
HMC Enhanced View: Collecting PEDBG from the HMC
https://www-01.ibm.com/support/docview.wss?uid=nas8N1022548
c.  Collect debug data from the VIOS lpars as padmin using  the "snap" command.
Collect Snap From a VIO Server
https://www-01.ibm.com/support/docview.wss?uid=isg3T1013064
d. Send the data to IBM for further analysis.

Document Location

Worldwide

[{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"HW1A1","label":"IBM Power Systems"},"Component":"","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF041","label":"HMC"},{"code":"PF012","label":"IBM i"},{"code":"PF016","label":"Linux"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB08","label":"Cognitive Systems"}}]

Document Information

Modified date:
12 October 2021

UID

ibm11111155

Page Feedback