Troubleshooting Web Console issues

The following table describes possible reasons for occurrence and troubleshooting tips for issues that most frequently occur with the Control Center Director Web console.

The quickest way to troubleshoot broken interface elements, misbehaving blocks, and interactive functionality issues is to check the browser console to view errors. To access developer tools right-click anywhere on the web page and select Inspect Element from the context menu. Click Network tab to view further details.

Alternatively, browser developer tools can also be accessed for:

  • Mozilla Firefox via Menu>Web Developer >Network
  • Google Chrome via Menu>More tools>Developer Tools>Network
  • Microsoft Edge via F12> Developer Tools > Network tab

After opening the browser console, you can start looking for errors. Errors will be displayed differently in each browser, but generally will be color coded, labeled, or marked with an identifying icon. If you find an error, make sure to copy the error name, location, and line number.

Table 1. Troubleshooting Web Console issues
Issue Possible Reason Solution/Workaround
Licenses
Some Connect:Direct servers show up under Licenses view as Unconfigured
  • One possible reason could be that you have added a Connect:Server version that IBM Control Center Director does not support.
  • Another possible reason could be that this Connect:Direct server is not configured to support IBM Control Center Director.
Unable to edit/delete Server Groups from within the Licenses>Server Groups tab Licensing view restricts users from editing server groups under the Server Groups tab in the left navigation pane. To edit or delete a Server group, click Servers>All Groups tab and hover over Server group.
Server count displayed against an environment type tab such as Production (left navigation pane) does not match entries displayed in server listing by license type displayed in the pane on the right. This could be possible because IBM Control Center Director records and displays server entry in the listing each time you restart a server.  
Multiple entries for the same server in a environment type  
No Licenses information displayed under the Licenses view IBM Control Center Director collects licensing metrics from a Connect:Direct server which could be located on a different server. Time difference between the two should be considered based on UTC time and not local time. Licensing view displays data by an offset from UTC that is, UTC+2:00
Servers
No Licenses information displayed under licenses tab in Servers > All Servers > View Server details One possible reason could be that the licenses data is made available by an offset from UTC that is, UTC+2:00. Also, the licensing tab under Server details page is restricted to display licensing data for only the last 24 hours. To view licensing metrics for a particular server go to Licenses > Production (any Environment type) and select a date from the date picker.
Timezone setting is already applied to the Connect:Direct server provided UTC offset. This could be possibly because this Server is auto-discovered by ICC Director.  

No value is populated in the Platform, Agent Port, Architecture, and CD Server Installation Location fields under the General tab in Servers > All Servers > View Server Details

It could be possible that the server details you are attempting to view are for a Connect:Direct server:
  • that is not configured properly to relay Connect:Direct Server properties to IBM Control Center Director
  • that was added via Web Console
  • version that is not supported by IBM Control Center Director
Icon status next to the server name is displayed in orange This implies that either the Connect:Direct Server or Agent is unavailable.  
Server listing view is missing the Delete option in the overflow menu against a server entry This is because you cannot delete a server while the Agent is still running. Ensure that you decommission the Server and then attempt to delete it from the Server listing view.
Server listing view is missing the Start option in the overflow menu against a server entry This is because you can start a server only when the Agent is still running and the Connect:Direct server is unavailable.  
Bulk deploy option is missing when I check 'Select all' check box under Servers > All Servers > View Server Details This possibly occurred because the Connect:Direct servers:
  • Were added via Web Console
  • Server composition is not homogenous that is, they are running on different platform or architecture
  • Version is not supported by IBM Control Center Director
  • Ensure that Connect:Direct servers in your deployment are configured to be dynamically discovered by Control Center Director
  • Ensure that server/server groups selected are a homogenous set in terms of platform and architecture
  • Install/Upgrade to a CD version that supports IBM Control Center Director. For more information see, see Compatibility Matrix
Delete option is missing when I check Select All check box under Servers > All Servers > View Server Details This is because you cannot delete a server while the Agent is still running. Ensure that you decommission the Agent and then attempt to delete it from the Server listing view.
The Server group with 0 Servers or Server group with mixed types of servers/server groups is not appearing in Control Center Director Web Console. NA The Server groups are classified based on NODE_TYPE_ID. Empty Server Groups or Server Groups with mixed type of servers/server groups have NODE TYPE_ID=-1. In Control Center Director, only Server Groups having either Sub CD Server Groups or/and CD servers have NODE_TYPE_ID=1 and will be visible. All other categories of Server Groups will be visible in Control Center Monitor. Read viewing server groups in Control Center Monitor for more details.
Deployments
After upgrade from a previous version to Control Center Director version 6.2 or later, the tabs/functionality like Deployments, Licenses, New Install are not appearing.

This is because after an grade the role permissions are set to No Access for these functionality except for the admin role.

The admin needs to re-visit/assign the required permissions to roles. For more information, refer Users and Roles Management.
No field data is populated under Server Group/Servers in Deployment > Deploy > Job
  • One possible reason could be that upgrade is not supported for servers that are added via IBM Control Center Director Web Console
  • Another possible reason could be that scheduling a Job to upgrade a server and/or server group is constrained by the Operating System and package version.
    Expected input for the following fields:
    • Operating System: Should match the OS for servers being upgraded.
    • Package Name: Should be a version higher than the current Connect:Direct server version.
Go to Deployment > Deploy Job and:
  • Select a Package name with a higher/equal package version
  • Enter an Operating Server type that matches the Server and/or Server Group being upgraded
Deploying a job returns the following error:

Package selected is not compatible with server(s) included in the selected server group(s)
Scheduling a Job to upgrade a server and/or server group is constrained by the package version the system encountered a Package mismatch. Go to Deployment > Deploy Job > Package Name.

Select a higher/equal package version and click Deploy.
A scheduled upgrade job did not move into In-Progress state.
  • One possible reason could be that the system running IBM Control Center Director is out of sync with the server time.
  • Other possible reason could be that because IBM Control Center Director picks scheduling jobs every minute, the Web Console has not displayed refreshed data.
  • Ensure that the time on system running ICC Director is in sync with your local time
  • ICC Director Engine is up
A scheduled Job fails
  • One possible reason could be that the package was possibly deleted from the designated directory while the job was still In Progress state
  • Another possible reason could be that Agent is down
  • Another possible reason could be that the Server for which the upgrade job was scheduled was deleted after the job was scheduled.
Connect:Direct packages do not show up under the Deployments tab.
  • View events related to packages in the event log files to isolate the issue.
  • One possible reason could be Web Console is not displaying refreshed data. Auto sense feature enables ICC director to detect and process the packages after they are added/removed into the designated directory. It can take up to 15 minutes for the packages to show up under the Deployment view.
  • Another possible reason could be that ICC Director was not configured correctly to define the directory where the Connect:Direct packages will be made available for auto-discovery.
  • Another possible reason could be that the package name is not in the prescribed standard IBM format as downloaded from Fix Central.
  • Another possible reason could be that you might have attempted to add a package that already exists in the designated directory possibly with a different name and/or same content. Packages once added to the designated folder cannot be replaced.
  • Administrators, note that auto sense scheduler is configured by default to run every 15 minutes. To change this value, modify package.folderWatchDelayInSecs parameter in application.properties
  • See Configuring Control Center Director.
  • Go to Fix Central
  • Remove the old package first and replace it with the new package.

Installing a new Connect:Direct server returns the following error:

CDAI050E Required parameter not provided: cdai_keystorePassword 
or
<Error: SPCLI> SPCG770E rc=8 Import error, label: 172.20.186.103 
- The Certificate validity interval has expired.>
This is possibly because the bundle was not downloaded properly. Download the bundle from the link again.

Refer New Install to understand the detailed process.
Untar of New Install task bundle or package fails and returns the error:.
Error: tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors

Some problem occurred in downloading bundle/package. Look for the cause of failure, open <name.tar> in any editor to get the cause of failure and download the bundle/package again:

There may be following reasons:
  • Bundle download failure:
    • Control Center Director node is down
    • The download link is archived or deleted.
  • Package download failure:
    • Load balancer and package are not configured properly.
    • Package is not available or deleted
    • Certificates used are invalid.
  • Bundle download solution:
    • Control Center Director node should be up and running.
    • Check the download link is in active state and not archived or deleted.
  • Package download solution:
    • If user has Load Balancer, then make sure that all the cluster nodes are using shared path for package. Refer Using RESTful APIs for Web Configurations.
    • Make sure that server.use-forwarded-for-with-proxy is set to true, when using load balancer.
    • Check if the package is available.
    • Make sure all the certificates used are valid.
All the packages of deployment do not show up in packages in new-install drop down This could be because all the packages listed in deployments may not be valid for new-install. New-Install supports the package version 6.1.0.1 or later. For more details, refer to New Install.
Unable to access KDB certificate file of new-installed Connect:Direct Windows after installation A possible reason could be unavailability of password. Open cd_srvr.ini file present in the extracted bundle. Read Parameter: CD_KEYSTORE_PWD
Unable to access KDB certificate file of new-installed Connect:Direct UNIX after installation A possible reason could be unavailability of password. Extract the bundle (tar) again, open options file, read parameter cdai_keystorePassword.
Unable to complete Connect:Direct installation using New Install feature.
Error: Attempt to connect to remote node <hostname/IP> failed. FRWL=N
This implies that Connect:Direct guidelines have not been followed. Read C:D iPv6 link.
While copying the shared link, you get localhost or 127.0.0.1 in URL or getting following error while trying to download new-install bundle:
curl: (7) Failed connect to 127.0.0.1:58083; Connection refused
This could be because CC-UI is not using hostname or machine IP and the server is down. Ensure that CC-UI is using hostname or machine IP and the server is up and running.
Getting the following error on adding the admin user ID while creating a New Install task:
<Error: Invalid admin userid.>

Admin user ID provided does not exist Connect:Direct UNIX.

Provide correct user name in Admin User ID, while creating the installer for New Install is a valid user on the machine where Connect:Direct server will be installed.
Unable to access requestor with admin user credentials. This implies that the Admin user name mentioned while creating Connect:Direct Windows installer is a not a valid user. Ensure that the Admin user name mentioned while creating Connect:Direct Windows installer is a valid user. Create a valid user and build the installer again for installation.
Certificate is not yet valid.
Error: SPCLI> SPCG773E rc=8 Node: .Local - error: The Certificate Label 'LabelName
validity interval not yet reached.
A possible reason could be that the Control Center Director nodes are not synchronized. Synchronize the time across all the communicating nodes. For more details, refer Using RESTful APIs for Web Configurations.
While using new install feature to install Connect:Direct, getting following errors:
Error: XSEC010I Return Code: 8 Feedback: 0
XAPI005I Return Code: 8 Feedback: 0
ndm_auth failure.
Failed to obtain connection to Connect:Direct server, exiting. 
Connect:Direct CLI Terminated...
20200820 09:02:36 8 CDAI022E 
Connect:Direct installation verification failed. 
Task is submit sample process.
A possible reason could be incorrect Connect:Direct configurations.
  • Port 1367 must be available.
  • Check the /etc/hosts file and make sure if there is no entry like --> ::1 adcenter-01 localhost. If there is an entry, either remove it or comment the line and try installing again.

For more details, refer Customization Worksheet.

Users and Roles

After upgrading from previous version Control Center Director to Control Center version 6.2 or later, all users are able to view/edit all Control Center Monitor functionality and can see a restricted user with DVG Manage permission.

This must be because, on upgrading from Control Center Director to Control Center version 6.2 or later, Base and Control Center Monitor permissions are set to Manage, by default. The admin needs to reassign/update the required permissions of the roles. For more information, refer Users and Roles Management.
User is unable to log into Web Console with correct ICC Director account credentials User tried to access the ICC Director Web Console from an IP addresses other than the one that the user is designated to use and was configured by the Administrator. Contact your administrator to find out your designated IP address as entered in Users and Roles>Create User>TCP IP address field.
ICC Director account is locked after consecutive failed login attempts IP is locked out to provide protection against brute force login. Attempt login again in 15 min (default). To configure the time duration that login attempts are blocked for modify authentication.lockduration in application.properties.
User is unable to log into Web Console using ICC Director account credentials (user ID). One possible reason could be that the user is configured to authenticate using SEAS credentials (Users and Roles)>Use External Authentication (SEAS) Attempt logging into the Web Console using SEAS credentials.
Account activation/password reset link-related
User did not receive an e-mail with password reset/Account activation link.
  • One possible reason could be that the user is trying to login using SEAS credentials. E-mails with account activation/password reset link are only sent if you login with a Control Center Director Account ID.
  • Other possible reason could be that as a security consideration ICC Director does not display an error message when an incorrect account ID is entered by the user. The user assumes that the ID was validated, and an e-mail was sent.
  • Another possible reason could be that your SMTP server configuration is incorrect.
  • Attempt password reset procedure with SEAS ID.
  • Attempt password reset procedure with a correct ICC Director ID or contact your administrator.
  • See Configuring Control Center Director.
User is unable to login in Control Center Web Console. One possible reach could be clustering of data in cache memory. Clear the cache memory and try again.
Web Console won't display refreshed Connect:Direct server upgrade and maintenance data when its servers are set to be auto-discovered by ICC Director
  • One possible reason could be due to incorrect certificate-based configuration
  • Make sure the OSA URL configured on Connect:Direct is same as ICC Director instance is using to track the Connect:Direct servers.
  • It is recommended to not configure same Connect:Direct Server to work with multiple instances of ICC Director. This causes the Connect:Direct Server to be registered with multiple instances of ICC Director instances.
  • Verify Certificate-based authentication configuration
  • To download error logs for a failed upgrade job, go to: Deployments>All Jobs>View Job details> Click entry next to Failed field to display Failed pop-up>Click download button.

    Extract the log file to view contents. The log file could either be:

    1. A single file that implies failure even before the upgrade started with details.

    2. Multiple log files which implies that upgrade started but encountered an error.

    Look for following primary log file _UpgradeByInstallAgent.log that includes all OSAs sent during the upgrade.

    The last OSA records the upgrade status giving failure details with detailed information on where the error occurred and possible reasons that caused the error to occur.

    For further details on log files specific to Agent Upgrade issues, see _UpgradeAgent_cdaiLog.log.

    For Connect:Direct server upgrade issues, see _Upgrade_cdaiLog.log.

Unable to activate password for newly created user or unable to login with newly created user. This could occur when SMTP server is not configured, or SMTP server is not accessible. Since the password is not activated, you cannot login with any newly created system-user.
  • Login via default super user i.e. admin

  • Login by SEAS users i.e. by configuring external authentication server

Forgot/lost Admin user password  
  1. Change property DEFAULT_USER_PASS_ENCRYPTED to false in InstallationInfo.properties file present under directory path //INSTALLATION-DIRECTORY//conf.
  2. Access the database and delete the admin user ID from the system. Using given command:
    Delete from CC_USER where NAME =’admin’;
  3. Now, simply run the config.bat and set the admin passphrase when “Config step: Default User 'admin' Configuration ...” prompt appears.
Configuration
Invalid Field Value. Allowed values SP800-131A_TRANSITION,SP800-131A_STRICT, SUITE_B-128, SUITE_B-192, DefaultToLN You might be trying to update secure+ node configuration of a legacy node. Secure+ node configuration for legacy node is not supported. Select relevant values from Security Mode drop down.
Expected search results are not obtained. You might be using Oracle database, where the search is case sensitive. If you are using Oracle database, you may set ORACLE_ENABLE_CASE_INSENSITIVE_SEARCH (from engine.properties) to true or false (default value), to perform case sensitive and insensitive search respectively. Fore more details, refer Column Management and Advanced Search.

For all other databases, the search is case insensitive.

Venafi
Error in downloading bundle from Control Center using Venafi as cert generator with below error in web:
  • Caused by: feign.FeignException$InternalServerError: status 500 reading

When 500 error code is returned from Venafi it means there is some issue in certificate generation. Check all the configurations in Venafi Settings screen. e.g. Country attribute should have 2-character value like US rather than USA. If “USA” is passed as value for Country attribute, Venafi certificate generation will fail and throw this error. Go to Venafi portal to the template used. Expand the tree and look for your certificate request. It would be showing error in the Current status field. E.g. This certificate cannot be processed while it is in an error state. Fix any errors, and then click Retry.

Delete this certificate. Now go to Venafi settings screen on Control Center and correct the Country value to US. Re-trigger the flow for generating Venafi certificate and it should work.

Also ensure correct API and CA files uploaded and check Truststore listing page. Check Test Connection working fine. Make sure Restarted CC after Save configuration.

Auto-renewal or manual renewal is failing and the web logs contain below error:

2024-10-22 09:00:00.089 INFO CDRQA-073 — [ool-5-thread-26] s.ServerCertificateAutoRenewJobScheduler : Start certificateAutoRenewalJob1! at:2024-10-22T03:30:00.088873615 2024-10-22 09:00:10.550 ERROR CDRQA-073 — [ool-5-thread-26] c.a.c.a.s.VenafiCAImpl :

Error when connecting Venafi for certificate signing. Exception : {} feign.RetryableException: Connection reset executing GET https://api.venafi.cloud/v1/useraccounts at feign.FeignException.errorExecuting(FeignException.java:213)

When using Venafi cloud trial users, then sometimes Venafi cloud might throw this exception. We are suspecting this error comes due to using trial user (we can confirm with Venafi team)
Note: Attached logs are SaaS trial account.
There is no fix for this. After some time, this starts working.

When we are trying test connection, we get below error:

Invalid Venafi parameters 2024-07-04 03:51:23.397 ERROR bb477624-5dc3-428b-8df7-d8b291b39c67 CDWLT-045 — [qtp55741515-454] .c.c.a.s.CertificateAuthorityServiceImpl : Venafi authentication failed.

Exception : {} com.venafi.vcert.sdk.VCertException: feign.FeignException$Unauthorized: status 401 reading Cloud#authorize

This could be due to incorrect values provided. The values can be:
  • Credentials
  • Certificates
Incorrect credentials can cause this issue. If credentials are correct then certificate used will be as below:
  • If certificates are provided as part of request then, it will use those for test connection.
  • If certificates are not provided, then it will try to fetch from truststore and used those for test connection
Ensure Venafi credentials correct. If certificate/s are provided as part of request, then ensure they are correct. If certificate/s are not provided as part of request, then ensure correct certificate are present in truststore.

Venafi certificate signing is failing, and the web logs contain the following error:

2024-07-04 22:08:34.789 ERROR b718049a-b32b-45cd-a8c0-0a885e17e727 CDWLT-045 — [tp55741515-4595] c.i.c.c.a.s.VenafiCAImpl : Error when connecting Venafi for certificate signing. Exception :

{} com.venafi.vcert.sdk.VCertException: status 404 reading Cloud#certificateDetails(String,String):

Unable to find certificate for key b79e1dc0-3a8c-11ef-8537-e59a38d8ac13 at com.venafi.vcert.sdk.VCertException.fromFeignException(VCertException.java:54)

at com.venafi.vcert.sdk.VCertClient.retrieveCertificate(VCertClient.java:219)

When using Venafi cloud trial users, then sometimes Venafi cloud might throw this exception. We are suspecting this error comes due to using trial user (we can confirm with Venafi team) There is no fix for this. After some time, this starts working.
When we are trying to install CD server, we get below error: \

curl: (35) error:0A000416:SSL routines::sslv3 alert certificate unknown Downloading package...done (2/3)

Extracting package...

ls: cannot access '*package*': No such file or directory
This could be due to the issuer certificate not being present in our CC truststore. The CA certificate, along with its chain, should be present in our CC truststore.

You can check this in the truststore listing page.

Make sure to restart Control Center after saving the configuration.
Renew CD certificate with manual or with auto renew scheduler job is failed. When trying renew CD certificate, sometimes it may failed . (Please refer audit log for more details related to renew )

When triggered CD certificate renew, secure keystore config job created for each CD server and it might failed.
Check exact issue/reason by referring audit log section.

When triggering CD certificate make sure CD server is UP (or ready) status.

Check CA is expired or not.