Troubleshooting

Important: You are not looking at the latest product documentation. Make sure you are reading the documentation that matches the version of the software that you are using. Switch to product version 2.1.3, 2.5, 2.5.1, 2.5.2 (latest)

General problems may occur when using the console to manage nodes, channels, or smart contracts. In many cases, you can recover from these problems by following a few easy steps.

This topic describes common issues that can occur when using the IBM Blockchain Platform console.

Issues during Deployment

Issues with the Console

Issues with your Nodes

My deployment fails when I try apply the security and access policies to my namespace

When I try to deploy the IBM Blockchain Platform v2.1.3 and apply the Security Context Constraint, clusterRole, or ClusterRoleBinding to my namespace, I encounter one of the following errors.

When I apply the file, I receive a user forbidden error:

securitycontextconstraints.security.openshift.io "e-block2" is forbidden: User cannot get securitycontextconstraints.security.openshift.io at the cluster scope: no RBAC policy matched

This problem occurs when you are not a cluster administrator. You must have a cluster administrator role and apply the security and access policies to your namespace.

When I try to apply resource file, I receive a parsing error:

error: error parsing ibp-scc.yaml: error converting YAML to JSON: yaml: line 10: mapping values are not allowed in this context

This problem occurs when there is a problem with the indents in your file. Refer to the documentation for the correct format for the security and access files.

My deployment fails when I try apply the custom resource definition of the console or operator

When I try to deploy the IBM Blockchain Platform v2.1.3 and apply the custom resource definition of the operator or the console, I encounter one of the following errors:

When I apply the custom resource file, I receive an image pull or image pull back-off error:

NAME                            READY     STATUS             RESTARTS   AGE
ibp-operator-769d94ffbc-w52n6   0/1       ImagePullBackOff   0          32s

This problem occurs when your deployment cannot pull the IBM Blockchain Platform images from the IBM Entitlement registry. This can happen because you provided an incorrect name of your secret to the imagePullSecrets: field, or if there was a problem with the url or the key that you provided to the entitlement key secret. This error can also occur if you supplied the wrong image tag to the file.

When I try to apply the custom resource file, I receive a parsing error:

error: error parsing console.yaml: error converting YAML to JSON: yaml: line 11: mapping values are not allowed in this context

This problem occurs when there is a problem with the indents in your file. Refer to the documentation for the correct format for the custom resource files of the console and operator.

Why are my console actions failing in my Chrome browser Version 77.0.3865.90 (Official Build) (64-bit)?

The console has been working successfully, but requests have started to fail. For example, after I create an ordering service and open it I see the error: Unable to get system channel. If you associated an identity without administrative privilege on the ordering service node, you will not be able to view or manage ordering service details.

This problem can be caused by a bug introduced by the Chrome browser Version 77.0.3865.90 (Official Build) (64-bit) that causes actions from the browser to fail.

To resolve this problem, open the console in a new browser tab in Chrome. Any identities that you saved in your console wallet will persist in the new browser tab. To avoid this problem you can upgrade your Chrome browser version. Ensure you have downloaded all of your wallet identities to your local machine before closing your browser. If this solution does not resolve your problem see Why is my channel creation failing or I am unable to add a new organization to my ordering service with the error "Unable to get system channel"?.

Why am I not able to log in to the console from my Chrome browser on Mac OS Catalina?

The console has been working successfully, but after I upgraded my Mac OS to Catalina, I can no longer log in to the console.

There are three ways to resolve this problem:

  1. Use a different supported browser with Catalina.
  2. Use your own TLS certificates when deploying on OpenShift Contain Platform or TLS certificates when deploying on Kubernetes or IBM Cloud Private.

  3. Run the following commands to generate a new key and certificate pair for the console that will fix the problem.

    1. Run the following command to get the pod that corresponds to the ibp console:

       kubectl get po

    2. Exec into the pod by running the command:

       kubectl get po <pod-name> -c optools bash

    3. Delete the console key and certificate by running the command:

       rm -f /certs/tls.key rm -f /certs/tls.crt

    4. Delete the console pod which causes it to restart by running the command:

       kubectl delete po <pod-name>

    When the pod restart completes, you should now be able to log in to your console URL from a Chrome Browser.

Why is my channel creation failing or I am unable to add a new organization to my ordering service with the error "Unable to get system channel"?

If you are not using your own TLS certificates to secure communications in your blockchain network, you need to accept the self-signed certificate that was generated for you. Otherwise, when you try to create a channel or add an organization to an ordering service the action fails. Channel creation fails with the error An error occurred when creating channel. submit config update failed: grpc code ???= response contains no code or message. Or, when you click on your ordering service, you see Unable to get system channel. If you associated an identity without administrative privilege on the ordering service node, you will not be able to view or manage ordering service details.

This problem occurs when the blockchain console is deployed without the advanced deployment option to use your own TLS certificates.

To resolve this problem, you need to accept the self-signed certificate in your browser.

  1. Copy your console URL, for example https://<PROJECT-NAME>-ibpconsole-console.abc.xyz.com:443, where the value of abc.xyz.com depends on your cloud provider.
  2. Replace -console with -proxy. The new URL looks similar to: https://<PROJECT-NAME>-ibpconsole-proxy.abc.xyz.com:443.
  3. Open a new tab in your browser and paste in the new URL.
  4. Accept the certificate.

You need to accept the certificate from this url to communicate with your nodes from your console and then log in as usual. When you switch to a new machine or a new browser, you need to repeat these steps.

When I hover over my node, the status is Status unavailable or Status unknown, what does this mean?

A CA, peer, or ordering node has a grey status box, meaning the status of the node is not available. Ideally, when you hover over any node, the node status should be Running.

This problem can occur if the node is newly created and the deployment process has not completed. If the node is a CA and the status has been grey for more than a few minutes, then it is likely that the deployment process has failed. Peers and ordering nodes take longer to deploy, but this condition can also occur when the health checker that runs against the peer or ordering nodes cannot contact the node. The request for status can fail with a timeout error because the node did not respond within a specific time period, the node could be down, or network connectivity is down.

If this is a new node, wait a few more minutes for the deployment to complete. You can try reloading the page in your browser to refresh the status. If the node is not new, examine the associated node logs for errors to determine the cause.

When I hover over my node, the status is Status undetectable, what does this mean?

The node status in the tile for the peer or ordering node is yellow, meaning the status of the node cannot be detected. Ideally, when you hover over any node, the node status should be Running.

This condition only occurs on peer and ordering nodes that were imported to the console and the health checker cannot run against the node. This status happens because an operations_url was not specified when the node was imported. An operations url is required for the node health checker to run. The node itself is likely Running, but because the operations url was not specified, its status cannot be determined.

You can resolve this problem by performing the following steps:

  1. Click the node tile to open it.
  2. Click the Settings icon.
  3. Click Associate identity, view and make note of the identity that is associated with this node. Click Cancel to close this panel.
  4. Click Export to generate a JSON file for the node.
  5. Edit the generated JSON file and enter the operations_url for the node. The value of the operations_url depends on how it was configured as well as various network settings. This value needs to be provided by the network administrator who deployed the node.
  6. Click Delete. This step removes the imported node from the console, but does not delete the actual node.
  7. From the Nodes tab, click Add Peer or Add ordering service followed by Import an existing peer or Import an existing Ordering service.
  8. Click Upload JSON and browse to the JSON file you just edited. Click Next.
  9. Associate the same identity you noted in step three.
  10. Click Add peer or Add ordering service. The health checker can now run against the node and report the status of the node.

Why did my smart contract installation, instantiation or upgrade fail?

It is possible you may experience an error when installing, instantiating or upgrading a smart contract. For example, when you try to install a smart contract on a peer, it fails with the error An error occurred when installing smart contract on peer.

You may receive this error if this version of the smart contract already exists on the peer, or if your peer is out of resources.

Why is the smart contract that I installed on the peer not listed in the UI?

A smart contract was installed on a peer but when you click on the Smart contracts tab, it is not listed.

This issue can occur when another user or application installs the smart contract onto the peer and you do not have the peer admin identity in your browser wallet.

In order to view the smart contracts installed on a peer, you need to be a peer admin. Ensure that the peer admin identity exists in your browser wallet. If it does not, you need to import it into your console wallet.

My nodes, channels, smart contracts, and identities have disappeared from the console. How can I get them back?

Nodes, identities, channels, or smart contracts (or some combination of these) that I successfully deployed into the console are no longer available.

One of the features of IBM Blockchain Platform is that you are now responsible for securing and managing your certificates. Therefore, they are only persisted in the browser local storage to allow you to manage the component. If you are using a private browser window and then switch to another browser or non-private browser window, the identities that you created will be gone from your wallet in the new browser session. Therefore, it is required that you export the identities from the wallet in your private browser session to your file system. You can then import them into your non-private browser session if they are needed. Otherwise, there is no way to recover them.

This problem can also occur when the console has lost contact with your Kubernetes cluster on IBM Cloud. This can happen if your console has not recently communicated with your cluster, or if you have made changes to your cluster that have overridden the settings used by the IBM Blockchain platform.

You can renew the communication between your console and the IBM Kubernetes service cluster on IBM Cloud by clicking on the service instance of your console in your Resource list and clicking the Refresh cluster button. When the link between your cluster and your console has been refreshed, click the Launch the IBM Blockchain Platform button.

Why am I getting the error Unable to authenticate with the enroll ID and secret you provided when I create a new organization MSP definition?

When you attempt to create a new organization MSP definition from the Organizations tab, you experience the error Unable to authenticate with the enroll ID and secret you provided.

This error occurs when the value that you specified for the enroll secret is not valid for the enroll ID that you selected in the Generate organization admin certificate section of the panel.

Verify that you have selected the correct organization admin enroll ID from enroll ID drop down list and enter the correct value for the enroll secret.

Why am I getting the error An error occurred when updating channel when I try to add an organization to my channel?

When you attempt to add another organization to a channel, the update fails with the message An error occurred when updating channel.

This error occurs when the selected Channel Updater MSP ID on the Update channel panel is not an admin of the channel.

On the Update channel panel, scroll down to the Channel Updater MSP ID and select the MSP ID that was specified when the channel was created or specify the MSP ID that is the admin of the channel.

When I log in to my console, why am I getting a 401 Unauthorized error?

When I try to log in to my console, I am unable to access the console from my browser. If I check the browser logs, I can find the error 401 unauthorized.

Your browser console session times out after 8 hours of inactivity. If a session becomes inactive, the console will prevent the inactive user from performing any actions.

If your session has become inactive, you can try simply refreshing your browser. If that does not work, close the browser including all tabs and windows. Open the URL again. You will be required to log in.

As a best practice, you should have already stored your certificates and identities on your file system. If you happen to be using an incognito window, all the certificates are deleted from the browser local storage when you close the browser. After you log in again you will need to re-import your identities and certificates.

Why is my first invoke of a smart contract returning the following error: no suitable peers available to initialize from?

When I try to invoke a smart contract from the Fabric SDK, the transaction fails and returns the following error:

error: [Network]: _initializeInternalChannel: no suitable peers available to initialize from Failed to submit transaction: Error: no suitable peers available to initialize from

This error occurs if you have not configured an anchor peer on your channel. Unless you have manually updated your connection profile, your application needs to use the Service Discovery feature to learn about the peers it needs to submit the transaction to.

Use the following steps to configure anchor peers on your channel. Also, you should verify that you are submitting the transactions against the correct channel and organization MSP id.

Why are my node operations failing after I create my peer or ordering service?

It is possible you may experience an error when managing an existing node. When that occurs, it is often useful to consult the node logs.

For example, when you try to operate the node, the action might fail.

After creating a new peer or ordering service, depending on your cluster storage configuration, it may take a few minutes for the nodes to be ready for operation.

Check your Kubernetes dashboard and ensure the peer or node status is Running. Then try your action again. If you are still experiencing problems after the node is up, check your node logs for errors.

Why does my peer or ordering node fail to start?

It is possible to experience this error under a variety of conditions.

The peer log includes:

[main] InitCmd -> ERRO 001 Cannot run peer because cannot init crypto, folder “/certs/msp” does not exist`

or the ordering node log contains:

Failed to initialize local MSP: admin 0 is invalid [The identity does not contain OU [CLIENT], MSP: [orderermsp],The identity does not contain OU [ADMIN], MSP: [orderermsp]]

What is the proper way to clean up a failed node deployment?

Sometimes a node can fail to deploy, for example, due to lack of resources in your Kubernetes cluster. After you understand the cause of the node deployment failure, you need to cleanup the failed node in your cluster.

Do not attempt to use Kubernetes commands to remove the node. Instead, it is extremely important that you use the IBM Blockchain Platform console or the APIs to remove the failed node to ensure that the associated metadata and storage are also cleaned up.

How can I view my smart contract container logs?

You may need to view your smart contract, or chaincode, container logs to debug a smart contract issue.

Follow these instructions to view your smart contract container logs.

Why is my CA, peer, or ordering node that is configured to use HSM not working?

This problem can happen when you try to deploy a CA, peer, or ordering node that is configured with HSM and the deployment fails. Or it can surface when a node that is configured for HSM stops working. The node, client, or SDK logs contain the following error:

{"code":1000,"message":"Private key not found [pkcs11: 0x30: CKR_DEVICE_ERROR]"}"

or

{"code":1000,"message":"Private key not found [pkcs11: 0xB3: CKR_SESSION_HANDLE_INVALID]"}"

This problem happens when the PKCS #11 proxy that is associated with the HSM is unreachable due to a network problem or if the proxy restarts after the node has connected to it.

To re-establish communications between the node and the proxy, restart the failing node by deleting the pod associated with the node. A new pod will be created and the connection with the PKCS #11 proxy is restored. Use the following steps to restart the failing node:

Replace:

My CA failed to upgrade, how can I fix it?

After upgrading from IBM Blockchain Platform v2.1.2 to v2.1.3, when I try to update my CA by clicking Update version, it fails with the error ECONNRESET. The CA logs include an error similar to the following text "error":"CA instance 'org1ca' encountered error: Code: 23 - failed to migrate ca: no matches for kind \"IBPCA\" in version \"ibp.com/v212\.

To resolve this problem, you need to restart the Operator pod in your cluster.

After the operator pod restarts, the CA node is successfully upgraded.

Why are my transactions returning an endorsement policy error: signature set did not satisfy policy?

When I invoke a smart contract to submit a transaction, the transaction returns the following endorsement policy failure:

returned error: VSCC error: endorsement policy failure, err: signature set did not satisfy policy

If you have recently joined a channel and installed the smart contract, this error occurs if you have not added your organization to the endorsement policy. Because your organization is not on the list of organizations who can endorse a transaction from the smart contract, the endorsement from your peers is rejected by the channel. If you encounter this problem, you can change the endorsement policy by upgrading the smart contract. For more information, see Specifying an endorsement policy and Upgrading a smart contract.

Why are the transactions I submit from VS Code failing with a No endorsement plan available error?

Transactions submitted from VS Code fail with an error similar to:

Error submitting transaction: No endorsement plan available for {"chaincodes":[{"name":"hello-world"}]}

This error occurs if you are using the Fabric Service Discovery feature but did not configure any anchor peers on your channel.

Follow step three of the private data topic in the Deploy a smart contract tutorial to configure your anchor peers.

Why are the transactions I submit from VS Code failing with an endorsement failure?

My smart contract endorsement proposals from my peer are failing from my client application with the endorsement error error: [Channel.js]: Channel:<channel_name> received discovery error:failed constructing descriptor for chaincodes:<name:"chaincode-name"> or [ERROR] Error submitting transaction: No endorsement plan available for {"chaincodes":[{"name":"MyAssetContract"}]}

Also in the endorsing peer logs I can see the error:

UTC [discovery] chaincodeQuery -> ERRO 23c Failed constructing descriptor for chaincode chaincodes:<name:"chaincode-name">,: cannot satisfy any principal combination

This error occurs when the peer's enroll id type does not match the smart contract endorsement policy that was configured when the smart contract was instantiated on the channel.

The only way to resolve this error is to delete the peer and create a new one with an enroll id that has the correct type peer. You can use the enroll id and secret from an existing user of type peer from the peer's CA or register a new user with type peer. Follow the instructions in the Build a network tutorial to create a new peer identity with the correct type and peer.