IC SunsetThe developerWorks Connections platform will be sunset on December 31, 2019. On January 1, 2020, this forum will no longer be available. More details available on our FAQ.
Topic
  • 7 replies
  • Latest Post - ‏2019-07-30T21:59:12Z by wolz
TMasteen
TMasteen
384 Posts

Pinned topic APAR IT18393

‏2019-06-30T12:19:03Z |

We want to upgrade SVC to v8.3.0.0.

SVC Update Utility returns with:

The tool has detected that one or more nodes is reporting duplicate USB ports due to APAR IT18393. 

I cannot find any information about this APAR. Has someone more info about this APAR and is it Ok to proceed with upgrade?

Thanks..

 

  • fincherjc
    fincherjc
    6 Posts

    Re: APAR IT18393

    ‏2019-07-04T01:00:50Z  

    Hello friend,

    This has to do with an error in the output of lsportusb and is usually cleared by removing and re-adding the offending node. I recommend opening a case with support to get more specific instructions to resolve pre-upgrade. I do not recommend pushing through without resolving this if you are using USB encryption.

  • TMasteen
    TMasteen
    384 Posts

    Re: APAR IT18393

    ‏2019-07-04T15:04:16Z  
    • fincherjc
    • ‏2019-07-04T01:00:50Z

    Hello friend,

    This has to do with an error in the output of lsportusb and is usually cleared by removing and re-adding the offending node. I recommend opening a case with support to get more specific instructions to resolve pre-upgrade. I do not recommend pushing through without resolving this if you are using USB encryption.

    Hello,

    A "normal" remove and addnode command of the node with duplicate USBports cleared the error.

    Still a bumpy workaround to fix this.... In this case three nodes of a 4-node cluster had the USBPort-error.

    Noticed that this check is added in upgradetest_29, Upgradetest_28_20 (code 8.2.1.4) does not stall on USBport errors.

  • wolz
    wolz
    3 Posts

    Re: APAR IT18393

    ‏2019-07-29T21:17:55Z  
    • TMasteen
    • ‏2019-07-04T15:04:16Z

    Hello,

    A "normal" remove and addnode command of the node with duplicate USBports cleared the error.

    Still a bumpy workaround to fix this.... In this case three nodes of a 4-node cluster had the USBPort-error.

    Noticed that this check is added in upgradetest_29, Upgradetest_28_20 (code 8.2.1.4) does not stall on USBport errors.

    What are the commands to do a "normal" remove and addnode?

    I'm at 7.8.1.8 and see only the following commands (on a new system with no pools/volumes/hosts defined yet):

    superuser>help | grep rmnode
    rmnodecanister - Deletes a node canister from the system.
    superuser>help | grep addnode
    superuser>

     

    (i.e. no rmnode, just rmnodecanister... and no corresponding addnode or addnodecanister)

  • TMasteen
    TMasteen
    384 Posts

    Re: APAR IT18393

    ‏2019-07-30T06:58:24Z  
    • wolz
    • ‏2019-07-29T21:17:55Z

    What are the commands to do a "normal" remove and addnode?

    I'm at 7.8.1.8 and see only the following commands (on a new system with no pools/volumes/hosts defined yet):

    superuser>help | grep rmnode
    rmnodecanister - Deletes a node canister from the system.
    superuser>help | grep addnode
    superuser>

     

    (i.e. no rmnode, just rmnodecanister... and no corresponding addnode or addnodecanister)

    Wolz, I was talking about a SVC - Cluster. I think you have a V7000, V50x0 or other storwize system.

    A SVC does have nodes (no nodecanisters). In your case you can use rmnodecanister\addnodecanister.

  • wolz
    wolz
    3 Posts

    Re: APAR IT18393

    ‏2019-07-30T12:49:27Z  
    • TMasteen
    • ‏2019-07-30T06:58:24Z

    Wolz, I was talking about a SVC - Cluster. I think you have a V7000, V50x0 or other storwize system.

    A SVC does have nodes (no nodecanisters). In your case you can use rmnodecanister\addnodecanister.

    Yes, I have a V7000 Storwize system (it's running 7.8.1.8, I want to update it to 7.8.1.10).  But, as noted above, I do not have an "addnodecanister" command available:

    superuser>help | grep canister
    cfgportip - Assigns an IP address to each node canister ethernet port for iSCSI I/O.
    chenclosurecanister - Changes the properties of an enclosure canister.
    chnodecanister - Changes the name and other attributes of a node canister.
    chnodecanisterhw - Updates the hardware configuration for a node canister.
    cleardumps - Cleans the dump directories on a specified node canister.
    cpdumps - Copies dump files to the configuration node canister from another node canister.
    lsdumps - Displays files in a dumps directory on one of the node canisters in the system.
    lsenclosurecanister - Displays status for each canister in an enclosure.
    lsfabric - Displays a report of the Fibre Channel connectivity between node canisters, controllers and hosts.
    lsiogrpcandidate - Displays I/O groups that can have node canisters added to them.
    lslivedump - Displays the livedump state of a node canister.
    lsnodecanister - Displays the node canisters that are part of the system.
    lsnodecanisterhw - Displays the hardware configuration of node canisters in the system.
    lsnodecanisterstats - Displays node canister statistics.
    lsnodecanistervpd - Displays the vital product data (VPD) for each node canister.
    lsportip - Displays the iSCSI IP addresses assigned for each port on each node canister in the system.
    lssasfabric - Displays which canisters are visible to a node canister, and the order of these canisters.
    lssystemstats - Displays the most recent values or a history of canister statistics.
    rmnodecanister - Deletes a node canister from the system.
    rmportip - Removes an iSCSI IP address from a node canister ethernet port.
    startstats - Changes the interval at which per-node canister statistics are collected.
    stopsystem - Shuts down a single node canister or the entire system.
    triggerlivedump - Captures a livedump to the internal disk on the node canister.
    superuser>

    There are commands for configuring/changing/listing information about node canisters, and there is a command to remove a node canister... but, once removed, there does not appear to be a command to add one back.

    I'm sure there *is* such a command, but it would be nice to verify what it is ahead of time (i.e. I don't want to start down a path, only to find out that yea, verily, there is no command to do what I want to do :-)

  • TMasteen
    TMasteen
    384 Posts

    Re: APAR IT18393

    ‏2019-07-30T13:47:29Z  
    • wolz
    • ‏2019-07-30T12:49:27Z

    Yes, I have a V7000 Storwize system (it's running 7.8.1.8, I want to update it to 7.8.1.10).  But, as noted above, I do not have an "addnodecanister" command available:

    superuser>help | grep canister
    cfgportip - Assigns an IP address to each node canister ethernet port for iSCSI I/O.
    chenclosurecanister - Changes the properties of an enclosure canister.
    chnodecanister - Changes the name and other attributes of a node canister.
    chnodecanisterhw - Updates the hardware configuration for a node canister.
    cleardumps - Cleans the dump directories on a specified node canister.
    cpdumps - Copies dump files to the configuration node canister from another node canister.
    lsdumps - Displays files in a dumps directory on one of the node canisters in the system.
    lsenclosurecanister - Displays status for each canister in an enclosure.
    lsfabric - Displays a report of the Fibre Channel connectivity between node canisters, controllers and hosts.
    lsiogrpcandidate - Displays I/O groups that can have node canisters added to them.
    lslivedump - Displays the livedump state of a node canister.
    lsnodecanister - Displays the node canisters that are part of the system.
    lsnodecanisterhw - Displays the hardware configuration of node canisters in the system.
    lsnodecanisterstats - Displays node canister statistics.
    lsnodecanistervpd - Displays the vital product data (VPD) for each node canister.
    lsportip - Displays the iSCSI IP addresses assigned for each port on each node canister in the system.
    lssasfabric - Displays which canisters are visible to a node canister, and the order of these canisters.
    lssystemstats - Displays the most recent values or a history of canister statistics.
    rmnodecanister - Deletes a node canister from the system.
    rmportip - Removes an iSCSI IP address from a node canister ethernet port.
    startstats - Changes the interval at which per-node canister statistics are collected.
    stopsystem - Shuts down a single node canister or the entire system.
    triggerlivedump - Captures a livedump to the internal disk on the node canister.
    superuser>

    There are commands for configuring/changing/listing information about node canisters, and there is a command to remove a node canister... but, once removed, there does not appear to be a command to add one back.

    I'm sure there *is* such a command, but it would be nice to verify what it is ahead of time (i.e. I don't want to start down a path, only to find out that yea, verily, there is no command to do what I want to do :-)

    Wolz, I did not notice this one... there is no addnodecanister command.. 

    Have a look at a maintenance task like replaceing a cmos-battery.  https://www.ibm.com/support/knowledgecenter/en/ST3FR7_8.3.0/com.ibm.storwize.v7000.830.doc/fab1_replace_cmos_battery.html

    This is a complete procedure to remove\add a canister from a cluster.

  • wolz
    wolz
    3 Posts

    Re: APAR IT18393

    ‏2019-07-30T21:59:12Z  
    • TMasteen
    • ‏2019-07-30T13:47:29Z

    Wolz, I did not notice this one... there is no addnodecanister command.. 

    Have a look at a maintenance task like replaceing a cmos-battery.  https://www.ibm.com/support/knowledgecenter/en/ST3FR7_8.3.0/com.ibm.storwize.v7000.830.doc/fab1_replace_cmos_battery.html

    This is a complete procedure to remove\add a canister from a cluster.

    Turns out, I have Gen1 canisters.  While the procedure to remove/add a Gen1 canister is (probably) largely identical to that for removing a Gen2 canister, I still needed a way to identify which canister needed to be removed. 

    Per PMR 93443,004,000 the way to identify the offending canister is to ssh into the V7000 and run "svcinfo lsportusb -delim :"  This will return a list of USB ports by node.  One node will have more ports than the other; that's the node that has to be removed from the system, as follows:

    1. Log in to the service GUI, select the node to be removed, select Enter Service State, and press Go

    2. Once the node is in service state, at the CLI run the command "rmnodecanister (nodename)"

    3. Wait awhile (the suggestion was to wait 30-40 miunutes).  If the node does not automatically come back online, refresh the service GUI and look for the node to be in service state with an error code of 690; if so, from the Actions menu choose Exit Service State and press Go.  Wait for both nodes to report as Active.

    At this point, the test utility should no longer report the APAR IT18393 error.