Topic
9 replies Latest Post - ‏2013-08-19T13:33:41Z by Dev_Dhoot
Dev_Dhoot
Dev_Dhoot
42 Posts
ACCEPTED ANSWER

Pinned topic Extreme Scale exception "com.ibm.websphere.objectgrid.TargetNotAvailableException"??

‏2013-08-06T13:00:41Z |

I am using IBM WebSphere Extreme Scale 8.6.0.2. I am seeing the following exception in the logs.

 

===

Caused by: com.ibm.websphere.objectgrid.TargetNotAvailableException: DefaultDomain:Grid:mapSet:k com.ibm.ws.xsspi.xio.exception.InvalidXIORefException [originating=x.x.x.x:4009;causedby=y.y.y.y:0;reqId=66054803;exid=32816360]: com.ibm.ws.xsspi.xio.exception.InvalidXIORefException:unable to find actor at index=40
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.sendMessage(XIOClientCoreMessageHandler.java:639) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.retry(XIOClientCoreMessageHandler.java:822) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.sendMessage(XIOClientCoreMessageHandler.java:721) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.retry(XIOClientCoreMessageHandler.java:822) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.sendMessage(XIOClientCoreMessageHandler.java:721) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
....
....
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.sendMessage(XIOClientCoreMessageHandler.java:721) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.retry(XIOClientCoreMessageHandler.java:822) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.sendMessage(XIOClientCoreMessageHandler.java:721) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.retry(XIOClientCoreMessageHandler.java:822) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.sendMessage(XIOClientCoreMessageHandler.java:721) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.retry(XIOClientCoreMessageHandler.java:822) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.sendMessage(XIOClientCoreMessageHandler.java:721) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.retry(XIOClientCoreMessageHandler.java:822) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.sendMessage(XIOClientCoreMessageHandler.java:721) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.retry(XIOClientCoreMessageHandler.java:822) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.sendMessage(XIOClientCoreMessageHandler.java:721) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.retry(XIOClientCoreMessageHandler.java:822) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.sendMessage(XIOClientCoreMessageHandler.java:721) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.client.CommonClientCoreMessageHandler.sendGetRequest(CommonClientCoreMessageHandler.java:224) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.client.RemoteTransactionCallbackImpl.mapGet(RemoteTransactionCallbackImpl.java:1942) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.client.RemoteCacheLoader.get(RemoteCacheLoader.java:462) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.map.BaseMap.getRaw(BaseMap.java:4785) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
... 147 common frames omitted
Caused by: com.ibm.ws.xsspi.xio.exception.InvalidXIORefException: com.ibm.ws.xsspi.xio.exception.InvalidXIORefException:unable to find actor at index=40
at sun.reflect.GeneratedConstructorAccessor73.newInstance(Unknown Source) ~[na:na]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) ~[na:1.6.0]
at java.lang.reflect.Constructor.newInstance(Constructor.java:516) ~[na:1.6.0]
at com.ibm.ws.xsspi.xio.exception.XIOExceptionConverter.constructException(XIOExceptionConverter.java:233) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.xsspi.xio.exception.XIOExceptionConverter.convertExceptionMessageToExceptionRecursive(XIOExceptionConverter.java:149) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.xsspi.xio.exception.XIOExceptionConverter.convertExceptionMessageToException(XIOExceptionConverter.java:133) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.xsspi.xio.dispatch.MessageInfo.throwException(MessageInfo.java:145) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.xsspi.xio.dispatch.MessageInfo.getMessage(MessageInfo.java:104) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.xsspi.xio.dispatch.MessageInfo.getMessage(MessageInfo.java:205) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]

===

I have tried looking into this exceptions but with no avail.

Can anyone help me identify the root cause behind this.

--Devendra

 

  • This reply was deleted by Dev_Dhoot 2013-08-07T15:12:26Z.
    • jhanders
      jhanders
      253 Posts
      ACCEPTED ANSWER

      Re: Extreme Scale exception "com.ibm.websphere.objectgrid.TargetNotAvailableException"??

      ‏2013-08-07T13:11:06Z  in response to Dev_Dhoot

      Likely something is wrong on the server side.  I would look to see if there are any error conditions for that specific partition or if it is not placed.  Using xscmd's showObjectGridPlacement and routetable's commands should help determine if there is an issue with the server side or client side for that particular partition.  There were updates I believe in the 8.6.0.3 release for invalid XIO ref errors.  Trying to use 8.6.0.3 may also resolve this issue for you.

      I hope that helps.

      Jared Anderson

      • Dev_Dhoot
        Dev_Dhoot
        42 Posts
        ACCEPTED ANSWER

        Re: Extreme Scale exception "com.ibm.websphere.objectgrid.TargetNotAvailableException"??

        ‏2013-08-07T14:16:19Z  in response to jhanders

        Jared,

        Thanks for the reply.

        I have executed the routetable command on the grid and found that the particular Map Set in question is not available.

        Now, I wonder would refreshing the client JVM  would resolve the issue or what else can be done to fix it.

         

        -Devendra

        • jhanders
          jhanders
          253 Posts
          ACCEPTED ANSWER

          Re: Extreme Scale exception "com.ibm.websphere.objectgrid.TargetNotAvailableException"??

          ‏2013-08-07T14:20:37Z  in response to Dev_Dhoot

          There could be a few problems.  I would look to see if there were failures during activation of a shard.  If you provide some logs it would be easier to help determine what is going on.

          • Dev_Dhoot
            Dev_Dhoot
            42 Posts
            ACCEPTED ANSWER

            Re: Extreme Scale exception "com.ibm.websphere.objectgrid.TargetNotAvailableException"??

            ‏2013-08-07T14:25:07Z  in response to jhanders

            Jared,

            Could you please guide me on what logs should I gather and provide you based on the exception of stack trace above.

            --Devendra

            • jhanders
              jhanders
              253 Posts
              ACCEPTED ANSWER

              Re: Extreme Scale exception "com.ibm.websphere.objectgrid.TargetNotAvailableException"??

              ‏2013-08-07T14:29:50Z  in response to Dev_Dhoot

              The container and catalog server logs directories.  Take the logs directory and all subsequent subdirectories.  This will include all FFDC as well as the server logs.

              • Dev_Dhoot
                Dev_Dhoot
                42 Posts
                ACCEPTED ANSWER

                Re: Extreme Scale exception "com.ibm.websphere.objectgrid.TargetNotAvailableException"??

                ‏2013-08-07T14:37:23Z  in response to jhanders

                Jared,

                Thanks for your quick response.

                It would not be possible for me to send the entire logs directory on the forum due to internal reasons.

                Could you provide me your official IBM id on which I can send you the logs in zip format.

                 

                Appreciate you understand my concern.

                 

                --Devendra

                • jhanders
                  jhanders
                  253 Posts
                  ACCEPTED ANSWER

                  Re: Extreme Scale exception "com.ibm.websphere.objectgrid.TargetNotAvailableException"??

                  ‏2013-08-07T14:47:49Z  in response to Dev_Dhoot

                  As to avoid spam, my email is the userid that you see for me at us dot ibm dot com.

                  • Dev_Dhoot
                    Dev_Dhoot
                    42 Posts
                    ACCEPTED ANSWER

                    Re: Extreme Scale exception "com.ibm.websphere.objectgrid.TargetNotAvailableException"??

                    ‏2013-08-08T12:38:35Z  in response to jhanders

                    I have shared across the logs with you.

                    • Dev_Dhoot
                      Dev_Dhoot
                      42 Posts
                      ACCEPTED ANSWER

                      Re: Extreme Scale exception "com.ibm.websphere.objectgrid.TargetNotAvailableException"??

                      ‏2013-08-19T13:33:41Z  in response to Dev_Dhoot

                      Jared,

                      We have  upgraded the  server to 8.6.0.3 but the issue still persists.

                       

                      ==

                      CWOBJ0006W: An exception occurred: com.ibm.websphere.objectgrid.TargetNotAvailableException: DefaultDomain:Grid:mapSet:7 com.ibm.ws.xsspi.xio.exception.InvalidXIORefException [originating=x.x.x.x.:4010;causedby=y.y.y.y:0;reqId=488932;exid=672283]: com.ibm.ws.xsspi.xio.exception.InvalidXIORefException:Target has an endpoint id=40e0a1408ffb4204e0000050569b01df but local id=416b71409024417ee0000050569b01df. Local process probably restarted.
                      at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.sendMessage(XIOClientCoreMessageHandler.java:639)
                      at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.retry(XIOClientCoreMessageHandler.java:822)

                      ===

                       

                      Can you let me know the fix for this issue. I guess PM92210 should have fixed the issue.

                      --Devendra