Topic
  • 9 replies
  • Latest Post - ‏2013-08-19T13:33:41Z by Dev_Dhoot
Dev_Dhoot
Dev_Dhoot
42 Posts

Pinned topic Extreme Scale exception "com.ibm.websphere.objectgrid.TargetNotAvailableException"??

‏2013-08-06T13:00:41Z |

I am using IBM WebSphere Extreme Scale 8.6.0.2. I am seeing the following exception in the logs.

 

===

Caused by: com.ibm.websphere.objectgrid.TargetNotAvailableException: DefaultDomain:Grid:mapSet:k com.ibm.ws.xsspi.xio.exception.InvalidXIORefException [originating=x.x.x.x:4009;causedby=y.y.y.y:0;reqId=66054803;exid=32816360]: com.ibm.ws.xsspi.xio.exception.InvalidXIORefException:unable to find actor at index=40
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.sendMessage(XIOClientCoreMessageHandler.java:639) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.retry(XIOClientCoreMessageHandler.java:822) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.sendMessage(XIOClientCoreMessageHandler.java:721) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.retry(XIOClientCoreMessageHandler.java:822) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.sendMessage(XIOClientCoreMessageHandler.java:721) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
....
....
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.sendMessage(XIOClientCoreMessageHandler.java:721) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.retry(XIOClientCoreMessageHandler.java:822) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.sendMessage(XIOClientCoreMessageHandler.java:721) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.retry(XIOClientCoreMessageHandler.java:822) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.sendMessage(XIOClientCoreMessageHandler.java:721) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.retry(XIOClientCoreMessageHandler.java:822) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.sendMessage(XIOClientCoreMessageHandler.java:721) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.retry(XIOClientCoreMessageHandler.java:822) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.sendMessage(XIOClientCoreMessageHandler.java:721) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.retry(XIOClientCoreMessageHandler.java:822) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.sendMessage(XIOClientCoreMessageHandler.java:721) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.retry(XIOClientCoreMessageHandler.java:822) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.sendMessage(XIOClientCoreMessageHandler.java:721) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.client.CommonClientCoreMessageHandler.sendGetRequest(CommonClientCoreMessageHandler.java:224) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.client.RemoteTransactionCallbackImpl.mapGet(RemoteTransactionCallbackImpl.java:1942) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.client.RemoteCacheLoader.get(RemoteCacheLoader.java:462) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.objectgrid.map.BaseMap.getRaw(BaseMap.java:4785) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
... 147 common frames omitted
Caused by: com.ibm.ws.xsspi.xio.exception.InvalidXIORefException: com.ibm.ws.xsspi.xio.exception.InvalidXIORefException:unable to find actor at index=40
at sun.reflect.GeneratedConstructorAccessor73.newInstance(Unknown Source) ~[na:na]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) ~[na:1.6.0]
at java.lang.reflect.Constructor.newInstance(Constructor.java:516) ~[na:1.6.0]
at com.ibm.ws.xsspi.xio.exception.XIOExceptionConverter.constructException(XIOExceptionConverter.java:233) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.xsspi.xio.exception.XIOExceptionConverter.convertExceptionMessageToExceptionRecursive(XIOExceptionConverter.java:149) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.xsspi.xio.exception.XIOExceptionConverter.convertExceptionMessageToException(XIOExceptionConverter.java:133) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.xsspi.xio.dispatch.MessageInfo.throwException(MessageInfo.java:145) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.xsspi.xio.dispatch.MessageInfo.getMessage(MessageInfo.java:104) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]
at com.ibm.ws.xsspi.xio.dispatch.MessageInfo.getMessage(MessageInfo.java:205) ~[wsogclient.jar:WXS8.6.0.2.XS [cf21326.29112045]]

===

I have tried looking into this exceptions but with no avail.

Can anyone help me identify the root cause behind this.

--Devendra

 

  • jhanders
    jhanders
    261 Posts

    Re: Extreme Scale exception "com.ibm.websphere.objectgrid.TargetNotAvailableException"??

    ‏2013-08-07T13:11:06Z  
    This reply was deleted by Dev_Dhoot 2013-08-07T15:12:26Z.

    Likely something is wrong on the server side.  I would look to see if there are any error conditions for that specific partition or if it is not placed.  Using xscmd's showObjectGridPlacement and routetable's commands should help determine if there is an issue with the server side or client side for that particular partition.  There were updates I believe in the 8.6.0.3 release for invalid XIO ref errors.  Trying to use 8.6.0.3 may also resolve this issue for you.

    I hope that helps.

    Jared Anderson

  • Dev_Dhoot
    Dev_Dhoot
    42 Posts

    Re: Extreme Scale exception "com.ibm.websphere.objectgrid.TargetNotAvailableException"??

    ‏2013-08-07T14:16:19Z  
    • jhanders
    • ‏2013-08-07T13:11:06Z

    Likely something is wrong on the server side.  I would look to see if there are any error conditions for that specific partition or if it is not placed.  Using xscmd's showObjectGridPlacement and routetable's commands should help determine if there is an issue with the server side or client side for that particular partition.  There were updates I believe in the 8.6.0.3 release for invalid XIO ref errors.  Trying to use 8.6.0.3 may also resolve this issue for you.

    I hope that helps.

    Jared Anderson

    Jared,

    Thanks for the reply.

    I have executed the routetable command on the grid and found that the particular Map Set in question is not available.

    Now, I wonder would refreshing the client JVM  would resolve the issue or what else can be done to fix it.

     

    -Devendra

  • jhanders
    jhanders
    261 Posts

    Re: Extreme Scale exception "com.ibm.websphere.objectgrid.TargetNotAvailableException"??

    ‏2013-08-07T14:20:37Z  
    • Dev_Dhoot
    • ‏2013-08-07T14:16:19Z

    Jared,

    Thanks for the reply.

    I have executed the routetable command on the grid and found that the particular Map Set in question is not available.

    Now, I wonder would refreshing the client JVM  would resolve the issue or what else can be done to fix it.

     

    -Devendra

    There could be a few problems.  I would look to see if there were failures during activation of a shard.  If you provide some logs it would be easier to help determine what is going on.

  • Dev_Dhoot
    Dev_Dhoot
    42 Posts

    Re: Extreme Scale exception "com.ibm.websphere.objectgrid.TargetNotAvailableException"??

    ‏2013-08-07T14:25:07Z  
    • jhanders
    • ‏2013-08-07T14:20:37Z

    There could be a few problems.  I would look to see if there were failures during activation of a shard.  If you provide some logs it would be easier to help determine what is going on.

    Jared,

    Could you please guide me on what logs should I gather and provide you based on the exception of stack trace above.

    --Devendra

  • jhanders
    jhanders
    261 Posts

    Re: Extreme Scale exception "com.ibm.websphere.objectgrid.TargetNotAvailableException"??

    ‏2013-08-07T14:29:50Z  
    • Dev_Dhoot
    • ‏2013-08-07T14:25:07Z

    Jared,

    Could you please guide me on what logs should I gather and provide you based on the exception of stack trace above.

    --Devendra

    The container and catalog server logs directories.  Take the logs directory and all subsequent subdirectories.  This will include all FFDC as well as the server logs.

  • Dev_Dhoot
    Dev_Dhoot
    42 Posts

    Re: Extreme Scale exception "com.ibm.websphere.objectgrid.TargetNotAvailableException"??

    ‏2013-08-07T14:37:23Z  
    • jhanders
    • ‏2013-08-07T14:29:50Z

    The container and catalog server logs directories.  Take the logs directory and all subsequent subdirectories.  This will include all FFDC as well as the server logs.

    Jared,

    Thanks for your quick response.

    It would not be possible for me to send the entire logs directory on the forum due to internal reasons.

    Could you provide me your official IBM id on which I can send you the logs in zip format.

     

    Appreciate you understand my concern.

     

    --Devendra

  • jhanders
    jhanders
    261 Posts

    Re: Extreme Scale exception "com.ibm.websphere.objectgrid.TargetNotAvailableException"??

    ‏2013-08-07T14:47:49Z  
    • Dev_Dhoot
    • ‏2013-08-07T14:37:23Z

    Jared,

    Thanks for your quick response.

    It would not be possible for me to send the entire logs directory on the forum due to internal reasons.

    Could you provide me your official IBM id on which I can send you the logs in zip format.

     

    Appreciate you understand my concern.

     

    --Devendra

    As to avoid spam, my email is the userid that you see for me at us dot ibm dot com.

  • Dev_Dhoot
    Dev_Dhoot
    42 Posts

    Re: Extreme Scale exception "com.ibm.websphere.objectgrid.TargetNotAvailableException"??

    ‏2013-08-08T12:38:35Z  
    • jhanders
    • ‏2013-08-07T14:47:49Z

    As to avoid spam, my email is the userid that you see for me at us dot ibm dot com.

    I have shared across the logs with you.

  • Dev_Dhoot
    Dev_Dhoot
    42 Posts

    Re: Extreme Scale exception "com.ibm.websphere.objectgrid.TargetNotAvailableException"??

    ‏2013-08-19T13:33:41Z  
    • Dev_Dhoot
    • ‏2013-08-08T12:38:35Z

    I have shared across the logs with you.

    Jared,

    We have  upgraded the  server to 8.6.0.3 but the issue still persists.

     

    ==

    CWOBJ0006W: An exception occurred: com.ibm.websphere.objectgrid.TargetNotAvailableException: DefaultDomain:Grid:mapSet:7 com.ibm.ws.xsspi.xio.exception.InvalidXIORefException [originating=x.x.x.x.:4010;causedby=y.y.y.y:0;reqId=488932;exid=672283]: com.ibm.ws.xsspi.xio.exception.InvalidXIORefException:Target has an endpoint id=40e0a1408ffb4204e0000050569b01df but local id=416b71409024417ee0000050569b01df. Local process probably restarted.
    at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.sendMessage(XIOClientCoreMessageHandler.java:639)
    at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.retry(XIOClientCoreMessageHandler.java:822)

    ===

     

    Can you let me know the fix for this issue. I guess PM92210 should have fixed the issue.

    --Devendra