Topic
IC4NOTICE: developerWorks Community will be offline May 29-30, 2015 while we upgrade to the latest version of IBM Connections. For more information, read our upgrade FAQ.
4 replies Latest Post - ‏2012-08-01T12:46:12Z by joshuawhite929
SystemAdmin
SystemAdmin
1485 Posts
ACCEPTED ANSWER

Pinned topic Issue serializing to remote server

‏2012-08-01T03:19:44Z |
I'm seeing a strange issue when serializing data from a local WAS managed client to a remote WAS managed XS server. The client side is configured using 0 buckets, so it forces extreme scale to read and write from the server. The server is configured using a copyMode of COPY_TO_BYTES. If I put something like a string key and a string value, there is no issue. When I try to put something like a single "complex" object that is just a custom object with a single string in it, I get an error that the server has timed out. I imagine that the object is very small.

Using this same configuration but deploying the client to the remote server that is running the XS instance, this is not a problem.

Changing the server side configuration to use a default copy mode (not specified) gets rid of this issue as well (Even when running from local machine to remote server).

Here is the stack trace when using the copy to bytes mode and sending "complex" data from my local machine to the remote server:

Caused by: com.ibm.websphere.objectgrid.ObjectGridRuntimeException: Timeout elapsed. One or more agent targets did not return in the specified time. The timeout expiration is determined by the Session's transaction timeout.
at com.ibm.ws.objectgrid.datagrid.AgentManagerImpl.timeout(AgentManagerImpl.java:1263)
at com.ibm.ws.objectgrid.datagrid.AgentCallbackState.checkRequestExpiry(AgentCallbackState.java:240)
at com.ibm.ws.objectgrid.datagrid.AgentCallbackState.block(AgentCallbackState.java:142)
at com.ibm.ws.objectgrid.datagrid.AgentManagerImpl.callMapAgent(AgentManagerImpl.java:375)
at com.ibm.ws.objectgrid.plugins.SerializationInfoCacheImpl.getInfoForClassDescriptor(SerializationInfoCacheImpl.java:339)
at com.ibm.ws.objectgrid.plugins.SerializationInfoCacheImpl.lookupByClassDescriptor(SerializationInfoCacheImpl.java:209)
at com.ibm.ws.objectgrid.map.BaseMap.objectToBytes(BaseMap.java:12083)
... 53 more

I'm not really sure exactly why the timeout is happening. It almost seems like there is an issue serializing the data on the server. Does anyone have any insight into this issue?

Thanks
Updated on 2012-08-01T12:46:12Z at 2012-08-01T12:46:12Z by joshuawhite929
  • jhanders
    jhanders
    257 Posts
    ACCEPTED ANSWER

    Re: Issue serializing to remote server

    ‏2012-08-01T03:57:39Z  in response to SystemAdmin
    The problem here is that an agent is used for copy to bytes function. As such the server ends up calling back to the client to communicate the result. Likely you will see an FFDC on the server side showing that a commandCallback method is throwing an exception. Likely there is a security setting that is preventing the XS server process to call back to the client process.

    I hope this helps.
    • joshuawhite929
      joshuawhite929
      52 Posts
      ACCEPTED ANSWER

      Re: Issue serializing to remote server

      ‏2012-08-01T11:41:23Z  in response to jhanders
      Jared,

      We will certainly take a look in the ffdc directory to see what we find. As James has mentioned, using the default copy mode works fine. Could you clarify for us the differences between using the default copymode and copy_to_bytes copymode under the hood (for serialization)? I would assume that both copymodes would have to perform the same/similar serialization process (default) when doing a "put". Is an agent used in one and not the other? (We understand the fundamental differences in how the data is stored).

      Also, if using a simple String object works using copy_to_bytes, can you help us understand how/why security might be a factor when using a more complicated object?

      Thanks,

      Joshua
      • jhanders
        jhanders
        257 Posts
        ACCEPTED ANSWER

        Re: Issue serializing to remote server

        ‏2012-08-01T11:59:13Z  in response to joshuawhite929
        Joshua,

        No problem. We only do the agent when doing the COPY_TO_BYTES CopyMode. We do not use it with the other CopyMode's. When you store a String with COPY_TO_BYTES we already know how we are going to serialize it. We have built-in type support to serialize simple Java objects efficiently and don't have to store any metadata on the server side. When using a custom object we flow the metadata to the server side with an agent. If you wrote an agent or did an ObjectMap.clear in your application you would see this same issue. There are changes coming in the future to help this.

        To understand more details when you do default serialization in Java they end up writing the class descriptor (java.io.ObjectStreamClass) to the stream. We end up writing that once to the server using an agent and put in a pointer to it in the bytes instead of having every entry with a value of that custom type have the class descriptor duplicated and wasting bytes.

        To avoid this problem you could write your own ObjectTransformer or DataSerializer and then you control what is put into the stream. If you don't do a ObjectOutputStream.writeObject() you won't have this issue because we will never flow the class descriptor to the grid.

        I hope this helps answer all your questions.

        Jared
        • joshuawhite929
          joshuawhite929
          52 Posts
          ACCEPTED ANSWER

          Re: Issue serializing to remote server

          ‏2012-08-01T12:46:12Z  in response to jhanders
          Jared,

          This may be a little off topic, but is this the reason why it is recommended to disconnect/reconnect if there is any type of failure/restart of extreme scale (when using copy_to_bytes)? If I remember right, the cached meta-data gets out of sync on the client side. Do you have any recommendations on how we should be detecting this and performing the disconnect/reconnect?

          Back on topic... thanks for the clarification. I am still trying to understand where the problem/timeout is occurring. Can you shed some light on what might be happening here? Why would security impact this operation?

          If we change to a custom serialization strategy, you seem to indicate that this problem will go away. Are you saying that the problem is specifically with writing the objects meta-data? So if we changed to a Hessian (as an example) serialization strategy, you think that this problem would go away?

          -Joshua