Topic
9 replies Latest Post - ‏2011-01-17T15:52:15Z by tonye1
tonye1
tonye1
5 Posts
ACCEPTED ANSWER

Pinned topic XC10 transaction failure on commit

‏2011-01-11T20:12:19Z |
Hi,
I'm doing an evaluation of an XC10 for IBM internal use. Unfortunately, even for my simple data grid, I'm having problems getting it to work correctly. I wonder if folks are aware and can give a work-around?

I have a simple grid where I use simple APIs to access. Here's a snippet of my API code:
try {
gridSession = grid.getSession();
ObjectMap map = gridSession.getMap(mapToUse);
gridSession.begin();
out.println("got a session...");
for (int j = 0;j < 64000; j++) { // yeah, I know it's 64 K worth of data...
ba[j] = '1';
}
map.put(sessionKey, ba);
gridSession.commit();

} catch (Exception e) {

This code works perfectly on my WXS 7.1 grid on AIX and Windows, but fails not on the XC10.

I can get this to successfully commit about every 2nd or 3rd API call. Usually, I get this on the commit:
com.ibm.websphere.objectgrid.ClientServerTransactionCallbackException:
Client Services - Exception during commit processing:
com.ibm.websphere.objectgrid.TargetNotAvailableException:
org.omg.CORBA.TRANSIENT: initial and forwarded IOR inaccessible
vmcid: IBM minor code: E07 completed: No

I've switched the API from "put" to "insert"...it makes no difference. I've tried tracing the XC10 with ObjectGrid*=all=enabled,
but I don't see the APIs being traced in the same way as I do on AIX/windows grid with the same trace spec.

I've attached the html and servlet I'm using for the test...but really, any API calls this way will show the problem.
I'm available internally to IBM - just bluepages search here:
http://w3.ibm.com/jct03019wt/bluepages/simpleSearch.wss?searchBy=Name&location=All+locations&searchFor=tony+efremenko

The XC10 I'm using is version 1.0.0, but has some firmware updates.I'm looking forward to hearing from someone.
Regards
-Tony Efremenko
http://w3.ibm.com/jct03019wt/bluepages/simpleSearch.wss?searchBy=Name&location=All+locations&searchFor=tony+efremenko
Updated on 2011-01-17T15:52:15Z at 2011-01-17T15:52:15Z by tonye1
  • bkmartin
    bkmartin
    11 Posts
    ACCEPTED ANSWER

    Re: XC10 transaction failure on commit

    ‏2011-01-12T21:04:37Z  in response to tonye1
    The error you are describing is caused when some partitions are "offline". Keys are hashed to partitions and when your key hashes to an offline partition, you are getting an error. The 1.0.0 firmware that was shipped with the product from the factory had bugs and a firmware upgrade is required when you first receive the appliance. You should apply the 1.0.0.4 firmware and do a clear-all and the problem should be resolved.

    Brian
    • bkmartin
      bkmartin
      11 Posts
      ACCEPTED ANSWER

      Re: XC10 transaction failure on commit

      ‏2011-01-12T21:06:25Z  in response to bkmartin
      I should have been more specific in that the "clear-all" is only required when upgrading from the factory 1.0.0 firmware. In general, as you move from 1.0.0.4 to 1.0.05, etc, no clear-all is required. This requirement (to clear-all from the factory firmware) is documented in the XC10 infocenter.

      Brian
      • tonye1
        tonye1
        5 Posts
        ACCEPTED ANSWER

        Re: XC10 transaction failure on commit

        ‏2011-01-13T16:39:46Z  in response to bkmartin
        Hi Brian,
        Unfortunately, I upgraded to the xx firmware level, and cleared events, but still see the problem
        (the machine was already at that firmware level, but I reloaded, just to be sure).

        Here's the interesting part of the stack trace:
        1/13/11 11:34:05:814 EST FFDC Exception:com.ibm.websphere.objectgrid.TargetNotAvailableException SourceId:com.ibm.ws.objectgrid.client.RemoteTransactionCallbackImpl.processReadWriteRequest ProbeId:304 Reporter:com.ibm.ws.objectgrid.client.RemoteTransactionCallbackImpl@73307330
        com.ibm.websphere.objectgrid.TargetNotAvailableException: org.omg.CORBA.TRANSIENT: initial and forwarded IOR inaccessible vmcid: IBM minor code: E07 completed: No
        at com.ibm.ws.objectgrid.client.ORBClientCoreMessageHandler.sendMessage(ORBClientCoreMessageHandler.java:508)
        at com.ibm.ws.objectgrid.client.ORBClientCoreMessageHandler.sendReadWriteRequest(ORBClientCoreMessageHandler.java:757)
        at com.ibm.ws.objectgrid.client.RemoteTransactionCallbackImpl.processReadWriteRequestAndResponse(RemoteTransactionCallbackImpl.java:961)
        at com.ibm.ws.objectgrid.client.RemoteTransactionCallbackImpl.commit(RemoteTransactionCallbackImpl.java:275)
        at com.ibm.ws.objectgrid.SessionImpl.commit(SessionImpl.java:1401)
        at XC10Servlet.doPost(XC10Servlet.java:220)
        (I've attached the ffdc file)

        I do accept your explanation of what's happening. Not sure how you prefer to proceed...I'm happy to work with your team for further
        debugging, if you like. We can also talk internally (outside the forum) about what's next.

        -Best
        Tony Efremenko, tonye1@us.ibm.com
  • SystemAdmin
    SystemAdmin
    25 Posts
    ACCEPTED ANSWER

    Re: XC10 transaction failure on commit

    ‏2011-01-13T19:46:06Z  in response to tonye1
    Can you please send me the logs from the XC10 appliance?

    Thanks,
    Tom
    gissel@us.ibm.com
    • tonye1
      tonye1
      5 Posts
      ACCEPTED ANSWER

      Re: XC10 transaction failure on commit

      ‏2011-01-13T20:10:26Z  in response to SystemAdmin
      Thanks, Tom, I sent you a link to the logs via Lotus Connections - Thanks for your help
      Tony Efremenko (tonye1@us.ibm.com)
  • SystemAdmin
    SystemAdmin
    25 Posts
    ACCEPTED ANSWER

    Re: XC10 transaction failure on commit

    ‏2011-01-13T21:43:40Z  in response to tonye1
    Hi Tony,

    The logs seem to indicate that one of the IPs configured on the XC10 is for a private network, 172.20.1.65, whereas the others are on another network. If you want to configure a private IP for administrative purposes that can be done but the interface needs to be marked as a "Private IP Address". After marking the IP as private, a restart of the appliance my be initiated.

    Thanks,
    Tom
    • tonye1
      tonye1
      5 Posts
      ACCEPTED ANSWER

      Re: XC10 transaction failure on commit

      ‏2011-01-14T16:49:29Z  in response to SystemAdmin
      Hi Tom,
      I was skeptical that the "private" IP had anything to do with it, but I disabled it (I just unchecked it - it was set up by somebody else) and rebooted and things do seem to be working much better. Before, about every third insert would hang - I've just run about 16 inserts,
      and it's doing pretty well.
      So my take back on this is "watch what IP interfaces you use" ? Considering I never actually came across that interface, I still don't
      understand why it matters, but success is success.
      If you want to clue me in on what the other ETH interface had to do with that, that's fine, but otherwise, I'll just close out this thread
      (Thanks very much for all your help!)- Tony Efremenko (tonye1@us.ibm.com)
      • bkmartin
        bkmartin
        11 Posts
        ACCEPTED ANSWER

        Re: XC10 transaction failure on commit

        ‏2011-01-14T22:30:08Z  in response to tonye1
        For completeness, let me explain why the private network configuration mattered. The XC10 has 4 network interfaces. If you configure all 4 interfaces and don't indicate that any of the adapters are "private", then the product assumes you want all 4 interfaces to be used for data grid traffic. As I mentioned earlier in this thread, the XC10 automatically partitions your data and along with that these partitions are assigned to configured network adapters in your XC10 so that the network traffic is also partitioned.

        Imagine there are 4 partitions (to keep the example simple) and you have configured 2 ethernet ports and a grid named MyTestGrid. The layout may look like this

        eth0: MyTestGrid:1 MyTestGrid:3
        eth1: MyTestGrid:2 MyTestGrid:4

        So requests that hash to partition 2 would go to the IP serviced by eth1 and requests that hash to partition 3 would go to eth0. In your case, your client was only able to access one of the two networks, so when a request hashed to a partition on the inaccessible network, then you received the targetnotavailable exception.

        I hope this clarifies the situation.

        Brian
        • tonye1
          tonye1
          5 Posts
          ACCEPTED ANSWER

          Re: XC10 transaction failure on commit

          ‏2011-01-17T15:52:15Z  in response to bkmartin
          Thanks, Brian, That does clarify it. Thanks for the insight.

          Please consider this thread closed, and thank you.
          -Tony Efremenko (tonye1@us.ibm.com)