Topic
  • 6 replies
  • Latest Post - ‏2013-12-24T11:50:17Z by HermannSW
DPUser77
DPUser77
70 Posts

Pinned topic Any limit on # of TPS that can be generated through DP in ideal conditions

‏2013-12-18T14:58:50Z |

Hi,

Is there any limit on # of TPS (transaction per second) that can be achieved through DP.

We need to use an ESB for cobol to xml transformation and exploring on both MB and DP  as a solution for the same.

In DP 'DataGlue' license is required for transformations but wanted to explore on the performance aspects, if we can achieve better performance through DP.

Please share your valuable thoughts on this and correct me if i am wrong. 

 

  • ted.jump
    ted.jump
    454 Posts

    Re: Any limit on # of TPS that can be generated through DP in ideal conditions

    ‏2013-12-18T15:45:50Z  

    I don't know what the official party line is but I've seen over 500 TPS through one XI52 appliance using firmware 6 to a very simple back-side service, with debug support active.

    I don't know if I have hit the high water mark yet, nor what the system utilization level was. We've only started performance profiling the related MPGW.

    It was clear in the trivial case that the test was client-bound as increasing concurrency (more client threads making concurrent requests from a single client host) scaled up throughput quite a bit.

  • HermannSW
    HermannSW
    6019 Posts

    Re: Any limit on # of TPS that can be generated through DP in ideal conditions

    ‏2013-12-18T23:08:43Z  
    • ted.jump
    • ‏2013-12-18T15:45:50Z

    I don't know what the official party line is but I've seen over 500 TPS through one XI52 appliance using firmware 6 to a very simple back-side service, with debug support active.

    I don't know if I have hit the high water mark yet, nor what the system utilization level was. We've only started performance profiling the related MPGW.

    It was clear in the trivial case that the test was client-bound as increasing concurrency (more client threads making concurrent requests from a single client host) scaled up throughput quite a bit.

    As alwys, you will have to measure yourself!

    I did find 3 XML files of sizes 24, 285 and 2870 bytes.

    I created two loopback passthru XML FWs on two XI52s on port 40000.

    And I did add a MPGW on dp1-l2:9001 doing "store:///identity.xsl" transformation on request and response rule against dp1-l3:40000.

    My dev workstation allows to run 1000 connections in parallel, and so I did run this command against the different endpoints:

    ab -n 100000 -c 1000 -p input.xml http://dpbox:port

     

    The dev workstation and both XI52s are connected via 1Gbps to network, and workstation to each box has 2 hops.

    Connection from dp1-l2 to dp1-l3 is done over 10Gbit direct cable connection (eth20).

    Find below the requests per second information, as well as CPU utilization on dp1-l2 box when doing the store:///identity.xsl transformations:

      ->dp1-l2:40000 ->dp1-l3:40000 ->dp1-l2:9001(identity.xsl)->dp1-l3:40000
    <-                      (identity.xsl)<-
     
    24 17099 16968 14116 5
    285 16694 16961 15047 11
    2870 12453 12593 12043 23
    request [bytes] [requests/sec] [requests/sec] [requests/sec] cpu usage (%)

     

    So all will depend on the complexity of you Cobol to XML and XML to Cobol  binary transformations and whether your backend can cope with what it gets from DataPower ...


    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/> <myFrameless/>

    Updated on 2013-12-18T23:19:18Z at 2013-12-18T23:19:18Z by HermannSW
  • JoeMorganNTST
    JoeMorganNTST
    427 Posts

    Re: Any limit on # of TPS that can be generated through DP in ideal conditions

    ‏2013-12-20T20:55:33Z  
    • HermannSW
    • ‏2013-12-18T23:08:43Z

    As alwys, you will have to measure yourself!

    I did find 3 XML files of sizes 24, 285 and 2870 bytes.

    I created two loopback passthru XML FWs on two XI52s on port 40000.

    And I did add a MPGW on dp1-l2:9001 doing "store:///identity.xsl" transformation on request and response rule against dp1-l3:40000.

    My dev workstation allows to run 1000 connections in parallel, and so I did run this command against the different endpoints:

    ab -n 100000 -c 1000 -p input.xml http://dpbox:port

     

    The dev workstation and both XI52s are connected via 1Gbps to network, and workstation to each box has 2 hops.

    Connection from dp1-l2 to dp1-l3 is done over 10Gbit direct cable connection (eth20).

    Find below the requests per second information, as well as CPU utilization on dp1-l2 box when doing the store:///identity.xsl transformations:

      ->dp1-l2:40000 ->dp1-l3:40000 ->dp1-l2:9001(identity.xsl)->dp1-l3:40000
    <-                      (identity.xsl)<-
     
    24 17099 16968 14116 5
    285 16694 16961 15047 11
    2870 12453 12593 12043 23
    request [bytes] [requests/sec] [requests/sec] [requests/sec] cpu usage (%)

     

    So all will depend on the complexity of you Cobol to XML and XML to Cobol  binary transformations and whether your backend can cope with what it gets from DataPower ...


    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/> <myFrameless/>

    Hermann,

    We've discussed using direct vable over the 10GB switch for DP to DP communication.

    How are you doing the 10Gbit direct cable connection?  Is this a cross-over cable, or does DP handle it like a 10GB switch??

    One other issue is when when have AO.  I can envision a cross-over scenario where we use both 10GB ports between 2 appliances, A1 -> B1, A1 -> B2, A2 -> B2 and A2 -> B1, but I can't see how that can solve for AO, and I'm guessing it won't.

    I'd like to hear your ideas.

     

  • HermannSW
    HermannSW
    6019 Posts

    Re: Any limit on # of TPS that can be generated through DP in ideal conditions

    ‏2013-12-20T23:38:19Z  

    Hermann,

    We've discussed using direct vable over the 10GB switch for DP to DP communication.

    How are you doing the 10Gbit direct cable connection?  Is this a cross-over cable, or does DP handle it like a 10GB switch??

    One other issue is when when have AO.  I can envision a cross-over scenario where we use both 10GB ports between 2 appliances, A1 -> B1, A1 -> B2, A2 -> B2 and A2 -> B1, but I can't see how that can solve for AO, and I'm guessing it won't.

    I'd like to hear your ideas.

     

    Hi Joe,

    I am not a networking guy, I requested a cable from network people at IBM Böblingen lab (Germany).
    The guy told me that the one I got is special in that it allows connectors be inserted in both orientations.

    But what I know (as I said, not a networking guy), you can use both, "normal" cable as well as "cross" cable
    to connect DataPower boxes. DataPower interfaces will deal with both and make the connection.

     

    I doubt that direct connections can help to replace AO.

    What I did use the direct cable for is access backup.
    If I cannot ssh into one box mgmt0/mgmt1 interface, I can do so via tcpproxy on other device.
    So instead of "ssh dp1-l3" I can login additionally via "ssh -p 23  dp1-l2".
    And I do have tcpproxy "9091" that allows to login to WebGUI of "the other" box over port 9091 instead of 9090.

    xi52(config)# show tcpproxy 23

    tcpproxy: 23 [up]
    ------------
     admin-state enabled
     ip-address 0.0.0.0
     Service Priority normal
     Port Number 23
     Remote Host 192.168.0.2
     Remote Port 22


    xi52(config)#


    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/> <myFrameless/>

  • JoeMorganNTST
    JoeMorganNTST
    427 Posts

    Re: Any limit on # of TPS that can be generated through DP in ideal conditions

    ‏2013-12-23T15:10:37Z  
    • HermannSW
    • ‏2013-12-20T23:38:19Z

    Hi Joe,

    I am not a networking guy, I requested a cable from network people at IBM Böblingen lab (Germany).
    The guy told me that the one I got is special in that it allows connectors be inserted in both orientations.

    But what I know (as I said, not a networking guy), you can use both, "normal" cable as well as "cross" cable
    to connect DataPower boxes. DataPower interfaces will deal with both and make the connection.

     

    I doubt that direct connections can help to replace AO.

    What I did use the direct cable for is access backup.
    If I cannot ssh into one box mgmt0/mgmt1 interface, I can do so via tcpproxy on other device.
    So instead of "ssh dp1-l3" I can login additionally via "ssh -p 23  dp1-l2".
    And I do have tcpproxy "9091" that allows to login to WebGUI of "the other" box over port 9091 instead of 9090.

    xi52(config)# show tcpproxy 23

    tcpproxy: 23 [up]
    ------------
     admin-state enabled
     ip-address 0.0.0.0
     Service Priority normal
     Port Number 23
     Remote Host 192.168.0.2
     Remote Port 22


    xi52(config)#


    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/> <myFrameless/>

    I don't think the cross-over could work with AO.  I imagine I'd have to build a back end load balancer, but my guess is it would be much faster moving traffic from the DMZ to the Trusted appliances this way.   I probably wouldn't be able to scale it beyond 2 pair of appliances either.  Be easier to get a 10GB switch and keep AO.

    I'm curious, why did you create backup access this way?  I know this would be nice in the event an admin cert is screwed up on the WebGUI.  But SSH?

     

  • HermannSW
    HermannSW
    6019 Posts

    Re: Any limit on # of TPS that can be generated through DP in ideal conditions

    ‏2013-12-24T11:50:17Z  

    I don't think the cross-over could work with AO.  I imagine I'd have to build a back end load balancer, but my guess is it would be much faster moving traffic from the DMZ to the Trusted appliances this way.   I probably wouldn't be able to scale it beyond 2 pair of appliances either.  Be easier to get a 10GB switch and keep AO.

    I'm curious, why did you create backup access this way?  I know this would be nice in the event an admin cert is screwed up on the WebGUI.  But SSH?

     

    Hi Joe,

    > I'm curious, why did you create backup access this way? 
    > I know this would be nice in the event an admin cert is screwed up on the WebGUI.  But SSH?
    >
    although my serial console server does provide 16 ports I do not have enogh serial cables to connect all my DataPower boxes.

    As you can see only one of my two XI52s is connected via serial (bottom box in photo, most left cable).

    Sometimes when I do reconfigure the boxes IP assignments or have other network issue not allowing me to ssh into the top box directly, I can ssh into it via "-p 23" over the bottom box.


    Before my rack moved to new location in basement computing center I was short of connections. I did use the "-p 23" trick to login to laptops connected via direct link to DataPower boxes from remote. That allowed to do heavy load and performance tests just over that link and not affecting others systems in computing center.


    Next reconfiguration will happen when I will be back in office after vacation. A techsales colleague did send me a palette with six 9004s he has no use for (EOS in 3/2014) -- but I in Level3 support and my colleagues in Level2 support in Böblingen lab may still use them for dealing with PMRs of customers having bought extended support.


    Btw, the cable that goes to the box two below top XI52 (cannot be seen as it is mounted a bit deeper in the rack) connects my last still active 9002 box (for historical reasons and because I have the space)  ;-)

    xi50# show version

             Serial: 00406 20899
            Version: XI50.3.7.1.4
              Build: 165191
         Build Date: 2008/12/11 00:17:09
     Watchdog Build: XI50.3.7.1.4
     Installed DPOS: XI50.3.7.1.4
       Running DPOS: XI50.3.7.1.4
    XML accelerator: xg3
       Machine Type: n/a
         Model Type: n/a
     Coproc library: 1.2
                 MQ: 6.0.0-0
      WebSphere JMS: 1.2.3

    xi50#



    ˙˙˙ sɐɯʇsᴉɹɥƆ ʎɹɹǝW


    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/> <myFrameless/>
     

    Updated on 2013-12-24T11:59:00Z at 2013-12-24T11:59:00Z by HermannSW