Comments (10)
  • Add a Comment
  • Edit
  • More Actions v
  • Quarantine this Entry

1 shrik commented Permalink

Peter, thanks for another great post. Very informative as usual! <br /> I have a question on the need for "MQRC_PUT_INHIBITED handling" on the application side. <br /> IF the request queue is MQ clustered and CLWLUSEQ is set to 'any", wouldn't MQ clustering automatically take care of ignoring the traffic to the 'disabled' queue ?

2 shrik commented Permalink

And obviously for the existing old connection to detect the disabled queue on fly, the MQopen should have specified Bind-Not-Fixed.

3 Peter Broadhurst commented Permalink

Hi Raki, you're quite right on the clustered put - assuming there's a cluster defined. Even if CLWLUSEQ(QMGR) is specified, messages will be routed to a remote cluster queue instance if the local one is PUT(DISABLED). <br /> Your point on DEFBIND(NOTFIXED) is a good one too. While in the 'scalable topology' article MQSC samples I specify this, there's nothing that discusses it in the articles themselves. <br /> You'll see I've put these into updated text for others.

4 Peter Broadhurst commented Permalink

Actually, while I was updating the text it struck me I need to think through this a bit more before updating the post. If you don't get MQRC_PUT_INHIBITED (which as you point out in a cluster you won't), then your applications will continue to _connect_ to the queue manager that is being quiesced. So that's where your reply queues are going to be hosted. So it's actually the MQRC_PUT_INHIBITED behavior you want here so the apps start connecting to the other queue manager and waiting there for their replies, hence I think you actually want the exception handling I mention, along with a QALIAS for your request on the QMGRs you connect to, to let you explicitly PUT(DISABLE) each qmgr separately.

5 shrik commented Permalink

I would like to make one distinction here is that the app could 'connect' to any of the available (by round-robin etc) clustered gateways BUT the message sent via one Gateway could land up in a different Gateway. Meaning the destinationQM could be different from the connectQM. In the connect- call, if destinationQM is empty, "only" then the clustered-load-balancing kicks in. In samples, for REQUESTs, I see destinationQM is always empty, so it always load-balances. For replies, as we earlier discussed, the destinationQM is the replyToQM picked from request, so the reply always reaches the right QM. The REPLY queue isnt in cluster and thats perfectly ok. <div>&nbsp;</div> here is the notes I have (as I understand how simple request, reply is working): <br /> "simple req-rep" working:--&gt; <br /> Message sent via QCF connection to (assume connected to GW1) REQUEST queue, same connection waits for response, using time-out- sets replyToQmgr (so sets to GW1- to the same as connected QMgr) <br /> the REQUEST queue is in cluster, so request could reach to GW2, if 'connection' was to GW1 <br /> MDB pulls from the REQUEST queue (GW1 or GW2- we have 2 MDBs), connects using another connection (this could connect to GW1 or GW2, doesnt matter) QCF posts to REPLY queue setting same destQmgr as replyQmgr <br /> REPLY queue is not in cluster, but it reaches the Gateway the initial requesting 'connection' was connected to, and waiting <br /> The waiting connection from request pulls the reply back from REPLY queue from same GW (GW1 in this case) <div>&nbsp;</div> Back to the discussion: <br /> If destinationQMgr is empty for "request" calls, couple of other things needs to occur for load-balancing. <br /> Load balancing totally handled by MQ gateway QM. The effective bind needs to be 'not-fixed'- meaning app could set 'not-fixed' or use queue-bind option, and then queue needs to have not-fixed. <br /> The other thing that needs to happen is the effective CLWLUSEQ needs to be either 'any'. If QMGR is set on REQUEST queue, the GW-QM attribute needs to be set to 'any'. So no issue occurs at all if queue is inhibited. Nothing goes there. <div>&nbsp;</div> From the tests I have run, what I noticed is that if app uses bind-as-queue and queue property has changed to 'not-fixed' after, in that situation the app will continue to send to inhibited queue and it may fail. App has to reconnect to start sending requests as 'not-fixed' <br /> Once app already connected as 'not-fixed' long before and after that the queue can be put-inhibited on fly, and for every message cluster checks if queue is disabled and doesnt send anything there if it is (Again, connectQM could be different as destinationQM, if in request destinationQM is empty). For remoteQmgr i believe it checks if its suspended or if cluster-rcvr is having an issue. <div>&nbsp;</div> I agree a bit with setting queue to inhibit until the replies are received or wait till time-out &amp; then bring the GW down. Not sure how easy it is to implement that way- getting all that logic onto start/stop scripts. Its only issue with some non-persistent replies that may be lost when QM restarts. if non-persistent, the app should retry any way (by design). If persistent, the reply is waiting on the XMITq somewhere. If app had timed out, by then it'll retry any way. I dont see why a req-rep has to be persistent but just in case. <div>&nbsp;</div> Thanks a lot for your help and responses. Really appreciate

6 shrik commented Permalink

actually i figured now after posting above reply (sorry about that) that your concern is not where the app connects to (as cluster routing avoids disabled queues) but more with apps connecting and awaiting on replies. you said: "when your applications will continue to _connect_ to the queue manager that is being quiesced" <div>&nbsp;</div> questions are: <br /> 1) does QM during quiesce let applications connect ? <br /> 2) is it an issue if app has to disconnect w/o waiting for reply ? (wouldnt it retry ?) <br /> 3) i agree if thats the case put-inhibit needs to be checked for, and put-disable on QM side before bringing the gateway down <br /> 4) would advanced req-rep have this issue at all, considering it has 2 MDB listeners for each gateway ?

7 Peter Broadhurst commented Permalink

Re (1) If you quiesce the whole queue manager, or stop the SVRCONN channel, then the apps will receive exceptions from both connect (good) &amp; receive (which is too late as they've already successfully sent the request). <br /> Re (2/3) my experience is that the retry is generally pushed back to the end-user in a sync. request/reply scenario, when there's an exception/timeout getting the reply. For example, if someone has pushed a button on a web page, that results in a app-server web container doing a sync. request/reply exchange, then that human would get and exception and have to retry (noting that they have no way of knowing whether the original request was a success or failure). The suggestion was to find a way to avoid this via operational controls for planned maintenance. <br /> Re (4) it would indeed. For the 'advanced' version of sync request/reply, you could put-disable the reply queues on the queue manager being quiesced, ensure the clients drain the reply queue and disconnect (via a SVRCONN stop) before the qmgr is stopped (to let the xmitqs drain). <div>&nbsp;</div> Even better than (4/'advanced' sync request/reply) would be two-way asynchronous messaging. e.g. decouple the requests from the replies, so there is never a thread 'hung' waiting for a response/error/timeout for one specific message. However, using two-way asynchronous messaging is about designing applications to exploit MQ's full potential, rather than something you can easily retrofit to an existing sync. request/reply scenario..

8 shrik commented Permalink

Peter - we discussed about put-inhibiting queues before the Gateway QM is brought down. I wanted to ask if instead of that what if the Gateway is suspended out of the MQ-cluster ?

9 shrik commented Permalink

Of all the different approaches listed in this article, if possible, I would like to hear your thoughts on comparing one other thing please i.e. the ability to scale the # of MQ gateways for 'existing' applications to accomodate future traffic increase. <div>&nbsp;</div> Currently say we have 2 gateways. All applications connect using 'code stub' approach to the 2 gateways and all is fine. I see this approach is a bit tightly coupled (unless there is some properties file kind of thing to seamlessly add or remove extra gateways in the shared library used by codestub) to the gateways. The problem is say the traffic quadrupled. Extra system resources on gateays is not sufficient. I believe in that situation, ccdt may be better option to add extra gateways w/o requiring any changes to the app side. I agree ccdt means: no XA support, may be a performance dent, and have issue with all connections not failing back. But if we have 4 gateways I think we can live with 3 gateways sharing all traffic using CLNTWGHT and 1 gateway is idle. <div>&nbsp;</div> What are your thoughts on this ?

10 Peter Broadhurst commented Permalink

I actually the challenge of adding extra queue managers between producers and consumers is more difficult on the consumer side than the producer side - and the consumer side really isn't discussed in this blog post at all. Putting more qmgrs with a 'sending gateway' role doesn't really help the overall throughput, if you don't have more qmgrs in the 'receiving gateway' role (noting this might be the same queue managers). CCDTs with multiple entries don't help on the consuming side, as any ambiguity in which QMGRs your consumer instances connect to could lead to stranded messages. That's really why I think two is the magic number here (Figure 4 in Part 1 of the 'MQ Scalable Topologies' developerWorks series). Personally, I also think on the sending side limiting the number of QMGRs each app instance connects to is good for problem determination, as you're limited to which QMGRs each app instance will be sending messages to. So my feeling is the best option is to pin each sending/consuming app instance to two QMGRs, but to virtualize which qmgrs those are using DNS records, a VIP, or CCDTs (for code-stub, use single matching entry CCDTs).