Topic
  • 6 replies
  • Latest Post - ‏2016-11-28T10:07:14Z by Anurag_Mahajan
Anurag_Mahajan
Anurag_Mahajan
6 Posts

Pinned topic Fix Connection issues to MQ

‏2016-11-21T11:02:04Z |

Hi

In our configuration there is one channel to which has all our test environments are connected to and i can see around 48000 + connections established when i run below command. The CPU utilization is too high.
echo "display conn(*) where (channel eq <Channel_Name>)" | runmqsc QM_Name | wc -l
48133


Now ,there are some connections which I am trying to kill using stop conn command but seems they getting re-created everytime. Also, looks like developers have used autoreconnect option in the code which is not allowing me to kill the connections.
>pr<property name="clientReconnectOptions"><util:constant static-field="com.ibm.msg.client.wmq.WMQConstants.WMQ_CLIENT_RECONNECT"/>    /property>

I do not want to make channel itself down, I have below queries
1. What would be the best approach to reduce these connections if they no longer needed ?
2. What if auto reconnection is disabled?Is it recommended? How to do that in application?

Please suggest as my MQ server's CPU is getting too high may be because of this.

 

Thanks

Anurag

  • MoragH
    MoragH
    131 Posts

    Re: Fix Connection issues to MQ

    ‏2016-11-22T05:11:23Z  

    If you have client applications which are connecting to this queue manager over the SVRCONN channel in question and you want to stop them from doing so anymore, you should disallow them from being able to connect in. Read up about CHLAUTH (channel authentication) for how to configure rules on your queue manager for which users/IP addresses/digital certificates are allowed to make connections. If you do this, then once you stop the connection and it attempts to auto-reconnect, it won't be able to anymore.

    Given that they are currently using this queue manager to connect to - are they going to be surprised when you put something in place that stops them from being able to anymore? There should be some discussion with these application owners as well to find out why they are connecting to your queue manager if you don't want them to anymore.

    Alternatively, your question might be asking how to reduce the number of connections any one application makes, and not allow them to make 100s or 1000s of connections? If that is what you are suffering from, I suggest you read up about MAXINST and MAXINSTC.

    If you can be more specific about your problem, I can comment further.

    Cheers
    Morag

  • Anurag_Mahajan
    Anurag_Mahajan
    6 Posts

    Re: Fix Connection issues to MQ

    ‏2016-11-23T07:50:16Z  
    • MoragH
    • ‏2016-11-22T05:11:23Z

    If you have client applications which are connecting to this queue manager over the SVRCONN channel in question and you want to stop them from doing so anymore, you should disallow them from being able to connect in. Read up about CHLAUTH (channel authentication) for how to configure rules on your queue manager for which users/IP addresses/digital certificates are allowed to make connections. If you do this, then once you stop the connection and it attempts to auto-reconnect, it won't be able to anymore.

    Given that they are currently using this queue manager to connect to - are they going to be surprised when you put something in place that stops them from being able to anymore? There should be some discussion with these application owners as well to find out why they are connecting to your queue manager if you don't want them to anymore.

    Alternatively, your question might be asking how to reduce the number of connections any one application makes, and not allow them to make 100s or 1000s of connections? If that is what you are suffering from, I suggest you read up about MAXINST and MAXINSTC.

    If you can be more specific about your problem, I can comment further.

    Cheers
    Morag

    Thanks for your response Morag.
    Actually the reason I want to do this is because we have new MQ servers machine now wherein we have setup everything (QM,Channels etc). Now we do not want to use our old MQ server machine & our client applications should connect to the newer MQ server rather than old one.

    I have changed the IP address , QM name , Channel name in the application's configuration so that it points to new MQ server. But I can see there are still few connections in the old MQ server which are referring to application server where our application is installed. I am wondering why this behavior exists even i have change application's configuration. So i thought of killing/stopping the connections in old MQ server since they are no longer needed but it never happen.

    Example:
     1 : display conn(*) type(CONN) where(CONNAME EQ '<IP_Address>')
    AMQ8276: Display Connection details.
       CONN(3E6C2D58099F4324)
       EXTCONN(414D51434D5147454E30315420202020)
       TYPE(CONN)
       CONNAME(<IP_Address>)

    The above output is from our old MQ server, but as i mention that I have changed the <IP_Address>,QM , channels in the application's configuration pointing to newer MQ server but still I can see the connection in old MQ server. Moreover now if i stop the above connection with ID 3E6C2D58099F4324, it says "Connection identifier not found."

    stop conn(3E6C2D58099F4324)
         1 : stop conn(3E6C2D58099F4324)
    AMQ8461: Connection identifier not found.

     

     

  • MoragH
    MoragH
    131 Posts

    Re: Fix Connection issues to MQ

    ‏2016-11-23T08:55:56Z  

    Thanks for your response Morag.
    Actually the reason I want to do this is because we have new MQ servers machine now wherein we have setup everything (QM,Channels etc). Now we do not want to use our old MQ server machine & our client applications should connect to the newer MQ server rather than old one.

    I have changed the IP address , QM name , Channel name in the application's configuration so that it points to new MQ server. But I can see there are still few connections in the old MQ server which are referring to application server where our application is installed. I am wondering why this behavior exists even i have change application's configuration. So i thought of killing/stopping the connections in old MQ server since they are no longer needed but it never happen.

    Example:
     1 : display conn(*) type(CONN) where(CONNAME EQ '<IP_Address>')
    AMQ8276: Display Connection details.
       CONN(3E6C2D58099F4324)
       EXTCONN(414D51434D5147454E30315420202020)
       TYPE(CONN)
       CONNAME(<IP_Address>)

    The above output is from our old MQ server, but as i mention that I have changed the <IP_Address>,QM , channels in the application's configuration pointing to newer MQ server but still I can see the connection in old MQ server. Moreover now if i stop the above connection with ID 3E6C2D58099F4324, it says "Connection identifier not found."

    stop conn(3E6C2D58099F4324)
         1 : stop conn(3E6C2D58099F4324)
    AMQ8461: Connection identifier not found.

     

     

    In your initial post you suggested that you didn't want to stop the channel, and you also stated that you had 48000 + connections established. If all the connections are supposed to be using the new server and have had their configuration changed, what is the harm in stopping the channel?

    You could, rather than using the STOP CONN command, use the STOP CHANNEL command with the IP address, so that it will only stop the connections from that IP address. The IP address you use in the command should be the one seen when you view DISPLAY CHSTATUS. That way you will only affect the channel instances coming from certain client machines. You will affect ALL the channel instances coming from any one client machine by doing this, but if nothing is supposed to be using this server, then surely you need to stop them all anyway?

    Cheers
    Morag

  • Anurag_Mahajan
    Anurag_Mahajan
    6 Posts

    Re: Fix Connection issues to MQ

    ‏2016-11-24T08:05:59Z  
    • MoragH
    • ‏2016-11-23T08:55:56Z

    In your initial post you suggested that you didn't want to stop the channel, and you also stated that you had 48000 + connections established. If all the connections are supposed to be using the new server and have had their configuration changed, what is the harm in stopping the channel?

    You could, rather than using the STOP CONN command, use the STOP CHANNEL command with the IP address, so that it will only stop the connections from that IP address. The IP address you use in the command should be the one seen when you view DISPLAY CHSTATUS. That way you will only affect the channel instances coming from certain client machines. You will affect ALL the channel instances coming from any one client machine by doing this, but if nothing is supposed to be using this server, then surely you need to stop them all anyway?

    Cheers
    Morag

    Thanks...As I have mentioned that there are several test environments(having separate server/IP address) which are connected to one single Channel. I have to stop the connections one by one environmentwise. That is the reason i cannot bring entire channel down.

    I think the other option you have suggested to STOP Channel with IP address should work in my case. I will try and let you know. Thanks a lot!

    One last question, what will be the affect of disabling Auto Reconnect option on our client applications? How / when application will pick the message from queue in that case?

    I really don't understand, why there are so many connections to QM from single IP address. Is it because of auto reconnect? What measures can be taken to avoid so many connections. Because i do not want this situation on our new MQ servers.

    IP -> number of connections

    <x.x.x.a1>  -> 371

    <x.x.x.a2>  -> 334


    After doubling the memory of server , doubles the amount of channel connections through that single channel which we are using for all environments. Earlier it was 800 and now its 1550 connections to this channel.

     

    Thanks
    Anurag

    Updated on 2016-11-24T08:42:24Z at 2016-11-24T08:42:24Z by Anurag_Mahajan
  • MoragH
    MoragH
    131 Posts

    Re: Fix Connection issues to MQ

    ‏2016-11-24T08:57:49Z  

    Thanks...As I have mentioned that there are several test environments(having separate server/IP address) which are connected to one single Channel. I have to stop the connections one by one environmentwise. That is the reason i cannot bring entire channel down.

    I think the other option you have suggested to STOP Channel with IP address should work in my case. I will try and let you know. Thanks a lot!

    One last question, what will be the affect of disabling Auto Reconnect option on our client applications? How / when application will pick the message from queue in that case?

    I really don't understand, why there are so many connections to QM from single IP address. Is it because of auto reconnect? What measures can be taken to avoid so many connections. Because i do not want this situation on our new MQ servers.

    IP -> number of connections

    <x.x.x.a1>  -> 371

    <x.x.x.a2>  -> 334


    After doubling the memory of server , doubles the amount of channel connections through that single channel which we are using for all environments. Earlier it was 800 and now its 1550 connections to this channel.

     

    Thanks
    Anurag

    I suspect one of the reasons why there might be lots of connections from one single IP address could be due to a badly written application that doesn't disconnect, but then makes another connection later. These connections build up and up until eventually you run out of something (sockets, file handles, slots in the status table, memory). You'll have to get your application writers to look into it. You can protect a queue manager from these types of badly written applications by using MAXINST and MAXINSTC. Read https://www.ibm.com/developerworks/community/blogs/aimsupport/entry/avoiding_runaway_numbers_of_mq_channels

    If you disable Auto Reconnect, then the application will simply have to make a deliberate connection again. Whether the application is coded to do this when it receives any kind of error, like a 2009 MQRC_BROKEN_CONNECTION for example is a question only the application writers can answer for you - I can't. If they can't handle such errors then the application would presumably make a new connection next time it is started up again.

    Cheers
    Morag

  • Anurag_Mahajan
    Anurag_Mahajan
    6 Posts

    Re: Fix Connection issues to MQ

    ‏2016-11-28T10:07:14Z  
    • MoragH
    • ‏2016-11-24T08:57:49Z

    I suspect one of the reasons why there might be lots of connections from one single IP address could be due to a badly written application that doesn't disconnect, but then makes another connection later. These connections build up and up until eventually you run out of something (sockets, file handles, slots in the status table, memory). You'll have to get your application writers to look into it. You can protect a queue manager from these types of badly written applications by using MAXINST and MAXINSTC. Read https://www.ibm.com/developerworks/community/blogs/aimsupport/entry/avoiding_runaway_numbers_of_mq_channels

    If you disable Auto Reconnect, then the application will simply have to make a deliberate connection again. Whether the application is coded to do this when it receives any kind of error, like a 2009 MQRC_BROKEN_CONNECTION for example is a question only the application writers can answer for you - I can't. If they can't handle such errors then the application would presumably make a new connection next time it is started up again.

    Cheers
    Morag

    Thank you so much Morag. I am very much convinced with your replies. Thanks again! Have a great day!

     

    Thanks

    Anurag