IBM Support

PowerVC 1.4.X - mariadb fails as the maximum sql sessions reached

Troubleshooting


Problem

Depending on the environment, PowerVC can hit the maximum allowed SQL connections.

Symptom

Neutron agents are down and all hosts are in Attention mode.

Diagnosing The Problem

Check var\log\neutron\server.log for any messages like:
2020-06-11 11:49:24.989 141799 WARNING neutron.db.agents_db [req-f4f5f0ee-4256-4a96-ad7f-17dd7fdecd67 - - - - -] Agent healthcheck: found 96 dead agents out of 96
Also var\log\neutron\server.log-DATE.gz:

2020-06-11 03:17:38.820 141776 ERROR oslo_messaging.rpc.server  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 437, in connect

2020-06-11 03:17:38.820 141776 ERROR oslo_messaging.rpc.server    return self.dbapi.connect(*cargs, **cparams)

2020-06-11 03:17:38.820 141776 ERROR oslo_messaging.rpc.server  File "/usr/lib64/python2.7/site-packages/MySQLdb/__init__.py", line 81, in Connect

2020-06-11 03:17:38.820 141776 ERROR oslo_messaging.rpc.server    return Connection(*args, **kwargs)

2020-06-11 03:17:38.820 141776 ERROR oslo_messaging.rpc.server  File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 193, in __init__

2020-06-11 03:17:38.820 141776 ERROR oslo_messaging.rpc.server    super(Connection, self).__init__(*args, **kwargs2)

2020-06-11 03:17:38.820 141776 ERROR oslo_messaging.rpc.server OperationalError: (_mysql_exceptions.OperationalError) (1040, 'Too many connections') (Background on this error at: http://sqlalche.me/e/e3q8)

Resolving The Problem

Restart PowerVC to resume normal operation.
Take backup of /etc/my.cnf.d/server.cnf and /etc/my.cnf
Update the below conf properties in both /etc/my.cnf.d/server.cnf and /etc/my.cnf under mysqld section:
max_connections = 50000
table_open_cache = 21000
Add below in /etc/systemd/system/mysqld.service.d/override_pvc.conf

TasksMax=55000
LimitNPROC=55000
Then run:
systemctl daemon-reload
powervc-services stop
powervc-services start

Document Location

Worldwide

[{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SSGRZY","label":"IBM Cloud PowerVC Manager"},"ARM Category":[{"code":"a8m50000000L0jDAAS","label":"PowerVC"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Version(s)","Line of Business":{"code":"LOB57","label":"Power"}}]

Document Information

Modified date:
29 June 2020

UID

ibm16241516

Page Feedback