Meaning#

The number of connected threads on a MySQL instance is approaching max_connections. New connections risk being refused with ER_CON_COUNT_ERROR.

Fires when:

max by (pod) (
  mysql_global_status_threads_connected{namespace="safetywing-<env>-infra"}
  / mysql_global_variables_max_connections{namespace="safetywing-<env>-infra"}
) > <ratio>

for: 10m, severity ticket, tier component.

Impact#

  • Once max_connections is hit, new clients get “Too many connections” and application requests fail.
  • Often a symptom of leaked/unclosed connections, an oversized client pool, or slow queries holding connections open.

Diagnosis#

kubectl config use-context hetzner
kubectl get mysqlcluster -n safetywing-<env>-infra
kubectl moco status -n safetywing-<env>-infra <cluster>

# Open a mysql shell to the primary and inspect live connections
kubectl moco mysql -n safetywing-<env>-infra -u moco-admin <cluster> -- \
  -e "SHOW PROCESSLIST;"
kubectl moco mysql -n safetywing-<env>-infra -u moco-admin <cluster> -- \
  -e "SHOW STATUS LIKE 'Threads_connected'; SHOW VARIABLES LIKE 'max_connections';"

# Group connections by host/user to find the offender
kubectl moco mysql -n safetywing-<env>-infra -u moco-admin <cluster> -- \
  -e "SELECT user, host, count(*) FROM information_schema.processlist GROUP BY user, host ORDER BY 3 DESC;"
# Current usage ratio per pod
max by (pod) (
  mysql_global_status_threads_connected{namespace="safetywing-<env>-infra"}
  / mysql_global_variables_max_connections{namespace="safetywing-<env>-infra"}
)

Mitigation#

  1. Identify the offending client(s) from SHOW PROCESSLIST / the grouped query above — usually one service with a misconfigured pool or leaked connections.
  2. Fix at the source: scale down the offending workload, tune its connection pool max size, or restart it to drop leaked connections.
  3. Kill stuck/sleeping connections if needed:
    KILL <process_id>;
  4. If demand is legitimate, raise max_connections in the MySQLCluster spec (MOCO reconciles it into the instances’ my.cnf):
    spec:
  mysqlConfigMapName: <name>   # or set under spec.podTemplate config
# in the referenced ConfigMap:
#   max_connections = <n>
    Ensure the instance has memory headroom — each connection consumes per-thread buffers.

References#

Backward MySQL MysqlDiskFillingUp Forward