MySQL on SafetyWing Runbooks

MysqlConnectionsSaturated

Mon, 01 Jan 0001 00:00:00 +0000

Meaning#

The number of connected threads on a MySQL instance is approaching max_connections. New connections risk being refused with ER_CON_COUNT_ERROR.

Fires when:

max by (pod) (
 mysql_global_status_threads_connected{namespace="safetywing-<env>-infra"}
 / mysql_global_variables_max_connections{namespace="safetywing-<env>-infra"}
) > <ratio>

for: 10m, severity ticket, tier component.

Impact#

Once max_connections is hit, new clients get “Too many connections” and application requests fail.
Often a symptom of leaked/unclosed connections, an oversized client pool, or slow queries holding connections open.

Diagnosis#

kubectl config use-context hetzner
kubectl get mysqlcluster -n safetywing-<env>-infra
kubectl moco status -n safetywing-<env>-infra <cluster>

# Open a mysql shell to the primary and inspect live connections
kubectl moco mysql -n safetywing-<env>-infra -u moco-admin <cluster> -- \
 -e "SHOW PROCESSLIST;"
kubectl moco mysql -n safetywing-<env>-infra -u moco-admin <cluster> -- \
 -e "SHOW STATUS LIKE 'Threads_connected'; SHOW VARIABLES LIKE 'max_connections';"

# Group connections by host/user to find the offender
kubectl moco mysql -n safetywing-<env>-infra -u moco-admin <cluster> -- \
 -e "SELECT user, host, count(*) FROM information_schema.processlist GROUP BY user, host ORDER BY 3 DESC;"

# Current usage ratio per pod
max by (pod) (
 mysql_global_status_threads_connected{namespace="safetywing-<env>-infra"}
 / mysql_global_variables_max_connections{namespace="safetywing-<env>-infra"}
)

Mitigation#

Identify the offending client(s) from SHOW PROCESSLIST / the grouped query above — usually one service with a misconfigured pool or leaked connections.
Fix at the source: scale down the offending workload, tune its connection pool max size, or restart it to drop leaked connections.
Kill stuck/sleeping connections if needed:
```
KILL <process_id>;
```
If demand is legitimate, raise max_connections in the MySQLCluster spec (MOCO reconciles it into the instances’ my.cnf):
```
spec:
 mysqlConfigMapName: <name>  # or set under spec.podTemplate config
# in the referenced ConfigMap:
# max_connections = <n>
```
Ensure the instance has memory headroom — each connection consumes per-thread buffers.

References#

MysqlDiskFillingUp

Mon, 01 Jan 0001 00:00:00 +0000

Meaning#

A mysql-data-* PersistentVolumeClaim is running low on free space. If it fills completely, mysqld will fail writes and may refuse to start.

Fires when:

min by (persistentvolumeclaim) (
 kubelet_volume_stats_available_bytes{persistentvolumeclaim=~"mysql-data-.*"}
 / kubelet_volume_stats_capacity_bytes{persistentvolumeclaim=~"mysql-data-.*"}
) < (1 - <ratio>)

for: 15m, severity ticket, tier component.

Impact#

A full data volume causes write errors and can crash the instance (MysqlInstanceDown).
Common culprits: accumulated binary logs, an oversized dataset, or relay logs/temp files on a lagging replica.

Diagnosis#

kubectl config use-context hetzner
kubectl get mysqlcluster -n safetywing-<env>-infra
kubectl get pvc -n safetywing-<env>-infra

# Inspect on-disk usage from inside the mysqld container
kubectl exec -n safetywing-<env>-infra <pod> -c mysqld -- df -h /var/lib/mysql
kubectl exec -n safetywing-<env>-infra <pod> -c mysqld -- \
 sh -c 'du -sh /var/lib/mysql/* | sort -h | tail -20'

# Binary log inventory
kubectl moco mysql -n safetywing-<env>-infra -u moco-admin <cluster> -- \
 -e "SHOW BINARY LOGS;"
# Largest tables
kubectl moco mysql -n safetywing-<env>-infra -u moco-admin <cluster> -- \
 -e "SELECT table_schema, table_name, ROUND((data_length+index_length)/1024/1024) AS mb FROM information_schema.tables ORDER BY mb DESC LIMIT 20;"

# Fraction free per PVC
min by (persistentvolumeclaim) (
 kubelet_volume_stats_available_bytes{persistentvolumeclaim=~"mysql-data-.*"}
 / kubelet_volume_stats_capacity_bytes{persistentvolumeclaim=~"mysql-data-.*"}
)

Mitigation#

Expand the PVC (preferred — the StorageClass supports volume expansion). Increase the volume request in the MySQLCluster volumeClaimTemplates; MOCO/Kubernetes resizes the PVC online:
```
spec:
 volumeClaimTemplates:
 - metadata:
 name: mysql-data
 spec:
 resources:
 requests:
 storage: <larger-size>
```
Apply via GitOps, then confirm with kubectl get pvc -n safetywing-<env>-infra.
Prune binary logs if they dominate usage (only purge logs already applied by all replicas):
```
PURGE BINARY LOGS BEFORE NOW() - INTERVAL 1 DAY;
```
For a durable fix, tune binlog_expire_logs_seconds in the cluster MySQL config.
Replica behind: a lagging replica accumulates relay logs — clearing the lag (MysqlReplicationLagHigh) lets them be purged.
Reclaim table space: drop unused data or run OPTIMIZE TABLE on bloated tables (note: requires temporary extra space, so resize first if very full).

References#

MysqlInstanceDown

Mon, 01 Jan 0001 00:00:00 +0000

Meaning#

A MySQL instance’s mysqld_exporter sidecar reports the server unreachable, so the mysqld process is down or not accepting connections.

Fires when:

max by (pod) (mysql_up{namespace="safetywing-<env>-infra"}) == 0

for: 5m, severity page, tier component.

Impact#

The affected instance serves no queries.
If the primary is down, MOCO must fail over before writes can resume; expect a short write outage.
If a replica is down, read capacity and replication redundancy are reduced.

Diagnosis#

kubectl config use-context hetzner

# Cluster + member roles (which pod is primary vs replica)
kubectl get mysqlcluster -n safetywing-<env>-infra
kubectl moco status -n safetywing-<env>-infra <cluster>

# Pod state and recent events (OOMKill, evictions, probe failures)
kubectl get pods -n safetywing-<env>-infra -l app.kubernetes.io/name=mysql -o wide
kubectl describe pod -n safetywing-<env>-infra <pod>

# Container logs — mysqld, the MOCO agent, and the exporter
kubectl logs -n safetywing-<env>-infra <pod> -c mysqld --tail=200
kubectl logs -n safetywing-<env>-infra <pod> -c agent --tail=200
kubectl logs -n safetywing-<env>-infra <pod> -c mysqld-exporter --tail=100

# Confirm which pod(s) are down
max by (pod) (mysql_up{namespace="safetywing-<env>-infra"})

Mitigation#

Check pod events for the root cause: OOMKilled (raise memory limits in the MySQLCluster .spec.podTemplate), node pressure/eviction, or failed PVC mount.
If disk is full, mysqld will refuse to start — see MysqlDiskFillingUp and expand the PVC first.
If the process crashed but the pod is healthy, restart it:
```
kubectl delete pod -n safetywing-<env>-infra <pod>
```
MOCO recreates the pod; verify it rejoins via kubectl moco status.
If the primary is down and not recovering, let MOCO fail over to a healthy replica; confirm a new primary was elected in kubectl moco status. Investigate the old primary before reintroducing it.

If MOCO cannot reconcile, inspect the operator:

kubectl logs -n moco-system deploy/moco-controller-manager --tail=200

References#

MysqlReplicationLagHigh

Mon, 01 Jan 0001 00:00:00 +0000

Meaning#

A MySQL replica is applying the primary’s binlog stream slower than it is produced, so its Seconds_Behind_Master has exceeded the threshold.

Fires when:

max by (pod) (
 mysql_slave_status_seconds_behind_master{namespace="safetywing-<env>-infra"}
) > <seconds>

for: 10m, severity ticket, tier component.

Impact#

Reads served by the lagging replica return stale data.
A failover to a lagging replica could lose recent writes (MOCO uses semi-sync, which bounds but does not eliminate this risk under degraded conditions).

Diagnosis#

kubectl config use-context hetzner
kubectl get mysqlcluster -n safetywing-<env>-infra
kubectl moco status -n safetywing-<env>-infra <cluster>

# Replication status on the lagging replica
kubectl moco mysql -n safetywing-<env>-infra -u moco-admin --index <n> <cluster> -- \
 -e "SHOW REPLICA STATUS\G"

# Container logs of the replica (mysqld + agent)
kubectl logs -n safetywing-<env>-infra <replica-pod> -c mysqld --tail=200
kubectl logs -n safetywing-<env>-infra <replica-pod> -c agent --tail=200

# Long-running transactions on the primary that bloat the binlog
kubectl moco mysql -n safetywing-<env>-infra -u moco-admin <cluster> -- \
 -e "SELECT * FROM information_schema.innodb_trx ORDER BY trx_started LIMIT 10;"

# Lag per pod
max by (pod) (mysql_slave_status_seconds_behind_master{namespace="safetywing-<env>-infra"})

In SHOW REPLICA STATUS, check Replica_IO_Running / Replica_SQL_Running (both should be Yes), Last_Error, and Seconds_Behind_Master.