https://runbooks.safetywing.dev/runbooks/Recent content in Runbooks on SafetyWing RunbooksHugoen-ushttps://runbooks.safetywing.dev/runbooks/catalog/Mon, 01 Jan 0001 00:00:00 +0000https://runbooks.safetywing.dev/runbooks/catalog/<h1 id="alert-catalog">Alert Catalog<a class="anchor" href="#alert-catalog">#</a></h1> <p>Every alert evaluated across SafetyWing clusters — <strong>29 custom</strong> (component / environment / platform tiers, owned by us) and <strong>133 stock</strong> (kube-prometheus-stack defaults). Custom alerts link to the runbook on this site; stock alerts link to the upstream <a href="https://runbooks.prometheus-operator.dev/">prometheus-operator runbooks</a>.</p> <blockquote class='book-hint '> <p>Generated from the live hetzner rule set + the infra-charts/cluster-monitors sources. Stock alerts are identical across clusters; custom alerts deploy per environment/cluster where the chart is enabled.</p> </blockquote><h2 id="safetywing-custom-alerts">SafetyWing custom alerts<a class="anchor" href="#safetywing-custom-alerts">#</a></h2> <h3 id="kafka--component-tier">Kafka <small>(component tier)</small><a class="anchor" href="#kafka--component-tier">#</a></h3> <table> <thead> <tr> <th>Alert</th> <th>Severity</th> <th>Runbook</th> </tr> </thead> <tbody> <tr> <td>KafkaOfflinePartitions</td> <td><code>page</code></td> <td><a href="https://runbooks.safetywing.dev/runbooks/kafka/kafkaofflinepartitions/">runbook</a></td> </tr> <tr> <td>KafkaNoActiveController</td> <td><code>page</code></td> <td><a href="https://runbooks.safetywing.dev/runbooks/kafka/kafkanoactivecontroller/">runbook</a></td> </tr> <tr> <td>KafkaUnderReplicatedPartitions</td> <td><code>ticket</code></td> <td><a href="https://runbooks.safetywing.dev/runbooks/kafka/kafkaunderreplicatedpartitions/">runbook</a></td> </tr> <tr> <td>KafkaConsumerGroupLagHigh</td> <td><code>ticket</code></td> <td><a href="https://runbooks.safetywing.dev/runbooks/kafka/kafkaconsumergrouplaghigh/">runbook</a></td> </tr> </tbody> </table> <h3 id="kafka-connect--component-tier">Kafka Connect <small>(component tier)</small><a class="anchor" href="#kafka-connect--component-tier">#</a></h3> <table> <thead> <tr> <th>Alert</th> <th>Severity</th> <th>Runbook</th> </tr> </thead> <tbody> <tr> <td>KafkaConnectFailedTasks</td> <td><code>page</code></td> <td><a href="https://runbooks.safetywing.dev/runbooks/kafka-connect/kafkaconnectfailedtasks/">runbook</a></td> </tr> <tr> <td>KafkaConnectWorkersDown</td> <td><code>page</code></td> <td><a href="https://runbooks.safetywing.dev/runbooks/kafka-connect/kafkaconnectworkersdown/">runbook</a></td> </tr> <tr> <td>KafkaConnectNoConnectors</td> <td><code>ticket</code></td> <td><a href="https://runbooks.safetywing.dev/runbooks/kafka-connect/kafkaconnectnoconnectors/">runbook</a></td> </tr> </tbody> </table> <h3 id="mysql--component-tier">MySQL <small>(component tier)</small><a class="anchor" href="#mysql--component-tier">#</a></h3> <table> <thead> <tr> <th>Alert</th> <th>Severity</th> <th>Runbook</th> </tr> </thead> <tbody> <tr> <td>MysqlInstanceDown</td> <td><code>page</code></td> <td><a href="https://runbooks.safetywing.dev/runbooks/mysql/mysqlinstancedown/">runbook</a></td> </tr> <tr> <td>MysqlConnectionsSaturated</td> <td><code>ticket</code></td> <td><a href="https://runbooks.safetywing.dev/runbooks/mysql/mysqlconnectionssaturated/">runbook</a></td> </tr> <tr> <td>MysqlReplicationLagHigh</td> <td><code>ticket</code></td> <td><a href="https://runbooks.safetywing.dev/runbooks/mysql/mysqlreplicationlaghigh/">runbook</a></td> </tr> <tr> <td>MysqlDiskFillingUp</td> <td><code>ticket</code></td> <td><a href="https://runbooks.safetywing.dev/runbooks/mysql/mysqldiskfillingup/">runbook</a></td> </tr> </tbody> </table> <h3 id="rabbitmq--component-tier">RabbitMQ <small>(component tier)</small><a class="anchor" href="#rabbitmq--component-tier">#</a></h3> <table> <thead> <tr> <th>Alert</th> <th>Severity</th> <th>Runbook</th> </tr> </thead> <tbody> <tr> <td>RabbitmqNodeDown</td> <td><code>page</code></td> <td><a href="https://runbooks.safetywing.dev/runbooks/rabbitmq/rabbitmqnodedown/">runbook</a></td> </tr> <tr> <td>RabbitmqMemoryAlarm</td> <td><code>page</code></td> <td><a href="https://runbooks.safetywing.dev/runbooks/rabbitmq/rabbitmqmemoryalarm/">runbook</a></td> </tr> <tr> <td>RabbitmqDiskAlarm</td> <td><code>page</code></td> <td><a href="https://runbooks.safetywing.dev/runbooks/rabbitmq/rabbitmqdiskalarm/">runbook</a></td> </tr> <tr> <td>RabbitmqQueueBacklog</td> <td><code>ticket</code></td> <td><a href="https://runbooks.safetywing.dev/runbooks/rabbitmq/rabbitmqqueuebacklog/">runbook</a></td> </tr> <tr> <td>RabbitmqQueueNoConsumers</td> <td><code>ticket</code></td> <td><a href="https://runbooks.safetywing.dev/runbooks/rabbitmq/rabbitmqqueuenoconsumers/">runbook</a></td> </tr> </tbody> </table> <h3 id="ceph--platform-tier">Ceph <small>(platform tier)</small><a class="anchor" href="#ceph--platform-tier">#</a></h3> <table> <thead> <tr> <th>Alert</th> <th>Severity</th> <th>Runbook</th> </tr> </thead> <tbody> <tr> <td>CephHealthError</td> <td><code>page</code></td> <td><a href="https://runbooks.safetywing.dev/runbooks/ceph/cephhealtherror/">runbook</a></td> </tr> <tr> <td>CephMonOutOfQuorum</td> <td><code>page</code></td> <td><a href="https://runbooks.safetywing.dev/runbooks/ceph/cephmonoutofquorum/">runbook</a></td> </tr> <tr> <td>CephHealthWarning</td> <td><code>ticket</code></td> <td><a href="https://runbooks.safetywing.dev/runbooks/ceph/cephhealthwarning/">runbook</a></td> </tr> <tr> <td>CephOSDDown</td> <td><code>ticket</code></td> <td><a href="https://runbooks.safetywing.dev/runbooks/ceph/cephosddown/">runbook</a></td> </tr> <tr> <td>CephClusterNearFull</td> <td><code>ticket</code></td> <td><a href="https://runbooks.safetywing.dev/runbooks/ceph/cephclusternearfull/">runbook</a></td> </tr> </tbody> </table> <h3 id="elasticsearch--platform-tier">Elasticsearch <small>(platform tier)</small><a class="anchor" href="#elasticsearch--platform-tier">#</a></h3> <table> <thead> <tr> <th>Alert</th> <th>Severity</th> <th>Runbook</th> </tr> </thead> <tbody> <tr> <td>ElasticsearchClusterRed</td> <td><code>page</code></td> <td><a href="https://runbooks.safetywing.dev/runbooks/elasticsearch/elasticsearchclusterred/">runbook</a></td> </tr> <tr> <td>ElasticsearchClusterYellow</td> <td><code>ticket</code></td> <td><a href="https://runbooks.safetywing.dev/runbooks/elasticsearch/elasticsearchclusteryellow/">runbook</a></td> </tr> <tr> <td>ElasticsearchHeapHigh</td> <td><code>ticket</code></td> <td><a href="https://runbooks.safetywing.dev/runbooks/elasticsearch/elasticsearchheaphigh/">runbook</a></td> </tr> <tr> <td>ElasticsearchDiskWatermark</td> <td><code>ticket</code></td> <td><a href="https://runbooks.safetywing.dev/runbooks/elasticsearch/elasticsearchdiskwatermark/">runbook</a></td> </tr> </tbody> </table> <h3 id="node--platform-tier">Node <small>(platform tier)</small><a class="anchor" href="#node--platform-tier">#</a></h3> <table> <thead> <tr> <th>Alert</th> <th>Severity</th> <th>Runbook</th> </tr> </thead> <tbody> <tr> <td>NodeFilesystemAlmostFull</td> <td><code>ticket</code></td> <td><a href="https://runbooks.safetywing.dev/runbooks/node/nodefilesystemalmostfull/">runbook</a></td> </tr> </tbody> </table> <h3 id="traefik--platform-tier">Traefik <small>(platform tier)</small><a class="anchor" href="#traefik--platform-tier">#</a></h3> <table> <thead> <tr> <th>Alert</th> <th>Severity</th> <th>Runbook</th> </tr> </thead> <tbody> <tr> <td>TraefikDown</td> <td><code>page</code></td> <td><a href="https://runbooks.safetywing.dev/runbooks/traefik/traefikdown/">runbook</a></td> </tr> <tr> <td>TraefikHigh5xxRate</td> <td><code>ticket</code></td> <td><a href="https://runbooks.safetywing.dev/runbooks/traefik/traefikhigh5xxrate/">runbook</a></td> </tr> </tbody> </table> <h3 id="environment--environment-tier">Environment <small>(environment tier)</small><a class="anchor" href="#environment--environment-tier">#</a></h3> <table> <thead> <tr> <th>Alert</th> <th>Severity</th> <th>Runbook</th> </tr> </thead> <tbody> <tr> <td>EnvironmentHigh5xxRate</td> <td><code>ticket</code></td> <td><a href="https://runbooks.safetywing.dev/runbooks/environment/environmenthigh5xxrate/">runbook</a></td> </tr> </tbody> </table> <h2 id="stock-alerts-kube-prometheus-stack">Stock alerts (kube-prometheus-stack)<a class="anchor" href="#stock-alerts-kube-prometheus-stack">#</a></h2> <p>Shipped by the kube-prometheus-stack <code>defaultRules</code>. Documented upstream — links go there.</p>