TraefikDown

Mon, 01 Jan 0001 00:00:00 +0000

Meaning#

Prometheus has no healthy Traefik scrape target. Either the Traefik edge ingress is down, or just its metrics endpoint/scraping is broken.

Fires when: there is no up == 1 series for any Traefik scrape job.

absent(up{job=~".*traefik.*"} == 1)

for: 10m, severity page, tier platform (cluster-wide, no environment label).

Impact#

Traefik is the edge ingress (a hostNetwork DaemonSet on the Hetzner cluster). If Traefik itself is down, all external HTTP(S) traffic into the cluster fails — every public domain behind it. If only metrics are broken, traffic may still flow but we are blind to edge health and 5xx alerting is degraded.

TraefikHigh5xxRate

Mon, 01 Jan 0001 00:00:00 +0000

Meaning#

The cluster-wide ratio of 5xx responses served by Traefik exceeds 5%. This is an edge-level signal aggregating all services behind Traefik.

Fires when: 5xx requests are more than 5% of all Traefik service requests.

sum(rate(traefik_service_requests_total{code=~"5.."}[5m]))
 / sum(rate(traefik_service_requests_total[5m])) > 0.05

for: 10m, severity ticket, tier platform (cluster-wide, no environment label).

Impact#

A meaningful fraction of requests entering through the edge are failing with server errors. Because this is cluster-wide, it usually points at either a broadly-impacting backend (a shared dependency) or one high-traffic service skewing the aggregate. Users across one or more environments see errors.

Traefik on SafetyWing Runbooks

TraefikDown

Meaning#

Impact#

TraefikHigh5xxRate

Meaning#

Impact#