Kafka on SafetyWing Runbooks

KafkaConsumerGroupLagHigh

Mon, 01 Jan 0001 00:00:00 +0000

Meaning#

A consumer group is falling behind the producers on a topic — the gap between the latest offset and the group’s committed offset (lag) has exceeded the threshold. Messages are being produced faster than they are consumed, so processing is delayed. The consumergroup and topic labels identify exactly which consumer and topic are affected.

Fires when: per-(consumergroup, topic) lag exceeds <threshold> for 15m. Severity ticket, tier component.

max by (consumergroup, topic) (kafka_consumergroup_lag{namespace="safetywing-<env>-infra"}) > <threshold>

Impact#

Delayed processing for the affected consumer group → stale downstream data, late side effects, growing end-to-end latency.
If lag keeps climbing, retention may expire un-consumed messages, causing permanent message loss.
Brokers retain more unconsumed data, increasing disk usage.

Diagnosis#

kubectl config use-context hetzner
kubectl get kafka,kafkanodepool -n safetywing-<env>-infra

# Describe the lagging group: per-partition LAG, CURRENT-OFFSET, LOG-END-OFFSET, CONSUMER-ID
kubectl exec -n safetywing-<env>-infra <broker-pod> -- \
 bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
 --describe --group <consumergroup>

# Are there active members, or is the group empty / rebalancing?
kubectl exec -n safetywing-<env>-infra <broker-pod> -- \
 bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
 --describe --group <consumergroup> --members --verbose

# Inspect the consuming application workload
kubectl get pods -A | grep <consumer-app>
kubectl logs -n <consumer-ns> <consumer-pod> --tail=200

Confirm trend and scope in Prometheus (prom-ep.hetzner.safetywing.dev):

KafkaNoActiveController

Mon, 01 Jan 0001 00:00:00 +0000

Meaning#

In KRaft mode exactly one controller node should be active (the metadata quorum leader). This alert means the cluster sees zero active controllers (no quorum leader) or more than one (split brain). Either state puts cluster metadata — topic, partition, ISR, and config state — at risk and blocks administrative operations.

Fires when: the summed active controller count across the namespace is not exactly 1 for 5m. Severity page, tier component.

KafkaOfflinePartitions

Mon, 01 Jan 0001 00:00:00 +0000

Meaning#

One or more partitions have no leader, so they cannot serve reads or writes. Any producer or consumer touching an offline partition is blocked, which usually means data loss risk and stalled traffic across affected topics.

Fires when: any broker reports a non-zero offline partition count for 5m. Severity page, tier component.

max(kafka_controller_kafkacontroller_offlinepartitionscount{namespace="safetywing-<env>-infra"}) > 0

Impact#

Produce and consume requests to offline partitions fail or hang.
Consumer groups stall on the affected partitions; lag grows.
Topics with offline partitions are effectively partially unavailable.
Often a symptom of multiple broker failures or unavailable replicas.

Diagnosis#

kubectl config use-context hetzner

# Strimzi CRs and broker pods
kubectl get kafka,kafkanodepool -n safetywing-<env>-infra
kubectl get pods -n safetywing-<env>-infra -l strimzi.io/cluster

# Which brokers are not Ready
kubectl get pods -n safetywing-<env>-infra -o wide | grep -v Running

# Broker / controller logs (look for leader election, ISR, disk errors)
kubectl logs -n safetywing-<env>-infra <broker-pod> --tail=200

# Cluster + topic state from inside a broker
kubectl exec -n safetywing-<env>-infra <broker-pod> -- \
 bin/kafka-topics.sh --bootstrap-server localhost:9092 \
 --describe --under-min-isr-partitions
kubectl exec -n safetywing-<env>-infra <broker-pod> -- \
 bin/kafka-topics.sh --bootstrap-server localhost:9092 \
 --describe --unavailable-partitions

Confirm scope in Prometheus (prom-ep.hetzner.safetywing.dev):

KafkaUnderReplicatedPartitions

Mon, 01 Jan 0001 00:00:00 +0000

Meaning#

Partitions have fewer in-sync replicas than their configured replication factor. The cluster is still serving traffic, but durability is reduced — losing one more broker could take partitions offline or lose data. Usually a broker is down, restarting, or lagging behind on replication.

Fires when: any broker reports a non-zero under-replicated partition count for 10m. Severity ticket, tier component.

max(kafka_server_replicamanager_underreplicatedpartitions{namespace="safetywing-<env>-infra"}) > 0

Impact#

Reduced fault tolerance: a single additional broker failure may cause offline partitions or data loss.
Producers using acks=all may slow down or block if the ISR drops below min.insync.replicas.
Sustained under-replication often precedes an OfflinePartitions page.

Diagnosis#

kubectl config use-context hetzner

# Strimzi CRs and broker pods
kubectl get kafka,kafkanodepool -n safetywing-<env>-infra
kubectl get pods -n safetywing-<env>-infra -l strimzi.io/cluster -o wide

# Any broker not Ready / restarting?
kubectl get pods -n safetywing-<env>-infra | grep -vE "Running|Completed"

# Which partitions are under-replicated / below min ISR
kubectl exec -n safetywing-<env>-infra <broker-pod> -- \
 bin/kafka-topics.sh --bootstrap-server localhost:9092 \
 --describe --under-replicated-partitions
kubectl exec -n safetywing-<env>-infra <broker-pod> -- \
 bin/kafka-topics.sh --bootstrap-server localhost:9092 \
 --describe --under-min-isr-partitions

# Broker logs: replica fetcher, ISR shrink/expand, disk
kubectl logs -n safetywing-<env>-infra <broker-pod> --tail=200 | grep -iE "ISR|replica|fetch"

Confirm scope in Prometheus (prom-ep.hetzner.safetywing.dev):