Kafka Connect on SafetyWing Runbooks

KafkaConnectFailedTasks

Mon, 01 Jan 0001 00:00:00 +0000

Meaning#

One or more Debezium CDC connector tasks have entered the FAILED state, so change capture for the affected connector is degraded or fully stopped. Fires when:

max(kafka_connect_worker_metrics_connector_failed_task_count{namespace="safetywing-<env>-infra"}) > 0

for: 10m, severity page, tier component.

Impact#

CDC from MOCO MySQL into Kafka is stalled for the failed connector. Downstream consumers stop receiving database changes: search indices fall behind, derived/mirror tables go stale, and any event-driven flow fed by these topics no longer reflects new writes. Lag grows until the task is recovered.

KafkaConnectNoConnectors

Mon, 01 Jan 0001 00:00:00 +0000

Meaning#

The Kafka Connect cluster is running but reports zero connectors, meaning no Debezium CDC source connector is deployed or running — CDC for the environment may be unconfigured or all connectors were removed. Fires when:

max(kafka_connect_worker_metrics_connector_count{namespace="safetywing-<env>-infra"}) == 0

for: 15m, severity ticket, tier component.

Impact#

No change capture is happening at all in this environment: no MySQL changes flow from MOCO MySQL into Kafka. Downstream consumers (search indices, mirror/derived tables, event-driven flows) receive nothing new. For a freshly provisioned env this may be expected during bring-up; for an established env it means CDC is silently broken.

KafkaConnectWorkersDown

Mon, 01 Jan 0001 00:00:00 +0000

Meaning#

Fewer Kafka Connect workers are reporting metrics than the number of replicas the kafka-cdc chart expects, indicating one or more Connect pods are down, crash-looping, or not scraping. Fires when:

count(kafka_connect_worker_metrics_connector_count{namespace="safetywing-<env>-infra"}) < <connect.replicas>

for: 5m, severity page, tier component.

Impact#

Reduced Connect capacity and resilience for the CDC pipeline. Tasks owned by the missing worker are rebalanced onto survivors (added load, possible throughput drop and lag); if the cluster is at one replica, CDC from MOCO MySQL into Kafka is fully stopped and downstream consumers (search indices, mirror tables, event flows) stop receiving DB changes.