Introduction
For any mission-critical application, monitoring is a non-negotiable requirement. Monitoring helps identify, diagnose, and resolve issues before they become catastrophic. When dealing with Apache Kafka, which is a distributed, high-throughput, and real-time processing system, it is all the more critical to have robust monitoring in place.
In this post, we will explore the tools and techniques to monitor Kafka effectively. You’ll get hands-on knowledge of Kafka’s built-in monitoring capabilities, the JMX interface, and popular open-source monitoring tools like Prometheus and Grafana.
Part 1: Built-In Monitoring Capabilities
Apache Kafka offers built-in command-line tools that allow you to check the status of your Kafka cluster.
Example 1: Check the Status of a Topic
You can use the kafka-topics.sh
script to describe the topic and get the status:
bin/kafka-topics.sh --describe --topic my_topic --bootstrap-server localhost:9092
Example 2: Check the Status of the Consumer Group
Similarly, the kafka-consumer-groups.sh
script allows you to check the status of consumer groups:
bin/kafka-consumer-groups.sh --describe --group my_group --bootstrap-server localhost:9092
Example 3: Check the Under-replicated Partitions
The kafka-topics.sh
script can also be used to check for under-replicated partitions, which can be an indicator of failing brokers:
bin/kafka-topics.sh --describe --under-replicated-partitions --bootstrap-server localhost:9092
Part 2: JMX Interface
Kafka also exposes a Java Management Extensions (JMX) interface that can be used to monitor its internal metrics.
Example 4: Enable JMX Interface
You can enable the JMX interface by setting the KAFKA_JMX_OPTS
environment variable:
export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=127.0.0.1 -Dcom.sun.management.jmxremote.rmi.port=9999"
Then, you can start your Kafka server:
bin/kafka-server-start.sh config/server.properties
Example 5: Monitor Kafka Metrics using JConsole
JConsole, which comes with JDK, can be used to connect to the JMX interface and monitor Kafka metrics.
jconsole localhost:9999
Part 3: Monitoring Kafka with Prometheus and Grafana
Prometheus, an open-source monitoring system, and Grafana, an open-source metric analytics & visualization suite, are commonly used for Kafka monitoring.
Example 6: Prometheus Kafka Exporter
Prometheus provides a Kafka exporter, which is a service that scrapes Kafka metrics and makes them available to the Prometheus server. Here is a basic configuration:
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: 'kafka'
static_configs:
- targets: ['localhost:7071']
Then, you can start the Kafka exporter:
./kafka_exporter --kafka.server=localhost:9092
Example 7: Prometheus Server
You can then configure the Prometheus server to scrape the metrics exposed by the Kafka exporter. Here’s an example of a Prometheus server configuration:
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- job_name: 'kafka'
static_configs:
- targets: ['localhost:7071']
Then, start the Prometheus server:
./prometheus --config.file=prometheus.yml
Example 8: Grafana Dashboard
Grafana can connect to the Prometheus server and visualize the collected metrics. First, add the Prometheus data source in Grafana:
{
"name": "Prometheus",
"type": "prometheus",
"url": "http://localhost:9090",
"access": "proxy"
}
You can then import a pre-configured Kafka dashboard or create your own.
Conclusion
Apache Kafka, as an integral part of many complex data pipelines, requires effective monitoring to ensure its health and performance. Kafka provides various means for monitoring, from built-in tools to JMX metrics, to integration with open-source monitoring solutions like Prometheus and Grafana.
By now, you should have a solid understanding of the different tools and techniques available for monitoring Kafka. However, always remember that each Kafka deployment is unique, and it’s essential to understand what metrics are important for your particular use case.
Subscribe to our email newsletter to get the latest posts delivered right to your email.