My top 5 tools to manage/develop with Apache Kafka
Kafka topic analyzer
kafka-topic-analyzer is a very simple to use CLI that gathers statistics about a Apache Kafka topic by reading it from beginning to end and counting various metrics.
I find it very useful when handling big topics, specially the ones with log compaction enabled, making very easy to see if adjusts to the topic configuration, like segment size, is required.
kafka-ui
kafka-ui is a free, open-source web UI to monitor and manage Apache Kafka clusters.
Having a UI can save you a lot of time when you need to check a topic config, its names, or broker addresses. I personally don't like to manage the topic configuration through it, I instead use this kind of interface as read only.
Bitnami Docker image
When I need to run some experiment and I need a Kafka broker running locally I always use the docker-compose configuration and images from bitnami.
On my previous post about Kafka and Python, I used bitnami/kafka to run Kafka locally and connect to it using a Python client.
Prometheus/Grafana and kafka-exporter
Maintaining a Kafka cluster requires you to have a good view of the current state of your cluster as well being able to see trends on its metrics. The simplest way to achieve this is by using the metrics provided by Kafka and exporting it to Prometheus and create some charts on grafana.
kafka-exporter extracts the metrics from JMX and make it available for Prometheus to ingest them.
kcat
From time to time you will need to verify messages from a given topic. This might happen and your are investigating some issue or developing a new feature that consumes or produces to a topic.
Kafka provides a console consumer, but it is a bit limited. For that you can use kcat instead. kcat is a generic non-JVM producer and consumer for Apache Kafka >=0.8, think of it as a netcat for Kafka.
It also allows you to query a broker metadata, or a topic offset by timestamp for instance.
docker run -it --rm \
edenhill/kcat \
-b kafka-broker:9092 \
-C \
-f '\nKey (%K bytes): %k\t\nValue (%S bytes): %s\n\Partition: %p\tOffset: %o\n--\n' \
-t testKey (1 bytes): 1
Value (88 bytes): {"order_id":1,"order_ts":1534772501276,"total_amount":10.50,...}
Partition: 0 Offset: 0
--Key (1 bytes): 2
Value (89 bytes): {"order_id":2,"order_ts":1534772605276,"total_amount":3.32,..}
Partition: 0 Offset: 1
--Key (1 bytes): 3
Value (90 bytes): {"order_id":3,"order_ts":1534772742276,"total_amount":21.00,..}
Partition: 0 Offset: 2
--
% Reached end of topic test [0] at offset 3
