Setting Up Kafka Streaming with Docker Compose

Set up a Kafka streaming environment using Docker Compose, and enhance your management capabilities with Confluent Control Center. Simplify development, testing, and monitoring with this comprehensive guide.

Introduction:

In this tutorial, we will walk you through setting up a Kafka streaming environment using Docker Compose. Kafka is a distributed streaming platform that can handle real-time data feeds. With Docker Compose, you can easily manage multi-container Docker applications. This guide will help you set up Kafka, Zookeeper, and the Confluent Control Center.

Prerequisites:

  • Docker and Docker Compose installed on your machine.
  • Basic knowledge of Docker and Kafka.

Step 1: Clone the Repository

First, clone the repository containing the Docker Compose file.

git clone https://github.com/DevelopersCoffee/kafka-streaming.git
cd kafka-streaming

Step 2: Review the Docker Compose File

The docker-compose.yaml file sets up the necessary services for Kafka streaming. Here is the content of the file: https://github.com/DevelopersCoffee/kafka-streaming/blob/main/docker-compose.yaml

Step 3: Start the Kafka Environment

Use Docker Compose to start the Kafka environment.

Description: Start the Kafka services using Docker Compose.

docker-compose -f ~/workspace/docker/docker-compose/kafka/docker-compose.yml up -d

This command will start Zookeeper, Kafka broker, and the Confluent Control Center in detached mode.

Step 4: Stop Kafka

Description: Stop the Kafka services using Docker Compose.

docker-compose -f ~/workspace/docker/docker-compose/kafka/docker-compose.yml down

Step 5: Verify the Setup

Once the services are up, you can verify the setup by checking the running containers.

docker ps

You should see the zookeeper, broker, and control-center containers running.

Confluent Control Center

Confluent Control Center is a web-based management and monitoring tool provided by Confluent for Apache Kafka. It allows you to manage and monitor your Kafka clusters, topics, and streaming applications efficiently. Here are some key features and details about Confluent Control Center:

Key Features
  1. Cluster Management:
    • Monitor the health and status of your Kafka clusters.
    • View broker statistics and metrics.
    • Manage broker configurations.
  2. Topic Management:
    • Create, modify, and delete topics.
    • View topic configurations and partitions.
    • Monitor topic-specific metrics such as partition offsets, replication status, and throughput.
  3. Consumer Monitoring:
    • Track consumer group lag and offsets.
    • Monitor the performance and health of consumer groups.
    • Identify slow consumers and potential bottlenecks.
  4. Schema Registry Integration:
    • Manage schemas for your Kafka topics.
    • View and edit schema versions.
    • Ensure data compatibility with schema evolution.
  5. KSQL Integration:
    • Create, manage, and monitor KSQL queries.
    • Visualize data streams and query results.
    • Perform real-time data transformations and analytics using KSQL.
  6. Alerting and Notifications:
    • Set up alerts for various metrics and events.
    • Receive notifications via email, Slack, or other channels.
    • Proactively monitor your Kafka environment to identify and resolve issues.
  7. Security and Auditing:
    • Manage user access and permissions.
    • Monitor access logs and audit trails.
    • Ensure compliance with security policies.

Accessing Control Center

Once you have started your Docker Compose setup, you can access the Confluent Control Center by navigating to http://localhost:9021 in your web browser.

Using Control Center
  1. Dashboard:
    • The dashboard provides an overview of your Kafka clusters, including cluster health, broker status, and key metrics.
  2. Topics:
    • Manage and monitor Kafka topics. View details like partition distribution, replication status, and data flow.
  3. Consumer Groups:
    • Monitor consumer groups, check their lag, and ensure that your consumers are processing data efficiently.
  4. Schemas:
    • Manage and validate schemas for your Kafka data streams. Ensure compatibility and track schema versions.
  5. KSQL:
    • Create and manage KSQL queries for real-time data processing. Visualize query results and data transformations.
  6. Alerts:
    • Set up alerts for various Kafka metrics. Get notified of any issues or anomalies in your Kafka environment.
  7. Settings:
    • Configure user access, permissions, and security settings. Monitor access logs and audit trails.

Step 6: Basic Kafka Operations

Now that your Kafka environment is set up, you can perform basic Kafka operations such as creating topics, producing messages, and consuming messages.

List Topics

Description: List all the topics in Kafka.

docker exec -it broker kafka-topics --list --bootstrap-server broker:29092
Create Topic

Description: Create a new topic named test-topic with 3 partitions and a replication factor of 1.

docker exec -it broker kafka-topics --create --topic test-topic --partitions 3 --replication-factor 1 --bootstrap-server broker:29092
Delete Topic

Description: Delete the existing topic named test-topic.

docker exec -it broker kafka-topics --delete --topic test-topic --bootstrap-server broker:29092
Publish Messages

Description: Publish messages to the topic test-topic. After running this command, type your messages and press Enter. Use Ctrl+C to exit.

docker exec -it broker kafka-console-producer --broker-list broker:29092 --topic test-topic
Publish Messages from a File

Description: Produce messages from a file named messages.txt located in /tmp to the topic test-topic.

# Create the messages.txt file in /tmp
echo -e "First message\nSecond message\nThird message" > /tmp/messages.txt

# Produce messages to Kafka from the file
docker exec -it broker kafka-console-producer --broker-list broker:29092 --topic test-topic < /tmp/messages.txt
Consume Messages

Description: Consume messages from the topic test-topic from the beginning.

docker exec -it broker kafka-console-consumer --bootstrap-server broker:29092 --topic test-topic --from-beginning
Consume Messages to a File

Description: Consume messages from the topic test-topic and save them to a file named output.txt in /tmp.

docker exec -it broker kafka-console-consumer --bootstrap-server broker:29092 --topic test-topic --from-beginning > /tmp/output.txt
Describe Topic

Description: Describe the topic test-topic to see details like partitions, replication, etc.

docker exec -it broker kafka-topics --describe --topic test-topic --bootstrap-server broker:29092
List Consumer Groups

Description: List all the consumer groups in Kafka.

docker exec -it broker kafka-consumer-groups --bootstrap-server broker:29092 --list
Describe Consumer Group

Description: Describe a specific consumer group named my-group.

docker exec -it broker kafka-consumer-groups --bootstrap-server broker:29092 --describe --group my-group
Reset Consumer Group Offsets

Description: Reset the offsets for a consumer group named my-group for a topic test-topic to the earliest.

docker exec -it broker kafka-consumer-groups --bootstrap-server broker:29092 --group my-group --topic test-topic --reset-offsets --to-earliest --execute
Check Kafka Broker Status

Description: Check the status of the Kafka broker.

docker exec -it broker kafka-broker-api-versions --bootstrap-server broker:29092
Describe Kafka Cluster

Description: Describe the Kafka cluster to get information about brokers, topics, partitions, etc.

docker exec -it broker kafka-cluster-metadata --bootstrap-server broker:29092
KafkaRestart Command

Description: This command will restart your Kafka Docker setup by stopping the services, removing volumes, and starting the services again.

docker-compose -f ~/workspace/docker/docker-compose/kafka/docker-compose.yml down
docker volume rm kafka_broker_data kafka_zookeeper_data
docker-compose -f ~/workspace/docker/docker-compose/kafka/docker-compose.yml up -d

These commands should provide a comprehensive guide to managing your Kafka environment using Docker Compose.

Conclusion:

By following these steps, you have successfully set up a Kafka streaming environment using Docker Compose. This setup simplifies development and testing by providing a straightforward configuration process. Additionally, integrating Confluent Control Center into your Kafka setup enhances your ability to manage and monitor your Kafka environment. Control Center offers a comprehensive set of features that ensure the health and performance of your Kafka clusters and streaming applications. For more detailed information and advanced configurations, refer to the Confluent Control Center documentation. If you have any questions or need further assistance, feel free to ask! Happy streaming!


For more tutorials and insights on Kafka and other technologies, stay tuned to DevelopersCoffee.com.

Reference:

https://kafka.apache.org/documentation/

https://docs.confluent.io/platform/current/control-center/index.html

https://docs.docker.com

https://docs.docker.com/compose

Leave a Reply

Your email address will not be published. Required fields are marked *