Cursory Introduction

Apache Kafka was adult past LinkedIn to handle their log files and handed over to the open up source customs in early 2011. It became the principal Apache project in October 2012. Apache Kafka is a distributed streaming platform, and has iii primal capabilities:

  • Publish and subscribe to streams of records, like to a message queue or enterprise messaging system
  • Store streams of records in a fault-tolerant, durable manner
  • Procedure streams of records as they occur

What is it used for?

Apache Kafka tin can exist used for various purposes in an arrangement, like equally a Messaging service, Real-fourth dimension stream processing, Log aggregation, Commit log service, Event sourcing, etc.

In order to use this engineering, we need to first set up our Kafka environment. In this demonstration we will set upward a Kafka environment consisting of 1 zookeeper and 3 brokers. We will create a simple Producer and Consumer using Coffee Spring boot.

System Architecture

  1. Zookeeper: is used past Kafka to maintain state between the nodes of the cluster.
  2. Brokers: The "pipes" in our pipeline, which store and emit data
  3. Producers: Send messages to the topic
  4. Consumer: Listen and read letters from topic

Installation

We volition set up our Kafka cluster in a Centos 7 environment.

I. Prerequisite

  1. Centos 7 Arrangement
  2. Java 1.8
    If you lot oasis't installed Java 1.8 in your system, please install it beginning
  3. kafka_2.11-2.3.0.tgz
    You tin download the file here.
  4. Spring kick application
    If you want to know how to build a jump boot awarding from scratch, you tin can refer to this link, or if you desire to but acquire how to create a Kafka Producer and Consumer using spring kicking please experience free to clone my projection instead.

II. Starting Zookeeper

First, excerpt kafka_2.11-2.3.0.tgz using this command:

tar -xvzf kafka_2.11-ii.3.0.tgz

Within the extracted kafka_2.11-2.3.0 folder, you will find a bin/zookeeper-server-starting time.sh file which is used to start the zookeeper and config/zookeeper.properties which provides the default configuration for the zookeeper server to run.

Showtime the zookeeper by running (inside the Kafka root folder):

nohup bin/zookeeper-server-kickoff.sh config/zookeeper.properties &>zookeper_nohup.out&

This command uses nohup to transfer the output log into a file (zookeeper_nohup.out). Y'all can monitor the log past tailing to that file using this command:

tail -2000f zookeeper_nohup.out

Three. Starting Brokers

In the same way that we started Zookeeper, there are ii files config/kafka-server-kickoff.sh to commencement the broker and config/server.backdrop to configure the broker server. From the architecture diagram to a higher place nosotros want to get-go 3 brokers, and all you lot need to practise is create 3 configuration properties files. Let'due south create them:

cp config/server.properties config/server1.backdrop
cp config/server.properties config/server2.properties
cp config/server.properties config/server3.properties

There are, however, 3 properties that must exist unique for each banker case:  banker.id, listeners, and log.dirs.

Change the higher up 3 properties for each copy of the file so that they are all unique.

server1.properties

broker.id=i
listeners=PLAINTEXT://172.xix.16.244:9093
log.dirs=/tmp/kafka-logs1

server2.properties

broker.id=2
listeners=PLAINTEXT://172.19.16.244:9094
log.dirs=/tmp/kafka-logs2

server3.properties

banker.id=3
listeners=PLAINTEXT://172.19.16.244:9095
log.dirs=/tmp/kafka-logs3

Important Note: 172.19.sixteen.224 is the host IP address (of your centos 7 system). You must set up the IP address or domain name in the listener holding or your cluster cannot exist accessed from outside of your host.


Now it's time to run the brokers.

Enter this control to run banker 1

nohup bin/kafka-server-start.sh config/server3.properties &> server3_log.out&

Enter this command to run broker two

nohup bin/kafka-server-start.sh config/server2.properties &> server2_log.out&

Enter this command to run broker 3

nohup bin/kafka-server-start.sh config/server3.backdrop &> server3_log.out&

Warning: Don't forget to open the brokers port in the centos firewall.  You can run the following command to open up those 3 ports (9093,9094,9095) in the firewall:

firewall-cmd --zone=public --add-port=9093/tcp –permanent
firewall-cmd --zone=public --add-port=9094/tcp --permanent
firewall-cmd --zone=public --add-port=9095/tcp --permanent
firewall-cmd --reload

Now nosotros have our brokers running. Before we tin can outset putting information into the cluster, nosotros demand to create a topic to which the data will belong.

Creating a Topic

To create a topic, run this control:

bin/kafka-topics.sh --create --topic bootcamp-topic --zookeeper localhost:2181 --partitions 3 --replication-factor 2

It will create a Kafka topic named 'bootcamp-topic'. The --partition statement describes how many brokers we volition use. Since we set up upward iii brokers, we can set this selection to iii.  The --replication-factor describes how many copies of your data yous crave. This is handy - in case one of the brokers goes downward, you all the same accept your information on the others.

IV. Testing, Connecting and Monitoring our Kafka using Kafka Tools

Now our Kafka is ready to utilize, we can monitor Kafka traffic using Kafka Tools that tin downloaded from http://www.kafkatool.com/download.html

In the Kafka Tools GUI, click File>>Add New Connexion

Make full in the following fields:
Cluster Name: You tin input anything - it'south simply a connection proper name
Zookeeper Host: IP accost or domain name where nosotros installed the Kafka cluster
Bootstrap server: your broker'south instance [IP,port] separated by comma

Click the Exam button to examination the connection. If it's a success, you lot tin can add the connection by pressing the Yep button on the popup dialog.

Right click your newly added connectedness and click Connect.

This is how it should look. There is no information in the topic yet - we will add together it later.

V. Creating a Producer

In this department, we will create the producer inside our bound-kick awarding. Yous can create and use your ain leap boot awarding from scratch, or clone my spring boot project from github.

First, we need to add the spring-Kafka library. Put this in your build.gradle :

Add the following properties in application.properties:

spring.kafka.bootstrap-servers=172.nineteen.xvi.224:9093,172.19.xvi.224:9094,172.19.16.224:9095
kafka.retries=three          
kafka.session.timeout=15000
kafka.my.topic=bootcamp-topic
kafka.auto.commit=true
kafka.offset.reset=latest
kafka.security.protocol=PLAINTEXT
bound.kafka.consumer.grouping-id=jcg-grouping

Now you can create a simple Restful API that will publish posted request trunk from the customer to Kafka.

Run spring boot application past using gradlewbootRun control, or you can build the jar file and execute information technology directly with the coffee -jar command.

Permit's try to hit our newly created Restful API service using the postman:

Let's check our message in Kafka tools:

Now we have 1 message in our topic.

Vi. Creating a Consumer

We volition create the consumer within the bound kick application.

The consumer is a simple bound service that listens to the Kafka topic. Whenever a new bulletin is sent to the topic, the listener captures it and saves to the database.

If we run the spring boot, and monitor the log, we will have something similar this whenever in that location is a message sent to the Kafka topic.

The message is then saved to the database.

Conclusion

Now you have Apache Kafka running on your CentOS server with 3 brokers and 1 zookeeper. You tin can also produce and consume letters from your jump kick application to your Kafka topic. This makes it much easier to implement asynchronous processes in your next projection. To learn more well-nigh Kafka, take a await at its documentation.

Author:
Arif Nazar Purwandaru - Annotator Developer