Using Kafka

Apache Kafka is a fast, scalable, durable, and distributed messaging system developed by LinkedIn. We will use Kafka using clj-kafka. It's a Clojure wrapper.

Getting ready

We need to install Kafka for this recipe. The installation steps of Kafka are as follows.

Downloading Kafka 0.8.2.2

First, download Kafka 0.8.2.2 using the wget command:

$ wget http://ftp.kddilabs.jp/infosystems/apache/kafka/0.8.2.2/kafka_2.11-0.8.2.2.tgz

Then, extract Kafka files from the downloaded TAR file:

$ tar xzvf kafka_2.11-0.8.2.2.tgz

Start ZooKeeper and Kafka server

Go to the Kafka directory and run ZooKeeper and Kafka server:

$ cd kafka_2.11-0.8.2.2 
$ bin/zookeeper-server-start.sh config/zookeeper.properties

The preceding ZooKeeper startup command will show you the following output:

Then, we will start kafka-server:

$ bin/kafka-server-start.sh config/server.properties

The preceding command will show you the following output:

Create topic

Create topic names test:

$ bin/kafka-topics.sh --create ...