Using Kafka with Sarama

Kafka is a popular distributed message queue with a lot of advanced functions for building distributed systems. This recipe will show how to write to a Kafka topic using a synchronous producer and how to consume the same topic using a partition consumer. This recipe will not explore different configurations of Kafka as that is a much wider topic, but I suggest beginning at https://kafka.apache.org/intro.

Getting ready

Configure your environment according to these steps:

1. Refer to the Getting ready section of the Goflow for dataflow programming recipe in this chapter.
2. Install Kafka using the steps mentioned at https://www.tutorialspoint.com/apache_kafka/apache_kafka_installation_steps.htm.
3. Alternatively, you can also access https://github.com/spotify/docker-kafka.
4. Run the go get gopkg.in/Shopify/sarama.v1 command.

How to do it...

These steps cover writing and running your application:

1. From your terminal/console application, create the chapter11/synckafka directory and navigate...