How to create a JSON serializer for Kafka Producer
We want to extend the Kafka Multithreaded
Producer example, and instead of dealing with a simple line of text, we want
to represent
each line as a Java Object. While sending Java Object to Kafka, you must serialize
it to byte[]. Hence, we want to create a JSON Serializer using
jackson-databind for
serializing Java Objects to byte[].
This example will create JSON Serializer to help you understand the details of
implementing a custom serializer.
This example is an excerpt from the Book Kafka Streams –
Real-time Stream Processing
For a detailed explanation of the example and much more, you can get access to
the Book using below link.
Creating JSON Producer for Kafka
We will be using com.fasterxml.jackson.databind library for implementing a JSON serializer. However, you are free to use any other JSON library such as Google’s Gson or something else of your choice. The code below shows a JSON serializer implementation.
You can access fully function project in our GitHub folder.
Using Kafka JSON Serializer
The example data file contains a CSV record. We wanted to read the CSV and convert it into a Java Object. The code below shows a simple function that reads a CSV file and creates a list of StockData object.
Once you have a list of objects, you are ready to use your JsonSerializer. To use your custom json serializer, you must set VALUE_SERIALIZER_CLASS_CONFIG as shown in the code below.
The example project is using jsonschema2pojo-maven-plugin to generate the StockData.java file using a JSON schema automatically. Rest of the code is straightforward and mostly inherited from our Producer Threads example.