Category: Apache Kafka

Apache Kafka – Develop Producer Java application No.66

Posted on by kumoman

How to develop Producer application by Java on Visual Studio Code is shown in this blog. Contents ▼1. Apache Kafka ▼2. Prerequisites▼3. Develop Producer Java application3-1. Make a directoryMake a directory to develop an application.3-2. Create a project using Apache Maven3-3. Remove the App.java and AppTest.java files3-4. Update pom.xml file3-5. Develop Producer application3-6. Execute Producer application3-7. Retrieve the generated data▼4. Reference▼1. Apache Kafka Ref: Apache Kafka ▼2. Prerequisites Install Apache Kafka 4 nodes No.42 Apache Kafka Word Count – Java No.44“2-3. Installing Apache Maven” ▼3. Develop Producer Java application This producer Java application write events from 1 to 100 into a topic. 3-1. Make a directoryMake a directory to develop … Continue reading “Apache Kafka – Develop Producer Java application No.66”

Apache Kafka Word Count – Java No.44

Posted on by kumoman

How to count the words in topic by Java is shown. Contents ▼1. Steps of counting words in a topic by Java▼2. Prerequisites2-1. Installing an Apache Kafka cluster2-2. Installing IntelliJ IDEA2-3. Installing Apache Maven2-4. Setting parameters of $KAFKABROKERS and $KAFKAZOOKEEPER2-5. Creating the topic for both input and output▼3. Counting words in a topic by Java3-1. Creating a project of IntelliJ IDEA3-2. Confirming the version 1.8 of JDK is used3-3. Putting the following information while creating a project3-5. Creating log4j.properties file in src folder under main -> resources and then adding below.3-6. Creating WordCountStream.java in src folder under main -> java3-7. Writing below in WordCountStream .java3-8. Adding parameters3-9. Building Project3-10. Starting … Continue reading “Apache Kafka Word Count – Java No.44”

Install Apache Kafka 4 nodes No.42

Posted on by kumoman

How to install Apache Kafka 4 nodes is shown here. 4 ubuntu nodes are used for Apache Kafka. Contents ▼1. What is Apache Kafka?▼2. Installing Apache Kafka with 4 nodes2-1. Preparing for 4 Ubuntu 20.04.01 LTS x64 nodes2-2. Updating hosts file on 4 nodes2-3. Installing Java 8 JDK on 4 nodes2-4. Installing Apache Kafka on 4 nodes2-5. Updating environment variables on 4 nodes2-6. Only on Zookeeper node, zookeeper.properties is set.2-7. Starting Zookeeper2-8. Updating rc.local due to start of Zookeeper automatically when OS is started2-9. Configuring Kafka Brokers2-10. Updating rc.local each Brokers. the steps are the same as ones above 2-8.▼3. Creating Topic, writing and read data▼4. Reference▼1. What is Apache … Continue reading “Install Apache Kafka 4 nodes No.42”

Apache Kafka – Consumer Java – Duplicate Messages No.78

Posted on by kumoman

In this blog, Duplicate messages in Consumer are shown using Java with false of enable.auto.commit property. Contents ▼1. What is enable.auto.commit property?▼2. Prerequisites▼3. Reproducing duplicate messages in Consumer3-1. Deploying Consumer Java application3-2. Runing Producer and write 5 messages in a topic3-3. Commenting out consumer.commitSync(map) in Consumer of 3.13-4. Running the consumer which was updated in 3.3 to get messages.3-5. Finding duplicate messagesRe-running the consumer which was updated in 3.3 caused the duplicate messages.3-6. Removing the commented-out line in 3-3 and re-running this consumerWe could not find the duplicate messages.5 messages were committed.▼4. Reference▼1. What is enable.auto.commit property? Setting enable.auto.commit means that offsets are committed automatically. if it is false, committing … Continue reading “Apache Kafka – Consumer Java – Duplicate Messages No.78”