Category: Big Data

Python – Put data in MySQL into Azure Blob Storage No.102

Posted on by kumoman

How to put the results of select query for MySQL into Azure Blob Storage using Python is shown in this blog. Contents ▼1. Saving data in MySQL into Azure Blob Storage▼2. Prerequisites▼3. Python code to save data in MySQL into Azure Blob Storage3-1. Creating python code3-2. Results of this code▼4. Reference▼1. Saving data in MySQL into Azure Blob Storage In the past, how to save data in MySQL into a local file using Python was shown in the following blog. in this case, the target of saving data is Azure Blob Storage. Python – Save data of MySQL into a local file  No.99 ▼2. Prerequisites How to create an environment … Continue reading “Python – Put data in MySQL into Azure Blob Storage No.102”

Python -Save data in SQL Server into Azure Blob Storage No96

Posted on by kumoman

How to save the results of select statement into Azure Blob Storage after connecting to SQL Server on Ubuntu through pyodbc using python is shown in this blog. Contents ▼1. Saving data in SQL Server on Ubuntu into Azure Blob Storage▼2. Prerequisites▼3. Developing Python code to save the results of select query in SQL Server into Azure Blob Storage3-1. Creating Python code3-2. Results of this code▼4. Reference▼1. Saving data in SQL Server on Ubuntu into Azure Blob Storage Based on the previous blog using Java as below, pyhon code is shown to save the results of select statement into Azure Blob Storage. Java – Save the results of select query … Continue reading “Python -Save data in SQL Server into Azure Blob Storage No96”

Java –Save records for MySQL into Azure Blob Storage No.86 

Posted on by kumoman

In this blog, how to save the results of select query for MySQL into Azure Blob Storage by Java. Contents ▼1. Prerequisites1-1. Install MySQL and create a table with some records▼2. Save the results of select for MySQL on Ubuntu into Azure Blob Storage by Java2-1. Create an Apache Maven project2-2. Remove the existed both App.java and AppTest.java2-3. Start Visual Studio Code2-4. Update pom.xml as below and save it (Ctrl+S)2-5. Save the results of select for MySQL into Azure Blob Storage2-6. Run the code▼3. Reference▼1. Prerequisites 1-1. Install MySQL and create a table with some records Java – Connect to MySQL on Ubuntu and run select query No.80 # Create a table … Continue reading “Java –Save records for MySQL into Azure Blob Storage No.86 “

Apache Kafka – Develop Producer Java application No.66

Posted on by kumoman

How to develop Producer application by Java on Visual Studio Code is shown in this blog. Contents ▼1. Apache Kafka ▼2. Prerequisites▼3. Develop Producer Java application3-1. Make a directoryMake a directory to develop an application.3-2. Create a project using Apache Maven3-3. Remove the App.java and AppTest.java files3-4. Update pom.xml file3-5. Develop Producer application3-6. Execute Producer application3-7. Retrieve the generated data▼4. Reference▼1. Apache Kafka Ref: Apache Kafka ▼2. Prerequisites Install Apache Kafka 4 nodes No.42 Apache Kafka Word Count – Java No.44“2-3. Installing Apache Maven” ▼3. Develop Producer Java application This producer Java application write events from 1 to 100 into a topic. 3-1. Make a directoryMake a directory to develop … Continue reading “Apache Kafka – Develop Producer Java application No.66”

Apache Kafka Word Count – Java No.44

Posted on by kumoman

How to count the words in topic by Java is shown. Contents ▼1. Steps of counting words in a topic by Java▼2. Prerequisites2-1. Installing an Apache Kafka cluster2-2. Installing IntelliJ IDEA2-3. Installing Apache Maven2-4. Setting parameters of $KAFKABROKERS and $KAFKAZOOKEEPER2-5. Creating the topic for both input and output▼3. Counting words in a topic by Java3-1. Creating a project of IntelliJ IDEA3-2. Confirming the version 1.8 of JDK is used3-3. Putting the following information while creating a project3-5. Creating log4j.properties file in src folder under main -> resources and then adding below.3-6. Creating WordCountStream.java in src folder under main -> java3-7. Writing below in WordCountStream .java3-8. Adding parameters3-9. Building Project3-10. Starting … Continue reading “Apache Kafka Word Count – Java No.44”

Install Apache Kafka 4 nodes No.42

Posted on by kumoman

How to install Apache Kafka 4 nodes is shown here. 4 ubuntu nodes are used for Apache Kafka. Contents ▼1. What is Apache Kafka?▼2. Installing Apache Kafka with 4 nodes2-1. Preparing for 4 Ubuntu 20.04.01 LTS x64 nodes2-2. Updating hosts file on 4 nodes2-3. Installing Java 8 JDK on 4 nodes2-4. Installing Apache Kafka on 4 nodes2-5. Updating environment variables on 4 nodes2-6. Only on Zookeeper node, zookeeper.properties is set.2-7. Starting Zookeeper2-8. Updating rc.local due to start of Zookeeper automatically when OS is started2-9. Configuring Kafka Brokers2-10. Updating rc.local each Brokers. the steps are the same as ones above 2-8.▼3. Creating Topic, writing and read data▼4. Reference▼1. What is Apache … Continue reading “Install Apache Kafka 4 nodes No.42”

Apache Kafka – Consumer Java – Duplicate Messages No.78

Posted on by kumoman

In this blog, Duplicate messages in Consumer are shown using Java with false of enable.auto.commit property. Contents ▼1. What is enable.auto.commit property?▼2. Prerequisites▼3. Reproducing duplicate messages in Consumer3-1. Deploying Consumer Java application3-2. Runing Producer and write 5 messages in a topic3-3. Commenting out consumer.commitSync(map) in Consumer of 3.13-4. Running the consumer which was updated in 3.3 to get messages.3-5. Finding duplicate messagesRe-running the consumer which was updated in 3.3 caused the duplicate messages.3-6. Removing the commented-out line in 3-3 and re-running this consumerWe could not find the duplicate messages.5 messages were committed.▼4. Reference▼1. What is enable.auto.commit property? Setting enable.auto.commit means that offsets are committed automatically. if it is false, committing … Continue reading “Apache Kafka – Consumer Java – Duplicate Messages No.78”

Java – Download files from Azure Blob Storage No.27

Posted on by kumoman

In this blog, How to download files from Azure Blob Storage using Java is shown. Contents ▼1. List file names and download files from Azure Blob Storage▼2. Prerequisites2-1. Installing Java 8 JDK2-2. Free trial Azure account for Azure Blob Storage2-3. Connecting to Azure Blob Storage using Azure Storage Explorer ▼3. List files and download these files from Azure Blob Storage3-1. Starting IntelliJ IDEA and confirming JDK version3-2. Updating pom.xml for operations of Azure Blob Storage3-3. Creating Java Code▼4. Reference▼1. List file names and download files from Azure Blob Storage Using IntelliJ IDEA 2021.1.2 (Community Edition) on Ubuntu , Java application lists files on Azure Blob Storage and downloads these files. … Continue reading “Java – Download files from Azure Blob Storage No.27”

Java – Save the results of select query on SQL Server into Azure Blob Storage No.75

Posted on by kumoman

I will show Java code that save results of select query on SQL Server into Azure Blob Storage. Contents ▼1. Copy data on SQL Server into Azure Blob Storage▼2. Prerequisites2-1. Installation of SQL Server on UbuntuRef: Quickstart: Install SQL Server and create a database on Ubuntu2-2. Installation of Visual Studio Code on Ubuntu2-3. Installation of Maven on Ubuntu2-4. Creating a sample data on SQL Server2-5. Executing a select query on SQL Server for Ubuntu▼3. Java code to copy results of a select query on SQL Server into Azure Blob Storage3-1. Making a directory3-2. Developing an Apache Maven project3-3. Removing the existing App.java and AppTest.java3-4. Starting Visual Studio Code3-5. Adding SQL … Continue reading “Java – Save the results of select query on SQL Server into Azure Blob Storage No.75”

Java – Save the results of select on Azure Cosmos DB SQL API into Azure Blob Storage No.74

Posted on by kumoman

I will show Java code that save results of select query as a JSON file into Azure Blob Storage. In the previous blog, I shared how to save results of select query on Cosmos DB as a JSON file into the local path. in this blog, I will save results into Azure Blob Storage. Contents ▼1. Migration data from Cosmos DB to Azure Blob Storage▼2. Prerequisites2-1. Free trial Azure Cosmos DB Account2-2. Install Visual Studio Code2-3. Install Maven2-4. Create a sample data▼3. Save the results of select on Azure Cosmos DB SQL API as JSON file into Azure Blob Storage using Java Code3-1. Make a directory3-2. Deploy an Apache Maven project3-3. Remove … Continue reading “Java – Save the results of select on Azure Cosmos DB SQL API into Azure Blob Storage No.74”