Java – Save the results of select on Azure Cosmos DB SQL API into Azure Blob Storage No.74

I will show Java code that save results of select query as a JSON file into Azure Blob Storage.

In the previous blog, I shared how to save results of select query on Cosmos DB as a JSON file into the local path. in this blog, I will save results into Azure Blob Storage.

▼1. Migration data from Cosmos DB to Azure Blob Storage

We can migrate data from Cosmos DB to Azure Blob Storage using dt.exe or dtui.exe easily. this document is helpful for these migration. but I will share you how to migrate data on Azure Cosmos DB into Azure Blob Storage using Java.

Ref:  Tutorial: Use Data migration tool to migrate your data to Azure Cosmos DB


▼2. Prerequisites

2-1. Free trial Azure Cosmos DB Account

You can use trial free Cosmos DB Account
https://azure.microsoft.com/ja-jp/try/cosmosdb/

2-2. Install Visual Studio Code

sudo snap install --classic code

https://code.visualstudio.com/docs/setup/linux

2-3. Install Maven

Ref: Apache Kafka Word Count 実装 – Java No.44
line of 2-3

2-4. Create a sample data

Ref: Java Azure Cosmos DB (SQL API) Database の作成方法 No.14

GeneratingData

▼3. Save the results of select on Azure Cosmos DB SQL API as JSON file into Azure Blob Storage using Java Code

3-1. Make a directory

Make a directory to create an application.

mkidr cosmosdbtoblob
cd cosmosdbtoblob

3-2. Deploy an Apache Maven project

cd cosmosdbtoblob
mvn archetype:generate  -DinteractiveMode=false -DgroupId=org.example.cosmosdb.sql -DartifactId=sqldemo -DarchetypeArtifactId=maven-archetype-quickstart
(A part of output)

[INFO] Scanning for projects...
[INFO] 
[INFO] ------------------< org.apache.maven:standalone-pom >--------------
[INFO] Building Maven Stub Project (No POM) 1
[INFO] --------------------------------[ pom ]----------------------------
[INFO] 
[INFO] >>> maven-archetype-plugin:3.2.0:generate (default-cli) > generate-sources @ standalone-pom >>>
[INFO] 
[INFO] <<< maven-archetype-plugin:3.2.0:generate (default-cli) < generate-sources @ standalone-pom <<<
[INFO] 
[INFO] 
[INFO] --- maven-archetype-plugin:3.2.0:generate (default-cli) @ standalone-pom ---
[INFO] Generating project in Batch mode
[INFO] -------------------------------------------------------------------
[INFO] Using following parameters for creating project from Old (1.x) Archetype: maven-archetype-quickstart:1.0
[INFO] -------------------------------------------------------------------
[INFO] Parameter: basedir, Value: /home/xxx/cosmosdbtoblob
[INFO] Parameter: package, Value: org.example.cosmosdb.sql
[INFO] Parameter: groupId, Value: org.example.cosmosdb.sql
[INFO] Parameter: artifactId, Value: sqldemo
[INFO] Parameter: packageName, Value: org.example.cosmosdb.sql
[INFO] Parameter: version, Value: 1.0-SNAPSHOT
[INFO] project created from Old (1.x) Archetype in dir: /home/xxx/cosmosdbtoblob/sqldemo
[INFO] -------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] -------------------------------------------------------------------
[INFO] Total time:  3.706 s
[INFO] Finished at: 2022-11-12T19:03:12+09:00
[INFO] -------------------------------------------------------------------

3-3. Remove the existing App.java and AppTest.java

rm ./sqldemo/src/main/java/org/example/comsosdb/sql/App.java
rm ./sqldemo/src/test/java/org/example/cosmosdb/sql/AppTest.java

3-4. Start Visual Studio Code

cd ./sqldemo
code .

3-5. Add Azure Cosmos DB and Azure Blob Storage libraries into pom.xml and save it (Ctrl+S)

  <properties>
    <maven.compiler.source>1.8</maven.compiler.source>
    <maven.compiler.target>1.8</maven.compiler.target>
  </properties>

  <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>com.azure</groupId>
      <artifactId>azure-cosmos</artifactId>
      <version>4.30.1</version>
    </dependency>
    <dependency>
      <groupId>com.azure</groupId>
      <artifactId>azure-identity</artifactId>
      <version>1.6.1</version>
    </dependency>
    <dependency>
      <groupId>com.azure</groupId>
      <artifactId>azure-storage-blob</artifactId>
      <version>12.13.0</version>
    </dependency>
  </dependencies>

3-6. Coding for saving results of select query on Azure Cosmos DB as a JSON file into Azure Blob Storage

copycosmosdbtoblob.java is created.

package org.example.cosmosdb.sql;

import com.azure.cosmos.*;
import com.azure.cosmos.models.CosmosQueryRequestOptions;
import com.azure.cosmos.util.CosmosPagedIterable;

import java.io.File;
import java.io.FileNotFoundException;
import java.io.PrintStream;
import com.azure.storage.blob.*;

public class copycosmosdbtoblob {
    public static class Family {
        public String id;
        public String lastName;
        public boolean isRegistered;

        public String getId() {
            return id;
        }

        public String getLastName() {
            return lastName;
        }

        public Family(String idv, String lastNamev, boolean isv) {
            id = idv;
            lastName = lastNamev;
            isRegistered = isv;
        }

        public String toJson(String idv, String lastNamev, boolean isv) {
            return "{\"id\":" + idv + ",\"lastName\":\"" + lastNamev + "\",\"isRegistered\":" + isv + "}";
        }

        public Family() {
        }
    }

    public static void main(String[] args) {
        String databaseName = "db1";
        String endpoint = "https://xxx.documents.azure.com:443";
        String masterkey = "yourkey";
        String containerName = "coll1";

        String connectStr = "DefaultEndpointsProtocol=https;AccountName=xxx;AccountKey=yourkey;EndpointSuffix=core.windows.net";
        String blobcontainerName = "files";
        String fileName = "sample5.json";
        String localPath = "/home/xxx/cosmosdbtoblob/sqldemo/";

        // Get a reference to a container
        BlobServiceClient BlobServiceClient = new BlobServiceClientBuilder().connectionString(connectStr).buildClient();
        BlobContainerClient containerClinet = BlobServiceClient.getBlobContainerClient(blobcontainerName);

        try (CosmosClient cosmosClient = new CosmosClientBuilder().endpoint(endpoint).key(masterkey)
                .gatewayMode(GatewayConnectionConfig.getDefaultConfig()).consistencyLevel(ConsistencyLevel.SESSION)
                .connectionSharingAcrossClientsEnabled(true).contentResponseOnWriteEnabled(true)
                .userAgentSuffix("Applicationtest").buildClient()) {
            CosmosDatabase database = cosmosClient.getDatabase(databaseName);
            CosmosContainer container = database.getContainer(containerName);

            // run select query
            String query = "SELECT * FROM c";
            CosmosQueryRequestOptions options = new CosmosQueryRequestOptions();
            options.setMaxDegreeOfParallelism(2);
            CosmosPagedIterable<Family> results = container.queryItems(query, options, Family.class);

            File file = new File("/home/xxx/cosmosdbtoblob/sqldemo/sample5.json");
            try {
                PrintStream ps = new PrintStream(file);
                System.setOut(ps);
            } catch (FileNotFoundException e) {
                e.printStackTrace();
            }

            System.out.print("{\"Records\":[");
            for (Family result : results) {
                // if first line, no comma
                if (result.getId().equals("1")) {
                    System.out.print(result.toJson(result.getId(), result.getLastName(), result.isRegistered));
                } else {
                    System.out.print("," + result.toJson(result.getId(), result.getLastName(), result.isRegistered));
                }
            }
        } catch (CosmosException e) {
            e.printStackTrace();
            System.err.println(e.getMessage());
        }
        System.out.println("]}");
        // close printstream
        System.out.close();

    BlobClient blobClient = containerClinet.getBlobClient(fileName);

    System.out.println("Uploading to Azure Blob storage as blob:\n\t" + blobClient.getBlobUrl());

    // Upload a file to the blob
    blobClient.uploadFromFile(localPath + "/" + fileName);
    }
}

3-7. Check the JSON file in the container on Azure Blob Storage

You may be able to see sample5.json as below.

{"Records":[{"id":1,"lastName":"Travase","isRegistered":false},{"id":2,"lastName":"Ken","isRegistered":true},{"id":3,"lastName":"Sai","isRegistered":false}]}

▼4. Reference

  1. Azure Cosmos DB を無料で試す https://docs.microsoft.com/ja-jp/azure/cosmos-db/optimize-dev-test#try-azure-cosmos-db-for-free
  2. https://code.visualstudio.com/docs/setup/linux
  3. Apache Kafka Word Count 実装 – Java No.44
  4. Java Azure Cosmos DB (SQL API) Database の作成方法 No.14
  5. Java Azure Cosmos DB (SQL) 検索結果をテキストに保存 No.72
  6. Java – Azure Cosmos DB (SQL) : save results of select into a JSON file No.73

That’s all. Have a nice day, ahead !!!

Leave a Reply

Your email address will not be published. Required fields are marked *