Amazon S3 物件儲存Java API操作記錄（Minio與S3 SDK兩種實現）

東北小狐狸發表於2023-04-27
S3物件JavaAPI

緣起

今年(2023年) 2月的時候做了個適配Amazon S3物件儲存介面的需求，由於4月份自學考試臨近，一直在備考就拖著沒總結記錄下，開發聯調過程中也出現過一些奇葩的問題，最近人剛從考試緩過來順手記錄一下。

S3物件儲存的基本概念

S3是什麼？

Amazon S3(Simple Storage Service)物件儲存出現得比較早且使用簡單的RESTful API，於是成為了物件儲存服務(Object Storage Service，OSS)業內的標準介面規範。

S3的邏輯模型

如下圖，我們可以把S3的儲存空間想象成無限的，想儲存一個任意格式的檔案到S3服務中，只需要知道要把它放到哪個桶(Bucket)中，它的名字（Object Id）應該是什麼。

按圖中的模型，可簡單理解為S3是由若干個桶（Bucket）組成，每個桶中包含若干個不同標識的物件（Object），還有就是統一的訪問入口(RESTful API)，這樣基本就足夠了。

Minio客戶端方式操作S3

詳細API文件：https://min.io/docs/minio/linux/developers/java/API.html

以下程式碼異常處理做了簡化，真實使用時請注意捕獲異常做處理。

引入依賴

Maven:

<dependency>
    <groupId>io.minio</groupId>
    <artifactId>minio</artifactId>
    <version>8.5.2</version>
</dependency>

Gradle:

dependencies {
    implementation("io.minio:minio:8.5.2")
}

初始化客戶端

private static final String HTTP_PROTOCOL = "http";

private MinioClient minioClient;
private String endpoint = "http://192.168.0.8:9200";
private String accessKey = "testKey";
private String secretKey = "testSecretKey";

public void init() throws MalformedURLException {
    URL endpointUrl = new URL(endpoint);
    try {
        // url上無埠號時，識別http為80埠，https為443埠
        int port = endpointUrl.getPort() != -1 ? endpointUrl.getPort() : endpointUrl.getDefaultPort();
        boolean security = HTTP_PROTOCOL.equals(endpointUrl.getProtocol()) ? false : true;
        //@formatter:off
        this.minioClient = MinioClient.builder().endpoint(endpointUrl.getHost(), port, security)
            .credentials(accessKey, secretKey).build();
        //@formatter:on
        // 忽略證照校驗，防止自簽名證照校驗失敗導致無法建立連線
        this.minioClient.ignoreCertCheck();
    } catch (Exception e) {
        e.printStackTrace();
    }
}

建桶

public boolean createBucket(String bucket) {
    try {
        minioClient.makeBucket(MakeBucketArgs.builder().bucket(bucket).build());
    } catch (Exception e) {
        e.printStackTrace();
        return false;
    }
    return true;
}

刪桶

public boolean deleteBucket(String bucket) {
    try {
        minioClient.removeBucket(RemoveBucketArgs.builder().bucket(bucket).build());
        logger.info("刪除桶[{}]成功", bucket);
    } catch (Exception e) {
        e.printStackTrace();
        return false;
    }
    return true;
}

判斷桶是否存在

public boolean bucketExists(String bucket) {
    try {
        return minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucket).build());
    } catch (Exception e) {
        e.printStackTrace();
        return false;
    }
}

上傳物件

public void upload(String bucket, String objectId, InputStream input) {
    try {
        //@formatter:off
        minioClient.putObject(PutObjectArgs.builder().bucket(bucket).object(objectId)
                              .stream(input, input.available(), -1)
                              .build());
        //@formatter:on
    } catch (Exception e) {
        e.printStackTrace();
    }
}

下載物件

提供兩個下載方法，一個將輸入流返回，另一個用引數輸出流寫出

public InputStream download(String bucket, String objectId) {
    try {
        return minioClient.getObject(GetObjectArgs.builder().bucket(bucket).object(objectId).build());
    } catch (Exception e) {
        e.printStackTrace();
    }
    return null;
}

public void download(String bucket, String objectId, OutputStream output) {
    //@formatter:off
    try (InputStream input = minioClient.getObject(
        GetObjectArgs.builder().bucket(bucket).object(objectId).build())) {
        IOUtils.copyLarge(input, output);
    } catch (Exception e) {
        e.printStackTrace();
    }
    //@formatter:on
}

刪除物件

public boolean deleteObject(String bucket, String objectId) {
    //@formatter:off
    try {
        minioClient.removeObject(RemoveObjectArgs.builder()
                                 .bucket(bucket).object(objectId).build());
    } catch (Exception e) {
        e.printStackTrace();
    }
    //@formatter:on
    return true;
}

判斷物件是否存在

public boolean objectExists(String bucket, String key) {
    //@formatter:off
    try {
        // minio客戶端未提供判斷物件是否存在的方法，此方法中呼叫出現異常時說明物件不存在
        minioClient.statObject(StatObjectArgs.builder()
                               .bucket(bucket).object(key).build());
    } catch (Exception e) {
        return false;
    }
    //@formatter:on
    return true;
}

完整程式碼

import java.io.InputStream;
import java.io.OutputStream;
import java.net.MalformedURLException;
import java.net.URL;

import org.apache.tomcat.util.http.fileupload.IOUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import io.minio.BucketExistsArgs;
import io.minio.GetObjectArgs;
import io.minio.MakeBucketArgs;
import io.minio.MinioClient;
import io.minio.PutObjectArgs;
import io.minio.RemoveBucketArgs;
import io.minio.RemoveObjectArgs;
import io.minio.StatObjectArgs;

public class S3MinioClientDemo {
    private static final Logger logger = LoggerFactory.getLogger(S3MinioClientDemo.class);
    private static final String HTTP_PROTOCOL = "http";

    private MinioClient minioClient;
    private String endpoint = "http://192.168.0.8:9200";
    private String accessKey = "testKey";
    private String secretKey = "testSecretKey";

    public void init() throws MalformedURLException {
        URL endpointUrl = new URL(endpoint);
        try {
            // url上無埠號時，識別http為80埠，https為443埠
            int port = endpointUrl.getPort() != -1 ? endpointUrl.getPort() : endpointUrl.getDefaultPort();
            boolean security = HTTP_PROTOCOL.equals(endpointUrl.getProtocol()) ? false : true;
            //@formatter:off
            this.minioClient = MinioClient.builder().endpoint(endpointUrl.getHost(), port, security)
                    .credentials(accessKey, secretKey).build();
            //@formatter:on
            // 忽略證照校驗，防止自簽名證照校驗失敗導致無法建立連線
            this.minioClient.ignoreCertCheck();
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    public boolean createBucket(String bucket) {
        try {
            boolean found = minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucket).build());
            if (found) {
                logger.info("桶名[{}]已存在", bucket);
                return false;
            }
            minioClient.makeBucket(MakeBucketArgs.builder().bucket(bucket).build());
        } catch (Exception e) {
            e.printStackTrace();
        }
        return true;
    }

    public boolean deleteBucket(String bucket) {
        try {
            minioClient.removeBucket(RemoveBucketArgs.builder().bucket(bucket).build());
            logger.info("刪除桶[{}]成功", bucket);
        } catch (Exception e) {
            e.printStackTrace();
            return false;
        }
        return true;
    }

    public boolean bucketExists(String bucket) {
        try {
            return minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucket).build());
        } catch (Exception e) {
            e.printStackTrace();
            return false;
        }
    }

    public void upload(String bucket, String objectId, InputStream input) {
        try {
          //@formatter:off
          minioClient.putObject(PutObjectArgs.builder().bucket(bucket).object(objectId)
                     .stream(input, input.available(), -1)
                     .build());
          //@formatter:on
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    public InputStream download(String bucket, String objectId) {
        try {
            return minioClient.getObject(GetObjectArgs.builder().bucket(bucket).object(objectId).build());
        } catch (Exception e) {
            e.printStackTrace();
        }
        return null;
    }

    public void download(String bucket, String objectId, OutputStream output) {
        //@formatter:off
        try (InputStream input = minioClient.getObject(
                GetObjectArgs.builder().bucket(bucket).object(objectId).build())) {
            IOUtils.copyLarge(input, output);
        } catch (Exception e) {
            e.printStackTrace();
        }
        //@formatter:on
    }

    public boolean objectExists(String bucket, String objectId) {
        //@formatter:off
        try {
            // minio客戶端未提供判斷物件是否存在的方法，此方法中呼叫出現異常時說明物件不存在
            minioClient.statObject(StatObjectArgs.builder()
                    .bucket(bucket).object(objectId).build());
        } catch (Exception e) {
            return false;
        }
        //@formatter:on
        return true;
    }

    public boolean deleteObject(String bucket, String objectId) {
        //@formatter:off
        try {
            minioClient.removeObject(RemoveObjectArgs.builder()
                    .bucket(bucket).object(objectId).build());
        } catch (Exception e) {
            e.printStackTrace();
        }
        //@formatter:on
        return true;
    }

    public void close() {
        minioClient = null;
    }

}

Amazon S3 SDK方式操作S3

官方API文件：https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html

這裡由於專案上提供的SDK和文件都是1.x的，這裡就暫時只提供1.x的程式碼

引入依賴

Maven:

<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-java-sdk-s3</artifactId>
    <version>1.11.300</version>
</dependency>

Gradle:

dependencies {   
    implementation 'com.amazonaws:aws-java-sdk-s3:1.11.300'
}

初始化客戶端

private static final Logger logger = LoggerFactory.getLogger(S3SdkDemo.class);

private AmazonS3 s3client;
private String endpoint = "http://192.168.0.8:9200";
private String accessKey = "testKey";
private String secretKey = "testSecretKey";

public void init() throws MalformedURLException {
    URL endpointUrl = new URL(endpoint);
    String protocol = endpointUrl.getProtocol();
    int port = endpointUrl.getPort() == -1 ? endpointUrl.getDefaultPort() : endpointUrl.getPort();

    ClientConfiguration clientConfig = new ClientConfiguration();
    clientConfig.setSignerOverride("S3SignerType");
    clientConfig.setProtocol(Protocol.valueOf(protocol.toUpperCase()));

    // 禁用證照檢查，避免https自簽證照校驗失敗
    System.setProperty("com.amazonaws.sdk.disableCertChecking", "true");
    // 遮蔽 AWS 的 MD5 校驗，避免校驗導致的下載丟擲異常問題
    System.setProperty("com.amazonaws.services.s3.disableGetObjectMD5Validation", "true");
    AWSCredentials awsCredentials = new BasicAWSCredentials(accessKey, secretKey);
    // 建立 S3Client 例項
    AmazonS3 s3client = new AmazonS3Client(awsCredentials, clientConfig);
    s3client.setEndpoint(endpointUrl.getHost() + ":" + port);
    s3client.setS3ClientOptions(S3ClientOptions.builder().setPathStyleAccess(true).build());
    this.s3client = s3client;
}

建桶

public boolean createBucket(String bucket)  {
    String bucketName = parseBucketName(bucket);
    try {
        if (s3client.doesBucketExist(bucketName)) {
            logger.warn("bucket[{}]已存在", bucketName);
            return false;
        }
        s3client.createBucket(bucketName);
    } catch (Exception e) {
        e.printStackTrace();
    }
    return true;
}

刪桶

public boolean deleteBucket(String bucket)  {
    try {
        s3client.deleteBucket(bucket);
        logger.info("刪除bucket[{}]成功", bucket);
    } catch (Exception e) {
        e.printStackTrace();
        return false;
    }
    return true;
}

判斷桶是否存在

public boolean bucketExists(String bucket)  {
    try {
        return s3client.doesBucketExist(bucket);
    } catch (Exception e) {
        e.printStackTrace();
    }
    return false;
}

上傳物件

public void upload(String bucket, String objectId, InputStream input)  {
    try {
        // 建立檔案上傳的後設資料
        ObjectMetadata meta = new ObjectMetadata();
        // 設定檔案上傳長度
        meta.setContentLength(input.available());
        // 上傳
        s3client.putObject(bucket, objectId, input, meta);
    } catch (Exception e) {
        e.printStackTrace();
    }
}

下載物件

public InputStream download(String bucket, String objectId)  {
    try {
        S3Object o = s3client.getObject(bucket, objectId);
        return o.getObjectContent();
    } catch (Exception e) {
        e.printStackTrace();
    }
    return null;
}

public void download(String bucket, String objectId, OutputStream out)  {
    S3Object o = s3client.getObject(bucket, objectId);
    try (InputStream in = o.getObjectContent()) {
        IOUtils.copyLarge(in, out);
    } catch (Exception e) {
        e.printStackTrace();
    }
}

刪除物件

public boolean deleteObject(String bucket, String objectId)  {
    try {
        s3client.deleteObject(bucket, objectId);
    } catch (Exception e) {
        e.printStackTrace();
        return false;
    }
    return true;
}

判斷物件是否存在

public boolean existObject(String bucket, String objectId)  {
    try {
        return s3client.doesObjectExist(bucket, objectId);
    } catch (Exception e) {
        e.printStackTrace();
        return false;
    }
}

完整程式碼

import java.io.InputStream;
import java.io.OutputStream;
import java.net.MalformedURLException;
import java.net.URL;

import org.apache.tomcat.util.http.fileupload.IOUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import com.amazonaws.ClientConfiguration;
import com.amazonaws.Protocol;
import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.S3ClientOptions;
import com.amazonaws.services.s3.model.ObjectMetadata;
import com.amazonaws.services.s3.model.S3Object;

/**
 * S3物件儲存官方SDK實現
 *
 * @author ZhangChenguang
 * @date 2023年2月2日
 */
@SuppressWarnings("deprecation")
public class S3SdkDemo {
    private static final Logger logger = LoggerFactory.getLogger(S3SdkDemo.class);

    private AmazonS3 s3client;
    private String endpoint = "http://192.168.0.8:9200";
    private String accessKey = "testKey";
    private String secretKey = "testSecretKey";
    
    public void init() throws MalformedURLException {
        URL endpointUrl = new URL(endpoint);
        String protocol = endpointUrl.getProtocol();
        int port = endpointUrl.getPort() == -1 ? endpointUrl.getDefaultPort() : endpointUrl.getPort();

        ClientConfiguration clientConfig = new ClientConfiguration();
        clientConfig.setSignerOverride("S3SignerType");
        clientConfig.setProtocol(Protocol.valueOf(protocol.toUpperCase()));

        // 禁用證照檢查，避免https自簽證照校驗失敗
        System.setProperty("com.amazonaws.sdk.disableCertChecking", "true");
        // 遮蔽 AWS 的 MD5 校驗，避免校驗導致的下載丟擲異常問題
        System.setProperty("com.amazonaws.services.s3.disableGetObjectMD5Validation", "true");
        AWSCredentials awsCredentials = new BasicAWSCredentials(accessKey, secretKey);
        // 建立 S3Client 例項
        AmazonS3 s3client = new AmazonS3Client(awsCredentials, clientConfig);
        s3client.setEndpoint(endpointUrl.getHost() + ":" + port);
        s3client.setS3ClientOptions(S3ClientOptions.builder().setPathStyleAccess(true).build());
        this.s3client = s3client;
    }

    public boolean createBucket(String bucket)  {
        try {
            s3client.createBucket(bucket);
        } catch (Exception e) {
            e.printStackTrace();
        }
        return true;
    }

    public boolean deleteBucket(String bucket)  {
        try {
            s3client.deleteBucket(bucket);
            logger.info("刪除bucket[{}]成功", bucket);
        } catch (Exception e) {
            e.printStackTrace();
            return false;
        }
        return true;
    }
    
    public boolean bucketExists(String bucket)  {
        try {
            return s3client.doesBucketExist(bucket);
        } catch (Exception e) {
           e.printStackTrace();
        }
        return false;
    }

    public void upload(String bucket, String objectId, InputStream input)  {
        try {
            // 建立檔案上傳的後設資料
            ObjectMetadata meta = new ObjectMetadata();
            // 設定檔案上傳長度
            meta.setContentLength(input.available());
            // 上傳
            s3client.putObject(bucket, objectId, input, meta);
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    public InputStream download(String bucket, String objectId)  {
        try {
            S3Object o = s3client.getObject(bucket, objectId);
            return o.getObjectContent();
        } catch (Exception e) {
            e.printStackTrace();
        }
        return null;
    }

    public void download(String bucket, String objectId, OutputStream out)  {
        S3Object o = s3client.getObject(bucket, objectId);
        try (InputStream in = o.getObjectContent()) {
            IOUtils.copyLarge(in, out);
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    public boolean existObject(String bucket, String objectId)  {
        try {
            return s3client.doesObjectExist(bucket, objectId);
        } catch (Exception e) {
            e.printStackTrace();
            return false;
        }
    }

    public boolean deleteObject(String bucket, String objectId)  {
        try {
            s3client.deleteObject(bucket, objectId);
        } catch (Exception e) {
            e.printStackTrace();
            return false;
        }
        return true;
    }

    public void close()  {
        s3client = null;
    }
}

遇到的問題

1、bucket名稱必須是小寫，不支援下劃線

  • 處理方式：寫方法轉換下bucket名稱，將大寫轉小寫，將下劃線替換為中劃線。

2、minio客戶端下載非官方S3儲存的檔案時，如果響應頭的Content-Length與實際檔案大小不符，會導致minio客戶端包裝的okhttp3報錯

報錯資訊：

Caused by: java.net.ProtocolException: unexpected end of stream
        at okhttp3.internal.http1.Http1ExchangeCodec$FixedLengthSource.read(Http1ExchangeCodec.java:430) ~[okhttp-3.14.9.jar:?]
        at okhttp3.internal.connection.Exchange$ResponseBodySource.read(Exchange.java:286) ~[okhttp-3.14.9.jar:?]
        at okio.RealBufferedSource$1.read(RealBufferedSource.java:447) ~[okio-1.17.2.jar:?]
        at com.jiuqi.nr.file.utils.FileUtils.writeInput2Output(FileUtils.java:83) ~[nr.file-2.5.7.jar:?]
        at com.jiuqi.nr.file.impl.FileAreaServiceImpl.download(FileAreaServiceImpl.java:395) ~[nr.file-2.5.7.jar:?]
        ... 122 more

抓包發現問題的圖：

最終換成了S3官方SDK可用了。

PS：客戶現場部署的S3是浪潮公司提供的，如果現場遇到這個情況，就不要固執去找對方對線了，完全沒用。。

總結

S3儲存的基本操作就記錄到這裡了，由於沒有S3儲存就沒嘗試官方SDK的V2版本，由於這些程式碼是總結時從業務程式碼裡抽取出來的，可能會有點問題，但大體思路已經有了。

希望對讀者有所用處，覺得寫得不錯和有幫到你，歡迎點個贊，您的支援就是我的鼓勵！

相關文章