【物件儲存】Minio本地執行和 golang客戶端基本操作

Mr.YF發表於2023-10-19

執行環境

OS和Golang版本：

go version go1.21.0 darwin/arm64

安裝

原始碼安裝

下載最新版本的原始碼，地址https://github.com/minio/minio後編譯

cd minio
go build main.go
# 得到 116M Oct 19 15:49 main

把 main 改名為 minio

二進位制安裝

參考https://www.minio.org.cn/docs/minio/macos/index.html的安裝步驟。

啟動

minio server --address=0.0.0.0:8877 ./data

控制檯輸出如下資訊

➜  minio minio server --address=0.0.0.0:8877 ./data


┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ You are running an older version of MinIO released 2 years ago ┃
┃ Update: Run `mc admin update`                                  ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

API: http://0.0.0.0:8877
RootUser: minioadmin
RootPass: minioadmin

Console: http://10.78.14.68:56056 http://192.168.255.10:56056 http://127.0.0.1:56056
RootUser: minioadmin
RootPass: minioadmin

Command-line: https://docs.min.io/docs/minio-client-quickstart-guide
   $ mc alias set myminio http://0.0.0.0:8877 minioadmin minioadmin

Documentation: https://docs.min.io

啟動控制檯頁面

使用瀏覽器開啟 http://127.0.0.1:56056，使用賬號密碼登入（minioadmin/minioadmin）

建立aksk

在左側選單中選擇 Account，點選Create Service Account，建立得到一個 aksk

建立Bucket

mc命令

mc命令安裝

參考https://min.io/docs/minio/linux/reference/minio-mc.html

mc命令執行

mc給本地的minio連結做一個別名

➜  ~ mc alias set myminio http://0.0.0.0:8877
Enter Access Key: KLN00KFT1K5EP9I39I9N
Enter Secret Key:
Added `myminio` successfully.

mc檢視 minio 節點資訊

➜  ~ mc admin info myminio
●  0.0.0.0:8877
   Uptime: 3 hours
   Version: 2021-08-05T22:01:19Z

mc 對 minio 做 ping 檢查連結是否 ok

➜  minio mc ping myminio
  1: http://0.0.0.0:8877:8877   min=9.36ms     max=9.36ms     average=9.36ms     errors=0   roundtrip=9.36ms
  2: http://0.0.0.0:8877:8877   min=0.64ms     max=9.36ms     average=5.00ms     errors=0   roundtrip=0.64ms

mc上傳檔案

➜  minio mc cp ./minio-dev.yaml myminio/mybucket/3.yaml
...o-dev.yaml: 1.46 KiB / 1.46 KiB ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 49.36 KiB/s 0s

mc下載檔案

➜  minio mc cp myminio/mybucket/3.yaml ./3-get.yaml
...ket/3.yaml: 1.46 KiB / 1.46 KiB ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 67.56 KiB/s 0s

mc監聽桶變化

➜  minio mc watch myminio/mybucket/
[2023-10-19T07:09:05.751Z] 1.5 KiB s3:ObjectCreated:Put http://0.0.0.0:8877/mybucket/3.yaml

[2023-10-19T07:27:35.675Z]   36 B s3:ObjectCreated:Put http://0.0.0.0:8877/mybucket/test/1.txt
[2023-10-19T07:28:46.813Z]        s3:ObjectAccessed:Get http://0.0.0.0:8877/mybucket/test/1.txt
[2023-10-19T07:28:58.157Z]        s3:ObjectAccessed:Head http://0.0.0.0:8877/mybucket/test/1.txt
[2023-10-19T08:11:38.631Z]        s3:ObjectAccessed:Head http://0.0.0.0:8877/mybucket/3.yaml
[2023-10-19T08:11:38.633Z]        s3:ObjectAccessed:Head http://0.0.0.0:8877/mybucket/3.yaml
[2023-10-19T08:11:38.634Z]        s3:ObjectAccessed:Head http://0.0.0.0:8877/mybucket/3.yaml
[2023-10-19T08:11:38.638Z]        s3:ObjectAccessed:Head http://0.0.0.0:8877/mybucket/3.yaml
[2023-10-19T08:11:38.644Z]        s3:ObjectAccessed:Head http://0.0.0.0:8877/mybucket/3.yaml
[2023-10-19T08:11:38.653Z]        s3:ObjectAccessed:Head http://0.0.0.0:8877/mybucket/3.yaml
[2023-10-19T08:11:38.655Z]        s3:ObjectAccessed:Get http://0.0.0.0:8877/mybucket/3.yaml

mc檢視桶統計資訊

➜  minio mc stat myminio/mybucket/
Name      : 2.yaml
Date      : 2023-10-19 14:59:57 CST
Size      : 1.5 KiB
ETag      : 34095c50340c4381e0fdc5fd61eecc76
Type      : file
Metadata  :
  Content-Type: text/yaml

Name      : 3.yaml
Date      : 2023-10-19 15:09:05 CST
Size      : 1.5 KiB
ETag      : 34095c50340c4381e0fdc5fd61eecc76
Type      : file
Metadata  :
  Content-Type: text/yaml

Name      : minio-dev.yaml
Date      : 2023-10-19 14:54:23 CST
Size      : 1.5 KiB
ETag      : 34095c50340c4381e0fdc5fd61eecc76
Type      : file
Metadata  :
  Content-Type: text/yaml

Name      : test/
Date      : 2023-10-19 16:13:06 CST
Type      : folder

這個etag 是怎麼計算的呢？其實就是檔案的 md5值

➜  minio md5 3-get.yaml
MD5 (3-get.yaml) = 34095c50340c4381e0fdc5fd61eecc76

mc列出桶中的檔案

➜  minio mc ls myminio/mybucket/
[2023-10-19 14:59:57 CST] 1.5KiB STANDARD 2.yaml
[2023-10-19 15:09:05 CST] 1.5KiB STANDARD 3.yaml
[2023-10-19 14:54:23 CST] 1.5KiB STANDARD minio-dev.yaml
[2023-10-19 16:18:25 CST]     0B test/

服務端檔案儲存

啟動時，將minio 的工作目錄設定到 data 下，在 data 目錄下主要有兩個目錄

.minio.sys 是minio系統資訊
- 包括桶定義和桶中的檔案索引目錄 ./.minio.sys/buckets/mybucket
- 賬號資訊和iam資訊 ./.minio.sys/config/iam/service-accounts/KLN00KFT1K5EP9I39I9N
mybucket 是建立的一個桶名稱

➜  data find .
.
./.minio.sys
./.minio.sys/buckets
./.minio.sys/buckets/.usage-cache.bin
./.minio.sys/buckets/.minio.sys
./.minio.sys/buckets/.minio.sys/buckets
./.minio.sys/buckets/.minio.sys/buckets/.usage-cache.bin
./.minio.sys/buckets/.minio.sys/buckets/.usage-cache.bin/fs.json
./.minio.sys/buckets/.minio.sys/buckets/.bloomcycle.bin
./.minio.sys/buckets/.minio.sys/buckets/.bloomcycle.bin/fs.json
./.minio.sys/buckets/.minio.sys/buckets/mybucket
./.minio.sys/buckets/.minio.sys/buckets/mybucket/.usage-cache.bin
./.minio.sys/buckets/.minio.sys/buckets/mybucket/.usage-cache.bin/fs.json
./.minio.sys/buckets/.minio.sys/buckets/.usage.json
./.minio.sys/buckets/.minio.sys/buckets/.usage.json/fs.json
./.minio.sys/buckets/.bloomcycle.bin
./.minio.sys/buckets/mybucket
./.minio.sys/buckets/mybucket/.usage-cache.bin
./.minio.sys/buckets/mybucket/test
./.minio.sys/buckets/mybucket/test/1.txt
./.minio.sys/buckets/mybucket/test/1.txt/fs.json
./.minio.sys/buckets/mybucket/minio-dev.yaml
./.minio.sys/buckets/mybucket/minio-dev.yaml/fs.json
./.minio.sys/buckets/mybucket/3.yaml
./.minio.sys/buckets/mybucket/3.yaml/fs.json
./.minio.sys/buckets/mybucket/2.yaml
./.minio.sys/buckets/mybucket/2.yaml/fs.json
./.minio.sys/buckets/mybucket/.metadata.bin
./.minio.sys/buckets/.tracker.bin
./.minio.sys/buckets/.usage.json
./.minio.sys/config
./.minio.sys/config/config.json
./.minio.sys/config/iam
./.minio.sys/config/iam/service-accounts
./.minio.sys/config/iam/service-accounts/KLN00KFT1K5EP9I39I9N
./.minio.sys/config/iam/service-accounts/KLN00KFT1K5EP9I39I9N/identity.json
./.minio.sys/config/iam/policydb
./.minio.sys/config/iam/policydb/sts-users
./.minio.sys/config/iam/policydb/sts-users/P1Y2O1AO30UYBE2UODBY.json
./.minio.sys/config/iam/policydb/users
./.minio.sys/config/iam/policydb/users/ak00123456789.json
./.minio.sys/config/iam/users
./.minio.sys/config/iam/users/ak00123456789
./.minio.sys/config/iam/users/ak00123456789/identity.json
./.minio.sys/config/iam/format.json
./.minio.sys/config/iam/sts
./.minio.sys/config/iam/sts/P1Y2O1AO30UYBE2UODBY
./.minio.sys/config/iam/sts/P1Y2O1AO30UYBE2UODBY/identity.json
./.minio.sys/multipart
./.minio.sys/format.json
./.minio.sys/tmp
./.minio.sys/tmp/07c1ffc6-ae6f-4a99-a57e-cb5e55530603
./mybucket
./mybucket/test
./mybucket/test/1.txt
./mybucket/minio-dev.yaml
./mybucket/3.yaml
./mybucket/2.yaml

aws-s3操作檔案

aws-s3的 sdk 程式碼簡單包裝

go.mod

module minio-demo

go 1.18

require (
    github.com/aws/aws-sdk-go v1.43.21
)

aws_s3.go

package minio

import (
    "bytes"
    "io"
    "time"

    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/credentials"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/s3"
)

const (
    //token暫時為空
    DefaultToken = ""
    //測試用的regin，呼叫方需自行配置
    DefaultRegion = "us-east-1"
)

// AwsS3 aws s3服務應用層客戶端
type AwsS3 struct {
    SecretId  string
    SecretKey string
    Region    string
    Bucket    string
    Endpoint  string
    Token     string
    Client    *s3.S3
}

// NewAwsS3 建立aws s3例項
func NewAwsS3(secretId, secretKey, region, bucket, endpoint, token string) *AwsS3 {
    var awsS3Instance AwsS3
    awsS3Instance.SecretId = secretId
    awsS3Instance.SecretKey = secretKey
    awsS3Instance.Region = region
    awsS3Instance.Bucket = bucket
    awsS3Instance.Endpoint = endpoint
    awsS3Instance.Token = token
    config := &aws.Config{
        Credentials: credentials.NewStaticCredentials(secretId, secretKey, token),
        Region:      aws.String(region),
        Endpoint:    aws.String(endpoint),
        //DisableSSL:       aws.Bool(true),
        S3ForcePathStyle: aws.Bool(true),
    }
    sess := session.Must(session.NewSession(config))
    awsS3Instance.Client = s3.New(sess)
    return &awsS3Instance
}

// PutObject 根據內容上傳檔案物件
func (a *AwsS3) PutObject(awsPath string, content []byte) (string, error) {
    putObjectInput := &s3.PutObjectInput{
        Bucket: aws.String(a.Bucket),
        Key:    aws.String(awsPath),
        Body:   aws.ReadSeekCloser(bytes.NewReader(content)),
    }
    resp, err := a.Client.PutObject(putObjectInput)
    if err != nil {
        return "", err
    }
    return *(resp.ETag), nil
}

// GetObject 下載檔案物件內容
func (a *AwsS3) GetObject(awsPath string) ([]byte, string, error) {
    getObjectInput := &s3.GetObjectInput{
        Bucket: aws.String(a.Bucket),
        Key:    aws.String(awsPath),
    }
    resp, err := a.Client.GetObject(getObjectInput)
    if err != nil {
        return nil, "", err
    }
    content, err := io.ReadAll(resp.Body)
    if err != nil {
        return nil, "", err
    }
    return content, *(resp.ETag), nil
}

// DeleteObject 刪除檔案物件
func (a *AwsS3) DeleteObject(awsPath string) error {
    deleteObject := &s3.DeleteObjectInput{
        Bucket: aws.String(a.Bucket),
        Key:    aws.String(awsPath),
    }
    _, err := a.Client.DeleteObject(deleteObject)
    if err != nil {
        return err
    }
    return nil
}

// HeadObject 獲取物件後設資料資訊，包括md5和上次修改時間
func (a *AwsS3) HeadObject(awsPath string) (string, *time.Time, error) {
    headObject := &s3.HeadObjectInput{
        Bucket: aws.String(a.Bucket),
        Key:    aws.String(awsPath),
    }
    resp, err := a.Client.HeadObject(headObject)
    if err != nil {
        return "", nil, err
    }
    return *(resp.ETag), resp.LastModified, nil
}

minio_test.go

package minio

import (
    "os"
    "testing"
)

// minio測試配置
var (
    SecretId  = "KLN00KFT1K5EP9I39I9N"
    SecretKey = "k******j"
    Region    = DefaultRegion
    Bucket    = "mybucket"
    Token     = DefaultToken
    Endpoint  = "http://127.0.0.1:8877"
)

var awsS3Instance = NewAwsS3(SecretId, SecretKey, Region, Bucket, Endpoint, Token)

上傳檔案 Put http://0.0.0.0:8877/mybucket/test/1.txt

func TestPutObject(t *testing.T) {
    // 測試時修改本地路徑
    localFilePath := "./testdata/1.txt"
    t.Logf("local file path %s", localFilePath)
    fileContent, err := os.ReadFile(localFilePath)
    if err != nil {
        t.Fatalf("read file error: %s!", err.Error())
        return
    }
    // 測試時修改aws路徑
    awsPath := "/test/1.txt"
    _, err = awsS3Instance.PutObject(awsPath, fileContent)
    if err != nil {
        t.Fatalf("put object error: %s", err.Error())
    }
    t.Logf("put object success")
}

下載檔案 Get http://0.0.0.0:8877/mybucket/test/1.txt

func TestGetObject(t *testing.T) {
    // 測試時修改aws路徑
    awsPath := "/test/1.txt"
    contentBytes, _, err := awsS3Instance.GetObject(awsPath)
    if err != nil {
        t.Fatalf("get object error: %s", err.Error())
    }

    //獲取當前系統根目錄
    if err != nil {
        t.Fatalf("get home dir error: %s!", err.Error())
        return
    }
    // 測試時修改本地路徑
    localFilePath := "./testdata/1-get.txt"
    err = os.WriteFile(localFilePath, contentBytes, 0644)
    if err != nil {
        t.Fatal("write error")
        return
    }
    t.Logf("get object success")
}

Head檔案 Head http://0.0.0.0:8877/mybucket/test/1.txt

func TestHeadObject(t *testing.T) {
    // 測試時修改aws路徑
    awsPath := "/test/1.txt"
    eTag, lastModifyTime, err := awsS3Instance.HeadObject(awsPath)
    if err != nil {
        t.Fatalf("head object error: %s", err.Error())
    }
    t.Logf("head object success,eTag : %s, lastModifyTime : %v", eTag, lastModifyTime)
}

刪除檔案 Delete http://0.0.0.0:8877/mybucket/test/1.txt

func TestDeleteObject(t *testing.T) {
    // 測試時修改aws路徑
    awsPath := "/test/1.txt"
    err := awsS3Instance.DeleteObject(awsPath)
    if err != nil {
        t.Fatalf("delete object error: %s", err.Error())
    }
    t.Logf("delete object success")
}

mc 監聽桶 mybucket 的變化，可以看出

➜  minio mc watch myminio/mybucket/

[2023-10-19T07:27:35.675Z]   36 B s3:ObjectCreated:Put http://0.0.0.0:8877/mybucket/test/1.txt
[2023-10-19T07:28:46.813Z]        s3:ObjectAccessed:Get http://0.0.0.0:8877/mybucket/test/1.txt
[2023-10-19T07:28:58.157Z]        s3:ObjectAccessed:Head http://0.0.0.0:8877/mybucket/test/1.txt
[2023-10-19T08:42:23.065Z]        s3:ObjectRemoved:Delete http://0.0.0.0:8877/mybucket/test/1.txt

minio控制檯頁面統計資訊

詳細文件參考

https://min.io/docs/minio/kubernetes/upstream/index.html?ref=docs-redirect&ref=con

done.

祝玩的開心~

minio-物件儲存
2021-08-23
物件
客戶端儲存筆記
2017-04-26
客戶端筆記
客戶端儲存那些事
2017-01-18
客戶端
客戶端資料儲存概述
2019-02-16
客戶端
本地MinIO儲存服務Java遠端呼叫上傳檔案
2023-11-22
Java
《客戶端儲存技術》總結
2017-03-15
客戶端
使用MinIO搭建物件儲存服務
2022-05-20
物件
《客戶端儲存技術》讀後感
2017-08-12
客戶端
C#執行緒本地儲存:LocalDataStoreSlot和ThreadLocal
2016-04-13
C#執行緒LDAASTthread
客戶端本地儲存的比較及使用window.name資料傳輸
2015-01-10
客戶端
VNC客戶端是Windows，VNC客戶端是Windows如何進行操作
2020-06-02
VNC客戶端Windows
【儲存】AIX儲存管理基本概念和操作（LVM）
2017-07-25
AILVM
Redis客戶端基本操作以及檢視慢查詢
2020-04-06
Redis客戶端
在客戶端儲存對EJB的遠端呼叫是否可行？
2006-03-14
客戶端
跟蹤客戶端執行的SQL
2014-03-01
客戶端SQL
如何用 Minio SDK 訪問滴滴雲物件儲存
2018-12-19
物件
JavaScript客戶端document物件和window物件的區別
2018-06-19
JavaScript客戶端物件
使用Java客戶端對Redis進行操作
2017-07-09
Java客戶端Redis
Kubernetes官方java客戶端之五：proto基本操作
2021-09-09
Java客戶端
Kubernetes官方java客戶端之六：OpenAPI基本操作
2021-01-11
Java客戶端API
客戶端(瀏覽器端)資料儲存技術概覽
2017-03-09
客戶端瀏覽器
AIX儲存管理基本概念和操作（一)
2015-08-22
AI
HTML5離線應用與客戶端儲存
2018-05-01
HTML客戶端
Elasticsearch的PHP客戶端操作
2021-09-09
ElasticsearchPHP客戶端
golang實現tcp客戶端服務端程式
2020-12-27
GolangTCP客戶端服務端
git客戶端建立本地公鑰
2015-10-09
Git客戶端
Spring 客戶端 IP 地址獲取及儲存細節
2020-08-04
Spring客戶端
超越 Cookie：當今的客戶端資料儲存技術
2019-08-06
Cookie客戶端
四種有能力取代Cookies的客戶端Web儲存方案
2013-10-21
Cookie客戶端Web
Asp.net直接儲存（下載）檔案到客戶端
2007-06-25
ASP.NET客戶端
HTML5的五種客戶端離線儲存方案
2015-11-06
HTML客戶端
分散式檔案儲存庫MinIO可還行？
2020-05-19
分散式
Windows下svn客戶端TortoiseSVN的安裝和操作
2009-05-21
Windows客戶端
ElasticSearch-命令列客戶端操作
2021-01-25
Elasticsearch命令列客戶端
走近原始碼：Redis命令執行過程（客戶端）
2019-01-21
原始碼Redis客戶端
XML基本操作-建立(DOM和LOINQ)和LINQ查詢和儲存
2018-05-17
XML
客戶端序列化JSON物件
2013-12-16
客戶端JSON物件
本地儲存
2017-10-08