Java GenericObjectPool 物件池化技術--SpringBoot sftp 連線池工具類

VipSoft發表於2023-04-07

通常一個物件建立、銷燬非常耗時的時候，我們不會頻繁的建立和銷燬它，而是考慮複用。複用物件的一種做法就是物件池，將建立好的物件放入池中維護起來，下次再用的時候直接拿池中已經建立好的物件繼續用，這就是池化的思想。

Apache Commons Pool是一個物件池的框架，他提供了一整套用於實現物件池化的API。它提供了三種物件池：GenericKeyedObjectPool，SoftReferenceObjectPool和GenericObjectPool，其中GenericObjectPool是我們最常用的物件池，內部實現也最複雜。

GenericObjectPool

GenericObjectPool 是一個通用物件池框架，我們可以藉助它實現一個健壯的物件池，UML圖如下所示：

GenericObjectPool 實現了ObjectPool介面，而ObjectPool就是物件池的核心介面，它定義了一個物件池應該實現的行為。

public interface ObjectPool<T> extends Closeable {
    /**
     * 從池中借走到一個物件
     */
    T borrowObject() throws Exception, NoSuchElementException, IllegalStateException;
    /**
     * 把物件歸還給物件池
     */
    void returnObject(T var1) throws Exception;
    /**
     * 驗證物件的有效性
     */
    void invalidateObject(T var1) throws Exception;

    /**
     * 往池中新增一個物件
     */
    void addObject() throws Exception, IllegalStateException, UnsupportedOperationException;
    /**
     * 返回物件池中有多少物件是空閒的，也就是能夠被借走的物件的數量。
     */
    int getNumIdle();
    /**
     * 返回物件池中有物件物件是活躍的，也就是已經被借走的，在使用中的物件的數量。
     */
    int getNumActive();
    /**
     * 清理物件池。注意是清理不是清空，該方法要求的是，清理所有空閒物件，釋放相關資源。
     */
    void clear() throws Exception, UnsupportedOperationException;
    /**
     * 關閉物件池。這個方法可以達到清空的效果，清理所有物件以及相關資源。
     */
    void close();
}

BasePooledObjectFactory

Java BasePooledObjectFactory 物件池化技術

使用GenericObjectPool只需要建立一個物件工廠類，繼承BasePooledObjectFactory並重寫它的create()和destroyObject()。
如下文中的：SftpPool.java

public interface PooledObjectFactory<T> {
    /**
     * 建立一個可由池提供服務的例項，並將其封裝在由池管理的PooledObject中。
     */
    PooledObject<T> makeObject() throws Exception;

    /**
     *  銷燬池不再需要的例項
     */
    void destroyObject(PooledObject<T> var1) throws Exception;

    /**
     * 確保例項可以安全地由池返回
     */
    boolean validateObject(PooledObject<T> var1);

    /**
     * 重新初始化池返回的例項
     */
    void activateObject(PooledObject<T> var1) throws Exception;

    /**
     * 取消初始化要返回到空閒物件池的例項
     */
    void passivateObject(PooledObject<T> var1) throws Exception;
}

配置類GenericObjectPoolConfig

GenericObjectPoolConfig是封裝GenericObject池配置的簡單“結構”，此類不是執行緒安全的；它僅用於提供建立池時使用的屬性。大多數情況，可以使用GenericObjectPoolConfig提供的預設引數就可以滿足日常的需求。

工作原理流程

構造方法
當我們執行構造方法時，主要工作就是建立了一個儲存物件的LinkedList型別容器，也就是概念意義上的“池”
從物件池中獲取物件
獲取池中的物件是透過borrowObject()命令，原始碼比較複雜，簡單而言就是去LinkedList中獲取一個物件，如果不存在的話，要呼叫構造方法中第一個引數Factory工廠類的makeObject()方法去建立一個物件再獲取，獲取到物件後要呼叫validateObject方法判斷該物件是否是可用的，如果是可用的才拿去使用。LinkedList容器減一
歸還物件到執行緒池
簡單而言就是先呼叫validateObject方法判斷該物件是否是可用的，如果可用則歸還到池中，LinkedList容器加一，如果是不可以的則呼叫destroyObject方法進行銷燬

上面三步就是最簡單的流程，由於取和還的流程步驟都在borrowObject和returnObject方法中固定的，所以我們只要重寫Factory工廠類的makeObject()和validateObject以及destroyObject方法即可實現最簡單的池的管理控制，透過構造方法傳入該Factory工廠類物件則可以建立最簡單的物件池管理類。這算是比較好的解耦設計模式，借和還的流程如下圖所示：

使用Demo

<dependency>
    <groupId>org.apache.commons</groupId>
    <artifactId>commons-pool2</artifactId>
    <version>2.7.0</version>
</dependency>

<!-- https://mvnrepository.com/artifact/com.jcraft/jsch -->
<dependency>
    <groupId>com.jcraft</groupId>
    <artifactId>jsch</artifactId>
    <version>0.1.55</version>
</dependency>

點選檢視程式碼

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <parent>
        <artifactId>vipsoft-parent</artifactId>
        <groupId>com.vipsoft.boot</groupId>
        <version>1.0-SNAPSHOT</version>
    </parent>
    <modelVersion>4.0.0</modelVersion>

    <artifactId>vipsoft-sftp</artifactId>
    <version>1.0-SNAPSHOT</version>

    <dependencies>

        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-pool2</artifactId>
            <version>2.7.0</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/com.jcraft/jsch -->
        <dependency>
            <groupId>com.jcraft</groupId>
            <artifactId>jsch</artifactId>
            <version>0.1.55</version>
        </dependency>

        <dependency>
            <groupId>org.eclipse.paho</groupId>
            <artifactId>org.eclipse.paho.client.mqttv3</artifactId>
            <version>1.2.5</version>
        </dependency>


        <dependency>
            <groupId>cn.hutool</groupId>
            <artifactId>hutool-all</artifactId>
            <version>5.3.6</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
            <exclusions>
                <exclusion>
                    <groupId>org.junit.vintage</groupId>
                    <artifactId>junit-vintage-engine</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>
</project>

application.yaml

server:
  port: 8088
  application:
    name: sftp Demo


sftp:
  host: 172.16.3.88 # 伺服器ip
  port: 22 # ssh埠
  username: root # 使用者名稱
  password: root # 密碼
  # 連線池引數
  pool:
    max-total: 10
    max-idle: 10
    min-idle: 5

SftpPoolException.java

package com.vipsoft.sftp.exception;


/**
 * sftp連線池異常
 */
public class SftpPoolException extends RuntimeException {

    private static final long serialVersionUID = 1L;

    /**
     * Constructs a new runtime exception with {@code null} as its
     * detail message.  The cause is not initialized, and may subsequently be
     * initialized by a call to {@link #initCause}.
     */
    public SftpPoolException() {
    }

    /**
     * Constructs a new runtime exception with the specified detail message.
     * The cause is not initialized, and may subsequently be initialized by a
     * call to {@link #initCause}.
     *
     * @param message the detail message. The detail message is saved for
     *                later retrieval by the {@link #getMessage()} method.
     */
    public SftpPoolException(String message) {
        super(message);
    }

    /**
     * Constructs a new runtime exception with the specified detail message and
     * cause.  <p>Note that the detail message associated with
     * {@code cause} is <i>not</i> automatically incorporated in
     * this runtime exception's detail message.
     *
     * @param message the detail message (which is saved for later retrieval
     *                by the {@link #getMessage()} method).
     * @param cause   the cause (which is saved for later retrieval by the
     *                {@link #getCause()} method).  (A <tt>null</tt> value is
     *                permitted, and indicates that the cause is nonexistent or
     *                unknown.)
     * @since 1.4
     */
    public SftpPoolException(String message, Throwable cause) {
        super(message, cause);
    }

    /**
     * Constructs a new runtime exception with the specified cause and a
     * detail message of <tt>(cause==null ? null : cause.toString())</tt>
     * (which typically contains the class and detail message of
     * <tt>cause</tt>).  This constructor is useful for runtime exceptions
     * that are little more than wrappers for other throwables.
     *
     * @param cause the cause (which is saved for later retrieval by the
     *              {@link #getCause()} method).  (A <tt>null</tt> value is
     *              permitted, and indicates that the cause is nonexistent or
     *              unknown.)
     * @since 1.4
     */
    public SftpPoolException(Throwable cause) {
        super(cause);
    }

    /**
     * Constructs a new runtime exception with the specified detail
     * message, cause, suppression enabled or disabled, and writable
     * stack trace enabled or disabled.
     *
     * @param message            the detail message.
     * @param cause              the cause.  (A {@code null} value is permitted,
     *                           and indicates that the cause is nonexistent or unknown.)
     * @param enableSuppression  whether or not suppression is enabled
     *                           or disabled
     * @param writableStackTrace whether or not the stack trace should
     *                           be writable
     * @since 1.7
     */
    public SftpPoolException(String message, Throwable cause, boolean enableSuppression, boolean writableStackTrace) {
        super(message, cause, enableSuppression, writableStackTrace);
    }

}

config

SftpConfig.java

package com.vipsoft.sftp.config;

import com.vipsoft.sftp.pool.SftpFactory;
import com.vipsoft.sftp.pool.SftpPool;
import com.vipsoft.sftp.utils.SftpUtil;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
@EnableConfigurationProperties(SftpProperties.class)
public class SftpConfig {
    // 工廠
    @Bean
    public SftpFactory sftpFactory(SftpProperties properties) {
        return new SftpFactory(properties);
    }

    // 連線池
    @Bean
    public SftpPool sftpPool(SftpFactory sftpFactory) {
        return new SftpPool(sftpFactory);
    }

    // 輔助類
    @Bean
    public SftpUtil sftpUtil(SftpPool sftpPool) {
        return new SftpUtil(sftpPool);
    }
}

SftpProperties.java

package com.vipsoft.sftp.config;

import com.jcraft.jsch.ChannelSftp;
import org.apache.commons.pool2.impl.GenericObjectPoolConfig;
import org.springframework.boot.context.properties.ConfigurationProperties;

@ConfigurationProperties(prefix = "sftp")
public class SftpProperties {

    private String host;
    private int port = 22;
    private String username = "root";
    private String password = "root";
    private Pool pool = new Pool();

    public String getHost() {
        return host;
    }

    public void setHost(String host) {
        this.host = host;
    }

    public int getPort() {
        return port;
    }

    public void setPort(int port) {
        this.port = port;
    }

    public String getUsername() {
        return username;
    }

    public void setUsername(String username) {
        this.username = username;
    }

    public String getPassword() {
        return password;
    }

    public void setPassword(String password) {
        this.password = password;
    }

    public Pool getPool() {
        return pool;
    }

    public void setPool(Pool pool) {
        this.pool = pool;
    }

    public static class Pool extends GenericObjectPoolConfig<ChannelSftp> {

        private int maxTotal = DEFAULT_MAX_TOTAL;
        private int maxIdle = DEFAULT_MAX_IDLE;
        private int minIdle = DEFAULT_MIN_IDLE;

        public Pool() {
            super();
        }
        @Override
        public int getMaxTotal() {
            return maxTotal;
        }
        @Override
        public void setMaxTotal(int maxTotal) {
            this.maxTotal = maxTotal;
        }
        @Override
        public int getMaxIdle() {
            return maxIdle;
        }
        @Override
        public void setMaxIdle(int maxIdle) {
            this.maxIdle = maxIdle;
        }
        @Override
        public int getMinIdle() {
            return minIdle;
        }
        @Override
        public void setMinIdle(int minIdle) {
            this.minIdle = minIdle;
        }
    }
}

Pool

SftpFactory.java

package com.vipsoft.sftp.pool;

import com.jcraft.jsch.ChannelSftp;
import com.jcraft.jsch.JSch;
import com.jcraft.jsch.JSchException;
import com.jcraft.jsch.Session;
import com.vipsoft.sftp.config.SftpProperties;
import com.vipsoft.sftp.exception.SftpPoolException;
import org.apache.commons.pool2.BasePooledObjectFactory;
import org.apache.commons.pool2.PooledObject;
import org.apache.commons.pool2.impl.DefaultPooledObject;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.Properties;

public class SftpFactory extends BasePooledObjectFactory<ChannelSftp> {

    private  final Logger logger = LoggerFactory.getLogger(this.getClass());

    private SftpProperties properties;

    public SftpProperties getProperties() {
        return properties;
    }

    public void setProperties(SftpProperties properties) {
        this.properties = properties;
    }

    public SftpFactory(SftpProperties properties) {
        this.properties = properties;
    }

    @Override
    public ChannelSftp create() {
        try {
            JSch jsch = new JSch();
            Session sshSession = jsch.getSession(properties.getUsername(), properties.getHost(), properties.getPort());
            sshSession.setPassword(properties.getPassword());
            Properties sshConfig = new Properties();
            sshConfig.put("StrictHostKeyChecking", "no");
            sshSession.setConfig(sshConfig);
            sshSession.connect();
            ChannelSftp channel = (ChannelSftp) sshSession.openChannel("sftp");
            channel.connect();
            return channel;
        } catch (JSchException e) {
            throw new SftpPoolException("連線sfpt失敗", e);
        }
    }

    @Override
    public PooledObject<ChannelSftp> wrap(ChannelSftp channelSftp) {
        return new DefaultPooledObject<>(channelSftp);
    }

    // 銷燬物件
    @Override
    public void destroyObject(PooledObject<ChannelSftp> p) {
        ChannelSftp channelSftp = p.getObject();
        channelSftp.disconnect();
    }

}

SftpPool.java

package com.vipsoft.sftp.pool;

import com.jcraft.jsch.ChannelSftp;
import org.apache.commons.pool2.impl.GenericObjectPool;

public class SftpPool<T> extends GenericObjectPool<ChannelSftp> {

    public SftpPool(SftpFactory factory) {
        super(factory,factory.getProperties().getPool());
    }

    /**
     * 獲取一個sftp連線物件
     * @return sftp連線物件
     */
    @Override
    public ChannelSftp borrowObject() throws Exception {
        return super.borrowObject();
    }

    /**
     * 歸還一個sftp連線物件
     * @param channelSftp sftp連線物件
     */
    @Override
    public void returnObject(ChannelSftp channelSftp) {
        if (channelSftp!=null) {
            super.returnObject(channelSftp);
        }
    }

}

Utils

ByteUtil.java

package com.vipsoft.sftp.utils;

import com.jcraft.jsch.ChannelSftp;
import com.jcraft.jsch.SftpException;
import com.vipsoft.sftp.exception.SftpPoolException;
import com.vipsoft.sftp.pool.SftpPool;

import java.io.InputStream;

public class SftpUtil {

    private SftpPool pool;

    public SftpUtil(SftpPool pool) {
        this.pool = pool;
    }

    /**
     * 下載檔案
     *
     * @param dir  遠端目錄
     * @param name 遠端檔名
     * @return 檔案位元組陣列
     */
    public byte[] download(String dir, String name) {
        ChannelSftp sftp = null;
        try {
            sftp = pool.borrowObject();
            sftp.cd(dir);
            InputStream in = sftp.get(name);
            return ByteUtil.inputStreamToByteArray(in);
        } catch (Exception e) {
            throw new SftpPoolException("sftp下載檔案出錯", e);
        } finally {
            pool.returnObject(sftp);
        }
    }

    /**
     * 上傳檔案
     *
     * @param dir  遠端目錄
     * @param name 遠端檔名
     * @param in   輸入流
     */
    public void upload(String dir, String name, InputStream in) {
        ChannelSftp sftp = null;
        try {
            sftp = pool.borrowObject();
            mkdirs(sftp, dir);
            sftp.cd(dir);
            sftp.put(in, name);
        } catch (Exception e) {
            throw new SftpPoolException("sftp上傳檔案出錯", e);
        } finally {
            pool.returnObject(sftp);
        }
    }

    /**
     * 刪除檔案
     *
     * @param dir  遠端目錄
     * @param name 遠端檔名
     */
    public void delete(String dir, String name) {
        ChannelSftp sftp = null;
        try {
            sftp = pool.borrowObject();
            sftp.cd(dir);
            sftp.rm(name);
        } catch (Exception e) {
            throw new SftpPoolException("sftp刪除檔案出錯", e);
        } finally {
            pool.returnObject(sftp);
        }
    }

    /**
     * 遞迴建立多級目錄
     *
     * @param dir 多級目錄
     */
    private void mkdirs(ChannelSftp sftp, String dir) {
        String[] folders = dir.split("/");
        try {
            sftp.cd("/");
            for (String folder : folders) {
                if (folder.length() > 0) {
                    try {
                        sftp.cd(folder);
                    } catch (Exception e) {
                        sftp.mkdir(folder);
                        sftp.cd(folder);
                    }
                }
            }
        } catch (SftpException e) {
            throw new SftpPoolException("sftp建立目錄出錯", e);
        }
    }

}

Test

SftpTest.java

package com.vipsoft.sftp;

import com.vipsoft.sftp.utils.SftpUtil;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;

@SpringBootTest
public class SftpTest {

    @Autowired
    private SftpUtil sftpUtil;

    @Test
    void downloadTest() {
        byte[] dockerfiles = sftpUtil.download("/opt/demo/", "Dockerfile");
        System.out.println("FileSize =>" + dockerfiles.length);
    }

}

物件池技術和通用實現GenericObjectPool
2021-07-11
物件Object
Java BasePooledObjectFactory 物件池化技術
2023-03-31
JavaObject物件
巧用GenericObjectPool建立自定義物件池
2023-03-17
Object物件
jsoup爬蟲技術+druid連線池
2018-03-21
JS爬蟲UI
Java技術分享：什麼是資料庫連線池？
2022-03-09
Java資料庫
資料庫連線池技術詳解
2018-08-19
資料庫
資料庫連線池_druid基本使用&工具類
2020-11-13
資料庫UI
走進JavaWeb技術世界3：JDBC的進化與連線池技術
2019-11-02
JavaWebJDBC
2-2. 線性池技術優化
2019-07-23
優化
Java篇-DBUtils與連線池
2021-09-09
Java
連線池
2024-05-26
SpringBoot專案整合阿里Druid連線池
2019-02-13
Spring Boot阿里UI
池化技術總結
2021-01-13
Jedis連線池究竟是何物｜得物技術
2023-12-26
Springboot 整合阿里資料庫連線池 druid
2021-09-03
Spring Boot阿里資料庫UI
java操作redis叢集連線池
2019-03-12
JavaRedis
django連線池
2024-08-04
Django
HTTP連線池
2018-05-07
HTTP
【JDBC】java連線池模擬測試連線oracle
2021-04-08
JDBCJavaOracle
springboot專案整合druid資料庫連線池
2021-04-18
Spring BootUI資料庫
Springboot 連線池wait_timeout超時設定
2018-05-27
Spring BootAI
Java Druid資料庫連線池+SpringJDBC
2020-10-29
JavaUI資料庫SpringJDBC
druid資料庫連線池的配置類
2022-01-18
UI資料庫
物件池Pools優化
2019-04-10
物件優化
ElasticSearch連線池建立
2020-12-18
Elasticsearch
自定義連線池
2020-12-31
從入門到掉坑：Go 記憶體池/物件池技術介紹
2020-06-04
Go記憶體物件
springboot之Druid連線池講解+mybatis整合+PageHelper整合
2020-11-30
Spring BootUIMyBatis
Java 客戶端 Jedis和JedisPool 連線池
2018-11-09
Java客戶端
commons-pool2 池化技術探究
2021-04-27
資料庫連線池-Druid資料庫連線池原始碼解析
2023-01-02
資料庫UI原始碼
Http持久連線與HttpClient連線池
2019-11-04
HTTPclient
連線池和連線數詳解
2018-04-04
go 語言連線池
2022-08-11
Go
Golang SQL連線池梳理
2020-07-31
GolangSQL
Tomcat 的 JDBC 連線池
2018-03-16
TomcatJDBC
springboot新增多資料來源連線池並配置Mybatis
2018-06-16
Spring BootMyBatis
《四資料庫連線池原始碼》手寫資料庫連線池
2019-04-01
資料庫原始碼