Sharding JDBC案例實戰

女友在高考發表於2022-05-15

基礎分庫

以下例項基於shardingsphere 4.1.0 + SpringBoot 2.2.5.RELEASE版本

依賴匯入：

<properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <project.compile.sourceEncoding>UTF-8</project.compile.sourceEncoding>
    <springboot.version>2.2.5.RELEASE</springboot.version>
    <shardingsphere.version>4.1.0</shardingsphere.version>
</properties>



    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-jdbc</artifactId>
            <version>${springboot.version}</version>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-jpa</artifactId>
            <version>${springboot.version}</version>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <version>${springboot.version}</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>8.0.13</version>
        </dependency>

        <dependency>
            <groupId>org.apache.shardingsphere</groupId>
            <artifactId>sharding-jdbc-spring-boot-starter</artifactId>
            <version>${shardingsphere.version}</version>
        </dependency>

    </dependencies>

場景：通過id欄位取餘分片到兩個資料庫

引入依賴

<dependency>
    <groupId>org.apache.shardingsphere</groupId>
    <artifactId>sharding-jdbc-spring-boot-starter</artifactId>
</dependency>

引數配置

spring.shardingsphere.datasource.names=ds0,ds1

spring.shardingsphere.datasource.ds0.type=com.zaxxer.hikari.HikariDataSource
spring.shardingsphere.datasource.ds0.driver-class-name=com.mysql.jdbc.Driver
spring.shardingsphere.datasource.ds0.jdbc-url=jdbc:mysql://localhost:3306/ds_0?useUnicode=true&characterEncoding=utf8&autoReconnect=true&allowMultiQueries=true&useSSL=false&serverTimezone=UTC
spring.shardingsphere.datasource.ds0.username=root
spring.shardingsphere.datasource.ds0.password=0490218292



spring.shardingsphere.datasource.ds1.type=com.zaxxer.hikari.HikariDataSource
spring.shardingsphere.datasource.ds1.driver-class-name=com.mysql.jdbc.Driver
spring.shardingsphere.datasource.ds1.jdbc-url=jdbc:mysql://localhost:3306/ds_1?useUnicode=true&characterEncoding=utf8&autoReconnect=true&allowMultiQueries=true&useSSL=false&serverTimezone=UTC
spring.shardingsphere.datasource.ds1.username=root
spring.shardingsphere.datasource.ds1.password=0490218292

spring.shardingsphere.sharding.tables.position.database-strategy.inline.sharding-column=id
spring.shardingsphere.sharding.tables.position.database-strategy.inline.algorithm-expression=ds$->{id % 2}

測試插入資料

@Test
    public void testAdd(){
        for (int i = 0; i <= 20; i++) {
            Position position=new Position();
            position.setId((long) i);
            position.setName("lagou"+i);
            position.setSalary("1000");
            position.setCity("beijing");
            positionRepository.save(position);
        }
    }

主鍵生成使用雪花演算法

id需要設定IDENTITY

@Id
@Column(name = "id")
@GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;

引數配置

增加id設定：

#id設定
spring.shardingsphere.sharding.tables.position.key-generator.column=id
spring.shardingsphere.sharding.tables.position.key-generator.type=SNOWFLAKE

測試

    @Test
    public void testAdd(){
        for (int i = 0; i <= 20; i++) {
            Position position=new Position();
            position.setName("lagou"+i);
            position.setSalary("1000");
            position.setCity("beijing");
            positionRepository.save(position);
        }
    }

自定義主鍵生成器

自定義主鍵生成器類

public class MyCustomId implements ShardingKeyGenerator {

    @Override
    public Comparable<?> generateKey() {
        return System.currentTimeMillis()+new Random().nextInt(100000);
    }

    @Override
    public String getType() {
        //自定義一個名稱
        return "MYID";
    }

    @Override
    public Properties getProperties() {
        return null;
    }

    @Override
    public void setProperties(Properties properties) {

    }

}

配置

在resources下建立META-INF/services目錄，並建立一個檔案，檔名為：org.apache.shardingsphere.spi.keygen.ShardingKeyGenerator

裡面寫自定義主鍵生成器的全類名

配置生成器型別的地方改為和我們自定義的生成器的型別一致

#id設定
spring.shardingsphere.sharding.tables.position.key-generator.column=id
spring.shardingsphere.sharding.tables.position.key-generator.type=MYID

兩表關聯的分庫

場景：職位表（position）和職位詳情表（position_detail）是關聯的兩個表，關聯關係是：position_detail.pid = position.id，那麼我們期望在插入資料後，根據職位Id進行查詢時能夠只查詢一個庫，而不是笛卡爾積的進行查詢。

完整的引數配置

spring.shardingsphere.datasource.names=ds0,ds1

spring.shardingsphere.datasource.ds0.type=com.zaxxer.hikari.HikariDataSource
spring.shardingsphere.datasource.ds0.driver-class-name=com.mysql.jdbc.Driver
spring.shardingsphere.datasource.ds0.jdbc-url=jdbc:mysql://localhost:3306/ds_0?useUnicode=true&characterEncoding=utf8&autoReconnect=true&allowMultiQueries=true&useSSL=false&serverTimezone=UTC
spring.shardingsphere.datasource.ds0.username=root
spring.shardingsphere.datasource.ds0.password=0490218292



spring.shardingsphere.datasource.ds1.type=com.zaxxer.hikari.HikariDataSource
spring.shardingsphere.datasource.ds1.driver-class-name=com.mysql.jdbc.Driver
spring.shardingsphere.datasource.ds1.jdbc-url=jdbc:mysql://localhost:3306/ds_1?useUnicode=true&characterEncoding=utf8&autoReconnect=true&allowMultiQueries=true&useSSL=false&serverTimezone=UTC
spring.shardingsphere.datasource.ds1.username=root
spring.shardingsphere.datasource.ds1.password=0490218292

#職位表設定
spring.shardingsphere.sharding.tables.position.database-strategy.inline.sharding-column=id
spring.shardingsphere.sharding.tables.position.database-strategy.inline.algorithm-expression=ds$->{id % 2}
#id設定
spring.shardingsphere.sharding.tables.position.key-generator.column=id
spring.shardingsphere.sharding.tables.position.key-generator.type=SNOWFLAKE
#職位表詳情設定
spring.shardingsphere.sharding.tables.position_detail.database-strategy.inline.sharding-column=pid
spring.shardingsphere.sharding.tables.position_detail.database-strategy.inline.algorithm-expression=ds$->{pid % 2}
#id設定
spring.shardingsphere.sharding.tables.position_detail.key-generator.column=id
spring.shardingsphere.sharding.tables.position_detail.key-generator.type=SNOWFLAKE

可以看出position的id的分片策略和position_detail的pid的分片策略一致。
2. 測試

@Test
public void testQueryPosition(){
    Object positionAndDetailById = positionRepository.findPositionAndDetailById(730545854473043968L);
    System.out.println(positionAndDetailById);
}

可以看出，只查詢了一個庫：

廣播表設定

場景：城市表屬於基礎表，資料量不大，每個庫都可以存一樣的資料。

廣播表配置

#廣播表設定
spring.shardingsphere.sharding.broadcast-tables=city
spring.shardingsphere.sharding.tables.city.key-generator.column=id
spring.shardingsphere.sharding.tables.city.key-generator.type=SNOWFLAKE

測試

@Test
public void testAddCity(){
    City city=new City();
    city.setName("成都");
    city.setProvince("四川");
    cityRepository.save(city);
}

和之前的不同，這一條資料的插入，兩個庫都有。且ID也是一致的。

分庫且分表

場景：我們有一個訂單表，可以根據公司id(companyId)進行分庫，然後在根據id進行分表。

引數配置

#訂單表分庫且分表
spring.shardingsphere.sharding.tables.b_order.database-strategy.inline.sharding-column=company_id
spring.shardingsphere.sharding.tables.b_order.database-strategy.inline.algorithm-expression=ds$->{company_id%2}
spring.shardingsphere.sharding.tables.b_order.table-strategy.inline.sharding-column=id
spring.shardingsphere.sharding.tables.b_order.table-strategy.inline.algorithm-expression=b_order${id % 2}
spring.shardingsphere.sharding.tables.b_order.actual-data-nodes=ds${0..1}.b_order${0..1}
#id設定
spring.shardingsphere.sharding.tables.b_order.key-generator.column=id
spring.shardingsphere.sharding.tables.b_order.key-generator.type=SNOWFLAKE

測試

    @Test
    @Repeat(100)
    public void testAddBOrder(){
        BOrder bOrder=new BOrder();
        bOrder.setDel(false);
        bOrder.setCompanyId(new Random().nextInt(10));
        bOrder.setPositionId(23);
        bOrder.setUserId(22);
        bOrder.setPublishUserId(11);
        bOrder.setResumeType(1);
        bOrder.setStatus("AUTO");
        bOrder.setCreateTime(new Date());
        bOrder.setOperateTime(new Date());
        bOrder.setWorkYear("2");
        bOrder.setName("lagou");
        bOrder.setPositionName("Java");
        bOrder.setResumeId(23443);
        bOrderRepository.save(bOrder);
    }

我們發現資料插入到了ds_0.b_order0、ds_0.b_order1、ds_1.b_order0、ds_1.b_order1四個node裡面。

讀寫分離

引數配置

spring.shardingsphere.datasource.names=master,slave0

spring.shardingsphere.datasource.master.type=com.zaxxer.hikari.HikariDataSource
spring.shardingsphere.datasource.master.driver-class-name=com.mysql.jdbc.Driver
spring.shardingsphere.datasource.master.jdbc-url=jdbc:mysql://localhost:3306/ds_0?useUnicode=true&characterEncoding=utf8&autoReconnect=true&allowMultiQueries=true&useSSL=false&serverTimezone=UTC
spring.shardingsphere.datasource.master.username=root
spring.shardingsphere.datasource.master.password=0490218292



spring.shardingsphere.datasource.slave0.type=com.zaxxer.hikari.HikariDataSource
spring.shardingsphere.datasource.slave0.driver-class-name=com.mysql.jdbc.Driver
spring.shardingsphere.datasource.slave0.jdbc-url=jdbc:mysql://localhost:3306/ds_0_slave?useUnicode=true&characterEncoding=utf8&autoReconnect=true&allowMultiQueries=true&useSSL=false&serverTimezone=UTC
spring.shardingsphere.datasource.slave0.username=root
spring.shardingsphere.datasource.slave0.password=0490218292

#讀寫分離
spring.shardingsphere.masterslave.name=datasource
spring.shardingsphere.masterslave.master-data-source-name=master
spring.shardingsphere.masterslave.slave-data-source-names=slave0
#多個讀庫時的負載均衡策略
spring.shardingsphere.masterslave.load-balance-algorithm-type=ROUND_ROBIN

查詢測試

    @Test
    public void test(){
        List<City> all = cityRepository.findAll();
        all.forEach(x->System.out.println(x));
    }

分庫分表+讀寫分離的引數配置

#資料來源
spring.shardingsphere.datasource.names=master0,slave0,slave1,master1,slave2,slave3

spring.shardingsphere.datasource.master0.type=com.zaxxer.hikari.HikariDataSource
spring.shardingsphere.datasource.master0.driver-class-name=com.mysql.jdbc.Driver
spring.shardingsphere.datasource.master0.jdbc-url=jdbc:mysql://localhost:3306/master0?useUnicode=true&characterEncoding=utf-8&useSSL=false
spring.shardingsphere.datasource.master0.username=root
spring.shardingsphere.datasource.master0.password=root

spring.shardingsphere.datasource.slave0.type=com.zaxxer.hikari.HikariDataSource
spring.shardingsphere.datasource.slave0.driver-class-name=com.mysql.jdbc.Driver
spring.shardingsphere.datasource.slave0.jdbc-url=jdbc:mysql://localhost:3306/slave0?useSSL=false
spring.shardingsphere.datasource.slave0.username=root
spring.shardingsphere.datasource.slave0.password=root

spring.shardingsphere.datasource.slave1.type=com.zaxxer.hikari.HikariDataSource
spring.shardingsphere.datasource.slave1.driver-class-name=com.mysql.jdbc.Driver
spring.shardingsphere.datasource.slave1.jdbc-url=jdbc:mysql://localhost:3306/slave1?useSSL=false
spring.shardingsphere.datasource.slave1.username=root
spring.shardingsphere.datasource.slave1.password=root

spring.shardingsphere.datasource.master1.type=com.zaxxer.hikari.HikariDataSource
spring.shardingsphere.datasource.master1.driver-class-name=com.mysql.jdbc.Driver
spring.shardingsphere.datasource.master1.jdbc-url=jdbc:mysql://localhost:3306/master1?useUnicode=true&characterEncoding=utf-8&useSSL=false
spring.shardingsphere.datasource.master1.username=root
spring.shardingsphere.datasource.master1.password=root

spring.shardingsphere.datasource.slave2.type=com.zaxxer.hikari.HikariDataSource
spring.shardingsphere.datasource.slave2.driver-class-name=com.mysql.jdbc.Driver
spring.shardingsphere.datasource.slave2.jdbc-url=jdbc:mysql://localhost:3306/slave2?useSSL=false
spring.shardingsphere.datasource.slave2.username=root
spring.shardingsphere.datasource.slave2.password=root

spring.shardingsphere.datasource.slave3.type=com.zaxxer.hikari.HikariDataSource
spring.shardingsphere.datasource.slave3.driver-class-name=com.mysql.jdbc.Driver
spring.shardingsphere.datasource.slave3.jdbc-url=jdbc:mysql://localhost:3306/slave3?useSSL=false
spring.shardingsphere.datasource.slave3.username=root
spring.shardingsphere.datasource.slave3.password=root

#分庫分表
spring.shardingsphere.sharding.tables.b_order.database-strategy.inline.sharding-column=company_id
spring.shardingsphere.sharding.tables.b_order.database-strategy.inline.algorithm-expression=master$->{company_id % 2}
spring.shardingsphere.sharding.tables.b_order.actual-data-nodes=master$->{0..1}.b_order$->{0..1}
spring.shardingsphere.sharding.tables.b_order.table-strategy.inline.sharding-column=id
spring.shardingsphere.sharding.tables.b_order.table-strategy.inline.algorithm-expression=b_order$->{id % 2}

#讀寫分離
spring.shardingsphere.sharding.master-slave-rules.master0.master-data-source-name=master0
spring.shardingsphere.sharding.master-slave-rules.master0.slave-data-source-names=slave0, slave1
spring.shardingsphere.sharding.master-slave-rules.master1.master-data-source-name=master1
spring.shardingsphere.sharding.master-slave-rules.master1.slave-data-source-names=slave2, slave3

強制路由

在一些應用場景中，分片條件並不存在於SQL，而存在於外部業務邏輯。因此需要提供一種通過在外部業務程式碼中指定路由配置的一種方式，在ShardingSphere中叫做Hint。如果使用Hint指定了強制分片路由，那麼SQL將會無視原有的分片邏輯，直接路由至指定的資料節點操作。

使用場景：

資料分片操作，如果分片鍵沒有在SQL或者資料表中，而是在業務邏輯程式碼中
讀寫分離操作，如果需要強制在主庫進行某些操作

自定義Hint實現類

public class MyHintShardingAlgorithm implements HintShardingAlgorithm<String> {


    @Override
    public Collection<String> doSharding(Collection<String> collection, HintShardingValue<String> hintShardingValue) {
        Collection<String> result=new ArrayList<>();
       if(hintShardingValue.getValues().contains("master")){
           ((ArrayList<String>) result).add("master");
       }else {
           ((ArrayList<String>) result).add("slave0");
       }
       return result;
    }
}

配置自定義的Hint類

spring.shardingsphere.sharding.tables.city.database-strategy.hint.algorithm-class-name=com.mmc.sharding.hint.MyHintShardingAlgorithm

測試

    @Test
    public void testHint(){
        HintManager hintManager = HintManager.getInstance();
        hintManager.addDatabaseShardingValue("city","master");
//        hintManager.setMasterRouteOnly();
        List<City> all = cityRepository.findAll();
        all.forEach(x->System.out.println(x));
    }

還可以使用hintManager.setMasterRouteOnly()指定僅路由到主庫。

測試過程中發現Hint的自定義策略和讀寫分離配置有衝突。配置了讀寫分離後自定義Hint類不生效了，僅hintManager.setMasterRouteOnly()還可以用。

資料加密

脫敏配置分為如下幾個：資料來源配置，加密器配置，脫敏表配置以及查詢屬性配置，其詳情如下圖所示：

資料來源配置：指Datasource的配置資訊
加密器配置：指使用什麼加密策略進行加解密。目前ShardingSphere內建了兩種加解密策略AES、MD5
脫敏表配置：指定哪個列用於儲存密文資料，哪個列存明文資料，以及在應用裡用哪個列（應用層sql裡使用的列名）
查詢屬性配置：當資料庫同時存了明文和密文的時候，該屬性開關用於決定是直接查詢資料庫表裡的明文，還是查密文然後通過解密後返回。

先建立個表

CREATE TABLE `c_user` (
  `Id` bigint(11) NOT NULL AUTO_INCREMENT,
  `name` varchar(256) DEFAULT NULL,
  `pwd_plain` varchar(256) DEFAULT NULL,
  `pwd_cipher` varchar(256) DEFAULT NULL,
  PRIMARY KEY (`Id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

建立實體類

@Entity
@Table(name = "c_user")
public class CUser implements Serializable {
    @Id
    @Column(name = "id")
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;

    @Column(name = "name")
    private String name;

    /**
     * 邏輯列名
     */
    @Column(name = "password")
    private String password;
}

引數配置

#定義資料表真實明文列
#spring.shardingsphere.encrypt.tables.c_user.columns.password.plain-column=pwd_plain
#定義資料表真實密文列
spring.shardingsphere.encrypt.tables.c_user.columns.password.cipher-column=pwd_cipher
#定義加密器，名稱為lagou_pwd
spring.shardingsphere.encrypt.encryptors.lagou_pwd.type=aes
spring.shardingsphere.encrypt.encryptors.lagou_pwd.props.aes.key.value=1234
#指定加密器，password是邏輯列名，與實體類中的欄位對應
spring.shardingsphere.encrypt.tables.c_user.columns.password.encryptor=lagou_pwd

測試

    @Test
    public void testEncrypt(){
        CUser cUser=new CUser();
        cUser.setName("阿百川");
        cUser.setPassword("123456");
        cUserRepository.save(cUser);
    }

    @Test
    public void testQueryByPassword(){
        List<CUser> byPassword = cUserRepository.findByPassword("123456");
        System.out.println(byPassword);
    }

資料庫存放的已經是密文了，通過明文密碼也可以查詢到資料了。

分散式事務

僅僅需要在測試方法上加上兩個註解：

    @Transactional(rollbackFor = Exception.class)
    @ShardingTransactionType(TransactionType.XA)

TransactionType有XA、BASE、LOCAL三種

@Test
    @Transactional(rollbackFor = Exception.class)
    @ShardingTransactionType(TransactionType.XA)
    public void testAddDetail(){
        for (int i = 0; i <= 3; i++) {
            Position position=new Position();
            position.setName("lagou"+i);
            position.setSalary("1000");
            position.setCity("beijing");
            positionRepository.save(position);


            if(i==3){
                throw new RuntimeException();
            }
            PositionDetail positionDetail=new PositionDetail();
            positionDetail.setPid(position.getId());
            positionDetail.setDescription("詳情");
            positionDetailRepository.save(positionDetail);
        }
    }

Sharding-JDBC
2017-03-07
JDBC
sharding-jdbc5.0.0分表實踐
2022-02-10
JDBC
非規範SQL的sharding-jdbc實踐
2018-12-19
SQLJDBC
【死磕Sharding-jdbc】—基於 SSM 整合sharding
2021-09-09
JDBCSSM
Spark SQL：JDBC資料來源複雜綜合案例實戰
2018-09-28
SparkSQLJDBC
Sharding-JDBC 核心之路由引擎
2020-10-15
JDBC路由
基於Sharding-Jdbc 實現的讀寫分離實現
2020-10-04
JDBC
Sharding-JDBC 原始碼之 SQL 解析
2020-12-24
JDBC原始碼SQL
Sharding-JDBC分片策略詳解（二）
2020-12-04
JDBC
Sharding-JDBC 快速入門第一課
2020-06-19
JDBC
Sharding-JDBC 使用入門和基本配置
2019-05-16
JDBC
資料庫中介軟體 Sharding-JDBC 原始碼分析 —— JDBC實現與讀寫分離
2017-10-22
資料庫JDBC原始碼
SpringBoot+MybatisPlus+Mysql+Sharding-JDBC分庫分表實踐
2022-03-01
Spring BootMyBatisMySqlJDBC
資料庫中介軟體sharding-jdbc實現讀寫分離
2020-10-06
資料庫JDBC
工作深度總結——分庫分表sharding-jdbc實踐路線
2018-03-01
JDBC
Sharding-JDBC 原始碼之啟動流程分析
2020-12-20
JDBC原始碼
資料量大了一定要分表，分庫分表元件Sharding-JDBC入門與專案實戰
2020-08-01
元件JDBC
採用Sharding-JDBC解決分庫分表
2021-01-26
JDBC
SpringBoot使用Sharding-JDBC讀寫分離
2019-02-12
Spring BootJDBC
SpringBoot使用Sharding-JDBC分庫分表
2019-02-01
Spring BootJDBC
Sharding-JDBC foreach批量更新事務異常
2020-11-15
JDBC
sharding-jdbc水平垂直分庫分表環境搭建
2020-10-17
JDBC
深度剖析Reflect + 實戰案例
2021-03-07
JQuery8：實戰案例
2018-11-14
jQuery
JavaScript8：實戰案例
2018-11-14
JavaScript
Sharding-JDBC原始碼解析與vivo的定製開發
2024-03-08
JDBC原始碼
Sharding-JDBC基本使用，整合Springboot實現分庫分表，讀寫分離
2021-10-29
JDBCSpring Boot
JDBC+MySQL入門實戰(實現CURD的例子)
2020-08-14
JDBCMySql
1.17 JavaScript8：實戰案例
2018-11-09
JavaScript
Laravel —— 服務注入實戰案例
2018-09-25
Laravel
4-陣列-案例實戰
2020-10-09
陣列
Hadoop2.7實戰v1.0之Eclipse+Hive2.0.0的JDBC案例(最詳細搭建)
2016-06-09
HadoopEclipseHiveJDBC
SpringBoot+Mybatis-Plus整合Sharding-JDBC5.1.1實現單庫分表【全網最新】
2022-05-30
Spring BootMyBatisJDBC
實戰SEO——實用技法與案例剖析
2011-09-19
ffmpeg實戰-音視訊合成案例
2021-06-17
vue2.0+Element-ui實戰案例
2018-11-03
VueUI
騰訊雲大資料實戰案例
2020-10-25
大資料
Python實戰小案例，值得收藏!
2023-05-15
Python