redis 原始碼分析:Jedis 哨兵模式連線原理

明月照江江發表於2023-10-02

1. 可以從單元測試開始入手

檢視類JedisSentinelPool
  private static final String MASTER_NAME = "mymaster";

  protected static final HostAndPort sentinel1 = HostAndPorts.getSentinelServers().get(1);
  protected static final HostAndPort sentinel2 = HostAndPorts.getSentinelServers().get(3);
  
  @Before
  public void setUp() throws Exception {
    sentinels.clear();

    sentinels.add(sentinel1.toString());
    sentinels.add(sentinel2.toString());
  }

  @Test
  public void repeatedSentinelPoolInitialization() {

    for (int i = 0; i < 20; ++i) {
      GenericObjectPoolConfig<Jedis> config = new GenericObjectPoolConfig<>();

      JedisSentinelPool pool = new JedisSentinelPool(MASTER_NAME, sentinels, config, 1000,
          "foobared", 2);
      pool.getResource().close();
      pool.destroy();
    }
  }

可以看到首先是建立了sentinel 的HostAndPort 物件,然後建立了連線池

2. 檢視 JedisSentinelPool 構造器,正式進入原始碼

  public JedisSentinelPool(String masterName, Set<String> sentinels,
      final GenericObjectPoolConfig<Jedis> poolConfig, int timeout, final String password,
      final int database) {
    this(masterName, sentinels, poolConfig, timeout, timeout, null, password, database);
  }
  
  ...
    public JedisSentinelPool(String masterName, Set<String> sentinels,
      final GenericObjectPoolConfig<Jedis> poolConfig,
      final int connectionTimeout, final int soTimeout, final int infiniteSoTimeout,
      final String user, final String password, final int database, final String clientName,
      final int sentinelConnectionTimeout, final int sentinelSoTimeout, final String sentinelUser,
      final String sentinelPassword, final String sentinelClientName) {
    this(masterName, parseHostAndPorts(sentinels), poolConfig,
        DefaultJedisClientConfig.builder().connectionTimeoutMillis(connectionTimeout)
            .socketTimeoutMillis(soTimeout).blockingSocketTimeoutMillis(infiniteSoTimeout)
            .user(user).password(password).database(database).clientName(clientName).build(),
        DefaultJedisClientConfig.builder().connectionTimeoutMillis(sentinelConnectionTimeout)
            .socketTimeoutMillis(sentinelSoTimeout).user(sentinelUser).password(sentinelPassword)
            .clientName(sentinelClientName).build()
    );
  }
  
  ...
    public JedisSentinelPool(String masterName, Set<HostAndPort> sentinels,
      final GenericObjectPoolConfig<Jedis> poolConfig, final JedisClientConfig masterClientConfig,
      final JedisClientConfig sentinelClientConfig) {
    this(masterName, sentinels, poolConfig, new JedisFactory(masterClientConfig), sentinelClientConfig);
  }
  
    public JedisSentinelPool(String masterName, Set<HostAndPort> sentinels,
      final GenericObjectPoolConfig<Jedis> poolConfig, final JedisFactory factory,
      final JedisClientConfig sentinelClientConfig) {
    super(poolConfig, factory);

    this.factory = factory;
    this.sentinelClientConfig = sentinelClientConfig;

    HostAndPort master = initSentinels(sentinels, masterName);
    initMaster(master);
  }

這裡執行了兩個重要方法
initSentinelsinitMaster

1. initSentinels 負責初始化sentinel ,並獲得master的地址
2. 有了master地址,就可以 initMaster 了
  private HostAndPort initSentinels(Set<HostAndPort> sentinels, final String masterName) {

    HostAndPort master = null;
    boolean sentinelAvailable = false;

    LOG.info("Trying to find master from available Sentinels...");

    for (HostAndPort sentinel : sentinels) {

      LOG.debug("Connecting to Sentinel {}", sentinel);
      //連線sentinel 節點
      try (Jedis jedis = new Jedis(sentinel, sentinelClientConfig)) {

        // 向sentinel傳送命令  sentinel get-master-addr-by-name mymaster
        List<String> masterAddr = jedis.sentinelGetMasterAddrByName(masterName);

        // connected to sentinel...
        sentinelAvailable = true;

        if (masterAddr == null || masterAddr.size() != 2) {
          LOG.warn("Can not get master addr, master name: {}. Sentinel: {}", masterName, sentinel);
          continue;
        }

        master = toHostAndPort(masterAddr);
        LOG.debug("Found Redis master at {}", master);
        break;
      } catch (JedisException e) {
        // resolves #1036, it should handle JedisException there's another chance
        // of raising JedisDataException
        LOG.warn(
          "Cannot get master address from sentinel running @ {}. Reason: {}. Trying next one.", sentinel, e);
      }
    }

    if (master == null) {
      if (sentinelAvailable) {
        // can connect to sentinel, but master name seems to not monitored
        throw new JedisException("Can connect to sentinel, but " + masterName
            + " seems to be not monitored...");
      } else {
        throw new JedisConnectionException("All sentinels down, cannot determine where is "
            + masterName + " master is running...");
      }
    }

    LOG.info("Redis master running at {}, starting Sentinel listeners...", master);

    for (HostAndPort sentinel : sentinels) {

      MasterListener masterListener = new MasterListener(masterName, sentinel.getHost(), sentinel.getPort());
      // whether MasterListener threads are alive or not, process can be stopped
      masterListener.setDaemon(true);
      masterListeners.add(masterListener);
      masterListener.start();
    }

    return master;
  }
這裡最終要的一步就是jedis.sentinelGetMasterAddrByName(masterName);,即向sentinel傳送命令

sentinel get-master-addr-by-name mymaster, 用來獲取master節點的地址,並將地址返回

然後initMaster(master);
  private void initMaster(HostAndPort master) {
    synchronized (initPoolLock) {
      if (!master.equals(currentHostMaster)) {
        currentHostMaster = master;
        // 這裡是容易忽略但非常關鍵的一步
        factory.setHostAndPort(currentHostMaster);
        // although we clear the pool, we still have to check the returned object in getResource,
        // this call only clears idle instances, not borrowed instances
        super.clear();

        LOG.info("Created JedisSentinelPool to master at {}", master);
      }
    }
  }
在這裡對factory的連線地址進行了設定(在之前這裡還是空值)

它是由 構造方法中的 new JedisFactory(masterClientConfig) 構造出來,在單元測試中我們得知masterClientConfig裡面的屬性都是空值

到這裡 sentinelpool 就構造完畢了,其實這裡還沒有初始化出一個連線到master節點的例項,我們繼續往後看

3. 單元測試中下一步 getResouce()

  @Override
  public Jedis getResource() {
    while (true) {
      // 關鍵一步
      Jedis jedis = super.getResource();
      // 這裡沒啥大用,容易誤導
      jedis.setDataSource(this);

      // get a reference because it can change concurrently
      final HostAndPort master = currentHostMaster;
      final HostAndPort connection = jedis.getClient().getHostAndPort();

      if (master.equals(connection)) {
        // connected to the correct master
        return jedis;
      } else {
        returnBrokenResource(jedis);
      }
    }
  }
這裡呼叫 super.getResource(), 父類是Pool, 而Pool 的物件一般是由Factory 構建出來
  public JedisSentinelPool(String masterName, Set<HostAndPort> sentinels,
      final GenericObjectPoolConfig<Jedis> poolConfig, final JedisClientConfig masterClientConfig,
      final JedisClientConfig sentinelClientConfig) {
    this(masterName, sentinels, poolConfig, new JedisFactory(masterClientConfig), sentinelClientConfig);
  }

  public JedisSentinelPool(String masterName, Set<HostAndPort> sentinels,
      final JedisFactory factory, final JedisClientConfig sentinelClientConfig) {
    super(factory);

    this.factory = factory;
    this.sentinelClientConfig = sentinelClientConfig;

    HostAndPort master = initSentinels(sentinels, masterName);
    initMaster(master);
  }

由此可知 factory 是 new JedisFactory(masterClientConfig), 並且由 父類子類都引用到,並且在 initMaster 方法中呼叫factory.setHostAndPort(currentHostMaster); 更新了master的地址。

而Pool extends GenericObjectPool , 這裡GenericObjectPool 來自包 org.apache.commons.pool2
  public T getResource() {
    try {
      return super.borrowObject();
    } catch (JedisException je) {
      throw je;
    } catch (Exception e) {
      throw new JedisException("Could not get a resource from the pool", e);
    }
  }

這裡borrowObject 時,實際是呼叫工廠的方法幹活,直接看工廠類JedisFactory

  @Override
  public PooledObject<Jedis> makeObject() throws Exception {
    Jedis jedis = null;
    try {
      jedis = new Jedis(jedisSocketFactory, clientConfig);
      return new DefaultPooledObject<>(jedis);
    } catch (JedisException je) {
      logger.debug("Error while makeObject", je);
      throw je;
    }
  }
在這裡會構建出Jedis物件 ,注意這裡的jedisSocketFactory物件,實在構造方法中構造出
  protected JedisFactory(final URI uri, final int connectionTimeout, final int soTimeout,
      final int infiniteSoTimeout, final String clientName, final SSLSocketFactory sslSocketFactory,
      final SSLParameters sslParameters, final HostnameVerifier hostnameVerifier) {
    if (!JedisURIHelper.isValid(uri)) {
      throw new InvalidURIException(String.format(
          "Cannot open Redis connection due invalid URI. %s", uri.toString()));
    }
    this.clientConfig = DefaultJedisClientConfig.builder().connectionTimeoutMillis(connectionTimeout)
        .socketTimeoutMillis(soTimeout).blockingSocketTimeoutMillis(infiniteSoTimeout)
        .user(JedisURIHelper.getUser(uri)).password(JedisURIHelper.getPassword(uri))
        .database(JedisURIHelper.getDBIndex(uri)).clientName(clientName)
        .protocol(JedisURIHelper.getRedisProtocol(uri))
        .ssl(JedisURIHelper.isRedisSSLScheme(uri)).sslSocketFactory(sslSocketFactory)
        .sslParameters(sslParameters).hostnameVerifier(hostnameVerifier).build();
    this.jedisSocketFactory = new DefaultJedisSocketFactory(new HostAndPort(uri.getHost(), uri.getPort()), this.clientConfig);
  }

  void setHostAndPort(final HostAndPort hostAndPort) {
    if (!(jedisSocketFactory instanceof DefaultJedisSocketFactory)) {
      throw new IllegalStateException("setHostAndPort method has limited capability.");
    }
    ((DefaultJedisSocketFactory) jedisSocketFactory).updateHostAndPort(hostAndPort);
  }

4. 再看Jedis


  public Jedis(final JedisSocketFactory jedisSocketFactory, final JedisClientConfig clientConfig) {
    connection = new Connection(jedisSocketFactory, clientConfig);
    RedisProtocol proto = clientConfig.getRedisProtocol();
    if (proto != null) commandObjects.setProtocol(proto);
  }

這裡直接構造出Connectrion 物件, 並傳入socketFactory

  public Connection(final JedisSocketFactory socketFactory, JedisClientConfig clientConfig) {
    this.socketFactory = socketFactory;
    this.soTimeout = clientConfig.getSocketTimeoutMillis();
    this.infiniteSoTimeout = clientConfig.getBlockingSocketTimeoutMillis();
    initializeFromClientConfig(clientConfig);
  }

在這個構造方法中執行關鍵方法initializeFromClientConfig

private void initializeFromClientConfig(final JedisClientConfig config) {
    try {
      connect();
	......
      
  }

connect()

  public void connect() throws JedisConnectionException {
    if (!isConnected()) {
      try {
        socket = socketFactory.createSocket();
        soTimeout = socket.getSoTimeout(); //?

        outputStream = new RedisOutputStream(socket.getOutputStream());
        inputStream = new RedisInputStream(socket.getInputStream());

        broken = false; // unset broken status when connection is (re)initialized

      } catch (JedisConnectionException jce) {

        setBroken();
        throw jce;

      } catch (IOException ioe) {

        setBroken();
        throw new JedisConnectionException("Failed to create input/output stream", ioe);

      } finally {

        if (broken) {
          IOUtils.closeQuietly(socket);
        }
      }
    }
  }

最終透過 soeckFactory 構建出socket,完成對redis 的master節點的連線

相關文章