flink維表關聯絡列之Redis維表關聯：實時查詢

點選上方藍

字關注~

       

在做維表關聯如果要求低延時，即維表資料的變更能夠被立刻感知到，所以就要求在查詢時沒有快取策略，直接查詢資料庫維表資訊。
本篇以實時查詢redis為例，要求redis 客戶端支援非同步查詢，可以使用io.lettuce包，支援redis不同模式：單點模式、sentinel模式、叢集模式，需要在pom中引入：

<dependency>
            <groupId>io.lettuce</groupId>
            <artifactId>lettuce-core</artifactId>
            <version>5.0.5.RELEASE</version>
</dependency>
<dependency>
            <groupId>io.netty</groupId>
            <artifactId>netty-all</artifactId>
            <version>4.1.24.Final</version>
 </dependency>

關於其不同模式的用法可以參考：https://juejin.im/post/5d8eb73ff265da5ba5329c66
裡面做了比較詳細的說明，為方便測試使用單點模式，仍以廣告業務為例，根據廣告位ID從redis裡面查詢對位的廣告主ID。

Redis中資料準備：

hmset 1 aid 1 cid 1
hmset 2 aid 1 cid 2

使用hash結構，key表示廣告位ID、aid表示廣告主ID、cid表示廣告計劃ID

定義RichAsyncFunction型別的RedisSide，非同步查詢Redis

class RedisSide extends RichAsyncFunction[AdData, AdData] {

  private var redisClient: RedisClient = _

  private var connection: StatefulRedisConnection[String, String] = _

  private var async: RedisAsyncCommands[String, String] = _

  override def open(parameters: Configuration): Unit = {

    val redisUri = "redis://localhost"
    redisClient = RedisClient.create(redisUri)
    connection = redisClient.connect()
    async = connection.async()
  }


  override def asyncInvoke(input: AdData, resultFuture: ResultFuture[AdData]): Unit = {

    val tid = input.tId.toString
    async.hgetall(tid).thenAccept(new Consumer[util.Map[String, String]]() {
      override def accept(t: util.Map[String, String]): Unit = {

        if (t == null || t.size() == 0) {
          resultFuture.complete(util.Arrays.asList(input))
          return
        }
        t.foreach(x => {
          if ("aid".equals(x._1)) {
            val aid = x._2.toInt
            var newData = AdData(aid, input.tId, input.clientId, input.actionType, input.time)
            resultFuture.complete(util.Arrays.asList(newData))
          }
        })
      }
    })
  }
  //關閉資源
  override def close(): Unit = {
    if (connection != null) connection.close()
    if (redisClient != null) redisClient.shutdown()
  }

}

主流程

case class AdData(aId: Int, tId: Int, clientId: String, actionType: Int, time: Long)

object Demo1 {

  def main(args: Array[String]): Unit = {

    val env = StreamExecutionEnvironment.getExecutionEnvironment
    env.setParallelism(1)

    val kafkaConfig = new Properties();
    kafkaConfig.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
    kafkaConfig.put(ConsumerConfig.GROUP_ID_CONFIG, "test1");
    val consumer = new FlinkKafkaConsumer[String]("topic1", new SimpleStringSchema(), kafkaConfig);
    val ds = env.addSource(consumer)
      .map(x => {
        val a: Array[String] = x.split(",")
        AdData(0, a(0).toInt, a(1), a(2).toInt, a(3).toLong) //預設給0
      })
 
    val redisSide: AsyncFunction[AdData, AdData] = new RedisSide
    AsyncDataStream.unorderedWait(ds, redisSide, 5L, SECONDS, 1000)
      .print()
    env.execute("Demo1")
  }
}

測試驗證
生產資料：

1,clientId1,1,1571646006000
3,clientId1,1,1571646006000

輸出：

AdData(1,1,clientId1,1,1571646006000)
AdData(0,3,clientId1,1,1571646006000)

驗證完畢，也算是補上維表系列裡面的空缺。

關注回覆Flink獲取更多資訊~