生產上有時候需要定期新增或修改某些篩選值.為了避免頻繁重啟任務,修改程式碼,可以將這些配置資訊放在mysql表中定期更新,然後通過傳送廣播流給下游.這樣可以減少更改頻率,降低重啟風險.

public class MysqlSource extends RichSourceFunction<Map<String, ValueConf>> {
    // 定時器間隔時間(ms)
    private static final CountDownLatch countDownLatch = new CountDownLatch(1);
    private final String sql;
    private QueryRunner qr;
    public SignalSource(String sql) {
        this.sql = sql;
    }
    @Override
    public void open(Configuration parameters) throws Exception {
        // 此處需要提前將mysql配置資訊儲存全域性變數中 env.getConfig().setGlobalJobParameters(params);
        ParameterTool params = (ParameterTool) getRuntimeContext().getExecutionConfig().getGlobalJobParameters();
        String jdbcUrl = params.getRequired("mysql.jdbc.url");
        String userName = params.getRequired("mysql.username");
        String password = params.getRequired("mysql.password");
        qr = new QueryRunner(JDBCUtils.getDataSource(jdbcUrl, userName, password));
    }
    @Override
    public void run(SourceContext<Map<String, ValueConf>> ctx) throws Exception {
        while (true) {
            Map<String, ValueConf> rs = qr.query(sql, new BeanListHandler<>(ValueConf.class)).stream().collect(Collectors.toMap(ValueConf::getCode, i -> i));
            ctx.collect(rs);
            // 每隔12小時執行一次
            countDownLatch.await(12, TimeUnit.HOURS);
        }
    }
    @Override
    public void cancel() {
    }}

將mysql資料來源並行度設定為1,只需要一個tm讀取mysql,減少了mysql連線數

MapStateDescriptor<String, ValueConf> descriptor=new MapStateDescriptor<>("mysql-descriptor",String.class,ValueConf.class);BroadcastStream<Map<String, ValueConf>> mysqlSource=env.addSource(new MysqlSource("querySql")).uid("MysqlSource").name("MysqlSource").setParallelism(1).broadcast(descriptor);

依賴
除了flink相關jar包,還需要mysql的,以下是我用的

    <properties>
        <commons.dbutils.version>1.6</commons.dbutils.version>
        <mysql.connector.version>8.0.16</mysql.connector.version>
        <hikaricp.version>4.0.3</hikaricp.version>
    </properties>
        <!--MySQL-java連線驅動-->
        <dependency>
            <groupId>http/:w
            <artifactId>mysql-connector-java</artifactId>
            <version>${mysql.connector.version}</version>
        </dependency>
        <dependency>
            <groupId>commons-dbutils</groupId>
            <artifactId>commons-dbutils</artifactId>
            <version>${commons.dbutils.version}</version>
        </dependency>
        <dependency>
            <groupId>com.zaxxer</groupId>
            <artifactId>HikariCP</artifactId>
            <version>${hikaricp.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-api</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

flink定時讀取mysql

相關文章