Presto Event Listener開發

叄金發表於2019-07-30
REST

簡介

同Hive Hook一樣，Presto也支援自定義實現Event Listener，用於偵聽Presto引擎執行查詢時發生的事件，並作出相應的處理。我們可以利用該功能實現諸如自定義日誌記錄、除錯和效能分析外掛，幫助我們更好的運維Presto叢集。但是不同於Hive Hook的是，在Presto叢集中，一次只能有一個Event Listener處於活動狀態。

Event Listener作為Plugin監聽以下事件：

  • Query Creation(查詢建立相關資訊)
  • Query completion (success or failure)(查詢執行相關資訊，包含成功查詢的細節資訊，失敗查詢的錯誤碼等資訊)
  • Split completion (success or failure)(split執行資訊，同理包含成功和失敗的細節資訊)

瞭解Hook及Listener模式的朋友對於其步驟應該很清楚了，我們只需要：

  1. 實現Presto Event Listener和EventListenerFactory介面。
  2. 正確的打包我們的jar。
  3. 部署，放到Presto指定目錄，修改配置檔案。

介面

  1. 實現EventListener，該類是我們的核心邏輯所在，供包含上面所說的三個事件：
public interface EventListener
{
    //query建立的詳細資訊
    default void queryCreated(QueryCreatedEvent queryCreatedEvent)
    {
    }
    //query執行的詳細資訊
    default void queryCompleted(QueryCompletedEvent queryCompletedEvent)
    {
    }
    //split執行的詳細資訊
    default void splitCompleted(SplitCompletedEvent splitCompletedEvent)
    {
    }
}
  1. 實現EventListenerFactory建立我們自己實現的EventListener
  2. 實現Plugin介面，實現getEventListenerFactories()方法，獲取我們自己實現的EventListenerFactory
  3. 新增配置資訊，為etc/event-listener.properties。其中event-listener.name為必備屬性，其他屬性為我們plugin所需要的資訊。

示例

由於叢集運維的需要，先需要將使用者的查詢歷史、查詢花費的時間等資訊進行統計，以便於後續對各個業務的查詢進行優先順序分級和評分，方便後續Presto叢集穩定性易用性的維護。這裡給出一個簡單的將這些資訊儲存到Mysql資料庫的樣例。

Maven Pom

<dependency>
      <groupId>com.facebook.presto</groupId>
      <artifactId>presto-spi</artifactId>
      <version>0.220</version>
      <scope>compile</scope>
    </dependency>

QueryEventListenerFactory

public class QueryEventListenerFactory implements EventListenerFactory {

  @Override
  public String getName() {
    return "query-event-listener";
  }

  @Override
  public EventListener create(Map<String, String> config) {
    if (!config.containsKey("jdbc.uri")) {
      throw new RuntimeException("/etc/event-listener.properties file missing jdbc.uri");
    }
    if (!config.containsKey("jdbc.user")) {
      throw new RuntimeException("/etc/event-listener.properties file missing jdbc.user");
    }
    if (!config.containsKey("jdbc.pwd")) {
      throw new RuntimeException("/etc/event-listener.properties file missing jdbc.pwd");
    }

    return new QueryEventListener(config);
  }
}

QueryEventPlugin

public class QueryEventPlugin implements Plugin {

  @Override
  public Iterable<EventListenerFactory> getEventListenerFactories() {
    EventListenerFactory listenerFactory = new QueryEventListenerFactory();
    return Arrays.asList(listenerFactory);
  }
}

QueryEventListener

public class QueryEventListener implements EventListener {

  private Map<String, String> config;
  private Connection connection;

  public QueryEventListener(Map<String, String> config) {
    this.config = new HashMap<>();
    this.config.putAll(config);
    init();
  }

  private void init() {
    try {
      if (connection == null || !connection.isValid(10)) {
        Class.forName("com.mysql.jdbc.Driver");
        connection = DriverManager
            .getConnection(config.get("jdbc.uri"), config.get("jdbc.user"), config.get("jdbc.pwd"));
      }
    } catch (SQLException | ClassNotFoundException e) {
      e.printStackTrace();
    }
  }

  @Override
  public void queryCreated(QueryCreatedEvent queryCreatedEvent) {
  }

  @Override
  public void queryCompleted(QueryCompletedEvent queryCompletedEvent) {
    String queryId = queryCompletedEvent.getMetadata().getQueryId();
    String querySql = queryCompletedEvent.getMetadata().getQuery();
    String queryState = queryCompletedEvent.getMetadata().getQueryState();
    String queryUser = queryCompletedEvent.getContext().getUser();
    long createTime = queryCompletedEvent.getCreateTime().toEpochMilli();
    long endTime = queryCompletedEvent.getEndTime().toEpochMilli();
    long startTime = queryCompletedEvent.getExecutionStartTime().toEpochMilli();
    //insert into query execution table

    long analysisTime = queryCompletedEvent.getStatistics().getAnalysisTime().orElse(Duration.ZERO)
        .toMillis();
    long cpuTime = queryCompletedEvent.getStatistics().getCpuTime().toMillis();
    long queuedTime = queryCompletedEvent.getStatistics().getQueuedTime().toMillis();
    long wallTime = queryCompletedEvent.getStatistics().getWallTime().toMillis();
    int completedSplits = queryCompletedEvent.getStatistics().getCompletedSplits();
    double cumulativeMemory = queryCompletedEvent.getStatistics().getCumulativeMemory();
    long outputBytes = queryCompletedEvent.getStatistics().getOutputBytes();
    long outputRows = queryCompletedEvent.getStatistics().getOutputRows();
    long totalBytes = queryCompletedEvent.getStatistics().getTotalBytes();
    long totalRows = queryCompletedEvent.getStatistics().getTotalRows();
    long writtenBytes = queryCompletedEvent.getStatistics().getWrittenBytes();
    long writtenRows = queryCompletedEvent.getStatistics().getWrittenRows();
    //insert into query info table
    
    queryCompletedEvent.getFailureInfo().ifPresent(queryFailureInfo -> {
      int code = queryFailureInfo.getErrorCode().getCode();
      String name = queryFailureInfo.getErrorCode().getName();
      String failureType = queryFailureInfo.getFailureType().orElse("").toUpperCase();
      String failureHost = queryFailureInfo.getFailureHost().orElse("").toUpperCase();
      String failureMessage = queryFailureInfo.getFailureMessage().orElse("").toUpperCase();
      String failureTask = queryFailureInfo.getFailureTask().orElse("").toUpperCase();
      String failuresJson = queryFailureInfo.getFailuresJson();
      // insert into failed query table
    });
  }


  @Override
  public void splitCompleted(SplitCompletedEvent splitCompletedEvent) {
    long createTime = splitCompletedEvent.getCreateTime().toEpochMilli();
    long endTime = splitCompletedEvent.getEndTime().orElse(Instant.MIN).toEpochMilli();
    String payload = splitCompletedEvent.getPayload();
    String queryId = splitCompletedEvent.getQueryId();
    String stageId = splitCompletedEvent.getStageId();
    long startTime = splitCompletedEvent.getStartTime().orElse(Instant.MIN).toEpochMilli();
    String taskId = splitCompletedEvent.getTaskId();
    long completedDataSizeBytes = splitCompletedEvent.getStatistics().getCompletedDataSizeBytes();
    long completedPositions = splitCompletedEvent.getStatistics().getCompletedPositions();
    long completedReadTime = splitCompletedEvent.getStatistics().getCompletedReadTime().toMillis();
    long cpuTime = splitCompletedEvent.getStatistics().getCpuTime().toMillis();
    long queuedTime = splitCompletedEvent.getStatistics().getQueuedTime().toMillis();
    long wallTime = splitCompletedEvent.getStatistics().getWallTime().toMillis();
    //insert into stage info table
  }

}

打包

  1. Presto使用服務提供者介面（SPI）來擴充套件Presto。Presto使用SPI載入聯結器，功能，型別和系統訪問控制。SPI通過後設資料檔案載入。我們還需要建立src/main/resources/META-INF/services/com.facebook.presto.spi.Plugin後設資料檔案。該檔案應包含我們外掛的類名如： com.ji3jin.presto.listener.QueryEventListener
  2. 執行mvn clean install打包

部署

  1. 建立配置檔案etc/event-listener.properties
event-listener.name=query-event-listener

jdbc.uri=jdbc:mysql://localhost:3306/presto_monitor
jdbc.user=presto
jdbc.pwd=presto123
  1. 在presto根目錄下建立query-event-listener目錄，名稱與我們上面event listener的name一致
  2. 將我們的jar包和mysql connector的jar包拷貝到上面建立的目錄
  3. 重新啟動Presto服務即可

好了，現在你可以執行查詢，然後就可以在Mysql中看到你的查詢歷史和相關時間的統計資訊了。如果你目前的工作對此也有需要，還等什麼，快動手實現一個吧。

歡迎關注我的公眾號：叄金大資料
Presto Event Listener開發

相關文章