Presto通過RESTful介面新增Connector

Gin.p發表於2018-08-19
REST

在實際使用Presto的過程中，經常會有以下的一些需求。

  • 新增一個新的Catalog
  • 對不再使用的Catalog希望把它刪除
  • 修改某個Catalog的引數

但在Presto中如果進行上述的修改，需要重啟Presto服務才可以生效，這給叢集維護帶來額外的工作量之外，還給上層應用帶來很不好的使用體驗。

如果還不能在開發環境很好執行Presto的話，參考在windows的IDEA執行Presto。
觀察PrestoServer的run方法，可以知道Presot分了多個模組，而且多個模組依賴airlift（Airlift framework for building REST services）的專案，倒不如說airlift是Presto的根基。說實在話，我對於Presto的理解可能還在管中窺豹的階段，但是不影響我對它做一些手腳。

1、新增一個Hello world測試介面

在presto-main中com.facebook.presto.server中新增一個CatalogResource，作用和Spring類似吧。

/*
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package com.facebook.presto.server;

import javax.inject.Inject;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.core.Response;

/**
 * @author Gin
 * @since 2018/8/15.
 */
@Path("/v1/catalog")
public class CatalogResource
{
    @Inject
    public CatalogResource()
    {
    }

    @GET
    @Path("test")
    public Response test()
    {
        return Response.ok("Hello world").build();
    }
}

在ServerMainModule的setup()的方法中最後一行註冊CatalogResource：

// Catalogs
jaxrsBinder(binder).bind(CatalogResource.class);

啟動server，訪問http://localhost:8080/v1/catalog/test，如果出現Hello World的話，那麼後面的步驟才行得通。

2、新增一個Add Connector的RESTful介面

新建CatalogInfo用於接收引數：

package com.facebook.presto.server;

import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonProperty;

import java.util.Map;

import static java.util.Objects.requireNonNull;

/**
 * @author Gin
 * @since 2018/8/17.
 */
public class CatalogInfo 
{

    private final String catalogName;

    private final String connectorName;

    private final Map<String, String> properties;

    @JsonCreator
    public CatalogInfo(
            @JsonProperty("catalogName") String catalogName,
            @JsonProperty("connectorName") String connectorName,
            @JsonProperty("properties")Map<String, String> properties)
    {
        this.catalogName = requireNonNull(catalogName, "catalogName is null");
        this.connectorName = requireNonNull(connectorName, "connectorName is null");
        this.properties = requireNonNull(properties, "properties is null");
    }

    @JsonProperty
    public String getCatalogName() {
        return catalogName;
    }

    @JsonProperty
    public String getConnectorName() {
        return connectorName;
    }

    @JsonProperty
    public Map<String, String> getProperties() {
        return properties;
    }
}

在CatalogResource中增加對應的介面，用到的服務用注入方法宣告。

/*
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package com.facebook.presto.server;

import com.facebook.presto.connector.ConnectorId;
import com.facebook.presto.connector.ConnectorManager;
import com.facebook.presto.metadata.InternalNodeManager;
import com.google.common.base.Joiner;
import com.google.common.base.Splitter;
import io.airlift.discovery.client.Announcer;
import io.airlift.discovery.client.ServiceAnnouncement;

import javax.inject.Inject;
import javax.ws.rs.*;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;

import java.util.LinkedHashMap;
import java.util.LinkedHashSet;
import java.util.Map;
import java.util.Set;

import static com.google.common.base.Strings.nullToEmpty;
import static io.airlift.discovery.client.ServiceAnnouncement.serviceAnnouncement;
import static java.util.Objects.requireNonNull;

/**
 * @author Gin
 * @since 2018/8/15.
 */
@Path("/v1/catalog")
public class CatalogResource
{
    private final ConnectorManager connectorManager;
    private final Announcer announcer;

    @Inject
    public CatalogResource(
            ConnectorManager connectorManager,
            Announcer announcer)
    {
        this.connectorManager = requireNonNull(connectorManager, "connectorManager is null");
        this.announcer = requireNonNull(announcer, "announcer is null");
    }

    @GET
    @Path("test")
    public Response test()
    {
        return Response.ok("Hello world").build();
    }

    @PUT
    @Consumes(MediaType.APPLICATION_JSON)
    @Produces(MediaType.APPLICATION_JSON)
    public Response createCatalog(CatalogInfo catalogInfo)
    {
        requireNonNull(catalogInfo, "catalogInfo is null");

        ConnectorId connectorId = connectorManager.createConnection(
                catalogInfo.getCatalogName(),
                catalogInfo.getConnectorName(),
                catalogInfo.getProperties());

        updateConnectorIdAnnouncement(announcer, connectorId);
        return Response.status(Response.Status.OK).build();
    }

    private static void updateConnectorIdAnnouncement(Announcer announcer, ConnectorId connectorId)
    {
        //
        // This code was copied from PrestoServer, and is a hack that should be removed when the connectorId property is removed
        //

        // get existing announcement
        ServiceAnnouncement announcement = getPrestoAnnouncement(announcer.getServiceAnnouncements());

        // update connectorIds property
        Map<String, String> properties = new LinkedHashMap<>(announcement.getProperties());
        String property = nullToEmpty(properties.get("connectorIds"));
        Set<String> connectorIds = new LinkedHashSet<>(Splitter.on(',').trimResults().omitEmptyStrings().splitToList(property));
        connectorIds.add(connectorId.toString());
        properties.put("connectorIds", Joiner.on(',').join(connectorIds));

        // update announcement
        announcer.removeServiceAnnouncement(announcement.getId());
        announcer.addServiceAnnouncement(serviceAnnouncement(announcement.getType()).addProperties(properties).build());
        announcer.forceAnnounce();
    }

    private static ServiceAnnouncement getPrestoAnnouncement(Set<ServiceAnnouncement> announcements)
    {
        for (ServiceAnnouncement announcement : announcements) {
            if (announcement.getType().equals("presto")) {
                return announcement;
            }
        }
        throw new RuntimeException("Presto announcement not found: " + announcements);
    }
}

3、測試RESTful介面

這步需要安裝需要的外掛，檢查外掛是否安裝。使用postman類似的東西，傳送application/json的PUT請求到http://localhost:8080/v1/catalog/，body為

{
    "catalogName": "test",
    "connectorName": "mysql",
    "properties": {
        "connection-url":"jdbc:mysql://localhost:3306",
        "connection-user":"root",
        "connection-password":"root"
    }
}

我們可以看到控制檯，重新輸出了connector的資訊：

2018-08-19T14:09:03.502+0800    INFO    main    com.facebook.presto.server.PrestoServer ======== SERVER STARTED ========
2018-08-19T14:09:23.496+0800    INFO    http-worker-133 Bootstrap   PROPERTY                  DEFAULT     RUNTIME                      DESCRIPTION
2018-08-19T14:09:23.496+0800    INFO    http-worker-133 Bootstrap   connection-password       [REDACTED]  [REDACTED]
2018-08-19T14:09:23.496+0800    INFO    http-worker-133 Bootstrap   connection-url            null        jdbc:mysql://localhost:3306
2018-08-19T14:09:23.496+0800    INFO    http-worker-133 Bootstrap   connection-user           null        root
2018-08-19T14:09:23.496+0800    INFO    http-worker-133 Bootstrap   allow-drop-table          false       false                        Allow connector to drop tables
2018-08-19T14:09:23.496+0800    INFO    http-worker-133 Bootstrap   mysql.auto-reconnect      true        true
2018-08-19T14:09:23.496+0800    INFO    http-worker-133 Bootstrap   mysql.connection-timeout  10.00s      10.00s
2018-08-19T14:09:23.496+0800    INFO    http-worker-133 Bootstrap   mysql.max-reconnects      3           3
2018-08-19T14:09:23.876+0800    INFO    http-worker-133 io.airlift.bootstrap.LifeCycleManager   Life cycle starting...
2018-08-19T14:09:23.877+0800    INFO    http-worker-133 io.airlift.bootstrap.LifeCycleManager   Life cycle startup complete. System ready.

接下來要怎麼利用，要看大家的腦洞了:)?

參考文獻：
Presto技術內幕

相關文章