Druid中coordinator的介紹

Xlucas發表於2017-09-08
UI

核心
1、主要介紹coordinate的作用
2、藉助coordinate完成一些查詢操作

coordinate的作用
druid協調節點主要負責管理segment和分發，更具體的說，druid協調節點和歷史節點進行通訊，根據配置進行載入segment或刪除segment。druid的協調節點負責載入新的segment，刪除過時的segment，管理segment副本和做segment的平衡。druid協調器定期執行，每次執行之間的時間是一個可配置的引數，每次druid協調器執行的時候，它將決定採取的適當行動之前評估叢集的當前狀態。和broker和historical節點相似，druid協調器和zookeeper連線獲取當前叢集的資訊，協調器還維護與包含segment和rules資訊的資料庫的連線，有效的segment儲存在segment表裡，並列出叢集中載入的所有segment，rules儲存在rule表中並提升如何處理segment。在historical節點沒有任何segment的時候，每層的可用historical首先按容量排序，最小容量伺服器具有最高優先順序，每個分配的segment總是分配給具有最小容量的節點，以保持節點之間的平衡，協調器在為其分配新的segment時不直接和historical進行通訊，而是在historical節點的載入佇列路徑下建立新segment的一些臨時資訊，一旦看到這個請求，historical將載入該segment並開始服務。

執行協調器
io.druid.cli.Main server coordinator

rules
可以根據一組規則從叢集自動載入和刪除segment

清理segment
每次執行，協調器將資料庫中有效的segment與當前叢集中segment進行比較，不在資料庫中但仍在叢集中的segment將會被標記並新增到刪除列表裡面，被覆蓋的segment
(版本太舊了，資料已被新的segment替代)也被刪除，如果資料庫中所有segment都被刪除，則協調器將不會刪除historical任何內容。

segment可用性
如果historical節點由於任何 原因重啟或者變得不可用，則druid協調器將監聽到一個節點已經丟失，並將該節點所服務的所有segment都視為丟棄，給定足夠的時間，可以將segment重新分配給叢集中
的其他historical節點，但是，每個segment都不會立即被刪除，有一個過渡的資料結構儲存具有關聯壽命的所有丟棄的segment，在生命週期內協調器不會重新分配segment，因此，如果historical
在短時間內變得不可以用並再次可用，historical節點在啟動並從快取記憶體中提取segment，而不會在整個叢集中重新分配這個segment

segment平衡
為了確保叢集中的歷史節點之間的分配均勻，協調器將在每次協調器執行時找到每個historical所有服務的segment的總大小，對於叢集中的每個historical節點，協調器將確定利用率最高的historical和利用率最低的historical,計算兩個節點之間利用率的差異百分比，如果超過某個閥值，則從最高使用節點移動到最低使用節點，每次協調器有執行，可以從一個節點移動到另一個節點的segment數量可配置

http請求
協調器公開了多個http請求進行互動

coordinate獲取叢集資訊
GET

/status

返回druid版本，載入的擴充套件，使用的記憶體，總記憶體和其他有關節點的有用資訊

[root@SZB-L0038788 druid-0.10.0]# curl -X GET  http://10.20.23.38:8081/status
{
    "version": "0.10.0",
    "modules": [
        {
            "name": "io.druid.storage.hdfs.HdfsStorageDruidModule",
            "artifact": "druid-hdfs-storage",
            "version": "0.10.0"
        },
        {
            "name": "io.druid.server.lookup.namespace.NamespaceExtractionModule",
            "artifact": "druid-lookups-cached-global",
            "version": "0.10.0"
        },
        {
            "name": "io.druid.firehose.kafka.KafkaEightDruidModule",
            "artifact": "druid-kafka-eight",
            "version": "0.10.0"
        },
        {
            "name": "io.druid.query.aggregation.datasketches.theta.oldapi.OldApiSketchModule",
            "artifact": "druid-datasketches",
            "version": "0.10.0"
        },
        {
            "name": "io.druid.metadata.storage.mysql.MySQLMetadataStorageModule",
            "artifact": "mysql-metadata-storage",
            "version": "0.10.0"
        },
        {
            "name": "io.druid.query.aggregation.datasketches.theta.SketchModule",
            "artifact": "druid-datasketches",
            "version": "0.10.0"
        },
        {
            "name": "io.druid.query.aggregation.histogram.ApproximateHistogramDruidModule",
            "artifact": "druid-histogram",
            "version": "0.10.0"
        }
    ],
    "memory": {
        "maxMemory": 266862592,
        "totalMemory": 266862592,
        "freeMemory": 157169296,
        "usedMemory": 109693296
    }
}

coordinater 資訊

/druid/coordinator/v1/leader

返回叢集當前leader 協調器

 [root@SZB-L0038788 druid-0.10.0]# curl 
 http://10.20.23.38:8081/druid/coordinator/v1/leader
 10.20.23.38:8081

/druid/coordinator/v1/isLeader

返回帶有 leader的json物件，為true或者false，表改伺服器是否是該叢集的leader協調器，此外，如果伺服器是當前的leader，則返回HTTP 200否則返回HTTP 400，如果你只希望
active leader在負載均衡上被考慮在服務上，則適用於負載均衡狀態檢查

/druid/coordinator/v1/loadstatus

返回在叢集中實際載入segment的百分比應該在叢集中載入segment的百分比

{
    "wikiticker123": 100,
    "app_auto_prem_qd_pp1": 100,
    "app_auto_prem_qd_pp3": 100
}

/druid/coordinator/v1/loadstatus?simple

返回待載入的segment，直到在叢集中載入的segment可用於查詢，不包括複製

{
    "wikiticker123": 0,
    "app_auto_prem_qd_pp1": 0,
    "app_auto_prem_qd_pp3": 0
}

/druid/coordinator/v1/loadstatus?full

返回每個DataSource載入的segment，直到在叢集中載入的segment都可以用，這個包括複製

{
    "_default_tier": {
        "wikiticker123": 1,
        "app_auto_prem_qd_pp1": 2706,
        "app_auto_prem_qd_pp3": 1
    }
}

/druid/coordinator/v1/loadqueue

返回每個歷史節點載入和刪除的segmentids

{
    "10.20.23.38:8083": {
        "segmentsToLoad": [],
        "segmentsToDrop": []
    }
}

/druid/coordinator/v1/loadqueue?simple

返回要載入和刪除的segment數，以及每個historical節點的總segment載入和總大小

{
    "10.20.23.38:8083": {
        "segmentsToLoad": 0,
        "segmentsToDrop": 0,
        "segmentsToLoadSize": 0,
        "segmentsToDropSize": 0
    }
}

/druid/coordinator/v1/loadqueue?full

返回要為每個historical載入和刪除的segment序列化json

{
    "10.20.23.38:8083": {
        "segmentsToLoad": [],
        "segmentsToDrop": []
    }
}

後設資料資訊

/druid/coordinator/v1/metadata/datasources

獲取叢集中DataSource的列表

[
    "app_auto_prem_qd_pp1",
    "app_auto_prem_qd_pp3",
    "wikiticker123"
]

/druid/coordinator/v1/metadata/datasources?includeDisabled

返回叢集中啟用和禁用的DataSource

[
    "app_auto_prem_qd_pp1",
    "app_auto_prem_qd_pp3",
    "wikiticker",
    "wikiticker11",
    "wikiticker123"
]

/druid/coordinator/v1/metadata/datasources?full

返回所有啟用的資料來源列表，其中包含儲存的後設資料儲存中的相關這些資料來源的所有後設資料

{
    "name": "wikiticker123",
    "properties": {
        "created": "2017-09-08T17:46:17.006+08:00"
    },
    "segments": [
        {
            "dataSource": "wikiticker123",
            "interval": "2015-09-12T08:00:00.000+08:00/2015-09-13T08:00:00.000+08:00",
            "version": "2017-08-03T17:43:40.525+08:00",
            "loadSpec": {
                "type": "hdfs",
                "path": "hdfs://10.20.23.29:9000/druid/index/output/wikiticker123/20150912T080000.000+0800_20150913T080000.000+0800/2017-08-03T17_43_40.525+08_00/0_index.zip"
            },
            "dimensions": "channel,cityName,comment,countryIsoCode,countryName,isAnonymous,isMinor,isNew,isRobot,isUnpatrolled,metroCode,namespace,page,regionIsoCode,regionName,user",
            "metrics": "count,added,deleted,delta,user_unique",
            "shardSpec": {
                "type": "none"
            },
            "binaryVersion": 9,
            "size": 5537818,
            "identifier": "wikiticker123_2015-09-12T08:00:00.000+08:00_2015-09-13T08:00:00.000+08:00_2017-08-03T17:43:40.525+08:00"
        }
    ]
}

/druid/coordinator/v1/metadata/datasources/{dataSourceName}

返回指定DataSource的完整後設資料

{
    "name": "wikiticker123",
    "properties": {
        "created": "2017-09-08T17:49:17.231+08:00"
    },
    "segments": [
        {
            "dataSource": "wikiticker123",
            "interval": "2015-09-12T08:00:00.000+08:00/2015-09-13T08:00:00.000+08:00",
            "version": "2017-08-03T17:43:40.525+08:00",
            "loadSpec": {
                "type": "hdfs",
                "path": "hdfs://10.20.23.29:9000/druid/index/output/wikiticker123/20150912T080000.000+0800_20150913T080000.000+0800/2017-08-03T17_43_40.525+08_00/0_index.zip"
            },
            "dimensions": "channel,cityName,comment,countryIsoCode,countryName,isAnonymous,isMinor,isNew,isRobot,isUnpatrolled,metroCode,namespace,page,regionIsoCode,regionName,user",
            "metrics": "count,added,deleted,delta,user_unique",
            "shardSpec": {
                "type": "none"
            },
            "binaryVersion": 9,
            "size": 5537818,
            "identifier": "wikiticker123_2015-09-12T08:00:00.000+08:00_2015-09-13T08:00:00.000+08:00_2017-08-03T17:43:40.525+08:00"
        }
    ]
}

/druid/coordinator/v1/metadata/datasources/{dataSourceName}/segments

返回後設資料中所有segment的列表

[
    "wikiticker123_2015-09-12T08:00:00.000+08:00_2015-09-13T08:00:00.000+08:00_2017-08-03T17:43:40.525+08:00"
]

/druid/coordinator/v1/metadata/datasources/{dataSourceName}/segments?full

返回後設資料所有segment的資訊

[
    {
        "dataSource": "wikiticker123",
        "interval": "2015-09-12T08:00:00.000+08:00/2015-09-13T08:00:00.000+08:00",
        "version": "2017-08-03T17:43:40.525+08:00",
        "loadSpec": {
            "type": "hdfs",
            "path": "hdfs://10.20.23.29:9000/druid/index/output/wikiticker123/20150912T080000.000+0800_20150913T080000.000+0800/2017-08-03T17_43_40.525+08_00/0_index.zip"
        },
        "dimensions": "channel,cityName,comment,countryIsoCode,countryName,isAnonymous,isMinor,isNew,isRobot,isUnpatrolled,metroCode,namespace,page,regionIsoCode,regionName,user",
        "metrics": "count,added,deleted,delta,user_unique",
        "shardSpec": {
            "type": "none"
        },
        "binaryVersion": 9,
        "size": 5537818,
        "identifier": "wikiticker123_2015-09-12T08:00:00.000+08:00_2015-09-13T08:00:00.000+08:00_2017-08-03T17:43:40.525+08:00"
    }
]

POST /druid/coordinator/v1/metadata/datasources/{dataSourceName}/segments

返回DataSource的所有segment列表，所有的segment重疊

例如  ["2012-01-01T00:00:00.000/2012-01-03T00:00:00.000", "2012-01-05T00:00:00.000/2012-01-07T00:00:00.000"]
[root@SZB-L0038788 druid-0.10.0]# curl -POST http://10.20.23.38:8081/druid/coordinator/v1/metadata/datasources/wikiticker123/segments
["wikiticker123_2015-09-12T08:00:00.000+08:00_2015-09-13T08:00:00.000+08:00_2017-08-03T17:43:40.525+08:00"]

POST
/druid/coordinator/v1/metadata/datasources/{dataSourceName}/segments?full

返回DataSource的所有segment列表的完整資訊，所有的segment重疊

[root@SZB-L0038788 druid-0.10.0]# curl -POST http://10.20.23.38:8081/druid/coordinator/v1/metadata/datasources/wikiticker123/segments?full
[
    {
        "dataSource": "wikiticker123",
        "interval": "2015-09-12T08:00:00.000+08:00/2015-09-13T08:00:00.000+08:00",
        "version": "2017-08-03T17:43:40.525+08:00",
        "loadSpec": {
            "type": "hdfs",
            "path": "hdfs://10.20.23.29:9000/druid/index/output/wikiticker123/20150912T080000.000+0800_20150913T080000.000+0800/2017-08-03T17_43_40.525+08_00/0_index.zip"
        },
        "dimensions": "channel,cityName,comment,countryIsoCode,countryName,isAnonymous,isMinor,isNew,isRobot,isUnpatrolled,metroCode,namespace,page,regionIsoCode,regionName,user",
        "metrics": "count,added,deleted,delta,user_unique",
        "shardSpec": {
            "type": "none"
        },
        "binaryVersion": 9,
        "size": 5537818,
        "identifier": "wikiticker123_2015-09-12T08:00:00.000+08:00_2015-09-13T08:00:00.000+08:00_2017-08-03T17:43:40.525+08:00"
    }
]

/druid/coordinator/v1/metadata/datasources/{dataSourceName}/segments/{segmentId}

返回儲存在後設資料儲存中指定的完成後設資料

curl -POST http://10.20.23.38:8081/druid/coordinator/v1/metadata/datasources/wikiticker123/segments/wikiticker123_2015-09-12T08:00:00.000+08:00_2015-09-13T08:00:00.000+08:00_2017-08-03T17:43:40.525+08:00
{
    "dataSource": "wikiticker123",
    "interval": "2015-09-12T08:00:00.000+08:00/2015-09-13T08:00:00.000+08:00",
    "version": "2017-08-03T17:43:40.525+08:00",
    "loadSpec": {
        "type": "hdfs",
        "path": "hdfs://10.20.23.29:9000/druid/index/output/wikiticker123/20150912T080000.000+0800_20150913T080000.000+0800/2017-08-03T17_43_40.525+08_00/0_index.zip"
    },
    "dimensions": "channel,cityName,comment,countryIsoCode,countryName,isAnonymous,isMinor,isNew,isRobot,isUnpatrolled,metroCode,namespace,page,regionIsoCode,regionName,user",
    "metrics": "count,added,deleted,delta,user_unique",
    "shardSpec": {
        "type": "none"
    },
    "binaryVersion": 9,
    "size": 5537818,
    "identifier": "wikiticker123_2015-09-12T08:00:00.000+08:00_2015-09-13T08:00:00.000+08:00_2017-08-03T17:43:40.525+08:00"
}

資料來源資訊

/druid/coordinator/v1/datasources

返回叢集中DataSource列表

[
    "app_auto_prem_qd_pp1",
    "app_auto_prem_qd_pp3",
    "wikiticker123"
]

/druid/coordinator/v1/datasources?simple

返回叢集中DataSource列表包含segment計算，總segment位元組大小，minTime和maxTime

[
    {
        "name": "app_auto_prem_qd_pp1",
        "properties": {
            "tiers": {
                "_default_tier": {
                    "size": 40878466,
                    "segmentCount": 2706
                }
            },
            "segments": {
                "maxTime": "2017-07-28T08:00:00.000+08:00",
                "size": 40878466,
                "minTime": "2003-02-25T08:00:00.000+08:00",
                "count": 2706
            }
        }
    },
    {
        "name": "app_auto_prem_qd_pp3",
        "properties": {
            "tiers": {
                "_default_tier": {
                    "size": 82445053,
                    "segmentCount": 1
                }
            },
            "segments": {
                "maxTime": "2017-08-23T08:00:00.000+08:00",
                "size": 82445053,
                "minTime": "2017-08-22T08:00:00.000+08:00",
                "count": 1
            }
        }
    },
    {
        "name": "wikiticker123",
        "properties": {
            "tiers": {
                "_default_tier": {
                    "size": 5537818,
                    "segmentCount": 1
                }
            },
            "segments": {
                "maxTime": "2015-09-13T08:00:00.000+08:00",
                "size": 5537818,
                "minTime": "2015-09-12T08:00:00.000+08:00",
                "count": 1
            }
        }
    }
]

/druid/coordinator/v1/datasources?full

返回叢集中DataSource列表，其中包含這些DataSource所有後設資料

{
    "name": "wikiticker123",
    "properties": {
        "created": "2017-09-08T17:46:17.006+08:00"
    },
    "segments": [
        {
            "dataSource": "wikiticker123",
            "interval": "2015-09-12T08:00:00.000+08:00/2015-09-13T08:00:00.000+08:00",
            "version": "2017-08-03T17:43:40.525+08:00",
            "loadSpec": {
                "type": "hdfs",
                "path": "hdfs://10.20.23.29:9000/druid/index/output/wikiticker123/20150912T080000.000+0800_20150913T080000.000+0800/2017-08-03T17_43_40.525+08_00/0_index.zip"
            },
            "dimensions": "channel,cityName,comment,countryIsoCode,countryName,isAnonymous,isMinor,isNew,isRobot,isUnpatrolled,metroCode,namespace,page,regionIsoCode,regionName,user",
            "metrics": "count,added,deleted,delta,user_unique",
            "shardSpec": {
                "type": "none"
            },
            "binaryVersion": 9,
            "size": 5537818,
            "identifier": "wikiticker123_2015-09-12T08:00:00.000+08:00_2015-09-13T08:00:00.000+08:00_2017-08-03T17:43:40.525+08:00"
        }
    ]
}

/druid/coordinator/v1/datasources/{dataSourceName}

返回指定DataSource的json物件，包括segment計數，總segment大小，minTime和maxTime

/druid/coordinator/v1/datasources/{dataSourceName}?full

返回DataSource完整後設資料

/druid/coordinator/v1/datasources/{dataSourceName}/intervals

返回一組segment間隔

/druid/coordinator/v1/datasources/{dataSourceName}/intervals?simple

返回指定DataSource的segment全部資訊

/druid/coordinator/v1/datasources/{dataSourceName}/intervals?full

返回DataSource對映到該間隔的伺服器名稱

/druid/coordinator/v1/datasources/{dataSourceName}/intervals/{interval}

返回間隔一組segmentID，需要注意 間隔引數有/修改成_例如2017-07-27T08:00:00.000+08:00/2017-07-28T08:00:00.000+08:00需要寫成 2017-07-27T08:00:00.000+08:00_2017-07-28T08:00:00.000+08:00

/druid/coordinator/v1/datasources/{dataSourceName}/intervals/{interval}?simple

將segment間隔的大小和數量返回為json

{
    "2015-09-12T08:00:00.000+08:00/2015-09-13T08:00:00.000+08:00": {
        "size": 5537818,
        "count": 1
    }
}

/druid/coordinator/v1/datasources/{dataSourceName}/intervals/{interval}?full

將segment的大小完整後設資料返回

{
    "2015-09-12T08:00:00.000+08:00/2015-09-13T08:00:00.000+08:00": {
        "wikiticker123_2015-09-12T08:00:00.000+08:00_2015-09-13T08:00:00.000+08:00_2017-08-03T17:43:40.525+08:00": {
            "metadata": {
                "dataSource": "wikiticker123",
                "interval": "2015-09-12T08:00:00.000+08:00/2015-09-13T08:00:00.000+08:00",
                "version": "2017-08-03T17:43:40.525+08:00",
                "loadSpec": {
                    "type": "hdfs",
                    "path": "hdfs://10.20.23.29:9000/druid/index/output/wikiticker123/20150912T080000.000+0800_20150913T080000.000+0800/2017-08-03T17_43_40.525+08_00/0_index.zip"
                },
                "dimensions": "channel,cityName,comment,countryIsoCode,countryName,isAnonymous,isMinor,isNew,isRobot,isUnpatrolled,metroCode,namespace,page,regionIsoCode,regionName,user",
                "metrics": "count,added,deleted,delta,user_unique",
                "shardSpec": {
                    "type": "none"
                },
                "binaryVersion": 9,
                "size": 5537818,
                "identifier": "wikiticker123_2015-09-12T08:00:00.000+08:00_2015-09-13T08:00:00.000+08:00_2017-08-03T17:43:40.525+08:00"
            },
            "servers": [
                "10.20.23.38:8083"
            ]
        }
    }
}

/druid/coordinator/v1/datasources/{dataSourceName}/intervals/{interval}/serverview

返回segment的大小完整資訊包含伺服器資訊

/druid/coordinator/v1/datasources/{dataSourceName}/segments

返回指定dataSource的所有segment列表

/druid/coordinator/v1/datasources/{dataSourceName}/segments?full

返回指定dataSource的所有segment列表的完整資訊

/druid/coordinator/v1/datasources/{dataSourceName}/segments/{segmentId}

返回指定dataSource的指定segment的完整資訊

/druid/coordinator/v1/datasources/{dataSourceName}/tiers

返回DataSource的規則

rules 規則

/druid/coordinator/v1/rules

返回叢集中所有規則

{
    "_default": [
        {
            "tieredReplicants": {
                "_default_tier": 2
            },
            "type": "loadForever"
        }
    ]
}

/druid/coordinator/v1/rules/{dataSourceName}

返回指定資料來源的規則

/druid/coordinator/v1/rules/{dataSourceName}?full

返回指定資料來源的所有規則

/druid/coordinator/v1/rules/history?interval=< interval>

返回所有DataSource的規則的審計歷史記錄，可以配置druid.audit.manager.auditHistoryMillis這個執行間隔預設是1周

{
    "error": "Invalid format: \"2015-09-12T08:00:00.000 08:00\" is malformed at \" 08:00\""
}

/druid/coordinator/v1/rules/history?count=< n>

返回所有資料來源的規則審計數目

/druid/coordinator/v1/rules/{dataSourceName}/history?interval=<
interval>

返回指定DataSource的規則的審計歷史記錄，可以配置druid.audit.manager.auditHistoryMillis這個執行間隔預設是1周

/druid/coordinator/v1/rules/{dataSourceName}/history?count=< n>

返回指定資料來源的規則審計數目

Intervals 間隔
需要注意分隔符有_替換 /

/druid/coordinator/v1/intervals

返回所有資料來源總大小和數量的間隔

/druid/coordinator/v1/intervals/{interval}

返回指定間隔的總大小和數量

curl http://10.20.23.38:8081/druid/coordinator/v1/intervals/2015-09-12T08:00:00.000+08:00_2015-09-13T08:00:00.000+08:00
{
    "size": 5550558,
    "count": 2
}

/druid/coordinator/v1/intervals/{interval}?simple

返回指定間隔的大小和數量

{
    "2015-09-12T08:00:00.000+08:00/2015-09-13T08:00:00.000+08:00": {
        "size": 5550558,
        "count": 2
    }
}

/druid/coordinator/v1/intervals/{interval}?full

返回指定間隔每個資料來源的大小和數量

{
    "2015-09-12T08:00:00.000+08:00/2015-09-13T08:00:00.000+08:00": {
        "wikiticker123": {
            "size": 5537818,
            "count": 1
        },
        "app_auto_prem_qd_pp1": {
            "size": 12740,
            "count": 1
        }
    }
}

POST

/druid/coordinator/v1/datasources/{dataSourceName}

返回指定資料來源規則和segment資訊

{
    "tiers": {
        "_default_tier": {
            "size": 5537818,
            "segmentCount": 1
        }
    },
    "segments": {
        "maxTime": "2015-09-13T08:00:00.000+08:00",
        "size": 5537818,
        "minTime": "2015-09-12T08:00:00.000+08:00",
        "count": 1
    }
}

/druid/coordinator/v1/datasources/{dataSourceName}/segments/{segmentId}

返回指定DataSource和指定segment的資訊

Rules

/druid/coordinator/v1/rules/{dataSourceName}

返回指定DataSource的規則

DELETE

/druid/coordinator/v1/datasources/{dataSourceName}

禁用datasource

/druid/coordinator/v1/datasources/{dataSourceName}/intervals/{interval}

@Deprecated.
/druid/coordinator/v1/datasources/{dataSourceName}?kill=true&interval={myISO8601Interval}

指定的間隔和DataSource執行kill

/druid/coordinator/v1/datasources/{dataSourceName}/segments/{segmentId}

禁用segment

相關文章