Sentry 監控 - Snuba 資料中臺本地開發環境配置實戰

為少發表於2021-12-02
開發環境

系列

克隆倉庫

分別克隆 getsentry/sentry 與 getsentry/snuba：

git clone https://github.com/getsentry/sentry.git
git clone https://github.com/getsentry/snuba.git

安裝系統依賴(以 Mac 為例)

Xcode CLI tools

xcode-select --install

Brewfile

進入 sentry 資料夾，你會看到一個 Brewfile 檔案：

cd sentry

Brewfile

# required to run devservices
cask 'docker'

brew 'pyenv'

# required for pyenv's python-build
brew 'openssl'
brew 'readline'

# required for yarn test -u
brew 'watchman'

# required to build some of sentry's dependencies
brew 'pkgconfig'
brew 'libxslt'
brew 'libxmlsec1'
brew 'geoip'

# Currently needed because on Big Sur there's no wheel for it
brew 'librdkafka'

# direnv isn't defined here, because we have it configured to check for a bootstrapped environment.
# If it's installed in the early steps of the setup process, it just leads to confusion.
# brew 'direnv'

tap 'homebrew/cask'

# required for acceptance testing
cask 'chromedriver'

如果你本地已經安裝了 Docker Desktop 並且已經啟動，可以把 cask 'docker' 註釋掉。

接下來，執行：

brew bundle --verbose

如果你之前本地沒有 Docker Desktop，則還需要手動啟動一下它：

open -g -a Docker.app

構建工具鏈

Sentry 依賴於 Python Wheels（包含二進位制擴充套件模組的包），官方為以下平臺分發：

  • Linux 相容 PEP-513 (manylinux1)
  • macOS 10.15 或更高版本

如果您的開發機器沒有執行上述系統之一，則需要安裝 Rust 工具鏈。 按照 https://www.rust-lang.org/tools/install 上的說明安裝編譯器和相關工具。安裝後，Sentry 安裝程式將自動使用 Rust 構建所有二進位制模組，無需額外配置。

官方通常會跟蹤最新的穩定 Rust 版本，該版本每六週更新一次。 因此，請確保通過偶爾執行來使您的 Rust 工具鏈保持最新：

rustup update stable

Python

Sentry 使用 pyenv 來安裝和管理 Python 版本。 它是在您執行 brew bundle 時安裝的。

要安裝所需版本的 Python，您需要執行以下命令。 這將需要一段時間，因為您的計算機實際上正在編譯 Python！

make setup-pyenv

這裡假設你是 Zsh 使用者。

如果您鍵入 which python，您應該看到類似 $HOME/.pyenv/shims/python 而不是 /usr/bin/python 的內容。這是因為以下內容已新增到您的啟動指令碼中：

cat ~/.zprofile，你會看到如下內容：

# MacPorts Installer addition on 2021-10-20_at_11:48:22: adding an appropriate PATH variable for use with MacPorts.
export PATH="/opt/local/bin:/opt/local/sbin:$PATH"
# Finished adapting your PATH environment variable for use with MacPorts.

# It is assumed that pyenv is installed via Brew, so this is all we need to do.
eval "$(pyenv init --path)"

虛擬環境

您現在已準備好建立 Python 虛擬環境。執行：

python -m venv .venv

並啟用虛擬環境：

source .venv/bin/activate

如果一切正常，執行 which python 現在應該會導致類似 /Users/you/sentry/.venv/bin/python 的結果。

Snuba 配置實戰

啟動 Snuba 相關依賴項容器

cd ../sentry
git checkout master
git pull

source .venv/bin/activate
sentry devservices up --exclude=snuba

# 11:17:59 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
# 11:18:01 [INFO] sentry.plugins.github: apps-not-configured
# > Pulling image 'postgres:9.6-alpine'
# > Pulling image 'yandex/clickhouse-server:20.3.9.70'
# > Not starting container 'sentry_relay' because it should be started on-demand with devserver.
# > Creating 'sentry_redis' volume
# > Creating 'sentry_zookeeper_6' volume
# > Creating 'sentry_kafka_6' volume
# > Creating container 'sentry_redis'
# > Creating container 'sentry_zookeeper'
# > Creating container 'sentry_kafka'
# > Starting container 'sentry_redis' (listening: ('127.0.0.1', 6379))
# > Starting container 'sentry_kafka' (listening: ('127.0.0.1', 9092))
# > Starting container 'sentry_zookeeper'
# > Creating 'sentry_clickhouse' volume
# > Creating container 'sentry_clickhouse'
# > Creating 'sentry_postgres' volume
# > Creating 'sentry_wal2json' volume
# > Starting container 'sentry_clickhouse' (listening: ('127.0.0.1', 9000), ('127.0.0.1', 9009), ('127.0.0.1', 8123))
# > Creating container 'sentry_postgres'
# > Starting container 'sentry_postgres' (listening: ('127.0.0.1', 5432))

這將在 master 上獲取最新版本的 Sentry，並調出所有 snuba 的依賴項。

Snuba 主要依賴 clickhouse，zookeeper，kafka，redis 相關容器。

docker ps 檢視一下：

1149a6f6ff23   postgres:9.6-alpine                  "docker-entrypoint.s…"   3 minutes ago   Up 3 minutes   127.0.0.1:5432->5432/tcp                                                       sentry_postgres
a7f3af7d52bb   yandex/clickhouse-server:20.3.9.70   "/entrypoint.sh"         3 minutes ago   Up 3 minutes   127.0.0.1:8123->8123/tcp, 127.0.0.1:9000->9000/tcp, 127.0.0.1:9009->9009/tcp   sentry_clickhouse
68913ee15c43   confluentinc/cp-zookeeper:6.2.0      "/etc/confluent/dock…"   3 minutes ago   Up 3 minutes   2181/tcp, 2888/tcp, 3888/tcp                                                   sentry_zookeeper
5a248eb26ed3   confluentinc/cp-kafka:6.2.0          "/etc/confluent/dock…"   3 minutes ago   Up 3 minutes   127.0.0.1:9092->9092/tcp                                                       sentry_kafka
0573aff7b5af   redis:5.0-alpine                     "docker-entrypoint.s…"   3 minutes ago   Up 3 minutes   127.0.0.1:6379->6379/tcp                                                       sentry_redis

設定 Snuba 虛擬環境

cd snuba
make pyenv-setup
python -m venv .venv
source .venv/bin/activate
pip install --upgrade pip==21.1.3
make develop

檢視遷移列表

snuba migrations list
#  system
#  [ ]  0001_migrations
#
#  events
#  [ ]  0001_events_initial
#  [ ]  0002_events_onpremise_compatibility
#  [ ]  0003_errors
#  [ ]  0004_errors_onpremise_compatibility
#  [ ]  0005_events_tags_hash_map (blocking)
#  [ ]  0006_errors_tags_hash_map (blocking)
#  [ ]  0007_groupedmessages
#  [ ]  0008_groupassignees
#  [ ]  0009_errors_add_http_fields
#  [ ]  0010_groupedmessages_onpremise_compatibility (blocking)
#  [ ]  0011_rebuild_errors
#  [ ]  0012_errors_make_level_nullable
#  [ ]  0013_errors_add_hierarchical_hashes
#  [ ]  0014_backfill_errors (blocking)
#  [ ]  0015_truncate_events
#
#  transactions
#  [ ]  0001_transactions
#  [ ]  0002_transactions_onpremise_fix_orderby_and_partitionby (blocking)
#  [ ]  0003_transactions_onpremise_fix_columns (blocking)
#  [ ]  0004_transactions_add_tags_hash_map (blocking)
#  [ ]  0005_transactions_add_measurements
#  [ ]  0006_transactions_add_http_fields
#  [ ]  0007_transactions_add_discover_cols
#  [ ]  0008_transactions_add_timestamp_index
#  [ ]  0009_transactions_fix_title_and_message
#  [ ]  0010_transactions_nullable_trace_id
#  [ ]  0011_transactions_add_span_op_breakdowns
#  [ ]  0012_transactions_add_spans
#
#  discover
#  [ ]  0001_discover_merge_table
#  [ ]  0002_discover_add_deleted_tags_hash_map
#  [ ]  0003_discover_fix_user_column
#  [ ]  0004_discover_fix_title_and_message
#  [ ]  0005_discover_fix_transaction_name
#  [ ]  0006_discover_add_trace_id
#  [ ]  0007_discover_add_span_id
#
#  outcomes
#  [ ]  0001_outcomes
#  [ ]  0002_outcomes_remove_size_and_bytes
#  [ ]  0003_outcomes_add_category_and_quantity
#  [ ]  0004_outcomes_matview_additions (blocking)
#
#  metrics
#  [ ]  0001_metrics_buckets
#  [ ]  0002_metrics_sets
#  [ ]  0003_counters_to_buckets
#  [ ]  0004_metrics_counters
#  [ ]  0005_metrics_distributions_buckets
#  [ ]  0006_metrics_distributions
#  [ ]  0007_metrics_sets_granularity_10
#  [ ]  0008_metrics_counters_granularity_10
#  [ ]  0009_metrics_distributions_granularity_10
#  [ ]  0010_metrics_sets_granularity_1h
#  [ ]  0011_metrics_counters_granularity_1h
#  [ ]  0012_metrics_distributions_granularity_1h
#  [ ]  0013_metrics_sets_granularity_1d
#  [ ]  0014_metrics_counters_granularity_1d
#  [ ]  0015_metrics_distributions_granularity_1d
#
#  sessions
#  [ ]  0001_sessions
#  [ ]  0002_sessions_aggregates
#  [ ]  0003_sessions_matview

執行遷移

snuba migrations migrate --force
#  ......
#  2021-12-01 19:45:57,557 Running migration: 0014_metrics_counters_granularity_1d
#  2021-12-01 19:45:57,575 Finished: 0014_metrics_counters_granularity_1d
#  2021-12-01 19:45:57,589 Running migration: 0015_metrics_distributions_granularity_1d
#  2021-12-01 19:45:57,610 Finished: 0015_metrics_distributions_granularity_1d
#  2021-12-01 19:45:57,623 Running migration: 0001_sessions
#  2021-12-01 19:45:57,656 Finished: 0001_sessions
#  2021-12-01 19:45:57,669 Running migration: 0002_sessions_aggregates
#  2021-12-01 19:45:57,770 Finished: 0002_sessions_aggregates
#  2021-12-01 19:45:57,792 Running migration: 0003_sessions_matview
#  2021-12-01 19:45:57,849 Finished: 0003_sessions_matview
#  Finished running migrations

檢查遷移

進入 Clickhouse 容器：

docker exec -it sentry_clickhouse clickhouse-client
# 執行如下 `sql` 語句：
select count() from sentry_local

# ClickHouse client version 20.3.9.70 (official build).
# Connecting to localhost:9000 as user default.
# Connected to ClickHouse server version 20.3.9 revision 54433.

# a7f3af7d52bb :) select count() from sentry_local

# SELECT count()
# FROM sentry_local

# ┌─count()─┐
# │       0 │
# └─────────┘

# 1 rows in set. Elapsed: 0.008 sec. 

# a7f3af7d52bb :)

檢視相關實體資料集

snuba entities list

# Declared Entities:
# discover
# events
# groups
# groupassignee
# groupedmessage
# metrics_sets
# metrics_counters
# metrics_distributions
# outcomes
# outcomes_raw
# sessions
# org_sessions
# spans
# transactions
# discover_transactions
# discover_events

啟動開發伺服器

此命令將啟動 api 和所有 Snuba 消費者以從 Kafka 攝取資料：

snuba devserver

轉到 http://localhost:1218/events/snql，你將會看到一個簡易的查詢 UI。

相關文章