openresty使用火焰圖排查效能問題

趙安家發表於2017-09-29
REST

本文主要是講解如何在ubuntu安裝最新Systemtap.以及繪製火焰圖

安裝除錯映象


# 匯入 GPG key
# 16.04 and higher
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C8CAB6595FDFF622

#older distributions
#sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys ECDCAD72428D7C01 

# 設定源
codename=$(lsb_release -c | awk  '{print $2}')
sudo tee /etc/apt/sources.list.d/ddebs.list << EOF
deb http://ddebs.ubuntu.com/ ${codename}      main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-security main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-updates  main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-proposed main restricted universe multiverse
EOF

# 更新
sudo apt-get update

# 安裝除錯映象
sudo apt-get install -y linux-image-$(uname -r)-dbgsym複製程式碼

安裝最新版 systemtap

$ sudo apt-get install -y build-essential zlib1g-dev elfutils libdw-dev gettext

# https://sourceware.org/elfutils/ftp/?C=M;O=D
$ wget https://sourceware.org/elfutils/ftp/0.170/elfutils-0.170.tar.bz2
$ tar xf elfutils-0.170.tar.bz2

# https://sourceware.org/systemtap/ftp/releases/?C=M;O=D
$ wget https://sourceware.org/systemtap/ftp/releases/systemtap-3.1.tar.gz
$ tar zxf systemtap-3.1.tar.gz

$ cd systemtap-3.1

$ ./configure --prefix=/opt/stap --disable-docs \
    --disable-publican --disable-refdocs CFLAGS="-g -O2" \
    --with-elfutils=../elfutils-0.170

$ make -j$(getconf _NPROCESSORS_ONLN) && sudo make install

# export STAP_HOME=/opt/stap/
# export PATH=$STAP_HOME:$PATH

# stap -V

Systemtap translator/driver (version 3.1/0.170, non-git sources)
Copyright (C) 2005-2017 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
tested kernel versions: 2.6.18 ... 4.10-rc8
enabled features: PYTHON2 PYTHON3 LIBXML2 NLS READLINE複製程式碼

測試是否生效

# stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}'
Pass 1: parsed user script and 465 library scripts using 77388virt/46648res/5256shr/41840data kb, in 80usr/30sys/333real ms.
Pass 2: analyzed script: 1 probe, 1 function, 7 embeds, 0 globals using 260440virt/231204res/6736shr/224892data kb, in 1680usr/350sys/7050real ms.
Pass 3: translated to C into "/tmp/stap8Lyxq5/stap_e1c4934460a3e749f6deefe95dd50015_2729_src.c" using 260440virt/231404res/6936shr/224892data kb, in 10usr/0sys/5real ms.
Pass 4: compiled C into "stap_e1c4934460a3e749f6deefe95dd50015_2729.ko" in 5260usr/420sys/7185real ms.
Pass 5: starting run.
read performed
Pass 5: run completed in 0usr/20sys/486real ms.複製程式碼

繪製火焰圖

下載各工具包

# git clone https://github.com/openresty/stapxx.git --depth=1 /opt/stapxx
# export STAP_PLUS_HOME=/opt/stapxx
# export PATH=$STAP_PLUS_HOME:$STAP_PLUS_HOME/samples:$PATH

# stap++ -e 'probe begin { println("hello") exit() }'

hello


# git clone https://github.com/openresty/openresty-systemtap-toolkit.git --depth=1 /opt/openresty-systemtap-toolkit

# git clone https://github.com/brendangregg/FlameGraph.git --depth=1 /opt/FlameGraph複製程式碼

繪製火焰圖

# /opt/stapxx/samples/lj-lua-stacks.sxx --arg time=120 --skip-badvars -x `ps --no-headers -fC nginx|awk '/worker/  {print$2}'| shuf | head -n 1` > /tmp/tmp.bt （-x 是要抓的程式的 pid， 探測結果輸出到 tmp.bt）
# /opt/openresty-systemtap-toolkit/fix-lua-bt tmp.bt > /tmp/flame.bt  (處理 lj-lua-stacks.sxx 的輸出，使其可讀性更佳)
# /opt/FlameGraph/stackcollapse-stap.pl /tmp/flame.bt > /tmp/flame.cbt
# /opt/FlameGraph/flamegraph.pl /tmp/flame.cbt > /tmp/flame.svg複製程式碼

為了突出效果，建議在執行stap++的時候，使用壓測工具，以便採集足夠的樣本

# ab -n 10000 -c 100 -k http://localhost/複製程式碼

用瀏覽器開啟 /tmp/flame.svg 儘量用 chrome firefox別用國產亂七八糟瀏覽器.

openresty/stapxx

## 使用 stap++ --args xx.sxx檢視具體引數

# stap++ --args /opt/stapxx/samples/lj-lua-stacks.sxx
    --arg depth=VALUE (default: 30)
    --arg detailed=VALUE (default: 0)
    --arg limit=VALUE (default: 1000)
    --arg min=VALUE (default: 2)
    --arg nointerp=VALUE (default: )
    --arg nojit=VALUE (default: )
    --arg probe=VALUE (default: timer.profile)
    --arg time=VALUE (default: )複製程式碼

具體指令碼用法，參見 openresty/stapxx#samples

openresty/openresty-systemtap-toolkit

這一系列指令碼很有用，比如可以用來看共享記憶體大小，使用情況，記憶體洩露情況，哪裡洩露的，不過部分指令碼需要在編譯的時候，開啟除錯或者增加依賴。具體參見readme.

如果要使用ngx-leaked-pools需要用到dtrace

$ apt install systemtap-sdt-dev -y
$ ./configure --prefix=/etc/openresty \
  --with-dtrace-probes複製程式碼

如果要用到ngx-pcrejit需要在編譯openresty時增加--with-pcre-opt=-g

重新編譯並make && make install 後會將原有的二進位制檔案重新命名為${openresty_home}/nginx/sbin/nginx.old，並建立一個${openresty_home}/nginx/sbin/nginx(新版)

$ kill -USR2 `cat /var/run/nginx.pid`複製程式碼

通過ps -fC nginx或者ps -fC openresty檢視新版本是否成功啟動

如果成功啟動，此時新舊版本同時接受請求

通過

$ kill -QUIT `cat /var/run/nginx.pid.oldbin`複製程式碼

平滑殺掉舊版

更多資料請自行谷歌、百度。或者參閱 下面的參考連線

參考連線

部落格 anjia.ml/2017/09/12/…
掘金 juejin.im/post/59ce27…
簡書 www.jianshu.com/p/008fde883…

相關文章