一.環境
1.1 jello@jello:~$ uname -a
Linux jello 4.4.0-98-generic #121-Ubuntu SMP Tue Oct 10 14:24:03 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
1.2 jello@jello-Inspiron-N4050:~$ lsb_release -a
Distributor ID: Ubuntu
Description: Ubuntu 16.04.3 LTS
Release: 16.04
Codename: xenial
二.準備工作.
2.1安裝systemtap (火焰圖依賴於此工具)
sudo apt-get install systemtap
2.2 查詢核心對應的debug包
jello@jello$ uname -r
4.4.0-98-generic
那麼接下來就是去http://ddebs.ubuntu.com/pool/main/l/linux/下載對應的debug包,找4.4.0-98-generic一致的
2.3 下載對應的debug包
wget http://ddebs.ubuntu.com/pool/main/l/linux/linux-image-4.4.0-98-generic-dbgsym_4.4.0-98.121_amd64.ddeb (這是筆者自己找到的對應下載路徑)
2.4 安裝debug包
sudo dpkg -i linux-image-4.4.0-98-generic-dbgsym_4.4.0-98.121_amd64.ddeb
2.5 安裝nginx
sudo apt-get install nginx
此時在瀏覽器中輸入localhost即可出現以下資訊表明安裝ok
Welcome to nginx!
If you see this page, the nginx web server is successfully installed and working. Further configuration is required.
For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.
Thank you for using nginx.
2.5 編寫systemtap指令碼nginx.systemtap,內容如下:
global s;
global quit=0;
probe timer.profile {
if (pid() == target()){
if (quit) {
foreach (i in s-) {
print_ustack(i);
printf("\t%d\n",@count(s[i]));
}
exit();
}
else {
s[ubacktrace()] <<< 1;
}
}
}
probe timer.s(20){
quit = 1
}
2.6 使用systemtap
sudo stap --ldd -d /usr/sbin/nginx --all-modules -D MAXMAPENTRIES=256 -D MAXACTION=20000 -D MAXTRACE=100 -D MAXSTRINGLEN=4096 -D MAXBACKTRACE=100 -x 2082 nginx.systemtap --vp 0001 > nginx.out
各引數解析:
--ldd，新增通過ldd解析出來的所有共享庫符號表資訊以便為probe到的使用者空間二進位制提供資訊或者以-d選項列出來，注意：這會使得probe模組相當的大
-d /usr/sbin/nginx，為給定的模組(這裡是nginx)新增符號表資訊到核心物件模組,這可能使能這些模組或者程式的象徵性traceback，即使他們沒有顯式probe到他們裡面
--all-modules,相當於為所有當前被載入的模組指定'-dkernel'和‘-d'選項
-D MAXMAPENTRIES=256 ,新增給定的c預處理直接到模組makefile中,這些能被用來限制以下描述的引數,MAXMAPENTRIES=256,設定預設全域性陣列的最大預設長度為256，通過了解，是用來指定堆疊大小的
-D MAXACTION=20000 設定單個probe hit包含執行的最大語句數目
-D MAXTRACE=100
-D MAXSTRINGLEN=4096 設定字串最大長度
-D MAXBACKTRACE=100 設定棧幀的最大數目
-x 2082 設定target()的pid為2082
--vp 0001
筆者執行之後出現以下錯誤資訊：
jello@jello:~$ sudo stap --ld -d /usr/sbin/nginx --all-modules -D MAXMAPENTRIES=256 -D MAXACTION=20000 -D MAXTRACE=100 -D MAXSTRINGLEN=4096 -D MAXBACKTRACE=100 -x 2082 nginx.systemtap --vp 0001 > nginx.out
In file included from /usr/share/systemtap/runtime/transport/control.c:14:0,
from /usr/share/systemtap/runtime/transport/transport.c:76,
from /usr/share/systemtap/runtime/linux/print.c:17,
from /usr/share/systemtap/runtime/print.c:17,
from /usr/share/systemtap/runtime/runtime_context.h:22,
from /tmp/stapiPADvo/stap_20923465b7e2011fc7e903a669485b1f_9417_src.c:221:
/usr/share/systemtap/runtime/transport/symbols.c: In function ‘_stp_module_update_self’:
/usr/share/systemtap/runtime/transport/symbols.c:243:44: error: ‘struct module’ has no member named ‘symtab’
if (attr->address == (unsigned long) mod->symtab)
^
/usr/share/systemtap/runtime/transport/symbols.c:245:9: error: ‘struct module’ has no member named ‘num_symtab’
mod->num_symtab * sizeof(mod->symtab[0]);
^
/usr/share/systemtap/runtime/transport/symbols.c:245:34: error: ‘struct module’ has no member named ‘symtab’
mod->num_symtab * sizeof(mod->symtab[0]);
^
make[1]: *** [/tmp/stapiPADvo/stap_20923465b7e2011fc7e903a669485b1f_9417_src.o] Error 1
make: *** [_module_/tmp/stapiPADvo] Error 2
WARNING: kbuild exited with status: 2
Pass 4: compiled C into "stap_20923465b7e2011fc7e903a669485b1f_9417.ko" in 9530usr/380sys/12191real ms.
Pass 4: compilation failed. [man error::pass4]
Tip: /usr/share/doc/systemtap/README.Debian should help you get started
從錯誤資訊來看是某個mod的某個欄位不存在，所以斷定是核心版本過高，但是systemtap版本過低導致的，因此升級systemtap版本
2.6.1 升級systemtap版本
2.6.1.1獲取最新systemtap程式碼
方案一.git clone git://sourceware.org/git/systemtap.git (下載速度比較慢,雖然配置FQ了，仍然慢，推薦方案二)
方案二.tsocks wget https://sourceware.org/systemtap/ftp/releases/systemtap-3.2.tar.gz (tsocks使能wget通過FQ下載原始碼，需要配置vpn進行FQ)
以上方案均需FQ，否則下載會非常慢!
2.6.1.2解壓systemtap原始碼
tar xvf systemtap-3.2.tar.gz
2.6.1.3 切換到解壓後的systemtap目錄
cd systemtap-3.2
2.6.1.4 進行編譯
2.6.1.4.1 根據README得知：
需要安裝帶有libdwfl庫的elfutils (版本號需要大於或等於0.151)才能正常配合systemtap;
2.6.1.4.2 安裝elfutils
sudo apt-get install elfutils
2.6.1.4.3 驗證elfutils的版本資訊
eu-objdump -V
2.6.1.4.4 安裝必不可少的編譯工具
sudo apt-get build-dep systemtap
2.6.1.4.5 建立編譯目錄
mkdir build
2.6.1.4.6 切換到build目錄
cd build
2.6.1.4.7 進行編譯前的配置
../configure
2.6.1.4.8 進行正式編譯
make all
2.6.1.4.8.1 編譯過程中出現以下問題
../bpf-translate.cxx:37:29: fatal error: elfutils/libebl.h: 沒有那個檔案或目錄
2.6.1.4.8.2 檢視bpf-translate.cxx
#if _ELFUTILS_PREREQ (0, 167)
#include <elfutils/libdwelf.h>
typedef Dwelf_Strent Stap_Strent;
typedef Dwelf_Strtab Stap_Strtab;
#define stap_strtab_init dwelf_strtab_init
#define stap_strtab_add(X,Y) dwelf_strtab_add(X,Y)
#define stap_strtab_free dwelf_strtab_free
#define stap_strtab_finalize dwelf_strtab_finalize
#define stap_strent_offset dwelf_strent_off
#else
#include <elfutils/libebl.h>
typedef Ebl_Strent Stap_Strent;
typedef Ebl_Strtab Stap_Strtab;
#define stap_strtab_init ebl_strtabinit
#define stap_strtab_add(X,Y) ebl_strtabadd(X,Y,0)
#define stap_strtab_free ebl_strtabfree
#define stap_strtab_finalize ebl_strtabfinalize
#define stap_strent_offset ebl_strtaboffset
#endif
2.6.1.4.8.2 修改以上內容為:
#if 1
#include <elfutils/libdwelf.h>
typedef Dwelf_Strent Stap_Strent;
typedef Dwelf_Strtab Stap_Strtab;
#define stap_strtab_init dwelf_strtab_init
#define stap_strtab_add(X,Y) dwelf_strtab_add(X,Y)
#define stap_strtab_free dwelf_strtab_free
#define stap_strtab_finalize dwelf_strtab_finalize
#define stap_strent_offset dwelf_strent_off
#else
#include <elfutils/libebl.h>
typedef Ebl_Strent Stap_Strent;
typedef Ebl_Strtab Stap_Strtab;
#define stap_strtab_init ebl_strtabinit
#define stap_strtab_add(X,Y) ebl_strtabadd(X,Y,0)
#define stap_strtab_free ebl_strtabfree
#define stap_strtab_finalize ebl_strtabfinalize
#define stap_strent_offset ebl_strtaboffset
#endif
2.6.1.4.8.3 通過以上修改出現以下錯誤資訊:
make all-recursive
make[1]: Entering directory '/home/jello/data/development/systemtap-3.2/build'
Making all in .
make[2]: Entering directory '/home/jello/data/development/systemtap-3.2/build'
CXX stap-bpf-translate.o
../bpf-translate.cxx:29:9: error: ‘Dwelf_Strent’ does not name a type
typedef Dwelf_Strent Stap_Strent;
^
../bpf-translate.cxx:30:9: error: ‘Dwelf_Strtab’ does not name a type
typedef Dwelf_Strtab Stap_Strtab;
^
../bpf-translate.cxx:1773:3: error: ‘Stap_Strent’ does not name a type
Stap_Strent *name_ent;
^
../bpf-translate.cxx: In constructor ‘bpf::BPF_Section::BPF_Section(const string&)’:
../bpf-translate.cxx:1782:22: error: class ‘bpf::BPF_Section’ does not have any field named ‘name_ent’
: scn(0), name(n), name_ent(0), data(0), free_data(false)
^
../bpf-translate.cxx: At global scope:
../bpf-translate.cxx:1794:3: error: ‘Stap_Strent’ does not name a type
Stap_Strent *name_ent;
^
../bpf-translate.cxx: In constructor ‘bpf::BPF_Symbol::BPF_Symbol(const string&, bpf::BPF_Section*, long int)’:
../bpf-translate.cxx:1801:14: error: class ‘bpf::BPF_Symbol’ does not have any field named ‘name_ent’
: name(n), name_ent(0)
^
../bpf-translate.cxx: At global scope:
../bpf-translate.cxx:1812:3: error: ‘Stap_Strtab’ does not name a type
Stap_Strtab *str_tab;
^
../bpf-translate.cxx: In constructor ‘bpf::BPF_Output::BPF_Output(int)’:
../bpf-translate.cxx:1827:5: error: class ‘bpf::BPF_Output’ does not have any field named ‘str_tab’
str_tab(stap_strtab_init(true))
^
../bpf-translate.cxx:1827:34: error: ‘dwelf_strtab_init’ was not declared in this scope
str_tab(stap_strtab_init(true))
^
../bpf-translate.cxx: In destructor ‘bpf::BPF_Output::~BPF_Output()’:
../bpf-translate.cxx:1835:20: error: ‘str_tab’ was not declared in this scope
stap_strtab_free(str_tab);
^
../bpf-translate.cxx:1835:27: error: ‘dwelf_strtab_free’ was not declared in this scope
stap_strtab_free(str_tab);
^
../bpf-translate.cxx: In member function ‘bpf::BPF_Section* bpf::BPF_Output::new_scn(const string&)’:
../bpf-translate.cxx:1854:6: error: ‘struct bpf::BPF_Section’ has no member named ‘name_ent’
n->name_ent = stap_strtab_add(str_tab, n->name.c_str());
^
../bpf-translate.cxx:1854:33: error: ‘str_tab’ was not declared in this scope
n->name_ent = stap_strtab_add(str_tab, n->name.c_str());
^
../bpf-translate.cxx:32:48: note: in definition of macro ‘stap_strtab_add’
#define stap_strtab_add(X,Y) dwelf_strtab_add(X,Y)
^
../bpf-translate.cxx:32:51: error: ‘dwelf_strtab_add’ was not declared in this scope
#define stap_strtab_add(X,Y) dwelf_strtab_add(X,Y)
^
../bpf-translate.cxx:1854:17: note: in expansion of macro ‘stap_strtab_add’
n->name_ent = stap_strtab_add(str_tab, n->name.c_str());
^
../bpf-translate.cxx: In member function ‘bpf::BPF_Symbol* bpf::BPF_Output::new_sym(const string&, bpf::BPF_Section*, long int)’:
../bpf-translate.cxx:1864:6: error: ‘struct bpf::BPF_Symbol’ has no member named ‘name_ent’
s->name_ent = stap_strtab_add(str_tab, s->name.c_str());
^
../bpf-translate.cxx:1864:33: error: ‘str_tab’ was not declared in this scope
s->name_ent = stap_strtab_add(str_tab, s->name.c_str());
^
../bpf-translate.cxx:32:48: note: in definition of macro ‘stap_strtab_add’
#define stap_strtab_add(X,Y) dwelf_strtab_add(X,Y)
^
../bpf-translate.cxx:32:51: error: ‘dwelf_strtab_add’ was not declared in this scope
#define stap_strtab_add(X,Y) dwelf_strtab_add(X,Y)
^
../bpf-translate.cxx:1864:17: note: in expansion of macro ‘stap_strtab_add’
s->name_ent = stap_strtab_add(str_tab, s->name.c_str());
^
../bpf-translate.cxx: In function ‘void bpf::output_symbols_sections(bpf::BPF_Output&)’:
../bpf-translate.cxx:2183:31: error: ‘struct bpf::BPF_Output’ has no member named ‘str_tab’
stap_strtab_finalize(eo.str_tab, str->data);
^
../bpf-translate.cxx:2183:49: error: ‘dwelf_strtab_finalize’ was not declared in this scope
stap_strtab_finalize(eo.str_tab, str->data);
^
../bpf-translate.cxx:2190:39: error: ‘struct bpf::BPF_Symbol’ has no member named ‘name_ent’
b->st_name = stap_strent_offset(s->name_ent);
^
../bpf-translate.cxx:2190:47: error: ‘dwelf_strent_off’ was not declared in this scope
b->st_name = stap_strent_offset(s->name_ent);
^
../bpf-translate.cxx:2196:29: error: ‘struct bpf::BPF_Output’ has no member named ‘str_tab’
stap_strtab_finalize(eo.str_tab, str->data);
^
../bpf-translate.cxx:2196:47: error: ‘dwelf_strtab_finalize’ was not declared in this scope
stap_strtab_finalize(eo.str_tab, str->data);
^
../bpf-translate.cxx:2203:48: error: ‘struct bpf::BPF_Section’ has no member named ‘name_ent’
s->shdr->sh_name = stap_strent_offset(s->name_ent);
^
../bpf-translate.cxx:2203:56: error: ‘dwelf_strent_off’ was not declared in this scope
s->shdr->sh_name = stap_strent_offset(s->name_ent);
從以上資訊可知：以上修改反而會引入更多的問題,因此看一下程式碼
2.6.1.4.8.4 將sytemtap的程式碼還原
2.6.1.4.8.5 清掉build目錄
rm -rf build
2.6.1.4.8.6 重新建立build目錄
mkdir build
2.6.1.4.8.7 切換到build目錄
cd build
2.6.1.4.8.8 查詢elfutils目錄
sudo find / |grep elfutils
筆者找到elfutils對應的標頭檔案路徑是/usr/include (預設使用apt-get install安裝的軟體對應的標頭檔案在/usr/include目錄下)
2.6.1.4.8.9 指定elfutils的標頭檔案路徑進行配置並且指定安裝目錄為/opt/
rm * -rf (需要清一下之前在build目錄下生成的檔案)
../configure --with-elfutils=/usr/include --prefix=/opt/
出現以下錯誤資訊:
checking for libvirt... no
checking for libxml2... yes
configure: WARNING: will not build systemtap virt support, cannot find libvirt headers
checking for python-config... /usr/bin/python-config
checking Python.h usability... yes
checking Python.h presence... yes
checking for Python.h... yes
checking for python3-config... /usr/bin/python3-config
checking Python.h usability... yes
checking Python.h presence... yes
checking for Python.h... yes
checking for jsonc... no
checking for ncurses... yes
checking for assembler .section "?" flags support... yes
checking linux/bpf.h usability... yes
checking linux/bpf.h presence... yes
checking for linux/bpf.h... yes
configure: error: No /usr/include/configure, forgot to run autoreconf -i?
從此資訊中發現缺少configure，應該與elfutils有關,那麼下載與當前elfutils對應的版本原始碼
2.6.1.4.8.10 檢視當前elfutils版本的方法:
eu-objdump -V
objdump (elfutils) 0.165
Copyright (C) 2012 Red Hat, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Ulrich Drepper.
版本為0.165
2.6.1.4.8.11 獲取elfutils的原始碼
筆者找到的0.165下載地址為:https://sourceware.org/elfutils/ftp/0.165/elfutils-0.165.tar.bz2
2.6.1.4.8.12 下載elfutils原始碼
tsocks wget https://sourceware.org/elfutils/ftp/0.165/elfutils-0.165.tar.bz2 (FQ下載才快)
2.6.1.4.8.13解壓elfutils
tar xvf elfutils-0.165.tar.bz2
2.6.1.4.8.14 指定elfutils原始碼的路徑(筆者的elfutils原始碼路徑在/home/jello/elfutils-0.165下)並指定systemtap的安裝路徑
../configure --with-elfutils=/home/jello/elfutils-0.165 --prefix=/opt/
2.6.1.4.8.15 開始編譯
make
2.6.1.4.8.16 編譯又出現錯誤
systemtap-3.2/build/python/../includes/sys -fPIC -I/usr/include/python2.7 -c HelperSDT/_HelperSDT.c -o /home/jello/systemtap-3.2/build/python/py2build/temp.linux-x86_64-2.7/HelperSDT/_HelperSDT.o
HelperSDT/_HelperSDT.c:10:21: fatal error: sys/sdt.h: 沒有那個檔案或目錄
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
Makefile:621: recipe for target 'all-local' failed
make[2]: *** [all-local] Error 1
make[2]: Leaving directory '/home/jello/systemtap-3.2/build/python'
Makefile:2012: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '/home/jello/systemtap-3.2/build'
Makefile:717: recipe for target 'all' failed
make: *** [all] Error 2
從以上資訊可知:沒有標頭檔案sys/sdt.h,那麼就查詢sys/sdt.h的路徑
2.6.1.4.8.17 查詢sys/sdt.h的路徑
sudo find /|grep "sys/sdt.h"
發現sys/sdt.h在systemtap-3.2的includes目錄下
2.6.1.4.8.18 修改build/python目錄下的Makefile
修改前的內容:
AM_CPPFLAGS = -I$(srcdir)/../includes \
-I$(abs_builddir)/../includes/sys修改後的內容:AM_CPPFLAGS = -I$(srcdir)/../includes \
-I$(abs_builddir)/../includes/sys \ -I$(abs_srcdir)/../includes
2.6.1.4.8.19 再次編譯又出現以下錯誤(cd ../../python; CFLAGS="-I../../python/../includes -I/home/jello/systemtap-3.2/build/python/../includes/sys -I/home/jello/systemtap-3.2/build/../python/../includes" /usr/bin/python3 setup.py build \
--build-base /home/jello/systemtap-3.2/build/python/py3build \
--verbose)
Traceback (most recent call last):
File "setup.py", line 9, in <module>
from setuptools import setup, Extension
ImportError: No module named 'setuptools'
Makefile:622: recipe for target 'all-local' failed
make[2]: *** [all-local] Error 1
make[2]: Leaving directory '/home/jello/data/development/systemtap-3.2/build/python'
Makefile:2012: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '/home/jello/data/development/systemtap-3.2/build'
Makefile:717: recipe for target 'all' failed
make: *** [all] Error 2
從以上資訊可知:python缺少setuptools模組,所以安裝setuptools模組,並且使用python3進行安裝
2.6.1.4.8.20 安裝setuptools模組
2.6.1.4.8.20.1 獲取模組
wget https://bootstrap.pypa.io/ez_setup.py
2.6.1.4.8.20.2 安裝模組
sudo python3 ez_setup.py2.6.1.4.8.21 繼續編譯
make一切ok2.6.1.4.9 安裝systemtapsudo make install2.6.1.4.10 解除安裝原來使用apt-get install安裝的systemtapsudo apt-get remove systemtap2.6.1.4.11 設定環境變數使新安裝的stap生效
vi /etc/profile往檔案最後加入一句:export "PATH=/opt/bin:$PATH"2.6.1.5 驗證systap是否安裝成功stap -VSystemtap translator/driver (version 3.2/0.165, non-git sources)
Copyright (C) 2005-2017 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
tested kernel versions: 2.6.18 ... 4.11
enabled features: AVAHI PYTHON2 PYTHON3 LIBSQLITE3 LIBXML2 NLS NSS READLINE以上資訊中發現版本為3.2/0.165，表明systemtap已經安裝成功
2.6.2 啟動stap
2.6.2.1 先切換為最高許可權
sudo su -l
2.6.2.2 啟動stap (之前寫過一個nginx.systemtap檔案，記得指定該檔案路徑，筆者將nginx.systemtap放在了當前目錄下，因此直接的指定檔案）
stap --ld -d /usr/sbin/nginx --all-modules -D MAXMAPENTRIES=256 -D MAXACTION=20000 -D MAXTRACE=100 -D MAXSTRINGLEN=4096 -D MAXBACKTRACE=100 -x 2082 nginx.systemtap --vp 0001 > nginx.out
2.7重開終端進行壓力測試(可選)
進行壓力測試:
ab -n 900000 -c 50 http://192.168.2.323/index.php /*192.168.2.323為筆者的本地ip地址，注意修改成自己的本地ip地址*/
2.8 FlameFraph
2.8.1 獲取FlameGraph
git clone https://github.com/brendangregg/FlameGraph.git
2.8.2
由於2.6.2.2一直未生成nginx.out,因此需要找一下nginx.systemtap是否有錯誤
未寫完，待續...