Linux下的一些I/O統計工具

me_lawrence發表於2015-09-21

http://blog.csdn.net/longxibendi/article/details/36004155

Linux 系統管理員，統計各類 IO 是一項必不可少的工作。其統計工具中 iostat 顯然又是最重要的一個統計手段。但是這裡 iostat 不是本文的重點，因為這個工具的使用在網路上已經有大量的教程，可以供大家參考。這裡主要是想介紹一些其他統計工具以來滿足不同的需求。

iostat

iostat 的功能異常強大，輸出項也特別多，比如下面這個例子：

<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">1</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">2</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">3</span>


Device: rrqm/s  wrqm/s  r/s     w/s    rkB/s    wkB/s    avgrq-sz avgqu-sz   await r_await w_await  svctm  %util

sda     0.00     0.50  173.50   73.50  3076.00   604.00    29.80   149.93    676.58   74.36 2098.15  4.05 100.00

其各項的含義分別是：

rrqm/s: 每秒進行 merge 的讀運算元目.即 delta(rmerge)/s
wrqm/s: 每秒進行 merge 的寫運算元目.即 delta(wmerge)/s
r/s: 每秒完成的讀 I/O 裝置次數.即 delta(rio)/s
w/s: 每秒完成的寫 I/O 裝置次數.即 delta(wio)/s
rsec/s: 每秒讀扇區數.即 delta(rsect)/s
wsec/s: 每秒寫扇區數.即 delta(wsect)/s
rkB/s: 每秒讀 K 位元組數.是 rsect/s 的一半,因為每扇區大小為 512 位元組.(需要計算)
wkB/s: 每秒寫 K 位元組數.是 wsect/s 的一半.(需要計算)
avgrq-sz: 平均每次裝置 I/O 操作的資料大小 (扇區).delta(rsect+wsect)/delta(rio+wio)
avgqu-sz: 平均 I/O 佇列長度.即 delta(aveq)/s/1000 (因為 aveq 的單位為毫秒).
await: 平均每次裝置 I/O 操作的等待時間 (毫秒).即 delta(ruse+wuse)/delta(rio+wio)
svctm: 平均每次裝置 I/O 操作的服務時間 (毫秒).即 delta(use)/delta(rio+wio)
%util: 一秒中有百分之多少的時間用於 I/O 操作,或者說一秒中有多少時間 I/O 佇列是非空的.即 delta(use)/s/1000 (因為 use 的單位為毫秒)

如果 %util 接近 100%,說明產生的 I/O 請求太多,I/O 系統已經滿負荷,該磁碟可能存在瓶頸.

idle 小於 70% IO 壓力就較大了,一般讀取速度有較多的 wait.

同時可以結合vmstat檢視檢視 b 引數(等待資源的程式數)和 wa 引數(IO 等待所佔用的 CPU 時間的百分比,高過 30%時 IO 壓力高)

另外 await 的引數也要多和 svctm 來參考。差的過高就一定有 IO 的問題.

avgrq-sz 也是個做 IO 調優時需要注意的地方,這個就是直接每次操作的資料的大小,如果次數多,但資料拿的小的話,其實 IO 也會很小.如果資料拿的大,才 IO 的資料會高.也可以通過 avgqu-sz × ( r/s or w/s ) = rsec/s or wsec/s.也就是講,讀定速度是這個來決定的.

svctm 一般要小於 await (因為同時等待的請求的等待時間被重複計算了),svctm 的大小一般和磁碟效能有關,CPU/記憶體的負荷也會對其有影響,請求過多也會間接導致 svctm 的增加.await 的大小一般取決於服務時間(svctm) 以及 I/O 佇列的長度和 I/O 請求的發出模式.如果 svctm 比較接近 await,說明 I/O 幾乎沒有等待時間；如果 await 遠大於 svctm,說明 I/O 佇列太長,應用得到的響應時間變慢,如果響應時間超過了使用者可以容許的範圍,這時可以考慮更換更快的磁碟,調整核心 elevator演算法,優化應用,或者升級 CPU.

佇列長度(avgqu-sz)也可作為衡量系統 I/O 負荷的指標,但由於 avgqu-sz 是按照單位時間的平均值,所以不能反映瞬間的 I/O 洪水.

有時間的話，我會單獨寫幾個帖子來說說iostat。

iodump

iodump 是一個統計每一個程式(執行緒)所消耗的磁碟 I/O 工具。這個一個 perl 指令碼，其原理時開啟有關 I/O 的核心記錄訊息開關，而後讀取訊息然後分析輸出。簡單使用步驟如下：

首先下載這個工具

wget http://aspersa.googlecode.com/svn/trunk/iodump

然後開啟有關 I/O 核心訊息的開關

echo 1 >/proc/sys/vm/block_dump

上述開關開啟後，核心會記錄下每一個 I/O 操作的訊息。我們只需要定時獲取並分析就好了，比如下面這樣

while true; do sleep 1; dmesg -c ; done |perl iodump

等待一段時間，然後通過ctrl+c來結束上述指令碼，你將獲得下面類似的資訊:

<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">1</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">2</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">3</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">4</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">5</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">6</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">7</span>


TASK                   PID      TOTAL       READ      WRITE      DIRTY DEVICES
postgres              5799       1919       1919          0          0 sda7
jbd2/sda7-8           1572         35          0         35          0 sda7
jbd2/sda2-8            250         32          0         32          0 sda2
flush-8:0             2229         31          0         31          0 sda2, sda7
postgres              4308          2          0          2          0 sda7
bash                  5804          1          0          1          0 sda2

上述輸出的單位為塊(block)，每塊的大小取決於建立檔案系統時指定的塊大小。比如我這個裡的 sda7 的 block 大小是 1KB。

iotop

iotop 是一個 Python 編寫的工具，有類似top工具的 UI，包括一些引數也和top類似。不過它對系統有一些要求，分別是：

Python ≥ 2.5 or Python ≥ 2.4 with the ctypes module
Kernel ≥ 2.6.20
Kernel uses options:
1. TASK_DELAY_ACCT
2. CONFIG_TASKSTATS
3. TASK_IO_ACCOUNTING
4. CONFIG_VM_EVENT_COUNTERS

如果是基於 RPM 包的系統，可以直接下載編譯好的二進位制包(here)或者二進位制原始碼包(here)

如果是 Debian/Ubuntu 系統，直接使用

sudo apt-get install iotop

即可（不得不說，Debian 系統提供的軟體真的是相當豐富呀)，其他系統則可以通過下面的指令下載原始碼，然後編譯

git clone git://repo.or.cz/iotop.git

具體的使用方法可以參考 iotop(8)手冊，下面是在我機器上的一個顯示：

<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">1</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">2</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">3</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">4</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">5</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">6</span>


iotop -o -u wgzhao
Total DISK READ:       2.15 M/s | Total DISK WRITE:    1601.15 K/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
 5984 be/4 wgzhao      2.15 M/s   70.55 K/s  0.00 % 83.67 % postgres: wgzhao pgbench [local] UPDATE
 4305 be/4 wgzhao      0.00 B/s  227.34 K/s  0.00 %  0.00 % postgres: writer process
 4308 be/4 wgzhao      0.00 B/s   90.15 K/s  0.00 %  0.00 % postgres: stats collector process

iopp

iopp 是另外一個統計每一個程式 I/O 的工具，使用 C 語言編寫，理論上應該比上述兩個重狙效率都要高。安裝方法很簡單，首先通過下面的指令下載原始碼:

git://github.com/markwkm/iopp.git

然後分別通過下面的指令編譯安裝

<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">1</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">2</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">3</span>


cmake CMakeLists.txt
make
make install DESTDIR=/usr

下面是一個使用例子

<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">1</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">2</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">3</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">4</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">5</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">6</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">7</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">8</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">9</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">10</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">11</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">12</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">13</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">14</span>


iopp -i -c 2
  pid    rchar    wchar    syscr    syscw   rbytes   wbytes  cwbytes command
 2144        0      296       40        8        0        0        0 /usr/sbin/LCDd
 2284        0        0        2        0        0        0        0 ha_logd: read process
 2299        0        0        2        0        0        0        0 ha_logd: write process
 2520        3        3        3        3        0        0        0 /usr/lib/virtualbox/vboxwebsrv
 2599        2        2        2        2        0        0        0 /usr/lib/virtualbox/VBoxSVC
 2675        0        0        1        0        0        0        0 runsvdir
 3177       16       16        4        2        0        0        0 /usr/bin/gnome-shell
 3192       16       16        4        2        0        0        0 nautilus
 3305      180      340      100       60        0        0        0 /usr/lib/icedove/icedove-bin
 3623     1393     1440        1        1        0        0        0 sshd: wgzhao@pts/0
 4305        0  4603904        0      562        0  4603904        0 postgres: writer process   
 6257  2064384  1892352      252      215  3719168   139264        0 postgres: wgzhao pgbench [local] UPDATE

上述輸出的各項含義是：

pid 程式 ID
rchar 將要從磁碟讀取的位元組數
wchar 已經寫入或應該要寫入磁碟的位元組數
syscr 讀 I/O 數
syscw 寫 I/O 數
rbytes 真正從磁碟讀取的位元組數
wbytes 真正寫入到磁碟的位元組數
cwbytes 因為清空頁面快取而導致沒有發生操作的位元組數
command 執行的命令

其中rbytes,wbytes,cwbytes會因給出-k或者-m引數，而顯示為rkb,wkb,cwkb或rmb,wmb,cwmb。command一列如果給出-c的引數則顯示完整的命令名而不僅僅只是命令本身。這些引數的使用和top類似。

更具體的可以參考 iopp(8)手冊。

dstat

dstat 號稱各種資源統計工具，其目的是想替代vmstat,iostat,netstat,ifstat等各種單一統計工具，從而做到All in one。 dstat 用 Python 語言編寫。

dstat 能夠清晰顯示每列的資訊，特別是單位及大小很明確，不會在單位換算上犯迷糊和失誤。最重要的是，因為它是基於模組化設計，因此我們可以很容易的寫一個外掛來收集我們需要的統計資訊。

另外，dstat 的輸出還可以匯出為CSV格式檔案，從而可以在電子表格工具裡分方便的生成統計圖形。

目前 dstat 的外掛已經相當多了，這是我機器上目前的輸出:

<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">1</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">2</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">3</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">4</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">5</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">6</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">7</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">8</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">9</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">10</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">11</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">12</span>


$ dstat  --list
internal:
  aio, cpu, cpu24, disk, disk24, disk24old, epoch, fs, int, int24, io, ipc, load, lock, mem, net, 
  page, page24, proc, raw, socket, swap, swapold, sys, tcp, time, udp, unix, vm
/usr/share/dstat:
  battery, battery-remain, cpufreq, dbus, disk-tps, disk-util, dstat, dstat-cpu, dstat-ctxt, 
  dstat-mem, fan, freespace, gpfs, gpfs-ops, helloworld, innodb-buffer, innodb-io, innodb-ops, lustre, 
  memcache-hits, mysql-io, mysql-keys, mysql5-cmds, mysql5-io, mysql5-keys, net-packets, nfs3, 
  nfs3-ops, nfsd3, nfsd3-ops, ntp, postfix, power, proc-count, qmail, rpc, rpcd, sendmail, snooze, 
  squid, test, thermal, top-bio, top-bio-adv, top-childwait, top-cpu, top-cpu-adv, top-cputime, 
  top-cputime-avg, top-int, top-io, top-io-adv, top-latency, top-latency-avg, top-mem, top-oom, utmp, 
  vm-memctl, vmk-hba, vmk-int, vmk-nic, vz-cpu, vz-io, vz-ubc, wifi

下面給出幾個使用的列子（實際輸出是帶彩色的，很容易識別）

dstat 的預設輸出

<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">1</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">2</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">3</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">4</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">5</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">6</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">7</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">8</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">9</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">10</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">11</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">12</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">13</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">14</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">15</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">16</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">17</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">18</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">19</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">20</span>


wgzhao-nb:~# dstat
You did not select any stats, using -cdngy by default.
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw 
  2   1  87  10   0   0| 816k  385k|   0     0 |   0     0 |2279  7048 
  5   1  78  16   0   0|2600k    0 | 140B  940B|   0     0 |5952    13k
  5   3  80  12   0   0|2896k  182k|  70B  358B|   0     0 |6074    14k
  4   2  78  16   0   0|2724k    0 |  70B  374B|   0     0 |5703    15k
  4   2  81  14   0   0|3008k    0 |  70B  358B|   0     0 |5924    13k
  5   1  80  14   0   0|1976k   17k|  70B  358B|   0     0 |5819    13k
  5   2  79  14   0   0|2056k    0 | 198B  374B|   0     0 |5618    13k
  4   2  79  15   0   0|2416k    0 |  70B  358B|   0     0 |5866    15k
  5   2  78  15   0   0|2528k    0 |  70B  358B|   0     0 |6356    14k
  5   2  78  16   0   0|2288k    0 |  70B  358B|   0     0 |6515    15k
  5   2  79  14   0   0|2656k 8192B|  70B  358B|   0     0 |6490    15k
  3   2  81  13   0   0|2296k    0 |  70B  374B|   0     0 |5573    15k
  4   3  76  17   0   1|2224k    0 |  70B  358B|   0     0 |5366    12k
  5   1  81  13   0   0|2208k    0 | 508B  358B|   0     0 |5403    13k
  4   2  79  15   0   0|2024k  182k|  70B  358B|   0     0 |5583    13k
  5   2  79  15   0   0|2148k   17k| 186B  490B|   0     0 |5400    12k

指定需要顯示的列

<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">1</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">2</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">3</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">4</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">5</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">6</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">7</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">8</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">9</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">10</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">11</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">12</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">13</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">14</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">15</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">16</span>


wgzhao-nb:~# dstat  -c --top-cpu -d --top-bio --top-latency
Module dstat_top_latency failed to load. (Kernel has no scheduler statistics, use at least 2.6.12)
----total-cpu-usage---- -most-expensive- -dsk/total- ----most-expensive----
usr sys idl wai hiq siq|  cpu process   | read  writ|  block i/o process   
  2   1  87  10   0   0|gnome-shell  0.7| 826k  384k|postgres    692k   52k
  4   2  79  16   0   0|postgres: wgz3.0|1744k  776k|postgres: w1744k   72k
  5   3  78  15   0   0|postgres: wgz5.0|3120k    0 |postgres: w3064k  136k
  6   2  73  19   0   0|postgres: wgz4.2|2608k  285k|postgres: w2608k  136k
  4   2  77  17   0   0|postgres: wgz3.5|2112k  848k|postgres: w2112k   88k
  3   2  71  25   0   0|postgres: wgz2.0| 944k 1049k|postgres: w 936k   48k
  3   2  58  37   0   0|postgres: wgz2.0| 920k 2070k|postgres: w 928k   64k
  3   2  62  34   0   0|postgres: wgz2.2|1496k  992k|postgres: w1608k   72k
  3   2  56  38   0   0|postgres: wgz3.0|1840k  645k|postgres: w1856k   88k
  3   2  78  17   0   0|postgres: wgz3.0|1420k 1200k|postgres: w1292k   80k
  5   2  80  12   0   1|postgres: wgz4.2|2628k    0 |postgres: w2636k  112k
  4   3  69  25   0   0|postgres: wgz3.8|2168k  576k|postgres: w2224k  104k

指定需要顯示的列，並同時將結果匯出到檔案

<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">1</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">2</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">3</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">4</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">5</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">6</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">7</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">8</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">9</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">10</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">11</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">12</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">13</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">14</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(88, 110, 117) !important;">15</span>


wgzhao-nb:~# dstat  --disk --mem --proc --io --sys --filesystem --tcp --vm --output dstat.csv
-dsk/total- ------memory-usage----- ---procs--- --io/total- ---system-- --filesystem- ----tcp-sockets---- -----virtual-memory----
 read  writ| used  buff  cach  free|run blk new| read  writ| int   csw |files  inodes|lis act syn tim clo|majpf minpf alloc  free
 844k  404k| 829M 19.4M 2920M  124M|  0 0.0 0.7|47.5  38.4 |2336  7185 | 4928  12286 | 11   3   0   0   2|   1   620   602   605 
2128k 1526k| 828M 19.4M 2915M  130M|  0 2.0   0| 111   157 |4588    14k| 4928  12285 | 11   3   0   0   2|   0  1859   995  2278 
 920k 2151k| 826M 19.4M 2917M  129M|  0 2.0   0|52.0   237 |3091  7540 | 4928  12284 | 11   3   0   0   2|   0  4448  2330  2144 
2124k 1003k| 826M 19.4M 2921M  126M|1.0 1.0   0| 135   106 |4705    14k| 4928  12284 | 11   3   0   0   2|   0   331   865     1 
2344k 1024k| 826M 19.4M 2924M  122M|1.0 2.0   0| 121   118 |4074    13k| 4928  12284 | 11   3   0   0   2|   0   249   953     1 
1572k 1624k| 827M 19.4M 2926M  120M|1.0 2.0   0|87.0   190 |3231    11k| 4928  12284 | 11   3   0   0   2|   0    98   530     1 
 916k  788k| 827M 19.4M 2928M  119M|  0 2.0   0|68.0  92.0 |3452  8709 | 4928  12284 | 11   3   0   0   2|   0   128   383     4 
2452k 1665k| 826M 19.4M 2931M  116M|1.0 1.0   0| 132   197 |4779    14k| 4928  12284 | 11   3   0   0   2|   0   208   822     1 
1552k 1328k| 827M 19.4M 2933M  114M|  0 2.0   0|97.0   156 |3762  9117 | 4928  12284 | 11   3   0   0   2|   0   133   473     1 
1192k 2024k| 827M 19.4M 2934M  112M|  0 2.0   0|81.0   239 |4068    11k| 4928  12284 | 11   3   0   0   2|   0   135   414     2 
2668k  584k| 827M 19.4M 2937M  109M|  0 2.0   0| 148  71.0 |4415    10k| 4928  12284 | 11   3   0   0   2|   0   174   870     4 
1712k  960k| 827M 19.4M 2940M  106M|  0 2.0   0| 122   113 |4454    14k| 4928  12284 | 11   3   0   0   2|   0   182   616     2

更詳細的用法，可以參考 dstat(1)手冊

Linux 下的I/O效能分析 iotop
2011-03-23
Linux
Linux下的5種I/O模型與3組I/O複用
2022-02-03
Linux模型
Linux下磁碟I/O測試
2021-08-31
Linux
在Linux下測試磁碟的I/O
2006-04-20
Linux
計算機I/O與I/O模型
2019-05-10
計算機模型
【工具】ORION I/O 測試工具
2018-04-28
系統程式設計 - I/O模型
2020-10-11
程式設計模型
Linux命令----分析系統I/O的瓶頸
2022-03-23
Linux
Linux下如何測試磁碟I/O: ( hdparm -t)
2006-08-17
Linux
系統級 I/O
2019-01-21
JAVA I/O系統
2018-11-13
Java
Linux系統監控之磁碟I/O篇
2012-12-17
Linux
Linux系統的Shell I/O重定向詳解(轉)
2007-08-16
Linux
Linux Shell程式設計（25）——I/O 重定向
2014-09-30
Linux程式設計
Linux I/O排程器
2019-03-04
Linux
深入 Linux I/O 重定向
2017-01-19
Linux
在Linux下開啟32位I/O和DMA(轉)
2007-08-12
Linux
Veritas Quick I/O and Cached Quick I/O
2009-03-19
UI
如何監測 Linux 的磁碟 I/O 效能
2022-09-19
Linux
如何更改Linux的I/O排程器
2018-03-13
Linux
【I/O scheduler】Linux的磁碟排程策略
2013-08-10
Linux
Linux裡五種I/O模型
2019-01-19
Linux模型
Linux libaio 非同步I/O
2017-11-23
LinuxAI非同步
linux I/O 瓶頸監控
2012-03-16
Linux
java的I/O
2011-11-26
Java
作業系統—I/O 模型
2020-10-09
作業系統模型
Linux 下使用 dd 命令進行硬碟 I/O 效能檢測
2015-08-29
Linux硬碟
[溫故而知新] 《Linux/Unix系統程式設計手冊》——檔案I/O
2016-10-23
Linux程式設計
AIX 下磁碟 I/O 效能分析[轉]
2014-01-14
AI
【AIX】AIX 下磁碟 I/O 效能分析
2012-06-14
AI
I/O埠和I/O記憶體
2012-05-25
記憶體
linux 下filesystemio_options disk_asynch_io 導致的I/O效能下降
2015-03-02
Linux
linux監測I/O效能－iostat
2010-08-27
LinuxiOS
HOW TO CHECK IF ASYNCHRONOUS I/O IS WORKING ON LINUX
2007-09-10
Linux
CPU-bound(計算密集型) 和I/O bound(I/O密集型)
2009-11-10
磁碟效能測試工具 flexible I/O tester
2011-11-09
Flex
設計模式中巧記I/O
2020-04-19
設計模式
Linux之《荒島餘生》（四）I/O篇
2018-12-22
Linux