[20130109]使用bbcp傳輸檔案.txt

[20130109]使用bbcp傳輸檔案.txt

今天看了一些blog，發現一些blog提到了bbcp。使用它來傳輸大檔案據說很快。

相關站點：
~abh/bbcp/bin/
~abh/bbcp/

--不行，可以自己下載編譯：
git clone ~abh/bbcp/bbcp.git
cd bbcp
make
cp bin/amd64_linux26/bbcp /usr/local/bin/bbcp


1.開始測試：
一看文件，暈！這麼多引數，使用簡單一點看看:
~abh/bbcp/

bbcp -P 2 /data/orcl/users01.dbf oracle@remote_host:/tmp/users01.dbf
--提示，執行檔名明明在那裡，為什麼不行。直接進入/usr/local/bin目錄執行，也一樣。
bash: bbcp: command not found
bbcp: bbcp unexpectedly terminated on XXX.XXX.XXX.XXX

--仔細看了相關文件
bbcp assumes the remote system's non-interactive environment contains the path to the bbcp utility.
This can be determined by with the following command:
ssh remotesystem which bbcp

--也就是遠端也需要這個一個執行檔案。將bbcp複製過去。

#  bbcp -P 2 /data/orcl/users01.dbf oracle@remote_host:/tmp/users01.dbf
oracle@xxX.xxx.xxx.xxx's password:
bbcp: Creating /tmp/users01.dbf
bbcp: 130109 16:50:04  5% done; 55.7 MB/s
bbcp: 130109 16:50:06  15% done; 63.2 MB/s
bbcp: 130109 16:50:08  25% done; 66.1 MB/s
bbcp: 130109 16:50:10  36% done; 71.3 MB/s
bbcp: 130109 16:50:12  47% done; 72.8 MB/s
bbcp: 130109 16:50:14  55% done; 70.0 MB/s
bbcp: 130109 16:50:16  65% done; 70.7 MB/s
bbcp: 130109 16:50:18  74% done; 70.4 MB/s
bbcp: 130109 16:50:20  84% done; 70.4 MB/s
bbcp: 130109 16:50:22  95% done; 71.6 MB/s

--檔案大小1.4G,大約20秒完成。
# du -sm /tmp/users01.dbf
1467    /tmp/users01.dbf

2.使用scp做一個比較:
scp     /data/orcl.20121016/users01.dbf oracle@192.168.101.209:/tmp/users01.dbf

users01.dbf 100% 1465MB  45.8MB/s   00:32

lftp也做了一個測試，稍微快一點，大約30秒。1536180224 bytes transferred in 30 seconds (48.95M/s)


3.另外bbcp有一個好處，如果複製檔案已經存在，提示檔案存在直接退出。
加入-f 引數
--force | -f     

forces the copy by erasing the target prior to copying the source file. By default, if the target already exists for the
source file, the copy fails. See also the –K option modifier.

如果在複製檔案時，在遠端機器使用ps -elf | grep bbcp 看。
# ps -elf | grep bbcp | grep -v grep
0 S oracle    7398  7397  1  81   0 - 12251 wait   17:04 ?        00:00:00 bbcp SNK
1 S oracle    7427  7398 68  76   0 - 22495 184466 17:04 ?        00:00:13 bbcp SNK

--會有2個程式。

看了一些文件從調整效能講重要的兩個引數（我自己認為）,我估計如果機器忙，頻寬不足，也許會影響這個網路。

1.2.1        Window Size (-w)
The first and most important option is –w. This determines the TCP window size as well as the default I/O size.  The
default of 128Kis usually good enough for LAN and larger values are likely to hurt performance.

1.2.2        Streams (-s)
The -- s options is the second most important tuning parameter. This specifies the number of parallel TCP streams. A
na?ve explanation would say that streams can make up for not having a large enough window. The idea is that if you can’
t get enough packets moving in a single window, then create multiple windows and run them simultaneously. This is only
partially true. While multiple streams do provide multiple windows, multiple windows also parallelize traffic with
independent time-outs, re-transmissions, and greatly improved I/O overlapping. This is a cumulative effect that
dramatically increases the overall bandwidth utilization. But, too much of a good thing can also be bad, as you will see
below.