一、簡介
MogileFS是一個開源的分散式檔案儲存系統，由LiveJournal旗下的Danga Interactive公司開發；它主要由三部分組成，第一部分是server端，server端主要包括mogilefsd和mogstored兩個應用程式。mogilefsd實現的是tracker，它通過資料庫來儲存後設資料資訊，包括站點domain、class、host等；mogstored是儲存節點(store node)，它其實是個WebDAV服務，預設監聽在7500埠，接受客戶端的檔案儲存請求。在MogileFS安裝完後，要執行mogadm工具將所有的store node註冊到mogilefsd的資料庫裡，mogilefsd會對這些節點進行管理和監控。第二部是mogilefs的utils,這部分主要是一些工具，比如mogadm、mogupload、mogfileinfo 等等；第三部分是客戶端API，mogilefs支援眾多程式語言的客戶端API介面，使用對應的客戶端API介面，我們可以編寫mogilefs的客戶端，從而實現來管理mogilefs上的檔案；
二、Mogilefs架構圖
提示：tracker的主要作用是代理前端使用者的請求到資料庫查詢相關檔案的元資訊，同時它還要監控註冊到tracker的storagenode的健康狀態，一旦後端storage掛掉，它會立刻把對應的storage節點上的device從叢集的中刪除，上線時在加入到叢集；客戶端通過向tracker傳送請求檔案的資訊，tracker通過去資料庫中找到對應的檔案所在storage節點的ip地址、埠、裝置id以及檔案fid，然後通過http連結的形式從tracker返回給客戶端，客戶端拿到這tracker的返回的檔案地址，然後在去對應的storage請求檔案，從而在客戶端就訪問到對應的檔案內容資訊；客戶端和tracker，storage之間通訊都是http協議，從本質上講storage就是一個web伺服器；所以對於客戶端我們可以使用web伺服器的反向代理，也可以是呼叫mogilefsAPI寫的客戶端程式；
三、Mogilefs安裝部署
環境說明
| 名稱 | 角色 | ip地址 | 埠 |
| node01 | tracker+mysql+storage | 192.168.0.41 | 7001/7500/7501/3306 |
| node02 | tracker+storage | 192.168.0.42 | 7001/7500/7501 |
| node03 | storage | 192.168.0.43 | 7500/7501 |
各server時間同步
提示：如果有必要，可以配置各節點用hosts檔案解析；
安裝環境依賴包
[root@node01 ~]# yum install perl-Net-Netmask perl-IO-stringy perl-Sys-Syslog perl-IO-AIO -y Loaded plugins: fastestmirror base | 3.6 kB 00:00:00 epel | 4.7 kB 00:00:00 extras | 2.9 kB 00:00:00 updates | 2.9 kB 00:00:00 Loading mirror speeds from cached hostfile * base: mirrors.aliyun.com * extras: mirrors.aliyun.com * updates: mirrors.aliyun.com Resolving Dependencies --> Running transaction check ---> Package perl-IO-AIO.x86_64 0:4.20-1.el7 will be installed --> Processing Dependency: perl(common::sense) for package: perl-IO-AIO-4.20-1.el7.x86_64 ---> Package perl-IO-stringy.noarch 0:2.110-22.el7 will be installed ---> Package perl-Net-Netmask.noarch 0:1.9015-13.el7 will be installed ---> Package perl-Sys-Syslog.x86_64 0:0.33-3.el7 will be installed --> Running transaction check ---> Package perl-common-sense.noarch 0:3.6-4.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved ======================================================================================================================================= Package Arch Version Repository Size ======================================================================================================================================= Installing: perl-IO-AIO x86_64 4.20-1.el7 epel 117 k perl-IO-stringy noarch 2.110-22.el7 base 71 k perl-Net-Netmask noarch 1.9015-13.el7 epel 25 k perl-Sys-Syslog x86_64 0.33-3.el7 base 42 k Installing for dependencies: perl-common-sense noarch 3.6-4.el7 epel 28 k Transaction Summary ======================================================================================================================================= Install 4 Packages (+1 Dependent package) Total download size: 283 k Installed size: 636 k Downloading packages: (1/5): perl-Sys-Syslog-0.33-3.el7.x86_64.rpm | 42 kB 00:00:00 (2/5): perl-Net-Netmask-1.9015-13.el7.noarch.rpm | 25 kB 00:00:00 (3/5): perl-IO-AIO-4.20-1.el7.x86_64.rpm | 117 kB 00:00:00 (4/5): perl-common-sense-3.6-4.el7.noarch.rpm | 28 kB 00:00:00 (5/5): perl-IO-stringy-2.110-22.el7.noarch.rpm | 71 kB 00:00:00 --------------------------------------------------------------------------------------------------------------------------------------- Total 642 kB/s | 283 kB 00:00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : perl-common-sense-3.6-4.el7.noarch 1/5 Installing : perl-IO-AIO-4.20-1.el7.x86_64 2/5 Installing : perl-Net-Netmask-1.9015-13.el7.noarch 3/5 Installing : perl-Sys-Syslog-0.33-3.el7.x86_64 4/5 Installing : perl-IO-stringy-2.110-22.el7.noarch 5/5 Verifying : perl-IO-AIO-4.20-1.el7.x86_64 1/5 Verifying : perl-common-sense-3.6-4.el7.noarch 2/5 Verifying : perl-IO-stringy-2.110-22.el7.noarch 3/5 Verifying : perl-Sys-Syslog-0.33-3.el7.x86_64 4/5 Verifying : perl-Net-Netmask-1.9015-13.el7.noarch 5/5 Installed: perl-IO-AIO.x86_64 0:4.20-1.el7 perl-IO-stringy.noarch 0:2.110-22.el7 perl-Net-Netmask.noarch 0:1.9015-13.el7 perl-Sys-Syslog.x86_64 0:0.33-3.el7 Dependency Installed: perl-common-sense.noarch 0:3.6-4.el7 Complete! [root@node01 ~]#
提示：三個節點都需要安裝環境依賴的包；
上傳rpm包到伺服器並安裝
[root@node01 ~]# rz rz waiting to receive. zmodem trl+C ȡ 100% 1 KB 1 KB/s 00:00:01 0 Errors.el7.centos.noarch.rpm... 100% 173 KB 173 KB/s 00:00:01 0 Errorsentos.noarch.rpm... 100% 26 KB 26 KB/s 00:00:01 0 Errorsentos.noarch.rpm... 100% 75 KB 75 KB/s 00:00:01 0 Errorsh.rpm... 100% 27 KB 27 KB/s 00:00:01 0 Errors.rpm... 100% 29 KB 29 KB/s 00:00:01 0 Errors.noarch.rpm... 100% 262 KB 262 KB/s 00:00:01 0 Errors [root@node01 ~]# ls MogileFS-Server-2.46-2.el7.centos.noarch.rpm perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm MogileFS-Server-mogilefsd-2.46-2.el7.centos.noarch.rpm perl-MogileFS-Client-1.14-1.el7.centos.noarch.rpm MogileFS-Server-mogstored-2.46-2.el7.centos.noarch.rpm perl-Perlbal-1.78-1.el6.noarch.rpm MogileFS-Utils-2.19-1.el7.centos.noarch.rpm [root@node01 ~]# yum install ./*.rpm -y Loaded plugins: fastestmirror Examining ./MogileFS-Server-2.46-2.el7.centos.noarch.rpm: MogileFS-Server-2.46-2.el7.centos.noarch Marking ./MogileFS-Server-2.46-2.el7.centos.noarch.rpm to be installed Examining ./MogileFS-Server-mogilefsd-2.46-2.el7.centos.noarch.rpm: MogileFS-Server-mogilefsd-2.46-2.el7.centos.noarch Marking ./MogileFS-Server-mogilefsd-2.46-2.el7.centos.noarch.rpm to be installed Examining ./MogileFS-Server-mogstored-2.46-2.el7.centos.noarch.rpm: MogileFS-Server-mogstored-2.46-2.el7.centos.noarch Marking ./MogileFS-Server-mogstored-2.46-2.el7.centos.noarch.rpm to be installed Examining ./MogileFS-Utils-2.19-1.el7.centos.noarch.rpm: MogileFS-Utils-2.19-1.el7.centos.noarch Marking ./MogileFS-Utils-2.19-1.el7.centos.noarch.rpm to be installed Examining ./perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm: perl-Danga-Socket-1.61-1.el6.rf.noarch Marking ./perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm to be installed Examining ./perl-MogileFS-Client-1.14-1.el7.centos.noarch.rpm: perl-MogileFS-Client-1.14-1.el7.centos.noarch Marking ./perl-MogileFS-Client-1.14-1.el7.centos.noarch.rpm to be installed Examining ./perl-Perlbal-1.78-1.el6.noarch.rpm: perl-Perlbal-1.78-1.el6.noarch Marking ./perl-Perlbal-1.78-1.el6.noarch.rpm to be installed Resolving Dependencies --> Running transaction check ---> Package MogileFS-Server.noarch 0:2.46-2.el7.centos will be installed ---> Package MogileFS-Server-mogilefsd.noarch 0:2.46-2.el7.centos will be installed --> Processing Dependency: perl(DBI) >= 1.44 for package: MogileFS-Server-mogilefsd-2.46-2.el7.centos.noarch Loading mirror speeds from cached hostfile * base: mirrors.aliyun.com * extras: mirrors.aliyun.com * updates: mirrors.aliyun.com --> Processing Dependency: perl(DBD::mysql) >= 3 for package: MogileFS-Server-mogilefsd-2.46-2.el7.centos.noarch ---> Package MogileFS-Server-mogstored.noarch 0:2.46-2.el7.centos will be installed ……省略部分內容…… Installed: MogileFS-Server.noarch 0:2.46-2.el7.centos MogileFS-Server-mogilefsd.noarch 0:2.46-2.el7.centos MogileFS-Server-mogstored.noarch 0:2.46-2.el7.centos MogileFS-Utils.noarch 0:2.19-1.el7.centos perl-Danga-Socket.noarch 0:1.61-1.el6.rf perl-MogileFS-Client.noarch 0:1.14-1.el7.centos perl-Perlbal.noarch 0:1.78-1.el6 Dependency Installed: mailcap.noarch 0:2.1.41-2.el7 perl-BSD-Resource.x86_64 0:1.29.07-1.el7 perl-Business-ISBN.noarch 0:2.06-2.el7 perl-Business-ISBN-Data.noarch 0:20120719.001-2.el7 perl-Compress-Raw-Bzip2.x86_64 0:2.061-3.el7 perl-Compress-Raw-Zlib.x86_64 1:2.061-4.el7 perl-DBD-MySQL.x86_64 0:4.023-6.el7 perl-DBI.x86_64 0:1.627-4.el7 perl-Data-Dumper.x86_64 0:2.145-3.el7 perl-Digest.noarch 0:1.17-245.el7 perl-Digest-MD5.x86_64 0:2.52-3.el7 perl-Encode-Locale.noarch 0:1.03-5.el7 perl-File-Listing.noarch 0:6.04-7.el7 perl-HTML-Parser.x86_64 0:3.71-4.el7 perl-HTML-Tagset.noarch 0:3.20-15.el7 perl-HTTP-Cookies.noarch 0:6.01-5.el7 perl-HTTP-Daemon.noarch 0:6.01-8.el7 perl-HTTP-Date.noarch 0:6.02-8.el7 perl-HTTP-Message.noarch 0:6.06-6.el7 perl-HTTP-Negotiate.noarch 0:6.01-5.el7 perl-IO-Compress.noarch 0:2.061-2.el7 perl-IO-HTML.noarch 0:1.00-2.el7 perl-IO-Socket-IP.noarch 0:0.21-5.el7 perl-IO-Socket-SSL.noarch 0:1.94-7.el7 perl-LWP-MediaTypes.noarch 0:6.02-2.el7 perl-Mozilla-CA.noarch 0:20130114-5.el7 perl-Net-Daemon.noarch 0:0.48-5.el7 perl-Net-HTTP.noarch 0:6.06-2.el7 perl-Net-LibIDN.x86_64 0:0.12-15.el7 perl-Net-SSLeay.x86_64 0:1.55-6.el7 perl-PlRPC.noarch 0:0.2020-14.el7 perl-Sys-Syscall.noarch 0:0.25-10.el7 perl-TimeDate.noarch 1:2.30-2.el7 perl-URI.noarch 0:1.60-9.el7 perl-WWW-RobotRules.noarch 0:6.02-5.el7 perl-libwww-perl.noarch 0:6.05-2.el7 Complete! [root@node01 ~]#
提示：安裝環境依賴和以上rpm包，首先要配置好epel源；因為有些依賴包，它來自epel源；相關包我已經上傳至我的網盤http://www.qiuhom.com/d/383f34f57cf847beb296/；
安裝mariadb-server
配置yum源
[root@node01 ~]# cat /etc/yum.repos.d/mairadb.repo [mariadb] name=mariadb repo baseurl=https://mirrors.tuna.tsinghua.edu.cn/mariadb//mariadb-10.1.46/yum/centos/7/x86_64/ gpgcheck=0 [root@node01 ~]#
安裝mariadb-server
[root@node01 ~]# yum install MariaDB-server -y Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.aliyun.com * extras: mirrors.aliyun.com * updates: mirrors.aliyun.com Resolving Dependencies --> Running transaction check ---> Package MariaDB-server.x86_64 0:10.1.46-1.el7.centos will be installed --> Processing Dependency: MariaDB-client for package: MariaDB-server-10.1.46-1.el7.centos.x86_64 --> Processing Dependency: rsync for package: MariaDB-server-10.1.46-1.el7.centos.x86_64 --> Processing Dependency: MariaDB-common for package: MariaDB-server-10.1.46-1.el7.centos.x86_64 --> Processing Dependency: galera for package: MariaDB-server-10.1.46-1.el7.centos.x86_64 --> Processing Dependency: libjemalloc.so.1()(64bit) for package: MariaDB-server-10.1.46-1.el7.centos.x86_64 --> Running transaction check ……省略部分內容…… Installed: MariaDB-server.x86_64 0:10.1.46-1.el7.centos MariaDB-shared.x86_64 0:10.1.46-1.el7.centos Dependency Installed: MariaDB-client.x86_64 0:10.1.46-1.el7.centos MariaDB-common.x86_64 0:10.1.46-1.el7.centos boost-program-options.x86_64 0:1.53.0-28.el7 galera.x86_64 0:25.3.29-1.rhel7.el7.centos jemalloc.x86_64 0:3.6.0-1.el7 rsync.x86_64 0:3.1.2-10.el7 Replaced: mariadb-libs.x86_64 1:5.5.56-2.el7 Complete! [root@node01 ~]#
提示：mariadb-server包名是大寫的MariaDB-server；
配置mariadb忽略名稱解析
設定mariadb開機自動啟動，並啟動mariadb
新增用於mogilefs訪問資料庫的賬號
初始化資料庫
[root@node01 ~]# mogdbsetup --help
Usage: mogdbsetup [opts]
Options:
Default Description
============ ===========================================
--verbose <off> Be verbose about what's happening.
--dbhost= localhost hostname or IP to database server.
--dbport= dbd default port number to database server.
--dbname= mogilefs database name to create/upgrade.
--dbrootuser= root Database administrator username. Only needed
for initial setup, not subsequent upgrades.
--dbrootpass= <blank> Database administrator password. Only needed
for initial setup, not subsequent upgrades.
--dbuser= mogile Regular database user to create and/or use
for MogileFS database. This is what the
mogilefsd trackers connect as.
--dbpass= <blank> You should change this, especially if your
database servers are accessible to other users
on the network. But they shouldn't be
if you're running MogileFS, because MogileFS
assumes your network is closed.
--type= MySQL Which MogileFS::Store implementation to use.
Available: MySQL, Postgres
--yes Run without questions.
[root@node01 ~]# mogdbsetup --dbhost=192.168.0.41 --dbport=3306 --dbname=mogilefs --dbrootpass=admin --dbuser=mogilefs --dbpass=mogilefs
This will attempt to setup or upgrade your MogileFS database.
It won't destroy existing data.
Run with --help for more information. Run with --yes to shut up these prompts.
Continue? [N/y]: y
Create/Upgrade database name 'mogilefs'? [Y/n]: y
Grant all privileges to user 'mogilefs', connecting from anywhere, to the mogilefs database 'mogilefs'? [Y/n]: n
Stopped.
[root@node01 ~]# mogdbsetup --dbhost=192.168.0.41 --dbport=3306 --dbname=mogilefs --dbrootpass=admin --dbuser=mogilefs --dbpass=mogilefs
This will attempt to setup or upgrade your MogileFS database.
It won't destroy existing data.
Run with --help for more information. Run with --yes to shut up these prompts.
Continue? [N/y]: y
[root@node01 ~]#
提示：如果不想讓mogdbsetup在資料庫中建立一個任何地址都可訪問的mogilefs的使用者，在最後的選項中使用n即可，然後在重複執行上述命令即可；如果不想互動式輸入y，也可以直接在命令後面用--yes；
驗證：在資料庫中檢視對應的庫是否都已建立，對應庫中是否都有表？
[root@node01 ~]# mysql Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 7 Server version: 10.1.46-MariaDB MariaDB Server Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mogilefs | | mysql | | performance_schema | | test | +--------------------+ 5 rows in set (0.01 sec) MariaDB [(none)]> use mogilefs Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed MariaDB [mogilefs]> show tables; +----------------------+ | Tables_in_mogilefs | +----------------------+ | checksum | | class | | device | | domain | | file | | file_on | | file_on_corrupt | | file_to_delete | | file_to_delete2 | | file_to_delete_later | | file_to_queue | | file_to_replicate | | fsck_log | | host | | server_settings | | tempfile | | unreachable_fids | +----------------------+ 17 rows in set (0.00 sec) MariaDB [mogilefs]>
到此資料庫就準備好了
配置mogilefsd連線資料庫
提示：我們只需要修改紅框中的內容即可；db_dsn用於指定連線資料庫的驅動，以及資料庫庫名和主機地址；db_user用於指定連線資料庫所使用的使用者，db_pass用於指定連線資料庫的密碼；listen用於指定mogilefsd監聽的ip地址和埠；後面的配置就是和啟動的程式數量的配置，這個可以根據業務的規模來設定；
啟動mogilefsd
提示：在centos7上啟動mogilefsd時，會報錯，我們可以不用管這個錯誤，實際上mogilefsd是啟動成功的，只要看到對應的埠和程式起來，表示mogilefsd啟動就沒有問題；
配置node01上的mogstored
[root@node01 ~]# cat /etc/mogilefs/mogstored.conf maxconns = 10000 httplisten = 0.0.0.0:7500 mgmtlisten = 0.0.0.0:7501 docroot = /data/mogilefs [root@node01 ~]# mkdir /data/mogilefs/ -pv mkdir: created directory ‘/data’ mkdir: created directory ‘/data/mogilefs/’ [root@node01 ~]# chown -R mogilefs.mogilefs /data/mogilefs/ [root@node01 ~]# ll -d /data/mogilefs/ drwxr-xr-x 2 mogilefs mogilefs 6 Sep 16 13:37 /data/mogilefs/ [root@node01 ~]#
啟動mogstored
提示：在centos7上啟動mogstored也會同啟動mogilefsd一樣報錯，這個主要是因為systemd的緣故，在centos6不會報錯；到此，node01上的mariadb、mogilefsd和mogstored都已經安裝配置好了；
複製安裝包到node02和node03
在node02上安裝
[root@node02 ~]# yum install perl-Net-Netmask perl-IO-stringy perl-Sys-Syslog perl-IO-AIO ./*.rpm -y Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.aliyun.com * extras: mirrors.aliyun.com * updates: mirrors.aliyun.com Package perl-Net-Netmask-1.9015-13.el7.noarch already installed and latest version Package perl-IO-stringy-2.110-22.el7.noarch already installed and latest version Package perl-Sys-Syslog-0.33-3.el7.x86_64 already installed and latest version Package perl-IO-AIO-4.20-1.el7.x86_64 already installed and latest version Examining ./MogileFS-Server-2.46-2.el7.centos.noarch.rpm: MogileFS-Server-2.46-2.el7.centos.noarch Marking ./MogileFS-Server-2.46-2.el7.centos.noarch.rpm to be installed Examining ./MogileFS-Server-mogilefsd-2.46-2.el7.centos.noarch.rpm: MogileFS-Server-mogilefsd-2.46-2.el7.centos.noarch Marking ./MogileFS-Server-mogilefsd-2.46-2.el7.centos.noarch.rpm to be installed Examining ./MogileFS-Server-mogstored-2.46-2.el7.centos.noarch.rpm: MogileFS-Server-mogstored-2.46-2.el7.centos.noarch Marking ./MogileFS-Server-mogstored-2.46-2.el7.centos.noarch.rpm to be installed Examining ./MogileFS-Utils-2.19-1.el7.centos.noarch.rpm: MogileFS-Utils-2.19-1.el7.centos.noarch Marking ./MogileFS-Utils-2.19-1.el7.centos.noarch.rpm to be installed Examining ./perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm: perl-Danga-Socket-1.61-1.el6.rf.noarch Marking ./perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm to be installed Examining ./perl-MogileFS-Client-1.14-1.el7.centos.noarch.rpm: perl-MogileFS-Client-1.14-1.el7.centos.noarch Marking ./perl-MogileFS-Client-1.14-1.el7.centos.noarch.rpm to be installed Examining ./perl-Perlbal-1.78-1.el6.noarch.rpm: perl-Perlbal-1.78-1.el6.noarch Marking ./perl-Perlbal-1.78-1.el6.noarch.rpm to be installed Resolving Dependencies --> Running transaction check ---> Package MogileFS-Server.noarch 0:2.46-2.el7.centos will be installed ---> Package MogileFS-Server-mogilefsd.noarch 0:2.46-2.el7.centos will be installed --> Processing Dependency: perl(DBI) >= 1.44 for package: MogileFS-Server-mogilefsd-2.46-2.el7.centos.noarch ……省略部分內容…… Installed: MogileFS-Server.noarch 0:2.46-2.el7.centos MogileFS-Server-mogilefsd.noarch 0:2.46-2.el7.centos MogileFS-Server-mogstored.noarch 0:2.46-2.el7.centos MogileFS-Utils.noarch 0:2.19-1.el7.centos perl-Danga-Socket.noarch 0:1.61-1.el6.rf perl-MogileFS-Client.noarch 0:1.14-1.el7.centos perl-Perlbal.noarch 0:1.78-1.el6 Dependency Installed: mailcap.noarch 0:2.1.41-2.el7 perl-BSD-Resource.x86_64 0:1.29.07-1.el7 perl-Business-ISBN.noarch 0:2.06-2.el7 perl-Business-ISBN-Data.noarch 0:20120719.001-2.el7 perl-Compress-Raw-Bzip2.x86_64 0:2.061-3.el7 perl-Compress-Raw-Zlib.x86_64 1:2.061-4.el7 perl-DBD-MySQL.x86_64 0:4.023-6.el7 perl-DBI.x86_64 0:1.627-4.el7 perl-Data-Dumper.x86_64 0:2.145-3.el7 perl-Digest.noarch 0:1.17-245.el7 perl-Digest-MD5.x86_64 0:2.52-3.el7 perl-Encode-Locale.noarch 0:1.03-5.el7 perl-File-Listing.noarch 0:6.04-7.el7 perl-HTML-Parser.x86_64 0:3.71-4.el7 perl-HTML-Tagset.noarch 0:3.20-15.el7 perl-HTTP-Cookies.noarch 0:6.01-5.el7 perl-HTTP-Daemon.noarch 0:6.01-8.el7 perl-HTTP-Date.noarch 0:6.02-8.el7 perl-HTTP-Message.noarch 0:6.06-6.el7 perl-HTTP-Negotiate.noarch 0:6.01-5.el7 perl-IO-Compress.noarch 0:2.061-2.el7 perl-IO-HTML.noarch 0:1.00-2.el7 perl-IO-Socket-IP.noarch 0:0.21-5.el7 perl-IO-Socket-SSL.noarch 0:1.94-7.el7 perl-LWP-MediaTypes.noarch 0:6.02-2.el7 perl-Mozilla-CA.noarch 0:20130114-5.el7 perl-Net-Daemon.noarch 0:0.48-5.el7 perl-Net-HTTP.noarch 0:6.06-2.el7 perl-Net-LibIDN.x86_64 0:0.12-15.el7 perl-Net-SSLeay.x86_64 0:1.55-6.el7 perl-PlRPC.noarch 0:0.2020-14.el7 perl-Sys-Syscall.noarch 0:0.25-10.el7 perl-TimeDate.noarch 1:2.30-2.el7 perl-URI.noarch 0:1.60-9.el7 perl-WWW-RobotRules.noarch 0:6.02-5.el7 perl-libwww-perl.noarch 0:6.05-2.el7 Complete! [root@node02 ~]#
把node01上的mogilefsd的配置檔案複製到node02上，然後啟動mogilefsd
把node01上的mogstored配置檔案複製到node02上，然後在node02上建立對應的目錄並修改屬主和屬組為mogilefs，然後啟動mogstored
到此node02上的mogilefsd和mogstored就安裝配置啟動好了；
在node03上安裝配置mogstored
[root@node03 ~]# ls MogileFS-Server-2.46-2.el7.centos.noarch.rpm perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm MogileFS-Server-mogilefsd-2.46-2.el7.centos.noarch.rpm perl-MogileFS-Client-1.14-1.el7.centos.noarch.rpm MogileFS-Server-mogstored-2.46-2.el7.centos.noarch.rpm perl-Perlbal-1.78-1.el6.noarch.rpm MogileFS-Utils-2.19-1.el7.centos.noarch.rpm [root@node03 ~]# yum install ./MogileFS-Server-mogstored-2.46-2.el7.centos.noarch.rpm ./MogileFS-Utils-2.19-1.el7.centos.noarch.rpm ./perl-*.rpm perl-Net-Netmask perl-IO-stringy perl-Sys-Syslog perl-IO-AIO -y Loaded plugins: fastestmirror Examining ./MogileFS-Server-mogstored-2.46-2.el7.centos.noarch.rpm: MogileFS-Server-mogstored-2.46-2.el7.centos.noarch Marking ./MogileFS-Server-mogstored-2.46-2.el7.centos.noarch.rpm to be installed Examining ./MogileFS-Utils-2.19-1.el7.centos.noarch.rpm: MogileFS-Utils-2.19-1.el7.centos.noarch Marking ./MogileFS-Utils-2.19-1.el7.centos.noarch.rpm to be installed Examining ./perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm: perl-Danga-Socket-1.61-1.el6.rf.noarch Marking ./perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm to be installed Examining ./perl-MogileFS-Client-1.14-1.el7.centos.noarch.rpm: perl-MogileFS-Client-1.14-1.el7.centos.noarch Marking ./perl-MogileFS-Client-1.14-1.el7.centos.noarch.rpm to be installed Examining ./perl-Perlbal-1.78-1.el6.noarch.rpm: perl-Perlbal-1.78-1.el6.noarch Marking ./perl-Perlbal-1.78-1.el6.noarch.rpm to be installed Loading mirror speeds from cached hostfile * base: mirrors.aliyun.com * extras: mirrors.aliyun.com * updates: mirrors.aliyun.com Package perl-Net-Netmask-1.9015-13.el7.noarch already installed and latest version Package perl-IO-stringy-2.110-22.el7.noarch already installed and latest version Package perl-Sys-Syslog-0.33-3.el7.x86_64 already installed and latest version Package perl-IO-AIO-4.20-1.el7.x86_64 already installed and latest version Resolving Dependencies --> Running transaction check ---> Package MogileFS-Server-mogstored.noarch 0:2.46-2.el7.centos will be installed ---> Package MogileFS-Utils.noarch 0:2.19-1.el7.centos will be installed --> Processing Dependency: perl(Compress::Zlib) for package: MogileFS-Utils-2.19-1.el7.centos.noarch ……省略部分內容…… Installed: MogileFS-Server-mogstored.noarch 0:2.46-2.el7.centos MogileFS-Utils.noarch 0:2.19-1.el7.centos perl-Danga-Socket.noarch 0:1.61-1.el6.rf perl-MogileFS-Client.noarch 0:1.14-1.el7.centos perl-Perlbal.noarch 0:1.78-1.el6 Dependency Installed: mailcap.noarch 0:2.1.41-2.el7 perl-BSD-Resource.x86_64 0:1.29.07-1.el7 perl-Business-ISBN.noarch 0:2.06-2.el7 perl-Business-ISBN-Data.noarch 0:20120719.001-2.el7 perl-Compress-Raw-Bzip2.x86_64 0:2.061-3.el7 perl-Compress-Raw-Zlib.x86_64 1:2.061-4.el7 perl-Data-Dumper.x86_64 0:2.145-3.el7 perl-Digest.noarch 0:1.17-245.el7 perl-Digest-MD5.x86_64 0:2.52-3.el7 perl-Encode-Locale.noarch 0:1.03-5.el7 perl-File-Listing.noarch 0:6.04-7.el7 perl-HTML-Parser.x86_64 0:3.71-4.el7 perl-HTML-Tagset.noarch 0:3.20-15.el7 perl-HTTP-Cookies.noarch 0:6.01-5.el7 perl-HTTP-Daemon.noarch 0:6.01-8.el7 perl-HTTP-Date.noarch 0:6.02-8.el7 perl-HTTP-Message.noarch 0:6.06-6.el7 perl-HTTP-Negotiate.noarch 0:6.01-5.el7 perl-IO-Compress.noarch 0:2.061-2.el7 perl-IO-HTML.noarch 0:1.00-2.el7 perl-IO-Socket-IP.noarch 0:0.21-5.el7 perl-IO-Socket-SSL.noarch 0:1.94-7.el7 perl-LWP-MediaTypes.noarch 0:6.02-2.el7 perl-Mozilla-CA.noarch 0:20130114-5.el7 perl-Net-HTTP.noarch 0:6.06-2.el7 perl-Net-LibIDN.x86_64 0:0.12-15.el7 perl-Net-SSLeay.x86_64 0:1.55-6.el7 perl-Sys-Syscall.noarch 0:0.25-10.el7 perl-TimeDate.noarch 1:2.30-2.el7 perl-URI.noarch 0:1.60-9.el7 perl-WWW-RobotRules.noarch 0:6.02-5.el7 perl-libwww-perl.noarch 0:6.05-2.el7 Complete! [root@node03 ~]#
複製node01上的mogstored配置檔案到node03上，然後在node03上建立對應的目錄並修改屬主和屬組為mogilefs，然後啟動mogstored
提示：在node03上沒有安裝mogilefs-server這個包，所以在對應系統上沒有mogilefs這個賬號，我們需要手動建立這個賬號；
啟動mogstored
到此三個節點的環境配置，以及服務都已經正常啟動了；
四、使用mogadm管理mogilefs
把三個storage節點註冊到tracker上
提示：如果叢集各節點沒有做hosts主機名解析,那麼對應的的節點就需要寫清楚ip地址即可，如果埠不是7500，我們還需要用--port去指定埠；
檢查叢集
提示：在tracker節點監聽在本機所有地址的7001，執行上述命令不需要加--trackers選項來指明tracker是誰，它預設會去找本機的7001；否則執行上述命令需要加上--trackers來指定tracker的地址和埠；
在非tracker節點執行檢查叢集的命令
建立devices
分別在3個mogstored節點的docroot說指定的目錄下建立以dev+數字的目錄，並將屬主和屬組修改成mogilefs
[root@node03 ~]# mkdir /data/mogilefs/dev3 [root@node03 ~]# chown -R mogilefs.mogilefs /data/mogilefs/dev3 [root@node03 ~]# ll -d /data/mogilefs/dev3 drwxr-xr-x 3 mogilefs mogilefs 37 Sep 16 21:02 /data/mogilefs/dev3 [root@node03 ~]#
新增device
檢視device
提示：如果在指定節點的docroot所指定的目錄下沒有建立dev+裝置編號的目錄，上面是看不到對應裝置的空間大小的；
建立domain
建立class
好了，到此我們就可以上傳檔案了
上傳檔案，把/etc/passwd上傳到files域中，使用html class
提示：mogupload這個工具是mogilefs的一個上傳檔案的工具，其中--domain用於指定域，就是把檔案放到那個名稱空間中；--class用於指定使用那個class策略來同步檔案，預設不指定使用default；--key用於指定客戶端訪問檔案的路徑；--file用於指定要上傳的檔案路徑；
檢視上傳檔案的資訊
提示：檢視上傳檔案的資訊需要用mogfileinfo這個工具，我們只需要指定trancker，domain以及檔案的key就可以看到，該檔案的class，devcount,domain,fid,key,length,以及訪問http地址；這裡需要說明一下，我們剛才在建立files域中的html這個class 我們指定了mindevcount=3，表示只要使用這個class上傳檔案，檔案至少有3個副本；而我們現在看到的只有一個，這是因為在centos7上，Sys-syscall這個包的版本過高，導致mogilefs異常；
降級Sys-Syscall，解決centos7上安裝mogilefs 後端mogstored 同步問題
安裝perl編譯環境
[root@node01 ~]# yum -y install make gcc unzip perl-DBD-MySQL perl perl-CPAN perl-YAML perl-Time-HiRes Loaded plugins: fastestmirror base | 3.6 kB 00:00:00 epel | 4.7 kB 00:00:00 extras | 2.9 kB 00:00:00 mariadb | 2.9 kB 00:00:00 updates | 2.9 kB 00:00:00 (1/2): epel/x86_64/updateinfo | 1.0 MB 00:00:00 (2/2): epel/x86_64/primary_db | 6.9 MB 00:00:00 Loading mirror speeds from cached hostfile * base: mirrors.aliyun.com * extras: mirrors.aliyun.com * updates: mirrors.aliyun.com Package gcc-4.8.5-39.el7.x86_64 already installed and latest version Package unzip-6.0-21.el7.x86_64 already installed and latest version Package perl-DBD-MySQL-4.023-6.el7.x86_64 already installed and latest version Package 4:perl-5.16.3-295.el7.x86_64 already installed and latest version Package 4:perl-Time-HiRes-1.9725-3.el7.x86_64 already installed and latest version Resolving Dependencies --> Running transaction check ---> Package make.x86_64 1:3.82-23.el7 will be updated ---> Package make.x86_64 1:3.82-24.el7 will be an update ---> Package perl-CPAN.noarch 0:1.9800-295.el7 will be installed --> Processing Dependency: perl(local::lib) for package: perl-CPAN-1.9800-295.el7.noarch ……省略部分內容…… Installed: perl-CPAN.noarch 0:1.9800-295.el7 perl-YAML.noarch 0:0.84-5.el7 Dependency Installed: gdbm-devel.x86_64 0:1.10-8.el7 libdb-devel.x86_64 0:5.3.21-25.el7 perl-Digest-SHA.x86_64 1:5.85-4.el7 perl-ExtUtils-Install.noarch 0:1.58-295.el7 perl-ExtUtils-MakeMaker.noarch 0:6.68-3.el7 perl-ExtUtils-Manifest.noarch 0:1.61-244.el7 perl-ExtUtils-ParseXS.noarch 1:3.18-3.el7 perl-Test-Harness.noarch 0:3.28-3.el7 perl-devel.x86_64 4:5.16.3-295.el7 perl-local-lib.noarch 0:1.008010-4.el7 pyparsing.noarch 0:1.5.6-9.el7 systemtap-sdt-devel.x86_64 0:4.0-11.el7 Updated: make.x86_64 1:3.82-24.el7 Dependency Updated: libdb.x86_64 0:5.3.21-25.el7 libdb-utils.x86_64 0:5.3.21-25.el7 Complete! [root@node01 ~]#
提示：在mogstored節點上都要安裝好perl環境依賴包；然後按照下面的步驟重新編譯安裝Sys-Syscall；然後重啟mogstored即可生效；
下載低版本的Sys-Syscall原始碼包，並解壓，進入到原始碼目錄
編譯安裝Sys-Syscall
驗證
重啟mogstored，mogilefsd
提示：所有storage節點都有重新編譯安裝Sys-Syscall，然後重啟mogstored；對應的tracker節點也需要重啟mogilefsd；
上傳檔案看看對應的副本數量是否還是隻有一個？
提示：可以看到對應的devcount和我們之前指定的class上的mindevcount就一樣了；
驗證：訪問指定上述任一一個地址，看看訪問到的檔案內容是否是我們上傳到檔案內容？
提示：可以看到我們上傳到檔案內容在tracker給我們返回的地址上能夠訪問到，並且裡面的內容也是一樣的；
驗證：停掉node02上的mogstored，看看對應的tracker是否能夠及時發現對應的storage不線上了呢？
提示：可以看到當node02上的mogstored掛掉以後，在檢查叢集，tracker就能及時的發現對應的裝置不線上了，這個有點類似nginx對後端web伺服器做健康狀態監測的感覺；也就說tracker在監視著註冊在tracker上的storage節點；
到此，mogilefs在centos7上的安裝部署，以及使用mogadm工具管理叢集就完成了，對於mogilefs通常我們上傳檔案是通過業務程式呼叫api實現，mogupload只是一個測試工具；對於檢視檔案，通常使用者也不會使用mogfieinfo返回的連結去檢視檔案，通常會使用一個web反向代理（nginx）去代理使用者的訪問；比如使用者要檢視/images/test.jpg，而在tracker返回的對應資源的連結中，一定不是/images/test.jpg，這個時候我們就需要用到web反向代理，讓其反向代理能夠幫助我們去tracker取到對應uri的資源連結，從而幫助我們去storage節點獲取到對應的檔案，然後再返回給使用者；最後還是建議，如果用到mogilefs，最好跑在centos6上，centos7系統環境較高，有些依賴可能成問題；在centos6上，我測試過，完全沒有上述的問題；我們只需要安裝perl-IO-AIO這一個依賴包就可以解決mogstored啟動問題；centos6上需要用到的rpm包可以去我的網盤下載http://www.qiuhom.com/d/47e8f7ac56a445f1bfc6/；