## Teuthology架構
自動化測試框架流程大致如下：

![teuthology](https://seo.s3.didiyunapi.com/路錦博/teuthology.png?x-s3-process=image/resize,p_21)

## 軟體組成
Teuthology 測試框架所使用到的軟體組成如下：

- Jenkins 持續整合工具
- teuthology Ceph 測試套件
- shaman 查詢提供軟體包的 chacra 節點，水平擴充套件和排程 chacra 節點
- chacra 提供不同架構二進位制包或檔案的管理 REST API 介面
## Teuthology部署
### 部署paddles/pulpito node
#### 安裝依賴

```
yum install git python-dev python-virtualenv postgresql postgresql-contrib postgresql-server-dev-all supervisor gcc epel-release
```

#### 配置postgresql資料庫
初始化資料庫

```
postgresql-setup initdb
systemctl start postgresql //啟動資料庫
systemctl enable postgresql
```

postgresql安裝並啟動後，會自動生成postgres使用者

```
su - postgres //進入該使用者
-bash-4.2$ psql //進入資料庫
postgres=# \password postgres //更改使用者postgres的密碼
```

//按照輸出的提示輸入密碼即可

建立資料庫例項paddles、shaman

```
postgres=#create database paddles; // 建立名為paddles的資料庫
postgres=#create database shaman; // 建立名為shaman的資料庫
postgres=#create database chacra; // 建立名為chacra的資料庫
```

#### 安裝配置paddles
建立paddles使用者，並克隆paddles專案

```
useradd -m paddles -g root -G root //新建使用者 paddles
su - paddles //切換到paddles使用者下
git clone https://github.com/ceph/paddles.git //克隆該專案
cd paddles
virtualenv ./virtualenv //建立python的沙盒環境
source ./virtualenv/bin/activate //進入沙盒環境
```

修改配置檔案，paddles中有配置檔案的例子，只需要複製一份並修改即可。

```
cp config.py.in config.py
```

需要修改的內容

```
job_log_href_templ= 'http://qa-proxy.ceph.com/teuthology/{run_name}/{job_id}/teuthology.log' # 將qa-proxy.ceph.com/teuthology改成自己對應日誌伺服器的地址
...
sqlalchemy = {
     # You may use SQLite for testing
    'url': ' postgresql://postgres:password@localhost/paddles',
    # When you set up PostreSQL, it will look more like:
    #'url': 'postgresql+psycopg2://USER:PASSWORD@DB_HOST/DB_NAME',
    'echo': True,
    'echo_pool': True,
    'pool_recycle': 3600,
    'encoding': 'utf-8'
}
```

修改/var/lib/pgsql/data/pg_hba.conf,解決初始化paddles環境時的報錯Ident authentication failed。

```
# "local" is for Unix domain socket connections only
local   all             all                                     trust
# IPv4 local connections:
host    all             all             127.0.0.1/32            md5
# IPv6 local connections:
host    all             all             ::1/128                 trust
```

修改完後，儲存並退出。然後安裝相關依賴，初始化環境。

```
$pip install -r requirements.txt //安裝相關依賴
$python setup.py develop // 初始化環境
$pecan populate config.py// 在postgresql建立paddles需要的表
```

配置資料遷移工具

```
$ cp alembic.ini.in alembic.ini
$ vi alembic.ini
    sqlalchemy.url = postgresql://postgres:password@localhost/paddles
$alembic stamp head//觸發遷移工具生效
```

透過supervisord部署paddles服務

```
#cat /etc/supervisord.d/paddles.ini
[program:paddles]
user=paddles
environment=HOME="/home/paddles", USER="paddles"
directory=/home/paddles/paddles
command=/home/paddles/paddles/virtualenv/bin/gunicorn_pecan -c gunicorn_config.py config.py
autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile = /home/paddles/paddles.out.log
stderr_logfile = /home/paddles/paddles.err.log
```

#### 安裝配置pulpito
建立pulpito使用者，切換到pulpito使用者進行部署

```
#useradd -m pulpito -g root -G root
#su - pulpito
$git clone https://github.com/ceph/pulpito.git
$cd pulpito
$virtualenv ./virtualenv
```

修改配置檔案，paddles中有配置檔案的例子，只需要複製一份並修改即可。

```
$cp config.py.in prod.py
$vi prod.py
```

主要的修改部分如下：

```
………
paddles_address = 'http://192.168.122.34:8080' #實際的paddles服務地址
………
'debug': False,
```

修改完後，儲存並退出。然後安裝相關依賴。

```
$source ./virtualenv/bin/activate
$pip install -r requirements.txt //安裝相關依賴
```

透過supervisord部署pulpito服務

```
#cat /etc/supervisord.d/pulpito.ini
[program:pulpito] 
user=pulpito 
directory=/home/pulpito/pulpito 
command=/home/pulpito/pulpito/virtualenv/bin/python run.py 
environment=HOME="/home/pulpito",USER="pulpito" 
autostart=true 
autorestart=true 
redirect_stderr=true 
stdout_logfile = /home/pulpito/pulpito.out.log 
stderr_logfile = /home/pulpito/pulpito.err.log
啟動supervisor
sudo supervisord -c /etc/supervisord.conf
sudo supervisorctl reload
sudo supervisorctl start all//啟動
sudo supervisorctl status//檢視狀態，可以看到paddles 和pulpito的程式啟動了
檢測服務
```

在瀏覽器或者終端，訪問 http://{ip}:8080/和http://{ip}:8081/就可以了，看到如下介面：

![pulpito](https://seo.s3.didiyunapi.com/路錦博/pulpito.jpg?x-s3-process=image/resize,p_21)

### 部署shaman
shaman的部署方式分為兩種一種是手動部署，另一種透過ansible，在此採用的是手動部署。

#### 安裝相關依賴

```
yum install epel-release gcc gcc-c++ glibc-devel make ncurses-devel openssl-devel autoconf java-1.8.0-openjdk-devel git wget wxBase.x86_64 erlang
```

#### 安裝RabbitMQ

```
wget https://www.rabbitmq.com/releases/rabbitmq-server/v3.6.1/rabbitmq-server-3.6.1-1.noarch.rpm 
rpm --import https://www.rabbitmq.com/rabbitmq-signing-key-public.asc 
yum install rabbitmq-server-3.6.1-1.noarch.rpm
systemctl start rabbitmq-server #啟動rabbitmq-server
systemctl enable rabbitmq-server #設定rabbitmq-server開機自啟動
```

#### 部署shaman
建立shaman使用者，並克隆shaman專案，然後建立沙盒環境。

```
useradd -m shaman -g root -G root //新建使用者shaman
su - shaman //切換到shaman使用者下
git clone https://github.com/ceph/shaman.git //克隆該專案
cd shaman
virtualenv ./virtualenv //建立python的沙盒環境
source ./virtualenv/bin/activate //進入沙盒環境
```

安裝一些在requirements.txt中沒有的軟體依賴

```
pip install --upgrade setuptools 
pip install --upgrade pip
pip install "cherrypy>=3.8.0,<9.0.0" 
pip install wsgiserver
```

修改配置檔案config/dev.py。

```
$vim config/dev.py
 
...
    ], 
    'debug': False,
} 
... 
sqlalchemy_w = { 
    #'url': 'sqlite:///dev.db', 
    'url': 'postgresql://postgres:password@localhost/shaman', 
    'echo': True, 
    'echo_pool': True, 
    'pool_recycle': 3600, 
    'encoding': 'utf-8' 
} 
sqlalchemy_ro = { 
    #'url': 'sqlite:///dev.db', 
    'url': 'postgresql://postgres:password@localhost/shaman', 
    'echo': True, 
    'echo_pool': True, 
    'pool_recycle': 3600, 
    'encoding': 'utf-8' 
} 
....
```

修改config/run.py，需要修改的內容為，其中服務IP根據具體改動：

```
simpleapp_wsgi_app = deploy('config/dev.py')
...
server = wsgiserver.CherryPyWSGIServer(('0.0.0.0', 8082), application, server_name='simpleapp')
```

初始化環境

```
$pip install -r requirements.txt //安裝相關依賴
$python setup.py develop // 初始化環境
$pecan populate config/dev.py// 在postgresql建立shaman需要的表
```

配置資料遷移工具

```
$ cp alembic.ini.in alembic.ini
$ vi alembic.ini
    sqlalchemy.url = postgresql://postgres:password@localhost/shaman
$alembic stamp head//觸發遷移工具生效
```

#### 配置supervisord
透過supervisord部署shaman服務

```
#cat /etc/supervisord.d/shaman.ini
[program:shaman] 
user=shaman 
directory=/home/shaman/shaman 
command=/home/shaman/shaman/virtualenv/bin/python config/run.py 
environment=HOME="/home/shaman",USER="shaman" 
autostart=true 
autorestart=true 
redirect_stderr=true 
stdout_logfile = /home/shaman/shaman.out.log 
stderr_logfile = /home/shaman/shaman.err.log
```

啟動shaman服務

```
sudo supervisorctl reread && sudo supervisorctl update shaman && sudo supervisorctl start shaman
```

#### 檢測服務
在瀏覽器或者終端，訪問 http://{ip}:8082/就可以了，看到如下介面：

![shaman](https://seo.s3.didiyunapi.com/路錦博/shaman.png?x-s3-process=image/resize,p_21)

### 部署chacra
chacra的部署較為複雜，採取的是ansible的方式，但是官方提供的部署檔案是針對ubuntu系統的，需要進行改動。

#### 修改playbooks相關檔案
修改deploy/playbooks/examples/deploy_test.yml中對應引數

```
development_server: true
api_user: "admin"
api_key: "secret"
graphite_api_key: '1234-asdf-1234'
callback_url: "http://{ip}:8082/api/repos/" #shaman的地址
callback_user: "admin"
callback_key: "secret"
health_ping_url: "http://{ip}:8082/api/nodes/" #shaman的地址
```

修改deploy/playbooks/roles/common/tasks/letsencrypt.yml

```
#刪除下面幾行
- name: install software-properties-common
  apt:
    name: software-properties-common
    state: latest
    update_cache: yes
  become: true

- name: add certbot PPA
  apt_repository:
    repo: "ppa:certbot/certbot"
  become: true
```

修改deploy/playbooks/roles/common/tasks/rabbitmq.yml

```
#刪除下面幾行
- name: add the rabbitmq official repo
  become: true
  apt_repository:
    repo: "deb http://www.rabbitmq.com/debian/ testing main"
    state: present

- name: add the rabbitmq public key
  become: true
  apt_key:
    url: "https://www.rabbitmq.com/rabbitmq-release-signing-key.asc"
    state: present

- name: update the apt cache
  become: true
  apt:
    update_cache: true
```

修改deploy/playbooks/roles/common/tasks/systemd.yml

```
# 將state由running改為started
service: name=chacra state=started enabled=yes
service: name=chacra-celery state=started enabled=yes
service: name=chacra-celerybeat state=started enabled=yes
```

修改deploy/playbooks/roles/common/templates/systemd/chacra-celery.service.j2

```
# 由於%I在systemd中有特殊的含義，所有在此將%I改為%%I
ExecStart={{ app_home }}/bin/celery multi start 5 -Q:1,2 poll_repos,celery -Q:3-5 build_repos -A async --logfile=/var/log/celery/%n%%I.log
ExecReload={{ app_home }}/bin/celery multi restart 5 -Q:1,2 poll_repos,celery -Q:3-5 build_repos -A async --logfile=/var/log/celery/%n%%I.log
```

#### 部署chacra
使用ansible-playbook部署ansible服務

#### 檢查服務
在瀏覽器或者終端，訪問 http://{ip}:8083/就可以了，看到如下介面：

![chacra](https://seo.s3.didiyunapi.com/路錦博/chacra.png?x-s3-process=image/resize,p_21)

### Teuthology Node
Teuthology部署中官方建議為兩個角色scheduler和worker。

#### 建立/etc/teuthology.yaml
內容如下：

```
lab_domain: didichuxing.com #testnode的域名字尾

archive_base: /home/teuthworker/archive # 儲存日誌資訊的地方

automated_scheduling: False
reserve_machines: 0

queue_host: 127.0.0.1
queue_port: 11300

lock_server: 'http://{ip}:8080' # paddles的地址
results_server: 'http://{ip}:8080' # paddles的地址

results_ui_server: 'http://{ip}:8081' # pulpito的地址

#results_sending_email: 'teuthology',
results_timeout: 43200

# Verify the packagee signatures
check_package_signatures: false

ceph_git_base_url: 'git@git.xiaojukeji.com:chishui/' # gitlab地址
ceph_git_url: 'git@git.xiaojukeji.com:chishui/deph.git'
ceph_qa_suite_git_url: 'git@git.xiaojukeji.com:chishui/deph.git'

use_conserver: False
conserver_master: 'conserver.front.sepia.ceph.com'
conserver_port: 3109

max_job_time: 259200  # 3 days

baseurl_template: 'http://{host}/{proj}-{pkg_type}-{dist}-{arch}-{flavor}/{uri}'
use_shaman: True
shaman_host: '{ip}:8082'

src_base_path: /home/teuthworker/src
teuthology_path: /home/teuthworker/src/teuthology_master

suite_verify_ceph_hash: False
suite_allow_missing_packages: False
```

#### 安裝依賴
#### yum -y install git python-devel python-pip python-virtualenv libevent-devel libvirt-python beanstalkd
啟動beanstackd佇列：

```
systemctl start beanstalkd
systemctl enable beanstalkd # 配置開機自啟動
```

#### 部署teuthology
建立teuthology與teuthworker兩個使用者：

```
useradd -m teuthology -g root -G root
useradd -m teuthworker -g root -G root
```

分別給兩個賬號授予passwordless sudo access許可權：

```
useradd ubuntu
vim /etc/sudoers
#新增下面兩行
teuthology   ALL=(ALL)         NOPASSWD: ALL
teuthworker  ALL=(ALL)         NOPASSWD: ALL
```

#### scheduler

```
su - teuthology
mkdir ~/src
git clone https://github.com/ceph/teuthology.git src/teuthology_master
pushd src/teuthology_master/
./bootstrap
popd
```

#### worker

```
su - teuthworker
mkdir ~/src
git clone https://github.com/ceph/teuthology.git src/teuthology_master
pushd src/teuthology_master/
./bootstrap
popd
mkdir ~/bin
wget -O ~/bin/worker_start https://raw.githubusercontent.com/ceph/teuthology/master/docs/_static/worker_start.sh
echo 'PATH="$HOME/src/teuthology_master/virtualenv/bin:$PATH"' >> ~/.profile
source ~/.profile
mkdir -p ~/archive/worker_logs
worker_start magna 1
```

使用worker_start來啟動一個plana的工作程式

```
worker_start plana 1
```

#### 配置teuthology和teuthworker使用者與test nodes間的免密登入

```
ssh-copy-id -i ~/.ssh/id_rsa.pub ubuntu@{machine_type}00x.example.com
```

### Test Nodes
#### 新增ubuntu使用者
在每個test node上建立ubuntu使用者，併為ubuntu使用者授予passwordless sudo access許可權。

```
useradd ubuntu
passwd ubuntu
New password: 
etype new password: 
vim /etc/sudoers
#新增下面一行
ubuntu All=(ALL) NOPASSWD:ALL
```

#### 配置ssh

```
#vim /etc/ssh/sshd_config
//更改為下面行
#HostKey /etc/ssh/ssh_host_key
# HostKeys for protocol version 2
HostKey /home/ubuntu/.ssh/id_rsa
#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_dsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
#HostKey /etc/ssh/ssh_host_ed25519_key
......
PermitRootLogin yes
......
PubkeyAuthentication yes
......
PasswordAuthentication yes
......
#systemctl restart sshd
```

#### 安裝依賴

```
yum install -y redhat-lsb yum-priorities genisoimage wget
```

### submitting nodes
在paddles中註冊test nodes

```
wget https://raw.githubusercontent.com/ceph/teuthology/master/docs/_static/create_nodes.py
```

修改create_nodes.py

```
paddles_url = 'http://{ip}:8080/nodes/' # paddles的地址

machine_type = 'plana'
lab_domain = 'example.com'
# Don't change the user. It won't work at this time.
user = 'ubuntu'
# We are populating 'plana001' -> 'plana003'
machine_index_range = range(1, 4) # 建立了'plana001' -> 'plana003' 3臺機器
......
```

日誌伺服器部署
新增nginx服務配置

```
server {
  listen 80;
  server_name {ip};
  default_type text/plain;
  access_log /var/log/nginx/log-access.log;
  error_log /var/log/nginx/log-error.log;
  location / {
    autoindex on;
    alias /home/teuthworker/archive/;
  }
}
```

重啟nginx服務

```
systemctl restart nginx
```

修改paddles配置中的job_log_href_templ的值為：

```
job_log_href_templ= 'http://{ip}/{run_name}/{job_id}/teuthology.log'
```
作者：路錦博
- [現在註冊滴滴雲，有機會可得30元無門檻滴滴出行券](https://www.didiyun.com/?channel=14816 )

- [新購雲服務1月5折 3月4.5折 6月低至4折](https://www.didiyun.com/?channel=14816 "新購雲服務1月5折 3月4折 6月低至3折")

- [滴滴雲使者招募，推薦最高返傭50%](https://www.didiyun.com/envoy/e-index.html?channel=14844)

Teuthology平臺部署手冊

相關文章