本文是 理解 WSGI 框架 的下篇，重點介紹 WSGI 框架下一些常用的 python module，並使用這些 module 編寫一個類似 nova-api 裡 WSGI 的簡單樣例，最後分析 nova 是如何使用這些 module 構建其 WSGI 框架。
- eventlet: python 的高併發網路庫
- paste.deploy: 用於發現和配置 WSGI application 和 server 的庫
- routes: 處理 http url mapping 的庫
Eventlet
Eventlet 是一個基於協程的 Python 高併發網路庫，和上篇文章所用的 wsgiref 相比，它具有更強大的功能和更好的效能，OpenStack 大量的使用 eventlet 以提供併發能力。它具有以下特點：
- 使用 epoll、kqueue 或 libevent 等 I/O 複用機制，對於非阻塞 I/O 具有良好的效能
- 基於協程(Coroutines)，和程式、執行緒相比，其切換開銷更小，具有更高的效能
- 簡單易用，特別是支援採用同步的方式編寫非同步的程式碼
Eventlet.wsgi
Eventlet WSGI 簡單易用，數行程式碼即可實現一個基於事件驅動的 WSGI server。本例主要使用了 eventlet.wsgi.server 函式：
|
1 2 3 4 5 6 7 8 9 |
eventlet.wsgi.server(sock, site, log=None, environ=None, max_size=None, max_http_version='HTTP/1.1', protocol=eventlet.wsgi.HttpProtocol, server_event=None, minimum_chunk_size=None, log_x_forwarded_for=True, custom_pool=None, keepalive=True, log_output=True, log_format='%(client_ip)s...', url_length_limit=8192, debug=True, socket_timeout=None, capitalize_response_headers=True) |
該函式的引數眾多，重點介紹以下 2 個引數：
- sock: 即 TCP Socket，通常由 eventlet.listen(‘IP’, PORT) 實現
- site: WSGI 的 application
回顧上篇文章內容，本例採用 callable 的 instance 實現一個 WSGI application，利用 eventlet.server 構建 WSGI server，如下：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
import eventlet from eventlet import wsgi class AnimalApplication(object): def __init__(self): pass def __call__(self, environ, start_response): start_response('200 OK', [('Content-Type', 'text/plain')]) return ['This is a animal applicaltion!rn'] if '__main__' == __name__: application = AnimalApplication() wsgi.server(eventlet.listen(('', 8080)), application) |
Eventlet.spawn
Eventlet.spawn 基於 greenthread，它通過建立一個協程來執行函式，從而提供併發處理能力。
|
1 2 |
eventlet.spawn(func, *args, **kw) |
加入該函式後，樣例如下：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
import eventlet from eventlet import wsgi class AnimalApplication(object): def __init__(self): pass def __call__(self, environ, start_response): start_response('200 OK', [('Content-Type', 'text/plain')]) return ['This is a animal applicaltion!rn'] if '__main__' == __name__: application = AnimalApplication() server = eventlet.spawn(wsgi.server, eventlet.listen(('', 8080)), application) server.wait() |
Paste.deploy
Paste.deploy 是一個使用者發現和配置 WSGI server 和 application 的 python 庫，它定義簡潔的 loadapp 函式，用於從配置檔案或者 python egg 中載入 WSGI 應用，它僅關注 application 的入口，不關心 application 的內部細節。
Paste.deploy 通常要求 application 實現一個 factory 的類方法，如下：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
import eventlet from eventlet import wsgi from paste.deploy import loadapp class AnimalApplication(object): def __init__(self): pass def __call__(self, environ, start_response): start_response('200 OK', [('Content-Type', 'text/plain')]) return ['This is a animal applicaltion!rn'] @classmethod def factory(cls, global_conf, **kwargs): return cls() if '__main__' == __name__: application = loadapp('config:/path/to/animal.ini') server = eventlet.spawn(wsgi.server, eventlet.listen(('', 8080)), application) server.wait() |
配置檔案的規則請參考官網介紹，相應的配置檔案如下，其中 app:animal 給出了 application 的入口，pipeline:animal_pipeline 用於配置 WSGI middleware。
|
1 2 3 4 5 6 7 8 9 10 |
[composite:main] use = egg:Paste#urlmap / = animal_pipeline [pipeline:animal_pipeline] pipeline = animal [app:animal] paste.app_factory = animal:AnimalApplication.factory |
現在我們新增一個 IPBlackMiddleware，用於限制某些 IP：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
class IPBlacklistMiddleware(object): def __init__(self, application): self.application = application def __call__(self, environ, start_response): ip_addr = environ.get('HTTP_HOST').split(':')[0] if ip_addr not in ('127.0.0.1'): start_response('403 Forbidden', [('Content-Type', 'text/plain')]) return ['Forbidden'] return self.application(environ, start_response) @classmethod def factory(cls, global_conf, **local_conf): def _factory(application): return cls(application) return _factory |
相關配置檔案：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
[composite:main] use = egg:Paste#urlmap / = animal_pipeline [pipeline:animal_pipeline] pipeline = ip_blacklist animal [filter:ip_blacklist] paste.filter_factory = animal:IPBlacklistMiddleware.factory [app:animal] paste.app_factory = animal:AnimalApplication.factory |
Route
Routes 是基於 ruby on rails 的 routes system 開發的 python 庫，它根據 http url 把請求對映到具體的方法，routes 簡單易用，可方便的構建 Restful 風格的 url。
本例增加 CatController 和 DogController，對於 url_path 為 cats 的 HTTP 請求，對映到 CatController 處理，對於 url_path 為 dogs 的 HTTP 請求，對映到 DogController 處理，最終樣例如下：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 |
import eventlet from eventlet import wsgi from paste.deploy import loadapp import routes import routes.middleware as middleware import webob.dec import webob.exc class Resource(object): def __init__(self, controller): self.controller = controller() @webob.dec.wsgify def __call__(self, req): match = req.environ['wsgiorg.routing_args'][1] action = match['action'] if hasattr(self.controller, action): method = getattr(self.controller, action) return method(req) return webob.exc.HTTPNotFound() class CatController(object): def index(self, req): return 'List cats.' def create(self, req): return 'create cat.' def delete(self, req): return 'delete cat.' def update(self, req): return 'update cat.' class DogController(object): def index(self, req): return 'List dogs.' def create(self, req): return 'create dog.' def delete(self, req): return 'delete dog.' def update(self, req): return 'update dog.' class AnimalApplication(object): def __init__(self): self.mapper = routes.Mapper() self.mapper.resource('cat', 'cats', controller=Resource(CatController)) self.mapper.resource('dog', 'dogs', controller=Resource(DogController)) self.router = middleware.RoutesMiddleware(self.dispatch, self.mapper) @webob.dec.wsgify def __call__(self, req): return self.router @classmethod def factory(cls, global_conf, **local_conf): return cls() @staticmethod @webob.dec.wsgify def dispatch(req): match = req.environ['wsgiorg.routing_args'][1] return match['controller'] if match else webob.exc.HTTPNotFound() class IPBlacklistMiddleware(object): def __init__(self, application): self.application = application def __call__(self, environ, start_response): ip_addr = environ.get('HTTP_HOST').split(':')[0] if ip_addr not in ('127.0.0.1'): start_response('403 Forbidden', [('Content-Type', 'text/plain')]) return ['Forbidden'] return self.application(environ, start_response) @classmethod def factory(cls, global_conf, **local_conf): def _factory(application): return cls(application) return _factory if '__main__' == __name__: application = loadapp('config:/path/to/animal.ini') server = eventlet.spawn(wsgi.server, eventlet.listen(('', 8080)), application) server.wait() |
測試如下：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
$ curl 127.0.0.1:8080/test The resource could not be found. $ curl 127.0.0.1:8080/cats List cats. $ curl -X POST 127.0.0.1:8080/cats create cat. $ curl -X PUT 127.0.0.1:8080/cats/kitty update cat. $ curl -X DELETE 127.0.0.1:8080/cats/kitty delete cat. $ curl 127.0.0.1:8080/dogs List dogs. $ curl -X DELETE 127.0.0.1:8080/dogs/wangcai delete dog. |
WSGI In Nova-api
WSGI Server
Nova-api(nova/cmd/api.py) 服務啟動時，初始化 nova/wsgi.py 中的類 Server，建立了 socket 監聽 IP 和埠，再由 eventlet.spawn 和 eventlet.wsgi.server 建立 WSGI server：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 |
class Server(object): """Server class to manage a WSGI server, serving a WSGI application.""" def __init__(self, name, app, host='0.0.0.0', port=0, pool_size=None, protocol=eventlet.wsgi.HttpProtocol, backlog=128, use_ssl=False, max_url_len=None): """Initialize, but do not start, a WSGI server.""" self.name = name self.app = app self._server = None self._protocol = protocol self._pool = eventlet.GreenPool(pool_size or self.default_pool_size) self._logger = logging.getLogger("nova.%s.wsgi.server" % self.name) self._wsgi_logger = logging.WritableLogger(self._logger) if backlog < 1: raise exception.InvalidInput( reason='The backlog must be more than 1') bind_addr = (host, port) # 建立 socket，監聽 IP 和埠 self._socket = eventlet.listen(bind_addr, family, backlog=backlog) def start(self): """Start serving a WSGI application. :returns: None """ # 構建所需引數 wsgi_kwargs = { 'func': eventlet.wsgi.server, 'sock': self._socket, 'site': self.app, 'protocol': self._protocol, 'custom_pool': self._pool, 'log': self._wsgi_logger, 'log_format': CONF.wsgi_log_format } if self._max_url_len: wsgi_kwargs['url_length_limit'] = self._max_url_len # 由 eventlet.sawn 啟動 server self._server = eventlet.spawn(**wsgi_kwargs) |
Application Side & Middleware
Application 的載入由 nova/wsgi.py 的類 Loader 完成，Loader 的 load_app 方法呼叫了 paste.deploy.loadapp 載入了 WSGI 的配置檔案 /etc/nova/api-paste.ini：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
class Loader(object): """Used to load WSGI applications from paste configurations.""" def __init__(self, config_path=None): # 獲取 WSGI 配置檔案的路徑 self.config_path = config_path or CONF.api_paste_config def load_app(self, name): # paste.deploy 讀取配置檔案並載入該配置 return paste.deploy.loadapp("config:%s" % self.config_path, name=name) |
配置檔案 api-paste.ini 如下所示，我們通常使用 v2 API，即 composite:openstack_compute_api_v2，也通常使用 keystone 做認證，即 keystone = faultwrap sizelimit authtoken keystonecontext ratelimit osapi_compute_app_v2，從 fautlwrap 到 ratelimit 均是 middleware，我們也可根據需求增加某些 middleware。
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
[composite:osapi_compute] use = call:nova.api.openstack.urlmap:urlmap_factory /v2: openstack_compute_api_v2 /v3: openstack_compute_api_v3 [composite:openstack_compute_api_v2] use = call:nova.api.auth:pipeline_factory noauth = faultwrap sizelimit noauth ratelimit osapi_compute_app_v2 keystone = faultwrap sizelimit authtoken keystonecontext ratelimit osapi_compute_app_v2 keystone_nolimit = faultwrap sizelimit authtoken keystonecontext osapi_compute_app_v2 [composite:openstack_compute_api_v3] ... [filter:keystonecontext] paste.filter_factory = nova.api.auth:NovaKeystoneContext.factory [filter:authtoken] paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory [app:osapi_compute_app_v2] paste.app_factory = nova.api.openstack.compute:APIRouter.factory [app:osapi_compute_app_v3] paste.app_factory = nova.api.openstack.compute:APIRouterV3.factory |
Routes
在 nova/api/openstack/compute/__init__.py 定義了類 APIRouter，它定義了各種 url 和 controller 之間的對映關係，最終由 nova/wsgi.py 的類 Router 載入這些 mapper。
nova/wsgi.py 中的 Router class 如下：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
class Router(object): """WSGI middleware that maps incoming requests to WSGI apps.""" def __init__(self, mapper): """Create a router for the given routes.Mapper.""" self.map = mapper self._router = routes.middleware.RoutesMiddleware(self._dispatch, self.map) @webob.dec.wsgify(RequestClass=Request) def __call__(self, req): """Route the incoming request to a controller based on self.map. If no match, return a 404. """ return self._router @staticmethod @webob.dec.wsgify(RequestClass=Request) def _dispatch(req): """Dispatch the request to the appropriate controller.""" match = req.environ['wsgiorg.routing_args'][1] if not match: return webob.exc.HTTPNotFound() app = match['controller'] return app |