Python中的greelet模組的執行緒安全問題 （ by quqi99 )

Python中的greelet模組的執行緒安全問題 （ by quqi99 )

作者：張華  發表於：2013-06-17
版權宣告：可以任意轉載，轉載時請務必以超連結形式標明文章原始出處和作者資訊及本版權宣告

( http://blog.csdn.net/quqi99 )

         最近遇到一件很有趣事情，FVT team在對openstack進行壓力測試時，偶爾的qunatum網路這塊會拋異常，從日誌看很無法理喻。經過長時間的摸索，找到的根源如下，見openstack社群的一個patch, https://review.openstack.org/#/c/23198/10/nova/network/quantumv2/__init__.py，這個patch做了這樣一件事，將new_client= client.Client(**params) （是httplib2.Http一個子類）也就是一個socket物件放在快取中在多個greenthread間共享了。你看完下面文章就知道是怎麼一回事了，摘自：http://blog.eventlet.net/2010/03/18/safety/

         我的想法，如果只是單純的nova boot可能還沒事，因為那是一個個程式。但像$nova/nova/compute/manager.py裡的_heal_instance_info_cache之類的periodic_task可是一個單獨的greenlet，那麼它和正常的nova boot為虛擬分配網路時所用的greenlet就有可能剛才執行到下面的情況。

        所謂greenlet實切上是使用者態的NIO非阻塞執行緒，使用者態說明它不是由作業系統核心來切換而是由python虛擬機器來切換的，一個greenthread就是一個死迴圈，哪個多執行緒的任務的IO準備好了就先處理哪個，這個任務的IO阻塞了它不會等，繼續做其他IO準備好了的任務。像下面例子中的第三種greenthread池的情況，一個池內的多個greentthread同一時間也只能有一個greenthread在執行。

         出現這個問題的根在於python語言對socket這些物件沒有做執行緒同步，這從另一個角度也就說明了greenthread效能高效的原因，socket本來就不應該被共享。執行緒之間可以通過共享物件本身的同步來避免競爭，對於比執行緒更小的greenthread的設計哲學本來就是自己擁有自己的資料結構，而不是去共享。這點有點類似於java中的ThreadLocal物件，一個Thread可以擁有自己的區域性資料結構(2013.11.13日更新：關於區域性資料結構，一個patch https://review.openstack.org/#/c/56075/　想做這件事）。在https://review.openstack.org/#/c/33555/ 這裡我和Chris有一個討論。

       2013.11.25更新，neutron這塊的程式碼後仿造java，在每個greenthread的local裡快取socket，但是又出現了上面的錯誤，這個patch （https://review.openstack.org/#/c/57509/2/nova/openstack/common/local.py ）將greenthread改成了普通的thread物件從而解決了問題。我的理解是 (不一定對)，greenthread底層使用的httplib2庫可能會存在前一個請求沒處理完又接受第二個請求的問題，由於socket這時對同一個greenthread是共享的，但socket本身由於使用的是green socket沒有像java的synchronized之類的同步機制，這樣有可能會出問題。改成普通的thread剛好可以利用語言級的socket自由的同步從而解決了問題。

總結一下：
httplib2.Http不應該在greenthread之間共享；可以每個greenthread一個httplib2.Http例項；也可以使用eventlet.pools.Pool機制還構建httplib2.Http例項池在不同greenthread之間作一定程度共享，pool會保證 一個httplib2.HTTP例項在服務完一個greenthread之後再被共他greenthread例項共享。

同樣類似的，還有這個問題， rados.Rados在不同的greenthread之間共享出了問題，這個patch（https://review.openstack.org/#/c/175555/）將它改成用tpool.Proxy來構建rados.Rados例項池的方法在不同的greenthread之間共享，但是rados.Rados這個例項來自python-rbd，它本身又會spawn thread去連線rados, 所以之前的改法造成了迴歸問題，見https://review.openstack.org/#/c/197710/。這樣又回到了不同的greenthrad共享rados.Rados一個例項，rados.Rados例項再去使用native python thread的同步功能，這會同時block掉這些greenthread。

One of the simple user errors that keeps on cropping up is accidentally having multiple greenthreads reading from the same socket at the same time.  It’s a simple thing to accidentally do; just create a shared resource that contains a socket and spawn at least two greenthreads to use it:

 import eventlet
 httplib2 = eventlet.import_patched('httplib2')
 shared_resource = httplib2.Http()
 def get_url():
     resp, content = shared_resource.request("http://eventlet.net")
     return content
 p = eventlet.GreenPile()
 p.spawn(get_url)
 p.spawn(get_url)
 results = list(p)
 assert results[0] == results[1]

Running this with Eventlet 0.9.7 results in an httplib.IncompleteRead exception being raised. It’s because both calls to get_url are divvying up the data from the socket between them, and neither is getting the full picture.  The IncompleteRead error is pretty hard to debug — you’ll have no idea why it’s doing that, and you’ll be frustrated.

What’s new in the tip of Eventlet’s trunk is that Eventlet itself will warn you with a clear error message when you try to do this. If you run the above code with development Eventlet (see sidebar for instructions on how to get it) you now get this error instead:

RuntimeError: Second simultaneous read on fileno 3 detected.  Unless
 you really know what you're doing, make sure that only one greenthread
 can read any particular socket.  Consider using a pools.Pool. If you do know
 what you're doing and want to disable this error, call 
 eventlet.debug.hub_multiple_reader_prevention(False)

Cool, huh? A little clearer about what exactly is going wrong here. And if you really want to do multiple readers or multiple writers on the same socket simultaneously, there’s a way to disable the protection.

Of course, the fix for this particular toy example is to have a single instance of Http() for every greenthread:

 import eventlet
 httplib2 = eventlet.import_patched('httplib2')
 def get_url():
     resp, content = httplib2.Http().request("http://eventlet.net")
     return content
 p = eventlet.GreenPile()
 p.spawn(get_url)
 p.spawn(get_url)
 results = list(p)
 assert results[0] == results[1]

But you probably created that shared_resource because you wanted to reuse Http() instances between requests. So you need some other way to sharing connections. This is what pools.Pool objects are for! Use them like this:

 from __future__ import with_statement
 import eventlet
 from eventlet import pools
 httplib2 = eventlet.import_patched('httplib2')
 
 httppool = pools.Pool()
 httppool.create = httplib2.Http
 
 def get_url():
     with httppool.item() as http:
         resp, content = http.request("http://eventlet.net")
         return content
 
 p = eventlet.GreenPile()
 p.spawn(get_url)
 p.spawn(get_url)
 results = list(p)
 assert results[0] == results[1]

The Pool class will guarantee that the Http instances are reused if possible, and that only one greenthread can access each at a time. If you’re looking for somewhat more advanced usage of this design pattern, take a look at the source code to Heroshi, a concurrent web crawler written on top of Eventlet