前言
原文發在這裡的 Python concurrent.future 使用教程及原始碼初剖
垃圾話
很久沒寫部落格了，想了想不能再划水，於是給自己定了一個目標，寫點 concurrent.future 的內容，於是這篇文章就是來聊聊 Python 3.2 中新增的 concurrent.future 模組。
正文
Python 的非同步處理
有一個 Python 開發工程師小明，在面試過程中，突然接到這樣一個需求：去請求幾個網站，拿到他們的資料，小明定睛一想，簡單啊，噼裡啪啦，他寫了如下的程式碼
import multiprocessing
import time
def request_url(query_url: str):
time.sleep(3) # 請求處理邏輯
if __name__ == `__main__`:
url_list = ["abc.com", "xyz.com"]
task_list = [multiprocessing.Process(target=request_url, args=(url,)) for url in url_list]
[task.start() for task in task_list]
[task.join() for task in task_list]複製程式碼Easy, 好了，現在新的需求來了，我們想獲取每一個請求結果，怎麼辦?小明想了想，又寫出如下的程式碼
import multiprocessing
import time
def request_url(query_url: str, result_dict: dict):
time.sleep(3) # 請求處理邏輯
result_dict[query_url] = {} # 返回結果
if __name__ == `__main__`:
process_manager = multiprocessing.Manager()
result_dict = process_manager.dict()
url_list = ["abc.com", "xyz.com"]
task_list = [multiprocessing.Process(target=request_url, args=(url, result_dict)) for url in url_list]
[task.start() for task in task_list]
[task.join() for task in task_list]
print(result_dict)複製程式碼好了，面試官說，恩看起來不錯，好了，我再改改題目，首先，我們不能阻塞主程式，主程式需要根據任務當前的狀態（結束/未結束）來及時的獲取對應的結果，怎麼改？，小明想了想，要不，我們直接用訊號量，讓任務完成後，向父程式傳送一個訊號量？然後直接暴力出奇跡？還有更簡單的方法麼？貌似沒了？最後面試官心理說了一句 naive ，臉上笑而不語，讓小明回去慢慢等訊息。
從小明的窘境，我們可以看出一個這樣的問題，就是我們最常用的 multiprocessing 或者是 threding 兩個模組，對於我們想實現非同步任務的場景來說，其實略有一點不友好的，我們往往需要做一些額外的工作，才能比較乾淨的實現一些非同步的需求。為了解決這樣的窘境，09 年 10 月，Brian Quinlan 先生提出了 PEP 3148 ，在這個提案中，他提出將我們常用的 multiprocessing 和 threding 模組進行進一步封裝，達成較好的支援非同步操作的目的。最終這個提案在 Python 3.2 中被引入。也就是我們今天要聊聊的 concurrent.future 。
Future 模式
在我們正式開始聊新模組之前，我們需要去了解關於 Future 模式的相關姿勢
首先 Future 模式，是什麼？
Future 其實是生產-消費者模型的一種擴充套件，在生產-消費者模型中，生產者不關心消費者什麼時候處理完資料，也不關心消費者處理的結果。比如我們經常寫出如下的程式碼
import multiprocessing, Queue
import os
import time
from multiprocessing import Process
from time import sleep
from random import randint
class Producer(multiprocessing.Process):
def __init__(self, queue):
multiprocessing.Process.__init__(self)
self.queue = queue
def run(self):
while True:
self.queue.put(`one product`)
print(multiprocessing.current_process().name + str(os.getpid()) + ` produced one product, the no of queue now is: %d` %self.queue.qsize())
sleep(randint(1, 3))
class Consumer(multiprocessing.Process):
def __init__(self, queue):
multiprocessing.Process.__init__(self)
self.queue = queue
def run(self):
while True:
d = self.queue.get(1)
if d != None:
print(multiprocessing.current_process().name + str(os.getpid()) + ` consumed %s, the no of queue now is: %d` %(d,self.queue.qsize()))
sleep(randint(1, 4))
continue
else:
break
#create queue
queue = multiprocessing.Queue(40)
if __name__ == "__main__":
print(`Excited!")
#create processes
processed = []
for i in range(3):
processed.append(Producer(queue))
processed.append(Consumer(queue))
#start processes
for i in range(len(processed)):
processed[i].start()
#join processes
for i in range(len(processed)):
processed[i].join()複製程式碼這就是生產-消費者模型的一個簡單的實現，我們利用一個 multiprocessing 中的 Queue 來作為通訊渠道，我們的生產者負責往佇列中傳入資料，消費者負責從佇列中獲取資料並處理。不過就如同上面所說的一樣，在這種模式中，生產者並不關心消費者何時處理完資料，也不關心處理的結果。而在 Future 中，我們可以讓生產者等待訊息處理完成，如果需要的話，我們還可以獲取相關的計算結果。
比如，大家可以看看下面這樣一段 Java 程式碼
package concurrent;
import java.util.concurrent.Callable;
public class DataProcessThread implements Callable<String> {
@Override
public String call() throws Exception {
// TODO Auto-generated method stub
Thread.sleep(10000);//模擬資料處理
return "資料返回";
}
}複製程式碼這是我們負責處理資料的程式碼。
package concurrent;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.FutureTask;
public class MainThread {
public static void main(String[] args) throws InterruptedException,
ExecutionException {
// TODO Auto-generated method stub
DataProcessThread dataProcessThread = new DataProcessThread();
FutureTask<String> future = new FutureTask<String>(dataProcessThread);
ExecutorService executor = Executors.newFixedThreadPool(1);
executor.submit(future);
Thread.sleep(10000);//模擬繼續處理自身其他業務
while (true) {
if (future.isDone()) {
System.out.println(future.get());
break;
}
}
executor.shutdown();
}
}複製程式碼這是我們主執行緒，大家可以看到，我們可以很方便的獲取資料處理任務的狀態。同時獲取相關的結果。
Python 中的 concurrent.futures
前面說了，在 Python 3.2 以後，concurrent.futures 是內建的模組，我們可以直接使用
Note: 如果你需要在 Python 2.7 中使用 concurrent.futures , 那麼請用 pip 進行安裝，
pip install futures
好了，準備就緒後，我們來看看怎麼使用這個東西呢
from concurrent.futures import ProcessPoolExecutor
import time
def return_future_result(message):
time.sleep(2)
return message
if __name__ == `__main__`:
pool = ProcessPoolExecutor(max_workers=2) # 建立一個最大可容納2個task的程式池
future1 = pool.submit(return_future_result, ("hello")) # 往程式池裡面加入一個task
future2 = pool.submit(return_future_result, ("world")) # 往程式池裡面加入一個task
print(future1.done()) # 判斷task1是否結束
time.sleep(3)
print(future2.done()) # 判斷task2是否結束
print(future1.result()) # 檢視task1返回的結果
print(future2.result()) # 檢視task2返回的結果複製程式碼首先 from concurrent.futures import ProcessPoolExecutor 從 concurrent.futures 引入 ProcessPoolExecutor 作為我們的程式池，處理我們後面的資料。(在 concurrent.futures 中，為我們提供了兩種 Executor ，一種是我們現在用的 ProcessPoolExecutor, 一種是 ThreadPoolExecutor 他們對外暴露的方法一致，大家可以根據自己的實際需求選用。)
緊接著，初始化一個最大容量為 2 的程式池。然後我們呼叫程式池中的 submit 方法提交一個任務。好了有意思的點來了，我們在呼叫 submit 方法後，得到了一個特殊的變數，這個變數是 Future 類的例項，代表著一個在未來完成的操作。換句話說，當 submit 返回 Future 例項的時候，我們的任務可能還沒有完成，我們可以通過呼叫 Future 例項中的 done 方法來獲取當前任務的執行狀態，如果任務結束後，我們可以通過 result 方法來獲取返回的結果。如果在執行後續的邏輯時，我們因為一些原因想要取消任務時，我們可以通過呼叫 cancel 方法來取消當前的任務。
現在新的問題來了，我們如果想要提交很多個任務應該怎麼辦呢？concurrent.future 為我們提供了 map 方法來方便我們批量新增任務。
import concurrent.futures
import requests
task_url = [(`http://www.baidu.com`, 40), (`http://example.com/`, 40), (`https://www.github.com/`, 40)]
def load_url(params: tuple):
return requests.get(params[0], timeout=params[1]).text
if __name__ == `__main__`:
with concurrent.futures.ProcessPoolExecutor(max_workers=3) as executor:
for url, data in zip(task_url, executor.map(load_url, task_url)):
print(`%r page is %d bytes` % (url, len(data)))複製程式碼恩，concurrent.future 中執行緒/程式池所提供的 map 方法和標準庫中的 map 函式使用方法一樣。
剖一下 concurrent.futures
前面講了怎麼使用 concurrent.futures 後，我們都比較好奇，concurrent.futures 是怎麼實現 Future 模式的。裡面是怎麼將任務和結果進行關聯的。我們現在開始從 submit 方法著手來簡單看一下 ProcessPoolExecutor 的實現。
首先，在初始化 ProcessPoolExecutor 時，它的 __init__ 方法中進行了一些關鍵變數的初始化操作。
class ProcessPoolExecutor(_base.Executor):
def __init__(self, max_workers=None):
"""Initializes a new ProcessPoolExecutor instance.
Args:
max_workers: The maximum number of processes that can be used to
execute the given calls. If None or not given then as many
worker processes will be created as the machine has processors.
"""
_check_system_limits()
if max_workers is None:
self._max_workers = os.cpu_count() or 1
else:
if max_workers <= 0:
raise ValueError("max_workers must be greater than 0")
self._max_workers = max_workers
# Make the call queue slightly larger than the number of processes to
# prevent the worker processes from idling. But don`t make it too big
# because futures in the call queue cannot be cancelled.
self._call_queue = multiprocessing.Queue(self._max_workers +
EXTRA_QUEUED_CALLS)
# Killed worker processes can produce spurious "broken pipe"
# tracebacks in the queue`s own worker thread. But we detect killed
# processes anyway, so silence the tracebacks.
self._call_queue._ignore_epipe = True
self._result_queue = SimpleQueue()
self._work_ids = queue.Queue()
self._queue_management_thread = None
# Map of pids to processes
self._processes = {}
# Shutdown is a two-step process.
self._shutdown_thread = False
self._shutdown_lock = threading.Lock()
self._broken = False
self._queue_count = 0
self._pending_work_items = {}複製程式碼好了，我們來看看我們今天的入口 submit 方法
def submit(self, fn, *args, **kwargs):
with self._shutdown_lock:
if self._broken:
raise BrokenProcessPool(`A child process terminated `
`abruptly, the process pool is not usable anymore`)
if self._shutdown_thread:
raise RuntimeError(`cannot schedule new futures after shutdown`)
f = _base.Future()
w = _WorkItem(f, fn, args, kwargs)
self._pending_work_items[self._queue_count] = w
self._work_ids.put(self._queue_count)
self._queue_count += 1
# Wake up queue management thread
self._result_queue.put(None)
self._start_queue_management_thread()
return f複製程式碼首先，傳入的引數 fn 是我們的處理函式，args 以及 kwargs 是我們要傳遞 fn 函式的引數。在 submit 函式最開始，首先根據 _broken 和 _shutdown_thread 的值來判斷當前程式池中處理程式的狀態以及目前程式池的狀態。如果處理程式突然被銷燬或者程式池已經被關閉，那麼將丟擲異常表明目前不再接受新的 submit 操作。
如果前面狀態沒有問題，首先，例項化 Future 類，然後將這個例項和處理函式和相關引數一起，作為引數來例項化 _WorkItem 類，然後將例項 w 作為 value ，_queue_count 作為 key 存入 _pending_work_items 中。然後呼叫 _start_queue_management_thread 方法開啟程式池中的管理執行緒。現在來看看這部分程式碼
def _start_queue_management_thread(self):
# When the executor gets lost, the weakref callback will wake up
# the queue management thread.
def weakref_cb(_, q=self._result_queue):
q.put(None)
if self._queue_management_thread is None:
# Start the processes so that their sentinels are known.
self._adjust_process_count()
self._queue_management_thread = threading.Thread(
target=_queue_management_worker,
args=(weakref.ref(self, weakref_cb),
self._processes,
self._pending_work_items,
self._work_ids,
self._call_queue,
self._result_queue))
self._queue_management_thread.daemon = True
self._queue_management_thread.start()
_threads_queues[self._queue_management_thread] = self._result_queue複製程式碼這一部分很簡單，首先執行 _adjust_process_count 方法，然後開啟一個守護執行緒，執行 _queue_management_worker 方法。我們首先來看看 _adjust_process_count 方法。
def _adjust_process_count(self):
for _ in range(len(self._processes), self._max_workers):
p = multiprocessing.Process(
target=_process_worker,
args=(self._call_queue,
self._result_queue))
p.start()
self._processes[p.pid] = p複製程式碼根據在 __init__ 方法中設定的 _max_workers 來開啟對應數量的程式，在程式中執行 _process_worker 函式。
恩，順藤摸瓜，我們先來看看 _process_worker 函式吧？
def _process_worker(call_queue, result_queue):
"""Evaluates calls from call_queue and places the results in result_queue.
This worker is run in a separate process.
Args:
call_queue: A multiprocessing.Queue of _CallItems that will be read and
evaluated by the worker.
result_queue: A multiprocessing.Queue of _ResultItems that will written
to by the worker.
shutdown: A multiprocessing.Event that will be set as a signal to the
worker that it should exit when call_queue is empty.
"""
while True:
call_item = call_queue.get(block=True)
if call_item is None:
# Wake up queue management thread
result_queue.put(os.getpid())
return
try:
r = call_item.fn(*call_item.args, **call_item.kwargs)
except BaseException as e:
exc = _ExceptionWithTraceback(e, e.__traceback__)
result_queue.put(_ResultItem(call_item.work_id, exception=exc))
else:
result_queue.put(_ResultItem(call_item.work_id,
result=r))複製程式碼首先，這裡搞了一個死迴圈，緊接著，我們從 call_queue 佇列中獲取一個 _WorkItem 例項，然後如果獲取的值為 None 的話，那麼證明沒有新的任務進來了，我們可以把當前程式的 pid 放入結果佇列中。然後結束程式。
如果收到了任務，那麼執行這個任務。不管是在執行過程中發生異常，亦或者是得到最終的結果，都將其封裝為 _ResultItem 例項，並將其放入結果佇列中。
好了，我們回到剛剛看了一半的 _start_queue_management_thread 函式，
def _start_queue_management_thread(self):
# When the executor gets lost, the weakref callback will wake up
# the queue management thread.
def weakref_cb(_, q=self._result_queue):
q.put(None)
if self._queue_management_thread is None:
# Start the processes so that their sentinels are known.
self._adjust_process_count()
self._queue_management_thread = threading.Thread(
target=_queue_management_worker,
args=(weakref.ref(self, weakref_cb),
self._processes,
self._pending_work_items,
self._work_ids,
self._call_queue,
self._result_queue))
self._queue_management_thread.daemon = True
self._queue_management_thread.start()
_threads_queues[self._queue_management_thread] = self._result_queue複製程式碼在執行完 _adjust_process_count 函式後，我們程式池中的 _processes 變數（它是一個 dict ）便關聯了一些處理程式。然後我們開啟一個後臺守護執行緒，來執行 _queue_management_worker 函式，我們給它傳了幾個變數，首先 _processes 是我們的程式對映，_pending_work_items 中存放著我們待處理任務，還有 _call_queue 和 _result_queue 。好了還有一個引數大家可能不太理解，就是 weakref.ref(self, weakref_cb) 這貨。
首先，Python 是一門具有垃圾回收機制的語言，有著 GC (Garbage Collection) 機制意味著我們在大多數時候，不太需要去關注記憶體的分配與回收。在 Python 中，什麼時候物件會被回收是由其引用計數所決定的。當引用計數為 0 的時候，這個物件會被回收。在有一些情況下，我們物件因為交叉引用或者其餘的一些原因，造成引用計數始終不為0，這意味著這個物件無法被回收。造成記憶體洩露
。因此區別於我們普通的引用，Python 中新增了一個引用機制叫做弱引用，弱引用的意義在於，某個變數持有一個物件，卻不會增加這個物件的引用計數。因此 weakref.ref(self, weakref_cb) 在大多數而言，等價於 self （至於這裡為什麼要使用弱引用，我們這裡先不講，會開一個單章來說）
好了，這一部分程式碼看完，我們來看看，_queue_management_worker 怎麼實現的
def _queue_management_worker(executor_reference,
processes,
pending_work_items,
work_ids_queue,
call_queue,
result_queue):
"""Manages the communication between this process and the worker processes.
This function is run in a local thread.
executor_reference: A weakref.ref to the ProcessPoolExecutor that owns
Args:
process: A list of the multiprocessing.Process instances used as
this thread. Used to determine if the ProcessPoolExecutor has been
garbage collected and that this function can exit.
workers.
pending_work_items: A dict mapping work ids to _WorkItems e.g.
{5: <_WorkItem...>, 6: <_WorkItem...>, ...}
work_ids_queue: A queue.Queue of work ids e.g. Queue([5, 6, ...]).
call_queue: A multiprocessing.Queue that will be filled with _CallItems
derived from _WorkItems for processing by the process workers.
result_queue: A multiprocessing.Queue of _ResultItems generated by the
process workers.
"""
executor = None
def shutting_down():
return _shutdown or executor is None or executor._shutdown_thread
def shutdown_worker():
# This is an upper bound
nb_children_alive = sum(p.is_alive() for p in processes.values())
for i in range(0, nb_children_alive):
call_queue.put_nowait(None)
# Release the queue`s resources as soon as possible.
call_queue.close()
# If .join() is not called on the created processes then
# some multiprocessing.Queue methods may deadlock on Mac OS X.
for p in processes.values():
p.join()
reader = result_queue._reader
while True:
_add_call_item_to_queue(pending_work_items,
work_ids_queue,
call_queue)
sentinels = [p.sentinel for p in processes.values()]
assert sentinels
ready = wait([reader] + sentinels)
if reader in ready:
result_item = reader.recv()
else:
# Mark the process pool broken so that submits fail right now.
executor = executor_reference()
if executor is not None:
executor._broken = True
executor._shutdown_thread = True
executor = None
# All futures in flight must be marked failed
for work_id, work_item in pending_work_items.items():
work_item.future.set_exception(
BrokenProcessPool(
"A process in the process pool was "
"terminated abruptly while the future was "
"running or pending."
))
# Delete references to object. See issue16284
del work_item
pending_work_items.clear()
# Terminate remaining workers forcibly: the queues or their
# locks may be in a dirty state and block forever.
for p in processes.values():
p.terminate()
shutdown_worker()
return
if isinstance(result_item, int):
# Clean shutdown of a worker using its PID
# (avoids marking the executor broken)
assert shutting_down()
p = processes.pop(result_item)
p.join()
if not processes:
shutdown_worker()
return
elif result_item is not None:
work_item = pending_work_items.pop(result_item.work_id, None)
# work_item can be None if another process terminated (see above)
if work_item is not None:
if result_item.exception:
work_item.future.set_exception(result_item.exception)
else:
work_item.future.set_result(result_item.result)
# Delete references to object. See issue16284
del work_item
# Check whether we should start shutting down.
executor = executor_reference()
# No more work items can be added if:
# - The interpreter is shutting down OR
# - The executor that owns this worker has been collected OR
# - The executor that owns this worker has been shutdown.
if shutting_down():
try:
# Since no new work items can be added, it is safe to shutdown
# this thread if there are no pending work items.
if not pending_work_items:
shutdown_worker()
return
except Full:
# This is not a problem: we will eventually be woken up (in
# result_queue.get()) and be able to send a sentinel again.
pass
executor = None複製程式碼熟悉的大迴圈，迴圈的第一步，利用 _add_call_item_to_queue 函式來將等待佇列中的任務加入到呼叫佇列中去，先來看看這一部分程式碼
def _add_call_item_to_queue(pending_work_items,
work_ids,
call_queue):
"""Fills call_queue with _WorkItems from pending_work_items.
This function never blocks.
Args:
pending_work_items: A dict mapping work ids to _WorkItems e.g.
{5: <_WorkItem...>, 6: <_WorkItem...>, ...}
work_ids: A queue.Queue of work ids e.g. Queue([5, 6, ...]). Work ids
are consumed and the corresponding _WorkItems from
pending_work_items are transformed into _CallItems and put in
call_queue.
call_queue: A multiprocessing.Queue that will be filled with _CallItems
derived from _WorkItems.
"""
while True:
if call_queue.full():
return
try:
work_id = work_ids.get(block=False)
except queue.Empty:
return
else:
work_item = pending_work_items[work_id]
if work_item.future.set_running_or_notify_cancel():
call_queue.put(_CallItem(work_id,
work_item.fn,
work_item.args,
work_item.kwargs),
block=True)
else:
del pending_work_items[work_id]
continue複製程式碼首先，判斷呼叫佇列是不是已經滿了，如果滿了，則放棄這次迴圈。緊接著從 work_id 佇列中取出，然後從等待任務中取出對應的 _WorkItem 例項。緊接著，呼叫例項中繫結的 Future 例項的 set_running_or_notify_cancel 方法來設定任務的狀態，緊接著將其扔入呼叫佇列中。
def set_running_or_notify_cancel(self):
"""Mark the future as running or process any cancel notifications.
Should only be used by Executor implementations and unit tests.
If the future has been cancelled (cancel() was called and returned
True) then any threads waiting on the future completing (though calls
to as_completed() or wait()) are notified and False is returned.
If the future was not cancelled then it is put in the running state
(future calls to running() will return True) and True is returned.
This method should be called by Executor implementations before
executing the work associated with this future. If this method returns
False then the work should not be executed.
Returns:
False if the Future was cancelled, True otherwise.
Raises:
RuntimeError: if this method was already called or if set_result()
or set_exception() was called.
"""
with self._condition:
if self._state == CANCELLED:
self._state = CANCELLED_AND_NOTIFIED
for waiter in self._waiters:
waiter.add_cancelled(self)
# self._condition.notify_all() is not necessary because
# self.cancel() triggers a notification.
return False
elif self._state == PENDING:
self._state = RUNNING
return True
else:
LOGGER.critical(`Future %s in unexpected state: %s`,
id(self),
self._state)
raise RuntimeError(`Future in unexpected state`)複製程式碼這一部分內容很簡單，檢查當前例項如果處於等待狀態，就返回 True ，如果處於被取消的狀態，就返回 False , 在 _add_call_item_to_queue 函式中，會將已經處於 cancel 狀態的 _WorkItem 從等待任務中移除。
好了，我們繼續回到 _queue_management_worker 函式中去，
def _queue_management_worker(executor_reference,
processes,
pending_work_items,
work_ids_queue,
call_queue,
result_queue):
"""Manages the communication between this process and the worker processes.
This function is run in a local thread.
executor_reference: A weakref.ref to the ProcessPoolExecutor that owns
Args:
process: A list of the multiprocessing.Process instances used as
this thread. Used to determine if the ProcessPoolExecutor has been
garbage collected and that this function can exit.
workers.
pending_work_items: A dict mapping work ids to _WorkItems e.g.
{5: <_WorkItem...>, 6: <_WorkItem...>, ...}
work_ids_queue: A queue.Queue of work ids e.g. Queue([5, 6, ...]).
call_queue: A multiprocessing.Queue that will be filled with _CallItems
derived from _WorkItems for processing by the process workers.
result_queue: A multiprocessing.Queue of _ResultItems generated by the
process workers.
"""
executor = None
def shutting_down():
return _shutdown or executor is None or executor._shutdown_thread
def shutdown_worker():
# This is an upper bound
nb_children_alive = sum(p.is_alive() for p in processes.values())
for i in range(0, nb_children_alive):
call_queue.put_nowait(None)
# Release the queue`s resources as soon as possible.
call_queue.close()
# If .join() is not called on the created processes then
# some multiprocessing.Queue methods may deadlock on Mac OS X.
for p in processes.values():
p.join()
reader = result_queue._reader
while True:
_add_call_item_to_queue(pending_work_items,
work_ids_queue,
call_queue)
sentinels = [p.sentinel for p in processes.values()]
assert sentinels
ready = wait([reader] + sentinels)
if reader in ready:
result_item = reader.recv()
else:
# Mark the process pool broken so that submits fail right now.
executor = executor_reference()
if executor is not None:
executor._broken = True
executor._shutdown_thread = True
executor = None
# All futures in flight must be marked failed
for work_id, work_item in pending_work_items.items():
work_item.future.set_exception(
BrokenProcessPool(
"A process in the process pool was "
"terminated abruptly while the future was "
"running or pending."
))
# Delete references to object. See issue16284
del work_item
pending_work_items.clear()
# Terminate remaining workers forcibly: the queues or their
# locks may be in a dirty state and block forever.
for p in processes.values():
p.terminate()
shutdown_worker()
return
if isinstance(result_item, int):
# Clean shutdown of a worker using its PID
# (avoids marking the executor broken)
assert shutting_down()
p = processes.pop(result_item)
p.join()
if not processes:
shutdown_worker()
return
elif result_item is not None:
work_item = pending_work_items.pop(result_item.work_id, None)
# work_item can be None if another process terminated (see above)
if work_item is not None:
if result_item.exception:
work_item.future.set_exception(result_item.exception)
else:
work_item.future.set_result(result_item.result)
# Delete references to object. See issue16284
del work_item
# Check whether we should start shutting down.
executor = executor_reference()
# No more work items can be added if:
# - The interpreter is shutting down OR
# - The executor that owns this worker has been collected OR
# - The executor that owns this worker has been shutdown.
if shutting_down():
try:
# Since no new work items can be added, it is safe to shutdown
# this thread if there are no pending work items.
if not pending_work_items:
shutdown_worker()
return
except Full:
# This is not a problem: we will eventually be woken up (in
# result_queue.get()) and be able to send a sentinel again.
pass
executor = None複製程式碼result_item 變數
我們看看
首先，大家可能在這裡有點疑問了
sentinels = [p.sentinel for p in processes.values()]
assert sentinels
ready = wait([reader] + sentinels)複製程式碼這個 wait 是什麼鬼啊，reader 又是什麼鬼啊。一步步來。首先，我們看到，前面，reader = result_queue._reader 也會引起大家的疑問，這裡我們 result_queue 是 multiprocess 裡面的 SimpleQueue 啊，它沒有 _reader 方法啊QAQ
class SimpleQueue(object):
def __init__(self, *, ctx):
self._reader, self._writer = connection.Pipe(duplex=False)
self._rlock = ctx.Lock()
self._poll = self._reader.poll
if sys.platform == `win32`:
self._wlock = None
else:
self._wlock = ctx.Lock()複製程式碼上面這貼出來的，是 SimpleQueue 的部分程式碼，我們可以很清楚的看到，SimpleQueue 本質是利用一個 Pipe 來進行程式間通訊的，然後 _reader 是讀取 Pipe 的一個變數。
Note : 大家可以複習下其餘幾種程式間通訊的方法了
好了，這一部分看懂後，我們來看看 wait 方法吧。
def wait(object_list, timeout=None):
```
Wait till an object in object_list is ready/readable.
Returns list of those objects in object_list which are ready/readable.
```
with _WaitSelector() as selector:
for obj in object_list:
selector.register(obj, selectors.EVENT_READ)
if timeout is not None:
deadline = time.time() + timeout
while True:
ready = selector.select(timeout)
if ready:
return [key.fileobj for (key, events) in ready]
else:
if timeout is not None:
timeout = deadline - time.time()
if timeout < 0:
return ready複製程式碼這一部分程式碼很簡單，首先將我們待讀取的物件，進行一次註冊，然後當 timeout 為 None 的時候，就一直等待到有物件讀取資料成功為止
好了，我們繼續回到前面的 _queue_management_worker 函式中去，來看看這樣一段程式碼
ready = wait([reader] + sentinels)
if reader in ready:
result_item = reader.recv()
else:
# Mark the process pool broken so that submits fail right now.
executor = executor_reference()
if executor is not None:
executor._broken = True
executor._shutdown_thread = True
executor = None
# All futures in flight must be marked failed
for work_id, work_item in pending_work_items.items():
work_item.future.set_exception(
BrokenProcessPool(
"A process in the process pool was "
"terminated abruptly while the future was "
"running or pending."
))
# Delete references to object. See issue16284
del work_item
pending_work_items.clear()
# Terminate remaining workers forcibly: the queues or their
# locks may be in a dirty state and block forever.
for p in processes.values():
p.terminate()
shutdown_worker()
return複製程式碼我們用 wait 函式來讀取一系列物件，因為我們沒有設定 Timeout ，所以當我們拿到可讀取物件的結果時，如果 result_queue._reader 沒有在列表中，那麼意味著，有處理程式突然異常關閉了，這個時候，我們開始執行後面的語句來執行目前程式池的關閉操作。如果在列表中，我們讀取資料，得到 result_item 變數
我們再看看下面的程式碼
if isinstance(result_item, int):
# Clean shutdown of a worker using its PID
# (avoids marking the executor broken)
assert shutting_down()
p = processes.pop(result_item)
p.join()
if not processes:
shutdown_worker()
return
elif result_item is not None:
work_item = pending_work_items.pop(result_item.work_id, None)
# work_item can be None if another process terminated (see above)
if work_item is not None:
if result_item.exception:
work_item.future.set_exception(result_item.exception)
else:
work_item.future.set_result(result_item.result)
# Delete references to object. See issue16284
del work_item複製程式碼首先，如果 result_item 變數是 int 型別的話，不知道大家還記不記得在 _process_worker 函式中有這樣一段邏輯
call_item = call_queue.get(block=True)
if call_item is None:
# Wake up queue management thread
result_queue.put(os.getpid())
return複製程式碼當呼叫佇列中沒有新的任務時，將程式 pid 放入 result_queue 中。那麼我們 result_item 如果值為 int 那麼意味著，我們之前任務處理工作已經完畢，於是開始清理，關閉我們的程式池。
如果 result_item 既不為 int 也不為 None , 那麼必然是 _ResultItem 的例項，我們根據 work_id 取出 _WorkItem 例項，並將產生的異常或者值和 _WorkItem 例項中的 Future 例項（也就是我們 submit 後返回的那貨）進行繫結。
最後，刪除這個 work_item ，完事兒，手工
最後
洋洋灑灑寫了一大篇辣雞文章，希望大家不要介意，其實我們能看到 concurrent.future 的實現，其實並沒有用什麼高深的黑魔法，但是其中細節值得我們一一品味，所以這篇文章我們先寫到這裡。後面有機會的話，我們再去看看 concurrent.future 其餘部分程式碼。也有蠻多值得品味的地方。