3種方式實現python多執行緒併發處理

大衛發表於2019-02-16
Python執行緒

標籤： python奇淫技巧

最優執行緒數

  • Ncpu=CPU的數量
  • Ucpu=目標CPU使用率
  • W/C=等待時間與計算時間的比率

為保持處理器達到期望的使用率，最優的執行緒池的大小等於
$$Nthreads=Ncpu*Ucpu*（1+W/C$$

  • cpu密集型任務，即$W<<C$，則$W/C≈0$，則$Nthreads=Ncpu*Ucpu$

如果希望CPU利用率為100%，則$Nthreads=Ncpu$

  • IO密集型任務，即系統大部分時間在跟I/O互動，而這個時間執行緒不會佔用CPU來處理，即在這個時間範圍內，可以由其他執行緒來使用CPU，因而可以多配置一些執行緒。
  • 混合型任務，二者都佔有一定的時間

線城池

對於任務數量不斷增加的程式，每有一個任務就生成一個執行緒，最終會導致執行緒數量的失控。對於任務數量不端增加的程式，固定執行緒數量的執行緒池是必要的。

方法一：使用threadpool模組

threadpool是一個比較老的模組了，支援py2 和 py3 。

import threadpool
import time

def sayhello (a):
    print("hello: "+a)
    time.sleep(2)

def main():
    global result
    seed=["a","b","c"]
    start=time.time()
    task_pool=threadpool.ThreadPool(5)
    requests=threadpool.makeRequests(sayhello,seed)
    for req in requests:
        task_pool.putRequest(req)
    task_pool.wait()
    end=time.time()
    time_m = end-start
    print("time: "+str(time_m))
    start1=time.time()
    for each in seed:
        sayhello(each)
    end1=time.time()
    print("time1: "+str(end1-start1))

if __name__ == `__main__`:
    main(

方法二：使用concurrent.futures模組

from concurrent.futures import ThreadPoolExecutor
import time

import time
from concurrent.futures import ThreadPoolExecutor, wait, as_completed

ll = []
def sayhello(a):
    print("hello: "+a)
    ll.append(a)
    time.sleep(0.8)

def main():
    seed=["a","b","c","e","f","g","h"]
    start1=time.time()
    for each in seed:
        sayhello(each)
    end1=time.time()
    print("time1: "+str(end1-start1))
    start2=time.time()
    with ThreadPoolExecutor(2) as executor:
        for each in seed:
            executor.submit(sayhello,each)
    end2=time.time()
    print("time2: "+str(end2-start2))

def main2():
    seed = ["a", "b", "c", "e", "f", "g", "h"]
    executor = ThreadPoolExecutor(max_workers=10)
    f_list = []
    for each in seed:
        future = executor.submit(sayhello, each)
        f_list.append(future)
    wait(f_list)
    print(ll)
    print(`主執行緒結束`)


def main3():
    seed = ["a", "b", "c", "e", "f", "g", "h"]
    with ThreadPoolExecutor(max_workers=2) as executor:
        f_list = []
        for each in seed:
            future = executor.submit(sayhello, each)
            f_list.append(future)
        wait(f_list,return_when=`ALL_COMPLETED`)
        print(ll)
        print(`主執行緒結束`)

if __name__ == `__main__`:
    main3()

方法三：使用vthread模組

參考：https://pypi.org/project/vthr…

demo1

import vthread
 
@vthread.pool(6)
def some(a,b,c):
    import time;time.sleep(1)
    print(a+b+c)
 
for i in range(10):
    some(i,i,i)

demo2:分組執行緒池

import vthread
pool_1 = vthread.pool(5,gqueue=1) # open a threadpool with 5 threads named 1
pool_2 = vthread.pool(2,gqueue=2) # open a threadpool with 2 threads named 2

@pool_1
def foolfunc1(num):
    time.sleep(1)
    print(f"foolstring1, test3 foolnumb1:{num}")

@pool_2
def foolfunc2(num):
    time.sleep(1)
    print(f"foolstring2, test3 foolnumb2:{num}")

@pool_2
def foolfunc3(num):
    time.sleep(1)
    print(f"foolstring3, test3 foolnumb3:{num}")

for i in range(10): foolfunc1(i)
for i in range(4): foolfunc2(i)
for i in range(2): foolfunc3(i)

相關文章