多程式的方式可以增加指令碼的併發處理能力， python 支援這種多程式的程式設計方式
在類unix系統中， python的os 模組內建了fork 函式用以建立子程式
fork 方式建立子程式
|
1 2 3 4 5 6 7 8 9 10 |
import os print "Process %s start ..." %(os.getpid()) pid = os.fork() if pid == 0: print "This is child process and my pid is %d, my father process is %d" %(os.getpid(), os.getppid()) else: print "This is Fater process, And Its child pid is %d" %(pid) |
執行結果
|
1 2 3 |
Process 4276 start ... This is Fater process, And Its child pid is 4277 This is child process and my pid is 4277, my father process is 4276 |
從結果可以看到， 從pid = os.fork() 開始， 下面的部分程式碼執行了兩次， 第一次是父程式執行， 第二次是子程式執行， 且子程式的fork的結果總是0， 所以這個也可以用來作為區分父程式或是子程式標誌
那麼變數在多個程式之間是否相互影響呢
import os
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
print "Process %s start ..." %(os.getpid()) pid = os.fork() source = 10 if pid == 0: print "This is child process and my pid is %d, my father process is %d" %(os.getpid(), os.getppid()) source = source - 6 print "child process source value is "+str(source) else: print "This is Fater process, And Its child pid is %d" %(pid) source = source - 1 print "father process source value is "+str(source) print "source value is "+str(source) |
執行的結果如下：
|
1 2 3 4 5 6 7 |
Process 4662 start ... This is Fater process, And Its child pid is 4663 This is child process and my pid is 4663, my father process is 4662 father process source value is 9 child process source value is 4 source value is 9 source value is 4 |
很明顯， 初始值為10的source 在父程式中值 減少了 1, 為9， 而子程式明顯source的初始值 是10, 也就是說多程式之間並沒有什麼相互影響
multiprocessing 方式建立子程式
fork 方式是僅在linux 下才有的介面， 在windows下並沒有， 那麼在windows下如何實現多程式呢， 這就用到了multiprocessing
multiprocessing 模組的Process 物件表示的是一個程式物件， 可以建立子程式並執行制定的函式
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
from multiprocessing import Process import os def pro_do(name, func): print "This is child process %d from parent process %d, and name is %s which is used for %s" %(os.getpid(), os.getppid(), name, func) if __name__ == "__main__": print "Parent process id %d" %(os.getpid()) #process 物件指定子程式將要執行的操作方法(pro_do), 以及該函式的物件列表args(必須是tuple格式， 且元素與pro_do的引數一一對應) pro = Process(target=pro_do, args=("test", "dev")) print "start child process" #啟動子程式 pro.start() #是否阻塞方式執行， 如果有， 則阻塞方式， 否則非阻塞 pro.join() #if has this, it's synchronous operation or asynchronous operation print "Process end" |
執行結果
|
1 2 3 4 |
Parent process id 4878 start child process This is child process 4879 from parent process 4878, and name is test which is used for dev Process end |
如果沒有pro.join()， 則表示非阻塞方式執行， 那麼最終的Process end的輸出位置就有可能出現在pro_do 方法執行之前了
|
1 2 3 4 |
Parent process id 4903 start child process Process end This is child process 4904 from parent process 4903, and name is test which is used for dev |
通過multiprocessing 的process物件建立多程式， 還可以從主程式中向子程式傳遞引數， 例如上面例子中的pro_do的引數
Pool 程式池
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
from multiprocessing import Pool import os, time def pro_do(process_num): print "child process id is %d" %(os.getpid()) time.sleep(6 - process_num) print "this is process %d" %(process_num) if __name__ == "__main__": print "Current process is %d" %(os.getpid()) p = Pool() for i in range(5): p.apply_async(pro_do, (i,)) #增加新的程式 p.close() # 禁止在增加新的程式 p.join() print "pool process done" |
輸出：
|
1 2 3 4 5 6 7 8 9 10 11 12 |
Current process is 19138 child process id is 19139 child process id is 19140 this is process 1 child process id is 19140 this is process 0 child process id is 19139 this is process 2 child process id is 19140 this is process 3 this is process 4 pool process done |
其中
|
1 2 |
child process id is 19139 child process id is 19140 |
是立即輸出的， 後面的依次在等待了sleep的時間後輸出 ， 之所以立即輸出了上面兩個是因為誒Pool 程式池預設是按照cpu的數量開啟子程式的， 我是在虛擬機器中執行， 只分配了兩核， 所以先立即啟動兩個子程式， 剩下的程式要等到前面的程式執行完成後才能啟動。
不過也可以在p=Poo() 中使用Pool(5)來指定啟動的子程式數量， 這樣輸出就是下面的了：
|
1 2 3 4 5 6 7 8 9 10 11 12 |
Current process is 19184 child process id is 19185 child process id is 19186 child process id is 19188 child process id is 19189 child process id is 19187 this is process 4 this is process 3 this is process 2 this is process 1 this is process 0 pool process done |
且
|
1 2 3 4 5 6 |
Current process is 19184 child process id is 19185 child process id is 19186 child process id is 19188 child process id is 19189 child process id is 19187 |
都是立即輸出的
程式間的通訊
父程式可以指定子程式執行的方法及其引數， 達到父程式向子程式傳遞訊息的單向通訊的目的， 那子程式之間或子程式怎麼向父程式通訊呢
Queue
Queue 是一種方式
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
from multiprocessing import Process, Queue import os, time def write_queue(q): for name in ["Yi_Zhi_Yu", "Tony" ,"San"]: print "put name %s to queue" %(name) q.put(name) time.sleep(2) print "write data finished" def read_queue(q): print "begin to read data" while True: name = q.get() print "get name %s from queue" %(name) if __name__ == "__main__": q = Queue() pw = Process(target=write_queue, args=(q,)) pr = Process(target=read_queue,args=(q,)) pw.start() pr.start() pw.join() #這個表示是否阻塞方式啟動程式， 如果要立即讀取的話， 兩個程式的啟動就應該是非阻塞式的， 所以pw在start後不能立即使用pw.join(), 要等pr start後方可 pr.terminate() #服務程式，強制停止 |
結果
|
1 2 3 4 5 6 7 8 |
put name Yi_Zhi_Yu to queue begin to read data get name Yi_Zhi_Yu from queue put name Tony to queue get name Tony from queue put name San to queue get name San from queue write data finished |
Pipe
另外還有Pipe
其原理參見http://ju.outofmemory.cn/entry/106041， 其只能作為兩個程式之間的通訊
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
#!/usr/bin/env python #encoding=utf-8 from multiprocessing import Process,Pipe import os,time,sys def send_pipe(p): names = ["Yi_Zhi_Yu", "Tony", "San"] for name in names: print "put name %s to Pipe" %(name) p.send(name) time.sleep(1) def recv_pipe(p): print "Try to read data in pipe" while True: name = p.recv() print "get name %s from pipe" %(name) if __name__ == "__main__": #pipe, one for send, one for read ps_pipe, pr_pipe = Pipe() #process ps = Process(target=send_pipe, args=(ps_pipe,)) pr = Process(target=recv_pipe, args=(pr_pipe,)) pr.start() ps.start() ps.join() pr.terminate() |
在例項化Pipe的時候， 會產生兩個ps_pipe(read-write Connection, handle 5)， pr_pipe(read-write Connection, handle 5) , 都可以作為傳送或者接受方， 一旦一個確認為攻， 另一個自然就是受了(之所以Pipe只能作為兩個程式之間的通訊方式， 原因也許就是這個)，產生的結果如下
|
1 2 3 4 5 6 7 |
Try to read data in pipe put name Yi_Zhi_Yu to Pipe get name Yi_Zhi_Yu from pipe put name Tony to Pipe get name Tony from pipe put name San to Pipe get name San from pipe |
還有一種Array, Value 的形式， 暫且不表， 有時間在折騰
以上均為python 學習筆記和練習， 如有錯誤， 歡迎指出