轉(zhuǎn)自：原文地址：http://www.cnblogs.com/alex3714/articles/5230609.html

本節(jié)內(nèi)容進程、與線程區(qū)別cpu運行原理python GIL全局解釋器鎖線程語法join線程鎖之Lock/Rlock/信號量將線程變?yōu)槭刈o進程Event事件　queue隊列生產(chǎn)者消費者模型Queue隊列開發(fā)一個線程池進程語法進程間通訊進程池　　　　 進程與線程什么是線程(thread)？線程是操作系統(tǒng)能夠進行運算調(diào)度的最小單位。它被包含在進程之中，是進程中的實際運作單位。一條線程指的是進程中一個單一順序的控制流，一個進程中可以并發(fā)多個線程，每條線程并行執(zhí)行不同的任務(wù)A thread is an execution context, which is all the information a CPU needs to execute a stream of instructions.Suppose you're reading a book, and you want to take a break right now, but you want to be able to come back and resume reading from the exact point where you stopped. One way to achieve that is by jotting down the page number, line number, and Word number. So your execution context for reading a book is these 3 numbers.If you have a roommate, and she's using the same technique, she can take the book while you're not using it, and resume reading from where she stopped. Then you can take it back, and resume it from where you were.Threads work in the same way. A CPU is giving you the illusion that it's doing multiple computations at the same time. It does that by spending a bit of time on each computation. It can do that because it has an execution context for each computation. Just like you can share a book with your friend, many tasks can share a CPU.On a more technical level, an execution context (therefore a thread) consists of the values of the CPU's registers.Last: threads are different from PRocesses. A thread is a context of execution, while a process is a bunch of resources associated with a computation. A process can have one or many threads.Clarification: the resources associated with a process include memory pages (all the threads in a process have the same view of the memory), file descriptors (e.g., open sockets), and security credentials (e.g., the ID of the user who started the process).什么是進程(process)？An executing instance of a program is called a process.Each process provides the resources needed to execute a program. A process has a virtual address space, executable code, open handles to system objects, a security context, a unique process identifier, environment variables, a priority class, minimum and maximum working set sizes, and at least one thread of execution. Each process is started with a single thread, often called the primary thread, but can create additional threads from any of its threads.進程與線程的區(qū)別？Threads share the address space of the process that created it; processes have their own address space.Threads have direct access to the data segment of its process; processes have their own copy of the data segment of the parent process.Threads can directly communicate with other threads of its process; processes must use interprocess communication to communicate with sibling processes.New threads are easily created; new processes require duplication of the parent process.Threads can exercise considerable control over threads of the same process; processes can only exercise control over child processes.Changes to the main thread (cancellation, priority change, etc.) may affect the behavior of the other threads of the process; changes to the parent process does not affect child processes.Python GIL(Global Interpreter Lock)　　In CPython, the global interpreter lock, or GIL, is a mutex that prevents multiple native threads from executing Python bytecodes at once. This lock is necessary mainly because CPython’s memory management is not thread-safe. (However, since the GIL exists, other features have grown to depend on the guarantees that it enforces.)上面的核心意思就是，無論你啟多少個線程，你有多少個cpu, Python在執(zhí)行的時候會淡定的在同一時刻只允許一個線程運行，擦。。。，那這還叫什么多線程呀？莫如此早的下結(jié)結(jié)論，聽我現(xiàn)場講。首先需要明確的一點是GIL并不是Python的特性，它是在實現(xiàn)Python解析器(CPython)時所引入的一個概念。就好比C++是一套語言（語法）標(biāo)準(zhǔn)，但是可以用不同的編譯器來編譯成可執(zhí)行代碼。有名的編譯器例如GCC，INTEL C++，Visual C++等。Python也一樣，同樣一段代碼可以通過CPython，PyPy，Psyco等不同的Python執(zhí)行環(huán)境來執(zhí)行。像其中的JPython就沒有GIL。然而因為CPython是大部分環(huán)境下默認(rèn)的Python執(zhí)行環(huán)境。所以在很多人的概念里CPython就是Python，也就想當(dāng)然的把GIL歸結(jié)為Python語言的缺陷。所以這里要先明確一點：GIL并不是Python的特性，Python完全可以不依賴于GIL這篇文章透徹的剖析了GIL對python多線程的影響，強烈推薦看一下：http://www.dabeaz.com/python/UnderstandingGIL.pdf  Python threading模塊線程有2種調(diào)用方式，如下：直接調(diào)用import threadingimport time def sayhi(num): #定義每個線程要運行的函數(shù)     print("running on number:%s" %num)     time.sleep(3) if __name__ == '__main__':     t1 = threading.Thread(target=sayhi,args=(1,)) #生成一個線程實例    t2 = threading.Thread(target=sayhi,args=(2,)) #生成另一個線程實例     t1.start() #啟動線程    t2.start() #啟動另一個線程     print(t1.getName()) #獲取線程名    print(t2.getName())繼承式調(diào)用import threadingimport time  class MyThread(threading.Thread):    def __init__(self,num):        threading.Thread.__init__(self)        self.num = num     def run(self):#定義每個線程要運行的函數(shù)         print("running on number:%s" %self.num)         time.sleep(3) if __name__ == '__main__':     t1 = MyThread(1)    t2 = MyThread(2)    t1.start()    t2.start()Join & DaemonSome threads do background tasks, like sending keepalive packets, or performing periodic garbage collection, or whatever. These are only useful when the main program is running, and it's okay to kill them off once the other, non-daemon, threads have exited.Without daemon threads, you'd have to keep track of them, and tell them to exit, before your program can completely quit. By setting them as daemon threads, you can let them run and forget about them, and when your program quits, any daemon threads are killed automatically.import timeimport threading def run(n):     print('[%s]------running----/n' % n)    time.sleep(2)    print('--done--') def main():    for i in range(5):        t = threading.Thread(target=run,args=[i,])        #time.sleep(1)        t.start()        t.join(1)        print('starting thread', t.getName())  m = threading.Thread(target=main,args=[])m.setDaemon(True) #將主線程設(shè)置為Daemon線程,它退出時,其它子線程會同時退出,不管是否執(zhí)行完任務(wù)m.start()#m.join(timeout=2)print("---main thread done----")Note：Daemon threads are abruptly stopped at shutdown. Their resources (such as open files, database transactions, etc.) may not be released properly. If you want your threads to stop gracefully, make them non-daemonic and use a suitable signalling mechanism such as an Event.　　  線程鎖(互斥鎖Mutex)一個進程下可以啟動多個線程，多個線程共享父進程的內(nèi)存空間，也就意味著每個線程可以訪問同一份數(shù)據(jù)，此時，如果2個線程同時要修改同一份數(shù)據(jù)，會出現(xiàn)什么狀況？import timeimport threading def addNum():    global num #在每個線程中都獲取這個全局變量    print('--get num:',num )    time.sleep(1)    num  -=1 #對此公共變量進行-1操作 num = 100  #設(shè)定一個共享變量thread_list = []for i in range(100):    t = threading.Thread(target=addNum)    t.start()    thread_list.append(t) for t in thread_list: #等待所有線程執(zhí)行完畢    t.join()  print('final num:', num )正常來講，這個num結(jié)果應(yīng)該是0， 但在python 2.7上多運行幾次，會發(fā)現(xiàn)，最后打印出來的num結(jié)果不總是0，為什么每次運行的結(jié)果不一樣呢？ 哈，很簡單，假設(shè)你有A,B兩個線程，此時都 要對num 進行減1操作， 由于2個線程是并發(fā)同時運行的，所以2個線程很有可能同時拿走了num=100這個初始變量交給cpu去運算，當(dāng)A線程去處完的結(jié)果是99，但此時B線程運算完的結(jié)果也是99，兩個線程同時CPU運算的結(jié)果再賦值給num變量后，結(jié)果就都是99。那怎么辦呢？ 很簡單，每個線程在要修改公共數(shù)據(jù)時，為了避免自己在還沒改完的時候別人也來修改此數(shù)據(jù)，可以給這個數(shù)據(jù)加一把鎖， 這樣其它線程想修改此數(shù)據(jù)時就必須等待你修改完畢并把鎖釋放掉后才能再訪問此數(shù)據(jù)。 *注：不要在3.x上運行，不知為什么，3.x上的結(jié)果總是正確的，可能是自動加了鎖加鎖版本import timeimport threading def addNum():    global num #在每個線程中都獲取這個全局變量    print('--get num:',num )    time.sleep(1)    lock.acquire() #修改數(shù)據(jù)前加鎖    num  -=1 #對此公共變量進行-1操作    lock.release() #修改后釋放 num = 100  #設(shè)定一個共享變量thread_list = []lock = threading.Lock() #生成全局鎖for i in range(100):    t = threading.Thread(target=addNum)    t.start()    thread_list.append(t) for t in thread_list: #等待所有線程執(zhí)行完畢    t.join() print('final num:', num )　　RLock（遞歸鎖）說白了就是在一個大鎖中還要再包含子鎖import threading,time def run1():    print("grab the first part data")    lock.acquire()    global num    num +=1    lock.release()    return numdef run2():    print("grab the second part data")    lock.acquire()    global  num2    num2+=1    lock.release()    return num2def run3():    lock.acquire()    res = run1()    print('--------between run1 and run2-----')    res2 = run2()    lock.release()    print(res,res2)  if __name__ == '__main__':     num,num2 = 0,0    lock = threading.RLock()    for i in range(10):        t = threading.Thread(target=run3)        t.start() while threading.active_count() != 1:    print(threading.active_count())else:    print('----all threads done---')    print(num,num2)　　Semaphore(信號量)互斥鎖 同時只允許一個線程更改數(shù)據(jù)，而Semaphore是同時允許一定數(shù)量的線程更改數(shù)據(jù) ，比如廁所有3個坑，那最多只允許3個人上廁所，后面的人只能等里面有人出來了才能再進去。import threading,time def run(n):    semaphore.acquire()    time.sleep(1)    print("run the thread: %s/n" %n)    semaphore.release() if __name__ == '__main__':     num= 0    semaphore  = threading.BoundedSemaphore(5) #最多允許5個線程同時運行    for i in range(20):        t = threading.Thread(target=run,args=(i,))        t.start() while threading.active_count() != 1:    pass #print threading.active_count()else:    print('----all threads done---')    print(num) EventsAn event is a simple synchronization object;the event represents an internal flag, and threadscan wait for the flag to be set, or set or clear the flag themselves.event = threading.Event()# a client thread can wait for the flag to be setevent.wait()# a server thread can set or reset itevent.set()event.clear()If the flag is set, the wait method doesn’t do anything.If the flag is cleared, wait will block until it becomes set again.Any number of threads may wait for the same event.通過Event來實現(xiàn)兩個或多個線程間的交互，下面是一個紅綠燈的例子，即起動一個線程做交通指揮燈，生成幾個線程做車輛，車輛行駛按紅燈停，綠燈行的規(guī)則。import threading,timeimport randomdef light():    if not event.isSet():        event.set() #wait就不阻塞 #綠燈狀態(tài)    count = 0    while True:        if count < 10:            print('/033[42;1m--green light on---/033[0m')        elif count <13:            print('/033[43;1m--yellow light on---/033[0m')        elif count <20:            if event.isSet():                event.clear()            print('/033[41;1m--red light on---/033[0m')        else:            count = 0            event.set() #打開綠燈        time.sleep(1)        count +=1def car(n):    while 1:        time.sleep(random.randrange(10))        if  event.isSet(): #綠燈            print("car [%s] is running.." % n)        else:            print("car [%s] is waiting for the red light.." %n)if __name__ == '__main__':    event = threading.Event()    Light = threading.Thread(target=light)    Light.start()    for i in range(3):        t = threading.Thread(target=car,args=(i,))        t.start()　　　　queue隊列 queue is especially useful in threaded programming when information must be exchanged safely between multiple threads.class queue.Queue(maxsize=0) #先入先出class queue.LifoQueue(maxsize=0) #last in fisrt out class queue.PriorityQueue(maxsize=0) #存儲數(shù)據(jù)時可設(shè)置優(yōu)先級的隊列Constructor for a priority queue. maxsize is an integer that sets the upperbound limit on the number of items that can be placed in the queue. Insertion will block once this size has been reached, until queue items are consumed. If maxsize is less than or equal to zero, the queue size is infinite.The lowest valued entries are retrieved first (the lowest valued entry is the one returned by sorted(list(entries))[0]). A typical pattern for entries is a tuple in the form: (priority_number, data).exception queue.EmptyException raised when non-blocking get() (or get_nowait()) is called on a Queue object which is empty.exception queue.FullException raised when non-blocking put() (or put_nowait()) is called on a Queue object which is full.Queue.qsize()Queue.empty() #return True if empty  Queue.full() # return True if full Queue.put(item, block=True, timeout=None)Put item into the queue. If optional args block is true and timeout is None (the default), block if necessary until a free slot is available. If timeout is a positive number, it blocks at most timeout seconds and raises the Full exception if no free slot was available within that time. Otherwise (block is false), put an item on the queue if a free slot is immediately available, else raise the Full exception (timeout is ignored in that case).Queue.put_nowait(item)Equivalent to put(item, False).Queue.get(block=True, timeout=None)Remove and return an item from the queue. If optional args block is true and timeout is None (the default), block if necessary until an item is available. If timeout is a positive number, it blocks at most timeout seconds and raises the Empty exception if no item was available within that time. Otherwise (block is false), return an item if one is immediately available, else raise the Empty exception (timeout is ignored in that case).Queue.get_nowait()Equivalent to get(False).Two methods are offered to support tracking whether enqueued tasks have been fully processed by daemon consumer threads.Queue.task_done()Indicate that a formerly enqueued task is complete. Used by queue consumer threads. For each get() used to fetch a task, a subsequent call to task_done() tells the queue that the processing on the task is complete.If a join() is currently blocking, it will resume when all items have been processed (meaning that a task_done() call was received for every item that had been put() into the queue).Raises a ValueError if called more times than there were items placed in the queue.Queue.join() block直到queue被消費完畢生產(chǎn)者消費者模型 import time,randomimport queue,threadingq = queue.Queue()def Producer(name):  count = 0  while count <20:    time.sleep(random.randrange(3))    q.put(count)    print('Producer %s has produced %s baozi..' %(name, count))    count +=1def Consumer(name):  count = 0  while count <20:    time.sleep(random.randrange(4))    if not q.empty():        data = q.get()        print(data)        print('/033[32;1mConsumer %s has eat %s baozi.../033[0m' %(name, data))    else:        print("-----no baozi anymore----")    count +=1p1 = threading.Thread(target=Producer, args=('A',))c1 = threading.Thread(target=Consumer, args=('B',))p1.start()c1.start()　　 多進程multiprocessingmultiprocessing is a package that supports spawning processes using an API similar to the threading module. The multiprocessing package offers both local and remote concurrency, effectively side-stepping the Global Interpreter Lock by using subprocesses instead of threads. Due to this, the multiprocessing module allows the programmer to fully leverage multiple processors on a given machine. It runs on both Unix and Windows.from multiprocessing import Processimport timedef f(name):    time.sleep(2)    print('hello', name) if __name__ == '__main__':    p = Process(target=f, args=('bob',))    p.start()    p.join()To show the individual process IDs involved, here is an expanded example:from multiprocessing import Processimport os def info(title):    print(title)    print('module name:', __name__)    print('parent process:', os.getppid())    print('process id:', os.getpid())    print("/n/n") def f(name):    info('/033[31;1mfunction f/033[0m')    print('hello', name) if __name__ == '__main__':    info('/033[32;1mmain process line/033[0m')    p = Process(target=f, args=('bob',))    p.start()    p.join() 進程間通訊　　不同進程間內(nèi)存是不共享的，要想實現(xiàn)兩個進程間的數(shù)據(jù)交換，可以用以下方法：Queues使用方法跟threading里的queue差不多from multiprocessing import Process, Queue def f(q):    q.put([42, None, 'hello']) if __name__ == '__main__':    q = Queue()    p = Process(target=f, args=(q,))    p.start()    print(q.get())    # prints "[42, None, 'hello']"    p.join()PipesThe Pipe() function returns a pair of connection objects connected by a pipe which by default is duplex (two-way). For example:from multiprocessing import Process, Pipe def f(conn):    conn.send([42, None, 'hello'])    conn.close() if __name__ == '__main__':    parent_conn, child_conn = Pipe()    p = Process(target=f, args=(child_conn,))    p.start()    print(parent_conn.recv())   # prints "[42, None, 'hello']"    p.join()The two connection objects returned by Pipe() represent the two ends of the pipe. Each connection object has send() and recv() methods (among others). Note that data in a pipe may become corrupted if two processes (or threads) try to read from or write to the same end of the pipe at the same time. Of course there is no risk of corruption from processes using different ends of the pipe at the same time. ManagersA manager object returned by Manager() controls a server process which holds Python objects and allows other processes to manipulate them using proxies.A manager returned by Manager() will support types list, dict, Namespace, Lock, RLock, Semaphore, BoundedSemaphore, Condition, Event, Barrier, Queue, Value and Array. For example,from multiprocessing import Process, Manager def f(d, l):    d[1] = '1'    d['2'] = 2    d[0.25] = None    l.append(1)    print(l) if __name__ == '__main__':    with Manager() as manager:        d = manager.dict()         l = manager.list(range(5))        p_list = []        for i in range(10):            p = Process(target=f, args=(d, l))            p.start()            p_list.append(p)        for res in p_list:            res.join()         print(d)        print(l)　　  進程同步Without using the lock output from the different processes is liable to get all mixed up.from multiprocessing import Process, Lock def f(l, i):    l.acquire()    try:        print('hello world', i)    finally:        l.release() if __name__ == '__main__':    lock = Lock()     for num in range(10):        Process(target=f, args=(lock, num)).start()　　 進程池　　進程池內(nèi)部維護一個進程序列，當(dāng)使用時，則去進程池中獲取一個進程，如果進程池序列中沒有可供使用的進進程，那么程序就會等待，直到進程池中有可用進程為止。進程池中有兩個方法：applyapply_asyncfrom  multiprocessing import Process,Poolimport time def Foo(i):    time.sleep(2)    return i+100 def Bar(arg):    print('-->exec done:',arg) pool = Pool(5) for i in range(10):    pool.apply_async(func=Foo, args=(i,),callback=Bar)    #pool.apply(func=Foo, args=(i,)) print('end')pool.close()pool.join()#進程池中進程執(zhí)行完畢后再關(guān)閉，如果注釋，那么程序直接關(guān)閉。