python 线程、进程

内容导读

互联网集市收集整理的这篇技术教程文章主要介绍了python 线程、进程，小编现在分享给大家，供广大互联网技能从业者学习和参考。文章包含1721字，纯文字阅读大概需要3分钟。

内容图文

1、队列、线程

import MySQLdb
import MySQLdb.cursors
import queue
import threading

def update_line_thread():
    connection = MySQLdb.connect(host='xx.xx.xx.xx', port=3306, user='', passwd='', db='db_name',
                             charset='utf8mb4', connect_timeout=100000, cursorclass=MySQLdb.cursors.SSCursor, autocommit=1)
    cursor = connection.cursor()
    sql = 'UPDATE table SET col_name=%s,where id=%s'
    while True:
        stroke,Id= q.get()
        keys = (stroke,Id)
        try:
            cursor.execute(sql, keys)
        except Exception as e:
            print(e,fid)
        q.task_done()
        
        
        
q = queue.Queue()
for i in range(10):
    t = threading.Thread(target=update_line_thread, daemon=True)
    t.start()
for ID in id2stroke:
    q.put((id2stroke[ID], ID))
q.join()

2、进程池 与 线程池

import MySQLdb
import MySQLdb.cursors
import pymongo
from multiprocessing.pool import Pool, ThreadPool

def gen_doc(paper_list):
    with ThreadPool(7) as tp:
        p1 = tp.apply_async(get_paper_info, (paper_list, ))
        p2 = tp.apply_async(get_paper_field, (paper_list, ))
        p3 = tp.apply_async(get_paper_author, (paper_list, ))

        
        paper_info = p1.get()
        paper_field = p2.get()
        paper_author = p3.get()

    
    for line in paper_info:
        yield { "key":"value"}


def update_batch(paper_list):
    conn= pymongo.MongoClient('xx,xx,xx,xx', 27017, username="", password="").db_name.collection_name
    for doc in gen_doc(paper_list):
        # conn.delete_one({'_id':doc['_id']})
        conn.replace_one({'_id': doc['_id']}, doc, upsert=True)


def update_by_list(paper_list):
    paper_list.sort()
    batch_size = 1000
    batch_list = [paper_list[begin:begin+batch_size] for begin in range(0, len(paper_list), batch_size)]
    print("Total batch num: ", len(batch_list))
    
    with Pool(25) as pool:
       # pool.map(update_batch, batch_list, chunksize=200)
        for idx, _ in enumerate(pool.imap_unordered(update_batch, batch_list)):
            if idx % 100 == 0:
                print(idx)

3、concurrent.futures

内容总结

以上是互联网集市为您收集整理的python 线程、进程全部内容，希望文章能够帮你解决python 线程、进程所遇到的程序开发问题。 如果觉得互联网集市技术教程内容还不错，欢迎将互联网集市网站推荐给程序员好友。

内容备注

版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 gblab@vip.qq.com 举报，一经查实，本站将立刻删除。

内容手机端