异步HTTP库grequests（gevent + requests）

简介

grequests是gevent和requests的封装库，实现了简单的异步HTTP请求。

用法

import grequests

HEADERS = {"User-Agent": "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36"}
urls = ['http://www.qq.com', 'http://www.baidu.com']


def exception_handler(request, exception):
    print(exception)
    print("Request Exception")


def run(url):
    """
    运行
    :param url:
    :return:
    """
    r = grequests.get(url, headers=HEADERS)  # HTTP GET请求，并自定义请求头
    r.encoding = "utf-8"  # 设置请求参数
    return r


reqs = []
for url in urls:
    # 设置线程池链接
    reqs.append(run(url))

# 获得Response对象列表
ls_res = grequests.map(reqs, exception_handler=exception_handler)
for res in ls_res:
    print(res.text)

参考

https://github.com/spyoungtech/grequests
https://github.com/requests/requests-threads
https://github.com/ross/requests-futures