龙卷风一般睡眠增加延迟

Question

我试图以异步方式在请求之间添加延迟。当我使用龙卷风gen.sleep（x）时，我的函数（启动）不会执行。如果我从产量gen.sleep（1.0）中删除产量，函数被调用，但没有添加延迟。如何在我的for循环中添加请求之间的延迟？我需要控制请求每秒外部API。如果使用time.sleep所有请求完成后，响应将延迟。尝试添加@gen.engine装饰器启动功能，但没有结果。

代码：

import collections
import tornado.httpclient


class BacklogClient(object):
    MAX_CONCURRENT_REQUESTS = 20

    def __init__(self, ioloop):
        self.ioloop = ioloop
        self.client = tornado.httpclient.AsyncHTTPClient(max_clients=self.MAX_CONCURRENT_REQUESTS)
        self.client.configure(None, defaults=dict(connect_timeout=20, request_timeout=30))
        self.backlog = collections.deque()
        self.concurrent_requests = 0

    def __get_callback(self, function):
        def wrapped(*args, **kwargs):
            self.concurrent_requests -= 1
            self.try_run_request()
            return function(*args, **kwargs)

        return wrapped

    def try_run_request(self):
        while self.backlog and self.concurrent_requests < self.MAX_CONCURRENT_REQUESTS:
            request, callback = self.backlog.popleft()
            self.client.fetch(request, callback=callback)
            self.concurrent_requests += 1

    def fetch(self, request, callback=None):
        wrapped = self.__get_callback(callback)

        self.backlog.append((request, wrapped))
        self.try_run_request()


import time
from tornado import ioloop, httpclient, gen


class TornadoBacklog:
    def __init__(self):

        self.queue = 0
        self.debug = 1
        self.toProcess = [
            'http://google.com',
            'http://yahoo.com',
            'http://nytimes.com',
            'http://msn.com',
            'http://cnn.com',
            'http://twitter.com',
            'http://facebook.com',
        ]


    def handle_request(self, response):

        print response.code
        if not self.backlog.backlog and self.backlog.concurrent_requests == 0:
            ioloop.IOLoop.instance().stop()


    def launch(self):
        self.ioloop = ioloop.IOLoop.current()
        self.backlog = BacklogClient(self.ioloop)

        for item in self.toProcess:
            yield gen.sleep(1.0)
            print item
            self.backlog.fetch(
                httpclient.HTTPRequest(
                    item,
                    method='GET',
                    headers=None,
                ),
                self.handle_request
            )

        self.ioloop.start()



def main():
    start_time = time.time()

    scraper = TornadoBacklog()
    scraper.launch()

    elapsed_time = time.time() - start_time
    print('Process took %f seconds processed %d items.' % (elapsed_time, len(scraper.toProcess)))


if __name__ == "__main__":
    main()

参考：https://github.com/tornadoweb/tornado/issues/1400

Answer 1

Tornado协同程序有两个组件：

1. 它们包含“收益率”声明

在您的"启动"功能上使用"协程"装饰器：

@gen.coroutine
def launch(self):

从头到尾运行龙卷风协同路线，如下所示：

tornado.ioloop.IOLoop.current().run_sync(launch)

从你的“启动”函数中删除对“ioloop.start”的调用：循环运行“启动”函数，反之亦然。