如何使用具有长时间运行函数的python [e-mail]

Question

我有一个web应用程序，也在做真正密集的数据处理。有些函数非常慢（想想几分钟）。

# code that is using to much memory because of the new threads being spawned 
def handler():
   thread = Thread(target = really_slow_function)
   thread.start()
   thread.join()
   return "done"

def really_slow_function():
   # this is an example of an intensive function
   # which should be treated as a blackbox
   sleep(100)
   return "done"

#code that doesn't scale because all the requests are block on that one slow request.
@gen.coroutine
def handler():
   yield really_slow_function()
   raise gen.Return("done")

def really_slow_function():
   # this is an example of an intensive function
   # which should be treated as a blackbox
   sleep(100)
   return "done"

这个重构的问题是tornado服务器阻塞了really_slow_function，同时无法为其他请求提供服务。

所以问题是：有没有一种方法可以重构处理程序，而不触及really_slow_function，也不创建新的线程/进程？

Answer 1

使用ThreadPoolExecutor（来自Concurrent.Futures包）在单独的线程中运行长时间运行的函数，而无需每次启动新线程。

async def handler():
    await IOLoop.current().run_in_executor(None, really_slow_function)
    return "done"

如果您想准确控制有多少线程有资格运行此函数，您可以创建自己的执行器并传递它，而不是none。