通过Python函数跟踪*最大*内存使用情况

Question

我想找出在调用函数期间（在Python中）分配的最大RAM量是多少。还有其他与跟踪RAM使用情况有关的问题：

建议使用哪个Python内存分析器？

如何剖析Python中的内存使用情况？

但是这些似乎允许您在heap()调用方法（例如guppy）时更多地跟踪内存使用情况。但是，我要跟踪的是外部库中的一个函数，该函数无法修改，并且会逐渐占用大量RAM，但是一旦函数执行完成，就可以释放它。有什么方法可以找出函数调用期间使用的RAM总量是多少？

Answer 1

这个问题似乎很有趣，它使我有理由研究Guppy / Heapy，对此我表示感谢。

我尝试了大约2个小时，让Heapy监视函数调用/进程，而又没有 零 运气地修改其源代码。

我确实找到了一种使用内置Python库完成任务的方法resource。请注意，文档没有指出RU_MAXRSS值返回的内容。另一个SO用户注意到它在kB中。运行Mac
OSX 7.3并看着我的系统资源在下面的测试代码中攀升，我相信返回的值以 字节为单位 ，而不是千 字节 。

我如何使用resource库来监视库调用的10000英尺视图是在一个单独的（可监视的）线程中启动该函数，并在主线程中跟踪该进程的系统资源。下面是您需要运行以对其进行测试的两个文件。

图书馆资源监视器 -
whatever_you_want.py

import resource
import time

from stoppable_thread import StoppableThread


class MyLibrarySniffingClass(StoppableThread):
    def __init__(self, target_lib_call, arg1, arg2):
        super(MyLibrarySniffingClass, self).__init__()
        self.target_function = target_lib_call
        self.arg1 = arg1
        self.arg2 = arg2
        self.results = None

    def startup(self):
        # Overload the startup function
        print "Calling the Target Library Function..."

    def cleanup(self):
        # Overload the cleanup function
        print "Library Call Complete"

    def mainloop(self):
        # Start the library Call
        self.results = self.target_function(self.arg1, self.arg2)

        # Kill the thread when complete
        self.stop()

def SomeLongRunningLibraryCall(arg1, arg2):
    max_dict_entries = 2500
    delay_per_entry = .005

    some_large_dictionary = {}
    dict_entry_count = 0

    while(1):
        time.sleep(delay_per_entry)
        dict_entry_count += 1
        some_large_dictionary[dict_entry_count]=range(10000)

        if len(some_large_dictionary) > max_dict_entries:
            break

    print arg1 + " " +  arg2
    return "Good Bye World"

if __name__ == "__main__":
    # Lib Testing Code
    mythread = MyLibrarySniffingClass(SomeLongRunningLibraryCall, "Hello", "World")
    mythread.start()

    start_mem = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
    delta_mem = 0
    max_memory = 0
    memory_usage_refresh = .005 # Seconds

    while(1):
        time.sleep(memory_usage_refresh)
        delta_mem = (resource.getrusage(resource.RUSAGE_SELF).ru_maxrss) - start_mem
        if delta_mem > max_memory:
            max_memory = delta_mem

        # Uncomment this line to see the memory usuage during run-time 
        # print "Memory Usage During Call: %d MB" % (delta_mem / 1000000.0)

        # Check to see if the library call is complete
        if mythread.isShutdown():
            print mythread.results
            break;

    print "\nMAX Memory Usage in MB: " + str(round(max_memory / 1000.0, 3))

停止的主题 -
stoppable_thread.py

import threading
import time

class StoppableThread(threading.Thread):
    def __init__(self):
        super(StoppableThread, self).__init__()
        self.daemon = True
        self.__monitor = threading.Event()
        self.__monitor.set()
        self.__has_shutdown = False

    def run(self):
        '''Overloads the threading.Thread.run'''
        # Call the User's Startup functions
        self.startup()

        # Loop until the thread is stopped
        while self.isRunning():
            self.mainloop()

        # Clean up
        self.cleanup()

        # Flag to the outside world that the thread has exited
        # AND that the cleanup is complete
        self.__has_shutdown = True

    def stop(self):
        self.__monitor.clear()

    def isRunning(self):
        return self.__monitor.isSet()

    def isShutdown(self):
        return self.__has_shutdown


    ###############################
    ### User Defined Functions ####
    ###############################

    def mainloop(self):
        '''
        Expected to be overwritten in a subclass!!
        Note that Stoppable while(1) is handled in the built in "run".
        '''
        pass

    def startup(self):
        '''Expected to be overwritten in a subclass!!'''
        pass

    def cleanup(self):
        '''Expected to be overwritten in a subclass!!'''
        pass