Python中重定向文件流的推荐方法？

Question

我正在为一个断断续续变化的sqlite数据库编写一个备份脚本。现在是这样的：

from bz2 import BZ2File
from datetime import datetime
from os.path import dirname, abspath, join
from hashlib import sha512
def backup_target_database(target):
    backup_dir = dirname(abspath(target))
    hash_file = join(backup_dir, 'last_hash')
    new_hash = sha512(open(target, 'rb').read()).digest()
    if new_hash != open(hash_file, 'rb').read():
        fmt = '%Y%m%d-%H%M.sqlite3.bz2'
        snapshot_file = join(backup_dir, datetime.now().strftime(fmt))
        BZ2File(snapshot_file, 'wb').write(open(target, 'rb').read())
        open(hash_file, 'wb').write(new_hash)

目前数据库的重量仅为20MB，因此在运行并将整个文件读入内存时（在检测到更改时执行两次）不会太费力，但我不想等到这成为一个问题。

使用Bashscript术语进行这种流管道的正确方法是什么？

Answer 1

首先，您的代码中存在重复（读取目标文件两次）。

您可以将shutil.copyfileobj和hashlib.update用于节省内存的例程。

from bz2 import BZ2File
from datetime import datetime
from hashlib import sha512
from os.path import dirname, abspath, join
from shutil import copyfileobj

def backup_target_database(target_path):
    backup_dir = dirname(abspath(target_path))
    hash_path = join(backup_dir, 'last_hash')
    old_hash = open(hash_path, 'rb').read()
    hasher = sha512()
    with open(target_path, 'rb') as target:
        while True:
            data = target.read(1024)
            if not data:
                break
            hasher.update(data)
        new_hash = hasher.digest()
    if new_hash != old_hash:
        fmt = '%Y%m%d-%H%M.sqlite3.bz2'
        snapshot_path = join(backup_dir, datetime.now().strftime(fmt))
        with open(target_path, 'rb') as target:
            with BZ2File(snapshot_path, 'wb', compresslevel=9) as snapshot:
                copyfileobj(target, snapshot)

（注意：我没有测试此代码。如果您有问题，请通知我）