替换文本文件中的标记列表的最佳方式

Question

我有一个文本文件（没有标点符号），文件大小约为100MB-1GB，以下是一些示例行：

please check in here
i have a full hd movie
see you again bye bye
press ctrl c to copy text to clipboard
i need your help
...

以及替换标记列表，如下所示:

check in -> check_in
full hd -> full_hd
bye bye -> bye_bye
ctrl c -> ctrl_c
...

在文本文件上替换后我想要的输出如下所示：

please check_in here
i have a full_hd movie
see you again bye_bye
press ctrl_c to copy text to clipboard
i need your help
...

replace_tokens = {'ctrl c': 'ctrl_c', ...} # a python dictionary
for line in open('text_file'):
  for token in replace_tokens:
    line = re.sub(r'\b{}\b'.format(token), replace_tokens[token])
    # Save line to file

该解决方案正在工作，但对于大量替换标记和大型文本文件，速度非常慢。有更好的解决方案吗？

Answer 1

>

正如其他人所建议的那样，生成单个正则表达式将消除内部循环。

regex = re.compile("|".join(r"\b{}\b".format(t) for t in tokens))

< code>re2库比内置的< code>re要快得多，尤其是在有大量标记和/或大量文本的情况下。

regex = re2.compile("|".join(r"\b{}\b".format(t) for t in tokens))

根据内存量和文件的可能大小，尝试一次读取整个内容可能是值得的，而不是一行一行地读取。尤其是如果行很短，即使您实际上没有进行任何面向行的处理，处理这些行也可能会消耗大量时间。

text = f.read()
text = regex.sub(lambda m: tokens[m.group(0)], text)

进一步的改进是使用< code > find all /< code > finditer 而不是< code>sub，然后使用< code > start /< code > end 偏移量输出原始文件的各个部分，与替换部分交错；这将避免在内存中有文本的两个副本。

text = f.read()
pos = 0
for m in regex.finditer(text):
    out_f.write(text[pos:m.start(0)])
    out_f.write(tokens[m.group(0)])
    pos = m.end(0)
out_f.write(text[pos:])

如果您的文本是换行的，您可能还需要考虑是否需要替换短语在一行中断开的实例；这可以通过“将整个文本读入内存”方法轻松完成。如果您需要这样做，但文件太大而无法读入内存，您可能需要改用面向单词的方法——一个函数读取文件并生成单个单词，另一个函数执行面向单词的有限状态机。

Answer 2

您至少可以通过以下方式消除内部循环的复杂性:

import re 

tokens={"check in":"check_in", "full hd":"full_hd",
"bye bye":"bye_bye","ctrl c":"ctrl_c"}

regex=re.compile("|".join([r"\b{}\b".format(t) for t in tokens]))

with open(your_file) as f:
    for line in f:
        line=regex.sub(lambda m: tokens[m.group(0)], line.rstrip())
        print(line)

打印：

please check_in here
i have a full_hd movie
see you again bye_bye
press ctrl_c to copy text to clipboard
i need your help

Answer 3

使用二进制文件和字符串替换，如下所示

• 将文件作为二进制文件处理，以减少文件转换的开销
• 使用字符串替换而不是正则表达式

代码

def process_binary(filename):
    """ Replace strings using binary and string replace
        Processing follows original code flow except using
        binary files and string replace """

    # Map using binary strings
    replace_tokens = {b'ctrl c': b'ctrl_c', b'full hd': b'full_hd', b'bye bye': b'bye_bye', b'check in': b'check_in'}

    outfile = append_id(filename, 'processed')

    with open(filename, 'rb') as fi, open(outfile, 'wb') as fo:
        for line in fi:
            for token in replace_tokens:
                line = line.replace(token, replace_tokens[token])
            fo.write(line)

def append_id(filename, id):
    " Convenience handler for generating name of output file "
    return "{0}_{2}.{1}".format(*filename.rsplit('.', 1) + [id])

性能比较

在124兆字节的文件上(通过复制发布的字符串生成):

• 发布的解决方案：82.8秒
• 避免正则表达式中的内部循环（DAWG post）：28.2秒
• 当前解决方案：9.5秒

当前解决方案：

• 与发布的解决方案相比提高了约 8.7 倍，并且
• 通过正则表达式约 3 倍（避免内循环）

总体趋势

测试代码

# Generate Data by replicating posted string
s = """please check in here
i have a full hd movie
see you again bye bye
press ctrl c to copy text to clipboard
i need your help
"""
with open('test_data.txt', 'w') as fo:
    for i in range(1000000):  # Repeat string 1M times
        fo.write(s)

# Time Posted Solution
from time import time
import re

def posted(filename):
    replace_tokens = {'ctrl c': 'ctrl_c', 'full hd': 'full_hd', 'bye bye': 'bye_bye', 'check in': 'check_in'}

    outfile = append_id(filename, 'posted')
    with open(filename, 'r') as fi, open(outfile, 'w') as fo:
        for line in fi:
            for token in replace_tokens:
                line = re.sub(r'\b{}\b'.format(token), replace_tokens[token], line)
            fo.write(line)

def append_id(filename, id):
    return "{0}_{2}.{1}".format(*filename.rsplit('.', 1) + [id])

t0 = time()
posted('test_data.txt')
print('Elapsed time: ', time() - t0)
# Elapsed time:  82.84100198745728

# Time Current Solution
from time import time

def process_binary(filename):
    replace_tokens = {b'ctrl c': b'ctrl_c', b'full hd': b'full_hd', b'bye bye': b'bye_bye', b'check in': b'check_in'}

    outfile = append_id(filename, 'processed')
    with open(filename, 'rb') as fi, open(outfile, 'wb') as fo:
        for line in fi:
            for token in replace_tokens:
                line = line.replace(token, replace_tokens[token])
            fo.write(line)

def append_id(filename, id):
    return "{0}_{2}.{1}".format(*filename.rsplit('.', 1) + [id])


t0 = time()
process_binary('test_data.txt')
print('Elapsed time: ', time() - t0)
# Elapsed time:  9.593998670578003

# Time Processing using Regex 
# Avoiding inner loop--see dawg posted answer

import re 

def process_regex(filename):
    tokens={"check in":"check_in", "full hd":"full_hd",
    "bye bye":"bye_bye","ctrl c":"ctrl_c"}

    regex=re.compile("|".join([r"\b{}\b".format(t) for t in tokens]))

    outfile = append_id(filename, 'regex')
    with open(filename, 'r') as fi, open(outfile, 'w') as fo:
        for line in fi:
            line = regex.sub(lambda m: tokens[m.group(0)], line)
            fo.write(line)

def append_id(filename, id):
    return "{0}_{2}.{1}".format(*filename.rsplit('.', 1) + [id])

t0 = time()
process_regex('test_data.txt')
print('Elapsed time: ', time() - t0)
# Elapsed time:  28.27900242805481