有没有办法删除字符串中重复和连续的单词/短语？

Question

有没有办法删除字符串中重复和连续的单词/短语？例如。

[输入]：foo foo bar foo bar

[out]：foobar foobar

我试过这个：

>>> s = 'this is a foo bar bar black sheep , have you any any wool woo , yes sir yes sir three bag woo wu wool'
>>> [i for i,j in zip(s.split(),s.split()[1:]) if i!=j]
['this', 'is', 'a', 'foo', 'bar', 'black', 'sheep', ',', 'have', 'you', 'any', 'wool', 'woo', ',', 'yes', 'sir', 'yes', 'sir', 'three', 'bag', 'woo', 'wu']
>>> " ".join([i for i,j in zip(s.split(),s.split()[1:]) if i!=j]+[s.split()[-1]])
'this is a foo bar black sheep , have you any wool woo , yes sir yes sir three bag woo wu'

当它变得更复杂，我想删除短语（假设短语最多由5个单词组成）时会发生什么？怎样才能做到呢？例如。

[输入]：foobar foobar foobar

[out]：foo bar

另一个例子：

[in]：这是一个句子这是一个短语重复的句子短语重复的地方。句子不是句子。

这是一个短语重复的句子。句子不是短语。

Answer 1

txt1 = 'this is a foo bar bar black sheep , have you any any wool woo , yes sir yes sir three bag woo wu wool'
txt2 =  'this is a sentence sentence sentence this is a sentence where phrases phrases duplicate where phrases duplicate'

def remove_duplicates(txt):
    result = []
    for word in txt.split():
        if word not in result:
            result.append(word)
    return ' '.join(result)

输出：

In [7]: remove_duplicate_words(txt1)                                                                                                                                  
Out[7]: 'this is a foo bar black sheep , have you any wool woo yes sir three bag wu'                                                                                  

In [8]: remove_duplicate_words(txt2)                                                                                                                                 
Out[8]: 'this is a sentence where phrases duplicate' 

Answer 2

我喜欢itertools。好像每次我想写东西的时候，itertools都已经有了。在这种情况下，groupby获取一个列表，并将该列表中重复的、连续的项分组到的元组中（项值、迭代器值）。在这里使用它，就像：

>>> s = 'this is a foo bar bar black sheep , have you any any wool woo , yes sir yes sir three bag woo wu wool'
>>> ' '.join(item[0] for item in groupby(s.split()))
'this is a foo bar black sheep , have you any wool woo , yes sir yes sir three bag woo wu wool'

让我们用一个函数来扩展它，该函数返回一个列表，并删除重复值：

from itertools import chain, groupby

def dedupe(lst):
    return list(chain(*[item[0] for item in groupby(lst)]))

这对一个单词的短语很好，但对较长的短语没有帮助。要怎么做？好的，首先，我们要检查较长的短语，跨过我们的原始短语：

def stride(lst, offset, length):
    if offset:
        yield lst[:offset]

    while True:
        yield lst[offset:offset + length]
        offset += length
        if offset >= len(lst):
            return

现在我们在做饭！好啊因此，我们的策略是首先删除所有单个单词的重复项。接下来，我们将删除两个重复的单词，从偏移量0开始，然后是1。在此之后，从偏移量0、1和2开始的三个单词重复，依此类推，直到我们找到五个单词重复：

def cleanse(list_of_words, max_phrase_length):
    for length in range(1, max_phrase_length + 1):
        for offset in range(length):
            list_of_words = dedupe(stride(list_of_words, offset, length))

    return list_of_words

总而言之：

from itertools import chain, groupby

def stride(lst, offset, length):
    if offset:
        yield lst[:offset]

    while True:
        yield lst[offset:offset + length]
        offset += length
        if offset >= len(lst):
            return

def dedupe(lst):
    return list(chain(*[item[0] for item in groupby(lst)]))

def cleanse(list_of_words, max_phrase_length):
    for length in range(1, max_phrase_length + 1):
        for offset in range(length):
            list_of_words = dedupe(stride(list_of_words, offset, length))

    return list_of_words

a = 'this is a sentence sentence sentence this is a sentence where phrases phrases duplicate where phrases duplicate . sentence are not prhases .'

b = 'this is a sentence where phrases duplicate . sentence are not prhases .'

print ' '.join(cleanse(a.split(), 5)) == b

Answer 3

您可以使用re模块来实现这一点。

>>> s = 'foo foo bar bar'
>>> re.sub(r'\b(.+)\s+\1\b', r'\1', s)
'foo bar'

>>> s = 'foo bar foo bar foo bar'
>>> re.sub(r'\b(.+)\s+\1\b', r'\1', s)
'foo bar foo bar'

如果要匹配任意数量的连续事件：

>>> s = 'foo bar foo bar foo bar'
>>> re.sub(r'\b(.+)(\s+\1\b)+', r'\1', s)
'foo bar'    

编辑最后一个例子的补充。要做到这一点，您必须在存在重复短语时调用re.sub。因此：

>>> s = 'this is a sentence sentence sentence this is a sentence where phrases phrases duplicate where phrases duplicate'
>>> while re.search(r'\b(.+)(\s+\1\b)+', s):
...   s = re.sub(r'\b(.+)(\s+\1\b)+', r'\1', s)
...
>>> s
'this is a sentence where phrases duplicate'