如何有效地删除字符串中连续重复的单词或短语[重复]

Question

我有一个字符串，其中包含重复出现的短语，或者它甚至可能是一个连续出现多次的单词。

尝试了各种方法，但找不到更好的节省时间和空间的方法。

这是我尝试过的方法

1. Groupby（）
2. re

String = "what type of people were most likely to be able to be able to be able to be able to be 1.35 ?"
s1 = " ".join([k for k,v in groupby(String.replace("</Sent>","").split())])
s2 = re.sub(r'\b(.+)(\s+\1\b)+', r'\1', String)

在我的情况下，他们两个似乎都不起作用

我的预期结果：

什么类型的人最有可能成为1.35？

这些是我提到的一些帖子

1. 有没有办法删除字符串中重复和连续的单词/短语？-不工作
2. 我怎么能删除重复的字在一个字符串与Python？-工程部分，但需要一个最佳的方式为大字符串也

请不要在上面的帖子中将我的问题标记为重复，因为我尝试了大多数实现，但没有找到有效的解决方案。

Answer 1

我非常肯定，在这种方法中，Python 3.7中保持了顺序，但我并不完全确定较旧的版本。

String = "what type of people were most likely to be able to be able to be able to be able to be 1.35 ?"
unique_words = dict.fromkeys(String.split())
print(' '.join(unique_words))
>>> what type of people were most likely to be able 1.35 ?

Answer 2

我会采用这种创造性的方法来寻找长度不断增长的复制品：

input = "what type of people were most likely to be able to be able to be able to be able to be 1.35 ?"
def combine_words(input,length):
    combined_inputs = []
    if len(splitted_input)>1:
        for i in range(len(input)-1):
            combined_inputs.append(input[i]+" "+last_word_of(splitted_input[i+1],length)) #add the last word of the right-neighbour (overlapping) sequence (before it has expanded), which is the next word in the original sentence
    return combined_inputs, length+1

def remove_duplicates(input, length):
    bool_broke=False #this means we didn't find any duplicates here
    for i in range(len(input) - length):
        if input[i]==input[i + length]: #found a duplicate piece of sentence!
            for j in range(0,length): #remove the overlapping sequences in reverse order
                del input[i + length - j]
            bool_broke = True
            break #break the for loop as the loop length does not matches the length of splitted_input anymore as we removed elements
    if bool_broke:
        return remove_duplicates(input, length) #if we found a duplicate, look for another duplicate of the same length
    return input

def last_word_of(input,length):
    splitted = input.split(" ")
    if len(splitted)==0:
        return input
    else:
        return splitted[length-1]

#make a list of strings which represent every sequence of word_length adjacent words
splitted_input = input.split(" ")
word_length = 1
splitted_input,word_length = combine_words(splitted_input,word_length)

intermediate_output = False

while len(splitted_input)>1:
    splitted_input = remove_duplicates(splitted_input,word_length) #look whether two sequences of length n (with distance n apart) are equal. If so, remove the n overlapping sequences
    splitted_input, word_length = combine_words(splitted_input,word_length) #make even bigger sequences
    if intermediate_output:
        print(splitted_input)
        print(word_length)
output = splitted_input[0] #In the end you have a list of length 1, with all possible lengths of repetitive words removed

输出流畅的

what type of people were most likely to be able to be 1.35 ?

尽管它不是期望的输出，但我不知道它如何识别删除前面3处出现的“to be”（长度为2）。