如何调整NLTK句子标记器

Question

我正在使用NLTK来分析一些经典文本，并且遇到麻烦按句子标记文本的麻烦。例如，这是我从 Moby
Dick
获得的摘录的内容：

import nltk
sent_tokenize = nltk.data.load('tokenizers/punkt/english.pickle')

'''
(Chapter 16)
A clam for supper? a cold clam; is THAT what you mean, Mrs. Hussey?" says I, "but
that's a rather cold and clammy reception in the winter time, ain't it, Mrs. Hussey?"
'''
sample = 'A clam for supper? a cold clam; is THAT what you mean, Mrs. Hussey?" says I, "but that\'s a rather cold and clammy reception in the winter time, ain\'t it, Mrs. Hussey?"'

print "\n-----\n".join(sent_tokenize.tokenize(sample))
'''
OUTPUT
"A clam for supper?
-----
a cold clam; is THAT what you mean, Mrs.
-----
Hussey?
-----
" says I, "but that\'s a rather cold and clammy reception in the winter time, ain\'t it, Mrs.
-----
Hussey?
-----
"
'''

考虑到梅尔维尔（Melville）的语法有些陈旧，我不希望在这里达到完美，但是NLTK应该能够处理终端双引号和标题，例如“
Mrs”。但是，由于令牌生成器是无监督训练算法的结果，所以我不知道该如何进行修补。

有人建议使用更好的句子标记器吗？我更喜欢可以破解的简单启发式方法，而不必训练自己的解析器。

Answer 1

您需要向令牌生成器提供缩写列表，如下所示：

from nltk.tokenize.punkt import PunktSentenceTokenizer, PunktParameters
punkt_param = PunktParameters()
punkt_param.abbrev_types = set(['dr', 'vs', 'mr', 'mrs', 'prof', 'inc'])
sentence_splitter = PunktSentenceTokenizer(punkt_param)
text = "is THAT what you mean, Mrs. Hussey?"
sentences = sentence_splitter.tokenize(text)

现在的句子是：

['is THAT what you mean, Mrs. Hussey?']

更新：如果句子的最后一个单词附有撇号或引号（例如 Hussey？’
），则此方法不起作用。因此，一种快速而又肮脏的方法是在撇号和引号之前加上空格，并在句子结尾的符号（。！？）之后：

text = text.replace('?"', '? "').replace('!"', '! "').replace('."', '. "')