使用antlr解析电子邮件

Question

我已经尝试了整整一周使用antlr构建一个语法，允许我解析电子邮件消息。

我的目标不是将整个电子邮件彻底解析为令牌，而是将其分解为相关部分。

这是我必须处理的文档格式。//描述不属于消息一部分的内联注释：

Subject : [SUBJECT_MARKER] + lorem ipsum...
// marks a message that needs to be parsed.
// Subject marker can be something like "help needed", "action required"

Body: 

// irrelevant text we can ignore, discard or skip
Hi George,
Hope you had a good weekend. Another fluff sentence ...
// end of irrelevant text


// beginning of the SECTION_TYPE_1. SECTION_TYPE_1 marker is "answers below:" 
[SECTION_TYPE_1]

Meaningful text block that needs capturing, made of many sentences: Lorem ipsum ...

[SENTENCE_MARKER] - Sentences that needs to be captured.
[SENTENCE_MARKER] - Sentences that needs to be captured.
[SENTENCE_MARKER] - Sentences that needs to be captured.

[SECTION_END_MARKER] // this is "\n\n"

// SENTENCE_MARKER can be "a)", "b)" or anything that is in the form "[a-zA-Z]')'"
// one important requirement is that this SENTENCE_MARKER matches only inside a section. Either SECTION_TYPE_1 or SECTION_TYPE_2


// alternatively instead of [SECTION_TYPE_1] we can have [SECTION_TYPE_2].
// if we have SECTION_TYPE_1 then try to parse SECTION_TYPE_1 else try to parse SECTION_TYPE_2.enter code here

[SECTION_TYPE_2] // beginning of the section type 1;

Meaningful text bloc that needs capturing. Many sentences Lorem ipsum ...

[SENTENCE_MARKER] - Sentences that needs to be captured.
[SENTENCE_MARKER] - Sentences that needs to be captured.
[SENTENCE_MARKER] - Sentences that needs to be captured.

[SECTION_END_MARKER] // same as above

我面临的问题如下：

• 我没有想出一个好方法来跳过消息开头的文本，并在找到标记后才开始应用解析规则。截面类型1
• 捕获节开始和句子标记之间节内的所有文本
• 在SECTION\u结束标记之后，忽略后面的所有文本

Answer 1

Antlr是结构化的、理想情况下结构明确的文本的解析器。除非您的源消息具有相对明确的功能来可靠地标记感兴趣的消息部分，否则Antlr不太可能工作。

更好的方法是使用自然语言处理器（NLP）包来识别每个句子或短语的形式和对象，从而识别感兴趣的句子或短语。斯坦福NLP软件包非常有名（Github）。

使现代化

必要的语法形式如下：

message : subject ( sec1 | sec2 | fluff )* EOF ;

subject : fluff* SUBJECT_MARKER subText EOL ;
subText : ( word | HWS )+ ;

sec1    : ( SECTION_TYPE_1 content )+ SECTION_END_MARKER     ;
sec2    : ( SECTION_TYPE_2 content )+ SECTION_END_MARKER     ;
content : ( word | ws )+ ;

word    : CHAR+ ;
ws      : ( EOL | HWS )+ ;

fluff   : . ;

SUBJECT_MARKER      : 'marker' ;
SECTION_TYPE_1      : 'text1' ;
SECTION_TYPE_2      : 'text2' ;
SENTENCE_MARKER     : [a-zA-Z0-9] ')' ;

EOL                 : '\r'? '\n';
HWS                 : [ \t] ;
CHAR                : . ;

成功将取决于各种标记的明确程度——并且肯定会有歧义。要么修改语法以显式处理歧义，要么推迟到树行走/分析阶段来解决。