为什么ANTLR省略了最后一个令牌*而*不产生错误？

Question

grammar Query;

startExpression
  : WS? expression WS? EOF
  ;

expression
  | maybeDefaultBooleanExpression
  ;

maybeDefaultBooleanExpression
  : defaultBooleanExpression
  | queryFragment
  ;

defaultBooleanExpression
  : nested += queryFragment (WS nested += queryFragment)+
  ;

queryFragment
  : unquotedQuery
  | quotedQuery
  ;

unquotedQuery
  : UNQUOTED
  ;

quotedQuery
  : QUOTED
  ;

UNQUOTED
  : UnquotedStartChar
    UnquotedChar*
  ;

fragment
UnquotedStartChar
  : EscapeSequence
  | ~( ' ' | '\r' | '\t' | '\u000C' | '\n' | '\\' | ':'
     | '"' | '\u201C' | '\u201D'    // DoubleQuote
     | '\'' | '\u2018' | '\u2019'   // SingleQuote
     | '(' | ')' | '[' | ']' | '{' | '}' | '~'
     | '&' | '|' | '!' | '^' | '?' | '*' | '/' | '+' | '-' | '$' )
  ;

fragment
UnquotedChar
  : EscapeSequence
  | ~( ' ' | '\r' | '\t' | '\u000C' | '\n' | '\\' | ':'
     | '"' | '\u201C' | '\u201D'    // DoubleQuote
     | '\'' | '\u2018' | '\u2019'   // SingleQuote
     | '(' | ')' | '[' | ']' | '{' | '}' | '~'
     | '&' | '|' | '!' | '^' | '?' | '*' )
  ;

QUOTED
  : '"'
    QuotedChar*
    '"'
  ;

fragment
QuotedChar
  : ~( '\\'
     |  | '\u201C' | '\u201D'      // DoubleQuote
     | '\r' | '\n' | '?' | '*' )
  ;


WS : ( ' ' | '\r' | '\t' | '\u000C' | '\n' )+;

    CharStream input = CharStreams.fromString("A \"");
    QueryLexer lexer = new QueryLexer(input);
    lexer.removeErrorListeners();
    CommonTokenStream tokens = new CommonTokenStream(lexer);
    System.out.println(tokens.LT(0));
    System.out.println(tokens.LT(1));
    System.out.println(tokens.LT(2));
    System.out.println(tokens.LT(3));

java.lang.StringIndexOutOfBoundsException: String index out of range: 4

    at java.lang.String.checkBounds(String.java:385)
    at java.lang.String.<init>(String.java:462)
    at org.antlr.v4.runtime.CodePointCharStream$CodePoint8BitCharStream.getText(CodePointCharStream.java:160)
    at org.antlr.v4.runtime.Lexer.notifyListeners(Lexer.java:360)
    at org.antlr.v4.runtime.Lexer.nextToken(Lexer.java:144)
    at org.antlr.v4.runtime.BufferedTokenStream.fetch(BufferedTokenStream.java:169)
    at org.antlr.v4.runtime.BufferedTokenStream.sync(BufferedTokenStream.java:152)
    at org.antlr.v4.runtime.CommonTokenStream.LT(CommonTokenStream.java:100)

        QueryParser parser = new QueryParser(tokens);
        parser.removeErrorListeners();
        parser.addErrorListener(LoggingErrorListener.get());
        parser.setErrorHandler(new BailErrorStrategy());

        // Performance hack as per the ANTLR v4 FAQ
        parser.getInterpreter().setPredictionMode(PredictionMode.SLL);
        ParseTree expression;
        try
        {
            expression = parser.startExpression();
        }
        catch (Exception e)
        {
            // It catches a StringIndexOutOfBoundsException here.

            parser.reset();
            parser.getInterpreter().setPredictionMode(PredictionMode.LL);
            expression = parser.startExpression();
        }

我得到：

tokens = {org.antlr.v4.runtime.CommonTokenStream@1811} 
 channel = 0
 tokenSource = {MyQueryLexer@1810} 
 tokens = {java.util.ArrayList@1816}  size = 3
  0 = {org.antlr.v4.runtime.CommonToken@1818} "[@0,0:0='A',<13>,1:0]"
  1 = {org.antlr.v4.runtime.CommonToken@1819} "[@1,1:1=' ',<32>,1:1]"
  2 = {org.antlr.v4.runtime.CommonToken@1820} "[@2,3:2='<EOF>',<-1>,1:3]"
 p = 2
 fetchedEOF = true

expression = {MyQueryParser$StartExpressionContext@1813} "[]"
 children = {java.util.ArrayList@1827}  size = 3
  0 = {MyQueryParser$ExpressionContext@1831} "[87]"
  1 = {org.antlr.v4.runtime.tree.TerminalNodeImpl@1832} " "
  2 = {org.antlr.v4.runtime.tree.TerminalNodeImpl@1833} "<EOF>"

在这里，我本来希望得到一个recognitionexception，但不知何故解析成功了，并且在末尾丢失了令牌数据的无效位。

问题是：

然后我进一步钻探，发现令牌#NextToken()抛出了一个异常，文档使它看起来不应该发生这种情况，所以我最终为此提交了一张罚单。

Answer 1

直到最近的构建，ANTLR4的自适应机制都具有这样的“特性”：如果令牌流中只有一个可行的替代方案，ANTLR4的自适应机制能够从单个令牌丢失和单个额外令牌解析中恢复。现在最近，显然这种行为已经改变了。因此，如果您像我一样使用旧的构建，您仍然会看到自适应解析。也许帕尔和哈威尔会解决这个问题。

像您一样，我认识到需要一个完美的输入流和零解析错误，不管是否被“忽略”。要创建一个“严格的解析器”，请遵循以下步骤：

>

创建一个名为“StricTerrorStrategy that inherit from/extended DefaulTerrorStrategy”的类。您需要重写Recover、RecoverInline和Sync方法。这里的底线是，我们会对任何出错的地方抛出异常，并且在额外的/丢失的令牌后不尝试重新同步代码。下面是我的C#代码，您的java看起来非常相似：

public class StrictErrorStrategy : DefaultErrorStrategy
{
    public override void Recover(Parser recognizer, RecognitionException e)
    {
    IToken token = recognizer.CurrentToken;
    string message = string.Format("parse error at line {0}, position {1} right before {2} ", token.Line, token.Column, GetTokenErrorDisplay(token));
    throw new Exception(message, e);
    }

public override IToken RecoverInline(Parser recognizer)
{
    IToken token = recognizer.CurrentToken;
    string message = string.Format("parse error at line {0}, position {1} right before {2} ", token.Line, token.Column, GetTokenErrorDisplay(token));
    throw new Exception(message, new InputMismatchException(recognizer));
}

public override void Sync(Parser recognizer) { /* do nothing to resync */}
}

创建一个实现单个方法的新lexer：

public class StrictLexer : <YourGeneratedLexerNameHere>
{
   public override void Recover(LexerNoViableAltException e)
    {
        string message = string.Format("lex error after token {0} at position {1}", _lasttoken.Text, e.StartIndex);
        throw new ParseCanceledException(BasicEnvironment.SyntaxError);
    }
}

使用你的lexer和策略：

  AntlrInputStream inputStream = new AntlrInputStream(stream);
  StrictLexer lexer = new BailLexer(inputStream);
  CommonTokenStream tokenStream = new CommonTokenStream(lexer);
  LISBASICParser parser = new LISBASICParser(tokenStream);
  parser.RemoveErrorListeners();
  parser.ErrorHandler = new StrictErrorStrategy();

这是很好的，来自我的一个项目的实际代码，它对语法错误有一个“零容忍规则”。我从特伦斯·帕尔关于ANTLR4的伟大著作中获得了代码和想法。