C#和ANTLR4:解析文件时处理“include”指令

Question

null

Answer 1

这个问题的一个解决方案是重写扫描程序的行为，特别是NextToken()方法。这是necassary，因为EOF令牌不能由ANTLR lexer语法处理（据我所知），并且任何附加到识别EOF的lexer规则的操作都被忽略（如下面的代码所示）。因此，有必要将此行为直接实现到扫描器方法中。

所以假设我们有一个解析器语法

parser grammar INCParserGrammar;

@parser::members {
        public static Stack<ICharStream> m_nestedfiles = new Stack<ICharStream>();
}

options { tokenVocab = INCLexerGrammar; }

/*
 * Parser Rules
 */

compileUnit
    :   (include_directives | ANY )+ ENDOFFILE
    ;

include_directives : INCLUDEPREFIX FILE DQUOTE
                     ;

应该在语法的成员中引入静态公共堆栈 （即MySpecialFileStack)。该堆栈将用于存储与参与解析的文件相关联的字符流。当使用include语句遇到新文件时，字符流被推送到这个堆栈

   lexer grammar INCLexerGrammar;

   @lexer::header {
    using System;
    using System.IO;
   }

   @lexer::members { 
    string file;
    ICharStream current;
    
   }


/*
 * Lexer Rules
 */
INCLUDEPREFIX : '#include'[ \t]+'"' {                                                 
                                      Mode(INCLexerGrammar.FILEMODE);
                                    };

// The following ruls has always less length matched string that the the rule above
ANY : ~[#]+ ;


ENDOFFILE : EOF { // Any actions in the this rule are ignored by the ANTLR lexer };


////////////////////////////////////////////////////////////////////////////////////////////////////////

mode FILEMODE;
FILE : [a-zA-Z][a-zA-Z0-9_]*'.'[a-zA-Z0-9_]+ {  file= Text;
                                                StreamReader s = new StreamReader(file);
                                                INCParserGrammar.m_nestedfiles.Push(_input);                                                
                                                current =new AntlrInputStream(s);           
                                            
                                             };
DQUOTE: '"'  {  
                this._input = current;
                Mode(INCLexerGrammar.DefaultMode);  };

if (this._input.La(1) == -1)
{
    if ( mySpecialFileStack.Count == 0 )
        this._hitEOF = true;
    else
        this._input = mySpecialFileStack.Pop();
}

NextToken（）方法重写的全部内容是

public override IToken NextToken() {
            int marker = this._input != null ? this._input.Mark() : throw new InvalidOperationException("nextToken requires a non-null input stream.");
            label_3:
            try {
                while (!this._hitEOF) {
                    this._token = (IToken)null;
                    this._channel = 0;
                    this._tokenStartCharIndex = this._input.Index;
                    this._tokenStartCharPositionInLine = this.Interpreter.Column;
                    this._tokenStartLine = this.Interpreter.Line;
                    this._text = (string)null;
                    do {
                        this._type = 0;
                        int num;
                        try {
                            num = this.Interpreter.Match(this._input, this._mode);
                        } catch (LexerNoViableAltException ex) {
                            this.NotifyListeners(ex);
                            this.Recover(ex);
                            num = -3;
                        }

                        if (this._input.La(1) == -1) {
                            if (INCParserGrammar.m_nestedfiles.Count == 0 ) {
                                this._hitEOF = true;
                            }
                            else
                            {
                                this._input = INCParserGrammar.m_nestedfiles.Pop();
                            }
                        }

                        if (this._type == 0)
                            this._type = num;
                        if (this._type == -3)
                            goto label_3;
                    }
                    while (this._type == -2);
                    if (this._token == null)
                        this.Emit();
                    return this._token;
                }
                this.EmitEOF();
                return this._token;
            } finally {
                this._input.Release(marker);
            }
        }

现在，当您在代码中识别出应该解析的文件时，只需添加以下操作

FILE
    : [a-zA-Z][a-zA-Z0-9_]*'.'[a-zA-Z0-9_]+ {
        StreamReader s = new StreamReader(Text);
        mySpecialFileStack.Push(_input);                                                
        _input = new AntlrInputStream(s);                                               
    };
    
DQUOTE: '"'  {  this._input = current;
            Mode(INCLexerGrammar.DefaultMode);  };
//***Warning:***
// Be careful when your file inclusion is enclosed inside quotes or other symbols, or if  
// the filename-to-be-included is not the last token that defines an inclusion: `_input`  
// should only be switched AFTER the inclusion detection is completely found (i.e. after  
// the closing quote has been recognized).  

最后，下面给出了主程序，很明显根文件首先添加到ICharStream堆栈中

 static void Main(string[] args) {
            var a = new StreamReader("./root.txt");
            var antlrInput = new AntlrInputStream(a);
            INCParserGrammar.m_nestedfiles.Push(antlrInput);
            var lexer = new INCLexerGrammar(antlrInput);
            var tokens = new BufferedTokenStream(lexer);
            var parser = new INCParserGrammar(tokens);
            parser.compileUnit();
            
        }