Tokenizer
优质
小牛编辑
115浏览
2023-12-01
This class takes a set of highlighting rules, and creates a tokenizer out of them. For more information, see the wiki on extending highlighters.
Constructors
Constructs a new tokenizer based on the given rules and flags.
| rules | Object | Required. The highlighting rules |
| flag | String | Required. Any additional regular expression flags to pass (like "i" for case insensitive) |
Methods
Returns an object containing two properties: tokens, which contains all the tokens; and state, the current state.
| line | Object | Required. |
| startState | Object | Required. |