Tokenizer
优质
小牛编辑
124浏览
2023-12-01
This class takes a set of highlighting rules, and creates a tokenizer out of them. For more information, see the wiki on extending highlighters.
Constructors
Constructs a new tokenizer based on the given rules and flags.
rules | Object | Required. The highlighting rules |
flag | String | Required. Any additional regular expression flags to pass (like "i" for case insensitive) |
Methods
Returns an object containing two properties: tokens
, which contains all the tokens; and state
, the current state.
line | Object | Required. |
startState | Object | Required. |