Tokenizer

优质
小牛编辑
124浏览
2023-12-01

This class takes a set of highlighting rules, and creates a tokenizer out of them. For more information, see the wiki on extending highlighters.

Constructors

Constructs a new tokenizer based on the given rules and flags.

rulesObjectRequired. The highlighting rules
flagStringRequired. Any additional regular expression flags to pass (like "i" for case insensitive)

Methods

Returns an object containing two properties: tokens, which contains all the tokens; and state, the current state.

lineObjectRequired.
startStateObjectRequired.