964b87f76e
not used directly right now, but basis for other simple tokenizers |
||
---|---|---|
.. | ||
regexp_tokenizer | ||
rune_tokenizer | ||
single_token | ||
unicode_word_boundary |
964b87f76e
not used directly right now, but basis for other simple tokenizers |
||
---|---|---|
.. | ||
regexp_tokenizer | ||
rune_tokenizer | ||
single_token | ||
unicode_word_boundary |