Marty Schoch 964b87f76e added rune tokenizer ... not used directly right now, but basis for other simple tokenizers		2014-08-07 22:14:26 -04:00
..
regexp_tokenizer	introduced token type	2014-07-31 13:54:12 -04:00
rune_tokenizer	added rune tokenizer	2014-08-07 22:14:26 -04:00
single_token	introduced token type	2014-07-31 13:54:12 -04:00
unicode_word_boundary	introduced token type	2014-07-31 13:54:12 -04:00