Steve Yen 325a616993 unicode.Tokenize() avoids array growth via array of arrays		2016-01-02 12:21:25 -08:00
..
exception	exception: fail if pattern is empty, name tokenizer in error	2015-10-27 18:53:03 +01:00
regexp_tokenizer	changed whitespace tokenizer to work better on cjk input	2014-09-07 14:11:01 -04:00
single_token	add newline between license and package	2014-09-02 10:54:50 -04:00
unicode	unicode.Tokenize() avoids array growth via array of arrays	2016-01-02 12:21:25 -08:00
web	add support for a "web" tokenizer/analyzer	2015-11-30 14:27:18 -05:00
whitespace_tokenizer	added benchmark for tokenizing English text	2014-10-17 18:07:01 -04:00