Marty Schoch 0b171c85da change "simple" analyzer to use "letter" tokenizer ... this change improves compatibility with the simple analyzer defined by Lucene. this has important implications for some perf tests as well as they often use the simple analyzer.		2016-03-31 15:13:17 -04:00
..
character	change "simple" analyzer to use "letter" tokenizer	2016-03-31 15:13:17 -04:00
exception	exception: fail if pattern is empty, name tokenizer in error	2015-10-27 18:53:03 +01:00
letter	change "simple" analyzer to use "letter" tokenizer	2016-03-31 15:13:17 -04:00
regexp_tokenizer	changed whitespace tokenizer to work better on cjk input	2014-09-07 14:11:01 -04:00
single_token	add newline between license and package	2014-09-02 10:54:50 -04:00
unicode	unicode.Tokenize() avoids array growth via array of arrays	2016-01-02 12:21:25 -08:00
web	add support for a "web" tokenizer/analyzer	2015-11-30 14:27:18 -05:00
whitespace_tokenizer	added benchmark for tokenizing English text	2014-10-17 18:07:01 -04:00