3 Commits

Author	SHA1	Message	Date
Marty Schoch	fedb46269e	updated whtitepsace to behave more like lucene/es	2016-06-10 15:30:43 -04:00
Marty Schoch	dcb90ad176	added benchmark for tokenizing English text	2014-10-17 18:07:01 -04:00
Marty Schoch	cb5ccd2b1d	fix whitespace tokenizer ... previously would fail to split ascii running into ideographic	2014-09-11 10:38:02 -04:00