0
0
bleve/analysis
Marty Schoch 44ff6ced8a improve perf of porter stemmer
1.  porter stemmer offers method to NOT do lowercasing, however
to use this we must convert to runes first ourself, so we did this

2.  now we can invoke the version that skips lowercasing, we
already do this ourselves before stemming through separate filter

due to the fact that the stemmer modifies the runes in place
we have no way to know if there were changes, thus we must
always encode back into the term byte slice

added unit test which catches the problem found

NOTE this uses analysis.BuildTermFromRunes so perf gain is
only visible with other PR also merged

future gains are possible if we udpate the stemmer to let us
know if changes were made, thus skipping re-encoding to
[]byte when no changes were actually made
2016-09-11 20:13:15 -04:00
..
analyzers change "simple" analyzer to use "letter" tokenizer 2016-03-31 15:13:17 -04:00
char_filters add newline between license and package 2014-09-02 10:54:50 -04:00
datetime_parsers major refactor of bleve configuration 2015-09-16 17:10:59 -04:00
language fix another test issue 2016-06-10 13:21:27 -04:00
token_filters improve perf of porter stemmer 2016-09-11 20:13:15 -04:00
token_map token_map: document it along with stop_token_filter 2015-11-05 14:07:54 +01:00
tokenizers updated whtitepsace to behave more like lucene/es 2016-06-10 15:30:43 -04:00
benchmark_test.go analyze locations only if includeTermVectors enabled 2016-01-05 12:46:46 -08:00
freq_test.go gofmt simplifications 2016-04-02 21:54:33 -04:00
freq.go copy locations on merge for more safe/predictable behavior 2016-01-19 14:21:48 -05:00
test_words.txt major refactor of analysis files, now wired up to registry 2014-08-13 21:14:47 -04:00
token_map_test.go fix issues identified by errcheck 2015-04-07 14:52:00 -04:00
token_map.go added some godoc documentation for the en analyzer 2015-11-18 15:28:57 +13:00
type.go Implemented boolean field support 2016-01-11 17:18:03 -08:00
util_test.go add newline between license and package 2014-09-02 10:54:50 -04:00
util.go fix issues with lucene stemmer 2015-03-11 11:14:29 -04:00