gibheer
/
bleve
0
0
Fork 0
Code
bleve/config.go

394 lines
22 KiB
Go
Raw Normal View History

major refactor, apologies for the large commit removed analyzers (these are now built as needed through config) removed html chacter filter (now built as needed through config) added missing license header changed constructor signature of filters that cannot return errors filter constructors that can have errors, now have Must variant which panics change cdl2 tokenizer into filter (should only see lower-case input) new top level index api, closes #5 refactored index tests to not rely directly on analyzers moved query objects to top-level new top level search api, closes #12 top score collector allows skipping results index mapping supports _all by default, closes #3 and closes #6 index mapping supports disabled sections, closes #7 new http sub package with reusable http.Handler's, closes #22
2014-07-30 18:30:38 +02:00
// Copyright (c) 2014 Couchbase, Inc.
// Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file
// except in compliance with the License. You may obtain a copy of the License at
// http://www.apache.org/licenses/LICENSE-2.0
// Unless required by applicable law or agreed to in writing, software distributed under the
// License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
// either express or implied. See the License for the specific language governing permissions
// and limitations under the License.
package bleve
import (
"fmt"
"regexp"
added support for date range fields and queries closes #9 and closes #11
2014-08-03 23:19:04 +02:00
"time"
major refactor, apologies for the large commit removed analyzers (these are now built as needed through config) removed html chacter filter (now built as needed through config) added missing license header changed constructor signature of filters that cannot return errors filter constructors that can have errors, now have Must variant which panics change cdl2 tokenizer into filter (should only see lower-case input) new top level index api, closes #5 refactored index tests to not rely directly on analyzers moved query objects to top-level new top level search api, closes #12 top score collector allows skipping results index mapping supports _all by default, closes #3 and closes #6 index mapping supports disabled sections, closes #7 new http sub package with reusable http.Handler's, closes #22
2014-07-30 18:30:38 +02:00
"github.com/couchbaselabs/bleve/analysis"
added support for date range fields and queries closes #9 and closes #11
2014-08-03 23:19:04 +02:00
"github.com/couchbaselabs/bleve/analysis/datetime_parsers/flexible_go"
major refactor, apologies for the large commit removed analyzers (these are now built as needed through config) removed html chacter filter (now built as needed through config) added missing license header changed constructor signature of filters that cannot return errors filter constructors that can have errors, now have Must variant which panics change cdl2 tokenizer into filter (should only see lower-case input) new top level index api, closes #5 refactored index tests to not rely directly on analyzers moved query objects to top-level new top level search api, closes #12 top score collector allows skipping results index mapping supports _all by default, closes #3 and closes #6 index mapping supports disabled sections, closes #7 new http sub package with reusable http.Handler's, closes #22
2014-07-30 18:30:38 +02:00
"github.com/couchbaselabs/bleve/analysis/char_filters/regexp_char_filter"
"github.com/couchbaselabs/bleve/analysis/tokenizers/regexp_tokenizer"
"github.com/couchbaselabs/bleve/analysis/tokenizers/single_token"
"github.com/couchbaselabs/bleve/analysis/tokenizers/unicode_word_boundary"
added apostrophe filter to improve turkish analyzer closes #27
2014-08-06 14:50:00 +02:00
"github.com/couchbaselabs/bleve/analysis/token_filters/apostrophe_filter"
added arabic normalizer closes #63
2014-08-12 00:35:35 +02:00
"github.com/couchbaselabs/bleve/analysis/token_filters/arabic_normalize"
major refactor, apologies for the large commit removed analyzers (these are now built as needed through config) removed html chacter filter (now built as needed through config) added missing license header changed constructor signature of filters that cannot return errors filter constructors that can have errors, now have Must variant which panics change cdl2 tokenizer into filter (should only see lower-case input) new top level index api, closes #5 refactored index tests to not rely directly on analyzers moved query objects to top-level new top level search api, closes #12 top score collector allows skipping results index mapping supports _all by default, closes #3 and closes #6 index mapping supports disabled sections, closes #7 new http sub package with reusable http.Handler's, closes #22
2014-07-30 18:30:38 +02:00
"github.com/couchbaselabs/bleve/analysis/token_filters/cld2"
added elision filter defined article word maps for french, italian, irish and catalan defined elision filters for these same languages updated analyers for french and italian to use this new filter irish and catalan still depend on other missing pieces closes #25
2014-08-04 01:17:35 +02:00
"github.com/couchbaselabs/bleve/analysis/token_filters/elision_filter"
added german normalizer updated german analyzer to use this normalizer closes #65
2014-08-12 01:25:37 +02:00
"github.com/couchbaselabs/bleve/analysis/token_filters/german_normalize"
added hindi normalizer closes #64
2014-08-12 01:51:47 +02:00
"github.com/couchbaselabs/bleve/analysis/token_filters/hindi_normalize"
added hindi stemmer closes #40
2014-08-12 04:29:47 +02:00
"github.com/couchbaselabs/bleve/analysis/token_filters/hindi_stemmer_filter"
major refactor, apologies for the large commit removed analyzers (these are now built as needed through config) removed html chacter filter (now built as needed through config) added missing license header changed constructor signature of filters that cannot return errors filter constructors that can have errors, now have Must variant which panics change cdl2 tokenizer into filter (should only see lower-case input) new top level index api, closes #5 refactored index tests to not rely directly on analyzers moved query objects to top-level new top level search api, closes #12 top score collector allows skipping results index mapping supports _all by default, closes #3 and closes #6 index mapping supports disabled sections, closes #7 new http sub package with reusable http.Handler's, closes #22
2014-07-30 18:30:38 +02:00
"github.com/couchbaselabs/bleve/analysis/token_filters/length_filter"
"github.com/couchbaselabs/bleve/analysis/token_filters/lower_case_filter"
added persian normalizer closes #67
2014-08-12 00:15:41 +02:00
"github.com/couchbaselabs/bleve/analysis/token_filters/persian_normalize"
added sorani normalizer and stemmer, now have analyzer closes #43
2014-08-08 15:38:28 +02:00
"github.com/couchbaselabs/bleve/analysis/token_filters/sorani_normalize"
"github.com/couchbaselabs/bleve/analysis/token_filters/sorani_stemmer_filter"
major refactor, apologies for the large commit removed analyzers (these are now built as needed through config) removed html chacter filter (now built as needed through config) added missing license header changed constructor signature of filters that cannot return errors filter constructors that can have errors, now have Must variant which panics change cdl2 tokenizer into filter (should only see lower-case input) new top level index api, closes #5 refactored index tests to not rely directly on analyzers moved query objects to top-level new top level search api, closes #12 top score collector allows skipping results index mapping supports _all by default, closes #3 and closes #6 index mapping supports disabled sections, closes #7 new http sub package with reusable http.Handler's, closes #22
2014-07-30 18:30:38 +02:00
"github.com/couchbaselabs/bleve/analysis/token_filters/stemmer_filter"
"github.com/couchbaselabs/bleve/analysis/token_filters/stop_words_filter"
added token truncate filter closes #49
2014-08-07 02:39:42 +02:00
"github.com/couchbaselabs/bleve/analysis/token_filters/truncate_token_filter"
added unicode normalization filter
2014-08-05 03:59:57 +02:00
"github.com/couchbaselabs/bleve/analysis/token_filters/unicode_normalize"
major refactor, apologies for the large commit removed analyzers (these are now built as needed through config) removed html chacter filter (now built as needed through config) added missing license header changed constructor signature of filters that cannot return errors filter constructors that can have errors, now have Must variant which panics change cdl2 tokenizer into filter (should only see lower-case input) new top level index api, closes #5 refactored index tests to not rely directly on analyzers moved query objects to top-level new top level search api, closes #12 top score collector allows skipping results index mapping supports _all by default, closes #3 and closes #6 index mapping supports disabled sections, closes #7 new http sub package with reusable http.Handler's, closes #22
2014-07-30 18:30:38 +02:00
"github.com/couchbaselabs/bleve/search"
)
type AnalysisConfig struct {
refactored StopWordsMap into WordMap so it can be reused the ElisionFilter will need a word list of articles and plan to reuse this
2014-08-03 23:46:35 +02:00
TokenMaps map[string]analysis.WordMap
added support for date range fields and queries closes #9 and closes #11
2014-08-03 23:19:04 +02:00
CharFilters map[string]analysis.CharFilter
Tokenizers map[string]analysis.Tokenizer
TokenFilters map[string]analysis.TokenFilter
Analyzers map[string]*analysis.Analyzer
DateTimeParsers map[string]analysis.DateTimeParser
major refactor, apologies for the large commit removed analyzers (these are now built as needed through config) removed html chacter filter (now built as needed through config) added missing license header changed constructor signature of filters that cannot return errors filter constructors that can have errors, now have Must variant which panics change cdl2 tokenizer into filter (should only see lower-case input) new top level index api, closes #5 refactored index tests to not rely directly on analyzers moved query objects to top-level new top level search api, closes #12 top score collector allows skipping results index mapping supports _all by default, closes #3 and closes #6 index mapping supports disabled sections, closes #7 new http sub package with reusable http.Handler's, closes #22
2014-07-30 18:30:38 +02:00
}
type HighlightConfig struct {
Highlighters map[string]search.Highlighter
}
introduced a config option to disable creating indexes if they don't already exist closes #23 and closes #24
2014-07-30 20:29:26 +02:00
type Configuration struct {
added support for date range fields and queries closes #9 and closes #11
2014-08-03 23:19:04 +02:00
Analysis *AnalysisConfig
DefaultAnalyzer *string
Highlight *HighlightConfig
DefaultHighlighter *string
CreateIfMissing bool
DefaultDateTimeFormat *string
added support for a default field this works at the config and index mapping levels
2014-08-06 14:23:29 +02:00
DefaultField *string
further streamlined the API introduced concept of byte array converters right now only wired up to top-level index mapping allowing the removal of the JSON methods, now at the top level we default to parsing []byte as JSON, override if thats not the behavior you want. future enhancements will allow use of these byte array converters to control how byte arrays are handled elsewhere in documents this would allow for handing binary attachments, etc in the future closes #59
2014-08-11 18:47:29 +02:00
ByteArrayConverters map[string]ByteArrayConverter
major refactor, apologies for the large commit removed analyzers (these are now built as needed through config) removed html chacter filter (now built as needed through config) added missing license header changed constructor signature of filters that cannot return errors filter constructors that can have errors, now have Must variant which panics change cdl2 tokenizer into filter (should only see lower-case input) new top level index api, closes #5 refactored index tests to not rely directly on analyzers moved query objects to top-level new top level search api, closes #12 top score collector allows skipping results index mapping supports _all by default, closes #3 and closes #6 index mapping supports disabled sections, closes #7 new http sub package with reusable http.Handler's, closes #22
2014-07-30 18:30:38 +02:00
}
introduced a config option to disable creating indexes if they don't already exist closes #23 and closes #24
2014-07-30 20:29:26 +02:00
func (c *Configuration) BuildNewAnalyzer(charFilterNames []string, tokenizerName string, tokenFilterNames []string) (*analysis.Analyzer, error) {
major refactor, apologies for the large commit removed analyzers (these are now built as needed through config) removed html chacter filter (now built as needed through config) added missing license header changed constructor signature of filters that cannot return errors filter constructors that can have errors, now have Must variant which panics change cdl2 tokenizer into filter (should only see lower-case input) new top level index api, closes #5 refactored index tests to not rely directly on analyzers moved query objects to top-level new top level search api, closes #12 top score collector allows skipping results index mapping supports _all by default, closes #3 and closes #6 index mapping supports disabled sections, closes #7 new http sub package with reusable http.Handler's, closes #22
2014-07-30 18:30:38 +02:00
rv := analysis.Analyzer{}
if len(charFilterNames) > 0 {
rv.CharFilters = make([]analysis.CharFilter, len(charFilterNames))
for i, charFilterName := range charFilterNames {
charFilter := c.Analysis.CharFilters[charFilterName]
if charFilter == nil {
return nil, fmt.Errorf("no character filter named `%s` registered", charFilterName)
}
rv.CharFilters[i] = charFilter
}
}
rv.Tokenizer = c.Analysis.Tokenizers[tokenizerName]
if rv.Tokenizer == nil {
return nil, fmt.Errorf("no tokenizer named `%s` registered", tokenizerName)
}
if len(tokenFilterNames) > 0 {
rv.TokenFilters = make([]analysis.TokenFilter, len(tokenFilterNames))
for i, tokenFilterName := range tokenFilterNames {
tokenFilter := c.Analysis.TokenFilters[tokenFilterName]
if tokenFilter == nil {
return nil, fmt.Errorf("no token filter named `%s` registered", tokenFilterName)
}
rv.TokenFilters[i] = tokenFilter
}
}
return &rv, nil
}
introduced a config option to disable creating indexes if they don't already exist closes #23 and closes #24
2014-07-30 20:29:26 +02:00
func (c *Configuration) MustBuildNewAnalyzer(charFilterNames []string, tokenizerName string, tokenFilterNames []string) *analysis.Analyzer {
major refactor, apologies for the large commit removed analyzers (these are now built as needed through config) removed html chacter filter (now built as needed through config) added missing license header changed constructor signature of filters that cannot return errors filter constructors that can have errors, now have Must variant which panics change cdl2 tokenizer into filter (should only see lower-case input) new top level index api, closes #5 refactored index tests to not rely directly on analyzers moved query objects to top-level new top level search api, closes #12 top score collector allows skipping results index mapping supports _all by default, closes #3 and closes #6 index mapping supports disabled sections, closes #7 new http sub package with reusable http.Handler's, closes #22
2014-07-30 18:30:38 +02:00
analyzer, err := c.BuildNewAnalyzer(charFilterNames, tokenizerName, tokenFilterNames)
if err != nil {
panic(err)
}
return analyzer
}
refactored StopWordsMap into WordMap so it can be reused the ElisionFilter will need a word list of articles and plan to reuse this
2014-08-03 23:46:35 +02:00
func (c *Configuration) MustLoadStopWords(stopWordsBytes []byte) analysis.WordMap {
rv := analysis.NewWordMap()
improved stop word support and related config stop words can be loaded from files/bytes, closes #19 stop words loaded for large list of languages, closes #20 defined language specific analyzers for as much as possible right now, closes #21 opened new issues for some of the remaining gaps
2014-07-31 01:29:52 +02:00
err := rv.LoadBytes(stopWordsBytes)
if err != nil {
panic(err)
}
return rv
}
introduced a config option to disable creating indexes if they don't already exist closes #23 and closes #24
2014-07-30 20:29:26 +02:00
func NewConfiguration() *Configuration {
return &Configuration{
major refactor, apologies for the large commit removed analyzers (these are now built as needed through config) removed html chacter filter (now built as needed through config) added missing license header changed constructor signature of filters that cannot return errors filter constructors that can have errors, now have Must variant which panics change cdl2 tokenizer into filter (should only see lower-case input) new top level index api, closes #5 refactored index tests to not rely directly on analyzers moved query objects to top-level new top level search api, closes #12 top score collector allows skipping results index mapping supports _all by default, closes #3 and closes #6 index mapping supports disabled sections, closes #7 new http sub package with reusable http.Handler's, closes #22
2014-07-30 18:30:38 +02:00
Analysis: &AnalysisConfig{
refactored StopWordsMap into WordMap so it can be reused the ElisionFilter will need a word list of articles and plan to reuse this
2014-08-03 23:46:35 +02:00
TokenMaps: make(map[string]analysis.WordMap),
added support for date range fields and queries closes #9 and closes #11
2014-08-03 23:19:04 +02:00
CharFilters: make(map[string]analysis.CharFilter),
Tokenizers: make(map[string]analysis.Tokenizer),
TokenFilters: make(map[string]analysis.TokenFilter),
Analyzers: make(map[string]*analysis.Analyzer),
DateTimeParsers: make(map[string]analysis.DateTimeParser),
major refactor, apologies for the large commit removed analyzers (these are now built as needed through config) removed html chacter filter (now built as needed through config) added missing license header changed constructor signature of filters that cannot return errors filter constructors that can have errors, now have Must variant which panics change cdl2 tokenizer into filter (should only see lower-case input) new top level index api, closes #5 refactored index tests to not rely directly on analyzers moved query objects to top-level new top level search api, closes #12 top score collector allows skipping results index mapping supports _all by default, closes #3 and closes #6 index mapping supports disabled sections, closes #7 new http sub package with reusable http.Handler's, closes #22
2014-07-30 18:30:38 +02:00
},
Highlight: &HighlightConfig{
Highlighters: make(map[string]search.Highlighter),
},
further streamlined the API introduced concept of byte array converters right now only wired up to top-level index mapping allowing the removal of the JSON methods, now at the top level we default to parsing []byte as JSON, override if thats not the behavior you want. future enhancements will allow use of these byte array converters to control how byte arrays are handled elsewhere in documents this would allow for handing binary attachments, etc in the future closes #59
2014-08-11 18:47:29 +02:00
ByteArrayConverters: make(map[string]ByteArrayConverter),
major refactor, apologies for the large commit removed analyzers (these are now built as needed through config) removed html chacter filter (now built as needed through config) added missing license header changed constructor signature of filters that cannot return errors filter constructors that can have errors, now have Must variant which panics change cdl2 tokenizer into filter (should only see lower-case input) new top level index api, closes #5 refactored index tests to not rely directly on analyzers moved query objects to top-level new top level search api, closes #12 top score collector allows skipping results index mapping supports _all by default, closes #3 and closes #6 index mapping supports disabled sections, closes #7 new http sub package with reusable http.Handler's, closes #22
2014-07-30 18:30:38 +02:00
}
}
introduced a config option to disable creating indexes if they don't already exist closes #23 and closes #24
2014-07-30 20:29:26 +02:00
var Config *Configuration
major refactor, apologies for the large commit removed analyzers (these are now built as needed through config) removed html chacter filter (now built as needed through config) added missing license header changed constructor signature of filters that cannot return errors filter constructors that can have errors, now have Must variant which panics change cdl2 tokenizer into filter (should only see lower-case input) new top level index api, closes #5 refactored index tests to not rely directly on analyzers moved query objects to top-level new top level search api, closes #12 top score collector allows skipping results index mapping supports _all by default, closes #3 and closes #6 index mapping supports disabled sections, closes #7 new http sub package with reusable http.Handler's, closes #22
2014-07-30 18:30:38 +02:00
func init() {
// build the default configuration
introduced a config option to disable creating indexes if they don't already exist closes #23 and closes #24
2014-07-30 20:29:26 +02:00
Config = NewConfiguration()
major refactor, apologies for the large commit removed analyzers (these are now built as needed through config) removed html chacter filter (now built as needed through config) added missing license header changed constructor signature of filters that cannot return errors filter constructors that can have errors, now have Must variant which panics change cdl2 tokenizer into filter (should only see lower-case input) new top level index api, closes #5 refactored index tests to not rely directly on analyzers moved query objects to top-level new top level search api, closes #12 top score collector allows skipping results index mapping supports _all by default, closes #3 and closes #6 index mapping supports disabled sections, closes #7 new http sub package with reusable http.Handler's, closes #22
2014-07-30 18:30:38 +02:00
further streamlined the API introduced concept of byte array converters right now only wired up to top-level index mapping allowing the removal of the JSON methods, now at the top level we default to parsing []byte as JSON, override if thats not the behavior you want. future enhancements will allow use of these byte array converters to control how byte arrays are handled elsewhere in documents this would allow for handing binary attachments, etc in the future closes #59
2014-08-11 18:47:29 +02:00
// register byte array converters
Config.ByteArrayConverters["string"] = NewStringByteArrayConverter()
Config.ByteArrayConverters["json"] = NewJSONByteArrayConverter()
Config.ByteArrayConverters["ignore"] = NewIgnoreByteArrayConverter()
improved stop word support and related config stop words can be loaded from files/bytes, closes #19 stop words loaded for large list of languages, closes #20 defined language specific analyzers for as much as possible right now, closes #21 opened new issues for some of the remaining gaps
2014-07-31 01:29:52 +02:00
// register stop token maps
refactored StopWordsMap into WordMap so it can be reused the ElisionFilter will need a word list of articles and plan to reuse this
2014-08-03 23:46:35 +02:00
Config.Analysis.TokenMaps["da_stop"] = Config.MustLoadStopWords(stop_words_filter.DanishStopWords)
Config.Analysis.TokenMaps["nl_stop"] = Config.MustLoadStopWords(stop_words_filter.DutchStopWords)
Config.Analysis.TokenMaps["en_stop"] = Config.MustLoadStopWords(stop_words_filter.EnglishStopWords)
Config.Analysis.TokenMaps["fi_stop"] = Config.MustLoadStopWords(stop_words_filter.FinnishStopWords)
Config.Analysis.TokenMaps["fr_stop"] = Config.MustLoadStopWords(stop_words_filter.FrenchStopWords)
Config.Analysis.TokenMaps["de_stop"] = Config.MustLoadStopWords(stop_words_filter.GermanStopWords)
Config.Analysis.TokenMaps["hu_stop"] = Config.MustLoadStopWords(stop_words_filter.HungarianStopWords)
Config.Analysis.TokenMaps["it_stop"] = Config.MustLoadStopWords(stop_words_filter.ItalianStopWords)
Config.Analysis.TokenMaps["no_stop"] = Config.MustLoadStopWords(stop_words_filter.NorwegianStopWords)
Config.Analysis.TokenMaps["pt_stop"] = Config.MustLoadStopWords(stop_words_filter.PortugueseStopWords)
Config.Analysis.TokenMaps["ro_stop"] = Config.MustLoadStopWords(stop_words_filter.RomanianStopWords)
Config.Analysis.TokenMaps["ru_stop"] = Config.MustLoadStopWords(stop_words_filter.RussianStopWords)
Config.Analysis.TokenMaps["es_stop"] = Config.MustLoadStopWords(stop_words_filter.SpanishStopWords)
Config.Analysis.TokenMaps["sv_stop"] = Config.MustLoadStopWords(stop_words_filter.SwedishStopWords)
Config.Analysis.TokenMaps["tr_stop"] = Config.MustLoadStopWords(stop_words_filter.TurkishStopWords)
Config.Analysis.TokenMaps["ar_stop"] = Config.MustLoadStopWords(stop_words_filter.ArabicStopWords)
Config.Analysis.TokenMaps["hy_stop"] = Config.MustLoadStopWords(stop_words_filter.ArmenianStopWords)
Config.Analysis.TokenMaps["eu_stop"] = Config.MustLoadStopWords(stop_words_filter.BasqueStopWords)
Config.Analysis.TokenMaps["bg_stop"] = Config.MustLoadStopWords(stop_words_filter.BulgarianStopWords)
Config.Analysis.TokenMaps["ca_stop"] = Config.MustLoadStopWords(stop_words_filter.CatalanStopWords)
Config.Analysis.TokenMaps["gl_stop"] = Config.MustLoadStopWords(stop_words_filter.GalicianStopWords)
Config.Analysis.TokenMaps["el_stop"] = Config.MustLoadStopWords(stop_words_filter.GreekStopWords)
Config.Analysis.TokenMaps["hi_stop"] = Config.MustLoadStopWords(stop_words_filter.HindiStopWords)
Config.Analysis.TokenMaps["id_stop"] = Config.MustLoadStopWords(stop_words_filter.IndonesianStopWords)
Config.Analysis.TokenMaps["ga_stop"] = Config.MustLoadStopWords(stop_words_filter.IrishStopWords)
Config.Analysis.TokenMaps["fa_stop"] = Config.MustLoadStopWords(stop_words_filter.PersianStopWords)
Config.Analysis.TokenMaps["ckb_stop"] = Config.MustLoadStopWords(stop_words_filter.SoraniStopWords)
Config.Analysis.TokenMaps["th_stop"] = Config.MustLoadStopWords(stop_words_filter.ThaiStopWords)
added czech stop word list closes #36
2014-08-08 04:32:49 +02:00
Config.Analysis.TokenMaps["cs_stop"] = Config.MustLoadStopWords(stop_words_filter.CzechStopWords)
improved stop word support and related config stop words can be loaded from files/bytes, closes #19 stop words loaded for large list of languages, closes #20 defined language specific analyzers for as much as possible right now, closes #21 opened new issues for some of the remaining gaps
2014-07-31 01:29:52 +02:00
added elision filter defined article word maps for french, italian, irish and catalan defined elision filters for these same languages updated analyers for french and italian to use this new filter irish and catalan still depend on other missing pieces closes #25
2014-08-04 01:17:35 +02:00
// register article token maps for elision filters
Config.Analysis.TokenMaps["fr_articles"] = Config.MustLoadStopWords(elision_filter.FrenchArticles)
Config.Analysis.TokenMaps["it_articles"] = Config.MustLoadStopWords(elision_filter.ItalianArticles)
Config.Analysis.TokenMaps["ca_articles"] = Config.MustLoadStopWords(elision_filter.CatalanArticles)
Config.Analysis.TokenMaps["ga_articles"] = Config.MustLoadStopWords(elision_filter.IrishArticles)
major refactor, apologies for the large commit removed analyzers (these are now built as needed through config) removed html chacter filter (now built as needed through config) added missing license header changed constructor signature of filters that cannot return errors filter constructors that can have errors, now have Must variant which panics change cdl2 tokenizer into filter (should only see lower-case input) new top level index api, closes #5 refactored index tests to not rely directly on analyzers moved query objects to top-level new top level search api, closes #12 top score collector allows skipping results index mapping supports _all by default, closes #3 and closes #6 index mapping supports disabled sections, closes #7 new http sub package with reusable http.Handler's, closes #22
2014-07-30 18:30:38 +02:00
// register char filters
htmlCharFilterRegexp := regexp.MustCompile(`</?[!\w]+((\s+\w+(\s*=\s*(?:".*?"|'.*?'|[^'">\s]+))?)+\s*|\s*)/?>`)
htmlCharFilter := regexp_char_filter.NewRegexpCharFilter(htmlCharFilterRegexp, []byte{' '})
introduced a config option to disable creating indexes if they don't already exist closes #23 and closes #24
2014-07-30 20:29:26 +02:00
Config.Analysis.CharFilters["html"] = htmlCharFilter
configured zero width non joiner char filter, and persian analyzer
2014-08-12 00:57:04 +02:00
zeroWidthNonJoinerRegexp := regexp.MustCompile(`\x{200C}`)
zeroWidthNonJoinerCharFilter := regexp_char_filter.NewRegexpCharFilter(zeroWidthNonJoinerRegexp, []byte{' '})
Config.Analysis.CharFilters["zero_width_spaces"] = zeroWidthNonJoinerCharFilter
major refactor, apologies for the large commit removed analyzers (these are now built as needed through config) removed html chacter filter (now built as needed through config) added missing license header changed constructor signature of filters that cannot return errors filter constructors that can have errors, now have Must variant which panics change cdl2 tokenizer into filter (should only see lower-case input) new top level index api, closes #5 refactored index tests to not rely directly on analyzers moved query objects to top-level new top level search api, closes #12 top score collector allows skipping results index mapping supports _all by default, closes #3 and closes #6 index mapping supports disabled sections, closes #7 new http sub package with reusable http.Handler's, closes #22
2014-07-30 18:30:38 +02:00
// register tokenizers
whitespaceTokenizerRegexp := regexp.MustCompile(`\w+`)
introduced a config option to disable creating indexes if they don't already exist closes #23 and closes #24
2014-07-30 20:29:26 +02:00
Config.Analysis.Tokenizers["single"] = single_token.NewSingleTokenTokenizer()
Config.Analysis.Tokenizers["unicode"] = unicode_word_boundary.NewUnicodeWordBoundaryTokenizer()
Config.Analysis.Tokenizers["unicode_th"] = unicode_word_boundary.NewUnicodeWordBoundaryCustomLocaleTokenizer("th_TH")
Config.Analysis.Tokenizers["whitespace"] = regexp_tokenizer.NewRegexpTokenizer(whitespaceTokenizerRegexp)
major refactor, apologies for the large commit removed analyzers (these are now built as needed through config) removed html chacter filter (now built as needed through config) added missing license header changed constructor signature of filters that cannot return errors filter constructors that can have errors, now have Must variant which panics change cdl2 tokenizer into filter (should only see lower-case input) new top level index api, closes #5 refactored index tests to not rely directly on analyzers moved query objects to top-level new top level search api, closes #12 top score collector allows skipping results index mapping supports _all by default, closes #3 and closes #6 index mapping supports disabled sections, closes #7 new http sub package with reusable http.Handler's, closes #22
2014-07-30 18:30:38 +02:00
// register token filters
introduced a config option to disable creating indexes if they don't already exist closes #23 and closes #24
2014-07-30 20:29:26 +02:00
Config.Analysis.TokenFilters["detect_lang"] = cld2.NewCld2Filter()
Config.Analysis.TokenFilters["short"] = length_filter.NewLengthFilter(3, -1)
Config.Analysis.TokenFilters["long"] = length_filter.NewLengthFilter(-1, 255)
Config.Analysis.TokenFilters["to_lower"] = lower_case_filter.NewLowerCaseFilter()
added apostrophe filter to improve turkish analyzer closes #27
2014-08-06 14:50:00 +02:00
Config.Analysis.TokenFilters["apostrophe"] = apostrophe_filter.NewApostropheFilter()
added token truncate filter closes #49
2014-08-07 02:39:42 +02:00
Config.Analysis.TokenFilters["truncate_token"] = truncate_token_filter.NewTruncateTokenFilter(25)
added apostrophe filter to improve turkish analyzer closes #27
2014-08-06 14:50:00 +02:00
// register stemmer filters
introduced a config option to disable creating indexes if they don't already exist closes #23 and closes #24
2014-07-30 20:29:26 +02:00
Config.Analysis.TokenFilters["stemmer_da"] = stemmer_filter.MustNewStemmerFilter("danish")
Config.Analysis.TokenFilters["stemmer_nl"] = stemmer_filter.MustNewStemmerFilter("dutch")
Config.Analysis.TokenFilters["stemmer_en"] = stemmer_filter.MustNewStemmerFilter("english")
Config.Analysis.TokenFilters["stemmer_fi"] = stemmer_filter.MustNewStemmerFilter("finnish")
Config.Analysis.TokenFilters["stemmer_fr"] = stemmer_filter.MustNewStemmerFilter("french")
Config.Analysis.TokenFilters["stemmer_de"] = stemmer_filter.MustNewStemmerFilter("german")
Config.Analysis.TokenFilters["stemmer_hu"] = stemmer_filter.MustNewStemmerFilter("hungarian")
Config.Analysis.TokenFilters["stemmer_it"] = stemmer_filter.MustNewStemmerFilter("italian")
Config.Analysis.TokenFilters["stemmer_no"] = stemmer_filter.MustNewStemmerFilter("norwegian")
Config.Analysis.TokenFilters["stemmer_porter"] = stemmer_filter.MustNewStemmerFilter("porter")
Config.Analysis.TokenFilters["stemmer_pt"] = stemmer_filter.MustNewStemmerFilter("portuguese")
Config.Analysis.TokenFilters["stemmer_ro"] = stemmer_filter.MustNewStemmerFilter("romanian")
Config.Analysis.TokenFilters["stemmer_ru"] = stemmer_filter.MustNewStemmerFilter("russian")
Config.Analysis.TokenFilters["stemmer_es"] = stemmer_filter.MustNewStemmerFilter("spanish")
Config.Analysis.TokenFilters["stemmer_sv"] = stemmer_filter.MustNewStemmerFilter("swedish")
Config.Analysis.TokenFilters["stemmer_tr"] = stemmer_filter.MustNewStemmerFilter("turkish")
added sorani normalizer and stemmer, now have analyzer closes #43
2014-08-08 15:38:28 +02:00
Config.Analysis.TokenFilters["stemmer_ckb"] = sorani_stemmer_filter.NewSoraniStemmerFilter()
added hindi stemmer closes #40
2014-08-12 04:29:47 +02:00
Config.Analysis.TokenFilters["stemmer_hi"] = hindi_stemmer_filter.NewHindiStemmerFilter()
improved stop word support and related config stop words can be loaded from files/bytes, closes #19 stop words loaded for large list of languages, closes #20 defined language specific analyzers for as much as possible right now, closes #21 opened new issues for some of the remaining gaps
2014-07-31 01:29:52 +02:00
added elision filter defined article word maps for french, italian, irish and catalan defined elision filters for these same languages updated analyers for french and italian to use this new filter irish and catalan still depend on other missing pieces closes #25
2014-08-04 01:17:35 +02:00
// register stop token filters
improved stop word support and related config stop words can be loaded from files/bytes, closes #19 stop words loaded for large list of languages, closes #20 defined language specific analyzers for as much as possible right now, closes #21 opened new issues for some of the remaining gaps
2014-07-31 01:29:52 +02:00
Config.Analysis.TokenFilters["stop_token_da"] = stop_words_filter.NewStopWordsFilter(
refactored StopWordsMap into WordMap so it can be reused the ElisionFilter will need a word list of articles and plan to reuse this
2014-08-03 23:46:35 +02:00
Config.Analysis.TokenMaps["da_stop"])
improved stop word support and related config stop words can be loaded from files/bytes, closes #19 stop words loaded for large list of languages, closes #20 defined language specific analyzers for as much as possible right now, closes #21 opened new issues for some of the remaining gaps
2014-07-31 01:29:52 +02:00
Config.Analysis.TokenFilters["stop_token_nl"] = stop_words_filter.NewStopWordsFilter(
refactored StopWordsMap into WordMap so it can be reused the ElisionFilter will need a word list of articles and plan to reuse this
2014-08-03 23:46:35 +02:00
Config.Analysis.TokenMaps["nl_stop"])
improved stop word support and related config stop words can be loaded from files/bytes, closes #19 stop words loaded for large list of languages, closes #20 defined language specific analyzers for as much as possible right now, closes #21 opened new issues for some of the remaining gaps
2014-07-31 01:29:52 +02:00
Config.Analysis.TokenFilters["stop_token_en"] = stop_words_filter.NewStopWordsFilter(
refactored StopWordsMap into WordMap so it can be reused the ElisionFilter will need a word list of articles and plan to reuse this
2014-08-03 23:46:35 +02:00
Config.Analysis.TokenMaps["en_stop"])
improved stop word support and related config stop words can be loaded from files/bytes, closes #19 stop words loaded for large list of languages, closes #20 defined language specific analyzers for as much as possible right now, closes #21 opened new issues for some of the remaining gaps
2014-07-31 01:29:52 +02:00
Config.Analysis.TokenFilters["stop_token_fi"] = stop_words_filter.NewStopWordsFilter(
refactored StopWordsMap into WordMap so it can be reused the ElisionFilter will need a word list of articles and plan to reuse this
2014-08-03 23:46:35 +02:00
Config.Analysis.TokenMaps["fi_stop"])
improved stop word support and related config stop words can be loaded from files/bytes, closes #19 stop words loaded for large list of languages, closes #20 defined language specific analyzers for as much as possible right now, closes #21 opened new issues for some of the remaining gaps
2014-07-31 01:29:52 +02:00
Config.Analysis.TokenFilters["stop_token_fr"] = stop_words_filter.NewStopWordsFilter(
refactored StopWordsMap into WordMap so it can be reused the ElisionFilter will need a word list of articles and plan to reuse this
2014-08-03 23:46:35 +02:00
Config.Analysis.TokenMaps["fr_stop"])
improved stop word support and related config stop words can be loaded from files/bytes, closes #19 stop words loaded for large list of languages, closes #20 defined language specific analyzers for as much as possible right now, closes #21 opened new issues for some of the remaining gaps
2014-07-31 01:29:52 +02:00
Config.Analysis.TokenFilters["stop_token_de"] = stop_words_filter.NewStopWordsFilter(
refactored StopWordsMap into WordMap so it can be reused the ElisionFilter will need a word list of articles and plan to reuse this
2014-08-03 23:46:35 +02:00
Config.Analysis.TokenMaps["de_stop"])
improved stop word support and related config stop words can be loaded from files/bytes, closes #19 stop words loaded for large list of languages, closes #20 defined language specific analyzers for as much as possible right now, closes #21 opened new issues for some of the remaining gaps
2014-07-31 01:29:52 +02:00
Config.Analysis.TokenFilters["stop_token_hu"] = stop_words_filter.NewStopWordsFilter(
refactored StopWordsMap into WordMap so it can be reused the ElisionFilter will need a word list of articles and plan to reuse this
2014-08-03 23:46:35 +02:00
Config.Analysis.TokenMaps["hu_stop"])
improved stop word support and related config stop words can be loaded from files/bytes, closes #19 stop words loaded for large list of languages, closes #20 defined language specific analyzers for as much as possible right now, closes #21 opened new issues for some of the remaining gaps
2014-07-31 01:29:52 +02:00
Config.Analysis.TokenFilters["stop_token_it"] = stop_words_filter.NewStopWordsFilter(
refactored StopWordsMap into WordMap so it can be reused the ElisionFilter will need a word list of articles and plan to reuse this
2014-08-03 23:46:35 +02:00
Config.Analysis.TokenMaps["it_stop"])
improved stop word support and related config stop words can be loaded from files/bytes, closes #19 stop words loaded for large list of languages, closes #20 defined language specific analyzers for as much as possible right now, closes #21 opened new issues for some of the remaining gaps
2014-07-31 01:29:52 +02:00
Config.Analysis.TokenFilters["stop_token_no"] = stop_words_filter.NewStopWordsFilter(
refactored StopWordsMap into WordMap so it can be reused the ElisionFilter will need a word list of articles and plan to reuse this
2014-08-03 23:46:35 +02:00
Config.Analysis.TokenMaps["no_stop"])
improved stop word support and related config stop words can be loaded from files/bytes, closes #19 stop words loaded for large list of languages, closes #20 defined language specific analyzers for as much as possible right now, closes #21 opened new issues for some of the remaining gaps
2014-07-31 01:29:52 +02:00
Config.Analysis.TokenFilters["stop_token_pt"] = stop_words_filter.NewStopWordsFilter(
refactored StopWordsMap into WordMap so it can be reused the ElisionFilter will need a word list of articles and plan to reuse this
2014-08-03 23:46:35 +02:00
Config.Analysis.TokenMaps["pt_stop"])
improved stop word support and related config stop words can be loaded from files/bytes, closes #19 stop words loaded for large list of languages, closes #20 defined language specific analyzers for as much as possible right now, closes #21 opened new issues for some of the remaining gaps
2014-07-31 01:29:52 +02:00
Config.Analysis.TokenFilters["stop_token_ro"] = stop_words_filter.NewStopWordsFilter(
refactored StopWordsMap into WordMap so it can be reused the ElisionFilter will need a word list of articles and plan to reuse this
2014-08-03 23:46:35 +02:00
Config.Analysis.TokenMaps["ro_stop"])
improved stop word support and related config stop words can be loaded from files/bytes, closes #19 stop words loaded for large list of languages, closes #20 defined language specific analyzers for as much as possible right now, closes #21 opened new issues for some of the remaining gaps
2014-07-31 01:29:52 +02:00
Config.Analysis.TokenFilters["stop_token_ru"] = stop_words_filter.NewStopWordsFilter(
refactored StopWordsMap into WordMap so it can be reused the ElisionFilter will need a word list of articles and plan to reuse this
2014-08-03 23:46:35 +02:00
Config.Analysis.TokenMaps["ru_stop"])
improved stop word support and related config stop words can be loaded from files/bytes, closes #19 stop words loaded for large list of languages, closes #20 defined language specific analyzers for as much as possible right now, closes #21 opened new issues for some of the remaining gaps
2014-07-31 01:29:52 +02:00
Config.Analysis.TokenFilters["stop_token_es"] = stop_words_filter.NewStopWordsFilter(
refactored StopWordsMap into WordMap so it can be reused the ElisionFilter will need a word list of articles and plan to reuse this
2014-08-03 23:46:35 +02:00
Config.Analysis.TokenMaps["es_stop"])
improved stop word support and related config stop words can be loaded from files/bytes, closes #19 stop words loaded for large list of languages, closes #20 defined language specific analyzers for as much as possible right now, closes #21 opened new issues for some of the remaining gaps
2014-07-31 01:29:52 +02:00
Config.Analysis.TokenFilters["stop_token_sv"] = stop_words_filter.NewStopWordsFilter(
refactored StopWordsMap into WordMap so it can be reused the ElisionFilter will need a word list of articles and plan to reuse this
2014-08-03 23:46:35 +02:00
Config.Analysis.TokenMaps["sv_stop"])
improved stop word support and related config stop words can be loaded from files/bytes, closes #19 stop words loaded for large list of languages, closes #20 defined language specific analyzers for as much as possible right now, closes #21 opened new issues for some of the remaining gaps
2014-07-31 01:29:52 +02:00
Config.Analysis.TokenFilters["stop_token_tr"] = stop_words_filter.NewStopWordsFilter(
refactored StopWordsMap into WordMap so it can be reused the ElisionFilter will need a word list of articles and plan to reuse this
2014-08-03 23:46:35 +02:00
Config.Analysis.TokenMaps["tr_stop_stop"])
improved stop word support and related config stop words can be loaded from files/bytes, closes #19 stop words loaded for large list of languages, closes #20 defined language specific analyzers for as much as possible right now, closes #21 opened new issues for some of the remaining gaps
2014-07-31 01:29:52 +02:00
Config.Analysis.TokenFilters["stop_token_ar"] = stop_words_filter.NewStopWordsFilter(
refactored StopWordsMap into WordMap so it can be reused the ElisionFilter will need a word list of articles and plan to reuse this
2014-08-03 23:46:35 +02:00
Config.Analysis.TokenMaps["ar_stop"])
improved stop word support and related config stop words can be loaded from files/bytes, closes #19 stop words loaded for large list of languages, closes #20 defined language specific analyzers for as much as possible right now, closes #21 opened new issues for some of the remaining gaps
2014-07-31 01:29:52 +02:00
Config.Analysis.TokenFilters["stop_token_hy"] = stop_words_filter.NewStopWordsFilter(
refactored StopWordsMap into WordMap so it can be reused the ElisionFilter will need a word list of articles and plan to reuse this
2014-08-03 23:46:35 +02:00
Config.Analysis.TokenMaps["hy_stop"])
improved stop word support and related config stop words can be loaded from files/bytes, closes #19 stop words loaded for large list of languages, closes #20 defined language specific analyzers for as much as possible right now, closes #21 opened new issues for some of the remaining gaps
2014-07-31 01:29:52 +02:00
Config.Analysis.TokenFilters["stop_token_eu"] = stop_words_filter.NewStopWordsFilter(
refactored StopWordsMap into WordMap so it can be reused the ElisionFilter will need a word list of articles and plan to reuse this
2014-08-03 23:46:35 +02:00
Config.Analysis.TokenMaps["eu_stop"])
improved stop word support and related config stop words can be loaded from files/bytes, closes #19 stop words loaded for large list of languages, closes #20 defined language specific analyzers for as much as possible right now, closes #21 opened new issues for some of the remaining gaps
2014-07-31 01:29:52 +02:00
Config.Analysis.TokenFilters["stop_token_bg"] = stop_words_filter.NewStopWordsFilter(
refactored StopWordsMap into WordMap so it can be reused the ElisionFilter will need a word list of articles and plan to reuse this
2014-08-03 23:46:35 +02:00
Config.Analysis.TokenMaps["bg_stop"])
improved stop word support and related config stop words can be loaded from files/bytes, closes #19 stop words loaded for large list of languages, closes #20 defined language specific analyzers for as much as possible right now, closes #21 opened new issues for some of the remaining gaps
2014-07-31 01:29:52 +02:00
Config.Analysis.TokenFilters["stop_token_ca"] = stop_words_filter.NewStopWordsFilter(
refactored StopWordsMap into WordMap so it can be reused the ElisionFilter will need a word list of articles and plan to reuse this
2014-08-03 23:46:35 +02:00
Config.Analysis.TokenMaps["ca_stop"])
improved stop word support and related config stop words can be loaded from files/bytes, closes #19 stop words loaded for large list of languages, closes #20 defined language specific analyzers for as much as possible right now, closes #21 opened new issues for some of the remaining gaps
2014-07-31 01:29:52 +02:00
Config.Analysis.TokenFilters["stop_token_gl"] = stop_words_filter.NewStopWordsFilter(
refactored StopWordsMap into WordMap so it can be reused the ElisionFilter will need a word list of articles and plan to reuse this
2014-08-03 23:46:35 +02:00
Config.Analysis.TokenMaps["gl_stop"])
improved stop word support and related config stop words can be loaded from files/bytes, closes #19 stop words loaded for large list of languages, closes #20 defined language specific analyzers for as much as possible right now, closes #21 opened new issues for some of the remaining gaps
2014-07-31 01:29:52 +02:00
Config.Analysis.TokenFilters["stop_token_el"] = stop_words_filter.NewStopWordsFilter(
refactored StopWordsMap into WordMap so it can be reused the ElisionFilter will need a word list of articles and plan to reuse this
2014-08-03 23:46:35 +02:00
Config.Analysis.TokenMaps["el_stop"])
improved stop word support and related config stop words can be loaded from files/bytes, closes #19 stop words loaded for large list of languages, closes #20 defined language specific analyzers for as much as possible right now, closes #21 opened new issues for some of the remaining gaps
2014-07-31 01:29:52 +02:00
Config.Analysis.TokenFilters["stop_token_hi"] = stop_words_filter.NewStopWordsFilter(
refactored StopWordsMap into WordMap so it can be reused the ElisionFilter will need a word list of articles and plan to reuse this
2014-08-03 23:46:35 +02:00
Config.Analysis.TokenMaps["hi_stop"])
improved stop word support and related config stop words can be loaded from files/bytes, closes #19 stop words loaded for large list of languages, closes #20 defined language specific analyzers for as much as possible right now, closes #21 opened new issues for some of the remaining gaps
2014-07-31 01:29:52 +02:00
Config.Analysis.TokenFilters["stop_token_id"] = stop_words_filter.NewStopWordsFilter(
refactored StopWordsMap into WordMap so it can be reused the ElisionFilter will need a word list of articles and plan to reuse this
2014-08-03 23:46:35 +02:00
Config.Analysis.TokenMaps["id_stop"])
improved stop word support and related config stop words can be loaded from files/bytes, closes #19 stop words loaded for large list of languages, closes #20 defined language specific analyzers for as much as possible right now, closes #21 opened new issues for some of the remaining gaps
2014-07-31 01:29:52 +02:00
Config.Analysis.TokenFilters["stop_token_ga"] = stop_words_filter.NewStopWordsFilter(
refactored StopWordsMap into WordMap so it can be reused the ElisionFilter will need a word list of articles and plan to reuse this
2014-08-03 23:46:35 +02:00
Config.Analysis.TokenMaps["ga_stop"])
improved stop word support and related config stop words can be loaded from files/bytes, closes #19 stop words loaded for large list of languages, closes #20 defined language specific analyzers for as much as possible right now, closes #21 opened new issues for some of the remaining gaps
2014-07-31 01:29:52 +02:00
Config.Analysis.TokenFilters["stop_token_fa"] = stop_words_filter.NewStopWordsFilter(
refactored StopWordsMap into WordMap so it can be reused the ElisionFilter will need a word list of articles and plan to reuse this
2014-08-03 23:46:35 +02:00
Config.Analysis.TokenMaps["fa_stop"])
improved stop word support and related config stop words can be loaded from files/bytes, closes #19 stop words loaded for large list of languages, closes #20 defined language specific analyzers for as much as possible right now, closes #21 opened new issues for some of the remaining gaps
2014-07-31 01:29:52 +02:00
Config.Analysis.TokenFilters["stop_token_ckb"] = stop_words_filter.NewStopWordsFilter(
refactored StopWordsMap into WordMap so it can be reused the ElisionFilter will need a word list of articles and plan to reuse this
2014-08-03 23:46:35 +02:00
Config.Analysis.TokenMaps["ckb_stop"])
improved stop word support and related config stop words can be loaded from files/bytes, closes #19 stop words loaded for large list of languages, closes #20 defined language specific analyzers for as much as possible right now, closes #21 opened new issues for some of the remaining gaps
2014-07-31 01:29:52 +02:00
Config.Analysis.TokenFilters["stop_token_th"] = stop_words_filter.NewStopWordsFilter(
refactored StopWordsMap into WordMap so it can be reused the ElisionFilter will need a word list of articles and plan to reuse this
2014-08-03 23:46:35 +02:00
Config.Analysis.TokenMaps["th_stop"])
added czech stop word list closes #36
2014-08-08 04:32:49 +02:00
Config.Analysis.TokenFilters["stop_token_cs"] = stop_words_filter.NewStopWordsFilter(
Config.Analysis.TokenMaps["cs_stop"])
major refactor, apologies for the large commit removed analyzers (these are now built as needed through config) removed html chacter filter (now built as needed through config) added missing license header changed constructor signature of filters that cannot return errors filter constructors that can have errors, now have Must variant which panics change cdl2 tokenizer into filter (should only see lower-case input) new top level index api, closes #5 refactored index tests to not rely directly on analyzers moved query objects to top-level new top level search api, closes #12 top score collector allows skipping results index mapping supports _all by default, closes #3 and closes #6 index mapping supports disabled sections, closes #7 new http sub package with reusable http.Handler's, closes #22
2014-07-30 18:30:38 +02:00
added elision filter defined article word maps for french, italian, irish and catalan defined elision filters for these same languages updated analyers for french and italian to use this new filter irish and catalan still depend on other missing pieces closes #25
2014-08-04 01:17:35 +02:00
// register elision filters
Config.Analysis.TokenFilters["elision_fr"] = elision_filter.NewElisionFilter(
Config.Analysis.TokenMaps["fr_articles"])
Config.Analysis.TokenFilters["elision_it"] = elision_filter.NewElisionFilter(
Config.Analysis.TokenMaps["it_articles"])
Config.Analysis.TokenFilters["elision_ca"] = elision_filter.NewElisionFilter(
Config.Analysis.TokenMaps["ca_articles"])
Config.Analysis.TokenFilters["elision_ga"] = elision_filter.NewElisionFilter(
Config.Analysis.TokenMaps["ga_articles"])
added unicode normalization filter
2014-08-05 03:59:57 +02:00
// register unicode normalizers
Config.Analysis.TokenFilters["normalize_nfc"] = unicode_normalize.MustNewUnicodeNormalizeFilter(unicode_normalize.NFC)
Config.Analysis.TokenFilters["normalize_nfd"] = unicode_normalize.MustNewUnicodeNormalizeFilter(unicode_normalize.NFD)
Config.Analysis.TokenFilters["normalize_nfkc"] = unicode_normalize.MustNewUnicodeNormalizeFilter(unicode_normalize.NFKC)
Config.Analysis.TokenFilters["normalize_nfkd"] = unicode_normalize.MustNewUnicodeNormalizeFilter(unicode_normalize.NFKD)
added sorani normalizer and stemmer, now have analyzer closes #43
2014-08-08 15:38:28 +02:00
Config.Analysis.TokenFilters["normalize_ckb"] = sorani_normalize.NewSoraniNormalizeFilter()
added persian normalizer closes #67
2014-08-12 00:15:41 +02:00
Config.Analysis.TokenFilters["normalize_fa"] = persian_normalize.NewPersianNormalizeFilter()
added arabic normalizer closes #63
2014-08-12 00:35:35 +02:00
Config.Analysis.TokenFilters["normalize_ar"] = arabic_normalize.NewArabicNormalizeFilter()
added german normalizer updated german analyzer to use this normalizer closes #65
2014-08-12 01:25:37 +02:00
Config.Analysis.TokenFilters["normalize_de"] = german_normalize.NewGermanNormalizeFilter()
added hindi normalizer closes #64
2014-08-12 01:51:47 +02:00
Config.Analysis.TokenFilters["normalize_hi"] = hindi_normalize.NewHindiNormalizeFilter()
added unicode normalization filter
2014-08-05 03:59:57 +02:00
major refactor, apologies for the large commit removed analyzers (these are now built as needed through config) removed html chacter filter (now built as needed through config) added missing license header changed constructor signature of filters that cannot return errors filter constructors that can have errors, now have Must variant which panics change cdl2 tokenizer into filter (should only see lower-case input) new top level index api, closes #5 refactored index tests to not rely directly on analyzers moved query objects to top-level new top level search api, closes #12 top score collector allows skipping results index mapping supports _all by default, closes #3 and closes #6 index mapping supports disabled sections, closes #7 new http sub package with reusable http.Handler's, closes #22
2014-07-30 18:30:38 +02:00
// register analyzers
introduced a config option to disable creating indexes if they don't already exist closes #23 and closes #24
2014-07-30 20:29:26 +02:00
keywordAnalyzer := Config.MustBuildNewAnalyzer([]string{}, "single", []string{})
Config.Analysis.Analyzers["keyword"] = keywordAnalyzer
simpleAnalyzer := Config.MustBuildNewAnalyzer([]string{}, "whitespace", []string{"to_lower"})
Config.Analysis.Analyzers["simple"] = simpleAnalyzer
improved stop word support and related config stop words can be loaded from files/bytes, closes #19 stop words loaded for large list of languages, closes #20 defined language specific analyzers for as much as possible right now, closes #21 opened new issues for some of the remaining gaps
2014-07-31 01:29:52 +02:00
standardAnalyzer := Config.MustBuildNewAnalyzer([]string{}, "whitespace", []string{"to_lower", "stop_token_en"})
introduced a config option to disable creating indexes if they don't already exist closes #23 and closes #24
2014-07-30 20:29:26 +02:00
Config.Analysis.Analyzers["standard"] = standardAnalyzer
detectLangAnalyzer := Config.MustBuildNewAnalyzer([]string{}, "single", []string{"to_lower", "detect_lang"})
Config.Analysis.Analyzers["detect_lang"] = detectLangAnalyzer
major refactor, apologies for the large commit removed analyzers (these are now built as needed through config) removed html chacter filter (now built as needed through config) added missing license header changed constructor signature of filters that cannot return errors filter constructors that can have errors, now have Must variant which panics change cdl2 tokenizer into filter (should only see lower-case input) new top level index api, closes #5 refactored index tests to not rely directly on analyzers moved query objects to top-level new top level search api, closes #12 top score collector allows skipping results index mapping supports _all by default, closes #3 and closes #6 index mapping supports disabled sections, closes #7 new http sub package with reusable http.Handler's, closes #22
2014-07-30 18:30:38 +02:00
improved stop word support and related config stop words can be loaded from files/bytes, closes #19 stop words loaded for large list of languages, closes #20 defined language specific analyzers for as much as possible right now, closes #21 opened new issues for some of the remaining gaps
2014-07-31 01:29:52 +02:00
// language specific analyzers
danishAnalyzer := Config.MustBuildNewAnalyzer([]string{}, "unicode", []string{"to_lower", "stop_token_da", "stemmer_da"})
Config.Analysis.Analyzers["da"] = danishAnalyzer
dutchAnalyzer := Config.MustBuildNewAnalyzer([]string{}, "unicode", []string{"to_lower", "stop_token_nl", "stemmer_nl"})
Config.Analysis.Analyzers["nl"] = dutchAnalyzer
englishAnalyzer := Config.MustBuildNewAnalyzer([]string{}, "unicode", []string{"to_lower", "stop_token_en", "stemmer_en"})
Config.Analysis.Analyzers["en"] = englishAnalyzer
finnishAnalyzer := Config.MustBuildNewAnalyzer([]string{}, "unicode", []string{"to_lower", "stop_token_fi", "stemmer_fi"})
Config.Analysis.Analyzers["fi"] = finnishAnalyzer
added elision filter defined article word maps for french, italian, irish and catalan defined elision filters for these same languages updated analyers for french and italian to use this new filter irish and catalan still depend on other missing pieces closes #25
2014-08-04 01:17:35 +02:00
frenchAnalyzer := Config.MustBuildNewAnalyzer([]string{}, "unicode", []string{"elision_fr", "to_lower", "stop_token_fr", "stemmer_fr"})
improved stop word support and related config stop words can be loaded from files/bytes, closes #19 stop words loaded for large list of languages, closes #20 defined language specific analyzers for as much as possible right now, closes #21 opened new issues for some of the remaining gaps
2014-07-31 01:29:52 +02:00
Config.Analysis.Analyzers["fr"] = frenchAnalyzer
added german normalizer updated german analyzer to use this normalizer closes #65
2014-08-12 01:25:37 +02:00
germanAnalyzer := Config.MustBuildNewAnalyzer([]string{}, "unicode", []string{"to_lower", "stop_token_de", "normalize_de", "stemmer_de"})
improved stop word support and related config stop words can be loaded from files/bytes, closes #19 stop words loaded for large list of languages, closes #20 defined language specific analyzers for as much as possible right now, closes #21 opened new issues for some of the remaining gaps
2014-07-31 01:29:52 +02:00
Config.Analysis.Analyzers["de"] = germanAnalyzer
hungarianAnalyzer := Config.MustBuildNewAnalyzer([]string{}, "unicode", []string{"to_lower", "stop_token_hu", "stemmer_hu"})
Config.Analysis.Analyzers["hu"] = hungarianAnalyzer
added elision filter defined article word maps for french, italian, irish and catalan defined elision filters for these same languages updated analyers for french and italian to use this new filter irish and catalan still depend on other missing pieces closes #25
2014-08-04 01:17:35 +02:00
italianAnalyzer := Config.MustBuildNewAnalyzer([]string{}, "unicode", []string{"elision_it", "to_lower", "stop_token_it", "stemmer_it"})
improved stop word support and related config stop words can be loaded from files/bytes, closes #19 stop words loaded for large list of languages, closes #20 defined language specific analyzers for as much as possible right now, closes #21 opened new issues for some of the remaining gaps
2014-07-31 01:29:52 +02:00
Config.Analysis.Analyzers["it"] = italianAnalyzer
norwegianAnalyzer := Config.MustBuildNewAnalyzer([]string{}, "unicode", []string{"to_lower", "stop_token_no", "stemmer_no"})
Config.Analysis.Analyzers["no"] = norwegianAnalyzer
portugueseAnalyzer := Config.MustBuildNewAnalyzer([]string{}, "unicode", []string{"to_lower", "stop_token_pt", "stemmer_pt"})
Config.Analysis.Analyzers["pt"] = portugueseAnalyzer
romanianAnalyzer := Config.MustBuildNewAnalyzer([]string{}, "unicode", []string{"to_lower", "stop_token_ro", "stemmer_ro"})
Config.Analysis.Analyzers["ro"] = romanianAnalyzer
russianAnalyzer := Config.MustBuildNewAnalyzer([]string{}, "unicode", []string{"to_lower", "stop_token_ru", "stemmer_ru"})
Config.Analysis.Analyzers["ru"] = russianAnalyzer
spanishAnalyzer := Config.MustBuildNewAnalyzer([]string{}, "unicode", []string{"to_lower", "stop_token_es", "stemmer_es"})
Config.Analysis.Analyzers["es"] = spanishAnalyzer
swedishAnalyzer := Config.MustBuildNewAnalyzer([]string{}, "unicode", []string{"to_lower", "stop_token_sv", "stemmer_sv"})
Config.Analysis.Analyzers["sv"] = swedishAnalyzer
added apostrophe filter to improve turkish analyzer closes #27
2014-08-06 14:50:00 +02:00
turkishAnalyzer := Config.MustBuildNewAnalyzer([]string{}, "unicode", []string{"apostrophe", "to_lower", "stop_token_tr", "stemmer_tr"})
improved stop word support and related config stop words can be loaded from files/bytes, closes #19 stop words loaded for large list of languages, closes #20 defined language specific analyzers for as much as possible right now, closes #21 opened new issues for some of the remaining gaps
2014-07-31 01:29:52 +02:00
Config.Analysis.Analyzers["tr"] = turkishAnalyzer
thaiAnalyzer := Config.MustBuildNewAnalyzer([]string{}, "unicode_th", []string{"to_lower", "stop_token_th"})
Config.Analysis.Analyzers["th"] = thaiAnalyzer
added sorani normalizer and stemmer, now have analyzer closes #43
2014-08-08 15:38:28 +02:00
soraniAnalyzer := Config.MustBuildNewAnalyzer([]string{}, "unicode", []string{"normalize_ckb", "to_lower", "stop_token_ckb", "stemmer_ckb"})
Config.Analysis.Analyzers["ckb"] = soraniAnalyzer
configured zero width non joiner char filter, and persian analyzer
2014-08-12 00:57:04 +02:00
persianAnalyzer := Config.MustBuildNewAnalyzer([]string{"zero_width_spaces"}, "unicode", []string{"to_lower", "normalize_ar", "normalize_fa", "stop_token_fa"})
Config.Analysis.Analyzers["fa"] = persianAnalyzer
improved stop word support and related config stop words can be loaded from files/bytes, closes #19 stop words loaded for large list of languages, closes #20 defined language specific analyzers for as much as possible right now, closes #21 opened new issues for some of the remaining gaps
2014-07-31 01:29:52 +02:00
major refactor, apologies for the large commit removed analyzers (these are now built as needed through config) removed html chacter filter (now built as needed through config) added missing license header changed constructor signature of filters that cannot return errors filter constructors that can have errors, now have Must variant which panics change cdl2 tokenizer into filter (should only see lower-case input) new top level index api, closes #5 refactored index tests to not rely directly on analyzers moved query objects to top-level new top level search api, closes #12 top score collector allows skipping results index mapping supports _all by default, closes #3 and closes #6 index mapping supports disabled sections, closes #7 new http sub package with reusable http.Handler's, closes #22
2014-07-30 18:30:38 +02:00
// register ansi highlighter
introduced a config option to disable creating indexes if they don't already exist closes #23 and closes #24
2014-07-30 20:29:26 +02:00
Config.Highlight.Highlighters["ansi"] = search.NewSimpleHighlighter()
major refactor, apologies for the large commit removed analyzers (these are now built as needed through config) removed html chacter filter (now built as needed through config) added missing license header changed constructor signature of filters that cannot return errors filter constructors that can have errors, now have Must variant which panics change cdl2 tokenizer into filter (should only see lower-case input) new top level index api, closes #5 refactored index tests to not rely directly on analyzers moved query objects to top-level new top level search api, closes #12 top score collector allows skipping results index mapping supports _all by default, closes #3 and closes #6 index mapping supports disabled sections, closes #7 new http sub package with reusable http.Handler's, closes #22
2014-07-30 18:30:38 +02:00
// register html highlighter
htmlFormatter := search.NewHTMLFragmentFormatterCustom(`<span class="highlight">`, `</span>`)
htmlHighlighter := search.NewSimpleHighlighter()
htmlHighlighter.SetFragmentFormatter(htmlFormatter)
introduced a config option to disable creating indexes if they don't already exist closes #23 and closes #24
2014-07-30 20:29:26 +02:00
Config.Highlight.Highlighters["html"] = htmlHighlighter
major refactor, apologies for the large commit removed analyzers (these are now built as needed through config) removed html chacter filter (now built as needed through config) added missing license header changed constructor signature of filters that cannot return errors filter constructors that can have errors, now have Must variant which panics change cdl2 tokenizer into filter (should only see lower-case input) new top level index api, closes #5 refactored index tests to not rely directly on analyzers moved query objects to top-level new top level search api, closes #12 top score collector allows skipping results index mapping supports _all by default, closes #3 and closes #6 index mapping supports disabled sections, closes #7 new http sub package with reusable http.Handler's, closes #22
2014-07-30 18:30:38 +02:00
// set the default analyzer
simpleAnalyzerName := "simple"
introduced a config option to disable creating indexes if they don't already exist closes #23 and closes #24
2014-07-30 20:29:26 +02:00
Config.DefaultAnalyzer = &simpleAnalyzerName
major refactor, apologies for the large commit removed analyzers (these are now built as needed through config) removed html chacter filter (now built as needed through config) added missing license header changed constructor signature of filters that cannot return errors filter constructors that can have errors, now have Must variant which panics change cdl2 tokenizer into filter (should only see lower-case input) new top level index api, closes #5 refactored index tests to not rely directly on analyzers moved query objects to top-level new top level search api, closes #12 top score collector allows skipping results index mapping supports _all by default, closes #3 and closes #6 index mapping supports disabled sections, closes #7 new http sub package with reusable http.Handler's, closes #22
2014-07-30 18:30:38 +02:00
// set the default highlighter
htmlHighlighterName := "html"
introduced a config option to disable creating indexes if they don't already exist closes #23 and closes #24
2014-07-30 20:29:26 +02:00
Config.DefaultHighlighter = &htmlHighlighterName
major refactor, apologies for the large commit removed analyzers (these are now built as needed through config) removed html chacter filter (now built as needed through config) added missing license header changed constructor signature of filters that cannot return errors filter constructors that can have errors, now have Must variant which panics change cdl2 tokenizer into filter (should only see lower-case input) new top level index api, closes #5 refactored index tests to not rely directly on analyzers moved query objects to top-level new top level search api, closes #12 top score collector allows skipping results index mapping supports _all by default, closes #3 and closes #6 index mapping supports disabled sections, closes #7 new http sub package with reusable http.Handler's, closes #22
2014-07-30 18:30:38 +02:00
introduced a config option to disable creating indexes if they don't already exist closes #23 and closes #24
2014-07-30 20:29:26 +02:00
// default CreateIfMissing to true
Config.CreateIfMissing = true
added support for date range fields and queries closes #9 and closes #11
2014-08-03 23:19:04 +02:00
// set up the built-in date time formats
rfc3339NoTimezone := "2006-01-02T15:04:05"
rfc3339NoTimezoneNoT := "2006-01-02 15:04:05"
rfc3339NoTime := "2006-01-02"
Config.Analysis.DateTimeParsers["dateTimeOptional"] = flexible_go.NewFlexibleGoDateTimeParser(
[]string{
time.RFC3339Nano,
time.RFC3339,
rfc3339NoTimezone,
rfc3339NoTimezoneNoT,
rfc3339NoTime,
})
dateTimeOptionalName := "dateTimeOptional"
Config.DefaultDateTimeFormat = &dateTimeOptionalName
added support for a default field this works at the config and index mapping levels
2014-08-06 14:23:29 +02:00
defaultField := "_all"
Config.DefaultField = &defaultField
major refactor, apologies for the large commit removed analyzers (these are now built as needed through config) removed html chacter filter (now built as needed through config) added missing license header changed constructor signature of filters that cannot return errors filter constructors that can have errors, now have Must variant which panics change cdl2 tokenizer into filter (should only see lower-case input) new top level index api, closes #5 refactored index tests to not rely directly on analyzers moved query objects to top-level new top level search api, closes #12 top score collector allows skipping results index mapping supports _all by default, closes #3 and closes #6 index mapping supports disabled sections, closes #7 new http sub package with reusable http.Handler's, closes #22
2014-07-30 18:30:38 +02:00
}