previously we parsed/returned large sections of the documents
back index row in order to compute facet information. this
would require parsing the protobuf of the entire back index row.
unfortunately this creates considerable garbage.
this new version introduces a visitor/callback approach to
working with data inside the back index row. the benefit
of this approach is that we can let the higher-level code
see values, prior to any copies of data being made or
intermediate garbage being created. implementations of
the callback must copy any value which they would like to
retain beyond the callback.
NOTE: this approach is duplicates code from the
automatically generated protobuf code
NOTE: this approach assumes that the "field" field be serialized
before the "terms" field. This is guaranteed by our currently
generated protobuf encoder, and is recommended by the protobuf
spec. But, decoders SHOULD support them occuring in any order,
which we do not.
previously, the only way to get numeric matching was with the
range operators :> :>= :< :<=
now, when we encounter field:val if the val can be parsed as a
number, then we do a disjunction search, which includes
searching for val as a text term and as an exact numeric value
1) disjunction and conjunction queries now support a
"query string mode". By default they do not operate
in this mode. When in this mode, any disjunct/conjunct
which evaluates to MatchNone searcher, will be removed
from the disjunction/conjunction. If the query ends
up with NO conjuncts/disjuncts, it will itself
return the MatchNone seacher.
2) boolean query also supports a query string mode. when in
this mode, the Must, Should and MustNot searchers are all put
into query string mode.
3) rewriting of negation only queries (like -foo) now take into
account the rewriting rules above, and those are handled first.
this means that we rewrite correctly in case of +stoword -foo
4) the empty query string is now valid, and returns 0 hits.
previously this was considered a validation error.
some downstream applications need to know which sub-commands
may alter the index. this new function allows them to be
identified, while not prescribing any particular behavior.
we now use a multiphrase query in all cases
internally its optimized to be the same as regular phrase query
anyway, and we simplly map all the tokens in the stream into
a multi-phrase query with the appropriate structure
when parsing json, when we encounter the key "terms", we first
try to parse as traditional phrase query, then if that fails,
we also try parsing it as multi-phrase
the logic of how a phrase search works should be an internal
detail of the phrase searcher. further, these changes will
allow proper scoring of phrase matches, which require access
to the underlying searcher objects, which were hidden in the
previous approach.
this originated from a misunderstanding of mine going back
several years. the values need not be float64 just because
we plan to serialize them as json.
there are still larger questions about what the right type should
be, and where should any conversions go. but, this commit
simply attempts to address the most egregious problems
