When performing a MultiSearch, we create child SearchRequests
from the original SearchRequest. In doing so we copy many fields.
But, copying of the SortOrder was incorrect, as this contains
state, and distint SortOrder objects must be used. This change
introduces a Copy() method to the SearchSort interface, and
to the SortOrder types. MultiSearch now creates a new copy of
the SortOrder for each child request.
- make geo queries accessible from top-level bleve
- update query parsing to support same geo point formats as
document parsing
- add constructor for easier sorting by geo distance in Go
- additional integration tests using alternate (GeoJSON) style points
New field type GeoPointField, or "geopoint" in mapping JSON.
Currently structs and maps are considered when a mapping explicitly
marks a field as type "geopoint". Several variants of "lon", "lng", and "lat"
are looked for in map keys, struct field names, or method names.
New query type GeoBoundingBoxQuery searches for documents which have a
GeoPointField indexed with a value that is inside the specified bounding box.
New query type GeoDistanceQuery searches for documents which have a
GeoPointField indexed with a value that is less than or equal to the
specified distance from the specified location.
New sort by method "geo_distance". Hits can be sorted by their distance
from the specified location.
New geo utility package with all routines ported from Lucene.
New FilteringSearcher, which wraps an existing Searcher, but filters
all hits with a user-provided callback.
previously we parsed/returned large sections of the documents
back index row in order to compute facet information. this
would require parsing the protobuf of the entire back index row.
unfortunately this creates considerable garbage.
this new version introduces a visitor/callback approach to
working with data inside the back index row. the benefit
of this approach is that we can let the higher-level code
see values, prior to any copies of data being made or
intermediate garbage being created. implementations of
the callback must copy any value which they would like to
retain beyond the callback.
NOTE: this approach is duplicates code from the
automatically generated protobuf code
NOTE: this approach assumes that the "field" field be serialized
before the "terms" field. This is guaranteed by our currently
generated protobuf encoder, and is recommended by the protobuf
spec. But, decoders SHOULD support them occuring in any order,
which we do not.
primary change is going back to sort values be []string
and not []interface{}, this avoid allocatiosn converting
into the interface{}
that sounds obvious, so why didn't we just do that first?
because a common (default) sort is score, which is naturally
a number, not a string (like terms). converting into the
number was also expensive, and the common case.
so, this solution also makes the change to NOT put the score
into the sort value list. instead you see the dummy value
"_score". this is just a placeholder, the actual sort impl
knows that field of the sort is the score, and will sort
using the actual score.
also, several other aspets of the benchmark were cleaned up
so that unnecessary allocations do not pollute the cpu profiles
Here are the updated benchmarks:
$ go test -run=xxx -bench=. -benchmem -cpuprofile=cpu.out
BenchmarkTop10of100000Scores-4 3000 465809 ns/op 2548 B/op 33 allocs/op
BenchmarkTop100of100000Scores-4 2000 626488 ns/op 21484 B/op 213 allocs/op
BenchmarkTop10of1000000Scores-4 300 5107658 ns/op 2560 B/op 33 allocs/op
BenchmarkTop100of1000000Scores-4 300 5275403 ns/op 21624 B/op 213 allocs/op
PASS
ok github.com/blevesearch/bleve/search/collectors 7.188s
Prior to this PR, master reported:
$ go test -run=xxx -bench=. -benchmem
BenchmarkTop10of100000Scores-4 3000 453269 ns/op 360161 B/op 42 allocs/op
BenchmarkTop100of100000Scores-4 2000 519131 ns/op 388275 B/op 219 allocs/op
BenchmarkTop10of1000000Scores-4 200 7459004 ns/op 4628236 B/op 52 allocs/op
BenchmarkTop100of1000000Scores-4 200 8064864 ns/op 4656596 B/op 232 allocs/op
PASS
ok github.com/blevesearch/bleve/search/collectors 7.385s
So, we're pretty close on the smaller datasets, and we scale better on the larger datasets.
We also show fewer allocations and bytes in all cases (some of this is artificial due to test cleanup).
this change means simple sort requirements no longer require
importing the search package (high-level API goal)
also the sort test at the top-level was changed to use this form
previously from JSON we would just deserialize strings like
"-abv" or "city" or "_id" or "_score" as simple sorts
on fields, ids or scores respectively
while this is simple and compact, it can be ambiguous (for
example if you have a field starting with - or if you have a field
named "_id" already. also, this simple syntax doesnt allow us
to specify more cmoplex options to deal with type/mode/missing
we keep support for the simple string syntax, but now also
recognize a more expressive syntax like:
{
"by": "field",
"field": "abv",
"desc": true,
"type": "string",
"mode": "min",
"missing": "first"
}
type, mode and missing are optional and default to
"auto", "default", and "last" respectively
The syntax used is an array of strings. The strings "_id" and
"_score" are special and reserved to mean sorting on the document
id and score repsectively. All other strings refer to the literal
field name with that value. If the string is prefixed with "-"
the order of that sort is descending, without it, it defaults to
ascending.
Examples:
"sort":["-abv","-_score"]
This will sort results in decreasing order of the "abv" field.
Results which have the same value of the "abv" field will then
be sorted by their score, also decreasing.
If no value for "sort" is provided in the search request the
default soring is the same as before, which is decreasing score.
Sreekanth Sivasankaran