7e36109b3c
MB-28162: Provide API to estimate memory needed to run a search query
...
This API (unexported) will estimate the amount of memory needed to execute
a search query over an index before the collector begins data collection.
Sample estimates for certain queries:
{Size: 10, BenchmarkUpsidedownSearchOverhead}
ESTIMATE BENCHMEM
TermQuery 4616 4796
MatchQuery 5210 5405
DisjunctionQuery (Match queries) 7700 8447
DisjunctionQuery (Term queries) 6514 6591
ConjunctionQuery (Match queries) 7524 8175
Nested disjunction query (disjunction of disjunctions) 10306 10708
…
2018-03-06 13:53:42 -08:00
c74e08f039
BREAKING API CHANGE - use stdlib context pkg
...
update all references to context to use std lib pkg
2018-02-27 11:33:43 -08:00
a4a34cc3b2
topn collector switch approach based on size+skip
...
we now use the slice store when size+skip <= 10
and use the heap store when size+skip > 10
here are the new perf numbers:
go test -run=xxx -bench=. -benchmem
BenchmarkTop10of0Scores-4 1000000 1150 ns/op 2304 B/op 15 allocs/op
BenchmarkTop10of3Scores-4 1000000 1417 ns/op 2304 B/op 18 allocs/op
BenchmarkTop10of10Scores-4 1000000 2133 ns/op 2312 B/op 25 allocs/op
BenchmarkTop10of25Scores-4 500000 3410 ns/op 2464 B/op 26 allocs/op
BenchmarkTop10of50Scores-4 300000 5174 ns/op 2464 B/op 26 allocs/op
BenchmarkTop10of10000Scores-4 5000 342955 ns/op 2488 B/op 26 allocs/op
BenchmarkTop100of0Scores-4 300000 4796 ns/op 18320 B/op 15 allocs/op
BenchmarkTop100of3Scores-4 300000 5160 ns/op 18352 B/op 19 allocs/op
BenchmarkTop100of10Scores-4 200000 6354 ns/op 18408 B/op 26 allocs/op
BenchmarkTop100of25Scores-4 200000 10023 ns/op 18568 B/op 41 allocs/op
BenchmarkTop100of50Scores-4 100000 16821 ns/op 18832 B/op 66 allocs/op
BenchmarkTop100of10000Scores-4 3000 508989 ns/op 19760 B/op 117 allocs/op
BenchmarkTop1000of10000Scores-4 1000 1814198 ns/op 184768 B/op 1017 allocs/op
BenchmarkTop10000of100000Scores-4 50 26623920 ns/op 1939592 B/op 19024 allocs/op
BenchmarkTop10of100000Scores-4 500 3730204 ns/op 2496 B/op 26 allocs/op
BenchmarkTop100of100000Scores-4 300 4057127 ns/op 19912 B/op 117 allocs/op
BenchmarkTop1000of100000Scores-4 200 6390180 ns/op 186200 B/op 1017 allocs/op
BenchmarkTop10000of1000000Scores-4 20 82785756 ns/op 1963897 B/op 19024 allocs/op
PASS
ok github.com/blevesearch/bleve/search/collector 31.537s
Previously with heap:
go test -run=xxx -bench=. -benchmem
BenchmarkTop10of0Scores-4 1000000 1216 ns/op 2288 B/op 15 allocs/op
BenchmarkTop10of3Scores-4 1000000 1593 ns/op 2320 B/op 19 allocs/op
BenchmarkTop10of10Scores-4 500000 2734 ns/op 2376 B/op 26 allocs/op
BenchmarkTop10of25Scores-4 300000 5077 ns/op 2520 B/op 27 allocs/op
BenchmarkTop10of50Scores-4 200000 6875 ns/op 2528 B/op 27 allocs/op
BenchmarkTop10of10000Scores-4 3000 351210 ns/op 2552 B/op 27 allocs/op
BenchmarkTop100of0Scores-4 300000 4846 ns/op 18304 B/op 15 allocs/op
BenchmarkTop100of3Scores-4 300000 5357 ns/op 18336 B/op 19 allocs/op
BenchmarkTop100of10Scores-4 200000 6462 ns/op 18392 B/op 26 allocs/op
BenchmarkTop100of25Scores-4 200000 10012 ns/op 18552 B/op 41 allocs/op
BenchmarkTop100of50Scores-4 100000 17089 ns/op 18816 B/op 66 allocs/op
BenchmarkTop100of10000Scores-4 3000 528193 ns/op 19744 B/op 117 allocs/op
BenchmarkTop1000of10000Scores-4 1000 1859447 ns/op 184752 B/op 1017 allocs/op
BenchmarkTop10000of100000Scores-4 50 28005664 ns/op 1939576 B/op 19024 allocs/op
BenchmarkTop10of100000Scores-4 300 4120091 ns/op 2560 B/op 27 allocs/op
BenchmarkTop100of100000Scores-4 300 4325227 ns/op 19896 B/op 117 allocs/op
BenchmarkTop1000of100000Scores-4 200 6799804 ns/op 186184 B/op 1017 allocs/op
BenchmarkTop10000of1000000Scores-4 20 88494230 ns/op 1963881 B/op 19024 allocs/op
PASS
ok github.com/blevesearch/bleve/search/collector 30.198s
Previously with slice:
go test -run=xxx -bench=. -benchmem
BenchmarkTop10of0Scores-4 1000000 1202 ns/op 2288 B/op 15 allocs/op
BenchmarkTop10of3Scores-4 1000000 1453 ns/op 2288 B/op 18 allocs/op
BenchmarkTop10of10Scores-4 1000000 2162 ns/op 2296 B/op 25 allocs/op
BenchmarkTop10of25Scores-4 500000 3420 ns/op 2448 B/op 26 allocs/op
BenchmarkTop10of50Scores-4 300000 5336 ns/op 2448 B/op 26 allocs/op
BenchmarkTop10of10000Scores-4 5000 356733 ns/op 2472 B/op 26 allocs/op
BenchmarkTop100of0Scores-4 300000 4877 ns/op 18304 B/op 15 allocs/op
BenchmarkTop100of3Scores-4 300000 5132 ns/op 18304 B/op 18 allocs/op
BenchmarkTop100of10Scores-4 200000 5787 ns/op 18312 B/op 25 allocs/op
BenchmarkTop100of25Scores-4 200000 8083 ns/op 18344 B/op 40 allocs/op
BenchmarkTop100of50Scores-4 100000 14419 ns/op 18400 B/op 65 allocs/op
BenchmarkTop100of10000Scores-4 2000 665401 ns/op 18848 B/op 116 allocs/op
BenchmarkTop1000of10000Scores-4 100 15417063 ns/op 176560 B/op 1016 allocs/op
BenchmarkTop10000of100000Scores-4 1 1860011022 ns/op 1857960 B/op 19023 allocs/op
BenchmarkTop10of100000Scores-4 300 4099276 ns/op 2480 B/op 26 allocs/op
BenchmarkTop100of100000Scores-4 300 4533645 ns/op 18984 B/op 116 allocs/op
BenchmarkTop1000of100000Scores-4 50 30519235 ns/op 178008 B/op 1016 allocs/op
BenchmarkTop10000of1000000Scores-4 1 3483977385 ns/op 1882072 B/op 19023 allocs/op
PASS
ok github.com/blevesearch/bleve/search/collector 31.666s
It appears that this sucessfully gets the best of both, in these particular benchmark sizes.
2017-04-27 08:57:13 -04:00
952572718e
switch collector store impl from slice to heap
...
Additional testing has shown that the heap collector performs
significantly better when larger numbers of hits are requested.
The heap is also faster (though very close) when fewer (10) hits
are requested.
Here are the numbers from my laptop:
slice:
go test -run=xxx -bench=. -benchmem
BenchmarkTop10of10000Scores-4 5000 396943 ns/op 2472 B/op 26 allocs/op
BenchmarkTop100of10000Scores-4 2000 630894 ns/op 18848 B/op 116 allocs/op
BenchmarkTop1000of10000Scores-4 100 14996445 ns/op 176552 B/op 1016 allocs/op
BenchmarkTop10000of100000Scores-4 1 1878796320 ns/op 1857768 B/op 19023 allocs/op
BenchmarkTop10of100000Scores-4 500 3858309 ns/op 2480 B/op 26 allocs/op
BenchmarkTop100of100000Scores-4 300 4270086 ns/op 19000 B/op 116 allocs/op
BenchmarkTop1000of100000Scores-4 50 30163705 ns/op 178024 B/op 1016 allocs/op
BenchmarkTop10000of1000000Scores-4 1 3429557237 ns/op 1882008 B/op 19023 allocs/op
PASS
ok github.com/blevesearch/bleve/search/collector 16.316s
heap:
go test -run=xxx -bench=. -benchmem
BenchmarkTop10of10000Scores-4 5000 341064 ns/op 2552 B/op 27 allocs/op
BenchmarkTop100of10000Scores-4 3000 501922 ns/op 19744 B/op 117 allocs/op
BenchmarkTop1000of10000Scores-4 1000 1759088 ns/op 184744 B/op 1017 allocs/op
BenchmarkTop10000of100000Scores-4 50 25954696 ns/op 1939608 B/op 19024 allocs/op
BenchmarkTop10of100000Scores-4 500 3814933 ns/op 2560 B/op 27 allocs/op
BenchmarkTop100of100000Scores-4 300 4009369 ns/op 19896 B/op 117 allocs/op
BenchmarkTop1000of100000Scores-4 200 6397276 ns/op 186184 B/op 1017 allocs/op
BenchmarkTop10000of1000000Scores-4 20 81815315 ns/op 1963912 B/op 19024 allocs/op
PASS
ok github.com/blevesearch/bleve/search/collector 14.980s
2017-03-24 09:38:06 -07:00
bc7d8e3b35
fix perf regression, unnecessarily loading backindex
2017-03-04 15:23:16 -05:00
0eba2a3f0c
reduce garbage created while processing facets
...
previously we parsed/returned large sections of the documents
back index row in order to compute facet information. this
would require parsing the protobuf of the entire back index row.
unfortunately this creates considerable garbage.
this new version introduces a visitor/callback approach to
working with data inside the back index row. the benefit
of this approach is that we can let the higher-level code
see values, prior to any copies of data being made or
intermediate garbage being created. implementations of
the callback must copy any value which they would like to
retain beyond the callback.
NOTE: this approach is duplicates code from the
automatically generated protobuf code
NOTE: this approach assumes that the "field" field be serialized
before the "terms" field. This is guaranteed by our currently
generated protobuf encoder, and is recommended by the protobuf
spec. But, decoders SHOULD support them occuring in any order,
which we do not.
2017-03-02 17:00:46 -05:00
2f48d7fb02
fix misspellings
2016-10-02 12:11:15 -04:00
3a276153a3
actually rename packages to singular, not just directory name
2016-10-02 10:29:39 -04:00
2332455bd2
nicer formatting of license header
2016-10-02 10:13:14 -04:00
6bf9dd59ab
BREAKING CHANGE - additional package renaming
...
i recently learned that package names should also prefer the
singular form, not the plural form
2016-10-01 17:20:59 -04:00
abhinavdangeti