65fed52d0b
Do not account IndexReader's size in the query RAM estimate
...
Since its just the pointer size of the IndexReader that is
being accounted for while estimating the RAM needed to
execute a search query, get rid of the Size() API in the
IndexReader interface.
2018-03-15 13:23:58 -07:00
d1b84d4578
Merge pull request #828 from blevesearch/minor_fixes
...
posting iterator array positions clean up
2018-03-15 09:31:15 -07:00
1775602958
posting iterator array positions clean up,
...
max segment size limit adjustment for hit-1
optimisation
2018-03-15 14:40:00 +05:30
985082d5d2
Merge pull request #824 from steveyen/reuse-interim-vellum
...
scorch zap optimize interim's reuse of vellum builders
2018-03-14 08:10:43 -07:00
7578ff7cb8
scorch zap optimize interim's reuse of vellum builders
...
Since interim structs are now sync.Pool'ed, we can now also hold onto
and reuse the associated vellum builder.
2018-03-14 07:49:28 -07:00
2c69a5651b
Merge pull request #825 from abhinavdangeti/master
...
MB-27385: De-duplicate the list of requested fields
2018-03-13 14:33:13 -07:00
715144d632
MB-27385: De-duplicate the list of requested fields
...
De-duplicate the list of fields provided by the client as part
of the search request, so as to not inadvertantly load the same
stored field more than once.
2018-03-13 14:19:02 -07:00
62afdf4ac1
Merge pull request #823 from blevesearch/max_segment_size
...
adding maxsegment size limit checks
2018-03-13 07:52:27 -07:00
debbcd7d47
adding maxsegment size limit checks
2018-03-13 17:35:54 +05:30
a526fe70f3
Merge pull request #822 from steveyen/scorch-reuse-roaring-and-sync-Pool
...
reuse roaring Bitmaps and also use sync.Pool for interim data
2018-03-12 13:20:51 -07:00
dbfc5e9130
scorch zap reuse interim freq/norm/loc slices
2018-03-12 10:04:11 -07:00
07901910e2
scorch zap reuse roaring Bitmap in prepareDicts() slice growth
...
In this change, if the postings/postingsLocs slices need to be grown,
then copy over and reuse any of the preallocated roaring Bitmap's from
the old slice.
2018-03-12 09:19:38 -07:00
b1f3969521
scorch zap reuse roaring Bitmap in postings lists
2018-03-12 09:18:11 -07:00
cad88096ca
scorch zap reuse roaring Bitmap during merge
2018-03-12 09:17:37 -07:00
c4ceffe584
scorch zap sync Pool for interim data
2018-03-12 09:17:37 -07:00
531800c479
scorch zap use roaring Add() instead of AddInt()
...
This change invokes Add() directly as AddInt() is a convenience
wrapper around Add().
2018-03-12 09:17:37 -07:00
f9545bef2f
Merge pull request #800 from blevesearch/numsnapshots_config
...
making NumSnapshotsToKeep configurable
2018-03-12 20:59:03 +05:30
90aa91105a
handling only int, float64 values
2018-03-12 20:24:51 +05:30
6df6a036d8
Merge pull request #817 from steveyen/zap-no-longer-uses-mem-segment
...
scorch zap no longer uses mem segment
2018-03-12 07:54:10 -07:00
2e122eba0c
Merge pull request #818 from steveyen/optimize-no-roaring-when-1-hit
...
scorch zap optimize to avoid bitmaps for 1-hit posting lists
2018-03-12 07:48:10 -07:00
2a20a36e15
scorch zap optimimze to avoid bitmaps for 1-hit posting lists
...
This commit avoids creating roaring.Bitmap's (which would have just a
single entry) when a postings list/iterator represents a single
"1-hit" encoding.
2018-03-10 06:33:09 -08:00
5abf7b7a19
scorch zap remove mem.Segment usage from persist / build.go
2018-03-09 15:23:58 -08:00
eade78be2f
scorch zap unit tests no longer use mem.Segment
2018-03-09 15:23:58 -08:00
e82774ad20
scorch zap AnalysisResultsToSegmentBase()
...
AnalysisResultsToSegmentBase() allows analysis results to be directly
converted into a zap-encoded SegmentBase, which can then be introduced
onto the root, avoiding the creation of mem.Segment data structures.
This leads to some reduction of garbage memory allocations.
The grouping and sorting and shaping of the postings list information
is taken from the mem.Segment codepaths.
The encoding of stored fields reuses functions from zap's merger,
which has the largest savings of garbage memory avoidance.
And, the encoding of tf/loc chunks, postings & dictionary information
also follows the approach used by zap's merger, which also has some
savings of garbage memory avoidance.
In future changes, the mem.Segment dependencies will be removed from
zap, which should result in a smaller codebase.
2018-03-09 15:22:30 -08:00
3884cf4d12
scorch zap writePostings() helper func refactored out
2018-03-09 13:29:28 -08:00
fa52ff856a
Merge pull request #814 from abhinavdangeti/master
...
Adding RoaringBitmap/roaring to the bleve vendor manifest
2018-03-09 10:57:12 -08:00
8e8c3ee8c9
Adding RoaringBitmap/roaring to the bleve vendor manifest
2018-03-09 10:39:49 -08:00
b04909d3ee
adding the integer parser utility
2018-03-09 11:05:17 +05:30
b38a61d4cf
Merge pull request #812 from abhinavdangeti/search-callbacks
...
MB-28562: Support search query callbacks before and after execution
2018-03-08 15:11:32 -08:00
614e6f19f0
Merge pull request #813 from steveyen/reuse-fieldLens-docMap
...
scorch mem processDocument reuses fieldLens/docMap arrays
2018-03-08 14:31:20 -08:00
40f63baeb9
MB-28562: Support search query callbacks before and after execution
...
+ SearchQueryStartCallback
+ SearchQueryEndCallback
2018-03-08 13:35:51 -08:00
25beba615d
scorch mem processDocument reuses fieldLens/docMap arrays
...
This change produces less garbage by switching from a map[uint16]'s to
array's for the fieldLens and docMap, and then reusing those arrays
across multiple processDocument() calls.
2018-03-08 13:04:51 -08:00
e9bbca4270
Merge pull request #811 from steveyen/chumbawamba
...
scorch zap optimize FST val encoding for terms with 1 hit
2018-03-08 13:04:24 -08:00
676c85c935
Merge pull request #810 from abhinavdangeti/master
...
MB-28163: Provide an API to estimate the RAM needed for SearchResult
2018-03-08 12:01:17 -08:00
eac9808990
scorch zap optimize FST val encoding for terms with 1 hit
...
NOTE: this is a scorch zap file format change / bump to version 4.
In this optimization, the uint64 val stored in the vellum FST (term
dictionary) now may either be a uint64 postingsOffset (same as before
this change) or a uint64 encoding of the docNum + norm (in the case
where a term appears in just a single doc).
2018-03-08 09:19:54 -08:00
33ef4ce35e
MB-28163: Provide an API to estimate the RAM needed for SearchResult
...
exported API: MemoryNeededForSearchResult(req *SearchRequest)
2018-03-07 12:08:16 -08:00
f04226d10b
Merge pull request #809 from steveyen/TestRoaringSizes
...
added TestRoaringSizes()
2018-03-07 12:01:24 -08:00
1e2bb14f13
added TestRoaringSizes()
2018-03-07 10:53:24 -08:00
0ec4a1935a
Merge pull request #808 from steveyen/more-scorch-optimizing
...
err fix and more scorch optimizing
2018-03-07 10:39:20 -08:00
06be1ad72e
Merge pull request #806 from abhinavdangeti/master
...
Fixing the scorch search request memory estimate
2018-03-07 10:11:24 -08:00
2b5da7a819
go fmt
2018-03-07 09:12:55 -08:00
59eb70d020
scorch zap remove unused chunkedIntCoder field
2018-03-07 09:11:10 -08:00
79f28b7c93
scorch fix persistDocValues() err return
2018-03-07 09:11:10 -08:00
8c0f402d4b
scorch zap optimize processDocument() loc inner loop
2018-03-07 09:11:10 -08:00
15242af465
Merge pull request #805 from steveyen/optimize-scorch-mem-processField
...
Optimize scorch processField() inner loop and writeRoaringWithLen()
2018-03-07 09:09:57 -08:00
c813165d4b
Merge pull request #798 from blevesearch/compaction_bytes_stats
...
adding compaction_written_bytes/sec stats to scorch
2018-03-07 22:29:13 +05:30
73ed8e248d
fixing the indentation issues.
...
looks like it happened during the web based conflict resolution..
2018-03-07 18:34:54 +05:30
e0369a3553
Merge branch 'master' into compaction_bytes_stats
2018-03-07 14:47:33 +05:30
2a9739ee1b
naming change, interface removal
2018-03-07 14:43:33 +05:30
5c721226cf
Fixing the scorch search request memory estimate
...
Do not re-account for certain referenced data in the zap structures.
New estimates:
ESTIMATE BENCHMEM
TermQuery 11396 12437
MatchQuery 12244 12951
DisjunctionQuery (Term queries) 20644 20709
2018-03-06 16:03:10 -08:00
abhinavdangeti