Steve Yen
d1b84d4578
Merge pull request #828 from blevesearch/minor_fixes
...
posting iterator array positions clean up
2018-03-15 09:31:15 -07:00
Sreekanth Sivasankaran
1775602958
posting iterator array positions clean up,
...
max segment size limit adjustment for hit-1
optimisation
2018-03-15 14:40:00 +05:30
Sreekanth Sivasankaran
441065a41b
comments,simplification
2018-03-15 13:11:29 +05:30
Steve Yen
4af65a7846
scorch zap prealloc buf via estimate from previous interim work
2018-03-14 09:32:14 -07:00
Steve Yen
985082d5d2
Merge pull request #824 from steveyen/reuse-interim-vellum
...
scorch zap optimize interim's reuse of vellum builders
2018-03-14 08:10:43 -07:00
Steve Yen
7578ff7cb8
scorch zap optimize interim's reuse of vellum builders
...
Since interim structs are now sync.Pool'ed, we can now also hold onto
and reuse the associated vellum builder.
2018-03-14 07:49:28 -07:00
Abhinav Dangeti
2c69a5651b
Merge pull request #825 from abhinavdangeti/master
...
MB-27385: De-duplicate the list of requested fields
2018-03-13 14:33:13 -07:00
abhinavdangeti
715144d632
MB-27385: De-duplicate the list of requested fields
...
De-duplicate the list of fields provided by the client as part
of the search request, so as to not inadvertantly load the same
stored field more than once.
2018-03-13 14:19:02 -07:00
Steve Yen
62afdf4ac1
Merge pull request #823 from blevesearch/max_segment_size
...
adding maxsegment size limit checks
2018-03-13 07:52:27 -07:00
Sreekanth Sivasankaran
debbcd7d47
adding maxsegment size limit checks
2018-03-13 17:35:54 +05:30
Sreekanth Sivasankaran
19318194fa
moving to new offset slice format
2018-03-13 14:06:48 +05:30
Sreekanth Sivasankaran
5271b582bb
Merge branch 'master' of https://github.com/blevesearch/bleve into loadchunk_minor
2018-03-13 11:59:29 +05:30
Steve Yen
a526fe70f3
Merge pull request #822 from steveyen/scorch-reuse-roaring-and-sync-Pool
...
reuse roaring Bitmaps and also use sync.Pool for interim data
2018-03-12 13:20:51 -07:00
Steve Yen
dbfc5e9130
scorch zap reuse interim freq/norm/loc slices
2018-03-12 10:04:11 -07:00
Steve Yen
07901910e2
scorch zap reuse roaring Bitmap in prepareDicts() slice growth
...
In this change, if the postings/postingsLocs slices need to be grown,
then copy over and reuse any of the preallocated roaring Bitmap's from
the old slice.
2018-03-12 09:19:38 -07:00
Steve Yen
b1f3969521
scorch zap reuse roaring Bitmap in postings lists
2018-03-12 09:18:11 -07:00
Steve Yen
cad88096ca
scorch zap reuse roaring Bitmap during merge
2018-03-12 09:17:37 -07:00
Steve Yen
c4ceffe584
scorch zap sync Pool for interim data
2018-03-12 09:17:37 -07:00
Steve Yen
531800c479
scorch zap use roaring Add() instead of AddInt()
...
This change invokes Add() directly as AddInt() is a convenience
wrapper around Add().
2018-03-12 09:17:37 -07:00
Sreekanth Sivasankaran
f9545bef2f
Merge pull request #800 from blevesearch/numsnapshots_config
...
making NumSnapshotsToKeep configurable
2018-03-12 20:59:03 +05:30
Sreekanth Sivasankaran
90aa91105a
handling only int, float64 values
2018-03-12 20:24:51 +05:30
Steve Yen
6df6a036d8
Merge pull request #817 from steveyen/zap-no-longer-uses-mem-segment
...
scorch zap no longer uses mem segment
2018-03-12 07:54:10 -07:00
Steve Yen
2e122eba0c
Merge pull request #818 from steveyen/optimize-no-roaring-when-1-hit
...
scorch zap optimize to avoid bitmaps for 1-hit posting lists
2018-03-12 07:48:10 -07:00
Sreekanth Sivasankaran
aaccf59191
docValue space savings
...
merging the doc value length and loc
slices into a single offset slice as that
is enough to compute the starting offset and
length of the the doc values data for a given
document inside a docValue chunk.
2018-03-12 15:36:46 +05:30
Steve Yen
2a20a36e15
scorch zap optimimze to avoid bitmaps for 1-hit posting lists
...
This commit avoids creating roaring.Bitmap's (which would have just a
single entry) when a postings list/iterator represents a single
"1-hit" encoding.
2018-03-10 06:33:09 -08:00
Steve Yen
5abf7b7a19
scorch zap remove mem.Segment usage from persist / build.go
2018-03-09 15:23:58 -08:00
Steve Yen
eade78be2f
scorch zap unit tests no longer use mem.Segment
2018-03-09 15:23:58 -08:00
Steve Yen
e82774ad20
scorch zap AnalysisResultsToSegmentBase()
...
AnalysisResultsToSegmentBase() allows analysis results to be directly
converted into a zap-encoded SegmentBase, which can then be introduced
onto the root, avoiding the creation of mem.Segment data structures.
This leads to some reduction of garbage memory allocations.
The grouping and sorting and shaping of the postings list information
is taken from the mem.Segment codepaths.
The encoding of stored fields reuses functions from zap's merger,
which has the largest savings of garbage memory avoidance.
And, the encoding of tf/loc chunks, postings & dictionary information
also follows the approach used by zap's merger, which also has some
savings of garbage memory avoidance.
In future changes, the mem.Segment dependencies will be removed from
zap, which should result in a smaller codebase.
2018-03-09 15:22:30 -08:00
Steve Yen
3884cf4d12
scorch zap writePostings() helper func refactored out
2018-03-09 13:29:28 -08:00
Abhinav Dangeti
fa52ff856a
Merge pull request #814 from abhinavdangeti/master
...
Adding RoaringBitmap/roaring to the bleve vendor manifest
2018-03-09 10:57:12 -08:00
abhinavdangeti
8e8c3ee8c9
Adding RoaringBitmap/roaring to the bleve vendor manifest
2018-03-09 10:39:49 -08:00
Sreekanth Sivasankaran
d6522e7e17
minor optimisation to loadChunk method
2018-03-09 16:10:39 +05:30
Sreekanth Sivasankaran
b04909d3ee
adding the integer parser utility
2018-03-09 11:05:17 +05:30
Abhinav Dangeti
b38a61d4cf
Merge pull request #812 from abhinavdangeti/search-callbacks
...
MB-28562: Support search query callbacks before and after execution
2018-03-08 15:11:32 -08:00
Steve Yen
614e6f19f0
Merge pull request #813 from steveyen/reuse-fieldLens-docMap
...
scorch mem processDocument reuses fieldLens/docMap arrays
2018-03-08 14:31:20 -08:00
abhinavdangeti
40f63baeb9
MB-28562: Support search query callbacks before and after execution
...
+ SearchQueryStartCallback
+ SearchQueryEndCallback
2018-03-08 13:35:51 -08:00
Steve Yen
25beba615d
scorch mem processDocument reuses fieldLens/docMap arrays
...
This change produces less garbage by switching from a map[uint16]'s to
array's for the fieldLens and docMap, and then reusing those arrays
across multiple processDocument() calls.
2018-03-08 13:04:51 -08:00
Steve Yen
e9bbca4270
Merge pull request #811 from steveyen/chumbawamba
...
scorch zap optimize FST val encoding for terms with 1 hit
2018-03-08 13:04:24 -08:00
Abhinav Dangeti
676c85c935
Merge pull request #810 from abhinavdangeti/master
...
MB-28163: Provide an API to estimate the RAM needed for SearchResult
2018-03-08 12:01:17 -08:00
Steve Yen
eac9808990
scorch zap optimize FST val encoding for terms with 1 hit
...
NOTE: this is a scorch zap file format change / bump to version 4.
In this optimization, the uint64 val stored in the vellum FST (term
dictionary) now may either be a uint64 postingsOffset (same as before
this change) or a uint64 encoding of the docNum + norm (in the case
where a term appears in just a single doc).
2018-03-08 09:19:54 -08:00
abhinavdangeti
33ef4ce35e
MB-28163: Provide an API to estimate the RAM needed for SearchResult
...
exported API: MemoryNeededForSearchResult(req *SearchRequest)
2018-03-07 12:08:16 -08:00
Steve Yen
f04226d10b
Merge pull request #809 from steveyen/TestRoaringSizes
...
added TestRoaringSizes()
2018-03-07 12:01:24 -08:00
Steve Yen
1e2bb14f13
added TestRoaringSizes()
2018-03-07 10:53:24 -08:00
Steve Yen
0ec4a1935a
Merge pull request #808 from steveyen/more-scorch-optimizing
...
err fix and more scorch optimizing
2018-03-07 10:39:20 -08:00
Abhinav Dangeti
06be1ad72e
Merge pull request #806 from abhinavdangeti/master
...
Fixing the scorch search request memory estimate
2018-03-07 10:11:24 -08:00
Steve Yen
2b5da7a819
go fmt
2018-03-07 09:12:55 -08:00
Steve Yen
59eb70d020
scorch zap remove unused chunkedIntCoder field
2018-03-07 09:11:10 -08:00
Steve Yen
79f28b7c93
scorch fix persistDocValues() err return
2018-03-07 09:11:10 -08:00
Steve Yen
8c0f402d4b
scorch zap optimize processDocument() loc inner loop
2018-03-07 09:11:10 -08:00
Steve Yen
15242af465
Merge pull request #805 from steveyen/optimize-scorch-mem-processField
...
Optimize scorch processField() inner loop and writeRoaringWithLen()
2018-03-07 09:09:57 -08:00