5df53c8e1f
scorch zap file merger uses 1MB buffered writer
...
pprof of bleve-blast was showing file merging was in syscall/write a
lot. The bufio.NewWriter() provides a default buffer size of 4K,
which is too small, and using bufio.NewWriterSize(1MB buffer size)
leads to syscall/write dropping out of the file merging flame graphs.
2018-03-16 11:49:53 -07:00
60bdf6d247
Return an error when the snapshotEpoch is invalid
...
Avoiding this stacktrace (SIGSEGV) while using bleve scorch cmd-line utility
when snapshotEpoch provided is invalid:
github.com/blevesearch/bleve/index/scorch.(*IndexSnapshot).Segments(...)
/Users/abhinavdangeti/Documents/couchbaseV/godeps/src/github.com/blevesearch/bleve/index/scorch/snapshot_index.go:56
github.com/blevesearch/bleve/cmd/bleve/cmd/scorch.glob..func1(0x1f347e0, 0xc4201f1400, 0x2, 0x2, 0x0, 0x0)
/Users/abhinavdangeti/Documents/couchbaseV/godeps/src/github.com/blevesearch/bleve/cmd/bleve/cmd/scorch/ascii.go:43 +0xe4
github.com/blevesearch/bleve/cmd/bleve/vendor/github.com/spf13/cobra.(*Command).execute(0x1f347e0, 0xc4201f12e0, 0x2, 0x2, 0x1f347e0, 0xc4201f12e0)
/Users/abhinavdangeti/Documents/couchbaseV/godeps/src/github.com/blevesearch/bleve/cmd/bleve/vendor/github.com/spf13/cobra/command.go:646 +0x3e8
github.com/blevesearch/bleve/cmd/bleve/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x1f334c0, 0x0, 0x0, 0x0)
/Users/abhinavdangeti/Documents/couchbaseV/godeps/src/github.com/blevesearch/bleve/cmd/bleve/vendor/github.com/spf13/cobra/command.go:737 +0x2fe
github.com/blevesearch/bleve/cmd/bleve/vendor/github.com/spf13/cobra.(*Command).Execute(0x1f334c0, 0x0, 0x0)
/Users/abhinavdangeti/Documents/couchbaseV/godeps/src/github.com/blevesearch/bleve/cmd/bleve/vendor/github.com/spf13/cobra/command.go:695 +0x2b
github.com/blevesearch/bleve/cmd/bleve/cmd.Execute()
/Users/abhinavdangeti/Documents/couchbaseV/godeps/src/github.com/blevesearch/bleve/cmd/bleve/cmd/root.go:74 +0x31
main.main()
/Users/abhinavdangeti/Documents/couchbaseV/goproj/src/github.com/couchbase/cbft/cmd/cbft-bleve/main.go:39 +0x1cb
2018-03-16 11:43:25 -07:00
3d1cea0556
Merge pull request #834 from steveyen/more-mem-reuse-and-optimizations
...
More mem reuse and optimizations
2018-03-16 11:41:10 -07:00
b411e65234
scorch zap optimize postingsIterator reuse of freq/locChunkOffsets
2018-03-16 11:22:50 -07:00
e52eb84e37
scorch zap optimize merge when deletion bitmap is empty
...
This change detects whether a deletion bitmap is empty, and treats
that as a nil bitmap, which allows further postings iterator codepaths
to avoid roaring bitmap operations (like, AndNot(docNums, drops)).
2018-03-16 11:22:50 -07:00
5411d9ae4f
Merge pull request #826 from steveyen/scorch-estimate-buf-size
...
estimate interim buffer size based on previous results
2018-03-16 11:22:42 -07:00
4f33b4be44
Merge pull request #832 from mschoch/rename-size-full
...
rename SizeFull to Size
2018-03-16 12:10:48 -04:00
11ff31c2f9
rename SizeFull to Size
2018-03-16 11:31:47 -04:00
9a87593fd7
Merge pull request #830 from mschoch/avoid-app-herder-hot-lock
...
memoize the size of an entire index snapshot
2018-03-16 11:28:22 -04:00
f1c26e29f0
Merge branch 'master' into avoid-app-herder-hot-lock
2018-03-16 10:30:34 -04:00
dee639ccc0
Merge pull request #829 from abhinavdangeti/master
...
Do not account IndexReader's size in the query RAM estimate
2018-03-16 10:28:24 -04:00
53bf29763b
Merge pull request #821 from blevesearch/minor_docvalue_space_savings
...
docValue space savings
2018-03-16 09:12:13 +05:30
53c3cab512
Merge branch 'master' into minor_docvalue_space_savings
2018-03-16 08:53:57 +05:30
23cebae5a8
Merge pull request #815 from blevesearch/loadchunk_minor
...
minor optimisation to loadChunk method
2018-03-16 08:15:37 +05:30
45e0e5c666
memoize the size of an entire index snapshot
...
by memoizing the size of index snapshots and their
constituent parts, we significantly reduce the amount
of time that the lock is held in the app_herder, when
calculating the total memory used
2018-03-15 17:25:05 -04:00
65fed52d0b
Do not account IndexReader's size in the query RAM estimate
...
Since its just the pointer size of the IndexReader that is
being accounted for while estimating the RAM needed to
execute a search query, get rid of the Size() API in the
IndexReader interface.
2018-03-15 13:23:58 -07:00
d1155c223a
zap version bump, changed the offset slice format
...
,UTs
2018-03-15 23:25:53 +05:30
d1b84d4578
Merge pull request #828 from blevesearch/minor_fixes
...
posting iterator array positions clean up
2018-03-15 09:31:15 -07:00
1775602958
posting iterator array positions clean up,
...
max segment size limit adjustment for hit-1
optimisation
2018-03-15 14:40:00 +05:30
441065a41b
comments,simplification
2018-03-15 13:11:29 +05:30
4af65a7846
scorch zap prealloc buf via estimate from previous interim work
2018-03-14 09:32:14 -07:00
985082d5d2
Merge pull request #824 from steveyen/reuse-interim-vellum
...
scorch zap optimize interim's reuse of vellum builders
2018-03-14 08:10:43 -07:00
7578ff7cb8
scorch zap optimize interim's reuse of vellum builders
...
Since interim structs are now sync.Pool'ed, we can now also hold onto
and reuse the associated vellum builder.
2018-03-14 07:49:28 -07:00
2c69a5651b
Merge pull request #825 from abhinavdangeti/master
...
MB-27385: De-duplicate the list of requested fields
2018-03-13 14:33:13 -07:00
715144d632
MB-27385: De-duplicate the list of requested fields
...
De-duplicate the list of fields provided by the client as part
of the search request, so as to not inadvertantly load the same
stored field more than once.
2018-03-13 14:19:02 -07:00
62afdf4ac1
Merge pull request #823 from blevesearch/max_segment_size
...
adding maxsegment size limit checks
2018-03-13 07:52:27 -07:00
debbcd7d47
adding maxsegment size limit checks
2018-03-13 17:35:54 +05:30
19318194fa
moving to new offset slice format
2018-03-13 14:06:48 +05:30
5271b582bb
Merge branch 'master' of https://github.com/blevesearch/bleve into loadchunk_minor
2018-03-13 11:59:29 +05:30
a526fe70f3
Merge pull request #822 from steveyen/scorch-reuse-roaring-and-sync-Pool
...
reuse roaring Bitmaps and also use sync.Pool for interim data
2018-03-12 13:20:51 -07:00
dbfc5e9130
scorch zap reuse interim freq/norm/loc slices
2018-03-12 10:04:11 -07:00
07901910e2
scorch zap reuse roaring Bitmap in prepareDicts() slice growth
...
In this change, if the postings/postingsLocs slices need to be grown,
then copy over and reuse any of the preallocated roaring Bitmap's from
the old slice.
2018-03-12 09:19:38 -07:00
b1f3969521
scorch zap reuse roaring Bitmap in postings lists
2018-03-12 09:18:11 -07:00
cad88096ca
scorch zap reuse roaring Bitmap during merge
2018-03-12 09:17:37 -07:00
c4ceffe584
scorch zap sync Pool for interim data
2018-03-12 09:17:37 -07:00
531800c479
scorch zap use roaring Add() instead of AddInt()
...
This change invokes Add() directly as AddInt() is a convenience
wrapper around Add().
2018-03-12 09:17:37 -07:00
f9545bef2f
Merge pull request #800 from blevesearch/numsnapshots_config
...
making NumSnapshotsToKeep configurable
2018-03-12 20:59:03 +05:30
90aa91105a
handling only int, float64 values
2018-03-12 20:24:51 +05:30
6df6a036d8
Merge pull request #817 from steveyen/zap-no-longer-uses-mem-segment
...
scorch zap no longer uses mem segment
2018-03-12 07:54:10 -07:00
2e122eba0c
Merge pull request #818 from steveyen/optimize-no-roaring-when-1-hit
...
scorch zap optimize to avoid bitmaps for 1-hit posting lists
2018-03-12 07:48:10 -07:00
aaccf59191
docValue space savings
...
merging the doc value length and loc
slices into a single offset slice as that
is enough to compute the starting offset and
length of the the doc values data for a given
document inside a docValue chunk.
2018-03-12 15:36:46 +05:30
2a20a36e15
scorch zap optimimze to avoid bitmaps for 1-hit posting lists
...
This commit avoids creating roaring.Bitmap's (which would have just a
single entry) when a postings list/iterator represents a single
"1-hit" encoding.
2018-03-10 06:33:09 -08:00
5abf7b7a19
scorch zap remove mem.Segment usage from persist / build.go
2018-03-09 15:23:58 -08:00
eade78be2f
scorch zap unit tests no longer use mem.Segment
2018-03-09 15:23:58 -08:00
e82774ad20
scorch zap AnalysisResultsToSegmentBase()
...
AnalysisResultsToSegmentBase() allows analysis results to be directly
converted into a zap-encoded SegmentBase, which can then be introduced
onto the root, avoiding the creation of mem.Segment data structures.
This leads to some reduction of garbage memory allocations.
The grouping and sorting and shaping of the postings list information
is taken from the mem.Segment codepaths.
The encoding of stored fields reuses functions from zap's merger,
which has the largest savings of garbage memory avoidance.
And, the encoding of tf/loc chunks, postings & dictionary information
also follows the approach used by zap's merger, which also has some
savings of garbage memory avoidance.
In future changes, the mem.Segment dependencies will be removed from
zap, which should result in a smaller codebase.
2018-03-09 15:22:30 -08:00
3884cf4d12
scorch zap writePostings() helper func refactored out
2018-03-09 13:29:28 -08:00
fa52ff856a
Merge pull request #814 from abhinavdangeti/master
...
Adding RoaringBitmap/roaring to the bleve vendor manifest
2018-03-09 10:57:12 -08:00
8e8c3ee8c9
Adding RoaringBitmap/roaring to the bleve vendor manifest
2018-03-09 10:39:49 -08:00
d6522e7e17
minor optimisation to loadChunk method
2018-03-09 16:10:39 +05:30
b04909d3ee
adding the integer parser utility
2018-03-09 11:05:17 +05:30
Steve Yen