Steve Yen
745575a6c1
scorch zap mergeStoredAndRemap uses array indexing, not append()
...
Since we have right array size preallocated, we don't need the extra
capacity checking of append().
2018-01-27 11:35:10 -08:00
Steve Yen
8dd17a3b20
scorch zap mergeStoredAndRemap uses continue for less indentation
2018-01-27 11:35:10 -08:00
Steve Yen
0041664bc4
scorch zap merge computeNewDocCount() optimize 1 variable
2018-01-27 11:35:10 -08:00
Steve Yen
6985db13a0
scorch zap merge reuses docNumbers array
2018-01-27 11:35:10 -08:00
Steve Yen
916bbf4125
scorch zap merge prealloc's docTermMap capacity
2018-01-27 11:35:10 -08:00
Steve Yen
56cdb68f35
scorch zap merge checks err2 not err
...
Also, optimize the appending of the termSeparator so that the
docTermMap is accessed and updated just once.
2018-01-27 11:35:10 -08:00
Steve Yen
3030d4edb5
scorch zap merge preallocs segNewDocNums capacity
2018-01-27 11:35:10 -08:00
Steve Yen
9038d75c98
scorch zap allocate govarint.U64Base128Encoder just once
...
Instead of allocating a govarint.U64Base128Encoder in the inner loop,
allocate it just once on the outside, as it appears that it's just a
thin wrapper around binary.PutUvarint().
2018-01-27 11:35:10 -08:00
Steve Yen
10dd5489c2
scorch zap Dict.postingsList() takes []byte for more mem control
...
This allows callers that already have a []byte term to avoid
string'ification garbage.
2018-01-27 11:35:10 -08:00
Steve Yen
6a17ff48c7
scorch zap removed uneeded []byte cast of term
2018-01-27 11:35:10 -08:00
Steve Yen
d389e2bb40
scorch zap merge file cleanup on error, and some minor prealloc's
2018-01-27 11:35:10 -08:00
Steve Yen
37121c3b49
scorch zap writeRoaringWithLen optimized with reused bufs
2018-01-27 11:35:10 -08:00
Steve Yen
5a035dc9aa
scorch zap in-memory segment representation (SegmentBase)
...
The zap SegmentBase struct is a refactoring of the zap Segment into
the subset of fields that are needed for read-only ops, without any
persistence related info. This allows us to use zap's optimized data
encoding as scorch's in-memory segments.
The zap Segment struct now embeds a zap SegmentBase struct, and layers
on persistence. Both the zap Segment and zap SegmentBase implement
scorch's Segment interface.
2018-01-27 11:35:10 -08:00
Sreekanth Sivasankaran
448201243a
removed redundant buf writer, and checks
2017-12-30 16:54:06 +05:30
Sreekanth Sivasankaran
c8df014c0c
Updated readme, zap version, added new docvalue cmd,
...
fixed the footer and fields cmd,
interface name updated
2017-12-29 21:39:29 +05:30
Sreekanth Sivasankaran
76f827f469
docValue persist changes
...
docValues are persisted along with the index,
in a columnar fashion per field with variable
sized chunking for quick look up.
-naive chunk level caching is added per field
-data part inside a chunk is snappy compressed
-metaHeader inside the chunk index the dv values
inside the uncompressed data part
-all the fields are docValue persisted in this iteration
2017-12-28 12:05:33 +05:30
Steve Yen
67e0e5973b
scorch mergeStoredAndRemap() memory reuse
...
In mergeStoredAndRemap(), instead of allocating new hashmaps for each
document, this commit reuses some arrays that are indexed by fieldId.
2017-12-20 15:18:22 -08:00
Steve Yen
c155255506
scorch optimize zap.Merge() to reuse some buffers
2017-12-20 14:59:53 -08:00
Steve Yen
f6b506134b
import couchbase/vellum instead of couchbaselabs/vellum
...
Also, scrubbed an old couchbaselabs/moss reference in comments.
Also, go fmt.
2017-12-19 10:49:57 -08:00
Marty Schoch
e1b0c61e2a
fix bug in handling iterator-done
2017-12-13 22:08:06 -05:00
Marty Schoch
a0e12b2640
add license to a few files missing it
2017-12-13 16:12:29 -05:00
Marty Schoch
85e15628ee
major refactoring of posting details
2017-12-13 16:10:06 -05:00
Marty Schoch
6e2207c445
additional refactoring of build/merge
2017-12-13 15:22:13 -05:00
Marty Schoch
289dc398bd
more refacotring of build/merge
2017-12-13 14:26:11 -05:00
Marty Schoch
1cd3fd7fbe
extrac common functionality between build/merge
2017-12-13 14:06:54 -05:00
Marty Schoch
f83c9f2a20
initial cut of merger that actually introduces changes
2017-12-13 13:41:03 -05:00
Marty Schoch
57121e40a8
fix issues identified by errcheck
2017-12-12 11:41:14 -05:00
Marty Schoch
665c3c80ff
initial cut of zap segment merging
2017-12-12 11:21:55 -05:00