0
0
Commit Graph

237 Commits

Author SHA1 Message Date
Steve Yen
88c740095b scorch optimizations for mem.PostingsIterator.Next() & docTermMap
Due to the usage rules of iterators, mem.PostingsIterator.Next() can
reuse its returned Postings instance.

Also, there's a micro optimization in persistDocValues() for one fewer
access to the docTermMap in the inner-loop.
2018-03-03 11:31:18 -08:00
Steve Yen
a5253bfe2b scorch persister goes through introducer to affect root
This change allows the introducer to become the only goroutine to
modify the root, which in turn allows the introducer to greatly reduce
its root lock holding surface area.
2018-03-02 16:14:28 -08:00
Marty Schoch
30acc55d05 remove unnecessary scorch reader wrapper
we now use *IndexSnapshot directly
2018-03-02 14:03:54 -08:00
Steve Yen
d61d9e4cf6 scorch stats MaxBatchIntroTime and TotBatchIntroTime 2018-03-02 13:33:06 -08:00
Steve Yen
868a66279e scorch indexing time stat
Looks like this was forgotten along the way -- the stat for analysis
time was tracked correctly, but indexing time wasn't.
2018-03-02 11:07:39 -08:00
Steve Yen
7e5bb0bd8d renamed to CurOnDiskBytes/Files as those are gauges 2018-03-01 14:13:43 -08:00
Marty Schoch
0363b24dd4 update to use new vellum Reset API 2018-03-01 09:37:39 -08:00
Steve Yen
39f9cee910
Merge pull request #789 from steveyen/sreekanth-cb-scorch_stats
adding stats for scorch, with no gauges
2018-02-28 17:41:10 -08:00
Steve Yen
1b661ef844 stats cleanup, renaming, gauges replaced with counters 2018-02-28 17:03:28 -08:00
Steve Yen
7d46d2c7ae scorch zap intcoder encoder is never nil 2018-02-28 10:09:21 -08:00
Sreekanth Sivasankaran
4b742505aa adding stats for scorch 2018-02-28 15:31:55 +05:30
Steve Yen
dd7d93ee5e scorch zap loadChunk reuses Location slices 2018-02-27 18:01:48 -08:00
Steve Yen
4dbb4b1495 scorch zap posting reuses freqNorm & loc reader and decoder 2018-02-27 18:01:48 -08:00
Steve Yen
a32362ba2e MB-28403: scorch introduceMerge doesn't prealloc segments capacity
There's now multiple competing merge activities (file-merging and
in-memory merging during persistence), so the simple math to
precalculate capacity for the slice of segments in introduceMerge() no
longer works for all cases and might have negative capacity.

This change removes that (sometimes wrong) precalculation, and instead
depends on append() to grow the slice correctly.
2018-02-27 15:14:34 -08:00
Steve Yen
3f1dcb6078 scorch zap merge optimize drops lookup to outside of loop 2018-02-27 09:23:29 -08:00
Steve Yen
99ed127176 scorch zap merge optimize newDocNums lookup to outside of loop
And, also a "go fmt".
2018-02-26 14:23:55 -08:00
Steve Yen
98d5d7bd81 scorch zap chunkedIntCoder optimizations
The optimizations / changes include...

- reuse of a memory buf when serializing varint's.

- reuse of a govarint.U64Base128Encoder instance, as it's a thin,
  wrapper around an underlying chunkBuf, so Reset()'s on the
  chunkBuf is enough for encoder reuse.

- chunkedIntcoder.Write() method was changed to invoke w.Write() less
  often by forming a larger, reused buf.  Profiling and analysis
  showed w.Write() was getting called a lot, often with tiny 1 or 2
  byte inputs.  The theory is w.Write() and its underlying memmove()
  can be more efficient when provided with larger bufs.

- some repeated code removal, by reusing the Close() method.
2018-02-26 14:17:09 -08:00
Steve Yen
ce2332e111 scorch zap merge reuses tf/locEncoder across terms
The finishTerm() helper func that's invoked on every outer loop resets
the tf/locEncoders so they can be safely reused.
2018-02-26 11:37:11 -08:00
Marty Schoch
eca31dfd27
Merge pull request #777 from sreekanth-cb/persister_pause
pausing persister until merging catches up
2018-02-26 14:36:07 -05:00
Sreekanth Sivasankaran
e02849fcda
fix the indentation 2018-02-26 16:21:33 +05:30
Sreekanth Sivasankaran
c45822347f
Merge branch 'master' into mergeplanner_options 2018-02-26 15:59:20 +05:30
Sreekanth Sivasankaran
e4cc79a9ad adopting json parsing on options,
fixed the inadvertant option modification
2018-02-26 15:56:30 +05:30
Sreekanth Sivasankaran
f0a65f041d cleaning up the wait loop 2018-02-25 20:58:53 +05:30
Sreekanth Sivasankaran
3a571ad283
Merge branch 'master' into persister_pause 2018-02-24 23:57:20 +05:30
Sreekanth Sivasankaran
874829759b
cleaning up the wait loop 2018-02-24 23:53:49 +05:30
Sreekanth Sivasankaran
4109e327ff
Merge pull request #771 from sreekanth-cb/merge_handling_empty_seg_tasks
Fix for empty segment merge handling
2018-02-24 10:48:31 +05:30
Sreekanth Sivasankaran
683e195ac4 adding empty segment handling during introduction
cleaning up the segment live size check
2018-02-24 07:03:27 +05:30
Steve Yen
c50d9b4023 scorch conditional merging during persistSnapshot()
As part of this change, there are nw helper methods --
persistSnapshotMaybeMerge() and persistSnapshotDirect().
2018-02-23 09:17:02 -08:00
Sreekanth Sivasankaran
a1db057656 configurable mergePlanner options
mergePlanner options are parsed from the
scorch configs parameters
2018-02-23 16:09:37 +05:30
Sreekanth Sivasankaran
a8ebf2a553 lowering epochDistance to 5,
fixing the lastMergedEpoch value updates
2018-02-21 17:25:14 +05:30
Steve Yen
a0b7508da7 scorch zap mergeSegmentBases() func
As part of this, zap.MergeToWriter() now returns more information --
enough so that callers can now create their own SegmentBase instances.

Also, the fieldsMap maintained and returned by zap.MergeToWriter() is
now a mapping from fieldName ==> fieldID+1 (instead of the previous
mapping from fieldName ==> fieldID).  This makes it similar to how
fieldsMap are handled in other parts of zap to avoid "zero value"
issues.
2018-02-19 14:13:31 -08:00
Steve Yen
720010783e scorch zap InitSegmentBase() helper func
Refactored out a zap.InitSegmentBase() func so that non-zap packages
can create SegmentBase instances.
2018-02-19 14:13:31 -08:00
Steve Yen
656220ca9d
Merge pull request #769 from steveyen/scorch-rollback-ignores-unsafeBatch
scorch rollback ignores unsafeBatch flag
2018-02-15 18:51:59 -08:00
Sreekanth Sivasankaran
606a270669 Fix for empty segment merge handling
Avoid creating new files with emtpy segments tasks
during the merge operation, skips the
incorrect appending of a newer segment during merge.
2018-02-15 16:44:20 +05:30
Sreekanth Sivasankaran
35611f4287
Merge branch 'master' into persister_pause 2018-02-14 16:53:06 +05:30
Sreekanth Sivasankaran
6f2797bec3 Adding a pause to persister until the merger
catches up
2018-02-14 16:39:26 +05:30
Steve Yen
030469a351
Merge pull request #767 from steveyen/persistSnapshot-err-handling
improvements to err handling in persistSnapshot(), etc
2018-02-13 14:53:42 -08:00
Steve Yen
57fc03258e scorch rollback ignores unsafeBatch flag
See also: https://github.com/blevesearch/bleve/issues/760
2018-02-13 10:21:42 -08:00
Steve Yen
fe544f3352 scorch zap merge uses enumerator for vellum.Iterator's 2018-02-12 21:28:46 -08:00
Steve Yen
a073424e5a scorch zap dict.postingsListFromOffset() method
A helper method that can create a PostingsList if the caller already
knows the postingsOffset.
2018-02-12 20:54:07 -08:00
Steve Yen
2158e06c40 scorch zap merge collects dicts & itrs in lock-step
The theory with this change is that the dicts and itrs should be
positionally in "lock-step" with paired entries.

And, since later code also uses the same array indexing to access the
drops and newDocNums, those also need to be positionally in pair-wise
lock-step, too.
2018-02-12 20:54:07 -08:00
Steve Yen
95a4f37e5c scorch zap enumerator impl that joins multiple vellum iterators
Unlike vellum's MergeIterator, the enumerator introduced in this
commit doesn't merge when there are matching keys across iterators.

Instead, the enumerator implementation provides a traversal of all the
tuples of (key, iteratorIndex, val) from the underlying vellum
iterators, ordered by key ASC, iteratorIndex ASC.
2018-02-12 20:54:06 -08:00
Steve Yen
e37c563c56 scorch zap merge move fieldDvLocsOffset var declaration
Move the var declaration to nearer where its used.
2018-02-08 18:03:09 -08:00
Steve Yen
f177f07613 scorch zap segment merging reuses prealloc'ed PostingsIterator
During zap segment merging, a new zap PostingsIterator was allocated
for every field X segment X term.

This change optimizes by reusing a single PostingsIterator instance
per persistMergedRest() invocation.

And, also unused fields are removed from the PostingsIterator.
2018-02-08 17:24:30 -08:00
Steve Yen
6f5f90cd41 scorch zap segment cleanup handling for some edge cases
Two cases in this commit...

If we're shutting down, the merger might not have handed off its
latest merged segment to the introducer yet, so the merger still owns
the segment and needs to Close() that segment itself.

In persistSnapshot(), there migth be cases where the persister might
not be able to swap in its newly persisted segments -- so, the
persistSnapshot() needs to Close() those segments itself.
2018-02-08 14:04:04 -08:00
Steve Yen
83272a9629 scorch persistSnapshot() err handling & propagation 2018-02-08 14:03:59 -08:00
Steve Yen
dee6a2b1c6 scorch persistSnapshot() consistently uses err to commit vs abort
Some codepaths in persistSnapshot() were saving errors into an err2
local variable, which might lead incorrectly to commit during an error
situation rather than abort.
2018-02-08 14:02:35 -08:00
Steve Yen
91ac0d011a scorch uses segment.id to encode boltdb sub-bucket key
fixes #764
2018-02-08 13:25:16 -08:00
Steve Yen
8a7990427f
Merge pull request #765 from steveyen/more-TestIndexRollback-fixes
fix for TestIndexRollback unit tests
2018-02-08 12:45:28 -08:00
Steve Yen
d0644fec12 scorch persistSnapshot comments update
See also: https://github.com/blevesearch/bleve/issues/763
2018-02-08 12:22:58 -08:00