4ebf3f1d44
Merge pull request #794 from steveyen/persister-uses-introducer
...
scorch persister goes through introducer to affect root
2018-03-02 16:23:17 -08:00
a5253bfe2b
scorch persister goes through introducer to affect root
...
This change allows the introducer to become the only goroutine to
modify the root, which in turn allows the introducer to greatly reduce
its root lock holding surface area.
2018-03-02 16:14:28 -08:00
33641ef9d3
Merge pull request #793 from mschoch/remove-reader
...
remove unnecessary scorch reader wrapper
2018-03-02 14:13:31 -08:00
30acc55d05
remove unnecessary scorch reader wrapper
...
we now use *IndexSnapshot directly
2018-03-02 14:03:54 -08:00
fd6bfb0113
Merge pull request #792 from steveyen/more-stats
...
adding more scorch related stats
2018-03-02 13:41:06 -08:00
d61d9e4cf6
scorch stats MaxBatchIntroTime and TotBatchIntroTime
2018-03-02 13:33:06 -08:00
868a66279e
scorch indexing time stat
...
Looks like this was forgotten along the way -- the stat for analysis
time was tracked correctly, but indexing time wasn't.
2018-03-02 11:07:39 -08:00
3fe7e2e4f4
Merge pull request #791 from steveyen/scorch-stats-gauge
...
renamed to CurOnDiskBytes/Files as those are gauges
2018-03-01 22:47:21 -08:00
7e5bb0bd8d
renamed to CurOnDiskBytes/Files as those are gauges
2018-03-01 14:13:43 -08:00
bbfda08cf7
Merge pull request #790 from mschoch/use-vellum-reset
...
update to use new vellum Reset API
2018-03-01 10:06:38 -08:00
0363b24dd4
update to use new vellum Reset API
2018-03-01 09:37:39 -08:00
39f9cee910
Merge pull request #789 from steveyen/sreekanth-cb-scorch_stats
...
adding stats for scorch, with no gauges
2018-02-28 17:41:10 -08:00
1e6243e21c
Merge pull request #788 from steveyen/intcoder-encoder-never-nil
...
scorch zap intcoder encoder is never nil
2018-02-28 17:32:25 -08:00
1b661ef844
stats cleanup, renaming, gauges replaced with counters
2018-02-28 17:03:28 -08:00
7d46d2c7ae
scorch zap intcoder encoder is never nil
2018-02-28 10:09:21 -08:00
4b742505aa
adding stats for scorch
2018-02-28 15:31:55 +05:30
56c2acd990
Merge pull request #775 from steveyen/posting-reuse-reader
...
Postings codepaths reuse readers and location slices
2018-02-27 19:26:20 -08:00
dd7d93ee5e
scorch zap loadChunk reuses Location slices
2018-02-27 18:01:48 -08:00
4dbb4b1495
scorch zap posting reuses freqNorm & loc reader and decoder
2018-02-27 18:01:48 -08:00
1733d4ee5e
Merge pull request #786 from steveyen/MB-28403
...
MB-28403: scorch introduceMerge doesn't prealloc segments capacity
2018-02-27 15:26:27 -08:00
a32362ba2e
MB-28403: scorch introduceMerge doesn't prealloc segments capacity
...
There's now multiple competing merge activities (file-merging and
in-memory merging during persistence), so the simple math to
precalculate capacity for the slice of segments in introduceMerge() no
longer works for all cases and might have negative capacity.
This change removes that (sometimes wrong) precalculation, and instead
depends on append() to grow the slice correctly.
2018-02-27 15:14:34 -08:00
b5ce0b046f
Merge pull request #785 from mschoch/use-new-context-pkg
...
BREAKING API CHANGE - use stdlib context pkg
2018-02-27 14:17:08 -08:00
8063132766
fix new issues found by go vet when using stdlib context pkg
2018-02-27 11:57:21 -08:00
c74e08f039
BREAKING API CHANGE - use stdlib context pkg
...
update all references to context to use std lib pkg
2018-02-27 11:33:43 -08:00
f58a205ae8
remove 1.6 from travis, add "1.10"
2018-02-27 11:29:16 -08:00
1a319cdf5b
Merge pull request #784 from steveyen/drops-loop-optimization
...
scorch zap merge optimize drops lookup to outside of loop
2018-02-27 09:52:56 -08:00
3f1dcb6078
scorch zap merge optimize drops lookup to outside of loop
2018-02-27 09:23:29 -08:00
b8bb7922eb
Merge pull request #782 from steveyen/scorch-intcoder-optimizations
...
Various scorch optimizations around merge & chunkedIntCoder
2018-02-26 17:57:00 -05:00
99ed127176
scorch zap merge optimize newDocNums lookup to outside of loop
...
And, also a "go fmt".
2018-02-26 14:23:55 -08:00
98d5d7bd81
scorch zap chunkedIntCoder optimizations
...
The optimizations / changes include...
- reuse of a memory buf when serializing varint's.
- reuse of a govarint.U64Base128Encoder instance, as it's a thin,
wrapper around an underlying chunkBuf, so Reset()'s on the
chunkBuf is enough for encoder reuse.
- chunkedIntcoder.Write() method was changed to invoke w.Write() less
often by forming a larger, reused buf. Profiling and analysis
showed w.Write() was getting called a lot, often with tiny 1 or 2
byte inputs. The theory is w.Write() and its underlying memmove()
can be more efficient when provided with larger bufs.
- some repeated code removal, by reusing the Close() method.
2018-02-26 14:17:09 -08:00
ce2332e111
scorch zap merge reuses tf/locEncoder across terms
...
The finishTerm() helper func that's invoked on every outer loop resets
the tf/locEncoders so they can be safely reused.
2018-02-26 11:37:11 -08:00
eca31dfd27
Merge pull request #777 from sreekanth-cb/persister_pause
...
pausing persister until merging catches up
2018-02-26 14:36:07 -05:00
002df80357
Merge pull request #780 from sreekanth-cb/mergeplanner_options
...
configurable mergePlanner options
2018-02-26 11:33:31 -08:00
e02849fcda
fix the indentation
2018-02-26 16:21:33 +05:30
c45822347f
Merge branch 'master' into mergeplanner_options
2018-02-26 15:59:20 +05:30
e4cc79a9ad
adopting json parsing on options,
...
fixed the inadvertant option modification
2018-02-26 15:56:30 +05:30
f0a65f041d
cleaning up the wait loop
2018-02-25 20:58:53 +05:30
3a571ad283
Merge branch 'master' into persister_pause
2018-02-24 23:57:20 +05:30
874829759b
cleaning up the wait loop
2018-02-24 23:53:49 +05:30
4109e327ff
Merge pull request #771 from sreekanth-cb/merge_handling_empty_seg_tasks
...
Fix for empty segment merge handling
2018-02-24 10:48:31 +05:30
683e195ac4
adding empty segment handling during introduction
...
cleaning up the segment live size check
2018-02-24 07:03:27 +05:30
1929ceb1f5
Merge pull request #781 from abhinavdangeti/upsidedown-missing-close
...
Handle case where store snapshot isn't closed in upsidedown's Batch() API
2018-02-23 15:02:44 -08:00
da70758635
Handle case where store snapshot isn't closed in upsidedown's Batch() API
2018-02-23 14:47:22 -08:00
19080c1ae5
Merge pull request #779 from steveyen/wip-in-mem-seg-merging
...
merging of in-memory segments during persistSnapshot
2018-02-23 14:02:02 -08:00
c50d9b4023
scorch conditional merging during persistSnapshot()
...
As part of this change, there are nw helper methods --
persistSnapshotMaybeMerge() and persistSnapshotDirect().
2018-02-23 09:17:02 -08:00
a1db057656
configurable mergePlanner options
...
mergePlanner options are parsed from the
scorch configs parameters
2018-02-23 16:09:37 +05:30
a8ebf2a553
lowering epochDistance to 5,
...
fixing the lastMergedEpoch value updates
2018-02-21 17:25:14 +05:30
a0b7508da7
scorch zap mergeSegmentBases() func
...
As part of this, zap.MergeToWriter() now returns more information --
enough so that callers can now create their own SegmentBase instances.
Also, the fieldsMap maintained and returned by zap.MergeToWriter() is
now a mapping from fieldName ==> fieldID+1 (instead of the previous
mapping from fieldName ==> fieldID). This makes it similar to how
fieldsMap are handled in other parts of zap to avoid "zero value"
issues.
2018-02-19 14:13:31 -08:00
720010783e
scorch zap InitSegmentBase() helper func
...
Refactored out a zap.InitSegmentBase() func so that non-zap packages
can create SegmentBase instances.
2018-02-19 14:13:31 -08:00
656220ca9d
Merge pull request #769 from steveyen/scorch-rollback-ignores-unsafeBatch
...
scorch rollback ignores unsafeBatch flag
2018-02-15 18:51:59 -08:00
Steve Yen