Sreekanth Sivasankaran
fa5de8e09a
making NumSnapshotsToKeep configurable
2018-03-06 16:22:11 +05:30
Steve Yen
4ebf3f1d44
Merge pull request #794 from steveyen/persister-uses-introducer
...
scorch persister goes through introducer to affect root
2018-03-02 16:23:17 -08:00
Steve Yen
a5253bfe2b
scorch persister goes through introducer to affect root
...
This change allows the introducer to become the only goroutine to
modify the root, which in turn allows the introducer to greatly reduce
its root lock holding surface area.
2018-03-02 16:14:28 -08:00
Marty Schoch
33641ef9d3
Merge pull request #793 from mschoch/remove-reader
...
remove unnecessary scorch reader wrapper
2018-03-02 14:13:31 -08:00
Marty Schoch
30acc55d05
remove unnecessary scorch reader wrapper
...
we now use *IndexSnapshot directly
2018-03-02 14:03:54 -08:00
Steve Yen
fd6bfb0113
Merge pull request #792 from steveyen/more-stats
...
adding more scorch related stats
2018-03-02 13:41:06 -08:00
Steve Yen
d61d9e4cf6
scorch stats MaxBatchIntroTime and TotBatchIntroTime
2018-03-02 13:33:06 -08:00
Steve Yen
868a66279e
scorch indexing time stat
...
Looks like this was forgotten along the way -- the stat for analysis
time was tracked correctly, but indexing time wasn't.
2018-03-02 11:07:39 -08:00
Steve Yen
3fe7e2e4f4
Merge pull request #791 from steveyen/scorch-stats-gauge
...
renamed to CurOnDiskBytes/Files as those are gauges
2018-03-01 22:47:21 -08:00
Steve Yen
7e5bb0bd8d
renamed to CurOnDiskBytes/Files as those are gauges
2018-03-01 14:13:43 -08:00
Marty Schoch
bbfda08cf7
Merge pull request #790 from mschoch/use-vellum-reset
...
update to use new vellum Reset API
2018-03-01 10:06:38 -08:00
Marty Schoch
0363b24dd4
update to use new vellum Reset API
2018-03-01 09:37:39 -08:00
Steve Yen
39f9cee910
Merge pull request #789 from steveyen/sreekanth-cb-scorch_stats
...
adding stats for scorch, with no gauges
2018-02-28 17:41:10 -08:00
Steve Yen
1e6243e21c
Merge pull request #788 from steveyen/intcoder-encoder-never-nil
...
scorch zap intcoder encoder is never nil
2018-02-28 17:32:25 -08:00
Steve Yen
1b661ef844
stats cleanup, renaming, gauges replaced with counters
2018-02-28 17:03:28 -08:00
Steve Yen
7d46d2c7ae
scorch zap intcoder encoder is never nil
2018-02-28 10:09:21 -08:00
Sreekanth Sivasankaran
4b742505aa
adding stats for scorch
2018-02-28 15:31:55 +05:30
Steve Yen
56c2acd990
Merge pull request #775 from steveyen/posting-reuse-reader
...
Postings codepaths reuse readers and location slices
2018-02-27 19:26:20 -08:00
Steve Yen
dd7d93ee5e
scorch zap loadChunk reuses Location slices
2018-02-27 18:01:48 -08:00
Steve Yen
4dbb4b1495
scorch zap posting reuses freqNorm & loc reader and decoder
2018-02-27 18:01:48 -08:00
Steve Yen
1733d4ee5e
Merge pull request #786 from steveyen/MB-28403
...
MB-28403: scorch introduceMerge doesn't prealloc segments capacity
2018-02-27 15:26:27 -08:00
Steve Yen
a32362ba2e
MB-28403: scorch introduceMerge doesn't prealloc segments capacity
...
There's now multiple competing merge activities (file-merging and
in-memory merging during persistence), so the simple math to
precalculate capacity for the slice of segments in introduceMerge() no
longer works for all cases and might have negative capacity.
This change removes that (sometimes wrong) precalculation, and instead
depends on append() to grow the slice correctly.
2018-02-27 15:14:34 -08:00
Marty Schoch
b5ce0b046f
Merge pull request #785 from mschoch/use-new-context-pkg
...
BREAKING API CHANGE - use stdlib context pkg
2018-02-27 14:17:08 -08:00
Marty Schoch
8063132766
fix new issues found by go vet when using stdlib context pkg
2018-02-27 11:57:21 -08:00
Marty Schoch
c74e08f039
BREAKING API CHANGE - use stdlib context pkg
...
update all references to context to use std lib pkg
2018-02-27 11:33:43 -08:00
Marty Schoch
f58a205ae8
remove 1.6 from travis, add "1.10"
2018-02-27 11:29:16 -08:00
Steve Yen
1a319cdf5b
Merge pull request #784 from steveyen/drops-loop-optimization
...
scorch zap merge optimize drops lookup to outside of loop
2018-02-27 09:52:56 -08:00
Steve Yen
3f1dcb6078
scorch zap merge optimize drops lookup to outside of loop
2018-02-27 09:23:29 -08:00
Marty Schoch
b8bb7922eb
Merge pull request #782 from steveyen/scorch-intcoder-optimizations
...
Various scorch optimizations around merge & chunkedIntCoder
2018-02-26 17:57:00 -05:00
Steve Yen
99ed127176
scorch zap merge optimize newDocNums lookup to outside of loop
...
And, also a "go fmt".
2018-02-26 14:23:55 -08:00
Steve Yen
98d5d7bd81
scorch zap chunkedIntCoder optimizations
...
The optimizations / changes include...
- reuse of a memory buf when serializing varint's.
- reuse of a govarint.U64Base128Encoder instance, as it's a thin,
wrapper around an underlying chunkBuf, so Reset()'s on the
chunkBuf is enough for encoder reuse.
- chunkedIntcoder.Write() method was changed to invoke w.Write() less
often by forming a larger, reused buf. Profiling and analysis
showed w.Write() was getting called a lot, often with tiny 1 or 2
byte inputs. The theory is w.Write() and its underlying memmove()
can be more efficient when provided with larger bufs.
- some repeated code removal, by reusing the Close() method.
2018-02-26 14:17:09 -08:00
Steve Yen
ce2332e111
scorch zap merge reuses tf/locEncoder across terms
...
The finishTerm() helper func that's invoked on every outer loop resets
the tf/locEncoders so they can be safely reused.
2018-02-26 11:37:11 -08:00
Marty Schoch
eca31dfd27
Merge pull request #777 from sreekanth-cb/persister_pause
...
pausing persister until merging catches up
2018-02-26 14:36:07 -05:00
Steve Yen
002df80357
Merge pull request #780 from sreekanth-cb/mergeplanner_options
...
configurable mergePlanner options
2018-02-26 11:33:31 -08:00
Sreekanth Sivasankaran
e02849fcda
fix the indentation
2018-02-26 16:21:33 +05:30
Sreekanth Sivasankaran
c45822347f
Merge branch 'master' into mergeplanner_options
2018-02-26 15:59:20 +05:30
Sreekanth Sivasankaran
e4cc79a9ad
adopting json parsing on options,
...
fixed the inadvertant option modification
2018-02-26 15:56:30 +05:30
Sreekanth Sivasankaran
f0a65f041d
cleaning up the wait loop
2018-02-25 20:58:53 +05:30
Sreekanth Sivasankaran
3a571ad283
Merge branch 'master' into persister_pause
2018-02-24 23:57:20 +05:30
Sreekanth Sivasankaran
874829759b
cleaning up the wait loop
2018-02-24 23:53:49 +05:30
Sreekanth Sivasankaran
4109e327ff
Merge pull request #771 from sreekanth-cb/merge_handling_empty_seg_tasks
...
Fix for empty segment merge handling
2018-02-24 10:48:31 +05:30
Sreekanth Sivasankaran
683e195ac4
adding empty segment handling during introduction
...
cleaning up the segment live size check
2018-02-24 07:03:27 +05:30
Abhinav Dangeti
1929ceb1f5
Merge pull request #781 from abhinavdangeti/upsidedown-missing-close
...
Handle case where store snapshot isn't closed in upsidedown's Batch() API
2018-02-23 15:02:44 -08:00
abhinavdangeti
da70758635
Handle case where store snapshot isn't closed in upsidedown's Batch() API
2018-02-23 14:47:22 -08:00
Steve Yen
19080c1ae5
Merge pull request #779 from steveyen/wip-in-mem-seg-merging
...
merging of in-memory segments during persistSnapshot
2018-02-23 14:02:02 -08:00
Steve Yen
c50d9b4023
scorch conditional merging during persistSnapshot()
...
As part of this change, there are nw helper methods --
persistSnapshotMaybeMerge() and persistSnapshotDirect().
2018-02-23 09:17:02 -08:00
Sreekanth Sivasankaran
a1db057656
configurable mergePlanner options
...
mergePlanner options are parsed from the
scorch configs parameters
2018-02-23 16:09:37 +05:30
Sreekanth Sivasankaran
a8ebf2a553
lowering epochDistance to 5,
...
fixing the lastMergedEpoch value updates
2018-02-21 17:25:14 +05:30
Steve Yen
a0b7508da7
scorch zap mergeSegmentBases() func
...
As part of this, zap.MergeToWriter() now returns more information --
enough so that callers can now create their own SegmentBase instances.
Also, the fieldsMap maintained and returned by zap.MergeToWriter() is
now a mapping from fieldName ==> fieldID+1 (instead of the previous
mapping from fieldName ==> fieldID). This makes it similar to how
fieldsMap are handled in other parts of zap to avoid "zero value"
issues.
2018-02-19 14:13:31 -08:00
Steve Yen
720010783e
scorch zap InitSegmentBase() helper func
...
Refactored out a zap.InitSegmentBase() func so that non-zap packages
can create SegmentBase instances.
2018-02-19 14:13:31 -08:00