Steve Yen
655268bec8
scorch zap postings iterator nextDocNum() helper method
...
Refactored out a nextDocNum() helper method from Next() that future
optimizations can use.
2018-03-06 07:55:26 -08:00
Steve Yen
502e64c256
scorch zap Posting doesn't use iterator field
2018-03-05 16:33:13 -08:00
Steve Yen
8f8fd511b7
scorch zap access freqs[offset] outside loop
2018-03-05 12:02:33 -08:00
Steve Yen
a338386a03
scorch build optimize freq/loc slice capacity
2018-03-05 12:02:33 -08:00
Steve Yen
856778ad7b
scorch zap build prealloc docNumbers capacity
2018-03-05 12:02:33 -08:00
Steve Yen
8c0881eab2
scorch zap build reuses mem postingsList/Iterator structs
2018-03-05 12:02:33 -08:00
Steve Yen
85761c6a57
go fmt
2018-03-05 12:02:33 -08:00
Steve Yen
d44c5ad568
scorch stats MaxBatchIntroTime bug fix and more timing stats
...
Added timing stats for in-mem zap merging and file-based zap merging.
2018-03-05 12:02:33 -08:00
Steve Yen
884da6f93a
scorch optimize mem processDocument() norm calculation
...
This change moves the norm calculation outside of the inner loop.
2018-03-03 11:58:30 -08:00
Steve Yen
6ae799052a
scorch mem optimize processDocument() stored field
2018-03-03 11:52:33 -08:00
Steve Yen
b7cfef81c9
scorch optimize mem processDocument() dict access
...
This change moves the dict lookup to outside of the loop.
2018-03-03 11:43:25 -08:00
Steve Yen
88c740095b
scorch optimizations for mem.PostingsIterator.Next() & docTermMap
...
Due to the usage rules of iterators, mem.PostingsIterator.Next() can
reuse its returned Postings instance.
Also, there's a micro optimization in persistDocValues() for one fewer
access to the docTermMap in the inner-loop.
2018-03-03 11:31:18 -08:00
Steve Yen
a5253bfe2b
scorch persister goes through introducer to affect root
...
This change allows the introducer to become the only goroutine to
modify the root, which in turn allows the introducer to greatly reduce
its root lock holding surface area.
2018-03-02 16:14:28 -08:00
Marty Schoch
30acc55d05
remove unnecessary scorch reader wrapper
...
we now use *IndexSnapshot directly
2018-03-02 14:03:54 -08:00
Steve Yen
d61d9e4cf6
scorch stats MaxBatchIntroTime and TotBatchIntroTime
2018-03-02 13:33:06 -08:00
Steve Yen
868a66279e
scorch indexing time stat
...
Looks like this was forgotten along the way -- the stat for analysis
time was tracked correctly, but indexing time wasn't.
2018-03-02 11:07:39 -08:00
Steve Yen
7e5bb0bd8d
renamed to CurOnDiskBytes/Files as those are gauges
2018-03-01 14:13:43 -08:00
Marty Schoch
0363b24dd4
update to use new vellum Reset API
2018-03-01 09:37:39 -08:00
Steve Yen
39f9cee910
Merge pull request #789 from steveyen/sreekanth-cb-scorch_stats
...
adding stats for scorch, with no gauges
2018-02-28 17:41:10 -08:00
Steve Yen
1b661ef844
stats cleanup, renaming, gauges replaced with counters
2018-02-28 17:03:28 -08:00
Steve Yen
7d46d2c7ae
scorch zap intcoder encoder is never nil
2018-02-28 10:09:21 -08:00
Sreekanth Sivasankaran
4b742505aa
adding stats for scorch
2018-02-28 15:31:55 +05:30
Steve Yen
dd7d93ee5e
scorch zap loadChunk reuses Location slices
2018-02-27 18:01:48 -08:00
Steve Yen
4dbb4b1495
scorch zap posting reuses freqNorm & loc reader and decoder
2018-02-27 18:01:48 -08:00
Steve Yen
a32362ba2e
MB-28403: scorch introduceMerge doesn't prealloc segments capacity
...
There's now multiple competing merge activities (file-merging and
in-memory merging during persistence), so the simple math to
precalculate capacity for the slice of segments in introduceMerge() no
longer works for all cases and might have negative capacity.
This change removes that (sometimes wrong) precalculation, and instead
depends on append() to grow the slice correctly.
2018-02-27 15:14:34 -08:00
Steve Yen
3f1dcb6078
scorch zap merge optimize drops lookup to outside of loop
2018-02-27 09:23:29 -08:00
Steve Yen
99ed127176
scorch zap merge optimize newDocNums lookup to outside of loop
...
And, also a "go fmt".
2018-02-26 14:23:55 -08:00
Steve Yen
98d5d7bd81
scorch zap chunkedIntCoder optimizations
...
The optimizations / changes include...
- reuse of a memory buf when serializing varint's.
- reuse of a govarint.U64Base128Encoder instance, as it's a thin,
wrapper around an underlying chunkBuf, so Reset()'s on the
chunkBuf is enough for encoder reuse.
- chunkedIntcoder.Write() method was changed to invoke w.Write() less
often by forming a larger, reused buf. Profiling and analysis
showed w.Write() was getting called a lot, often with tiny 1 or 2
byte inputs. The theory is w.Write() and its underlying memmove()
can be more efficient when provided with larger bufs.
- some repeated code removal, by reusing the Close() method.
2018-02-26 14:17:09 -08:00
Steve Yen
ce2332e111
scorch zap merge reuses tf/locEncoder across terms
...
The finishTerm() helper func that's invoked on every outer loop resets
the tf/locEncoders so they can be safely reused.
2018-02-26 11:37:11 -08:00
Marty Schoch
eca31dfd27
Merge pull request #777 from sreekanth-cb/persister_pause
...
pausing persister until merging catches up
2018-02-26 14:36:07 -05:00
Sreekanth Sivasankaran
e02849fcda
fix the indentation
2018-02-26 16:21:33 +05:30
Sreekanth Sivasankaran
c45822347f
Merge branch 'master' into mergeplanner_options
2018-02-26 15:59:20 +05:30
Sreekanth Sivasankaran
e4cc79a9ad
adopting json parsing on options,
...
fixed the inadvertant option modification
2018-02-26 15:56:30 +05:30
Sreekanth Sivasankaran
f0a65f041d
cleaning up the wait loop
2018-02-25 20:58:53 +05:30
Sreekanth Sivasankaran
3a571ad283
Merge branch 'master' into persister_pause
2018-02-24 23:57:20 +05:30
Sreekanth Sivasankaran
874829759b
cleaning up the wait loop
2018-02-24 23:53:49 +05:30
Sreekanth Sivasankaran
4109e327ff
Merge pull request #771 from sreekanth-cb/merge_handling_empty_seg_tasks
...
Fix for empty segment merge handling
2018-02-24 10:48:31 +05:30
Sreekanth Sivasankaran
683e195ac4
adding empty segment handling during introduction
...
cleaning up the segment live size check
2018-02-24 07:03:27 +05:30
abhinavdangeti
da70758635
Handle case where store snapshot isn't closed in upsidedown's Batch() API
2018-02-23 14:47:22 -08:00
Steve Yen
c50d9b4023
scorch conditional merging during persistSnapshot()
...
As part of this change, there are nw helper methods --
persistSnapshotMaybeMerge() and persistSnapshotDirect().
2018-02-23 09:17:02 -08:00
Sreekanth Sivasankaran
a1db057656
configurable mergePlanner options
...
mergePlanner options are parsed from the
scorch configs parameters
2018-02-23 16:09:37 +05:30
Sreekanth Sivasankaran
a8ebf2a553
lowering epochDistance to 5,
...
fixing the lastMergedEpoch value updates
2018-02-21 17:25:14 +05:30
Steve Yen
a0b7508da7
scorch zap mergeSegmentBases() func
...
As part of this, zap.MergeToWriter() now returns more information --
enough so that callers can now create their own SegmentBase instances.
Also, the fieldsMap maintained and returned by zap.MergeToWriter() is
now a mapping from fieldName ==> fieldID+1 (instead of the previous
mapping from fieldName ==> fieldID). This makes it similar to how
fieldsMap are handled in other parts of zap to avoid "zero value"
issues.
2018-02-19 14:13:31 -08:00
Steve Yen
720010783e
scorch zap InitSegmentBase() helper func
...
Refactored out a zap.InitSegmentBase() func so that non-zap packages
can create SegmentBase instances.
2018-02-19 14:13:31 -08:00
Steve Yen
656220ca9d
Merge pull request #769 from steveyen/scorch-rollback-ignores-unsafeBatch
...
scorch rollback ignores unsafeBatch flag
2018-02-15 18:51:59 -08:00
Sreekanth Sivasankaran
606a270669
Fix for empty segment merge handling
...
Avoid creating new files with emtpy segments tasks
during the merge operation, skips the
incorrect appending of a newer segment during merge.
2018-02-15 16:44:20 +05:30
Sreekanth Sivasankaran
35611f4287
Merge branch 'master' into persister_pause
2018-02-14 16:53:06 +05:30
Sreekanth Sivasankaran
6f2797bec3
Adding a pause to persister until the merger
...
catches up
2018-02-14 16:39:26 +05:30
Steve Yen
030469a351
Merge pull request #767 from steveyen/persistSnapshot-err-handling
...
improvements to err handling in persistSnapshot(), etc
2018-02-13 14:53:42 -08:00
Steve Yen
57fc03258e
scorch rollback ignores unsafeBatch flag
...
See also: https://github.com/blevesearch/bleve/issues/760
2018-02-13 10:21:42 -08:00