Marty Schoch
23a323bc9d
add support for numPlainTextBytesIndexed metric
2016-03-05 14:05:08 -05:00
Marty Schoch
81780f97d0
add term search stats
2016-03-05 07:50:25 -05:00
Marty Schoch
147debaa12
expose metrics and moss stats wrapping underlying stats as well
2016-03-04 13:43:39 -05:00
Steve Yen
f6d1bd2c87
moss option MaxPreMergerBatches renamed
2016-03-03 11:18:30 -08:00
Steve Yen
7d67d89a9c
MB-18441 - moss lower-level iterator starts positioned on current
...
The iterator starts off positioned so that Current() is correct, so
invoking Next() right off the bat was incorrect.
2016-03-01 21:45:48 -08:00
Steve Yen
a29dd25a48
upside_down dict row value size accounts for large uvarint's
...
This is somewhat unlikely, but if a term is (incredibly) popular, its
uvarint count value representation might go beyond 8 bytes.
Some KVStore implementations (like forestdb) provide a BatchEx cgo
optimization that depends on proper preallocated counting, so this
change provides a proper worst-case estimate based on the max-unvarint
of 10 bytes instead of the previously incorrect 8 bytes.
2016-02-22 11:52:51 -08:00
Steve Yen
dd1718fa78
index/store/moss uses AllocMerge() instead of Merge()
...
Performance optimization. Before this change, by using Merge()
instead of AllocMerge(), moss's internal batch buf's would be
wastefully, dramatically grown during append()'s to a mis-sized buf.
2016-02-22 11:48:02 -08:00
Steve Yen
ea1a52464d
more index/store/moss err handling
2016-02-20 14:25:42 -08:00
Steve Yen
eb315fa500
integrate index/store/moss KV store
2016-02-20 14:25:42 -08:00
Marty Schoch
208b700e17
add missing build tag guarding cznicb benchmark
2016-02-09 15:57:35 -05:00
Marty Schoch
40c95513b7
add support for including kvstore stats
2016-02-05 12:26:19 -05:00
Marty Schoch
c5dea9e882
fix accessing store via Advanced() method which was broken
2016-02-02 11:54:18 -05:00
Marty Schoch
710d06e974
add support for native C merge operators
2016-01-27 17:51:07 -05:00
Steve Yen
d97e3caf4f
fix comment typo
2016-01-22 09:04:24 -08:00
Steve Yen
d5de1d3da1
metrics implements BatchEx correctly
2016-01-21 11:00:41 -08:00
Marty Schoch
fc34a97875
copy locations on merge for more safe/predictable behavior
...
fixes #328
2016-01-19 14:21:48 -05:00
Steve Yen
035d9d0e40
unneeded cast and parens
2016-01-17 00:16:05 -08:00
Marty Schoch
1335eb2a7b
Merge pull request #322 from steveyen/WIP-perf-20160113
...
KVReader.MultiGet and KVWriter.NewBatchEx API's
2016-01-15 14:28:59 -05:00
opennota
8517feb1c6
Fix some typos
2016-01-15 05:46:27 +07:00
Silvan Jegen
d326898f7b
Remove unneeded brackets
2016-01-14 16:41:41 +01:00
Steve Yen
6849e538be
upside_down and firestorm use new NewBatchEx() API
...
With this change, the upside_down batchRows() and firestorm
batchRows() now use the new KVWriter.NewBatchEx() API, which can
improve performance by reducing the number of cgo hops.
2016-01-13 23:08:20 -08:00
Steve Yen
d94ccf2d74
added KVWriter.NewBatchEx() method
2016-01-13 16:19:04 -08:00
Steve Yen
fb048f6c64
added KVReader.MultiGet() method
2016-01-13 15:12:10 -08:00
Steve Yen
8dc067b1d9
go fmt
2016-01-13 15:11:50 -08:00
Steve Yen
fe39b3fd13
avoid fieldTermFreqs loop if no composite fields
2016-01-13 14:45:04 -08:00
Marty Schoch
af25e724f6
Merge branch 'master' of https://github.com/slavikm/bleve into slavikm-master
2016-01-13 16:10:59 -05:00
Silvan Jegen
35ac2b2bee
Run go fmt ./...
2016-01-12 22:15:50 +01:00
Steve Yen
0e72b949b3
upside_down batchRows() takes array of arrays
...
In order to spend less time in append(), this change in upside_down
(similar to another recent performance change in firestorm) builds up
an array of arrays as the eventual input to batchRows().
2016-01-11 18:11:21 -08:00
slavikm
680be52f87
Implemented boolean field support
2016-01-11 17:18:03 -08:00
Steve Yen
7ce7d98cba
upside_down merge dictionary deltas before using batch.Merge()
...
This change performs more dictionary delta incr/decr math in
batchRows() instead of in the KVStore ExecuteBatch() machinery.
2016-01-11 16:52:07 -08:00
Steve Yen
94273d5fa9
upside_down process internal rows earlier
...
With this change, internal rows are processed while we're waiting for
backIndex rows to be retrieved.
2016-01-11 16:25:35 -08:00
Steve Yen
bb5cd8f3d6
upside_down merge backIndexRow concurrently
...
Previously, the code would gather all the backIndexRows before
processing them. This change instead merges the backIndexRows
concurrently on the theory that we might as well make progress on
compute & processing tasks while waiting for the rest of the back
index rows to be fetched from the KVStore.
2016-01-10 18:50:42 -08:00
Steve Yen
c3b5246b0c
upside_down track analysis time tighter; and comments
2016-01-10 15:36:54 -08:00
Steve Yen
d3dd40d334
upside_down retrieves backindex concurrently with analysis
...
Start backindex reading concurrently with analysi to try to utilize
more I/O bandwidth.
The analysis time vs indexing time stats tracking are also now "off",
since there's now concurrency between those actiivties.
One tradeoff is that the lock area in upside_down Batch() is increased
as part of this change.
2016-01-10 15:18:28 -08:00
Steve Yen
bff95eef70
firestorm close kvwriter sooner
2016-01-10 15:18:27 -08:00
Steve Yen
860de28a28
fix memory leak by closing batches in batchRows()
2016-01-07 17:59:42 -08:00
Steve Yen
70105477cf
added Close() method to KVBatch interface
2016-01-07 17:54:21 -08:00
Marty Schoch
48fcd5a7d5
Merge branch 'WIP-perf-20160106' of https://github.com/steveyen/bleve into steveyen-WIP-perf-20160106
2016-01-07 15:40:29 -05:00
Marty Schoch
665f5c58e1
fix errcheck violation
2016-01-07 11:11:43 -05:00
Marty Schoch
e54db33346
try testing slightly different way
2016-01-07 11:06:18 -05:00
Marty Schoch
cd940cc375
add another check to try to understand test failure on travis
2016-01-07 10:45:20 -05:00
Steve Yen
846912d083
upside_down udc.termVectorsFromTokenFreq rows append optimization
2016-01-07 00:48:34 -08:00
Steve Yen
8b980bd2ef
firestorm avoid extra goroutine, similar to upside_down
2016-01-07 00:43:27 -08:00
Steve Yen
fbd0e7bfe9
upside_down backIndexTermEntries precalloc'ed capacity
2016-01-07 00:23:25 -08:00
Steve Yen
4eee8821f9
upside_down storeField/indexField append to provided arrays
...
Taking another optimization from firestorm, upside_down's
storeField()/indexField() funcs now also append() to passed-in arrays
rather than always allocating their own arrays.
2016-01-07 00:13:46 -08:00
Steve Yen
1af2927967
upside_down gets analysis perf rows optimizations from firestorm
2016-01-06 23:53:13 -08:00
Steve Yen
82b8b3468e
upside_down analysis converts to docIDBytes once
2016-01-06 23:38:02 -08:00
Steve Yen
d6a997d8c1
firestorm gtreap lookup once per snapshot docID
...
Previously, firestorm would lookup docID's in the inFlight gtreap for
every candidate docNum, and this change moves the lookup to outside of
the loop.
2016-01-06 16:46:15 -08:00
Steve Yen
024848ac91
firestorm valid docNum finding, fixes #310
2016-01-06 16:04:56 -08:00
Steve Yen
7df07f94fa
firestorm use the ParseKey() funcs to avoid unneeded value parsing
...
With this change, the row allocation also happens only once per loop,
instead of once per item.
2016-01-06 15:53:12 -08:00