Steve Yen
6849e538be
upside_down and firestorm use new NewBatchEx() API
...
With this change, the upside_down batchRows() and firestorm
batchRows() now use the new KVWriter.NewBatchEx() API, which can
improve performance by reducing the number of cgo hops.
2016-01-13 23:08:20 -08:00
Steve Yen
d94ccf2d74
added KVWriter.NewBatchEx() method
2016-01-13 16:19:04 -08:00
Steve Yen
fb048f6c64
added KVReader.MultiGet() method
2016-01-13 15:12:10 -08:00
Steve Yen
8dc067b1d9
go fmt
2016-01-13 15:11:50 -08:00
Steve Yen
fe39b3fd13
avoid fieldTermFreqs loop if no composite fields
2016-01-13 14:45:04 -08:00
Marty Schoch
b286466787
Merge branch 'slavikm-master'
2016-01-13 17:13:28 -05:00
Marty Schoch
737a07ddcf
update dynamic boolean fields to honor Index/Store Dynamic property
2016-01-13 17:12:40 -05:00
Marty Schoch
2479ddef2e
fixed errcheck issues
2016-01-13 17:10:13 -05:00
Marty Schoch
af25e724f6
Merge branch 'master' of https://github.com/slavikm/bleve into slavikm-master
2016-01-13 16:10:59 -05:00
Marty Schoch
81f8509890
add the ability to not store dynamic fields globally
2016-01-12 17:27:11 -05:00
Marty Schoch
b7c03dae1a
boolean query defaults to minShould of 0
...
fixes #258
2016-01-12 16:30:10 -05:00
Marty Schoch
847360f1d0
Merge pull request #320 from Shugyousha/gofmt
...
Run go fmt ./...
2016-01-12 16:24:03 -05:00
Silvan Jegen
35ac2b2bee
Run go fmt ./...
2016-01-12 22:15:50 +01:00
slavikm
c4f9828b78
fixed typo in comment
2016-01-12 11:31:53 -08:00
Marty Schoch
c3ddf038ab
Merge pull request #316 from steveyen/WIP-perf-20160110
...
upside-down backindex read concurrency
2016-01-12 12:53:37 -05:00
Steve Yen
0e72b949b3
upside_down batchRows() takes array of arrays
...
In order to spend less time in append(), this change in upside_down
(similar to another recent performance change in firestorm) builds up
an array of arrays as the eventual input to batchRows().
2016-01-11 18:11:21 -08:00
slavikm
680be52f87
Implemented boolean field support
2016-01-11 17:18:03 -08:00
Steve Yen
7ce7d98cba
upside_down merge dictionary deltas before using batch.Merge()
...
This change performs more dictionary delta incr/decr math in
batchRows() instead of in the KVStore ExecuteBatch() machinery.
2016-01-11 16:52:07 -08:00
Steve Yen
94273d5fa9
upside_down process internal rows earlier
...
With this change, internal rows are processed while we're waiting for
backIndex rows to be retrieved.
2016-01-11 16:25:35 -08:00
Steve Yen
bb5cd8f3d6
upside_down merge backIndexRow concurrently
...
Previously, the code would gather all the backIndexRows before
processing them. This change instead merges the backIndexRows
concurrently on the theory that we might as well make progress on
compute & processing tasks while waiting for the rest of the back
index rows to be fetched from the KVStore.
2016-01-10 18:50:42 -08:00
Steve Yen
c3b5246b0c
upside_down track analysis time tighter; and comments
2016-01-10 15:36:54 -08:00
Steve Yen
d3dd40d334
upside_down retrieves backindex concurrently with analysis
...
Start backindex reading concurrently with analysi to try to utilize
more I/O bandwidth.
The analysis time vs indexing time stats tracking are also now "off",
since there's now concurrency between those actiivties.
One tradeoff is that the lock area in upside_down Batch() is increased
as part of this change.
2016-01-10 15:18:28 -08:00
Steve Yen
bff95eef70
firestorm close kvwriter sooner
2016-01-10 15:18:27 -08:00
Marty Schoch
d533b326f6
Merge pull request #315 from steveyen/WIP-perf-20160107
...
KVBatch Close() method
2016-01-08 09:47:37 -05:00
Steve Yen
860de28a28
fix memory leak by closing batches in batchRows()
2016-01-07 17:59:42 -08:00
Steve Yen
70105477cf
added Close() method to KVBatch interface
2016-01-07 17:54:21 -08:00
Marty Schoch
2b947e8c14
Merge branch 'steveyen-WIP-perf-20160106'
2016-01-07 15:42:53 -05:00
Marty Schoch
48fcd5a7d5
Merge branch 'WIP-perf-20160106' of https://github.com/steveyen/bleve into steveyen-WIP-perf-20160106
2016-01-07 15:40:29 -05:00
Marty Schoch
665f5c58e1
fix errcheck violation
2016-01-07 11:11:43 -05:00
Marty Schoch
e54db33346
try testing slightly different way
2016-01-07 11:06:18 -05:00
Marty Schoch
cd940cc375
add another check to try to understand test failure on travis
2016-01-07 10:45:20 -05:00
Marty Schoch
155e496121
switch to containerized builds
...
switch to go 1.5 (required by errcheck now)
do not install go vet (part of Go now)
2016-01-07 09:57:23 -05:00
Steve Yen
846912d083
upside_down udc.termVectorsFromTokenFreq rows append optimization
2016-01-07 00:48:34 -08:00
Steve Yen
8b980bd2ef
firestorm avoid extra goroutine, similar to upside_down
2016-01-07 00:43:27 -08:00
Steve Yen
fbd0e7bfe9
upside_down backIndexTermEntries precalloc'ed capacity
2016-01-07 00:23:25 -08:00
Steve Yen
4eee8821f9
upside_down storeField/indexField append to provided arrays
...
Taking another optimization from firestorm, upside_down's
storeField()/indexField() funcs now also append() to passed-in arrays
rather than always allocating their own arrays.
2016-01-07 00:13:46 -08:00
Steve Yen
1af2927967
upside_down gets analysis perf rows optimizations from firestorm
2016-01-06 23:53:13 -08:00
Steve Yen
82b8b3468e
upside_down analysis converts to docIDBytes once
2016-01-06 23:38:02 -08:00
Steve Yen
d6a997d8c1
firestorm gtreap lookup once per snapshot docID
...
Previously, firestorm would lookup docID's in the inFlight gtreap for
every candidate docNum, and this change moves the lookup to outside of
the loop.
2016-01-06 16:46:15 -08:00
Steve Yen
024848ac91
firestorm valid docNum finding, fixes #310
2016-01-06 16:04:56 -08:00
Steve Yen
7df07f94fa
firestorm use the ParseKey() funcs to avoid unneeded value parsing
...
With this change, the row allocation also happens only once per loop,
instead of once per item.
2016-01-06 15:53:12 -08:00
Steve Yen
009d59222a
firestorm StoredRow.ParseKey() func
2016-01-06 15:46:26 -08:00
Steve Yen
8389027ae8
firestorm TermFreqRow.ParseKey() func
2016-01-06 15:32:09 -08:00
Marty Schoch
83cd8da394
Merge pull request #307 from steveyen/WIP-perf-20160105
...
analyze locations only if includeTermVectors enabled
2016-01-05 16:04:59 -05:00
Steve Yen
89d17f01ef
analyze locations only if includeTermVectors enabled
...
With this change, TermLocations are computed and maintained only if
includeTermVectors is enabled, for higher performance.
2016-01-05 12:46:46 -08:00
Marty Schoch
e5c1af4164
add travis config to run integration tests against firestorm
2016-01-05 13:00:36 -05:00
Marty Schoch
ab67b2f642
Merge pull request #267 from pmezard/doc-dump-methods
...
index: document DumpAll, DumpDoc and DumpFields methods
2016-01-05 09:55:35 -05:00
Marty Schoch
db7363fba1
Merge pull request #305 from steveyen/WIP-perf-20160102
...
perf 20160102
2016-01-05 08:54:47 -05:00
Steve Yen
70b7e73c82
firestorm compensator inFlight.Get() might return nil
2016-01-03 10:21:54 -08:00
Steve Yen
fb8c9a7475
firestorm.Batch() collects [][]IndexRows instead of []IndexRow
...
Rather than append() all received rows into a flat []IndexRow during
the result gathering loop, this change instead collects the analysis
result rows into a [][]IndexRow, which avoids extra copying.
As part of this, firestorm batchRows() now takes the [][]IndexRow as
its input.
2016-01-02 12:30:47 -08:00