Silvan Jegen
d326898f7b
Remove unneeded brackets
2016-01-14 16:41:41 +01:00
Marty Schoch
af25e724f6
Merge branch 'master' of https://github.com/slavikm/bleve into slavikm-master
2016-01-13 16:10:59 -05:00
Steve Yen
0e72b949b3
upside_down batchRows() takes array of arrays
...
In order to spend less time in append(), this change in upside_down
(similar to another recent performance change in firestorm) builds up
an array of arrays as the eventual input to batchRows().
2016-01-11 18:11:21 -08:00
slavikm
680be52f87
Implemented boolean field support
2016-01-11 17:18:03 -08:00
Steve Yen
7ce7d98cba
upside_down merge dictionary deltas before using batch.Merge()
...
This change performs more dictionary delta incr/decr math in
batchRows() instead of in the KVStore ExecuteBatch() machinery.
2016-01-11 16:52:07 -08:00
Steve Yen
94273d5fa9
upside_down process internal rows earlier
...
With this change, internal rows are processed while we're waiting for
backIndex rows to be retrieved.
2016-01-11 16:25:35 -08:00
Steve Yen
bb5cd8f3d6
upside_down merge backIndexRow concurrently
...
Previously, the code would gather all the backIndexRows before
processing them. This change instead merges the backIndexRows
concurrently on the theory that we might as well make progress on
compute & processing tasks while waiting for the rest of the back
index rows to be fetched from the KVStore.
2016-01-10 18:50:42 -08:00
Steve Yen
c3b5246b0c
upside_down track analysis time tighter; and comments
2016-01-10 15:36:54 -08:00
Steve Yen
d3dd40d334
upside_down retrieves backindex concurrently with analysis
...
Start backindex reading concurrently with analysi to try to utilize
more I/O bandwidth.
The analysis time vs indexing time stats tracking are also now "off",
since there's now concurrency between those actiivties.
One tradeoff is that the lock area in upside_down Batch() is increased
as part of this change.
2016-01-10 15:18:28 -08:00
Steve Yen
860de28a28
fix memory leak by closing batches in batchRows()
2016-01-07 17:59:42 -08:00
Steve Yen
846912d083
upside_down udc.termVectorsFromTokenFreq rows append optimization
2016-01-07 00:48:34 -08:00
Steve Yen
8b980bd2ef
firestorm avoid extra goroutine, similar to upside_down
2016-01-07 00:43:27 -08:00
Steve Yen
fbd0e7bfe9
upside_down backIndexTermEntries precalloc'ed capacity
2016-01-07 00:23:25 -08:00
Steve Yen
4eee8821f9
upside_down storeField/indexField append to provided arrays
...
Taking another optimization from firestorm, upside_down's
storeField()/indexField() funcs now also append() to passed-in arrays
rather than always allocating their own arrays.
2016-01-07 00:13:46 -08:00
Steve Yen
1af2927967
upside_down gets analysis perf rows optimizations from firestorm
2016-01-06 23:53:13 -08:00
Steve Yen
82b8b3468e
upside_down analysis converts to docIDBytes once
2016-01-06 23:38:02 -08:00
Steve Yen
89d17f01ef
analyze locations only if includeTermVectors enabled
...
With this change, TermLocations are computed and maintained only if
includeTermVectors is enabled, for higher performance.
2016-01-05 12:46:46 -08:00
Marty Schoch
8efbd556a3
fix indexing bug with data coming from arrays
...
fixes #295
2015-12-21 14:59:32 -05:00
Marty Schoch
a73a178923
fix incorrect prefix search behavior
...
avoids double incrementing of end term when reading term dict
fixes #293
2015-12-04 14:07:16 -05:00
Patrick Mezard
e85c9c542e
row: expose TermFrequencyRow term and freq fields
...
Rows content is an implementation detail of bleve index and may change
in the future. That said, they also contains information valuable to
assess the quality of the index or understand its performances. So, as
long as we agree that type asserting rows should only be done if you
know what you are doing and are ready to deal with future changes, I see
no reason to hide the row fields from external packages.
Fix #268
2015-11-17 17:21:26 +01:00
Marty Schoch
30651065e9
fix panic on insufficiently sized buffer
...
adds test case to reproduce original problem
fixes #264
2015-10-30 18:25:38 -04:00
Marty Schoch
2bd3ef4080
copy relevant k/v pairs before advancing underlying iterator
2015-10-28 12:23:54 -04:00
Marty Schoch
d1b07f4909
fix dump methods to properly copy keys and values
2015-10-28 12:06:44 -04:00
Marty Schoch
6cc21346dc
fix errcheck issues
2015-10-19 14:27:03 -04:00
Marty Schoch
817c317c90
Merge branch 'master' into newkvstore
2015-10-19 12:04:07 -04:00
Marty Schoch
faceecf87b
make row buffer size constant/configurable
...
also handle case where it is insufficiently sized
2015-10-19 12:03:38 -04:00
Marty Schoch
c9471d5739
Merge pull request #244 from kevgs/master
...
reducing allocation count
2015-10-16 15:51:30 -04:00
Marty Schoch
e6d0fc8d95
Merge pull request #247 from pmezard/remove-update-goroutine
...
upside_down: no need for a goroutine to enqueue AnalysisWork
2015-10-16 10:15:55 -04:00
Marty Schoch
4c6bc23043
rewrite to keep using same buffer when possible
2015-10-13 14:04:56 -07:00
Marty Schoch
8de860bf12
2 more places that used old Key()
2015-10-13 12:35:08 -07:00
Marty Schoch
5f594d1acc
Merge branch 'master' into newkvstore
2015-10-12 18:07:04 -07:00
Marty Schoch
08572e4925
move literals outside loop for more predicatble test results
2015-10-12 18:06:38 -07:00
Patrick Mezard
8c928539ee
upside_down: no need for a goroutine to enqueue AnalysisWork
...
It boils down to:
1. client sends some work and a notification channel to a single worker,
then waits.
2. worker processes the work
3. worker sends the result to the client using the notification channel
I do not see any problem with this, even with unbuffered channels.
2015-10-12 10:42:14 +02:00
Marty Schoch
95e06538f3
fix benchmarks for the x kvstores
2015-10-09 11:09:42 -04:00
Marty Schoch
0f05d1d3ca
Merge branch 'master' into newkvstore
2015-10-09 10:33:41 -04:00
Patrick Mezard
aee82f8b49
upside_down: simplify return code in batchRows()
2015-10-09 09:57:12 +02:00
Marty Schoch
e28eb749d7
bump up buffer size
2015-10-06 16:45:38 -04:00
Marty Schoch
71cbb13e07
modify code to reuse buffer for kv generation
2015-10-05 17:49:50 -04:00
Kosov Eugene
a61c350888
reducing allocation count
2015-10-05 22:57:10 +03:00
Marty Schoch
d06b526cbf
more refactoring
2015-09-28 16:50:27 -04:00
Marty Schoch
900f1b4a67
major kvstore interface and impl overhaul
...
clarified the interface contract
2015-09-23 11:25:47 -07:00
Marty Schoch
f81b2be334
major refactor of bleve configuration
...
see #221 for full details
2015-09-16 17:10:59 -04:00
Marty Schoch
17c64d37c7
add similar benchmarks from firestorm
2015-09-10 08:13:52 -04:00
Marty Schoch
dbb93b75a4
refactoring to allow pluggable index encodings
...
this lays the foundation for supporting the new firestorm
indexing scheme. i'm merging these changes ahead of
the rest of the firestorm branch so i can continue
to make changes to the analysis pipeline in parallel
2015-09-02 13:12:08 -04:00
Marty Schoch
3e60ca24ec
support using end key on forestdb iterator for term freq lookup
...
also additoanl forestdb configs
2015-08-18 16:22:02 -04:00
Marty Schoch
01667dfff3
faster protobufs with gogo
2015-08-12 13:18:23 -04:00
Marty Schoch
7df66b4857
fix broken benchmark cause by index row encoding change
2015-08-06 14:48:04 -04:00
Marty Schoch
9db850a53e
Merge branch 'fix/MaxVarintLen64' of https://github.com/tukdesk/bleve into tukdesk-fix/MaxVarintLen64
2015-07-31 15:16:16 -04:00
Marty Schoch
3682c25467
update to correctly work with composite fields
...
also updated search results to return array positions
2015-07-31 11:16:11 -04:00
Marty Schoch
c1c4941dde
Merge branch 'feature/term_vector' of https://github.com/tukdesk/bleve into tukdesk-feature/term_vector
2015-07-29 14:31:15 -04:00