0
0
Commit Graph

121 Commits

Author SHA1 Message Date
Steve Yen
1d63c55f7c parse mossLowerLevelMaxBatchSize only when lower-level-store exists 2016-03-09 10:09:15 -08:00
Steve Yen
76b9365928 added moss RegistryCollectionOptions
The moss RegistryCollectionOptions allows applications to register
moss-related callback API functions and other advanced feature usage
at process initialization time.

For example, this could be used for moss's OnError(), OnEvent() and
logging callback options.
2016-03-09 09:40:29 -08:00
Marty Schoch
d7292ed891 add support for gathering stats via map for easier consumption 2016-03-07 18:37:46 -05:00
Marty Schoch
147debaa12 expose metrics and moss stats wrapping underlying stats as well 2016-03-04 13:43:39 -05:00
Steve Yen
f6d1bd2c87 moss option MaxPreMergerBatches renamed 2016-03-03 11:18:30 -08:00
Steve Yen
7d67d89a9c MB-18441 - moss lower-level iterator starts positioned on current
The iterator starts off positioned so that Current() is correct, so
invoking Next() right off the bat was incorrect.
2016-03-01 21:45:48 -08:00
Steve Yen
dd1718fa78 index/store/moss uses AllocMerge() instead of Merge()
Performance optimization.  Before this change, by using Merge()
instead of AllocMerge(), moss's internal batch buf's would be
wastefully, dramatically grown during append()'s to a mis-sized buf.
2016-02-22 11:48:02 -08:00
Steve Yen
ea1a52464d more index/store/moss err handling 2016-02-20 14:25:42 -08:00
Steve Yen
eb315fa500 integrate index/store/moss KV store 2016-02-20 14:25:42 -08:00
Marty Schoch
40c95513b7 add support for including kvstore stats 2016-02-05 12:26:19 -05:00
Marty Schoch
710d06e974 add support for native C merge operators 2016-01-27 17:51:07 -05:00
Steve Yen
d97e3caf4f fix comment typo 2016-01-22 09:04:24 -08:00
Steve Yen
d5de1d3da1 metrics implements BatchEx correctly 2016-01-21 11:00:41 -08:00
Marty Schoch
1335eb2a7b Merge pull request #322 from steveyen/WIP-perf-20160113
KVReader.MultiGet and KVWriter.NewBatchEx API's
2016-01-15 14:28:59 -05:00
opennota
8517feb1c6 Fix some typos 2016-01-15 05:46:27 +07:00
Steve Yen
d94ccf2d74 added KVWriter.NewBatchEx() method 2016-01-13 16:19:04 -08:00
Steve Yen
fb048f6c64 added KVReader.MultiGet() method 2016-01-13 15:12:10 -08:00
Steve Yen
70105477cf added Close() method to KVBatch interface 2016-01-07 17:54:21 -08:00
Marty Schoch
ff11f83842 properly handle errors inside metrics kvstore reporting 2015-11-24 12:52:03 -05:00
Kosov Eugene
45e670b99b BoltDB wrapper nano optimization which makes code a bit prettier too 2015-11-05 00:27:28 +03:00
Marty Schoch
1a978a4591 fix go vet issues and cleanup reader/iterator 2015-10-26 16:41:58 -04:00
Marty Schoch
f0d282f5f8 add test case for seeing prefix iterators outside of range
similar to #256 except for prefix iterators
includes fix for boltdb and gtreap which had incorrect behavior
2015-10-26 16:14:29 -04:00
Patrick Mezard
da72d0c2b9 store_test: deduplicate store initialization 2015-10-20 19:21:01 +02:00
Patrick Mezard
873f483804 gtreap: RangeIterator.Seek should not move before start 2015-10-20 19:12:30 +02:00
Patrick Mezard
5d7628ba3b boltdb: fix RangeIterator outside of range seeks
Two issues:
- Seeking before i.start and iterating returned keys before i.start
- Seeking after the store last key did not invalidate the iterator and
  could cause infinite loops.
2015-10-20 19:09:51 +02:00
Patrick Mezard
aada2e7333 store_test: test RangeIterator.Seek on goleveldb 2015-10-20 19:09:38 +02:00
Marty Schoch
f0ee9a3c66 removed commented code and unused functions 2015-10-19 11:13:03 -04:00
Marty Schoch
0f05d1d3ca Merge branch 'master' into newkvstore 2015-10-09 10:33:41 -04:00
Patrick Mezard
9d5407be13 boltdb: add "nosync" option to force boltdb.DB.NoSync=true
Use this option when rebuilding indexes from scratch. In my small case
(~17000 json documents), it reduces indexing from 520s to 250s.

I did not add any test, short of forced indexing termination it only
has performance effects, which are hard to test. And unknown options are
currently ignored.

Issue #240
2015-10-03 14:26:48 +02:00
Marty Schoch
d06b526cbf more refactoring 2015-09-28 16:50:27 -04:00
Marty Schoch
900f1b4a67 major kvstore interface and impl overhaul
clarified the interface contract
2015-09-23 11:25:47 -07:00
Marty Schoch
f81b2be334 major refactor of bleve configuration
see #221 for full details
2015-09-16 17:10:59 -04:00
Marty Schoch
7ad7659ce5 add support for using null kvstore outside of bleve internals 2015-09-02 11:50:06 -04:00
Marty Schoch
07d37ca38a add important rocksdb config options 2015-09-02 11:49:42 -04:00
Marty Schoch
3e60ca24ec support using end key on forestdb iterator for term freq lookup
also additoanl forestdb configs
2015-08-18 16:22:02 -04:00
Marty Schoch
d3dda3d0ea fixup config parsing and add new options 2015-08-12 13:18:23 -04:00
Marty Schoch
bf8dcae76b removing build tags 2015-07-28 18:59:10 -04:00
Marty Schoch
17ef48f82a switching back to the canonical goleveldb repo 2015-07-08 12:21:17 -06:00
Marty Schoch
bf80f4628e fix bug in curent goleveldb (must copy during iteration)
also changed over to mschoch fork of goleveldb (temporary)

the change to my fork is pending some read-only issues described
here:  https://github.com/syndtr/goleveldb/issues/111

hopefully we can find a path forward, and get that addressed upstream
2015-07-06 18:00:05 -04:00
Marty Schoch
7be7ecdf8e fix batch indexing bug, incremented docCount before commit
fixes #211
2015-06-08 14:14:05 -04:00
Marty Schoch
a52d3b5c07 put in hack to allow boltdb reader isolation test to pass
in boltdb, long readers *MAY* block a writer.  in particular if
the write requires additional allocation, it must acquire a lock
already held by the reader.  in general this is not a problem
for bleve (though it can affect performance in some cases), but
it is a problem for the reader isolation test.  this commit
adds a hack to try and avoid the need for additional allocation
closes #208
2015-05-21 11:39:59 -04:00
Steve Yen
e98ae8ab71 update metrics store to latest kvstore api 2015-04-27 11:01:53 -07:00
Marty Schoch
16f538d7b7 close documents returned by iterator before losing their reference
fixes #194
2015-04-24 17:48:21 -04:00
Marty Schoch
b54a59139c change forestdb imports to couchbase not couchbaselabs 2015-04-24 17:35:01 -04:00
Marty Schoch
452fea6a24 adding initial impl of rocksdb kv store 2015-04-24 17:19:44 -04:00
Marty Schoch
a9c07acbfa refactor of kvstore api to support native merge in rocksdb
refactor to share code in emulated batch
refactor to share code in emulated merge
refactor index kvstore benchmarks to share more code
refactor index kvstore benchmarks to be more repeatable
2015-04-24 17:13:50 -04:00
indraniel
a62320a50e + fix goleveldb's BytesSafeAfterClose() on reader
- it should be set to false
2015-04-10 15:45:22 -05:00
Marty Schoch
5f66bd84c7 fix issues identified by errcheck 2015-04-10 14:59:05 -04:00
indraniel
54ab493b3e + correctly copy bytes from the goleveldb store
- this is part of a recent bleve KVStore API change.

    See the following two google group threads for more details:

    * [help adding goleveldb as an alternative Key/Value store for bleve][1]
    * [bleve search performance improvement][2]

    [1]: https://groups.google.com/forum/#!topic/bleve/aHZ8gmihLiY
    [2]: https://groups.google.com/forum/#!topic/bleve/aTyqsSnbhik
2015-04-10 11:25:23 -05:00
indraniel
81bef38cce Revert "+ make copies of the []bytes returned by goleveldb"
This reverts commit cb8c1741289a0f00b30733e0d52d9d81d1199603.

This commit is no longer desired. The KV store API has been changed to
better address this issue.

For more details, see the google group conversation thread at:

https://groups.google.com/forum/#!topic/bleve/aHZ8gmihLiY
2015-04-10 11:12:44 -05:00