Marty Schoch
d06b526cbf
more refactoring
2015-09-28 16:50:27 -04:00
Marty Schoch
900f1b4a67
major kvstore interface and impl overhaul
...
clarified the interface contract
2015-09-23 11:25:47 -07:00
Marty Schoch
f81b2be334
major refactor of bleve configuration
...
see #221 for full details
2015-09-16 17:10:59 -04:00
Marty Schoch
7ad7659ce5
add support for using null kvstore outside of bleve internals
2015-09-02 11:50:06 -04:00
Marty Schoch
07d37ca38a
add important rocksdb config options
2015-09-02 11:49:42 -04:00
Marty Schoch
3e60ca24ec
support using end key on forestdb iterator for term freq lookup
...
also additoanl forestdb configs
2015-08-18 16:22:02 -04:00
Marty Schoch
d3dda3d0ea
fixup config parsing and add new options
2015-08-12 13:18:23 -04:00
Marty Schoch
bf8dcae76b
removing build tags
2015-07-28 18:59:10 -04:00
Marty Schoch
17ef48f82a
switching back to the canonical goleveldb repo
2015-07-08 12:21:17 -06:00
Marty Schoch
bf80f4628e
fix bug in curent goleveldb (must copy during iteration)
...
also changed over to mschoch fork of goleveldb (temporary)
the change to my fork is pending some read-only issues described
here: https://github.com/syndtr/goleveldb/issues/111
hopefully we can find a path forward, and get that addressed upstream
2015-07-06 18:00:05 -04:00
Marty Schoch
7be7ecdf8e
fix batch indexing bug, incremented docCount before commit
...
fixes #211
2015-06-08 14:14:05 -04:00
Marty Schoch
a52d3b5c07
put in hack to allow boltdb reader isolation test to pass
...
in boltdb, long readers *MAY* block a writer. in particular if
the write requires additional allocation, it must acquire a lock
already held by the reader. in general this is not a problem
for bleve (though it can affect performance in some cases), but
it is a problem for the reader isolation test. this commit
adds a hack to try and avoid the need for additional allocation
closes #208
2015-05-21 11:39:59 -04:00
Steve Yen
e98ae8ab71
update metrics store to latest kvstore api
2015-04-27 11:01:53 -07:00
Marty Schoch
16f538d7b7
close documents returned by iterator before losing their reference
...
fixes #194
2015-04-24 17:48:21 -04:00
Marty Schoch
b54a59139c
change forestdb imports to couchbase not couchbaselabs
2015-04-24 17:35:01 -04:00
Marty Schoch
452fea6a24
adding initial impl of rocksdb kv store
2015-04-24 17:19:44 -04:00
Marty Schoch
a9c07acbfa
refactor of kvstore api to support native merge in rocksdb
...
refactor to share code in emulated batch
refactor to share code in emulated merge
refactor index kvstore benchmarks to share more code
refactor index kvstore benchmarks to be more repeatable
2015-04-24 17:13:50 -04:00
indraniel
a62320a50e
+ fix goleveldb's BytesSafeAfterClose() on reader
...
- it should be set to false
2015-04-10 15:45:22 -05:00
Marty Schoch
5f66bd84c7
fix issues identified by errcheck
2015-04-10 14:59:05 -04:00
indraniel
54ab493b3e
+ correctly copy bytes from the goleveldb store
...
- this is part of a recent bleve KVStore API change.
See the following two google group threads for more details:
* [help adding goleveldb as an alternative Key/Value store for bleve][1]
* [bleve search performance improvement][2]
[1]: https://groups.google.com/forum/#!topic/bleve/aHZ8gmihLiY
[2]: https://groups.google.com/forum/#!topic/bleve/aTyqsSnbhik
2015-04-10 11:25:23 -05:00
indraniel
81bef38cce
Revert "+ make copies of the []bytes returned by goleveldb"
...
This reverts commit cb8c1741289a0f00b30733e0d52d9d81d1199603.
This commit is no longer desired. The KV store API has been changed to
better address this issue.
For more details, see the google group conversation thread at:
https://groups.google.com/forum/#!topic/bleve/aHZ8gmihLiY
2015-04-10 11:12:44 -05:00
indraniel
3a70401835
+ make copies of the []bytes returned by goleveldb
...
- The byte strings returned by goleveldb aren't necessarily safe. See
the following google group thread:
https://groups.google.com/forum/#!topic/bleve/aHZ8gmihLiY
This code change is based on the gist created here:
https://groups.google.com/forum/#!topic/bleve/aHZ8gmihLiY
2015-04-10 11:08:02 -05:00
indraniel
a0a2a61050
+ keep 'get' consistent with levigo implementation
...
- this change keeps the method behavior consistent with the
levigo/leveldb implementation.
- don't issue an err if a key isn't found
2015-04-10 11:08:02 -05:00
indraniel
5e55fa2866
+ keep 'getWithSnapshot' consistent with levigo implementation
...
- this change keeps the method behavior consistent with the
levigo/leveldb implementation.
- the leveldb store_test.go and goleveldb store_test.go are now
identical.
2015-04-10 11:08:02 -05:00
indraniel
caa19e6c36
+ initial stub of goleveldb package
...
- This is a first-pass introduction. Things may not be working
correctly yet.
2015-04-10 11:08:02 -05:00
Marty Schoch
ab24772bf0
fix issues identified by errcheck
...
part of #169
2015-04-07 16:34:29 -04:00
Marty Schoch
56c4a09de1
fix issues identified by errcheck
...
part of #169
2015-04-07 15:39:56 -04:00
Marty Schoch
93e01a803e
fix issues identified by errcheck
...
part of #169
2015-04-07 14:52:00 -04:00
Marty Schoch
dd921d31e3
undoing f92ab131e4
...
we now guarantee bytes were copied earlier in the chain
the kv store is NOT responsible for making an additional copy
closes #181
2015-04-07 11:12:28 -04:00
Marty Schoch
443c0252e0
fix another metrics BytesSafeAfterClose() loop
...
closes #184
2015-04-03 21:17:23 -04:00
Steve Yen
efc39a6857
fix metrics BytesSafeAfterClose() loop
...
fixes issue 184
2015-04-03 16:36:32 -07:00
Marty Schoch
867110e03b
major improvements to index row encoding
...
improvements uncovered some issues with how k/v data was copied
or not. to address this, kv abstraction layer now lets impl
specify if the bytes returned are safe to use after a reader
(or writer since writers are also readers) are closed
See index/store/KVReader - BytesSafeAfterClose() bool
false is the safe value if you're not sure
it will cause index impls to copy the data
Some kv impls already have created a copy a the C-api barrier
in which case they can safely return true.
Overall this yields ~25% speedup for searches with leveldb.
It yields ~10% speedup for boltdb.
Returning stored fields is now slower with boltdb, as previously
we were returning unsafe bytes.
2015-04-03 16:50:48 -04:00
Steve Yen
dbf50b7f29
KVStore gtreap allows only 1 writer at a time
2015-03-26 16:40:18 -07:00
Steve Yen
f92ab131e4
KVStore gtreap implementation copies value bytes
2015-03-26 14:46:37 -07:00
Steve Yen
78453dab7d
metrics KVStore now tracks last 100 errors
2015-03-19 18:41:16 -07:00
Steve Yen
38ee9be353
added some batch size 1000 microbenchmarks
2015-01-30 15:58:39 -08:00
Steve Yen
7d6a6aeaa8
single append for inmem KVStore batch
2015-01-29 11:14:08 -08:00
Steve Yen
5a30d36b17
cznicb KVStore uses Put() for faster read-modify-write
2015-01-29 11:02:01 -08:00
Steve Yen
b054cddf76
gtreap KVStore does 1 append for batch Set/Delete
2015-01-29 10:49:39 -08:00
Steve Yen
05d222f490
cznicb KVStore batch uses <2 appends per Set/Delete
2015-01-29 10:22:13 -08:00
Steve Yen
c5c59e61f4
make leveldb faster with non-zero sized batch
2015-01-29 10:20:26 -08:00
Steve Yen
1c1774d4ad
throw away data even faster in null KVStore
2015-01-29 10:17:21 -08:00
Steve Yen
782ad94e01
added debug tag for metrics KVStore
2015-01-16 11:18:40 -08:00
Marty Schoch
eebc8e7825
more debuging around forestdb snapshots
2015-01-16 14:18:28 -05:00
Marty Schoch
ba978ea27e
improving log messages
2015-01-16 14:07:47 -05:00
Marty Schoch
09fe749913
default to autocompaction for forestdb
2015-01-16 13:35:43 -05:00
Steve Yen
12dc2aff93
add go1.4 build tag to cznicb KVStore
...
This is because github.com/cznic/b depends on sync.Pool.
2015-01-15 15:54:25 -08:00
Steve Yen
11ee0209ad
no leading zeros for metrics CSV output
2015-01-15 15:09:53 -08:00
Steve Yen
202191201c
added WriteCSV() to metrics KVStore
2015-01-15 14:11:15 -08:00
Steve Yen
9be4e217bc
metrics KVStore tracks perf metrics on a wrapped KVStore
2015-01-15 11:42:41 -08:00