Marty Schoch
16f538d7b7
close documents returned by iterator before losing their reference
...
fixes #194
2015-04-24 17:48:21 -04:00
Marty Schoch
b54a59139c
change forestdb imports to couchbase not couchbaselabs
2015-04-24 17:35:01 -04:00
Marty Schoch
ee47d1c21a
standardize on including 1000 sized batches
2015-04-24 17:31:34 -04:00
Marty Schoch
452fea6a24
adding initial impl of rocksdb kv store
2015-04-24 17:19:44 -04:00
Marty Schoch
a9c07acbfa
refactor of kvstore api to support native merge in rocksdb
...
refactor to share code in emulated batch
refactor to share code in emulated merge
refactor index kvstore benchmarks to share more code
refactor index kvstore benchmarks to be more repeatable
2015-04-24 17:13:50 -04:00
indraniel
a62320a50e
+ fix goleveldb's BytesSafeAfterClose() on reader
...
- it should be set to false
2015-04-10 15:45:22 -05:00
Marty Schoch
d5dc66313f
change variable name conflicting when both LevelDB bencharmks run
2015-04-10 15:03:44 -04:00
Marty Schoch
d5caad4405
changed GoLevelDB benchmark names to be different from LevelDB
...
this will allow for easier comparision when running both
versions at the same time
2015-04-10 15:00:56 -04:00
Marty Schoch
5f66bd84c7
fix issues identified by errcheck
2015-04-10 14:59:05 -04:00
indraniel
54ab493b3e
+ correctly copy bytes from the goleveldb store
...
- this is part of a recent bleve KVStore API change.
See the following two google group threads for more details:
* [help adding goleveldb as an alternative Key/Value store for bleve][1]
* [bleve search performance improvement][2]
[1]: https://groups.google.com/forum/#!topic/bleve/aHZ8gmihLiY
[2]: https://groups.google.com/forum/#!topic/bleve/aTyqsSnbhik
2015-04-10 11:25:23 -05:00
indraniel
81bef38cce
Revert "+ make copies of the []bytes returned by goleveldb"
...
This reverts commit cb8c1741289a0f00b30733e0d52d9d81d1199603.
This commit is no longer desired. The KV store API has been changed to
better address this issue.
For more details, see the google group conversation thread at:
https://groups.google.com/forum/#!topic/bleve/aHZ8gmihLiY
2015-04-10 11:12:44 -05:00
indraniel
3a70401835
+ make copies of the []bytes returned by goleveldb
...
- The byte strings returned by goleveldb aren't necessarily safe. See
the following google group thread:
https://groups.google.com/forum/#!topic/bleve/aHZ8gmihLiY
This code change is based on the gist created here:
https://groups.google.com/forum/#!topic/bleve/aHZ8gmihLiY
2015-04-10 11:08:02 -05:00
indraniel
a88d714778
+ add a goleveldb index updside-down benchmark test
2015-04-10 11:08:02 -05:00
indraniel
a0a2a61050
+ keep 'get' consistent with levigo implementation
...
- this change keeps the method behavior consistent with the
levigo/leveldb implementation.
- don't issue an err if a key isn't found
2015-04-10 11:08:02 -05:00
indraniel
5e55fa2866
+ keep 'getWithSnapshot' consistent with levigo implementation
...
- this change keeps the method behavior consistent with the
levigo/leveldb implementation.
- the leveldb store_test.go and goleveldb store_test.go are now
identical.
2015-04-10 11:08:02 -05:00
indraniel
caa19e6c36
+ initial stub of goleveldb package
...
- This is a first-pass introduction. Things may not be working
correctly yet.
2015-04-10 11:08:02 -05:00
Marty Schoch
8581e73cef
added String method for Batch
...
also changed Batch methods to pointer receiver
closes #180
2015-04-08 10:41:42 -04:00
Marty Schoch
539aeb8dc7
fix errors identified by errcheck
...
part of #169
2015-04-07 18:05:41 -04:00
Marty Schoch
ba6b3c8bb3
fix more issues identified by errcheck
...
part of #169
2015-04-07 16:45:23 -04:00
Marty Schoch
ab24772bf0
fix issues identified by errcheck
...
part of #169
2015-04-07 16:34:29 -04:00
Marty Schoch
56c4a09de1
fix issues identified by errcheck
...
part of #169
2015-04-07 15:39:56 -04:00
Marty Schoch
93e01a803e
fix issues identified by errcheck
...
part of #169
2015-04-07 14:52:00 -04:00
Marty Schoch
f1ec73e764
fix issues identified by errcheck
...
part of #169
2015-04-07 13:26:54 -04:00
Marty Schoch
56a30a3574
fix issues identified by errcheck
...
part of #169
2015-04-07 13:05:47 -04:00
Marty Schoch
d2e9409413
fix issues identified by errcheck
...
part of #169
2015-04-07 12:04:59 -04:00
Marty Schoch
dd921d31e3
undoing f92ab131e4
...
we now guarantee bytes were copied earlier in the chain
the kv store is NOT responsible for making an additional copy
closes #181
2015-04-07 11:12:28 -04:00
Marty Schoch
443c0252e0
fix another metrics BytesSafeAfterClose() loop
...
closes #184
2015-04-03 21:17:23 -04:00
Steve Yen
efc39a6857
fix metrics BytesSafeAfterClose() loop
...
fixes issue 184
2015-04-03 16:36:32 -07:00
Marty Schoch
867110e03b
major improvements to index row encoding
...
improvements uncovered some issues with how k/v data was copied
or not. to address this, kv abstraction layer now lets impl
specify if the bytes returned are safe to use after a reader
(or writer since writers are also readers) are closed
See index/store/KVReader - BytesSafeAfterClose() bool
false is the safe value if you're not sure
it will cause index impls to copy the data
Some kv impls already have created a copy a the C-api barrier
in which case they can safely return true.
Overall this yields ~25% speedup for searches with leveldb.
It yields ~10% speedup for boltdb.
Returning stored fields is now slower with boltdb, as previously
we were returning unsafe bytes.
2015-04-03 16:50:48 -04:00
Steve Yen
dbf50b7f29
KVStore gtreap allows only 1 writer at a time
2015-03-26 16:40:18 -07:00
Steve Yen
f92ab131e4
KVStore gtreap implementation copies value bytes
2015-03-26 14:46:37 -07:00
Steve Yen
78453dab7d
metrics KVStore now tracks last 100 errors
2015-03-19 18:41:16 -07:00
Marty Schoch
a44a7c01af
rewrite to used fixed size []byte instead of buffer
...
removes unchecked errors in calls to buffer.Write
and also benchmarks considerably faster
2015-03-11 15:12:13 -04:00
Marty Schoch
522f9d5cc7
significant change to index format, support dictionary rows
...
this introduces disk format v4
now the summary rows for a term are stored in their own
"dictionary row" format, previously the same information
was stored in special term frequency rows
this now allows us to easily iterate all the terms for a field
in sorted order (useful for many other fuzzy data structures)
at the top-level of bleve you can now browse terms within a field
using the following api on the Index interface:
FieldDict(field string) (index.FieldDict, error)
FieldDictRange(field string, startTerm []byte, endTerm []byte) (index.FieldDict, error)
FieldDictPrefix(field string, termPrefix []byte) (index.FieldDict, error)
fixes #127
2015-03-10 16:22:19 -04:00
Marty Schoch
4e14f4e4ef
change path for forestdb test to correctly cleanup
...
this is due to forestdb auto-compaction using the provided
path as just the prefix, so if we're not careful we end
up with many stray files laying around
here, we create a sub-directory first, and just nuke the
whole subdir when we're done
2015-03-10 14:05:58 -04:00
Marty Schoch
300ec79c96
first pass at checking errors that were ignored
...
part of #169
2015-03-06 14:46:29 -05:00
Marty Schoch
a2ad7634f2
update term freq rows to use varint where possible
...
benchmark old ns/op new ns/op delta
BenchmarkLevelDBIndexing1Workers 1138292 657901 -42.20%
BenchmarkLevelDBIndexing2Workers 1619323 647628 -60.01%
BenchmarkLevelDBIndexing4Workers 1172845 636478 -45.73%
BenchmarkLevelDBIndexing1Workers10Batch 465556545 448153394 -3.74%
BenchmarkLevelDBIndexing2Workers10Batch 504203911 449657355 -10.82%
BenchmarkLevelDBIndexing4Workers10Batch 510766435 439839335 -13.89%
BenchmarkLevelDBIndexing1Workers100Batch 307657846 268976464 -12.57%
BenchmarkLevelDBIndexing2Workers100Batch 302257400 269110215 -10.97%
BenchmarkLevelDBIndexing4Workers100Batch 305320485 259084902 -15.14%
BenchmarkLevelDBIndexing1Workers1000Batch 301320576 258070231 -14.35%
BenchmarkLevelDBIndexing2Workers1000Batch 334174454 261175641 -21.84%
BenchmarkLevelDBIndexing4Workers1000Batch 267732436 261461739 -2.34%
closes #165
2015-03-06 13:00:53 -05:00
Marty Schoch
c566d34264
bump index format version number, start checking version on open
2015-02-17 17:16:31 +05:30
Steve Yen
38ee9be353
added some batch size 1000 microbenchmarks
2015-01-30 15:58:39 -08:00
Steve Yen
7d6a6aeaa8
single append for inmem KVStore batch
2015-01-29 11:14:08 -08:00
Steve Yen
5a30d36b17
cznicb KVStore uses Put() for faster read-modify-write
2015-01-29 11:02:01 -08:00
Steve Yen
b054cddf76
gtreap KVStore does 1 append for batch Set/Delete
2015-01-29 10:49:39 -08:00
Steve Yen
05d222f490
cznicb KVStore batch uses <2 appends per Set/Delete
2015-01-29 10:22:13 -08:00
Steve Yen
c5c59e61f4
make leveldb faster with non-zero sized batch
2015-01-29 10:20:26 -08:00
Steve Yen
1c1774d4ad
throw away data even faster in null KVStore
2015-01-29 10:17:21 -08:00
Steve Yen
782ad94e01
added debug tag for metrics KVStore
2015-01-16 11:18:40 -08:00
Marty Schoch
eebc8e7825
more debuging around forestdb snapshots
2015-01-16 14:18:28 -05:00
Marty Schoch
ba978ea27e
improving log messages
2015-01-16 14:07:47 -05:00
Marty Schoch
09fe749913
default to autocompaction for forestdb
2015-01-16 13:35:43 -05:00
Steve Yen
12dc2aff93
add go1.4 build tag to cznicb KVStore
...
This is because github.com/cznic/b depends on sync.Pool.
2015-01-15 15:54:25 -08:00