Marty Schoch
8581e73cef
added String method for Batch
...
also changed Batch methods to pointer receiver
closes #180
2015-04-08 10:41:42 -04:00
Marty Schoch
539aeb8dc7
fix errors identified by errcheck
...
part of #169
2015-04-07 18:05:41 -04:00
Marty Schoch
ba6b3c8bb3
fix more issues identified by errcheck
...
part of #169
2015-04-07 16:45:23 -04:00
Marty Schoch
ab24772bf0
fix issues identified by errcheck
...
part of #169
2015-04-07 16:34:29 -04:00
Marty Schoch
56c4a09de1
fix issues identified by errcheck
...
part of #169
2015-04-07 15:39:56 -04:00
Marty Schoch
93e01a803e
fix issues identified by errcheck
...
part of #169
2015-04-07 14:52:00 -04:00
Marty Schoch
f1ec73e764
fix issues identified by errcheck
...
part of #169
2015-04-07 13:26:54 -04:00
Marty Schoch
56a30a3574
fix issues identified by errcheck
...
part of #169
2015-04-07 13:05:47 -04:00
Marty Schoch
d2e9409413
fix issues identified by errcheck
...
part of #169
2015-04-07 12:04:59 -04:00
Marty Schoch
dd921d31e3
undoing f92ab131e4
...
we now guarantee bytes were copied earlier in the chain
the kv store is NOT responsible for making an additional copy
closes #181
2015-04-07 11:12:28 -04:00
Marty Schoch
443c0252e0
fix another metrics BytesSafeAfterClose() loop
...
closes #184
2015-04-03 21:17:23 -04:00
Steve Yen
efc39a6857
fix metrics BytesSafeAfterClose() loop
...
fixes issue 184
2015-04-03 16:36:32 -07:00
Marty Schoch
867110e03b
major improvements to index row encoding
...
improvements uncovered some issues with how k/v data was copied
or not. to address this, kv abstraction layer now lets impl
specify if the bytes returned are safe to use after a reader
(or writer since writers are also readers) are closed
See index/store/KVReader - BytesSafeAfterClose() bool
false is the safe value if you're not sure
it will cause index impls to copy the data
Some kv impls already have created a copy a the C-api barrier
in which case they can safely return true.
Overall this yields ~25% speedup for searches with leveldb.
It yields ~10% speedup for boltdb.
Returning stored fields is now slower with boltdb, as previously
we were returning unsafe bytes.
2015-04-03 16:50:48 -04:00
Steve Yen
dbf50b7f29
KVStore gtreap allows only 1 writer at a time
2015-03-26 16:40:18 -07:00
Steve Yen
f92ab131e4
KVStore gtreap implementation copies value bytes
2015-03-26 14:46:37 -07:00
Steve Yen
78453dab7d
metrics KVStore now tracks last 100 errors
2015-03-19 18:41:16 -07:00
Marty Schoch
a44a7c01af
rewrite to used fixed size []byte instead of buffer
...
removes unchecked errors in calls to buffer.Write
and also benchmarks considerably faster
2015-03-11 15:12:13 -04:00
Marty Schoch
522f9d5cc7
significant change to index format, support dictionary rows
...
this introduces disk format v4
now the summary rows for a term are stored in their own
"dictionary row" format, previously the same information
was stored in special term frequency rows
this now allows us to easily iterate all the terms for a field
in sorted order (useful for many other fuzzy data structures)
at the top-level of bleve you can now browse terms within a field
using the following api on the Index interface:
FieldDict(field string) (index.FieldDict, error)
FieldDictRange(field string, startTerm []byte, endTerm []byte) (index.FieldDict, error)
FieldDictPrefix(field string, termPrefix []byte) (index.FieldDict, error)
fixes #127
2015-03-10 16:22:19 -04:00
Marty Schoch
4e14f4e4ef
change path for forestdb test to correctly cleanup
...
this is due to forestdb auto-compaction using the provided
path as just the prefix, so if we're not careful we end
up with many stray files laying around
here, we create a sub-directory first, and just nuke the
whole subdir when we're done
2015-03-10 14:05:58 -04:00
Marty Schoch
300ec79c96
first pass at checking errors that were ignored
...
part of #169
2015-03-06 14:46:29 -05:00
Marty Schoch
a2ad7634f2
update term freq rows to use varint where possible
...
benchmark old ns/op new ns/op delta
BenchmarkLevelDBIndexing1Workers 1138292 657901 -42.20%
BenchmarkLevelDBIndexing2Workers 1619323 647628 -60.01%
BenchmarkLevelDBIndexing4Workers 1172845 636478 -45.73%
BenchmarkLevelDBIndexing1Workers10Batch 465556545 448153394 -3.74%
BenchmarkLevelDBIndexing2Workers10Batch 504203911 449657355 -10.82%
BenchmarkLevelDBIndexing4Workers10Batch 510766435 439839335 -13.89%
BenchmarkLevelDBIndexing1Workers100Batch 307657846 268976464 -12.57%
BenchmarkLevelDBIndexing2Workers100Batch 302257400 269110215 -10.97%
BenchmarkLevelDBIndexing4Workers100Batch 305320485 259084902 -15.14%
BenchmarkLevelDBIndexing1Workers1000Batch 301320576 258070231 -14.35%
BenchmarkLevelDBIndexing2Workers1000Batch 334174454 261175641 -21.84%
BenchmarkLevelDBIndexing4Workers1000Batch 267732436 261461739 -2.34%
closes #165
2015-03-06 13:00:53 -05:00
Marty Schoch
c566d34264
bump index format version number, start checking version on open
2015-02-17 17:16:31 +05:30
Steve Yen
38ee9be353
added some batch size 1000 microbenchmarks
2015-01-30 15:58:39 -08:00
Steve Yen
7d6a6aeaa8
single append for inmem KVStore batch
2015-01-29 11:14:08 -08:00
Steve Yen
5a30d36b17
cznicb KVStore uses Put() for faster read-modify-write
2015-01-29 11:02:01 -08:00
Steve Yen
b054cddf76
gtreap KVStore does 1 append for batch Set/Delete
2015-01-29 10:49:39 -08:00
Steve Yen
05d222f490
cznicb KVStore batch uses <2 appends per Set/Delete
2015-01-29 10:22:13 -08:00
Steve Yen
c5c59e61f4
make leveldb faster with non-zero sized batch
2015-01-29 10:20:26 -08:00
Steve Yen
1c1774d4ad
throw away data even faster in null KVStore
2015-01-29 10:17:21 -08:00
Steve Yen
782ad94e01
added debug tag for metrics KVStore
2015-01-16 11:18:40 -08:00
Marty Schoch
eebc8e7825
more debuging around forestdb snapshots
2015-01-16 14:18:28 -05:00
Marty Schoch
ba978ea27e
improving log messages
2015-01-16 14:07:47 -05:00
Marty Schoch
09fe749913
default to autocompaction for forestdb
2015-01-16 13:35:43 -05:00
Steve Yen
12dc2aff93
add go1.4 build tag to cznicb KVStore
...
This is because github.com/cznic/b depends on sync.Pool.
2015-01-15 15:54:25 -08:00
Steve Yen
11ee0209ad
no leading zeros for metrics CSV output
2015-01-15 15:09:53 -08:00
Steve Yen
202191201c
added WriteCSV() to metrics KVStore
2015-01-15 14:11:15 -08:00
Steve Yen
9be4e217bc
metrics KVStore tracks perf metrics on a wrapped KVStore
2015-01-15 11:42:41 -08:00
Steve Yen
ea0a8657f3
added cznicb in-memory kvstore (no reader isolation)
2015-01-13 17:35:28 -08:00
Marty Schoch
362d240b09
added configurable options to leveldb
2015-01-13 16:24:51 -05:00
Steve Yen
d6e6f655c9
initialize forestdb config if provided
2015-01-13 12:03:24 -08:00
Steve Yen
1fa80ffc40
pass config to forestdb Open()
2015-01-13 11:04:02 -08:00
Steve Yen
3a00a968f2
close levigo's read & write options
2015-01-12 18:42:19 -08:00
Steve Yen
c20726bb93
close levigo.Options when db is closed
2015-01-12 18:42:19 -08:00
Steve Yen
603c3af8bb
added gtreap in-memory, copy-on-write KVStore
2015-01-12 11:26:21 -08:00
Marty Schoch
d68c52e621
adding forestdb benchmark
2015-01-12 12:56:37 -05:00
Steve Yen
ae3600aeea
expose forestdb rollback methods
2015-01-06 18:59:02 -08:00
Steve Yen
5467e0a385
forestdb registered name fixed
2015-01-06 17:36:05 -08:00
Marty Schoch
38bdcbeb62
update to new forestdb iterator api
2014-12-27 13:15:14 -08:00
Silvan Jegen
ef18dfe4cd
Fix typos in comments and strings
2014-12-18 18:43:12 +01:00
Sergey Avseyev
a8351be5a6
Update protobuf imports
2014-12-10 01:24:59 +03:00
