gibheer/bleve
0
0
Fork 0
Code
772 Commits 10 Branches 0 Tags 12 MiB
2b947e8c14
Commit Graph

262 Commits

Author SHA1 Message Date
Patrick Mezard
f2b3d5698e index: document TermFieldReader interface 2015-10-27 18:53:03 +01:00
Patrick Mezard
3df789d258 index: document empty strings behaviour when calling DocIDReader() 2015-10-27 18:53:03 +01:00
Marty Schoch
1a978a4591 fix go vet issues and cleanup reader/iterator 2015-10-26 16:41:58 -04:00
Marty Schoch
f0d282f5f8 add test case for seeing prefix iterators outside of range 
similar to #256 except for prefix iterators
includes fix for boltdb and gtreap which had incorrect behavior
 2015-10-26 16:14:29 -04:00
Patrick Mezard
5100e00f20 doc: DocIDReader.Advance() is no longer implementation dependent 2015-10-20 20:32:23 +02:00
Patrick Mezard
2fa334fc27 doc: talk about "documents" not "indexed or stored documents" 2015-10-20 20:24:24 +02:00
Patrick Mezard
b174c137fd doc: document DocIDReader, and some Index bits 2015-10-20 20:24:24 +02:00
Patrick Mezard
da72d0c2b9 store_test: deduplicate store initialization 2015-10-20 19:21:01 +02:00
Patrick Mezard
873f483804 gtreap: RangeIterator.Seek should not move before start 2015-10-20 19:12:30 +02:00
Patrick Mezard
5d7628ba3b boltdb: fix RangeIterator outside of range seeks 
Two issues:
- Seeking before i.start and iterating returned keys before i.start
- Seeking after the store last key did not invalidate the iterator and
  could cause infinite loops.
 2015-10-20 19:09:51 +02:00
Patrick Mezard
aada2e7333 store_test: test RangeIterator.Seek on goleveldb 2015-10-20 19:09:38 +02:00
Marty Schoch
6cc21346dc fix errcheck issues 2015-10-19 14:27:03 -04:00
Marty Schoch
817c317c90 Merge branch 'master' into newkvstore 2015-10-19 12:04:07 -04:00
Marty Schoch
faceecf87b make row buffer size constant/configurable 
also handle case where it is insufficiently sized
 2015-10-19 12:03:38 -04:00
Marty Schoch
f0ee9a3c66 removed commented code and unused functions 2015-10-19 11:13:03 -04:00
Marty Schoch
c9471d5739 Merge pull request #244 from kevgs/master 
reducing allocation count
 2015-10-16 15:51:30 -04:00
Marty Schoch
e6d0fc8d95 Merge pull request #247 from pmezard/remove-update-goroutine 
upside_down: no need for a goroutine to enqueue AnalysisWork
 2015-10-16 10:15:55 -04:00
Marty Schoch
4c6bc23043 rewrite to keep using same buffer when possible 2015-10-13 14:04:56 -07:00
Marty Schoch
8de860bf12 2 more places that used old Key() 2015-10-13 12:35:08 -07:00
Marty Schoch
5f594d1acc Merge branch 'master' into newkvstore 2015-10-12 18:07:04 -07:00
Marty Schoch
08572e4925 move literals outside loop for more predicatble test results 2015-10-12 18:06:38 -07:00
Patrick Mezard
8c928539ee upside_down: no need for a goroutine to enqueue AnalysisWork 
It boils down to:
1. client sends some work and a notification channel to a single worker,
   then waits.
2. worker processes the work
3. worker sends the result to the client using the notification channel

I do not see any problem with this, even with unbuffered channels.
 2015-10-12 10:42:14 +02:00
Marty Schoch
95e06538f3 fix benchmarks for the x kvstores 2015-10-09 11:09:42 -04:00
Marty Schoch
0f05d1d3ca Merge branch 'master' into newkvstore 2015-10-09 10:33:41 -04:00
Patrick Mezard
aee82f8b49 upside_down: simplify return code in batchRows() 2015-10-09 09:57:12 +02:00
Marty Schoch
e28eb749d7 bump up buffer size 2015-10-06 16:45:38 -04:00
Marty Schoch
71cbb13e07 modify code to reuse buffer for kv generation 2015-10-05 17:49:50 -04:00
Kosov Eugene
a61c350888 reducing allocation count 2015-10-05 22:57:10 +03:00
Patrick Mezard
9d5407be13 boltdb: add "nosync" option to force boltdb.DB.NoSync=true 
Use this option when rebuilding indexes from scratch. In my small case
(~17000 json documents), it reduces indexing from 520s to 250s.

I did not add any test, short of forced indexing termination it only
has performance effects, which are hard to test. And unknown options are
currently ignored.

Issue #240
 2015-10-03 14:26:48 +02:00
Marty Schoch
d06b526cbf more refactoring 2015-09-28 16:50:27 -04:00
Marty Schoch
66aa1b020a Merge branch 'master' into firestorm 2015-09-23 11:32:25 -07:00
Marty Schoch
900f1b4a67 major kvstore interface and impl overhaul 
clarified the interface contract
 2015-09-23 11:25:47 -07:00
Marty Schoch
f81b2be334 major refactor of bleve configuration 
see #221 for full details
 2015-09-16 17:10:59 -04:00
Marty Schoch
c308f611cf skip unnecessary map before slice 
benchmark            old ns/op     new ns/op     delta
BenchmarkBatch-4     16950972      16377194      -3.38%

benchmark            old allocs     new allocs     delta
BenchmarkBatch-4     136164         136161         -0.00%

benchmark            old bytes     new bytes     delta
BenchmarkBatch-4     7168872       7109691       -0.83%
 2015-09-10 08:21:26 -04:00
Marty Schoch
f6f1628b15 avoid doing unnecessary work: 
benchmark            old ns/op     new ns/op     delta
BenchmarkBatch-4     20738739      17047158      -17.80%

benchmark            old allocs     new allocs     delta
BenchmarkBatch-4     136423         136160         -0.19%

benchmark            old bytes     new bytes     delta
BenchmarkBatch-4     20277781      7168772       -64.65%
 2015-09-10 08:19:05 -04:00
Marty Schoch
c8538c835f Merge branch 'master' into firestorm 2015-09-10 08:14:14 -04:00
Marty Schoch
17c64d37c7 add similar benchmarks from firestorm 2015-09-10 08:13:52 -04:00
Marty Schoch
1e4d637761 adding more benchmarks 2015-09-10 08:01:11 -04:00
Marty Schoch
f74ed6a9ae Merge remote-tracking branch 'origin' into firestorm 
cathching up with changes from master
 2015-09-02 13:29:03 -04:00
Marty Schoch
dbb93b75a4 refactoring to allow pluggable index encodings 
this lays the foundation for supporting the new firestorm
indexing scheme.  i'm merging these changes ahead of
the rest of the firestorm branch so i can continue
to make changes to the analysis pipeline in parallel
 2015-09-02 13:12:08 -04:00
Marty Schoch
7ad7659ce5 add support for using null kvstore outside of bleve internals 2015-09-02 11:50:06 -04:00
Marty Schoch
07d37ca38a add important rocksdb config options 2015-09-02 11:49:42 -04:00
Marty Schoch
18151862b5 fix go vet issues 2015-08-25 15:13:13 -04:00
Marty Schoch
84811cf5a0 made index type configurable + first version of firestorm 2015-08-25 14:52:42 -04:00
Marty Schoch
3e60ca24ec support using end key on forestdb iterator for term freq lookup 
also additoanl forestdb configs
 2015-08-18 16:22:02 -04:00
Marty Schoch
ae19d77b04 updated protobuf defs to be valid 2015-08-17 15:37:13 -04:00
Marty Schoch
1187436e46 changed Stored row Values to also use protobuf 2015-08-17 09:48:40 -04:00
Marty Schoch
8d8a05a842 fix more issues 2015-08-14 16:27:00 -04:00
Marty Schoch
e0802a2b39 fixed the worst of the formatting 2015-08-14 16:17:48 -04:00
Marty Schoch
f4df56eb7c add first draft of firestorm proposal 2015-08-14 16:09:19 -04:00
Marty Schoch
d3dda3d0ea fixup config parsing and add new options 2015-08-12 13:18:23 -04:00
Marty Schoch
01667dfff3 faster protobufs with gogo 2015-08-12 13:18:23 -04:00
Marty Schoch
7df66b4857 fix broken benchmark cause by index row encoding change 2015-08-06 14:48:04 -04:00
Marty Schoch
9db850a53e Merge branch 'fix/MaxVarintLen64' of https://github.com/tukdesk/bleve into tukdesk-fix/MaxVarintLen64 2015-07-31 15:16:16 -04:00
Marty Schoch
3682c25467 update to correctly work with composite fields 
also updated search results to return array positions
 2015-07-31 11:16:11 -04:00
Marty Schoch
c1c4941dde Merge branch 'feature/term_vector' of https://github.com/tukdesk/bleve into tukdesk-feature/term_vector 2015-07-29 14:31:15 -04:00
Marty Schoch
bf8dcae76b removing build tags 2015-07-28 18:59:10 -04:00
Marty Schoch
1b28f6218b additional row validation 2015-07-13 15:22:54 -04:00
Marty Schoch
17ef48f82a switching back to the canonical goleveldb repo 2015-07-08 12:21:17 -06:00
Marty Schoch
bf80f4628e fix bug in curent goleveldb (must copy during iteration) 
also changed over to mschoch fork of goleveldb (temporary)

the change to my fork is pending some read-only issues described
here:  https://github.com/syndtr/goleveldb/issues/111

hopefully we can find a path forward, and get that addressed upstream
 2015-07-06 18:00:05 -04:00
Marty Schoch
7be7ecdf8e fix batch indexing bug, incremented docCount before commit 
fixes #211
 2015-06-08 14:14:05 -04:00
Marty Schoch
2768c2da3c fix previous sloppy fix which hadn't been adequately tested 2015-05-27 19:15:55 -07:00
Marty Schoch
201fb91171 fix up to correctly trim off separator 
even though it should never be present
 2015-05-27 19:10:12 -07:00
Marty Schoch
a58592ceff fix case where NewBackIndexRowKV returns nil, nil 
the logic for reading the docID from the keys
in this row relies on the keys NEVER containing
the byte separator character (0xff), this is OK
as we require that all keys be valid utf-8
however, it turns out that in the case where this
rule was violated, we would panic, because we
return nil, nil and later try to print the doc id
 2015-05-27 19:04:57 -07:00
dtynn
59c97ae577 use binary.MaxVarintLen64 2015-05-26 15:35:31 +08:00
Marty Schoch
e0887f9113 fix tests which deadlock boltdb due to deferred cleanup 
fixes #209
 2015-05-21 12:29:31 -04:00
Marty Schoch
a52d3b5c07 put in hack to allow boltdb reader isolation test to pass 
in boltdb, long readers *MAY* block a writer.  in particular if
the write requires additional allocation, it must acquire a lock
already held by the reader.  in general this is not a problem
for bleve (though it can affect performance in some cases), but
it is a problem for the reader isolation test.  this commit
adds a hack to try and avoid the need for additional allocation
closes #208
 2015-05-21 11:39:59 -04:00
dtynn
b4f7496031 update the index format version number 2015-05-18 15:16:35 +08:00
dtynn
89dc2c22bc update TermVector 2015-05-17 13:07:14 +08:00
Marty Schoch
8f70def63b properly use the stored array positions when loading a document 
fixes #205
 2015-05-15 15:47:54 -04:00
Marty Schoch
328bc73ed0 clarify Batch is not threadsafe in docs 
in some limited cases we can detect unsafe usage
in these cases, do not trip over ourselves and panic
instead return a strongly typed error upside_down.UnsafeBatchUseDetected
also, introduced Batch.Reset() to allow batch reuse
this is currently still experimental
closes #195
 2015-05-15 15:04:52 -04:00
Marty Schoch
57cd67fa88 fix data race on index metadata (docCount) 
closes #198
 2015-05-08 08:07:20 -04:00
Marty Schoch
57358088ec fix row merging bug 
trying to be clever, we reused the memory allocated for the left
operand when doing partial merges
this had been tested to be safe, in general.  however, the
implementation was then written such that we always reused
globally defined operands, this meant that we mutated
the operands which were intended to always represent
+1/-1
this then cascades quickly to making increment/decrement
values much larger/smaller than they should be
related to #197
 2015-05-06 11:00:04 -04:00
Marty Schoch
30a0ba1f9b fix bug, dictionary row encoding buffer too small 
we incorrectly created a []byte of length 8
but the max for a uvarint is 10
closes #197
 2015-05-06 10:04:02 -04:00
Steve Yen
e98ae8ab71 update metrics store to latest kvstore api 2015-04-27 11:01:53 -07:00
Marty Schoch
16f538d7b7 close documents returned by iterator before losing their reference 
fixes #194
 2015-04-24 17:48:21 -04:00
Marty Schoch
b54a59139c change forestdb imports to couchbase not couchbaselabs 2015-04-24 17:35:01 -04:00
Marty Schoch
ee47d1c21a standardize on including 1000 sized batches 2015-04-24 17:31:34 -04:00
Marty Schoch
452fea6a24 adding initial impl of rocksdb kv store 2015-04-24 17:19:44 -04:00
Marty Schoch
a9c07acbfa refactor of kvstore api to support native merge in rocksdb 
refactor to share code in emulated batch
refactor to share code in emulated merge
refactor index kvstore benchmarks to share more code
refactor index kvstore benchmarks to be more repeatable
 2015-04-24 17:13:50 -04:00
indraniel
a62320a50e + fix goleveldb's BytesSafeAfterClose() on reader 
- it should be set to false
 2015-04-10 15:45:22 -05:00
Marty Schoch
d5dc66313f change variable name conflicting when both LevelDB bencharmks run 2015-04-10 15:03:44 -04:00
Marty Schoch
d5caad4405 changed GoLevelDB benchmark names to be different from LevelDB 
this will allow for easier comparision when running both
versions at the same time
 2015-04-10 15:00:56 -04:00
Marty Schoch
5f66bd84c7 fix issues identified by errcheck 2015-04-10 14:59:05 -04:00
indraniel
54ab493b3e + correctly copy bytes from the goleveldb store 
- this is part of a recent bleve KVStore API change.

    See the following two google group threads for more details:

    * [help adding goleveldb as an alternative Key/Value store for bleve][1]
    * [bleve search performance improvement][2]

    [1]: https://groups.google.com/forum/#!topic/bleve/aHZ8gmihLiY
    [2]: https://groups.google.com/forum/#!topic/bleve/aTyqsSnbhik
 2015-04-10 11:25:23 -05:00
indraniel
81bef38cce Revert "+ make copies of the []bytes returned by goleveldb" 
This reverts commit cb8c1741289a0f00b30733e0d52d9d81d1199603.

This commit is no longer desired. The KV store API has been changed to
better address this issue.

For more details, see the google group conversation thread at:

https://groups.google.com/forum/#!topic/bleve/aHZ8gmihLiY
 2015-04-10 11:12:44 -05:00
indraniel
3a70401835 + make copies of the []bytes returned by goleveldb 
- The byte strings returned by goleveldb aren't necessarily safe.  See
    the following google group thread:

    https://groups.google.com/forum/#!topic/bleve/aHZ8gmihLiY

    This code change is based on the gist created here:

    https://groups.google.com/forum/#!topic/bleve/aHZ8gmihLiY
 2015-04-10 11:08:02 -05:00
indraniel
a88d714778 + add a goleveldb index updside-down benchmark test 2015-04-10 11:08:02 -05:00
indraniel
a0a2a61050 + keep 'get' consistent with levigo implementation 
- this change keeps the method behavior consistent with the
     levigo/leveldb implementation.

   - don't issue an err if a key isn't found
 2015-04-10 11:08:02 -05:00
indraniel
5e55fa2866 + keep 'getWithSnapshot' consistent with levigo implementation 
- this change keeps the method behavior consistent with the
     levigo/leveldb implementation.

   - the leveldb store_test.go and goleveldb store_test.go are now
     identical.
 2015-04-10 11:08:02 -05:00
indraniel
caa19e6c36 + initial stub of goleveldb package 
- This is a first-pass introduction. Things may not be working
    correctly yet.
 2015-04-10 11:08:02 -05:00
Marty Schoch
8581e73cef added String method for Batch 
also changed Batch methods to pointer receiver
closes #180
 2015-04-08 10:41:42 -04:00
Marty Schoch
539aeb8dc7 fix errors identified by errcheck 
part of #169
 2015-04-07 18:05:41 -04:00
Marty Schoch
ba6b3c8bb3 fix more issues identified by errcheck 
part of #169
 2015-04-07 16:45:23 -04:00
Marty Schoch
ab24772bf0 fix issues identified by errcheck 
part of #169
 2015-04-07 16:34:29 -04:00
Marty Schoch
56c4a09de1 fix issues identified by errcheck 
part of #169
 2015-04-07 15:39:56 -04:00
Marty Schoch
93e01a803e fix issues identified by errcheck 
part of #169
 2015-04-07 14:52:00 -04:00
Marty Schoch
f1ec73e764 fix issues identified by errcheck 
part of #169
 2015-04-07 13:26:54 -04:00
Marty Schoch
56a30a3574 fix issues identified by errcheck 
part of #169
 2015-04-07 13:05:47 -04:00
Marty Schoch
d2e9409413 fix issues identified by errcheck 
part of #169
 2015-04-07 12:04:59 -04:00