Marty Schoch
b5aa4ed22b
return err not panic
2017-12-14 17:41:02 -05:00
Steve Yen
506aa1c325
scorch fix data race w/ AddEligibleForRemoval
...
Found from "go test -race ./..."
WARNING: DATA RACE
Read at 0x00c420088060 by goroutine 48:
github.com/blevesearch/bleve/index/scorch.(*Scorch).AddEligibleForRemoval()
/Users/steveyen/go/src/github.com/blevesearch/bleve/index/scorch/scorch.go:348 +0x6d
Previous write at 0x00c420088060 by goroutine 31:
github.com/blevesearch/bleve/index/scorch.(*Scorch).loadFromBolt.func1()
/Users/steveyen/go/src/github.com/blevesearch/bleve/index/scorch/persister.go:332 +0x87b
github.com/boltdb/bolt.(*DB).View()
/Users/steveyen/go/src/github.com/boltdb/bolt/db.go:629 +0xc1
github.com/blevesearch/bleve/index/scorch.(*Scorch).loadFromBolt()
/Users/steveyen/go/src/github.com/blevesearch/bleve/index/scorch/persister.go:290 +0xa1
github.com/blevesearch/bleve/index/scorch.(*Scorch).Open()
/Users/steveyen/go/src/github.com/blevesearch/bleve/index/scorch/scorch.go:121 +0x77f
github.com/blevesearch/bleve/index/scorch.TestIndexOpenReopen()
/Users/steveyen/go/src/github.com/blevesearch/bleve/index/scorch/scorch_test.go:115 +0x1351
testing.tRunner()
/usr/local/Cellar/go/1.9/libexec/src/testing/testing.go:746 +0x16c
Goroutine 48 (running) created at:
github.com/blevesearch/bleve/index/scorch.(*IndexSnapshot).DecRef()
/Users/steveyen/go/src/github.com/blevesearch/bleve/index/scorch/snapshot_index.go:72 +0x23e
github.com/blevesearch/bleve/index/scorch.(*Scorch).loadFromBolt.func1()
/Users/steveyen/go/src/github.com/blevesearch/bleve/index/scorch/persister.go:330 +0x8f4
github.com/boltdb/bolt.(*DB).View()
/Users/steveyen/go/src/github.com/boltdb/bolt/db.go:629 +0xc1
github.com/blevesearch/bleve/index/scorch.(*Scorch).loadFromBolt()
/Users/steveyen/go/src/github.com/blevesearch/bleve/index/scorch/persister.go:290 +0xa1
github.com/blevesearch/bleve/index/scorch.(*Scorch).Open()
/Users/steveyen/go/src/github.com/blevesearch/bleve/index/scorch/scorch.go:121 +0x77f
github.com/blevesearch/bleve/index/scorch.TestIndexOpenReopen()
/Users/steveyen/go/src/github.com/blevesearch/bleve/index/scorch/scorch_test.go:115 +0x1351
testing.tRunner()
/usr/local/Cellar/go/1.9/libexec/src/testing/testing.go:746 +0x16c
2017-12-14 14:40:33 -08:00
Marty Schoch
6ab27e4afa
quick hack to disable safe batches in fts
2017-12-14 17:19:50 -05:00
Marty Schoch
eb4ef2278e
Merge pull request #656 from steveyen/scorch
...
scorch fix for TestSortMatchSearch
2017-12-14 17:04:54 -05:00
Steve Yen
eb2f541d4f
scorch filters _id from Reader.Document() results
2017-12-14 13:52:28 -08:00
Steve Yen
a8884e1011
scorch fix for TestSortMatchSearch
...
The cachedDocs preparation has to happen for all docs in the field,
not just on the currently requested docNum.
Also, as part of this commit, there's a loop optimization where we no
longer use bytes.Split() on the terms buffer, thus avoiding garbage
creation.
2017-12-14 13:22:13 -08:00
Marty Schoch
5bca9f06b9
Merge pull request #655 from steveyen/scorch
...
scorch tracks zap files that can't be removed yet
2017-12-14 14:24:56 -05:00
Steve Yen
2be5eb4427
scorch tracks zap files that can't be removed yet
...
A race & solution found by Marty Schoch... consider a case when the
merger might grab a nextSegmentID, like 4, but takes awhile to
complete. Meanwhile, the persister grabs the nextSegmentID of 5, but
finishes its persistence work fast, and then loops to cleanup any old
files. The simple approach of checking a "highest segment ID" of 5 is
wrong now, because the deleter now thinks that segment 4's zap file is
(incorrectly) ok to delete.
The solution in this commit is to track an ephemeral map of filenames
which are ineligibleForRemoval, because they're still being written
(by the merger) and haven't been fully incorporated into the rootBolt
yet.
The merger adds to that ineligibleForRemoval map as it starts a merged
zap file, the persister cleans up entries from that map when it
persists zap filenames into the rootBolt, and the deleter (part of the
persister's loop) consults the map before performing any actual zap
file deletions.
2017-12-14 10:49:33 -08:00
Marty Schoch
bd742caf65
don't try to close a nil segment if err opening
2017-12-14 10:29:19 -05:00
Marty Schoch
149a26b5c1
merge deletion and cacheddocs fixes discussed in meeting
2017-12-14 10:27:39 -05:00
Marty Schoch
a79b450e0c
Merge branch 'sreekanth-cb-scorch_visit_fts' into scorch
2017-12-14 10:24:50 -05:00
Marty Schoch
364a1877fb
Merge branch 'scorch_visit_fts' of https://github.com/sreekanth-cb/bleve into sreekanth-cb-scorch_visit_fts
2017-12-14 08:17:24 -05:00
Marty Schoch
a4acd53c54
try to make racey test safe
2017-12-14 07:53:26 -05:00
Sreekanth Sivasankaran
95b65ade3e
getting right internalID for doc in UT
2017-12-14 17:16:47 +05:30
Sreekanth Sivasankaran
1066ee7d22
DocumentVisitFieldTerms Scorch implementation level1
2017-12-14 12:38:29 +05:30
Marty Schoch
2b92e5ff99
Merge pull request #653 from steveyen/scorch
...
scorch cleanup of the rootBolt of old snapshots
2017-12-13 22:47:14 -05:00
Marty Schoch
e1b0c61e2a
fix bug in handling iterator-done
2017-12-13 22:08:06 -05:00
Steve Yen
b7dff6669f
scorch cleanup of *.zap files not listed in the rootBolt
2017-12-13 17:09:50 -08:00
Steve Yen
c0cc46a2be
scorch cleanup of the rootBolt of old snapshots
...
A new global variable, NumSnapshotsToKeep, represents the default
number of old snapshots that each scorch instance should maintain -- 0
is the default. Apps that need rollback'ability may want to increase
this value in early initialization.
The Scorch.eligibleForRemoval field tracks epoches which are safe to
delete from the rootBolt. The eligibleForRemoval is appended to
whenever the ref-count on an IndexSnapshot drops to 0.
On startup, eligibleForRemoval is also initialized with any older
epoch's found in the rootBolt.
The newly introduced Scorch.removeOldSnapshots() method is called on
every cycle of the persisterLoop(), where it maintains the
eligibleForRemoval slice to under a size defined by the
NumSnapshotsToKeep.
A future commit will remove actual storage files in order to match the
"source of truth" information found in the rootBolt.
2017-12-13 15:53:31 -08:00
Marty Schoch
def77cb548
Merge pull request #652 from damz/pr/closest-doc-mapping
...
mapping: Fix closestDocMapping selecting wrong mapping
2017-12-13 18:27:55 -05:00
Damien Tournoud
74b882640e
mapping: Fix closestDocMapping selecting wrong mapping
2017-12-13 15:19:21 -08:00
Marty Schoch
8ffa978ce4
Merge pull request #651 from steveyen/scorch
...
scorch ref-counting
2017-12-13 18:01:13 -05:00
Steve Yen
c13ff85aaf
scorch ref-counting
...
Future commits will provide actual cleanup when ref-counts reach 0.
2017-12-13 14:48:07 -08:00
Marty Schoch
50471003dc
basic refactoring of introducer to make it more readable
2017-12-13 16:30:39 -05:00
Marty Schoch
a0e12b2640
add license to a few files missing it
2017-12-13 16:12:29 -05:00
Marty Schoch
85e15628ee
major refactoring of posting details
2017-12-13 16:10:06 -05:00
Marty Schoch
6e2207c445
additional refactoring of build/merge
2017-12-13 15:22:13 -05:00
Marty Schoch
50441e5065
refactor to reuse shared code
2017-12-13 14:41:20 -05:00
Marty Schoch
289dc398bd
more refacotring of build/merge
2017-12-13 14:26:11 -05:00
Marty Schoch
1cd3fd7fbe
extrac common functionality between build/merge
2017-12-13 14:06:54 -05:00
Marty Schoch
cd45487cb3
fsync rootBolt when persisting snapshot
2017-12-13 13:55:06 -05:00
Marty Schoch
f83c9f2a20
initial cut of merger that actually introduces changes
2017-12-13 13:41:03 -05:00
Marty Schoch
c15c3c11cd
extra protection if dict address is 0 (empty segment)
2017-12-13 13:31:18 -05:00
Marty Schoch
a681314740
Merge pull request #649 from steveyen/scorch
...
mergeplan: scoring implemented
2017-12-13 07:30:28 -05:00
Steve Yen
be7dd36ac6
mergeplan: more tests and bargraph tweaks
2017-12-12 10:37:27 -08:00
Steve Yen
59a1e26300
mergeplan: scoring implemented
2017-12-12 10:37:27 -08:00
Marty Schoch
57121e40a8
fix issues identified by errcheck
2017-12-12 11:41:14 -05:00
Marty Schoch
665c3c80ff
initial cut of zap segment merging
2017-12-12 11:21:55 -05:00
Marty Schoch
927216df8c
fix postings list count impl
2017-12-12 08:42:13 -05:00
Marty Schoch
3873237e46
try newer version of bolt (seeing random crashes on travis)
2017-12-11 22:09:26 -05:00
Marty Schoch
8fbf0f271b
Merge pull request #648 from steveyen/scorch
...
merge_plan: a placeholder planner that merges everything
2017-12-11 18:08:19 -05:00
Steve Yen
3461fb741f
mergeplan: a placeholder planner that merges all segments
...
A stepping stone to fleshing out the API contract.
2017-12-11 14:53:08 -08:00
Marty Schoch
58ef21a88a
fix golint issue
2017-12-11 16:24:46 -05:00
Marty Schoch
f246e0e4c0
update README for zap file format changes
2017-12-11 16:22:29 -05:00
Marty Schoch
74b2eeb14d
refactor where we do some work so we can return error
2017-12-11 15:59:36 -05:00
Marty Schoch
f13b786609
fix up issues to get all bleve unit tests passing for scorch
...
make scorch default
2017-12-11 15:47:41 -05:00
Marty Schoch
00722aa299
disable http unit test which relied on debug functionality
2017-12-11 15:38:44 -05:00
Marty Schoch
d7eb223e14
remove bolt segment format
...
upcomning breaking changes and no desire to maintain
2017-12-11 10:20:26 -05:00
Marty Schoch
eada7b209b
fix test issue identified by sreekanth
2017-12-11 10:16:56 -05:00
Marty Schoch
8280859bb8
handle read-only and in-mem only cases
2017-12-11 09:07:01 -05:00