gibheer
/
bleve
0
0
Fork 0
Code

scorch zap file merger uses 1MB buffered writer 

pprof of bleve-blast was showing file merging was in syscall/write a
lot.  The bufio.NewWriter() provides a default buffer size of 4K,
which is too small, and using bufio.NewWriterSize(1MB buffer size)
leads to syscall/write dropping out of the file merging flame graphs.
This commit is contained in:
Steve Yen 2018-03-16 11:49:50 -07:00
parent 3d1cea0556
commit 5df53c8e1f
1 changed files with 3 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
index/scorch/segment/zap/merge.go
View File

@ -29,6 +29,8 @@ import (
"github.com/golang/snappy"
)
var DefaultFileMergerBufferSize = 1024 * 1024
const docDropped = math.MaxUint64 // sentinel docNum to represent a deleted doc
// Merge takes a slice of zap segments and bit masks describing which
@ -55,7 +57,7 @@ func Merge(segments []*Segment, drops []*roaring.Bitmap, path string,
}
// buffer the output
br := bufio.NewWriter(f)
br := bufio.NewWriterSize(f, DefaultFileMergerBufferSize)
// wrap it for counting (tracking offsets)
cr := NewCountHashWriter(br)