Taking another optimization from firestorm, upside_down's
storeField()/indexField() funcs now also append() to passed-in arrays
rather than always allocating their own arrays.
Rather than append() all received rows into a flat []IndexRow during
the result gathering loop, this change instead collects the analysis
result rows into a [][]IndexRow, which avoids extra copying.
As part of this, firestorm batchRows() now takes the [][]IndexRow as
its input.
The new analyzeField() helper func is used for both regular fields and
for composite fields.
With this change, all analysis is done up front, for both regular
fields and composite fields.
After analysis, this change counts up all the row capacity needed and
extends the AnalysisResult.Rows in one shot, as opposed to the
previous approach of dynamically growing the array as needed during
append()'s.
Also, in this change, the TermFreqRow for _id is added first, which
seems more correct.
This uses the "backing array" technique to allocate many TermFreqRow's
at the front of firestorm.indexField(), instead of the previous
one-by-one, as-needed TermFreqRow allocation approach.
Results from micro-benchmark, null-firestorm, bleve-blast has this
change producing a ~half MB/sec improvement.
Previously, the firestorm.Batch() would notify the lookuper goroutine
on a document by document basis. If the lookuper input channel became
full, then that would block the firestorm.Batch() operation.
With this change, lookuper is notified once, with a "batch" that is an
[]*InFlightItem.
This change also reuses that same []*InFlightItem to invoke the
compensator.MutateBatch().
This also has the advantage of only converting the docID's from string
to []byte just once, outside of the lock that's used by the
compensator.
Micro-benchmark of this change with null-firestorm bleve-blast does
not show large impact, neither degradation or improvement.
After this change, with null kvstore micro-benchmark...
GOMAXPROCS=8 ./bleve-blast -source=../../tmp/enwiki.txt \
-count=100000 -numAnalyzers=8 -numIndexers=8 \
-config=../../configs/null-firestorm.json -batch=100
Then TermFreqRow key and value methods dissapear as large boxes from
the cpu profile graphs.
The field cache is expected to be the authority on which field
names are identified by which identifier. This code was
optimized for the most common case in which fields already
exist. However, if we deterimine the field is missing with
the read lock (shared), we incorrectly immediately proceed
to create a new row with the write lock (exclusive). The
problem is that multiple goroutines might have come to
the same conclusion, and they all proceed to add rows. The two
choices were to do the whole operation with the write lock, or
recheck the value again with the write lock. We have chosen
to repeat the check inside the write-lock, as this optimizes
for what we believe to be the most common case, in which most
fields will already exist.
Rows content is an implementation detail of bleve index and may change
in the future. That said, they also contains information valuable to
assess the quality of the index or understand its performances. So, as
long as we agree that type asserting rows should only be done if you
know what you are doing and are ready to deal with future changes, I see
no reason to hide the row fields from external packages.
Fix#268
