This is attempt #2 of the optimization that replaces the locsBitmap,
without any changes from the original commit attempt. A commit that
follows this one contains the actual fix.
See also...
- commit 621b58dd83 (the 1st attempt)
- commit 49a4ee60ba (the revert)
-------------
The original commit message body from 621b58 was...
NOTE: this is a zap file format change.
The separate "postings locations" roaring Bitmap that encoded whether
a posting has locations info is now replaced by the least significant
bit in the freq varint encoded in the freq-norm chunkedIntCoder.
encode/decodeFreqHasLocs() are added as helper functions.
Testing with the cbft application led to cbft process exits...
AsyncError exit()... error reading location field: EOF --
main.initBleveOptions.func1() at init_bleve.go:85
This reverts commit 621b58dd83.
NOTE: this is a zap file format change.
The separate "postings locations" roaring Bitmap that encoded whether
a posting has locations info is now replaced by the least significant
bit in the freq varint encoded in the freq-norm chunkedIntCoder.
encode/decodeFreqHasLocs() are added as helper functions.
correctly handle/print additional loc bitmap address
this fixes bitmap length that is output
instantiate roaring bitmap and print it out
removed some unnecessary debug logging
updated dict command to print 1-hit encoded vals
this makes dict command usable for seeing which
doc ids are in a segment and their corresponding doc number
merging the doc value length and loc
slices into a single offset slice as that
is enough to compute the starting offset and
length of the the doc values data for a given
document inside a docValue chunk.
NOTE: this is a scorch zap file format change / bump to version 4.
In this optimization, the uint64 val stored in the vellum FST (term
dictionary) now may either be a uint64 postingsOffset (same as before
this change) or a uint64 encoding of the docNum + norm (in the case
where a term appears in just a single doc).
zap command-line tool added to main bleve command-line tool
this required physical relocation due to the vendoring used
only on the bleve command-line tool (unforseen limitation)
a new scorch command-line tool has also been introduced
and for the same reasons it is physically store under
the top-level bleve command-line tool as well
some downstream applications need to know which sub-commands
may alter the index. this new function allows them to be
identified, while not prescribing any particular behavior.
our policy has been to vendor the libraries required for the
command-line portion of the bleve command-line tool. this
included spf13/cobra etc, unfortunately we missed this library
as it is only required for windows users.
bleve check was a consistency checking tool originally developed
as a part of cbft. currently it checks that the term dictionary
counts match the number of postings for the term. in the future
additional checks could be added. this tool has been back
ported to bleve as we've now adopted a single common tool for
both cbft and bleve.
Steve Yen