1dc466a800
often the result stream was the same length, so can reuse the existing token stream also, in cases where a new stream was required, set capacity to the length of the input stream. most output stream are at least as long as the input, so this may avoid some subsequent resizing |
||
---|---|---|
.. | ||
cld2_filter_test.go | ||
cld2_filter.cc | ||
cld2_filter.go | ||
cld2_filter.h | ||
compile_cld2.sh | ||
README.md |
cld2 token filter
A bleve token filter which passes the text of each token and passes it to the cld2 library. The library determines what it thinks the language most likely is. The ISO-639 language code replaces the token term.
In normal usage, you use this with the "single" tokenizer, so there is only one input token. Further, you should precede it with the "to_lower" filter so that the input term is in all lower-case unicode characters.
Building
-
Acquire the source to cld2 in this directory.
$ svn checkout -r 167 http://cld2.googlecode.com/svn/trunk/ cld2-read-only
-
Build cld2
As dynamic library
$ cd cld2-read-only/internal/ $ ./compile_libs.sh $ cp *.so /usr/local/lib $ cd ../..
Or static library
$ ./compile_cld2.sh $ cp *.a /usr/local/lib
-
Run the unit tests
$ go test -v === RUN TestCld2Filter --- PASS: TestCld2Filter (0.00 seconds) PASS ok github.com/couchbaselabs/bleve/analysis/token_filters/cld2 0.033s