Steve Yen 918732f3d8 unicode.Tokenize() allocs backing array of Tokens ... Previously, unicode.Tokenize() would allocate a Token one-by-one, on an as-needed basis. This change allocates a "backing array" of Tokens, so that it goes to the runtime object allocator much less often. It takes a heuristic guess as to the backing array size by using the average token (segment) length seen so far. Results from micro-benchmark (null-firestorm, bleve-blast) seem to give perhaps less than ~0.5 MB/second throughput improvement.		2016-01-02 12:21:25 -08:00
..
unicode_test.go	add test to cover kana/ideographic case	2014-11-26 08:42:40 -05:00
unicode.go	unicode.Tokenize() allocs backing array of Tokens	2016-01-02 12:21:25 -08:00