3 Commits

Author	SHA1	Message	Date
Silvan Jegen	29bdc094a9	Use byte positions instead of character positions	2014-09-14 13:19:30 +02:00
Silvan Jegen	a8ec7f7af2	Add tests for the Kagome tokenizer	2014-09-13 17:45:22 +02:00
Silvan Jegen	ebf100c097	Add the Kagome tokenizer for Japanese	2014-09-13 17:45:19 +02:00