tokenizers 0.10.1

Provides an implementation of today's most used tokenizers, with a focus on performances and versatility.
Documentation 
{
  "git": {
    "sha1": "af66d6fc6f14153b21fbccd1774e00adfa05742a"
  }
}