tokenizers 0.19.0

Provides an implementation of today's most used tokenizers, with a focus on performances and versatility.
Documentation 
{
  "git": {
    "sha1": "e59020d642b48148516090c04abe9545510e28b3"
  },
  "path_in_vcs": "tokenizers"
}