vtext 0.2.0

NLP with Rust
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

.. vtext documentation master file, created by
   sphinx-quickstart on Sun Apr 14 23:28:11 2019.
   You can adapt this file completely to your liking, but it should at least
   contain the root `toctree` directive.

vtext
=====

NLP in Rust with Python bindings

This package aims to provide a high performance toolkit for ingesting textual data for
machine learning applications.

Features
--------

 - Tokenization: Regexp tokenizer, Unicode segmentation + language specific rules
 - Stemming: Snowball (in Python 15-20x faster than NLTK)
 - Token counting: converting token counts to sparse matrices for use
   in machine learning libraries. Similar to `CountVectorizer` and
   `HashingVectorizer` in scikit-learn.
 - Levenshtein edit distance; Sørensen-Dice, Jaro, Jaro Winkler string similarities

.. toctree::
   :maxdepth: 2
   :caption: Contents

   install
   benchmarks
   rust-api
   python-api
   contributing