unitoken 0.1.7

Deprecated: use ffbpe for Unicode-aware, streaming BPE training and encoding
Documentation 
1
2
3
4
5
6
7

fn main() {
  println!("cargo:rerun-if-changed=build.rs");
  println!(
    "cargo:warning=unitoken has moved to ffbpe and is no longer maintained; \
     migrate with `cargo add ffbpe`"
  );
}