cpd-tokenizer-0.1.2
- cpd-tokenizer 0.1.2
- Docs.rs crate page
- MIT
- Links
- crates.io
- Source
- Owners
- kucherenko
- Dependencies
- cpd-core ^0.1.2 normal
  log ^0.4 normal
  oxc_allocator ^0.133 normal
  oxc_parser ^0.133 normal
  oxc_span ^0.133 normal
  xxhash-rust ^0.8 normal
  serde_json ^1 dev
- Versions
- 27.78% of the crate is documented
Go to latest version
Platform
- x86_64-unknown-linux-gnu
Feature flags

docs.rs

Rust

Skip to main content

Module generic

cpd_tokenizer0.1.2

Module generic

Module Items

Functions

In crate cpd_tokenizer

Module generic

Functions§

tokenize_generic: Tokenize source in the given format. Never panics on empty input.