cpd-tokenizer-0.1.5
- cpd-tokenizer 0.1.5
- Permalink
- Docs.rs crate page
- MIT
- Links
- crates.io
- Source
- Owners
- kucherenko
- Dependencies
- cpd-core ^0.1.4 normal
  log ^0.4 normal
  oxc_allocator ^0.133 normal
  oxc_parser ^0.133 normal
  oxc_span ^0.133 normal
  regex ^1 normal
  xxhash-rust ^0.8 normal
  serde_json ^1 dev
- Versions
- 31.58% of the crate is documented
Platform
- x86_64-unknown-linux-gnu
Feature flags

docs.rs

Rust

Skip to main content

tokenize_markdown

cpd_tokenizer0.1.5

In cpd_tokenizer::markdown

cpd_tokenizer::markdown

Function tokenize_markdown

pub fn tokenize_markdown(source: &str, mode: Mode) -> Vec<Token>