cpd-tokenizer 0.1.5

Source code tokenizers for cpd

Coverage
31.58%
18 out of 57 items documented0 out of 28 items with examples
Size
Source code size: 135.26 kB This is the summed size of all the files inside the crates.io package for this release.
Documentation size: 1.78 MB This is the summed size of all files generated by rustdoc for all configured targets
Ø build duration
this release: 18s Average build duration of successful builds.
all releases: 19s Average build duration of successful builds in releases after 2024-10-23.
Links
crates.io
Dependencies
Versions
Owners

cpd-tokenizer

Source code tokenizers for cpd — a fast copy/paste detector for code.

This crate provides language-aware tokenization for 200+ programming languages, producing token streams suitable for duplicate detection. It includes:

Generic tokenizer (comments, strings, keywords, punctuation)
JavaScript/TypeScript/JSX/TSX tokenizer (Oxc-based)
Markdown tokenizer (code fences, front matter, embedded languages)
SFC tokenizer (Vue, Svelte, Astro)
Embedded language detection and cross-format tokenization

This crate is not intended to be used directly; see the jscpd crate for the full CLI.

License

MIT