ipfrs-semantic-0.2.0
- ipfrs-semantic 0.2.0
- Docs.rs crate page
- Apache-2.0
- 16 June 2026
- Links
- Homepage
- Repository
- crates.io
- Source
- Owners
- cool-japan
- Dependencies
- anyhow ^1.0 normal
  async-trait ^0.1 normal
  bytes ^1.11 normal
  cid ^0.11 normal
  dashmap ^6.2 normal
  futures ^0.3 normal
  hnsw_rs ^0.3 normal
  ipfrs-core ^0.2.0 normal
  ipfrs-network ^0.2.0 normal
  ipfrs-storage ^0.2.0 normal
  ipfrs-tensorlogic ^0.2.0 normal
  lru ^0.18 normal
  memmap2 ^0.9 normal
  multihash ^0.19 normal
  nalgebra ^0.35 normal
  oxicode ^0.2.4 normal
  parking_lot ^0.12 normal
  rand ^0.10 normal
  rayon ^1.12 normal
  serde ^1.0 normal
  serde_json ^1.0 normal
  thiserror ^2.0 normal
  tokio ^1.52 normal
  tracing ^0.1 normal
  uuid ^1.23 normal
  criterion ^0.8 dev
  libp2p ^0.56 dev
  multihash-codetable ^0.2 dev
  rand ^0.10 dev
- Versions
- 94.02% of the crate is documented
Platform
- x86_64-unknown-linux-gnu
Feature flags

docs.rs

Rust

Skip to main content

tokenize

ipfrs_semantic0.2.0

In ipfrs_semantic::document_summarizer

ipfrs_semantic::document_summarizer

Function tokenize

pub fn tokenize(text: &str) -> Vec<String>

Expand description

Tokenize text into lowercase alphanumeric tokens.