spg-engine-7.37.21
- spg-engine 7.37.21
- Docs.rs crate page
- MIT OR Apache-2.0
- 14 August 2026
- Links
- Repository
- crates.io
- Source
- Owners
- doracawl
- Dependencies
- bumpalo ^3 normal
  hashbrown ^0.15 normal
  hmac ^0.12 normal
  icu_collator ^2 normal
  icu_locale_core ^2 normal
  libm ^0.2 normal
  md-5 ^0.10 normal
  sha1 ^0.10 normal
  sha2 ^0.10 normal
  spg-crypto ^7.34 normal
  spg-sql ^7.34 normal
  spg-storage ^7.34 normal
  unicode-normalization ^0.1 normal
  criterion ^0.5 dev
  spg-tzif ^7.34 dev
- Versions
- 61.55% of the crate is documented
Go to latest version
Platform
- x86_64-unknown-linux-gnu
Feature flags

docs.rs

Rust

Skip to main content

tokenize

spg_engine7.37.21

In spg_engine::fts

spg_engine::fts

Function tokenize

pub fn tokenize(text: &str) -> Vec<String>

Expand description

The old tokenizer, kept for callers that want bare words.