unobtanium-segmenter 0.5.2

A text segmentation toolbox for search applications inspired by charabia and tantivy.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

// SPDX-FileCopyrightText: 2026 Slatian
//
// SPDX-License-Identifier: LGPL-3.0-only

use crate::SegmentedToken;
use crate::UseOrSubdivide;

/// Allows the segmenter to be part of a [SubdivisionMap][crate::SubdivisionMap] iterator and be chained using the [chain_segmenter][crate::chain::ChainSegmenter::chain_segmenter] methods.
pub trait Segmenter {
	/// The iterator type returned by the `subdivide` function if it has multiple results.
	///
	/// Setting this to `IntoIter<SegmentedToken<'a>>` for [std::vec::IntoIter] is a sane default. Allowing you to generate a valid iterator calling `into_iter()` on a Vec.
	type SubdivisionIter<'a>: Iterator<Item = SegmentedToken<'a>>;

	/// A method that should split the given `token` into zero, one or more subtokens.
	///
	/// Have a look at the [UseOrSubdivide] enum, which is similar to an [Option],
	/// but can also return an owned iterator with multiple items in it.
	fn subdivide<'a>(
		&self,
		token: SegmentedToken<'a>,
	) -> UseOrSubdivide<SegmentedToken<'a>, Self::SubdivisionIter<'a>>;
}