1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
//! The input-side counterpart to [`FeatureFrom`](crate::traits::FeatureFrom).
//!
//! Every featurizer in this crate ultimately consumes a contiguous slice
//! `&[Token]`. [`AsTokens`] names exactly that requirement — "this type can be
//! viewed as a slice of tokens" — so a single blanket impl per featurizer
//! covers `&str`, `&String`, `&[T]`, `&Vec<T>`, and `&[T; N]` at once, instead
//! of one hand-written (or macro-generated) forwarding impl per container type.
//!
//! # Extending featurization to your own types
//! Implementing `AsTokens` for a local type makes it featurizable by *every*
//! featurizer and combinator in the crate, present and future:
//! ```
//! use creature_feature::traits::{AsTokens, Ftzr};
//! use creature_feature::ftzrs::bislice;
//!
//! struct Dna(Vec<u8>);
//! impl AsTokens for Dna {
//! type Token = u8;
//! fn as_tokens(&self) -> &[u8] { &self.0 }
//! }
//!
//! let dna = Dna(vec![b'A', b'C', b'G', b'T']);
//! let feats: Vec<&[u8]> = bislice().featurize(&dna);
//! assert_eq!(feats.len(), 3);
//! ```
//!
//! # Caveat
//! Because the featurizer impls are blanket over `D: AsTokens`, a type that
//! implements `AsTokens` can no longer carry a *bespoke* `IterFtzr<&YourType>`
//! impl for a given featurizer — the blanket already covers it (this is the
//! usual coherence trade-off, not a limitation specific to this crate).
/// A type that can be viewed as a contiguous slice of tokens.
///
/// This is the single trait that dispatches every featurizer across all
/// supported input containers. See the [module docs](self) for the rationale
/// and for how to extend featurization to your own types.