Struct rust_tokenizers::TokenIdsWithSpecialTokens [−][src]
Encoded input with special tokens
Intermediate tokenization steps before truncation to a maximum length, after encoding and addition of special tokens
Fields
token_ids: Vec<i64>
Vector of token IDs
segment_ids: Vec<i8>
Vector segments ids (for example for BERT segments are separated with a [SEP] marker, each incrementing the segment ID). This vector has the same length as token_ids.
special_tokens_mask: Vec<i8>
Flags tokens as special tokens (1) or not (0). This vector has the same length as token_ids.
token_offsets: Vec<Option<Offset>>
Offset information (as start and end positions) in relation to the original text. Tokens that can not be related to the original source are registered as None.
reference_offsets: Vec<Vec<OffsetSize>>
Offset information (as a sequence of positions) in relation to the original text. Tokens that can not be related to the original source are registered as None.
mask: Vec<Mask>
Masks tokens providing information on the type of tokens. This vector has the same length as token_ids.
Trait Implementations
impl Clone for TokenIdsWithSpecialTokens
[src]
fn clone(&self) -> TokenIdsWithSpecialTokens
[src]
pub fn clone_from(&mut self, source: &Self)
1.0.0[src]
impl Debug for TokenIdsWithSpecialTokens
[src]
Auto Trait Implementations
impl RefUnwindSafe for TokenIdsWithSpecialTokens
impl Send for TokenIdsWithSpecialTokens
impl Sync for TokenIdsWithSpecialTokens
impl Unpin for TokenIdsWithSpecialTokens
impl UnwindSafe for TokenIdsWithSpecialTokens
Blanket Implementations
impl<T> Any for T where
T: 'static + ?Sized,
[src]
T: 'static + ?Sized,
impl<T> Borrow<T> for T where
T: ?Sized,
[src]
T: ?Sized,
impl<T> BorrowMut<T> for T where
T: ?Sized,
[src]
T: ?Sized,
pub fn borrow_mut(&mut self) -> &mut T
[src]
impl<T> From<T> for T
[src]
impl<T, U> Into<U> for T where
U: From<T>,
[src]
U: From<T>,
impl<T> Pointable for T
pub const ALIGN: usize
type Init = T
The type for initializers.
pub unsafe fn init(init: <T as Pointable>::Init) -> usize
pub unsafe fn deref<'a>(ptr: usize) -> &'a T
pub unsafe fn deref_mut<'a>(ptr: usize) -> &'a mut T
pub unsafe fn drop(ptr: usize)
impl<T> ToOwned for T where
T: Clone,
[src]
T: Clone,
type Owned = T
The resulting type after obtaining ownership.
pub fn to_owned(&self) -> T
[src]
pub fn clone_into(&self, target: &mut T)
[src]
impl<T, U> TryFrom<U> for T where
U: Into<T>,
[src]
U: Into<T>,
type Error = Infallible
The type returned in the event of a conversion error.
pub fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>
[src]
impl<T, U> TryInto<U> for T where
U: TryFrom<T>,
[src]
U: TryFrom<T>,