Struct Node 

pub struct Node<T: Chunk> {
    pub path: PathBuf,
    pub chunk: T,
    pub vectors: Option<HashMap<EmbeddedField, Embedding>>,
    pub sparse_vectors: Option<HashMap<EmbeddedField, SparseEmbedding>>,
    pub metadata: Metadata,
    pub embed_mode: EmbedMode,
    pub original_size: usize,
    pub offset: usize,
    pub parent_id: Option<Uuid>,
}

Represents a unit of data in the indexing process.

Node encapsulates all necessary information for a single unit of data being processed in the indexing pipeline. It includes fields for an identifier, file path, data chunk, optional vector representation, and metadata.

Fields

path: PathBuf

File path associated with the node.

chunk: T

Data chunk contained in the node.

vectors: Option<HashMap<EmbeddedField, Embedding>>

Optional vector representation of embedded data.

sparse_vectors: Option<HashMap<EmbeddedField, SparseEmbedding>>

Optional sparse vector representation of embedded data.

metadata: Metadata

Metadata associated with the node.

embed_mode: EmbedMode

Mode of embedding data Chunk and Metadata

original_size: usize

Size of the input this node was originally derived from in bytes

offset: usize

Offset of the chunk relative to the start of the input this node was originally derived from in bytes

parent_id: Option<Uuid>

Id of the first node this node was derived from. Set when a node is created via Node::build_from_other (chunking, conversions) so derived nodes keep pointing at the node that entered the pipeline.

Implementations

impl<T: Chunk> Node<T>

pub fn build_from_other(node: &Node<T>) -> NodeBuilder<T>

Builds a new instance of Node, returning a NodeBuilder. Copies over the fields from the provided Node.

The new node points at the node that entered the pipeline via Node::parent_id.

pub fn builder<VALUE: Chunk + Clone>() -> NodeBuilder<VALUE>

Creates a new instance of NodeBuilder.

pub fn new(chunk: impl Into<String>) -> Node<String>

Creates a new instance of Node with the specified data chunk.

The other fields are set to their default values.

pub fn with_metadata(&mut self, metadata: impl Into<Metadata>) -> &mut Self

pub fn with_vectors( &mut self, vectors: impl Into<HashMap<EmbeddedField, Embedding>>, ) -> &mut Self

pub fn with_sparse_vectors( &mut self, sparse_vectors: impl Into<HashMap<EmbeddedField, SparseEmbedding>>, ) -> &mut Self

pub fn id(&self) -> Uuid

Retrieve the identifier of the node.

Calculates the identifier of the node based on its path and chunk as bytes, returning a UUID (v3).

WARN: Does not memoize the id. Use sparingly.

pub fn parent_id(&self) -> Option<Uuid>

Returns the id of the first node this node was derived from, if any.

impl Node<String>

pub fn into_chunks( self, chunks: impl IntoIterator<Item = String>, ) -> impl Iterator<Item = Self>

Derives nodes from chunks while retaining the first ancestor as their parent.

Resolves the parent before removing the input text, then reuses the remaining fields for every output without cloning the full input text.

pub fn as_embeddables(&self) -> Vec<(EmbeddedField, String)>

Creates embeddable data depending on chosen EmbedMode.

Returns

Embeddable data mapped to their EmbeddedField.

Trait Implementations

impl<T: Clone + Chunk> Clone for Node<T>

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<T: Chunk> Debug for Node<T>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the node for debugging purposes.

This method is used to provide a human-readable representation of the node when debugging. The vector field is displayed as the number of elements in the vector if present.

impl<T: Default + Chunk> Default for Node<T>

fn default() -> Self

Returns the “default value” for a type.

impl<'de, T> Deserialize<'de> for Node<T>

where T: Deserialize<'de> + Chunk,

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl<T: Into<String>> From<T> for Node<String>

fn from(value: T) -> Self

Converts to this type from the input type.

impl Hash for Node<String>

fn hash<H: Hasher>(&self, state: &mut H)

Hashes the node based on its path and chunk.

This method is used by the calculate_hash method to generate a hash value for the node.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl<T: PartialEq + Chunk> PartialEq for Node<T>

fn eq(&self, other: &Self) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl<T> Serialize for Node<T>

where T: Serialize + Chunk,

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl<T: PartialEq + Chunk> StructuralPartialEq for Node<T>