Struct Node

pub struct Node {
    pub path: PathBuf,
    pub chunk: String,
    pub vectors: Option<HashMap<EmbeddedField, Embedding>>,
    pub sparse_vectors: Option<HashMap<EmbeddedField, SparseEmbedding>>,
    pub metadata: Metadata,
    pub embed_mode: EmbedMode,
    pub original_size: usize,
    pub offset: usize,
}

Represents a unit of data in the indexing process.

Node encapsulates all necessary information for a single unit of data being processed in the indexing pipeline. It includes fields for an identifier, file path, data chunk, optional vector representation, and metadata.

Fields

path: PathBuf

File path associated with the node.

chunk: String

Data chunk contained in the node.

vectors: Option<HashMap<EmbeddedField, Embedding>>

Optional vector representation of embedded data.

sparse_vectors: Option<HashMap<EmbeddedField, SparseEmbedding>>

Optional sparse vector representation of embedded data.

metadata: Metadata

Metadata associated with the node.

embed_mode: EmbedMode

Mode of embedding data Chunk and Metadata

original_size: usize

Size of the input this node was originally derived from in bytes

offset: usize

Offset of the chunk relative to the start of the input this node was originally derived from in bytes

Implementations

impl Node

pub fn build_from_other(node: &Node) -> NodeBuilder

Builds a new instance of Node, returning a NodeBuilder. Copies over the fields from the provided Node.

pub fn builder() -> NodeBuilder

Creates a new instance of NodeBuilder.

pub fn new(chunk: impl Into<String>) -> Node

Creates a new instance of Node with the specified data chunk.

The other fields are set to their default values.

pub fn with_metadata(&mut self, metadata: impl Into<Metadata>) -> &mut Self

pub fn with_vectors( &mut self, vectors: impl Into<HashMap<EmbeddedField, Embedding>>, ) -> &mut Self

pub fn with_sparse_vectors( &mut self, sparse_vectors: impl Into<HashMap<EmbeddedField, SparseEmbedding>>, ) -> &mut Self

pub fn as_embeddables(&self) -> Vec<(EmbeddedField, String)>

Creates embeddable data depending on chosen EmbedMode.

Returns

Embeddable data mapped to their EmbeddedField.

pub fn id(&self) -> Uuid

Retrieve the identifier of the node.

Calculates the identifier of the node based on its path and chunk as bytes, returning a UUID (v3).

WARN: Does not memoize the id. Use sparingly.

Trait Implementations

impl Clone for Node

fn clone(&self) -> Node

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Node

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the node for debugging purposes.

This method is used to provide a human-readable representation of the node when debugging. The vector field is displayed as the number of elements in the vector if present.

impl Default for Node

fn default() -> Node

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for Node

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl<T: Into<String>> From<T> for Node

fn from(value: T) -> Self

Converts to this type from the input type.

impl Hash for Node

fn hash<H: Hasher>(&self, state: &mut H)

Hashes the node based on its path and chunk.

This method is used by the calculate_hash method to generate a hash value for the node.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl PartialEq for Node

fn eq(&self, other: &Node) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Serialize for Node

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl StructuralPartialEq for Node