Struct CognifyConfig 

pub struct CognifyConfig {

    pub max_chunk_size: usize,
    pub chunk_overlap: usize,
    pub chunk_strategy: ChunkStrategy,
    pub chunks_per_batch: usize,
    pub max_parallel_extractions: usize,
    pub custom_extraction_prompt: Option<String>,
    pub enable_summarization: bool,
    pub summarization_batch_size: usize,
    pub embed_triplets: bool,
    pub embedding_batch_size: usize,
    pub vector_collection_prefix: String,
    pub incremental_loading: bool,
    pub use_pipeline_cache: bool,
    pub temporal_cognify: bool,
    pub create_web_page_nodes: bool,
    pub data_per_batch: usize,
    pub token_counter_kind: TokenCounterKind,
    pub graph_schema: Option<Value>,
    pub summary_schema: Option<Value>,
    pub custom_chunker: Option<CustomChunker>,
    pub transcriber: Option<TranscriberHandle>,
}

Configuration for the cognify pipeline.

Design Principles:

1. NO hardcoded values in pipeline code - everything flows through config
2. NO environment variable reading in components (only in config construction if needed)
3. Sensible defaults matching cognee behavior
4. Builder pattern for easy customization

What is NOT in this config:

• Storage/Database/LLM/Embedding instances (passed as Arc to pipeline constructor)
• Runtime data (data_items, dataset_id, etc. - passed to cognify() method)
• Provider-specific API keys (handled by provider implementations, not pipeline config)

Fields

max_chunk_size: usize

Maximum chunk size in tokens.

The sentinel value 1500 means “auto-calculate at pipeline time” via CognifyConfig::auto_chunk_size. The pipeline in tasks.rs replaces the sentinel with the computed value before executing — matching Python’s get_max_chunk_tokens() behaviour where chunk_size=None at the cognify entry point always triggers auto-calculation. The computed value depends on the active embedding engine: ≈512 for the local ONNX/BGE default (512-token sequence limit) and 8191 for an OpenAI-compatible engine at its default max_completion_tokens (8191), both clamped by the LLM term (8192).

Pass an explicit value via CognifyConfig::with_chunk_size to override the auto-calculation; any value other than the sentinel is used as-is.

chunk_overlap: usize

Overlap between chunks (in tokens). Python default: 10 (from ChunkConfig.chunk_overlap) Used when chunk_strategy is RECURSIVE or LANGCHAIN

chunk_strategy: ChunkStrategy

Chunking strategy. Python default: ChunkStrategy.PARAGRAPH Options: Paragraph (sentence-aware), Recursive (character-based with overlap)

chunks_per_batch: usize

Number of chunks to process in a single batch during graph extraction. Python default: 100 (cognify parameter) Controls memory usage vs parallelism tradeoff

max_parallel_extractions: usize

Maximum number of parallel tasks for graph extraction within a batch. Python default: No explicit limit (uses asyncio.gather) Rust: Prevents spawning too many tokio tasks

custom_extraction_prompt: Option<String>

Custom prompt for entity/relationship extraction. Python parameter: custom_prompt (optional) If None, uses default prompts from cognee_llm

enable_summarization: bool

Enable text summarization stage. Python behavior: Always runs if summarization_model is set Default: true (matches Python)

summarization_batch_size: usize

Batch size for summarization (parallel summary generation). Python default: No explicit batching (processes all chunks in parallel) Rust: Prevents spawning too many tasks

embed_triplets: bool

Whether to generate and index triplet embeddings. Triplets are formatted as “source › relationship › target” Python config: CognifyConfig.triplet_embedding (default: False)

embedding_batch_size: usize

Batch size for embedding generation (all types: chunks, entities, summaries, triplets). Python default: varies by provider (36 for OpenAI, 100 for others) Controls how many texts are embedded in a single API call

vector_collection_prefix: String

Vector collection name prefix. Python default: Uses type names directly (“Entity”, “DocumentChunk”, etc.) Allows customization for multi-tenant or versioned deployments

incremental_loading: bool

Enable incremental loading - only process new/changed data. When true, tracks processed data IDs to avoid reprocessing. Python parameter: incremental_loading (default: True)

use_pipeline_cache: bool

Enable pipeline-level caching. When true, skips datasets whose latest pipeline run status is Completed. Requires a database connection to be provided. Python parameter: use_pipeline_cache (default: False)

temporal_cognify: bool

Enable temporal graph construction. Python parameter: temporal_cognify (default: False) Extracts events and timestamps for temporal reasoning

create_web_page_nodes: bool

Create WebPage/WebSite provenance nodes for URL-sourced documents.

When true, documents whose external metadata was produced by URL ingestion create deterministic WebPage and WebSite nodes plus DocumentChunk -> SOURCED_FROM -> WebPage and WebPage -> PART_OF -> WebSite edges.

data_per_batch: usize

Batch size for data processing in temporal cognify. Python parameter: data_per_batch (default: 20)

token_counter_kind: TokenCounterKind

How to count tokens when chunking text. Default is determined at construction time via TokenCounterKind::from_env.

graph_schema: Option<Value>

Optional JSON Schema for custom graph extraction model.

When Some, the LLM uses this schema instead of the default KnowledgeGraph schema for entity/relationship extraction. Extracted data is stored as-is in chunk metadata.

Mirrors Python’s graph_model parameter.

summary_schema: Option<Value>

Optional JSON schema for the summarization output.

Mirrors Python’s CognifyConfig.summarization_model (a Pydantic class, default SummarizedContent). When Some, the summarization stage requests this schema from the LLM instead of the built-in SummarizedContent shape. The schema must contain a string summary field — the pipeline reads summary to build each TextSummary (Python parity).

Validated at setter/builder time via validate_summary_schema.

custom_chunker: Option<CustomChunker>

Pluggable chunker callback.

When Some, this function is called instead of the built-in paragraph/recursive chunking. The callback receives the text and max token count, and returns a list of chunk strings.

Mirrors Python’s chunker parameter.

transcriber: Option<TranscriberHandle>

Optional transcriber for audio/video document processing.

When Some, this transcriber is used to convert audio content into text before chunking and graph extraction. Only takes effect when processing documents classified as audio type.

Implementations

impl CognifyConfig

pub fn with_chunk_size(self, size: usize) -> CognifyConfig

Set maximum chunk size in tokens.

pub fn with_chunk_overlap(self, overlap: usize) -> CognifyConfig

Set chunk overlap (for recursive chunking).

pub fn with_chunk_strategy(self, strategy: ChunkStrategy) -> CognifyConfig

Set chunking strategy.

pub fn with_chunks_per_batch(self, batch_size: usize) -> CognifyConfig

Set number of chunks per batch during graph extraction.

pub fn with_max_parallel_extractions(self, limit: usize) -> CognifyConfig

Set maximum parallel extractions.

pub fn with_custom_prompt(self, prompt: String) -> CognifyConfig

Set custom extraction prompt.

pub fn with_summarization(self, enable: bool) -> CognifyConfig

Enable or disable summarization.

pub fn with_summarization_batch_size(self, batch_size: usize) -> CognifyConfig

Set summarization batch size.

pub fn with_triplet_embeddings(self, enable: bool) -> CognifyConfig

Enable or disable triplet embeddings.

pub fn with_embedding_batch_size(self, batch_size: usize) -> CognifyConfig

Set embedding batch size.

pub fn with_collection_prefix(self, prefix: String) -> CognifyConfig

Set vector collection prefix.

pub fn with_incremental_loading(self, enable: bool) -> CognifyConfig

Enable or disable incremental loading.

pub fn with_pipeline_cache(self, enable: bool) -> CognifyConfig

Enable or disable pipeline-level caching.

pub fn with_temporal_cognify(self, enable: bool) -> CognifyConfig

Enable or disable temporal cognify.

pub fn with_web_page_nodes(self, enable: bool) -> CognifyConfig

Enable or disable WebPage/WebSite provenance graph construction.

pub fn with_data_per_batch(self, batch_size: usize) -> CognifyConfig

Set data per batch for temporal processing.

pub fn with_token_counter(self, kind: TokenCounterKind) -> CognifyConfig

Set the token counter implementation to use during chunking.

pub fn with_graph_schema(self, schema: Value) -> CognifyConfig

Set a custom JSON Schema for graph extraction.

pub fn with_summary_schema( self, schema: Value, ) -> Result<CognifyConfig, ConfigError>

Set a custom JSON schema for summarization output (Python summarization_model parity).

The schema must contain a string summary field — the pipeline reads summary to build each TextSummary. Returns an error if the schema lacks that field so callers catch the misconfiguration early rather than mid-pipeline.

pub fn with_custom_chunker( self, chunker: Arc<dyn Fn(&str, usize) -> Vec<String> + Sync + Send>, ) -> CognifyConfig

Set a custom chunker callback.

pub fn with_transcriber( self, transcriber: Arc<dyn Transcriber>, ) -> CognifyConfig

Set a transcriber for audio document processing.

pub fn auto_chunk_size( embedding_engine: &dyn EmbeddingEngine, _llm: &dyn Llm, ) -> usize

Auto-calculate max_chunk_size, mirroring Python’s get_max_chunk_tokens() from cognee/infrastructure/llm/utils.py:

llm_cutoff_point = llm_max_completion_tokens // 2   # Python default: 16384 → 8192
max_chunk_tokens = min(embedding_engine.max_completion_tokens, llm_cutoff_point)

Python uses completion-token budgets (not context windows):

• embedding_engine.max_completion_tokens — the engine’s configured token limit. Python’s EmbeddingConfig default is 8191 (embeddings/config.py:81), passed to the engine by the factory; the engine class’s own __init__ default of 512 is overridden in that path. Rust mirrors this: EmbeddingConfig.max_completion_tokens defaults to 8191.
• llm_max_completion_tokens = 16384 (infrastructure/llm/config.py:51).
• So for an OpenAI-compatible engine: min(8191, 8192) = 8191. For the local ONNX/BGE engine, max_sequence_length() is the model’s 512-token limit, so min(512, 8192) = 512. The embedding term is the binding one in both cases.

The Rust Llm trait exposes only max_context_length() (a context window), not a completion-token limit. Rather than divide an unrelated quantity, we use Python’s LLM completion-token constant (16384) directly. The embedding term (max_sequence_length() — 512 for BGE, the configured max_completion_tokens for OpenAI-compatible) is binding in all practical configurations, so the LLM argument is currently unused (_llm).

Result is at least 1.

pub fn with_auto_chunk_size( self, embedding_engine: &dyn EmbeddingEngine, llm: &dyn Llm, ) -> CognifyConfig

Set max_chunk_size by auto-calculating from embedding and LLM capabilities.

See auto_chunk_size for the formula used.

pub fn validate(&self) -> Result<(), ConfigError>

Validate configuration parameters.

Returns an error if any parameters are invalid.

Trait Implementations

impl Clone for CognifyConfig

fn clone(&self) -> CognifyConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for CognifyConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Default for CognifyConfig

fn default() -> CognifyConfig

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for CognifyConfig

fn deserialize<__D>( __deserializer: __D, ) -> Result<CognifyConfig, <__D as Deserializer<'de>>::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for CognifyConfig

fn serialize<__S>( &self, __serializer: __S, ) -> Result<<__S as Serializer>::Ok, <__S as Serializer>::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.