Struct NewDataset 

pub struct NewDataset {
    pub original_id: String,
    pub source_portal: String,
    pub url: String,
    pub title: String,
    pub description: Option<String>,
    pub embedding: Option<Vector>,
    pub metadata: Value,
    pub content_hash: String,
}

Data Transfer Object for inserting or updating datasets.

This structure is used when creating new datasets or updating existing ones. Unlike Dataset, it doesn’t include database-generated fields like id or timestamps. The embedding field uses pgvector’s Vector for database storage.

Examples

use ceres_core::NewDataset;
use serde_json::json;

let title = "My Dataset";
let description = Some("Description here".to_string());
let content_hash = NewDataset::compute_content_hash(title, description.as_deref());

let dataset = NewDataset {
    original_id: "dataset-123".to_string(),
    source_portal: "https://dati.gov.it".to_string(),
    url: "https://dati.gov.it/dataset/my-data".to_string(),
    title: title.to_string(),
    description,
    embedding: None,
    metadata: json!({"tags": ["open-data", "italy"]}),
    content_hash,
};

assert_eq!(dataset.title, "My Dataset");
assert!(dataset.embedding.is_none());
assert_eq!(dataset.content_hash.len(), 64); // SHA-256 = 64 hex chars

Fields

• original_id - Original identifier from the source portal
• source_portal - Base URL of the originating CKAN portal
• url - Public landing page URL for the dataset
• title - Human-readable dataset title
• description - Optional detailed description
• embedding - Optional vector of 768 floats (pgvector)
• metadata - Additional metadata as JSON
• content_hash - SHA-256 hash of title + description for delta detection

Fields

original_id: String

Original identifier from the source portal

source_portal: String

Base URL of the originating CKAN portal

url: String

Public landing page URL for the dataset

title: String

Human-readable dataset title

description: Option<String>

Optional detailed description

embedding: Option<Vector>

Optional vector of 768 floats (converted to pgvector on storage)

metadata: Value

Additional metadata as JSON

content_hash: String

SHA-256 hash of title + description for delta detection

Implementations

impl NewDataset

pub fn compute_content_hash(title: &str, description: Option<&str>) -> String

Computes a SHA-256 hash of the content (title + description) for delta detection.

This hash is used to determine if the dataset content has changed since the last harvest, avoiding unnecessary embedding regeneration.

Arguments

• title - The dataset title
• description - Optional dataset description

Returns

A 64-character lowercase hexadecimal string representing the SHA-256 hash.

Trait Implementations

impl Clone for NewDataset

fn clone(&self) -> NewDataset

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for NewDataset

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Serialize for NewDataset

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.