Struct DatasetEngineManager 

pub struct DatasetEngineManager { /* private fields */ }

Top-level coordinator for all dataset tables.

Manages a registry of table metadata, lazy-loads engine actors on demand, and evicts idle engines based on TTL and a hard cap.

Implementations

impl DatasetEngineManager

pub async fn new( storage_settings: &ObjectStorageSettings, ) -> Result<Self, DatasetEngineError>

pub async fn with_config( storage_settings: &ObjectStorageSettings, engine_ttl_secs: u64, max_active_engines: usize, flush_interval_secs: u64, max_buffer_rows: usize, refresh_interval_secs: u64, ) -> Result<Self, DatasetEngineError>

Create a manager with custom configuration (primarily for testing).

pub async fn register_dataset( &self, registration: &DatasetRegistration, ) -> Result<RegistrationResult, DatasetEngineError>

Register a dataset schema. Idempotent. Does NOT spawn an engine — that’s lazy on first write/query.

pub async fn insert_batch( &self, namespace: &DatasetNamespace, fingerprint: &DatasetFingerprint, batch: RecordBatch, ) -> Result<(), DatasetEngineError>

Insert a RecordBatch into a dataset table. Activates the engine on demand if not already active.

pub async fn query( &self, sql: &str, ) -> Result<Vec<RecordBatch>, DatasetEngineError>

Execute a SQL query against the shared query context.

Only SELECT statements are allowed. All other statement types (DDL, DML, SHOW, etc.) are rejected at parse time.

pub fn list_datasets(&self) -> Vec<DatasetRegistration>

List all registered datasets (from registry cache, not just active engines).

pub fn get_dataset_info( &self, namespace: &DatasetNamespace, ) -> Option<DatasetRegistration>

Get registration info for a specific dataset.

pub fn list_catalogs(&self) -> Vec<CatalogSummary>

List all distinct catalogs with schema and table counts.

pub fn list_schemas(&self, catalog: &str) -> Vec<SchemaSummary>

List schemas within a catalog with table counts.

pub fn list_tables( &self, catalog: &str, schema_name: &str, ) -> Vec<TableSummaryInfo>

List tables within a catalog.schema with summary info.

pub async fn get_table_detail( &self, namespace: &DatasetNamespace, ) -> Result<TableDetail, DatasetEngineError>

Get detailed info for a table: columns, partition info, and Delta stats.

pub async fn preview_table( &self, namespace: &DatasetNamespace, max_rows: usize, ) -> Result<Vec<RecordBatch>, DatasetEngineError>

Preview a table’s data (SELECT * LIMIT max_rows).

pub async fn execute_query( &self, sql: &str, query_id: &str, max_rows: usize, ) -> Result<QueryResult, DatasetEngineError>

Execute a SQL query with row limits, cancellation support, and metadata.

pub async fn cancel_query(&self, query_id: &str) -> bool

Cancel a running query by ID.

pub async fn explain_query( &self, sql: &str, analyze: bool, max_rows: usize, ) -> Result<ExplainResult, DatasetEngineError>

Generate a structured query plan, optionally with ANALYZE execution.

pub async fn shutdown(&self)

Shutdown all active engines gracefully.

pub fn start_reaper_loop( self: &Arc<Self>, shutdown_rx: Receiver<()>, ) -> impl Future<Output = ()> + Send + 'static

Start the reaper loop that evicts idle engines.

Returns a future suitable for TaskManager::spawn(). The loop exits when the shutdown receiver fires.

pub fn start_discovery_loop( self: &Arc<Self>, shutdown_rx: Receiver<()>, ) -> impl Future<Output = ()> + Send + 'static

Start the discovery loop that refreshes the registry from other pods.

Returns a future suitable for TaskManager::spawn(). The loop exits when the shutdown receiver fires.

pub fn query_ctx(&self) -> &Arc<SessionContext>

Access the shared query context (for Phase 3 gRPC/HTTP integration).

pub fn registry(&self) -> &Arc<DatasetRegistry>

Access the registry (for Phase 3 gRPC/HTTP integration).

pub fn active_engine_count(&self) -> usize

Number of currently active engines.