tenshift-core 0.1.2

Thread-safe, backpressure-aware data loading pipeline for iterative processing
Documentation
1
2
3
4
5
6
# BACKLOG

DONE 2026-07-17 | src/sources/image_folder.rs:74, src/sources/mod.rs:83, src/sources/ring_source.rs:69 | tenshift-core | silent-fallback | medium | REPOPULATE 2026-07-16 (consolidated, one root cause x3 source constructors): every filesystem source discards directory-traversal errors and silently trains on an incomplete dataset - image_folder.rs:74 drops read_dir/entry-iteration errors, GlobSource::new (mod.rs:83) and RingSource::from_glob (ring_source.rs:69) both `filter_map(Result::ok)` away glob GlobErrors. An unreadable subdir/permission error yields a partial path list with no operator-visible signal (Law 10 silent input loss in the data pipeline). | FIXED: added ONE-PLACE `collect_glob_files(pattern) -> Result<Vec<PathBuf>>` in sources/mod.rs that surfaces each glob::GlobError as `Error::Io` instead of `filter_map(Result::ok)`; `GlobSource::new` and `RingSource::from_glob` both route through it. image_folder.rs:74 now propagates read_dir + per-entry errors via a `read_failed` closure -> `Error::ReadFailed`. Proving tests (all green, 33 passed exit 0): `collect_glob_files_returns_only_regular_files`, `collect_glob_files_rejects_invalid_pattern`, `unreadable_class_dir_fails_loud_not_silently_skipped` (chmod 0o000 class dir, root-guarded). | status=done
DONE 2026-07-17 | src/pipeline/worker.rs:147 | tenshift-core | performance | medium | REPOPULATE 2026-07-16: The stateless transform worker loop allocates two fresh Vecs (buf1, buf2) with capacity 64 for every sample, causing significant heap allocation overhead in the hot data processing loop | FIXED: hoisted buf1/buf2 out of the per-sample loop (allocated once per worker thread alongside `output`); the catch_unwind closure now BORROWS them (and `transforms`) via AssertUnwindSafe instead of moving+reallocating, and results are drained with `output.append(&mut buf1)` which keeps buf1's allocation. Also removed the per-sample `Arc::clone(&transforms)`. Proving test `apply_stateless_reuses_scratch_buffers_across_samples` runs 500 samples through a 2-transform chain and asserts buf1's backing pointer AND capacity are unchanged throughout (a fresh per-sample Vec would differ). While here, fixed a co-located Law-10 silent fallback: `if let Ok(mut lock) = w_fatal.lock()` silently dropped the fatal error on a poisoned mutex (a failed epoch would look clean) — extracted ONE-PLACE `record_first_fatal_error` that recovers the poison via PoisonError::into_inner; both worker fatal sites route through it; tests `record_first_fatal_error_keeps_only_the_first` and `_recovers_a_poisoned_lock`. 36 passed exit 0. | status=done
DONE 2026-07-17 | src/pipeline/worker.rs:179,205 | tenshift-core | silent-fallback | high | FOUND 2026-07-17 while fixing worker.rs:147: `if let Ok(mut lock) = w_fatal.lock()` at both ErrorPolicy::Fail sites silently discarded the fatal error if the fatal-error mutex was poisoned (another worker panicked holding it), so a fatally-failed epoch could complete looking clean (Law 10). | FIXED alongside the row above via `record_first_fatal_error` (poison-recovering). | status=done
DONE 2026-07-17 | src/pipeline/collector/accumulate.rs:16 | tenshift-core | performance | medium | REPOPULATE 2026-07-16: SchemaSignature::from_sample allocates a new Vec and clones the String name and shape Vec for every field of every sample during batch schema validation, causing significant allocation overhead in the hot collector loop | FIXED: deleted SchemaSignature entirely; `validate_batch_schema` now compares each sample against the reference sample directly via `schema_matches`, looking up fields by name on the reference (order-independent, since field names are unique per sample) — zero allocation on the happy path (only error-message construction allocates). Proving tests: order-independent accept, field-count/dtype/shape mismatch rejects, and a renamed-field reject (caught by name lookup, not silently accepted). 41 passed exit 0. | status=done