pub enum JobType {
EmailSync {
identities: Option<Vec<String>>,
local_output: Option<PathBuf>,
remote_output: Option<String>,
encryption_key: Option<String>,
concurrency: Option<usize>,
upload_concurrency: Option<usize>,
max_connections_per_identity: Option<usize>,
upload_only: bool,
report_bucket: Option<String>,
yes: bool,
},
DecryptFiles {
input_dir: Option<PathBuf>,
output_dir: Option<PathBuf>,
encryption_key: Option<String>,
concurrency: Option<usize>,
report_bucket: Option<String>,
yes: bool,
},
EmailPull {
identities: Option<Vec<String>>,
local_output: Option<PathBuf>,
remote_output: Option<String>,
concurrency: Option<usize>,
upload_concurrency: Option<usize>,
max_connections_per_identity: Option<usize>,
upload_only: bool,
report_bucket: Option<String>,
yes: bool,
},
PullTransform {Show 14 fields
source_bucket: Option<String>,
local_output: Option<PathBuf>,
remote_output: Option<String>,
encryption_key: Option<String>,
file_types: Option<Vec<String>>,
expand_zips: Option<Vec<String>>,
image_format: Option<String>,
video_format: Option<String>,
audio_format: Option<String>,
concurrency: Option<usize>,
upload_concurrency: Option<usize>,
upload_only: bool,
report_bucket: Option<String>,
yes: bool,
},
Deduplicate {
source_bucket: Option<String>,
local_output: Option<PathBuf>,
remote_output: Option<String>,
concurrency: Option<usize>,
upload_concurrency: Option<usize>,
upload_only: bool,
report_bucket: Option<String>,
yes: bool,
},
Reduce {
source_bucket: Option<String>,
local_output: Option<PathBuf>,
remote_output: Option<String>,
concurrency: Option<usize>,
upload_concurrency: Option<usize>,
upload_only: bool,
force_valuable: Vec<String>,
force_reproducible: Vec<String>,
report_bucket: Option<String>,
yes: bool,
},
Import {
source: Option<String>,
destination: Option<String>,
local_output: Option<PathBuf>,
report_bucket: Option<String>,
yes: bool,
},
}Variants§
EmailSync
Fetch, transform, deduplicate, and optionally upload mail for one or
more authenticated email identities. Replaces pigeon email sync
(ADR-0021).
Fields
identities: Option<Vec<String>>Aliases of the identities to sync, comma-separated. Interactively selected from the authenticated identities when omitted and stdin is a terminal; required otherwise.
local_output: Option<PathBuf>Local directory to stage and store output under, shared across every selected identity (each gets its own subdirectory underneath). Defaults to a directory under the OS temp directory when omitted.
remote_output: Option<String>Alias of a configured bucket-config (see pigeon dataops bucket-config new) to upload each identity’s local result tree
to, once its local fetch/transform/dedupe phase is complete.
encryption_key: Option<String>Alias of a configured encryption key (see pigeon keyring add encryption-key) to use, overriding the target bucket-config’s
own default (if any). Interactively selected/confirmed when
omitted; falls back to the bucket’s default non-interactively
(ADR-0027).
concurrency: Option<usize>Maximum number of fetch/transform workers to run concurrently, spanning every selected identity’s every mailbox. Interactively prompted (with a rough time estimate) when omitted and stdin is a terminal; required otherwise.
upload_concurrency: Option<usize>Maximum number of files to upload concurrently, independent of
--concurrency (which sizes IMAP fetch/transform work) – the
upload phase is network-round-trip-bound, not IMAP-bound, so it
benefits from its own, separately-tuned concurrency (ADR-0091).
Interactively prompted when omitted and stdin is a terminal;
defaults to 16 otherwise. With --upload-only, this is the only
concurrency flag that has any effect.
max_connections_per_identity: Option<usize>Maximum number of simultaneous IMAP connections opened to any
one identity, regardless of --concurrency (ADR-0071) – caps
worker concurrency per-account rather than only globally, so a
mailbox with enough pending batches can’t cause more than this
many workers to log in to the same account at once and trip a
provider’s simultaneous-connection limit. Defaults to 6 (well
under Gmail’s documented 15-connection cap) when omitted; not
interactively prompted.
upload_only: boolResumes uploading already-completed local runs for the selected
identities instead of starting new ones: skips the IMAP
connect/fetch/transform/dedup phases entirely (and the
per-identity IMAP credentials they’d otherwise need) and
uploads straight from each identity’s existing local result
tree, picking up where a prior run’s upload phase left off via
the same .staging/.uploaded index per identity (ADR-0090). An
identity with no completed local run is skipped with a warning
rather than failing the whole command. Requires a mandatory
--remote-output.
report_bucket: Option<String>Alias of a configured bucket-config this run’s report, the
shared observability log, and a transcript of its printed
output are uploaded to, always unencrypted, under a
YYYY-MM-DD-job-name-{run-id}/ prefix (ADR-0100). Mandatory –
interactively selected when omitted and stdin is a terminal;
required otherwise.
DecryptFiles
Decrypts every *.enc file under --input-dir into --output-dir
(.enc suffix stripped, relative structure preserved), using a
configured encryption key (ADR-0028).
Fields
input_dir: Option<PathBuf>Directory containing *.enc files to decrypt. Interactively
prompted when omitted and stdin is a terminal; required
otherwise.
output_dir: Option<PathBuf>Directory decrypted files are written under, mirroring
--input-dir’s relative structure. Must not be the same
directory as --input-dir. Interactively prompted when omitted
and stdin is a terminal; required otherwise.
encryption_key: Option<String>Alias of a configured encryption key (see pigeon keyring add encryption-key) to decrypt with. Interactively selected when
omitted and stdin is a terminal; required otherwise.
report_bucket: Option<String>Alias of a configured bucket-config this run’s report, the
shared observability log, and a transcript of its printed
output are uploaded to, always unencrypted, under a
YYYY-MM-DD-job-name-{run-id}/ prefix (ADR-0100). Mandatory –
interactively selected when omitted and stdin is a terminal;
required otherwise.
EmailPull
Fetches raw .eml files and unpacked attachments (no Markdown/
frontmatter transform) for one or more authenticated email
identities, deduplicating attachments by content, and optionally
uploads the result unencrypted to a bucket-config (ADR-0081).
Fields
identities: Option<Vec<String>>Aliases of the identities to pull, comma-separated. Interactively selected from the authenticated identities when omitted and stdin is a terminal; required otherwise.
local_output: Option<PathBuf>Local directory to stage and store output under, shared across every selected identity. Defaults to a directory under the OS temp directory when omitted.
remote_output: Option<String>Alias of a configured bucket-config to upload each identity’s local result tree to, once its local fetch/dedupe phase is complete. Always uploaded unencrypted – this job never offers encryption (ADR-0081).
concurrency: Option<usize>Maximum number of fetch/extract workers to run concurrently, spanning every selected identity’s every mailbox. Interactively prompted (with a rough time estimate) when omitted and stdin is a terminal; required otherwise.
upload_concurrency: Option<usize>Maximum number of files to upload concurrently, independent of
--concurrency (which sizes IMAP fetch/extract work) – the
upload phase is network-round-trip-bound, not IMAP-bound, so it
benefits from its own, separately-tuned concurrency (ADR-0091).
Interactively prompted when omitted and stdin is a terminal;
defaults to 16 otherwise. With --upload-only, this is the only
concurrency flag that has any effect.
max_connections_per_identity: Option<usize>Maximum number of simultaneous IMAP connections opened to any
one identity, regardless of --concurrency. Defaults to 6 when
omitted; not interactively prompted.
upload_only: boolResumes uploading already-completed local runs for the selected
identities instead of starting new ones: skips the IMAP
connect/fetch/dedup phases entirely (and the per-identity IMAP
credentials they’d otherwise need) and uploads straight from
each identity’s existing local result tree, picking up where a
prior run’s upload phase left off via the same
.staging/.uploaded index per identity (ADR-0090). An identity
with no completed local run is skipped with a warning rather
than failing the whole command. Requires a mandatory
--remote-output.
report_bucket: Option<String>Alias of a configured bucket-config this run’s report, the
shared observability log, and a transcript of its printed
output are uploaded to, always unencrypted, under a
YYYY-MM-DD-job-name-{run-id}/ prefix (ADR-0100). Mandatory –
interactively selected when omitted and stdin is a terminal;
required otherwise.
PullTransform
Recursively pulls every object from a bucket-config, expands zips,
recodes media into a size-optimized canonical format per category
(photo/screenshot -> jpg, video -> mp4, audio -> m4a), dates and
dedups everything by content, and organizes the result by extension
– then optionally encrypts and uploads it to a (possibly
different) bucket-config (ADR-0074). Requires ffmpeg/ffprobe on
PATH.
Fields
source_bucket: Option<String>Alias of a configured bucket-config (see pigeon keyring add bucket) to pull from. Interactively selected from the
configured bucket-configs when omitted and stdin is a terminal;
required otherwise.
local_output: Option<PathBuf>Local directory to stage and store output under. Defaults to a directory under the OS temp directory when omitted.
remote_output: Option<String>Alias of a configured bucket-config to upload the organized result to, once local processing is complete.
encryption_key: Option<String>Alias of a configured encryption key, overriding the target bucket-config’s own default (if any).
file_types: Option<Vec<String>>File extensions to pull/transform/upload, comma-separated (e.g.
jpg,mp4,pdf; use the literal none for extensionless keys).
Everything else is left pending, untouched, for a future run –
never checkpointed as done (ADR-0077). Interactively selected
(all pre-checked) from the pending-summary table when omitted
and stdin is a terminal; defaults to everything otherwise.
expand_zips: Option<Vec<String>>Keys of pending zip objects to expand and transform; comma-separated. Every other pending zip is uploaded as-is, untouched (ADR-0077). Interactively selected (all pre-checked) when omitted and stdin is a terminal; defaults to expanding every pending zip otherwise.
upload_concurrency: Option<usize>Maximum number of files to upload concurrently, independent of
--concurrency (which sizes download/recode work) – the upload
phase is network-round-trip-bound, not CPU-bound, so it benefits
from its own, separately-tuned concurrency (ADR-0091).
Interactively prompted when omitted and stdin is a terminal;
defaults to 16 otherwise. With --upload-only, this is the only
concurrency flag that has any effect.
upload_only: boolResumes uploading an already-completed local pull-transform run
instead of starting a new one: skips the bucket listing/
download/classify/recode/placement phases entirely (and the
source bucket credentials and ffmpeg/ffprobe check they’d
otherwise need) and uploads straight from an existing
--local-output, picking up where a prior run’s upload phase
left off via the same .staging/.uploaded index (ADR-0090).
Requires a --local-output from a completed prior run (its
.processed checkpoint must exist and it must hold at least
one placed-content subdirectory) and a mandatory
--remote-output.
report_bucket: Option<String>Alias of a configured bucket-config this run’s report, the
shared observability log, and a transcript of its printed
output are uploaded to, always unencrypted, under a
YYYY-MM-DD-job-name-{run-id}/ prefix (ADR-0100). Mandatory –
interactively selected when omitted and stdin is a terminal;
required otherwise.
Deduplicate
Recursively scans a bucket, always inflates every zip found (the
zip container itself is never uploaded, only its inflated
contents), content-hashes every file bucket-wide to keep one
byte-identical copy of each, writes a human-readable merge report,
and optionally uploads the result unencrypted to a (possibly
different) bucket-config (ADR-0082). Unlike pull-transform, every
file is always processed and every zip is always expanded – there
is no file-type or zip-expansion selection, and this job never
offers encryption.
Fields
source_bucket: Option<String>Alias of a configured bucket-config to pull from. Interactively selected from the configured bucket-configs when omitted and stdin is a terminal; required otherwise.
local_output: Option<PathBuf>Local directory to stage and store output under. Defaults to a directory under the OS temp directory when omitted.
remote_output: Option<String>Alias of a configured bucket-config to upload the deduped result to, once local processing is complete. Always uploaded unencrypted.
upload_concurrency: Option<usize>Maximum number of files to upload concurrently, independent of
--concurrency (which sizes download/hash work) – the upload
phase is network-round-trip-bound, not CPU-bound, so it benefits
from its own, separately-tuned concurrency (ADR-0091).
Interactively prompted when omitted and stdin is a terminal;
defaults to 16 otherwise. With --upload-only, this is the only
concurrency flag that has any effect.
upload_only: boolResumes uploading an already-completed local deduplicate run instead
of starting a new one: skips the bucket listing/download/hash/
placement phases entirely (and the source bucket credentials
they’d otherwise need) and uploads straight from an existing
--local-output’s result/ tree, picking up where a prior
run’s upload phase left off via the same .staging/.uploaded
index (ADR-0089). Requires a --local-output from a completed
prior run (its .staging/.processed checkpoint must exist and
its result/ must be non-empty) and a mandatory
--remote-output.
report_bucket: Option<String>Alias of a configured bucket-config this run’s report, the
shared observability log, and a transcript of its printed
output are uploaded to, always unencrypted, under a
YYYY-MM-DD-job-name-{run-id}/ prefix (ADR-0100). Mandatory –
interactively selected when omitted and stdin is a terminal;
required otherwise.
Reduce
Runs after deduplicate: recursively scans a source bucket
already organized into top-level <extension>/ folders, classifies
each extension as either genuinely valuable or an artifact/piece of
media (TV, movie, software installer, disk image) that’s easily
reproduced from an external canonical source, and forwards only the
valuable extensions’ objects to a mandatory destination bucket
(ADR-0096). Reproducible extensions are never even downloaded. Never
offers encryption; never expands zips (input is already flat).
Fields
source_bucket: Option<String>Alias of a configured bucket-config to pull from. Interactively selected from the configured bucket-configs when omitted and stdin is a terminal; required otherwise.
local_output: Option<PathBuf>Local directory to stage and store output under. Defaults to a directory under the OS temp directory when omitted.
remote_output: Option<String>Alias of a configured bucket-config to upload the forwarded result to. Always uploaded unencrypted. Interactively selected from the configured bucket-configs when omitted and stdin is a terminal; required otherwise – uploading is mandatory for this job.
upload_concurrency: Option<usize>Maximum number of files to upload concurrently, independent of
--concurrency (which sizes download work) – the upload phase
is network-round-trip-bound, not CPU-bound, so it benefits from
its own, separately-tuned concurrency (ADR-0091). Interactively
prompted when omitted and stdin is a terminal; defaults to 16
otherwise. With --upload-only, this is the only concurrency
flag that has any effect.
upload_only: boolResumes uploading an already-completed local reduce run instead
of starting a new one: skips the bucket listing/download/
placement phases entirely (and the source bucket credentials
they’d otherwise need) and uploads straight from an existing
--local-output’s result/ tree, picking up where a prior
run’s upload phase left off via the same .staging/.uploaded
index (ADR-0090). Requires a --local-output from a completed
prior run (its .staging/.processed checkpoint must exist and
its result/ must be non-empty).
force_valuable: Vec<String>Extension (without the leading dot, e.g. mp3) to always treat
as valuable regardless of the built-in classification table.
Repeatable.
force_reproducible: Vec<String>Extension (without the leading dot, e.g. pdf) to always treat
as a reproducible artifact/media file regardless of the
built-in classification table. Repeatable.
report_bucket: Option<String>Alias of a configured bucket-config this run’s report, the
shared observability log, and a transcript of its printed
output are uploaded to, always unencrypted, under a
YYYY-MM-DD-job-name-{run-id}/ prefix (ADR-0100). Mandatory –
interactively selected when omitted and stdin is a terminal;
required otherwise.
Import
Copies data from a configurable source to a configurable
destination by shelling out to the external rclone binary, with
its performance/retry flags fixed (not configurable here). Pigeon
manages no rclone credentials/config – --source/--destination
are raw remote:path strings passed straight through to rclone copy’s argv; rclone.conf is provisioned by an external process,
outside this crate’s scope (ADR-0101). Requires rclone on PATH.
Fields
source: Option<String>rclone source, e.g. source:media/. Interactively prompted
when omitted and stdin is a terminal; required otherwise.
destination: Option<String>rclone destination, e.g. destination:. Interactively
prompted when omitted and stdin is a terminal; required
otherwise.
local_output: Option<PathBuf>Local directory this run’s rclone log (also serving as this job’s report) and transcript are written under. Defaults to a directory under the OS temp directory when omitted. Unlike every other job, this is not a staging area for transferred data – rclone transfers directly source -> destination with no pigeon-side staging.
report_bucket: Option<String>Alias of a configured bucket-config this run’s report (the
rclone log itself, for this job), the shared observability log,
and a transcript of its printed output are uploaded to, always
unencrypted, under a YYYY-MM-DD-job-name-{run-id}/ prefix
(ADR-0100). Mandatory – interactively selected when omitted
and stdin is a terminal; required otherwise.
Trait Implementations§
Source§impl FromArgMatches for JobType
impl FromArgMatches for JobType
Source§fn from_arg_matches(__clap_arg_matches: &ArgMatches) -> Result<Self, Error>
fn from_arg_matches(__clap_arg_matches: &ArgMatches) -> Result<Self, Error>
Source§fn from_arg_matches_mut(
__clap_arg_matches: &mut ArgMatches,
) -> Result<Self, Error>
fn from_arg_matches_mut( __clap_arg_matches: &mut ArgMatches, ) -> Result<Self, Error>
Source§fn update_from_arg_matches(
&mut self,
__clap_arg_matches: &ArgMatches,
) -> Result<(), Error>
fn update_from_arg_matches( &mut self, __clap_arg_matches: &ArgMatches, ) -> Result<(), Error>
ArgMatches to self.Source§fn update_from_arg_matches_mut<'b>(
&mut self,
__clap_arg_matches: &mut ArgMatches,
) -> Result<(), Error>
fn update_from_arg_matches_mut<'b>( &mut self, __clap_arg_matches: &mut ArgMatches, ) -> Result<(), Error>
ArgMatches to self.Source§impl Subcommand for JobType
impl Subcommand for JobType
Source§fn augment_subcommands<'b>(__clap_app: Command) -> Command
fn augment_subcommands<'b>(__clap_app: Command) -> Command
Source§fn augment_subcommands_for_update<'b>(__clap_app: Command) -> Command
fn augment_subcommands_for_update<'b>(__clap_app: Command) -> Command
Command so it can instantiate self via
FromArgMatches::update_from_arg_matches_mut Read moreSource§fn has_subcommand(__clap_name: &str) -> bool
fn has_subcommand(__clap_name: &str) -> bool
Self can parse a specific subcommandAuto Trait Implementations§
impl Freeze for JobType
impl RefUnwindSafe for JobType
impl Send for JobType
impl Sync for JobType
impl Unpin for JobType
impl UnsafeUnpin for JobType
impl UnwindSafe for JobType
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
Source§impl<T> Instrument for T
impl<T> Instrument for T
Source§fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
Source§fn in_current_span(self) -> Instrumented<Self> ⓘ
fn in_current_span(self) -> Instrumented<Self> ⓘ
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more