# Transcript subcommand
`omni-dev transcript` fetches captions and transcripts from external media
platforms. YouTube is the only source today; the CLI namespace and the
underlying library are designed so additional sources (Vimeo, podcast RSS,
generic VTT/SRT URLs, …) can be added without restructuring.
## CLI usage
### Auto-detecting `fetch`
`omni-dev transcript fetch <url>` probes every registered source and dispatches
to the one that recognises the locator — no provider name required. It accepts
the same flags as the per-source `fetch` below and is the shortest path when you
have a URL and don't care which source handles it:
```bash
# Source auto-detected from the locator shape.
omni-dev transcript fetch https://www.youtube.com/watch?v=jNQXAC9IVRw
omni-dev transcript fetch jNQXAC9IVRw --format vtt -o me-at-the-zoo.vtt
```
A locator no registered source recognises fails with `InvalidLocator` before any
network call. With only YouTube registered today, `fetch <url>` resolves to the
YouTube source whenever the URL is a YouTube locator; the dispatch generalises
the moment a second source is added (see "Adding a new source").
### Per-source subcommands
The provider is the first positional argument so per-source flags and help
output stay clean:
```bash
# Fetch captions for an unrestricted, captioned video.
omni-dev transcript youtube fetch https://www.youtube.com/watch?v=jNQXAC9IVRw
# Same video, written to disk as WebVTT.
omni-dev transcript youtube fetch jNQXAC9IVRw \
-o vtt --out-file me-at-the-zoo.vtt
# Fall through to auto-generated captions when no manual track exists.
omni-dev transcript youtube fetch <url> --auto
# Synthesise a translated track when no native track matches the language.
omni-dev transcript youtube fetch <url> --lang fr --translate fr
# List every caption track on a video, with `kind` distinguishing manual
# from auto-generated.
omni-dev transcript youtube list-langs <url>
# Show top-level metadata (title, channel, duration, available languages).
omni-dev transcript youtube info <url> -o json
# Sync every captioned video in one or more channels to a directory,
# incrementally. Writes a transcript and a metadata sidecar per video.
omni-dev transcript youtube sync @RickAstleyYT --out ./transcripts --auto
# Re-fetch metadata sidecars older than two days (missing ones are always
# backfilled; this also refreshes stale ones).
omni-dev transcript youtube sync @RickAstleyYT --out ./transcripts \
--refresh-metadata-older-than "2 days ago"
```
### `fetch` flags
| `--lang <code>` | `en` | Preferred language. Prefix fallback applies — `en` matches `en-US`. |
| `-o`, `--output <fmt>` | `srt` | One of `srt`, `vtt`, `txt`, `json`. |
| `--auto` | off | Allow falling through to auto-generated (ASR) captions. |
| `--translate <lang>`| — | Synthesise a translated track in `<lang>` when no native track matches. |
| `--out-file <path>` | stdout | Write the rendered transcript to a file instead of stdout. |
### `sync` output layout and metadata sidecars
`sync` enumerates a channel's videos and writes, per video, into
`<out>/<channel-id>/`:
- a **transcript** `<video-id>.<lang>.<format>` (e.g. `dQw4w9WgXcQ.en.srt`); and
- a **metadata sidecar** `<video-id>.meta.yaml` — one per video,
language-independent, written atomically (temp file + rename).
"Already synced" is filesystem state: an existing transcript file means the
transcript is skipped. Sidecars are planned by a separate **directory scan** of
`<out>/<channel-id>/` (every transcript file is a synced video; `*.meta.yaml`
and in-flight `.*`/`*.tmp` files are ignored), so backfill and refresh cover the
full set of already-synced videos **without** re-enumerating the channel
(`--full`) and **without** touching the bot-gated transcript path. A video with
no usable transcript leaves no anchor file and so gets no sidecar.
Metadata is fetched with a single **WEB-client** `/player` call — un-gated, no
`visitorData` bootstrap — which carries the `microformat` block (publish date,
like count, category) that the `ANDROID_VR` transcript path lacks. Metadata
failures are tallied separately and never block or fail transcript syncing.
A sidecar looks like:
```yaml
schema: 1
video_id: dQw4w9WgXcQ
title: Rick Astley - Never Gonna Give You Up (Official Video) (4K Remaster)
channel: Rick Astley
channel_id: UCuAXFkgsw1L7xaCfnd5JJOw
channel_url: http://www.youtube.com/@RickAstleyYT
category: Music
published_at: 2009-10-24T23:57:33-07:00
duration_seconds: 213
description: |
The official video for "Never Gonna Give You Up" by Rick Astley.
keywords:
- rick astley
view_count: 1781429760
like_count: 19148727
is_live_content: false
is_unlisted: false
thumbnail_url: https://i.ytimg.com/vi/dQw4w9WgXcQ/maxresdefault.jpg
fetched_at: 2026-06-11T03:12:45Z
```
`fetched_at` (UTC) is the snapshot time for the point-in-time `view_count` /
`like_count`, and the staleness key for refresh. `schema: 1` versions the
format. `microformat`-sourced fields (`like_count`, `category`, `published_at`,
`is_unlisted`, …) are omitted when absent — `like_count` when ratings are
disabled, all of them for private/removed videos (the sidecar is then written
from `videoDetails` alone).
#### `sync` metadata flags
| Flag | Default | Effect |
|---------------------------------------|---------|------------------------------------------------------------------------------------------------|
| `--refresh-metadata-older-than <spec>`| — | Re-fetch sidecars whose `fetched_at` predates `<spec>`. Missing sidecars are always backfilled. |
`<spec>` accepts an absolute date (`YYYY-MM-DD`, midnight UTC), a full RFC 3339
timestamp, or a relative spec `<N> <unit>[s] ago` (units: `minute`, `hour`,
`day`, `week`, `month`, `year`) resolved against now. Without the flag, no
refresh occurs; missing sidecars are still downloaded. An invalid spec errors at
plan time.
### Locator forms
`<url>` accepts any of:
- `https://www.youtube.com/watch?v=<id>` (extra query params ignored)
- `https://youtu.be/<id>` (with optional trailing query / fragment)
- `https://www.youtube.com/shorts/<id>`
- `https://www.youtube.com/embed/<id>`
- A bare 11-character video ID like `jNQXAC9IVRw`
### Errors
Failures surface as typed variants of
[`TranscriptError`](https://docs.rs/omni-dev/latest/omni_dev/transcript/enum.TranscriptError.html)
rather than generic HTTP errors:
| Variant | When |
|-------------------------------|-----------------------------------------------------------------------|
| `InvalidLocator` | URL did not parse, or bare ID failed validation. |
| `LanguageNotFound` | No track matched `--lang` (manual or, with `--auto`, ASR). |
| `AutoCaptionsRequireOptIn` | Only ASR matched, but `--auto` was not passed. |
| `PlayabilityRefused` | Age-gated, region-locked, removed, or login-required (carries status).|
| `MissingVisitorData` | YouTube watch-page format drifted; bootstrap regex needs retuning. |
| `ParseError` | InnerTube or json3 response did not match the expected shape. |
| `Http` | Non-2xx response from YouTube. |
## Library architecture
The library lives at [`src/transcript/`](../src/transcript/) and has no
`clap` dependency — it is reusable by other commands or external consumers.
The CLI in [`src/cli/transcript/`](../src/cli/transcript/) is a thin layer
that bridges `clap` argument parsing to library types.
```
src/
transcript/ # library: no clap
cue.rs # Cue { start_ms, end_ms, text }
detect.rs # detect(url) → Box<dyn TranscriptSource>
error.rs # TranscriptError + Result alias
source.rs # TranscriptSource trait + value types
format.rs # Format enum + dispatch
format/{srt,vtt,txt,json}.rs # source-agnostic converters
sources/
youtube.rs # impl TranscriptSource for Youtube
youtube/{url,player_response,timedtext,innertube,watch_page}.rs
cli/transcript/ # CLI: clap dispatch only
mod.rs # TranscriptCommand + TranscriptSubcommands
fetch.rs # provider-less auto-detecting `fetch`
format.rs # CliFormat ↔ Format bridge
youtube/{mod,fetch,info,list_langs}.rs
```
The trait contract:
```rust
#[async_trait]
pub trait TranscriptSource: Send + Sync {
fn name(&self) -> &'static str;
fn matches(url: &str) -> bool where Self: Sized;
async fn fetch(&self, locator: &str, opts: &FetchOpts) -> Result<Transcript>;
async fn list_languages(&self, locator: &str) -> Result<Vec<LanguageInfo>>;
async fn info(&self, locator: &str) -> Result<MediaInfo>;
}
```
`matches` is `where Self: Sized` so it stays out of the `dyn` vtable —
sources can be used through `Box<dyn TranscriptSource>`. That is what powers
the auto-detecting `omni-dev transcript fetch <url>` path:
[`detect`](../src/transcript/detect.rs) probes each registered source's
`matches` in order and returns the first as a `Box<dyn TranscriptSource>`,
which the CLI then drives through the trait's `fetch`/`list_languages`/`info`
(`matches` itself is static, so `detect` names each source concretely rather
than calling through the box). Delivered in #1187.
Format converters take `&[Cue]` and never reach back into a source, so
they are reused as-is by every implementation.
## Adding a new source
Adding a source is intentionally small: one library module, one CLI
module, two single-line additions to enums. The trait, the value types,
and the format converters are not touched. The recipe below walks through
adding a stub `vimeo` source.
### 1. Library module
Create `src/transcript/sources/vimeo.rs`:
```rust
//! Vimeo TranscriptSource — stub.
use async_trait::async_trait;
use crate::transcript::error::Result;
use crate::transcript::source::{
FetchOpts, LanguageInfo, MediaInfo, Transcript, TranscriptSource,
};
/// Vimeo transcript source.
pub struct Vimeo {
http: reqwest::Client,
}
impl Vimeo {
/// Construct a Vimeo source with default HTTP settings.
pub fn new() -> Result<Self> {
Ok(Self {
http: reqwest::Client::builder().build()?,
})
}
}
#[async_trait]
impl TranscriptSource for Vimeo {
fn name(&self) -> &'static str {
"vimeo"
}
fn matches(url: &str) -> bool {
url.contains("vimeo.com/")
}
async fn fetch(&self, _locator: &str, _opts: &FetchOpts) -> Result<Transcript> {
todo!("call Vimeo's text-tracks API, parse to Vec<Cue>")
}
async fn list_languages(&self, _locator: &str) -> Result<Vec<LanguageInfo>> {
todo!()
}
async fn info(&self, _locator: &str) -> Result<MediaInfo> {
todo!()
}
}
```
Register the module by adding one line to
[`src/transcript/sources.rs`](../src/transcript/sources.rs):
```rust
pub mod vimeo;
```
Then add one probe arm to [`detect`](../src/transcript/detect.rs) so the
provider-less `transcript fetch <url>` path can route to it (order matters only
if two sources could claim the same locator — put the more specific first):
```rust
if Vimeo::matches(url) {
return Ok(Box::new(Vimeo::new()?));
}
```
That's the entire library surface. Note what is *not* needed:
- No new error variants — `TranscriptError` already covers parse / HTTP /
language-not-found / playability-refused.
- No new format converters — the four shipped formats consume `&[Cue]`.
- No changes to `TranscriptSource`, `Transcript`, `Cue`, or `FetchOpts`.
### 2. CLI module
Create `src/cli/transcript/vimeo/mod.rs` mirroring the YouTube layout
(`fetch.rs`, `info.rs`, `list_langs.rs`). Each subcommand instantiates the
source and dispatches:
```rust
//! Vimeo transcript subcommands.
pub mod fetch;
pub mod info;
pub mod list_langs;
use anyhow::Result;
use clap::{Parser, Subcommand};
#[derive(Parser)]
pub struct VimeoCommand {
#[command(subcommand)]
pub command: VimeoSubcommands,
}
#[derive(Subcommand)]
pub enum VimeoSubcommands {
Fetch(fetch::FetchCommand),
ListLangs(list_langs::ListLangsCommand),
Info(info::InfoCommand),
}
impl VimeoCommand {
pub async fn execute(self) -> Result<()> {
match self.command {
VimeoSubcommands::Fetch(cmd) => cmd.execute().await,
VimeoSubcommands::ListLangs(cmd) => cmd.execute().await,
VimeoSubcommands::Info(cmd) => cmd.execute().await,
}
}
}
```
The individual subcommand structs follow the same shape as
[`src/cli/transcript/youtube/fetch.rs`](../src/cli/transcript/youtube/fetch.rs)
— construct the source via `Vimeo::new()?`, call the trait method, and
hand the result to the same `Format::render` and `print_table` helpers
the YouTube subcommands already use.
### 3. Top-level wiring
Two single-line edits in
[`src/cli/transcript/mod.rs`](../src/cli/transcript/mod.rs):
```rust
pub mod vimeo;
pub enum TranscriptSubcommands {
Youtube(youtube::YoutubeCommand),
Vimeo(vimeo::VimeoCommand), // ← new
}
impl TranscriptCommand {
pub async fn execute(self) -> Result<()> {
match self.command {
TranscriptSubcommands::Youtube(cmd) => cmd.execute().await,
TranscriptSubcommands::Vimeo(cmd) => cmd.execute().await, // ← new
}
}
}
```
After landing CLI changes, run the
[`update-snapshots`](../.claude/skills/update-snapshots/SKILL.md) skill to
refresh
[`tests/snapshots/integration_test__help_all_output.snap`](../tests/snapshots/integration_test__help_all_output.snap).
### 4. Tests
Per [STYLE-0009](STYLE_GUIDE.md), tests live in `#[cfg(test)] mod tests`
inside each source file. The YouTube source ships a two-layer pattern that
ports cleanly to a new source:
1. **Offline parsers** — fixture-driven `#[test]` cases for URL parsing,
API-response parsing, track selection, and any source-specific format
variant. Fixtures live next to the source in `fixtures/`.
2. **HTTP layer** — `#[tokio::test]` cases that drive
`Source::with_base_url(server.uri())` against a `wiremock::MockServer`
serving the source's endpoints.
See [`src/transcript/sources/youtube.rs`](../src/transcript/sources/youtube.rs)
for the worked layout, including `expect(1)` mocks that pin caching
behaviour and golden-output round-trips that compare HTTP and offline
pipelines against the same `.srt` reference.
An online integration test against the live platform is gated on
`#[cfg(online_tests)]` (declared in `Cargo.toml`'s `[lints.rust]`), so
`cargo test` and `cargo test --all-features` neither compile nor run it.
Operators run it manually with
`RUSTFLAGS='--cfg online_tests' cargo test online_<source>_against_public_video`.
## YouTube refresh signals
The YouTube source pins client constants that drift over months as
YouTube tightens its bot-detection signals. When `/player` starts
returning empty or refused responses for known-healthy videos, refresh:
- [`CLIENT_VERSION`](../src/transcript/sources/youtube/innertube.rs) and
the matching version token in
[`USER_AGENT`](../src/transcript/sources/youtube.rs) — bump to the value
currently shipped by the Oculus YouTube app.
- [`INNERTUBE_API_KEY`](../src/transcript/sources/youtube/innertube.rs) —
refresh if the ANDROID-family key starts being rejected.
- The `visitorData` regex in
[`watch_page.rs`](../src/transcript/sources/youtube/watch_page.rs) — if
the watch-page `ytcfg.set` block changes shape, the bootstrap will
surface `MissingVisitorData` rather than silently fall through.
The `BROWSER_USER_AGENT` used for the watch-page bootstrap is independent
of the InnerTube User-Agent — they target different YouTube surfaces and
must not be conflated.
The metadata sidecar path (`sync`) reads a different surface again: the
**WEB-client** `/player` response and its `microformat.playerMicroformatRenderer`
block, parsed in
[`metadata.rs`](../src/transcript/sources/youtube/metadata.rs). This shape can
drift independently of the `ANDROID_VR` constants above. If sidecars start
coming back empty or [`metadata::parse`] begins failing for known-healthy
videos, re-check the field paths there (`publishDate`, `likeCount`, the
`microformat` nesting) against a live WEB response — count fields in particular
have flipped between JSON string and number forms before, which the parser
tolerates but is the first thing to verify.
[`metadata::parse`]: ../src/transcript/sources/youtube/metadata.rs