# drone-rs Requirements
This document defines the requirements for `drone-rs`, a Rust SDK for drone flight
control, telemetry, and communication protocols. It covers the SDK's overall scope
and architecture, then details the first feature to be implemented: **object
tracking and following**.
## 1. Background & motivation
The sibling project [`../q10-drone-controller`](../../q10-drone-controller) reverse-engineered
a consumer drone's proprietary WiFi protocol (packet capture → decode → handshake
replay) and built a Python/OpenCV/PyTorch stack on top: manual joystick control, a
custom video-frame reassembly pipeline, Lucas-Kanade obstacle detection, and a
PilotNet-style behavioural-cloning autopilot.
`drone-rs` generalizes that experience into a reusable Rust SDK: a protocol-agnostic
transport layer (starting from the concrete Q10 protocol, extensible to MAVLink-based
flight controllers later), a video/sensor pipeline, and a perception layer that swaps
PyTorch out for the [`vision-rs`](../../vision-rs) computer-vision SDK. The lessons learned
there — motors need a throttle offset before they engage, manual input must always
pre-empt autonomy, protocol timing is rigid — carry over directly into the
requirements below.
## 2. Guiding principles
- **Safety first.** Every autonomous behavior operates inside an explicit safety
envelope (max throttle/velocity/altitude) and yields immediately to manual input.
Loss of the command link or loss of a tracked target degrades to a safe state
(hover or land), never to "do nothing and hope."
- **Protocol-agnostic transport.** Flight commands and telemetry are expressed
against traits, not a specific drone's wire format. The first backend targets the
Q10-style raw UDP protocol; MAVLink (PX4/ArduPilot) is a planned second backend.
- **Pluggable perception.** Detection and tracking are traits. The SDK does not
hard-depend on any one model or inference backend — `vision-rs` is the reference
implementation for detection, not a required dependency of `drone-rs` itself.
- **Idiomatic, `unsafe`-minimal Rust.** Follow `AGENT.md` throughout: borrow over
clone, `thiserror` for library errors, no `unwrap`/`expect` outside tests, iterators
over index loops, documented `SAFETY:` invariants for any `unsafe` block.
- **Testable without hardware.** Transport and video sources are mockable so the
full command/perception/behavior pipeline can run in CI without a physical drone.
- **Small core, optional weight.** Heavy dependencies (ML inference, CUDA-backed
detectors) sit behind feature flags so the core SDK stays lightweight for users who
only need transport and telemetry.
## 3. Target platforms
- **Development/CI**: Linux x86_64.
- **Companion computers**: Raspberry Pi class and Jetson-class ARM boards (matching
`q10-drone-controller`'s Pi Zero 2 deployment target), in addition to x86_64.
- **Airframes**: consumer WiFi-protocol drones (Q10-class, informed by
`q10-drone-controller`'s reverse-engineering) for v1; MAVLink-speaking flight
controllers (PX4, ArduPilot) as a later transport backend.
## 4. System architecture overview
```
┌─────────────────────────────────────────────────────────────────────┐
│ Behaviors │
│ (object-follow, hover, future: waypoint, orbit, RTL) │
├───────────────────────────┬───────────────────────────────────────┤
│ Perception │ Safety / Failsafe │
│ (Detector, Tracker) │ (envelope clamp, watchdog, override) │
├───────────────────────────┴───────────────────────────────────────┤
│ Video/Sensor Pipeline │ Command & Control API │
│ (frame source, decode, │ (arm/disarm, takeoff/land, roll/ │
│ ring buffer) │ pitch/throttle/yaw, setpoints) │
├─────────────────────────────┴───────────────────────────────────────┤
│ Telemetry & State │
│ (attitude, battery, link quality, event log) │
├───────────────────────────────────────────────────────────────────┤
│ Transport │
│ (Q10 raw-UDP backend today; MAVLink backend planned) │
└───────────────────────────────────────────────────────────────────┘
```
Each horizontal layer is defined by traits; the Q10 backend and a mock backend are
the two initial implementations of the Transport layer, and a mock frame source is
the initial implementation of the video pipeline for tests.
## 5. Functional requirements by subsystem
### 5.1 Transport
- A `Transport` trait abstracting connection lifecycle (connect, heartbeat/keepalive,
disconnect) and command send / telemetry receive, independent of wire format.
- A concrete backend for the Q10-style protocol: magic-byte-prefixed packets,
heartbeat every ~500ms, an explicit handshake sequence, source-IP binding
requirements documented per-backend (not assumed generically).
- Automatic reconnection with backoff; a stale/lost connection surfaces as a
telemetry event, not a silent failure.
- A mock/in-memory backend for tests and simulation.
### 5.2 Command & control
- Primitives for the four core axes (roll, pitch, throttle, yaw) as normalized,
documented ranges — not raw protocol bytes — with each backend responsible for its
own byte-level mapping (e.g. the Q10 backend's empirically-measured cruise/ceiling
thresholds).
- Arm/disarm and takeoff/land where the backend supports them; throttle-only control
where it doesn't (as with the Q10, which has no discrete takeoff command).
- All outgoing commands pass through the safety envelope (§5.6) before transmission.
- Manual input (when present) always overrides autonomous/behavior-driven commands
for that control cycle.
### 5.3 Telemetry & state
- Attitude, battery level, and link quality/RSSI where the backend exposes them.
- An append-only, bounded event/state log (state transitions, failsafe triggers,
connection changes) for post-flight debugging, mirroring
`q10-drone-controller`'s event ring buffer.
- A typed state machine for connection/flight state (e.g. disconnected → handshake →
settling → ready → flying), queryable by behaviors and safety logic.
### 5.4 Video / sensor pipeline
- A `FrameSource` trait yielding decoded frames (resolution/format documented
per-source); a Q10-style backend handles the fragmented "headless JPEG" reassembly
described in `q10-drone-controller`'s README.
- A small ring buffer decoupling frame production from consumption so a slow
consumer (e.g. inference) drops frames instead of blocking capture.
- A mock frame source (static images or a recorded sequence) for tests.
### 5.5 Perception (detection & tracking)
- A `Detector` trait: given a frame, returns zero or more detections (bounding box,
class, confidence). `vision-rs` is the reference implementation; the trait itself
has no dependency on it.
- A `Tracker` trait: given a frame's detections, returns tracked objects with stable
IDs across frames (handles association, brief occlusion, and track loss).
- Both traits are feature-gated from the core SDK so a user who only needs transport
and telemetry doesn't pull in an inference stack.
### 5.6 Safety & failsafe
- A configurable safety envelope: max throttle, max velocity/altitude where
applicable, clamps every outgoing command regardless of source (manual or
autonomous).
- A command-link watchdog: if heartbeat/telemetry goes stale beyond a configurable
timeout, the SDK forces a safe state (hover if capable, otherwise a controlled
throttle ramp-down) rather than continuing to execute stale commands.
- Any active behavior (e.g. object-follow) defines its own loss-of-input failsafe
(see §6.4) in addition to the link watchdog above.
- Geofencing and obstacle avoidance are out of scope for v1 (tracked in §7) but the
safety-envelope hook is designed to accommodate them without an API break.
### 5.7 Simulation & testing
- The full pipeline (transport → telemetry → video → perception → behavior →
command) must be exercisable in CI using the mock transport and mock frame source,
with no physical drone or live model required.
- Golden/fixture-based tests for protocol encode/decode (packet layouts) per
backend, independent of live hardware.
## 6. Feature 1: Object tracking and following
The first user-facing feature built on the architecture above: point the drone's
camera at a target, and have the drone keep that target framed and at a roughly
constant distance, flying itself to do so.
### 6.1 User story
As an operator, I select (or the system auto-selects) a target in the video feed;
the drone then autonomously adjusts roll/pitch/throttle/yaw each frame to keep the
target centered and at a consistent apparent size, until I take back manual control,
the target is lost for too long, or a safety limit is hit.
### 6.2 Functional requirements
- **Target acquisition.** Support at minimum one of:
- auto-select: the highest-confidence detection of a configured class (e.g.
"person") in the first usable frame, or
- manual-select: an operator-supplied initial bounding box, associated with the
detection nearest to it.
- **Target identity.** Once acquired, the `Tracker` (§5.5) maintains the same track
ID across frames; the follow behavior always steers toward the currently-tracked
ID, not merely "the latest detection of this class."
- **Centering.** Compute a lateral/vertical error between the target's bounding-box
center and the frame center each cycle; convert that error into yaw and
throttle/pitch corrections (proportional control is sufficient for v1; no
requirement for a specific control-theory approach beyond bounded, stable output).
- **Distance-keeping.** Use bounding-box size (area or height, whichever proves more
stable in practice) as a proxy for distance; drive forward/back (pitch) to hold a
configured target size, without a depth sensor or GPS.
- **Track loss handling.** Define a grace period after the target's ID disappears
from tracker output: hold the last commanded attitude (or hover, if the backend
supports it) during the grace period; if the target hasn't reappeared by the end
of it, abort the behavior to a safe state (hover/land per §5.6) rather than
continuing to fly blind.
- **Manual override.** Any manual input immediately suspends the follow behavior for
that session; resuming requires an explicit re-engage, not merely the absence of
further manual input.
- **Safety envelope.** All commands the follow behavior issues are subject to §5.6's
clamps; the behavior itself does not bypass or widen the configured envelope.
- **Configurability.** Target class, desired bounding-box size (distance), and
centering/loss-of-track thresholds are configuration, not hardcoded.
### 6.3 Non-functional requirements
- **Latency.** The frame-capture → detect → track → command-issue loop should run
fast enough to track a slowly moving target (walking pace) without visible
overshoot on reference hardware (Raspberry Pi / Jetson class); exact numeric
budgets are to be established once a reference `Detector` implementation (e.g. a
`vision-rs` model) is benchmarked on target hardware, rather than assumed here.
- **Degraded operation.** If perception falls behind the video frame rate, the
behavior must degrade gracefully (process the latest frame, drop stale ones — see
§5.4's ring buffer) rather than accumulate lag.
- **Determinism for testing.** The follow behavior's control logic (error → command
mapping) must be unit-testable against synthetic detection/tracker sequences,
without a live camera, model, or drone.
### 6.4 Interfaces (conceptual sketch)
Illustrative only — exact signatures are decided during implementation, not fixed
by this document.
```rust
trait FrameSource {
fn next_frame(&mut self) -> Result<Frame, FrameSourceError>;
}
trait Detector {
fn detect(&mut self, frame: &Frame) -> Result<Vec<Detection>, DetectorError>;
}
trait Tracker {
fn update(&mut self, detections: &[Detection]) -> Vec<TrackedObject>;
}
trait FollowController {
/// Computes the next command from the currently tracked target, or `None`
/// if the target is within its loss-of-track grace period / lost.
fn next_command(&mut self, target: Option<&TrackedObject>) -> FollowOutcome;
}
```
### 6.5 Out of scope for v1
- Simultaneously following multiple targets.
- Obstacle avoidance while following (depends on a perception capability not yet
built; tracked as a future feature in §7).
- Gimbal control — v1 assumes a fixed, body-mounted camera and aims the whole
airframe.
- GPS-based or GPS-denied SLAM-based following — v1 is vision-only, matching the
Q10-class target hardware which has no onboard GPS.
- Orbit/circle-around-target mode (a natural follow-up once basic follow is stable).
## 7. Future features (backlog)
Not detailed here; tracked as beads issues (`br`) once prioritized:
- MAVLink transport backend (PX4/ArduPilot).
- Geofencing and obstacle avoidance.
- Waypoint/mission planning.
- Gimbal control.
- Orbit / circle-around-target follow mode.
- Return-to-home and richer failsafe policies.
- Ground control station integration.
- Multi-drone / swarm coordination.
- Black-box flight logging and replay.