zond-engine 0.17.0

The core networking library powering Zond. Raw packet injection, protocol parsing, service and OS fingerprinting, and asynchronous network scanning.
Documentation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
// Copyright (c) 2026 Erik Lening (hollowpointer) and Contributors
//
// This file is part of Zond Engine, licensed under the GNU Affero General
// Public License, version 3 or later. See the LICENSE file for details, or
// <https://www.gnu.org/licenses/agpl-3.0.html>.
//
// SPDX-License-Identifier: AGPL-3.0-or-later

//! # Reading a document as findings
//!
//! The rest of [`import`](crate::import) reads a document for the targets to scan
//! next: addresses and ports, everything else skipped. This reads the same kind
//! of document for what the scan that produced it found, and builds the
//! [`ScanReport`] that scan would have produced.
//!
//! ## Why both, and why they stay apart
//!
//! Rescanning what a document found and reading what it found are different jobs
//! with opposite instincts. The target readers are narrow on purpose:
//! [`import::json`](crate::import::json) reads four fields, which leaves the
//! exported schema free to move while nothing promises to read most of it.
//! Widening them to carry findings would spend that freedom on a different
//! caller.
//!
//! So the readers here are separate, and the two directions share only the
//! parsing machinery that has no opinion about either, meaning `xml` for the
//! documents that are XML.
//!
//! ## What this unlocks
//!
//! [`diff`](crate::diff) compares two [`ScanReport`]s and asks nothing about
//! where either came from. A scan this process ran, a scan read back out of a
//! [`journal`](crate::journal) and a scan read back out of a file are the same
//! input to it. This module puts the third in reach, and a file is what people
//! archive, where a journal is state a machine keeps and prunes.
//!
//! So: last quarter's nmap output against tonight's scan is one call, two nmap
//! files against each other is one call, and an exported report from a build
//! that has since been upgraded against a fresh one is one call.
//!
//! ## A report is not evidence that this engine produced it
//!
//! A [`ScanReport`] built here is attributed to whatever wrote the document, so
//! `nmap 7.94` rather than this crate, through
//! [`ScanReport::recorded`](crate::report::ScanReport::recorded), and
//! [`Provenance::engine_version`](crate::diff::Provenance::engine_version) hands
//! that back unchanged. Nothing downstream should read a report as proof this
//! engine's scanners ran.
//!
//! ## The same bounds, and the same refusal to open anything
//!
//! Everything the module documentation of [`import`](crate::import) says applies
//! here. A reader takes a [`BufRead`] and never opens a file,
//! [`ImportLimits`] are part of the call rather than a constant, and exceeding
//! one is an error naming what exceeded it.

#[cfg(feature = "import-json")]
pub mod json;

#[cfg(feature = "import-nmap")]
pub mod nmap;

use std::io::BufRead;
use std::path::Path;

use crate::import::{ImportError, ImportLimits};
use crate::report::ScanReport;

/// Reads a document as the report of the scan that produced it.
///
/// The mirror of [`Exporter`](crate::export::Exporter), which writes one. A reader
/// takes bytes from wherever the caller got them and returns the whole report,
/// since a report is a document with a shape rather than a stream of independent
/// records: the phase it belongs to is stated once, at the top.
pub trait ReportReader {
    /// Reads `input` as one report.
    fn read(&self, input: &mut dyn BufRead) -> Result<ScanReport, ImportError>;
}

/// What a report reader is allowed to spend.
///
/// A struct rather than a bare [`ImportLimits`] so that a policy this side needs
/// and the target side does not stays an additive change.
/// [`max_document_bytes`](Self::max_document_bytes) is the first such policy.
#[must_use]
#[non_exhaustive]
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct ReportOptions {
    /// The bounds shared with the target readers.
    ///
    /// Not every one of them means something to every reader here, because a
    /// report is a document with a shape and a target list is a stream of
    /// expressions. [`max_addresses`](ImportLimits::max_addresses) bounds the
    /// hosts a document may name and binds both readers.
    /// [`max_line_bytes`](ImportLimits::max_line_bytes) binds the line-oriented
    /// nmap reader; a JSON document is one value and has no lines to bound, so
    /// [`max_document_bytes`](Self::max_document_bytes) is what stands in its
    /// place. [`max_tokens`](ImportLimits::max_tokens) counts target
    /// expressions and a report holds none, so nothing here reads it.
    pub limits: ImportLimits,

    /// The most bytes one document may be read from.
    ///
    /// Every reader here parses a whole document before it returns one, so this
    /// bounds what an untrusted file may make the process do. It is checked as
    /// the bytes are consumed, so a document past the ceiling is refused on the
    /// way in rather than after it has been held.
    ///
    /// It bounds the document, and the process holds a multiple of it. Size
    /// this to what the process can afford rather than to a file you are willing
    /// to read. Measured, one host whose `ips` array carries four million
    /// addresses: 59.9 MB of document, 502 MB resident at the peak, 8.4 times. A
    /// `Vec<String>` of pointer-sized headers plus a heap allocation per address,
    /// then the same strings moved into the record, then a `BTreeSet<IpAddr>`
    /// built beside them before the record is freed. Nothing there is unbounded
    /// or non-linear and the ratio is a constant, but at the 256 MiB default a
    /// document this admits can leave the process holding something like 2 GB.
    ///
    /// The default is 256 MiB. An exported report of a hundred thousand hosts
    /// with their services and findings runs to a few tens of megabytes, so the
    /// ceiling is several times past anything a scan produces and still a
    /// bound. Raise it with [`with_max_document_bytes`](Self::with_max_document_bytes)
    /// for a document that has been vetted, or pass [`u64::MAX`] to lift it.
    pub max_document_bytes: u64,
}

/// 256 MiB. See [`ReportOptions::max_document_bytes`].
const DEFAULT_MAX_DOCUMENT_BYTES: u64 = 256 * 1024 * 1024;

impl Default for ReportOptions {
    fn default() -> Self {
        Self {
            limits: ImportLimits::default(),
            max_document_bytes: DEFAULT_MAX_DOCUMENT_BYTES,
        }
    }
}

impl ReportOptions {
    /// The defaults.
    pub fn new() -> Self {
        Self::default()
    }

    /// Sets the bounds.
    pub fn with_limits(mut self, limits: ImportLimits) -> Self {
        self.limits = limits;
        self
    }

    /// Sets the whole-document ceiling.
    pub fn with_max_document_bytes(mut self, bytes: u64) -> Self {
        self.max_document_bytes = bytes;
        self
    }
}

/// A reader that refuses to hand out more than `limit` bytes.
///
/// Wrapped around the caller's input at the dispatch rather than inside each
/// reader, so the ceiling holds for every format and a format added later
/// inherits it instead of having to remember it.
struct Bounded<'a> {
    inner: &'a mut dyn BufRead,
    left: u64,
    /// Whether the budget ran out, which is all the dispatch needs back. A flag
    /// rather than a distinguishable error: the refusal travels out through
    /// `serde_json` and through [`xml`](crate::import::xml), and both rewrite an
    /// I/O failure into an error of their own. Asking afterwards is exact where
    /// reading the
    /// message that came back would be a guess.
    exhausted: bool,
}

impl<'a> Bounded<'a> {
    fn new(inner: &'a mut dyn BufRead, limit: u64) -> Self {
        Self {
            inner,
            left: limit,
            exhausted: false,
        }
    }

    fn over_budget(&mut self) -> std::io::Error {
        self.exhausted = true;
        std::io::Error::new(
            std::io::ErrorKind::InvalidData,
            "the document is past its byte ceiling",
        )
    }
}

impl Bounded<'_> {
    /// Whether a reader asking for more bytes is asking for more than the
    /// budget, rather than asking how it ends.
    ///
    /// A spent budget is not by itself an overrun. A document of exactly
    /// `max_document_bytes` has been handed over whole, and a parser then asks
    /// once more because that is how it learns there is nothing after the value
    /// it read. Refusing that reading would make the ceiling refuse the largest
    /// document it is supposed to admit.
    fn overran(&mut self) -> std::io::Result<bool> {
        Ok(self.left == 0 && !self.inner.fill_buf()?.is_empty())
    }
}

impl std::io::Read for Bounded<'_> {
    fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
        if self.overran()? {
            return Err(self.over_budget());
        }
        let ceiling = usize::try_from(self.left).unwrap_or(usize::MAX);
        let take = buf.len().min(ceiling);
        let read = self.inner.read(&mut buf[..take])?;
        self.left -= read as u64;
        Ok(read)
    }
}

impl BufRead for Bounded<'_> {
    fn fill_buf(&mut self) -> std::io::Result<&[u8]> {
        if self.overran()? {
            return Err(self.over_budget());
        }
        let ceiling = usize::try_from(self.left).unwrap_or(usize::MAX);
        let available = self.inner.fill_buf()?;
        Ok(&available[..available.len().min(ceiling)])
    }

    fn consume(&mut self, amount: usize) {
        self.left = self.left.saturating_sub(amount as u64);
        self.inner.consume(amount);
    }
}

/// A document format a report can be read from.
///
/// The mirror of [`ImportFormat`](crate::import::ImportFormat) for this
/// direction, and shorter: a report is a document some scanner wrote, and only
/// the formats that carry findings appear here. There is no list format, because
/// a list of addresses is not a report of anything.
#[non_exhaustive]
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum ReportFormat {
    /// This engine's own exported JSON, as a single document.
    #[cfg(feature = "import-json")]
    Json,
    /// The same data one record per line, which is what
    /// [`export::jsonl`](crate::export::jsonl) writes.
    ///
    /// Read here as well as in [`ImportFormat`](crate::import::ImportFormat).
    /// The format exists so a scan cut short still leaves a readable file, and a
    /// file that can only be read as the
    /// targets it names is not that.
    #[cfg(feature = "import-json")]
    JsonLines,
    /// Nmap's XML, which this engine's own nmap exporter also writes.
    #[cfg(feature = "import-nmap")]
    Nmap,
}

impl ReportFormat {
    /// The format a file extension names, if it is one this build reads.
    pub fn from_extension(extension: &str) -> Option<Self> {
        match extension
            .trim_start_matches('.')
            .to_ascii_lowercase()
            .as_str()
        {
            #[cfg(feature = "import-json")]
            "json" => Some(ReportFormat::Json),
            // `ndjson` is the other name the same format goes by, matching what
            // the exporter and the target-side reader both accept.
            #[cfg(feature = "import-json")]
            "jsonl" | "ndjson" => Some(ReportFormat::JsonLines),
            #[cfg(feature = "import-nmap")]
            "xml" => Some(ReportFormat::Nmap),
            _ => None,
        }
    }

    /// The canonical file extension for this format, without a leading dot.
    pub fn extension(self) -> &'static str {
        match self {
            #[cfg(feature = "import-json")]
            ReportFormat::Json => "json",
            #[cfg(feature = "import-json")]
            ReportFormat::JsonLines => "jsonl",
            #[cfg(feature = "import-nmap")]
            ReportFormat::Nmap => "xml",
        }
    }

    /// Every report format this build can read.
    ///
    /// Front ends use this to describe their own capabilities, as
    /// [`ImportFormat::all`](crate::import::ImportFormat::all) does: a help
    /// text listing formats the binary was not built with is worse than none.
    pub fn all() -> &'static [ReportFormat] {
        &[
            #[cfg(feature = "import-json")]
            ReportFormat::Json,
            #[cfg(feature = "import-json")]
            ReportFormat::JsonLines,
            #[cfg(feature = "import-nmap")]
            ReportFormat::Nmap,
        ]
    }

    /// The format a path's extension names.
    pub fn from_path(path: &Path) -> Option<Self> {
        path.extension()
            .and_then(|extension| extension.to_str())
            .and_then(Self::from_extension)
    }

    /// The format the start of `input` implies, without consuming it.
    ///
    /// Almost one byte: a report document is an object or an element and nothing
    /// else. Not a content sniff beyond that, since which document this is
    /// belongs to the reader, and each refuses one it does
    /// not recognise by naming what it found.
    ///
    /// The exception is the two JSON shapes, which both open with a brace. The
    /// record-per-line one names itself in its first record, so that tag is what
    /// separates them, exactly as
    /// [`ImportFormat::sniff`](crate::import::ImportFormat::sniff) separates them
    /// in the other direction. Without it a record-per-line export is read as a
    /// single document, its first line parses, its hosts are never reached, and
    /// what comes back is a correctly attributed report of a scan that found
    /// nothing.
    pub fn sniff(input: &mut dyn BufRead) -> Result<Self, ImportError> {
        /// How a record-per-line document's header record names itself, written
        /// as the compact exporter writes it.
        #[cfg(feature = "import-json")]
        const REPORT_TAG: &[u8] = br#""type":"report""#;

        let available = input.fill_buf()?;
        // Excel's mark, which says nothing about the format behind it. Stripped
        // here as the target side strips it, or a document saved by a Windows
        // editor is refused as neither format before either reader sees it.
        let prefix = crate::import::without_bom(available).trim_ascii_start();

        // Bound before the arms, because a build with only one of the two
        // features has no arm to read it and an unused binding there is a
        // warning nobody can act on.
        let _ = &prefix;

        #[cfg(feature = "import-nmap")]
        if prefix.first() == Some(&b'<') {
            return Ok(ReportFormat::Nmap);
        }

        #[cfg(feature = "import-json")]
        if prefix.first() == Some(&b'{') {
            let head = &prefix[..prefix.len().min(256)];
            let tagged = head
                .windows(REPORT_TAG.len())
                .any(|window| window == REPORT_TAG);
            return Ok(if tagged {
                ReportFormat::JsonLines
            } else {
                ReportFormat::Json
            });
        }

        Err(ImportError::Malformed {
            format: "report",
            origin: crate::import::ImportOrigin::unknown(),
            message: "the input begins as neither a JSON document nor an XML one".to_string(),
        })
    }

    /// The format at `path` if its extension names one, and otherwise whatever
    /// the input begins as.
    ///
    /// The extension wins because it is what the person who saved the file meant.
    pub fn resolve(path: Option<&Path>, input: &mut dyn BufRead) -> Result<Self, ImportError> {
        match path.and_then(Self::from_path) {
            Some(format) => Ok(format),
            None => Self::sniff(input),
        }
    }

    /// Reads `input` as a report in this format.
    ///
    /// Refuses a document past
    /// [`ReportOptions::max_document_bytes`] before any reader sees the whole of
    /// it, and refuses one naming more hosts than
    /// [`ImportLimits::max_addresses`] allows.
    pub fn read(
        self,
        input: &mut dyn BufRead,
        options: ReportOptions,
    ) -> Result<ScanReport, ImportError> {
        let limit = options.max_document_bytes;
        let mut bounded = Bounded::new(input, limit);

        let read = match self {
            #[cfg(feature = "import-json")]
            ReportFormat::Json => json::JsonReportReader::new(options).read(&mut bounded),
            #[cfg(feature = "import-json")]
            ReportFormat::JsonLines => json::JsonLinesReportReader::new(options).read(&mut bounded),
            #[cfg(feature = "import-nmap")]
            ReportFormat::Nmap => nmap::NmapXmlReportReader::new(options).read(&mut bounded),
        };

        match read {
            Err(_) if bounded.exhausted => Err(ImportError::DocumentTooLarge { limit }),
            other => other,
        }
    }
}

#[cfg(all(test, feature = "import-json", feature = "import-nmap"))]
mod tests {
    use std::io::Cursor;
    use std::path::Path;

    use super::*;

    #[test]
    fn an_extension_names_the_format() {
        assert_eq!(
            ReportFormat::from_path(Path::new("engagement/scan.xml")),
            Some(ReportFormat::Nmap)
        );
        assert_eq!(
            ReportFormat::from_path(Path::new("scan.JSON")),
            Some(ReportFormat::Json)
        );
        assert_eq!(ReportFormat::from_path(Path::new("scan.txt")), None);
    }

    #[test]
    fn the_first_byte_names_the_format_when_nothing_else_does() {
        let mut json = Cursor::new(b"  {\"schema_version\":1}".as_slice());
        assert_eq!(ReportFormat::sniff(&mut json).unwrap(), ReportFormat::Json);

        let mut xml = Cursor::new(b"<?xml version=\"1.0\"?><nmaprun/>".as_slice());
        assert_eq!(ReportFormat::sniff(&mut xml).unwrap(), ReportFormat::Nmap);

        let mut neither = Cursor::new(b"192.0.2.1\n".as_slice());
        assert!(ReportFormat::sniff(&mut neither).is_err());
    }

    #[test]
    fn sniffing_leaves_the_input_where_it_found_it() {
        let document = b"<?xml version=\"1.0\"?><nmaprun/>";
        let mut input = Cursor::new(document.as_slice());

        let format = ReportFormat::sniff(&mut input).unwrap();
        let report = format.read(&mut input, ReportOptions::new());

        assert!(
            report.is_ok(),
            "a sniff must not consume the bytes the reader needs: {report:?}"
        );
    }

    /// A document a Windows editor saved is still the document it is.
    ///
    /// A sniff that stripped no mark would refuse a report saved through
    /// `Out-File` as neither format, nmap XML included, which the reader
    /// behind it reads without complaint.
    #[test]
    fn a_byte_order_mark_hides_neither_format() {
        let marked = |text: &str| {
            let mut bytes = crate::format::UTF8_BOM.to_vec();
            bytes.extend_from_slice(text.as_bytes());
            Cursor::new(bytes)
        };

        for (document, expected) in [
            (r#"{"schema_version":1}"#, ReportFormat::Json),
            (
                "{\"type\":\"report\",\"schema_version\":1}\n",
                ReportFormat::JsonLines,
            ),
            (r#"<?xml version="1.0"?><nmaprun/>"#, ReportFormat::Nmap),
        ] {
            assert_eq!(
                ReportFormat::sniff(&mut marked(document)).expect("sniffs"),
                expected,
                "{document}"
            );
        }

        // And the reader named still reads what the sniff was looking at.
        let mut input = marked(r#"<?xml version="1.0"?><nmaprun/>"#);
        let format = ReportFormat::sniff(&mut input).expect("sniffs");
        assert!(
            format.read(&mut input, ReportOptions::new()).is_ok(),
            "the format was recognised and then refused the same bytes"
        );
    }

    /// The extension is what the person who saved the file meant.
    #[test]
    fn an_extension_outranks_what_the_bytes_look_like() {
        let mut input = Cursor::new(b"{}".as_slice());
        assert_eq!(
            ReportFormat::resolve(Some(Path::new("scan.xml")), &mut input).unwrap(),
            ReportFormat::Nmap
        );
    }
}