Module verify 

Expected results: a specification that checks itself against the literature.

The expect list of a specification states what the results must be, where they come from, and how close they must be. comnum verify (or verify) runs the specification and checks every expectation on every condition it applies to:

expect:
  - name: BER of Gray 16-QAM over AWGN
    source: Proakis & Salehi, Digital Communications, 5th ed., eq. (4.3-30)
    where: {m: 16}                     # the conditions it applies to (default: all)
    metric: ber
    theory: {function: ber, kind: qam, m: $m, ebn0_db: $ebn0_db}
    interval: 0.999                    # the expected value is inside the 99.9 % interval
    rtol: 0.05                         # … or within 5 % of the estimate
    min_count: 1000                    # skip the conditions with fewer errors
  - name: Effective SNR of linear equalisation, Fig. 5 of Häger & Pfister (2021)
    where: {receiver: cdc}
    metric: snr_eff_db
    curve: {along: power_dbm, x: [-8, -6, -4], y: [17.40, 16.96, 14.50]}
    atol: 0.5

The expected value is one of:

• value: a number;
• curve: a published curve y(x) along a variable of the grid, interpolated linearly; the conditions outside the curve, or on a missing point (null), are not checked;
• reference: another metric of the same condition (a theoretical value that a block records, for instance ce_ofdm_rx.ser_impulsive);
• theory: a closed form evaluated on the condition ($name substitutes a grid variable): ber, ser (with kind, m, and ebn0_db or snr_db), mi, gmi (with kind, m, snr_db, and mb_lambda for the Maxwell-Boltzmann priors of a shaped source), and for a search the inverses ebn0_for_ber, ebn0_for_ser, snr_for_ber, snr_for_ser (with kind, m, target; see comnum_mapping::theory::ebn0_for_ber), and gn_snr_db, the SNR of the centre channel of a WDM link by the GN model (with fiber, merged over the fiber of the specification as for the fibre blocks, n_channels, spacing_hz, symbol_rate, rolloff, power_dbm and coherent, see comnum_channel::optical::gn; a raman mapping in fiber gives the GN model with the Raman power profiles; format, a constellation with mb_lambda for shaped symbols, gives the EGN model), and ber_rayleigh, the BER of BPSK or QPSK with maximal-ratio combining of branches Rayleigh branches whose mean $E_{b}/N_{0}$ is ebn0_db plus offset_db (with m, kind; see comnum_mapping::theory::ber_rayleigh_mrc), and dvbs2_esn0_db, the ideal $E_{s}/N_{0}$ of a DVB-S2 MODCOD at a PER of $10^{-7}$ (Table 13 of EN 302 307-1, with modulation, rate) plus offset_db.

On a specification with a search, the expectations are about the searched variable (metric: ebn0_db for a search along ebn0_db) and are checked against its value and confidence interval (Expectation::check_search).

Every function takes its own parameters, and an unknown one is an error (branches on ber, say); kind is read without case.

An expectation passes when the measured value $v$ satisfies $|v - e| \le a + r |e|$ (atol: a, rtol: r), or, with factor: f, when $e/f \le v \le f e$ (error rates over several decades), or, with interval: c, when $e$ lies in the confidence interval of level $c$ of the estimate; at least one of atol, rtol, factor and interval is required. min_count skips the conditions whose estimate counts fewer errors (or values), and measured_range: [lo, hi] those whose measured value is outside the range (the domain of validity of an approximation, or the values precise enough).

An expectation that checks no condition is an error, not a pass: a where that matches no condition (a misspelt variable or value), a curve outside the grid, or conditions all skipped by min_count or measured_range verify nothing, and the message names the expectation and the step that left no condition.

report is the verification report that comnum verify writes to DIR/<name>.verify.json and that the Python verify_spec returns: the header of the reports of the runs (the seed, the confidence, the version and commit of the library, the specification) and every check.

Structs

Check

The outcome of one expectation on one condition.

Expectation

An expectation of a specification, validated.

Interval

A measured value and its confidence interval.

Tolerance

The acceptance rule of a measured value $v$ against an expected value $e$: $|v - e| \le a + r |e|$ (with atol $a$ or rtol $r$), or $e/f \le v \le f e$ (with factor $f$, for error rates over several decades); either rule suffices.

Enums

Verified

What verify_spec ran: the conditions of the grid, or the search.

Functions

report

The verification report of a specification: the header of the reports of its runs (comnum_sim::loggers::run_header: the name, the seed, the batch, the threads, the stopping rule and the confidence of spec with the options opts, the version and commit of the library, the specification), passed (every check passed) and checks (one object per check: name, source, condition, metric, expected, value, lo, hi, passed), as JSON text.

verify

Check every expectation of a list on the results of a run.

verify_search

Check every expectation of a list on the operating points of a search.

verify_spec

Run a specification (its search, when it has one) and check its expectations: what comnum verify and the Python verify_spec do.

write_report

Write the verification report of a specification (report) to path, its parent directories created.