Struct Population 

pub struct Population {
    pub root: PathBuf,
    pub language: Language,
    pub statements: Vec<Statement>,
    pub tests: Vec<TestRef>,
    /* private fields */
}

Fields

root: PathBuf

language: Language

statements: Vec<Statement>

tests: Vec<TestRef>

Implementations

impl Population

pub fn with_files(self, files: BTreeMap<String, String>) -> Self

Read these files from here rather than the checkout: a run’s sources as they were assessed, once the checkout has moved on.

pub fn files_read(&self) -> BTreeMap<String, String>

Every file read so far, as it was read.

pub fn order(&self, statement: usize, k: usize) -> Vec<usize>

The tests to ask about a statement, in order: tests whose file quotes text the statement produces first, one per file; then tests in a file named like the source; then one per describe block, round-robin over files. Without the first rule, the alphabetically first files filled the sample: on supergateway 15 of 179 tests were asked about a log line, none of them the test that compares the whole startup log.

pub fn last_order( &self, statement: usize, asked: &[usize], k: usize, ) -> Vec<usize>

The tests to ask last, when none of asked was judged to fail: of the rest, those whose name shares the most words with the statement’s line, and among those the tests that ran the least code.

A statement hundreds of tests run is mostly run in passing, by end-to-end tests on their way somewhere else. The tests written for it tend to say so in their name (corsOrigin: a false value for return corsOrigin(...)) and to run little besides it. Asking a few more in the usual order found few of them; replacing some of the usual tests with these lost statements the usual tests catch.

pub fn test_code(&self, test: usize) -> String

The test’s code: the file’s first lines and the test’s own body, found by its title, comment lines left out.

pub fn helpers(&self, test: usize, questions: usize) -> BTreeMap<String, String>

Test-side files the test imports (helpers, fixtures), cut to a budget that grows with the number of questions. None for JavaScript, TypeScript and Python (see asks_within_the_function): on supergateway they were a sixth of what was sent and changed 4 verdicts in 101, two each way.

pub fn code_run(&self, test: usize, asked: &[usize]) -> BTreeMap<String, String>

The source this test ran: every statement it executed marked ▶, four code lines around each asked statement, comment lines left out, within a budget that grows with the number of questions. Two lines around every executed statement and eight around each asked one, in a budget four times this, cost 28% more over 3,532 checked statements in 14 projects and were no more accurate.

In JavaScript, TypeScript and Python that is cut to the function each asked statement is in (see asks_within_the_function): what the test ran elsewhere in the file, and in other files, is left out.

pub fn questions(&self, statement: usize) -> Vec<String>

The questions for one statement: one per change (two for an if).

pub fn request(&self, test: usize, asked: &[usize], model: &str) -> Value

The request for one test and the statements asked about it.

pub fn pair_key( &self, statement: usize, test: usize, model: &str, salt: &str, ) -> String

What Jev sees for one statement asked of one test, line numbers aside: the same code renumbered is the same question. Equal keys, equal question. salt carries what changes every answer at once: the run’s dependency and configuration fingerprints.