Crate bears_health

Expand description

The check module contains unit and integration tests for the library.

The bears library constructs collections of valid parameter values by downloading the associated files from the BEA server into the BEA_DATA directory. Set the value of the BEA_DATA environmental variable in a .env file at the project root.

In order to construct API requests, we first must download the range of valid parameter values for each dataset. The minimum set of steps to gather the valid parameter values is:

use bears::check;

// Download valid parameter values into the BEA_DATA directory.
fn main() -> Result<(), bears::BeaErr> {
    // Get a list of available datasets.
    check::datasets_to_json()?;
    // Get the parameter names for each dataset.
    check::parameters_to_json()?;
    // Get the valid values for each parameter name.
    check::parameter_values_to_json()?;
    // Get the valid values for GDPbyIndustry filtered by table id.
    check::values_gdp_filtered()?;
    // Get the valid values for UnderlyingGDPbyIndustry filtered by table id.
    check::values_ugdp_filtered()?;
    Ok(())
}

Once the range of valid parameter values is available in the BEA_DATA directory, you can download the data for each dataset into the data folder.

use bears::check;

// Download data for each dataset into the `data` folder of the `BEA_DATA` directory.
fn main() -> Result<(), bears::BeaErr> {
    // No pre-existing download history.  Try every combo and see what is avialable.
    check::datasets_download_initial()?;
    // A download history exists.  Use it to download successes and ignore calls that will fail.
    check::datasets_download_with_history()?;
    Ok(())
}

§Submodules

The check module is divided into a series of private submodules, documented here:

§Datasets

The parameter values for datasets indicate the valid range of dataset names. There are three unit tests associated with datasets:

datasets_to_json downloads the valid values and saves the response to datasets.json using the GetDataSetList BEA method.
datasets_from_file deserializes datasets.json into the Dataset type.
check_datasets validates that all returned dataset values have corresponding variants in the Dataset type.

Running datasets_to_json is sufficient to provide the program with the information necessary to construct API calls. The datasets_from_file test confirms that deserialization will succeed, and remains in the testing suite to detect any regression. The check_datasets test will fail if the BEA adds any new datasets to their API, providing an alert when the bears library has become out of date.

§Parameters

The BEA GetParameterList method contains the set of parameter names associated with each dataset. Used to further query valid values for each name using the GetParameterValues or GetParameterValuesFiltered method.

There are three unit tests for the GetParameterList method:

parameters_to_json downloads valid values for each dataset, saving the responses to the parameters folder in the BEA_DATA directory.
parameters_from_file deserializes the JSON files in the parameters folder in the BeaResponse type.
parameter_names converts each ParameterName variant into a string slice using the to_string implementation, then from a string slice back into a ParameterName using the FromStr implementation.

The parameters_from_file test confirms that deserialization will succeed, and remains in the testing suite to detect any regression.

§Parameter Values

These tests obtain valid values for parameter names in a given dataset using the BEA GetParameterValues method.

Primary tests for this suite:

parameter_values_to_json downloads valid values for each dataset, saving the responses to the parameter_values folder of the BEA_DATA directory.
parameter_values_from_file deserializes the JSON files in the parameter_values folder of the BEA_DATA directory into the BeaResponse type.

§Values

The values suite contains additional tests for the GetParameterValues and GetParameterValuesFiltered methods. values is an admittedly bad names. Most of the names in the check module are problematic in that they simply add an “s” to the end of an existing type module, making typos surprisingly easy.

Key tests in the values suite:

[`values_filtered_subset’] downloads the subset of parameter values confirmed return data.
values_filtered attempts to download every parameter name. Not all names are implemented, so some requests will not succeed.
values_gdp_filtered downloads industry and year values for the GDPbyIndustry dataset filtered by table id.
values_ugdp_filtered downloads industry and year values for the UnderlyingGDPbyIndustry dataset filtered by table id.

The JSON files returned by the values_filtered_subset method contain the same information produced by parameter_values_to_json. We default to reading from the parameter_values folder populated by the parameter_values_to_json function.

The values_gdp_filtered function is required to construct API calls for the GDPbyIndustry dataset, and the values_ugdp_filtered function is required to construct API calls for the UnderlyingGDPbyIndustry dataset.

In addition:

The api_error function deserializes a BEA API error as a BeaResponse. Present to detect regressions during testing.
The requests_exceeded function deserializes a BEA requests exceeded API error as a BeaResponse. Present to detect regressions during testing.

§Data

Key tests in the data suite:

datasets_download_initial downloads data for each dataset into the data folder of the BEA_DATA directory. Tries every permutation of parameter values, including combinations that are not implemented. An exploratory download used to discover new datasets. The sizes of downloads in this method are unknown, so requests may exceed the 100MB per minute rate limit set by the BEA server. At present the program will abort upon receiving a Results::RequestsExceeded status.
datasets_download_with_history downloads data for each dataset into the data folder of the BEA_DATA directory, using the download History to select the subset of implemented endpoints from the set of possible API calls. Since the targets have known size, the request rate is metered to prevent exceeding the 100MB per minute download limit set by the BEA server.
datasets_initial_load loads all successful records from the download History. Used for initial reads when a load History is not available.
datasets_initial_load_continued loads all successful records from the download Hisotry that are not already present in the load History.
datasets_retry_load attempts to reload failures in the load History.

Use the datasets_download_initial and datasets_download_with_history functions to download data. The datasets_initial_load* and datasets_retry_load functions are used internally to detect regressions during testing.

In addition:

Use download_history to inspect a download history by printing it to the console. Used during initial development and potential future schema changes.
Use [naics] to confirm the naics_codes.csv file is present in the BEA_DATA directory and loads without issue. Present to detect regressions during testing.

§Queues

inspect_queues generates the request queue for each dataset and reports its length to the console. Used to verify that iterators generate sets of the expected length.

§Histories

download_summary prints summary statistics from the download history for each dataset. Used to generate the numbers for the progress report in the readme.md file.

Structs§

FixedAssetKeys: Contains the value sets for each field in [FixedAssetDatum] contained in a [FixedAssetData].
GdpKeys: Holds unique values from each field of GdpDatum in a BTreeSet for comparison between expected and observed values in BEA parameters and responses.
IipKeys: Holds unique values from each field of IipDatum in a BTreeSet for comparison between expected and observed values in BEA parameters and responses.
IoKeys: Holds unique values from each field of InputOutputDatum in a BTreeSet for comparison between expected and observed values in BEA parameters and responses.

Traits§

Set

Functions§

api_error: Calls a known bad combination of parameters to generate an API Error as a response. Writes the JSON representation of the error to the BEA_DATA directory.
check_aoc_sta: Checks that each “AreaOrCountry” parameter value matches an enum variant in AocSta. Returns an error if a value does not match a known variant.
check_components: Checks that each “Component” parameter value matches an enum variant in Component. Returns an error if a value does not match a known variant.
check_datasets: Checks that each dataset returned from the call matches an enum variant in Dataset Returns an error if the datasets do not match.
check_indicators: Checks that each “Indicator” parameter value matches an enum variant in Indicator. Returns an error if a value does not match a known variant.
check_investments: Checks that each “TypeOfInvestment” parameter value matches an enum variant in Investment. Returns an error if a value does not match a known variant.
check_naics_category: Checks that each Naics category code matches a variant of NaicsCategory. The category code is the leading four digits in the naics industry code. Returns an error if a value does not match a known variant.
check_naics_industry: Checks that each Naics industry code matches a variant of NaicsIndustry. The industry code is the full six-digit naics industry code. Returns an error if a value does not match a known variant.
check_naics_sectors: Checks that each Naics sector code matches a variant of NaicsSector. The sector code is the leading two digits in the industry code. Returns an error if a value does not match a known variant.
check_naics_subcategory: Checks that each Naics subcategory code matches a variant of NaicsSubcategory. The subcategory code is the leading five digits in the naics industry code. Returns an error if a value does not match a known variant.
check_naics_subsectors: Checks that each Naics subsector code matches a variant of NaicsSubsector. The subsector code is the leading three digits in the naics industry code. Returns an error if a value does not match a known variant.
data_from_json
data_to_json: Pings the BEA API.
datasets_download_initial: Attempt to download all configurations using the associated iterator for the dataset. Cannot meter by file size, susceptible to exceeding the 100MB per minute rate limit of the BEA server.
datasets_download_mne_initial
datasets_download_with_history: Download existing files of a known size from the download History. Metered to prevent exceeding the 100MB per minute rate limit set by the BEA server.
datasets_from_file: reads response and native format from file avoids making api calls to bea used to test internal parsing of responses
datasets_initial_load: Attempts to load all files in the download History, without respect to the load History.
datasets_initial_load_continued: Attempts to load all files in the download History that are not yet in the load History.
datasets_retry_load: Attempts to reload errors in the load History. Run on TRACE level to gather more data on specific a specific file.
datasets_to_json: Queries valid parameter values for the dataset parameter. Reads response to json using serde_json. Saves the result as dataset.json in the BEA_DATA directory. Pings the BEA API.
debug_gdpbyindustry: Specialty function for debugging deserialization errors with the GDPbyIndustry tables. Attempts to load each file in the download history. Move the problematic files into the download history (backing up as needed).
difference
download_history
download_summary: Prints summary data from the Load History of the currently implemented datasets. Used for reporting and to assess storage requirements.
inspect_queues
missing
next_mne_error
parameter_names: The parameter_names test verifies that translation to and from &str is idempotent.
parameter_value_filtered
parameter_value_from_json: Load parameter values from the file at path and parse into the BeaResponse type.
parameter_values_from_file: Load parameter values from the parameter_values folder of the BEA_DATA and serialize into the BeaResponse type.
parameter_values_to_json: Download valid parameter values for each dataset from the BEA server using the GetParameterValues method into the parameter_values folder of the BEA_DATA directory.
parameters_from_file: reads response and native format from file avoids making api calls to bea used to test internal parsing of responses
parameters_to_json: For each variant of Dataset, request the parameters. Write the results in JSON format to the BEA_DATA directory.
params
requests_exceeded: Attempts to parse the file requests_exceeded.json in the BEA_DATA directory to the BeaResponse type. Verifies that the program accurately recognizing this error condition from the BEA server.
unused
values_filtered: For each variant of [Dataset], request the valid range of values for each parameter name. BEA has not implemented this method for all parameters, so we expect some calls to fail.
values_filtered_subset: For each variant of [Dataset], request the valid range of values for each parameter name. The subset variant of this method only requests data for datasets where the BEA has implemented a response for each parameter name associated with the dataset.
values_gdp_filtered: Two parameters in the GdpByIndustry dataset have valid input sets that vary by table_id, namely Year and Industry. Obtain table ids using Method::GetParameterValues prior to running this check. For these two parameters, we obtain a response for each table_id and write the result to a folder in the BEA_DATA directory.
values_ugdp_filtered: Two parameters in the UnderlyingGdpByIndustry dataset have valid input sets that vary by table_id, namely Year and Industry. Obtain table ids using Method::GetParameterValues prior to running this check. For these two parameters, we obtain a response for each table_id and write the result to a folder in the BEA_DATA directory.