gis_puller
==========
[![CI](https://github.com/pinkforest/gis_puller/actions/workflows/CI.yml/badge.svg)](https://github.com/pinkforest/gis_puller/actions/workflows/CI.yml)
[![Crates.io](https://img.shields.io/crates/v/gis_puller.svg)](https://crates.io/crates/gis_puller)
[![Docs](https://docs.rs/gis_puller/badge.svg)](https://docs.rs/gis_puller)
[![Deps](https://deps.rs/repo/github/pinkforest/gis_puller/status.svg)](https://deps.rs/repo/github/pinkforest/gis_puller)
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![License](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
![MSRV](https://img.shields.io/badge/MSRV-1.47.0-blue)
This Rust crate aims to abstract and implement a general global Geographic Information System (GIS) data catalog and acquisition functionality.
Acquired catalog will follow the data provider derived structural separations as-is.
To build on top the below crates can be in future used to consume the acquired data and beyond:
- gis_loader will be available to load/transform these acquired dataset(s) by this crate into various data stores
- gis_api will be available to provide gRPC API to consume the loaded dataset(s)
Everything should be considered unstable including the interface definitions.
## GIS Data Sources Supported
- Australia - Geoscape consumed via https://data.gov.au/
## GIS DataSets Catalogued
- Suburb/Locality Boundaries: Australia (GDA94, GDA2020, GeoJSON)
- Address Forward/Reverse Lookup Data: Australia (GNAF)
Also see DIFFICULTIES.md
## Environment Variables
- USE_MODE = Environment "default" gets merged with either "development" | "integration" | "production" runtime configuration
- GIS_CONFIG_PATH = Filesystem path where the GIS datasource configuration files live
- TEST_DATA_PATH = Filesystem path where the mocks read data from including the "replay" recordings
## Usage
Using Rust Cargo - requires [cargo-edit](https://crates.io/crates/cargo-edit)
```bash
cargo add gis_puller
```
Or edit Cargo.toml by hand:
```toml
[dependencies]
gis_puller = "0.1.0"
```
Create documentation
```bash
env GIS_CONFIG_PATH=config RUN_MODE=development cargo doc --no-deps --open
```
Run tests (includes doctests and runs examples) with either replay or mocked datasources
```bash
env GIS_CONFIG_PATH=config TEST_DATA_PATH=test_data RUN_MODE=development cargo test
```
Run [clippy](https://github.com/rust-lang/rust-clippy) for additional lints
```bash
cargo clippy --all-targets --all-features
```
Run [tarpaulin](https://github.com/xd009642/tarpaulin) for code/examples test coverage
```bash
env GIS_CONFIG_PATH=config TEST_DATA_PATH=test_data RUN_MODE=development cargo +nightly tarpaulin --run-types Tests,Doctests,Benchmarks,Examples,Lib,Bins -v
```
NOTE: __Tarpaulin requires +nightly: rustup toolchain install nightly__
@TODO binary crate -- To use in shell with gis_puller image Docker e.g. with Australia localities boundaries datasets obtained from data.gov.au:
```bash
docker run --rm -t -i -e env GIS_PULLER_CONFIGPATH=/gis/etc/gis_puller GIS_PULLER_DATAPATH=/gis/data gis_puller gis_puller pull --all au locality/boundary
```
Or to use as a library:
- TBD: [Interfaces](INTERFACES.md) to appear in future gis_puller::Puller
Currently with AU locality/boundary dataset I doctest fetcher_matcher the below in development the below with recorded result replayed as a mock:
```rust
mod mocks;
use mocks::http_server;
use httpmock::Method::GET;
use tokio::task;
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
}).await?.unwrap();
// @TODO: replace this mess with a macro_rules!
let mut au_fetcher_matcher = (*au_config).fetcher_matcher.clone();
let mock_server = mocks::http_server::serve().await?;
let mock_path = format!("/replay/data.gov.au/api/v0/search/datasets");
let mock_query = format!("localities-boundaries");
let inject_replay = format!("test_data/replay/au/locality_boundaries.json");
let inject_url = format!("http://localhost:{}{}", mock_server.port(), &mock_path);
au_fetcher_matcher.rest_url = Some(inject_url);
au_fetcher_matcher.query = Some(mock_query.clone());
let mock = mock_server.mock(|when, then| {
when.method(GET)
.path(&mock_path)
.query_param("query", &mock_query);
then.status(200)
.header("Content-Type", "application/json")
.body_from_file(&inject_replay);
});
let catalog = gis_puller::au::fetcher_matcher(&au_fetcher_matcher).await?;
println!("Catalog = {:#?}", catalog);
Ok(())
}
```
Scratchpad for future interface:
```ignore
let puller_au = gis_puller::Puller::load('au').await?;
let sources_au = puller_au.sources('locality/boundary').await?;
// make sources_au Iterable for .iter()
sources_au.pull_all('/tmp').await?;
sources_au.pull_one('nsw').await?;
```
[cargo-edit]: https://crates.io/crates/cargo-edit
The below features can be toggled via Cargo:
- 'xxx': Enables xxx
## License
Licensed under either of:
* Apache License, Version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0)
* MIT license ([LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT)
### Contribution
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.