Birli
A Rust library for common preprocessing tasks performed in the data pipeline of the Murchison Widefield Array (MWA), located on the land of the Wajarri Yamatji people in Murchison Shire, Western Australia.
Birl reads MWA correlator visibilities in the gpufits file format using mwalib, which supports the existing "legacy" MWA correlator, as well as the in-development "MWAX" correlator.
Birli is the Wajarri word for lightning, a common cause of outages at the MWA, and a great descriptor for the speed which this library intends to deliver.
Installation
Prerequisites
- A Rust compiler with a version >= 1.51.0 - https://www.rust-lang.org/tools/install
- AOFlagger >= 3.0 (Ubuntu > 21.04: apt install aoflagger-dev)
- CFitsIO >= 3.49 (Ubuntu > 20.10: apt install libcfitsio-dev)
- LibERFA >= 1.7.1 (Ubuntu > 20.04: apt install liberfa-dev)
for OS-specific instructions, check out the linux and macOS CI Scripts; the Makefile.toml; and the Dockerfile as these are tested regularly. The instructions below may be updated less frequently, but are better documented.
(Debian/Ubuntu) Linux Setup
# Prerequisites for rustup, cargo and cargo-make
# Run the Rustup install script, profile=default, toolchain=stable
|
# Cargo make uses Makefile.toml to automate development tasks
# Use multiple cores when compiling C/C++ libraries
# Install prerequisite C/C++ libraries
# Ensure that rust can find the C/C++ libraries.
# AOFlagger and CFitsIO default to /usr/local/lib,
# however packages installed with apt (LibERFA) end up in /usr/lib/x86_64-linux-gnu/,
# so we need both.
MacOS Setup
# Install homebrew
# Run the Rustup install script, profile=default, toolchain=stable
|
# Cargo make uses Makefile.toml to automate development tasks
# Add the MWATelescope homebrew tap
# Install prerequisite libraries
Windows Setup
Unfortunately most of the prerequisites aren't available on Windows. However, WSL is great, and there is a docker image! You could use VSCode remote for WSL or Docker. Your best best is Ubuntu LTS
Installing the binary
This creates a birli binary in $HOME/.cargo/bin
Troubleshooting
Test suite
Having issues with Birli? run the test suite to narrow down your issue.
Dependencies
Experiencing segfaults? I can guarantee it's because of one of the C library dependencies. Make sure you have the right versions of all the libraries. These are specified in Prerequisites.
Get library versions with:
# cfitsio: ???
If you have something like CASA installed from apt, it's going to put an
ancient cfitsio library version in /usr/lib/x86_64-linux-gnu/, to get around
this, you must export LD_LIBRARY_PATH=/usr/local/lib/:/usr/lib/x86_64-linux-gnu/ in the shell so that Birli can find the correct library version.
Logging
You can enable additional logging on individual Rust modules by setting the RUST_LOG environment variable. For example:
RUST_LOG=trace RUST_LOG=birli=debug RUST_LOG=birli::io=error For more examples, see the env_logger docs
The default log level in info
Docker
Couldn't get it working on your environment? You can always run Birli in Docker
Want to open a shell within a fully provisioned Birli development environment? Easy!
Note: This mounts the current directory to /app in the Docker image, meaning both of these systems share the same
target folder. so if your host system is a different
architecture than Docker, you may need to cargo clean each time you switch between these environments. You may also want to temporarily disable any linters or language servers that use
the trait bound Jones<f32>: AbsDiffEq<_> is not satisfied
if you see an error that looks like this:
error[E0277]: the trait bound `Jones<f32>: AbsDiffEq<_>` is not satisfied
--> src/corrections.rs:1029:9
|
1029 | / assert_abs_diff_eq!(
1030 | | *jones_array.get((3, 3, 1)).unwrap(),
1031 | | &Jones::from([
1032 | | Complex::new(rot_1_xx_3_3_re, rot_1_xx_3_3_im),
... |
1036 | | ])
1037 | | );
| |__________^ the trait `AbsDiffEq<_>` is not implemented for `Jones<f32>`
|
= note: this error originates in the macro `abs_diff_eq` (in Nightly builds, run with -Z macro-backtrace for more info)
try
Usage
birli -h
USAGE:
birli [SUBCOMMAND]
FLAGS:
-h, --help Prints help information
-V, --version Prints version information
SUBCOMMANDS:
aoflagger flag visibilities with aoFlagger
help Prints this message or the help of the given subcommand(s)
birli aoflagger -h
flag visibilities with aoFlagger
USAGE:
birli aoflagger [FLAGS] [OPTIONS] <fits-files>... -m <metafits>
FLAGS:
--emulate-cotter Use Cotter's value for array position instead of MWAlib for direct comparison with
Cotter.
-h, --help Prints help information
--no-cable-delay Do not perform cable length corrections.
--no-geometric-delay Do not perform geometric length corrections.
-V, --version Prints version information
OPTIONS:
-f <flag-template> Sets the template used to name flag files. Percents are substituted for the zero-prefixed
GPUBox ID, which can be up to 3 characters log. Similar to -o in Cotter. Example:
FlagFile%%%.mwaf
-m <metafits> Sets the metafits file.
-u <uvfits-out> Filename for uvfits output. Similar to -o in Cotter. Example: 1196175296.uvfits
ARGS:
<fits-files>...
Cable Delay Corrections
Cable delay correction involves adjusting visibility phases to correct for the differences in electrical length of the cable between each tile and it's receiver.
Legacy MWA correlator observations do not typically have cable delays applied, however MWAX observations can. The CABLEDEL key in the metafits describes what geometric delays have been applied.
By default, Birli will apply cable length corrections. You can use --no-cable-delay to disable this.
A baseline's cable lengths are determined by the difference between a baseline's rfInput electrical lengths, as specified the the TILEDATA HDU of the metafits. Complex visibilities are phase-shifted by an angle determined by the electrical length, and the channel's frequency.
let angle = -2.0 * PI * electrical_length_m * freq_hz / SPEED_OF_LIGHT_IN_VACUUM_M_PER_S;
Geometric Delay Corrections (AKA Phase Tracking)
Geometric correction involves adjusting visibility phases to correct for the differences in distance that light from the phase center has to travel to reach each tile.
Legacy MWA correlator observations are not typically phase tracked, however MWAX observations can have phase tracking applied. The GEODEL card in the metafits describes what geometric delays have been applied.
By default, Birli will apply geometric corrections at the phase center if they have not already been applied. It determines the observations phase center from the RAPHASE and DECPHASE cards in the metafits. If these are not available, the pointing center cards (RA and DEC) from the metafits are used. You can use --no-geometric-delay to disable this.
A baseline's geometric length is determined by the w component of it's UVW fourier-space vector, after applying precession and nutation to it's tiles' positions and the phase center to the J2000 epoch, accounting for stellar aberration. Complex visibilities are phase-shifted by an angle determined by the w-component, and the channel's frequency.
let angle = -2.0 * PI * uvw.w * freq_hz / SPEED_OF_LIGHT_IN_VACUUM_M_PER_S;
Cotter Emulation
The --emulate-cotter flag ensures that outputs match Cotter as much as possible. You should only use this flag if you need to perform a direct comparison with Cotter.
By default, Birli will use the MWA array position from MWALib in order to calculate UVWs and geometric corrections. This is more accurate than the one that Cotter uses, and is the main source of error when doing direct comparisons.
This flag is used as part of the tests in src/main.rs to validate that Birli's output matches that of Cotter to within an acceptable margin.
Example: RFI Flagging, corrections and UVFits/mwaf output
In this example, we use the aoflagger subcommand to:
- Perform RFI flagging using the MWA-default flagging strategy
- Perform geometric and cable length corrections
- Output flags to .mwaf (
-f) - Output visibilities to .uvfits (
-u)
Since Cotter can't output flags and uvfits at the same time, the equivalent Cotter commands would be:
# output flags
# output uvfits
Contributing
Pull requests are welcome! Please do your best to ensure that the high standards of test coverage are maintained.
Before each commit, use cargo make ci to ensure your code is formatted correctly.
Acknowledgement
This scientific work uses data obtained from the Murchison Radio-astronomy Observatory. We acknowledge the Wajarri Yamatji people as the traditional owners of the Observatory site.
Coverage
This repo is approved by...
Release Checklist
- pipeline is green
- update
RELEASES.md - update
package.versioninCargo.toml - commit
-
git tag -a $tag -m $tag -
git push -
git push --tags