Birli
A Rust library for common preprocessing tasks performed in the data pipeline of the Murchison Widefield Array (MWA), located on the land of the Wajarri Yamatji people in Murchison Shire, Western Australia.
Birl reads MWA correlator visibilities in the gpufits file format using mwalib, which supports the existing "legacy" MWA correlator, as well as the in-development "MWAX" correlator.
Birli is the Wajarri word for lightning, a common cause of outages at the MWA, and a great descriptor for the speed which this library intends to deliver.
Installation
Prerequisites
- A Rust compiler with a version >= 1.58.0 - https://www.rust-lang.org/tools/install
- AOFlagger >= 3.0 (Ubuntu > 21.04: apt install aoflagger-dev)
- CFitsIO >= 3.49 (Ubuntu > 20.10: apt install libcfitsio-dev)
- LibERFA >= 1.7.1 (Ubuntu > 20.04: apt install liberfa-dev)
for OS-specific instructions, check out the linux and macOS CI Scripts; the Makefile.toml; and the Dockerfile as these are tested regularly. The instructions below may be updated less frequently, but are better documented.
(Debian/Ubuntu) Linux Setup
# Prerequisites for rustup, cargo and cargo-make
# Run the Rustup install script, profile=default, toolchain=stable
|
# Cargo make uses Makefile.toml to automate development tasks
# Use multiple cores when compiling C/C++ libraries
# Install prerequisite C/C++ libraries
# Ensure that rust can find the C/C++ libraries.
# AOFlagger and CFitsIO default to /usr/local/lib,
# however packages installed with apt (LibERFA) end up in /usr/lib/x86_64-linux-gnu/,
# so we need both.
MacOS Setup
# Install homebrew
# Run the Rustup install script, profile=default, toolchain=stable
|
# Cargo make uses Makefile.toml to automate development tasks
# Add the MWATelescope homebrew tap
# Install prerequisite libraries
Windows Setup
Unfortunately most of the prerequisites aren't available on Windows. However, WSL is great, and there is a docker image! You could use VSCode remote for WSL or Docker. Your best best is Ubuntu LTS
Installing the binary
This creates a birli
binary with the aoflagger
feature enabled in $HOME/.cargo/bin
Troubleshooting
Test suite
Having issues with Birli? run the test suite to narrow down your issue.
Dependencies
Experiencing segfaults? I can guarantee it's because of one of the C library dependencies. Make sure you have the right versions of all the libraries. These are specified in Prerequisites.
Get library versions on linux with:
If you have something like CASA installed from apt, it's going to put an
ancient cfitsio library version in /usr/lib/x86_64-linux-gnu/
, to get around
this, you must export LD_LIBRARY_PATH=/usr/local/lib/
in the shell so that Birli can find the correct library version.
Logging
You can enable additional logging on individual Rust modules by setting the RUST_LOG
environment variable. For example:
RUST_LOG=trace RUST_LOG=birli=debug RUST_LOG=birli::io=error
For more examples, see the env_logger docs
The default log level in info
Docker
Couldn't get it working on your environment? You can always run Birli in Docker
Want to open a shell within a fully provisioned Birli development environment? Easy!
Note: This mounts the current directory to /app
in the Docker image, meaning both of these systems share the same
target
folder. so if your host system is a different
architecture than Docker, you may need to cargo clean
each time you switch between these environments. You may also want to temporarily disable any linters or language servers that use
Singularity on HPC
# - load the singularity module
# - cd into your preferred sif file location, e.g. /pawsey/mwa/singularity/birli
# - create a .sif file from the latest mwatelescope/birli docker image
# - run birli within the singularity image
Singularity on HPC (debug mode)
This will give you much more information about any problem you're having with Birli, however the debug build is not optimised, and is much slower.
# - request an interactive HPC session
# - load the singularity module
# - cd into your preferred sif file location, e.g. /pawsey/mwa/singularity/birli
# - create a .sif file from the latest mwatelescope/birli docker image
# - run birli within the singularity image
then within this shell
# - enable lots of logs
# - run birli in debug mode with GDB
# > run
Usage
birli -h
USAGE:
birli [OPTIONS] --metafits <PATH> <PATHS>...
OPTIONS:
--apply-di-cal <PATH> Apply DI calibration solutions to the data before averaging
--dry-run Just print the summary and exit
--emulate-cotter Use Cotter's array position, not MWAlib's
-h, --help Print help information
--no-draw-progress do not show progress bars
--phase-centre <RA> <DEC> Override Phase centre from metafits (degrees)
--pointing-centre Use pointing instead phase centre
-V, --version Print version information
INPUT:
-m, --metafits <PATH> Metadata file for the observation
<PATHS>... GPUBox files to process
SELECTION:
--no-sel-autos [WIP] Deselect autocorrelations
--no-sel-flagged-ants [WIP] Deselect flagged antennas
--sel-time <MIN> <MAX> [WIP] Timestep index range (inclusive) to select
RESOURCE LIMITS:
--max-memory <GIBIBYTES> [WIP] Estimate --time-chunk with <GIBIBYTES> GiB each chunk.
--time-chunk <STEPS> [WIP] Process observation in chunks of <STEPS> timesteps.
FLAGGING:
--flag-antennae <ANTS>... [WIP] Flag antenna indices
--flag-autos [WIP] Flag auto correlations
--flag-coarse-chans <CHANS>... [WIP] Flag additional coarse chan indices
--flag-dc [WIP] Force flagging of DC centre chans
--flag-edge-chans <COUNT> [WIP] Flag <COUNT> fine chans on the ends of each coarse
--flag-edge-width <KHZ> [WIP] Flag bandwidth [kHz] at the ends of each coarse chan
--flag-end <SECONDS> [WIP] Flag seconds before the last provided time
--flag-end-steps <COUNT> [WIP] Flag <COUNT> steps before the last provided
--flag-fine-chans <CHANS>... [WIP] Flag fine chan indices in each coarse chan
--flag-init <SECONDS> [WIP] Flag <SECONDS> after first common time (quack time)
--flag-init-steps <COUNT> [WIP] Flag <COUNT> steps after first common time
--flag-times <STEPS>... [WIP] Flag additional time steps
--no-flag-dc [WIP] Do not flag DC centre chans
--no-flag-metafits [WIP] Ignore antenna flags in metafits
CORRECTION:
--no-cable-delay Do not perform cable length corrections
--no-digital-gains Do not perform digital gains corrections
--no-geometric-delay Do not perform geometric corrections
--passband <PATH> [WIP] Apply passband corrections from <PATH>
AVERAGING:
--avg-freq-factor <FACTOR> Average <FACTOR> channels per averaged channel
--avg-freq-res <KHZ> Frequency resolution of averaged data
--avg-time-factor <FACTOR> Average <FACTOR> timesteps per averaged timestep
--avg-time-res <SECONDS> Time resolution of averaged data
OUTPUT:
-f, --flag-template <TEMPLATE> The template used to name flag files. Percents are substituted
for the zero-prefixed GPUBox ID, which can be up to 3
characters long. Example: FlagFile%%%.mwaf
-M, --ms-out <PATH> Path for measurement set output
-u, --uvfits-out <PATH> Path for uvfits output
AOFLAGGER:
--aoflagger-strategy <PATH> Strategy to use for RFI Flagging
--no-rfi Do not perform RFI Flagging with aoflagger
Note: the aoflagged options are only available when the aoflagger feature is enabled.
Operations are performed in the order described by the following sections.
Cable Delay Corrections
Cable delay correction involves adjusting visibility phases to correct for the differences in electrical length of the cable between each tile and it's receiver.
Legacy MWA correlator observations do not typically have cable delays applied, however MWAX observations can. The CABLEDEL
key in the metafits describes what geometric delays have been applied.
By default, Birli will apply cable length corrections. You can use --no-cable-delay
to disable this.
A baseline's cable lengths are determined by the difference between a baseline's rfInput electrical lengths, as specified the the TILEDATA
HDU of the metafits. Complex visibilities are phase-shifted by an angle determined by the electrical length, and the channel's frequency.
let angle = -2.0 * PI * electrical_length_m * freq_hz / SPEED_OF_LIGHT_IN_VACUUM_M_PER_S;
RFI Flagging.
By default, Birli will flag the data using the default MWA strategy in AOFlagger. You can use the
--no-rfi
option to disable this, or the --aoflagger-strategy
option to proived your own strategy
file.
Geometric Delay Corrections (AKA Phase Tracking)
Geometric correction involves adjusting visibility phases to correct for the differences in distance that light from the phase center has to travel to reach each tile.
Legacy MWA correlator observations are not typically phase tracked, however MWAX observations can have phase tracking applied. The GEODEL
card in the metafits describes what geometric delays have been applied.
By default, Birli will apply geometric corrections at the phase center if they have not already been applied. It determines the observations phase center from the RAPHASE
and DECPHASE
cards in the metafits. If these are not available, the pointing center cards (RA
and DEC
) from the metafits are used. You can use --no-geometric-delay
to disable geometric corrections, as well as the --phase-centre
and --pointing-centre
options to override the phase center.
A baseline's geometric length is determined by the w component of it's UVW fourier-space vector, after applying precession and nutation to it's tiles' positions and the phase center to the J2000 epoch, accounting for stellar aberration. Complex visibilities are phase-shifted by an angle determined by the w-component, and the channel's frequency.
let angle = -2.0 * PI * uvw.w * freq_hz / SPEED_OF_LIGHT_IN_VACUUM_M_PER_S;
Calibration
Birli can apply direction independent calibration solutions using the --apply-di-cal
flag. Solutions are applied before averaging. The number of channels in the un-averaged visibilities must be an integer multiple of the number of channels in the calibration solutions file. Unlike Cotter, Birli will handle calibration solutions where a NaN
value is present by flagging any visibilities where a NaN is present.
Currently, only the MWA aocal format (.bin), historically generated by the calibrate
binary in the mwa-reduce
package is supported. This format is described here, however due to the ambiguous definition of the startTime and endTime fields, their values are ignored and so only a single timeblock of solutions can be applied.
Cotter Emulation
The --emulate-cotter
flag ensures that outputs match Cotter as much as possible. You should only use this flag if you need to perform a direct comparison with Cotter.
By default, Birli will use the MWA array position from MWALib in order to calculate UVWs and geometric corrections. This is more accurate than the one that Cotter uses, and is the main source of error when doing direct comparisons.
This flag is used as part of the tests in src/main.rs
to validate that Birli's output matches that of Cotter to within an acceptable margin.
Averaging
To average the data in time or frequency by a given whole number factor, you can provide the --avg-time-factor
or --avg-freq-factor
options. This can also be achieved with the --avg-time-res
and
--avg-freq-res
options which take a duration [seconds] or ammount of bandwidth [kHz]
respectively. This second group of options will choose the closest whole number averaging factor
based on the resolution of the input data.
Output
Birli can output visibility data to uvfits or measurement set with --ms-out
(-M
) or
--uvfits-out
(-u
). It can also output flags for each coarse channel in .mwaf format with
--flag-template
(-f
), where the %
characters in the template argument are replaced with
the same zero-prefixed coarse channel identifiers that are used to identify the coarse channel
GPUBox files that the coarse channel data came from. For legacy data, use two percentage characters,
since the coarse channel identifier is the GPUBox number. However, for MWAX data, the coarse channel
identifier is the channel number, which needs three digits.
Comparison with Cotter
The following table shows how Birli options map onto Cotter options:
Birli | Cotter | Cotter Description |
---|---|---|
--version | -version | Output version and exit. |
-m | --metafits/-m | Read meta data from given fits filename.. |
-f,-u,-M | -o | Save output to given filename. Default is 'preprocessed.ms'. If the files' extension is .uvfits, it will be outputted in uvfits format and extension .mwaf is the flag-only format for input into the RTS. |
--no-rfi | -norfi | Disable RFI detection. |
--aoflagger-strategy | -flag-strategy | Use the specified aoflagger strategy. |
--no-cable-delay | -nocablelength | Do not perform cable length corrections. |
--no-geom | -nogeom | Disable geometric corrections. |
--phase-centre | -centre | Set alternative phase centre, e.g. -centre 00h00m00.0s 00d00m00.0s. |
--pointing-centre | -usepcentre | Centre on pointing centre. |
--avg-time-res | -timeres | Average nr of sec of timesteps together before writing to measurement set. |
--avg-freq-res | -freqres | Average kHz bandwidth of channels together before writing to measurement set. When averaging: flagging, collecting statistics and cable length fixes are done at highest resolution. UVW positions are recalculated for new timesteps. |
--apply-di-cal | -full-apply | Apply a solution file before averaging. The solution file should have as many channels as the observation. |
--no-digital-gains | -nosbgains | Do not correct for the digital gains. |
--max-memory | -absmem | Use at most the given amount of memory, specified in gigabytes. |
--flag-channel-edge-width | -edgewidth | Flag the given width of edge channels of each sub-band (default: 80 kHz). |
--flag-init | -initflag | Specify number of seconds to flag at beginning of observation (default: use QUACKTIME provided by the metafits/mwalib). |
--flag-times-end | -endflag | Specify number of seconds to flag extra at end of observation (default: 0s). |
--flag-dc | -flagdcchannels | Flag the centre channel of each sub-band (currently the default). |
--no-flag-dc | -noflagdcchannels | Do not flag the centre channel of each sub-band. |
--flag-antennae ... | -flagantenna | Mark the comma-separated list of zero-indexed antennae as flagged antennae. |
--flag-coarse-chans ... | -flagsubband | Flag the comma-separated list of zero-indexed sub-bands. |
--no-sel-autos | -noautos | Do not output auto-correlations. |
(default unless --flag-autos) | -noflagautos | Do not flag auto-correlations (default for uvfits file output). |
(default) | -nostats | Disable collecting statistics (default for uvfits file output). |
(default unless --no-sel-flagged-ants) | -noantennapruning | Do not remove the flagged antennae. |
(default) | -allowmissing | Do not abort when not all GPU box files are available (default is to abort). |
(--passband , default: unitary) | -sbpassband | Read the sub-band passband from given file instead of using default passband. (default passband does a reasonably good job) |
(TBD) | -flagfiles | Use previously writted MWAF files to skip RFI detection. Name should have two percentage symbols (%%), which will be replaced by GPU numbers. |
Birli will eventually perform all the same default preprocessing steps as Cotter when no flags are provided. The exception is that we have not yet implemented passband correction, flagging of edge / centre fine channels / quack timesteps / auto-correlations, pruning of flagged antennas. This means that birli <in/out args>
is equivalent to:
<in/out
There is no intention of replicating the following options Birli at this point, but feel free to open an issue if these are important to you.:
- Coarse channel selection (
-sbcount
,-sbstart
): This can be done by simply changing which coarse channel files are given in the CLI arguments) - Dysco compression (
-use-dysco
,-dysco-config
) - Manual metadata specification (
-a
,-h
,-i
): This information is readily available from metafits. -offline-gpubox-format
- Quality statistics (
-saveqs
,-histograms
,-skipwrite
,-nostats
) -noflagmissings
: If an HDU is missing, it should always be flagged.-apply
: only-full-apply
is supported.-noalign
: gpuboxes are always aligned.- CPU limit (
-j
): Birli uses crossbeam for concurrency which intelligency uses the compute resources available. Strict resource limits can be achieved with cgroups. - Memory percentage limit (
-mem
): Only-absmem
is supported. Determining memory limits on HPC systems is unreliable, so we recommend manually specifying a memory limit instead.
Example: RFI Flagging, corrections, averaging, output
In this example, we use the aoflagger subcommand to:
- Perform RFI flagging using the MWA-default flagging strategy
- Perform geometric and cable length corrections
- average the data to 4 seconds, 160khz
- disable passband correction
- Output visibilities to .uvfits (
-u
)
The equivalent Cotter commands would be:
# output uvfits
Contributing
Pull requests are welcome! Please do your best to ensure that the high standards of test coverage are maintained.
Before each commit, use cargo make ci
to ensure your code is formatted correctly.
Acknowledgement
This scientific work uses data obtained from the Murchison Radio-astronomy Observatory. We acknowledge the Wajarri Yamatji people as the traditional owners of the Observatory site.
Coverage
This repo is approved by...
Release Checklist
- pipeline is green
- update
RELEASES.md
- update
package.version
inCargo.toml
-
cargo make pre_commit
- commit (include Cargo.toml)
-
git tag -a $tag -m $tag
-
git push
-
git push --tags