# lal spec
`lal` is a simple command line tool that works on folders with a valid `manifest.json`, and accepts the following commands:
- [`lal fetch`](#lal-fetch) - fetch dependencies from `manifest.json` into `INPUT`
- [`lal update`](#lal-update-components) - update arbitrary dependencies into `INPUT`
- [`lal status`](#lal-status) - print current INPUT dependencies with origin
- [`lal verify`](#lal-verify) - verify manifest validity + verify flat lockfile dependency tree
- [`lal env`](#lal-env-environment) - control build environment
- [`lal build [name]`](#lal-build-name-flags) - run canonical build in docker with current directory mounted
- [`lal shell`](#lal-shell) - enter container environment mounting current directory
- [`lal run`](#lal-run-name) - runs a non-build script through lal shell
- [`lal configure`](#lal-configure-defaults) - generate configuration file
- [`lal init`](#lal-init) - generate manifest file
- [`lal stash`](#lal-stash-name) - copies current `OUTPUT` to cache
- [`lal upgrade`](#lal-upgrade) - performs an upgrade check
- [`lal clean`](#lal-clean) - cleans up cache directory
- [`lal export`](#lal-export-component) - obtain a raw tarball from artifactory
- [`lal query`](#lal-query-component) - list versions of a component on artifactory
- [`lal remove`](#lal-remove-components) - remove components from `INPUT` and `manifest.json`
- [`lal publish`](#lal-publish) - publish release builds to artifactory
- [`lal propagate`](#lal-propagate-component) - works out steps to propagate dependencies
## Manifest
A per-repo file. Format looks like this (here annotated with illegal comments):
```json
{
"name": "libwebsockets", // name of repo
"environment": "centos", // name of environment found in the lal config
"supportedEnvironments": ["centos", "xenial"],
"components": { // map of components and default configuration
"libwebsockets": {
"defaultConfig": "release",
"configurations": ["release", "clang"]
},
"websockets_tests": {
"defaultConfig": "coverage",
"configurations": ["coverage", "release", "clang"]
}
},
"dependencies": {
"ciscossl": 42
},
"devDependencies": {
"gtest": 42
}
}
```
## Lockfile
A per-build file auto-generated by `lal build` and will reduce the lockfiles generated from dependencies to provide aggregated information.
```json
{
"name": "edonus",
"config": "release",
"container": {
"name": "edonusdevelopers/centos_build",
"tag": "2016-04-03"
},
"environment": "centos",
"tool": "0.10.0", // from `lal --version`
"version": "5", // from --with-version or "EXPERIMENTAL-{randomhex}"
"sha": "0ee0ee225d107076ed4b00368805d987baac9c4d", // from --with-sha
"dependencies": {
"libwebsockets": {
"name": "libwebsockets",
"config": "release",
"container": {
"name": "edonusdevelopers/centos_build",
"tag": "2016-04-03"
},
"environment": "centos",
"version": "47",
"tool": "0.10.0",
"dependencies": {
"ciscossl": {
"name": "ciscossl",
"config": "release",
"container": {
"name": "edonusdevelopers/centos_build",
"tag": "2016-04-03"
},
"environment": "centos",
"version": "200",
"tool": "0.10.0",
"dependencies": {}
}
}
},
"ciscossl": {
"name": "ciscossl",
"config": "release",
"container": {
"name": "edonusdevelopers/centos_build",
"tag": "2016-04-03"
},
"environment": "centos",
"version": "200",
"tool": "0.10.0",
"dependencies": {}
}
}
}
```
This struct is fully recursive in the sense that every value in the dependencies hash is also a valid lockfile.
## Config
A per-machine configuration file in `~/.lal/config` generated by `lal configure`. This is an example of environments, artifactory settings and mounts for a hypothetical edonus team.
```json
{
"artifactory": {
"server": "https://engci-maven-master.cisco.com/artifactory",
"group": "CME-release",
"vgroup": "https://engci-maven.cisco.com/artifactory/CME-group"
},
"cache": "/home/devuser/.lal/cache",
"environments": {
"centos": { "container": "edonusdevelopers/centos_build", "tag": "latest" },
"xenial": { "container": "edonusdevelopers/build_xenial", "tag": "latest" }
},
"upgradeCheck": "2016-06-30T12:20:10.126707483+00:00",
"mounts": [
{
"src": "/mnt/tools",
"dest": "/tools",
"readonly": true
}
]
}
```
Every repository is required to specify the name of one of the specified environments in their `manifest.json`.
The `upgradeCheck` value is updated automatically by `lal upgrade`.
## .lal/opts
A per-repo temporary file primarily for `lal env` that overrides the current environment.
```json
{
"env": "xenial"
}
```
When this file exists, every `lal` command will use this environment rather than the default one. It is created by [`lal-env`](#lal-env-environment).
This file is intended to be gitignored because it overrides `manifest.environment`.
## Caching
The local cache is populated by fetches from the registry, or calls to `stash` them.
```sh
~/.lal/cache $ tree .
├── environments
│ ├── centos
│ │ └── ciscossl
│ │ └── 6
│ │ └── ciscossl.tar
│ └── xenial
│ └── ciscossl
│ └── 6
│ └── ciscossl.tar
└── stash
└── ciscossl
└── asan
└── ciscossl.tar
```
Sources:
- `environments` are components from the registry under a specific environment namespace
- `stash` are tarballs of OUTPUT of builds when doing `lal stash <name>`
## Versioning
As implied by the structure of the Manifest, Lockfile, and cache directories, the *only* versioning scheme supported by `lal` is a monotonically increasing integer sequence.
### Subcommands
#### lal status
Provides list of dependencies currently in `INPUT`.
If they are not in the manifest they will be listed as _extraneous_.
If they are stashed dependencies they will be listed in yellow origin
Extra flags:
- `--full` or `-f`: print the full dependency tree
- `--origin` or `-o`: print version and environment origin of artifact
- `--time` or `-t`: print build time of artifact
Alias: `lal ls`
#### lal env [environment]
Subcommand that controls the current environment. This is a sticky, repo-wide setting stored in `$PWD/.lal/opts` when working with non-standard environments.
```sh
$ lal env
xenial
# every lal command will defer the environments key in `manifest.json` by default
$ lal env set zesty # writes { "environment": "zesty" } to .lal/opts
$ lal env update # invokes docker pull of the zesty image
# now every lal command will warn if `manifest.environment != lal env`
$ lal fetch # fetches from zesty
$ lal build # build using zesty components
$ lal shell # enters zesty shell
$ lal run unit-test websocket_server_test 5 asan # runs unit test in zesty shell
$ lal env reset # deletes `.lal/opts` if it exists
# lal now behaves as usual, doing all commands in the described environment in manifest.json
```
This is an advanced command for people developing on temporary, non-standard environments.
If you would like to override the environment on a command-by-command basis, there is an [option](#universal-options) for that as well.
#### lal build [name] [flags]
Runs the `BUILD` script in the current directory in the container.
If no arguments are suppplied it will run `./BUILD $name $config` where `name` is the value of `name` in the manifest, and `config` is the name of the component's `defaultConfig`.
E.g. `lal build` in say a gtest repo will probably call `./BUILD gtest release` in the container.
`lal build` will run `lal verify` and abort if this fails. When using stashed components, you should build with `--simple-verify` or `-s` for short. This will allow stashed versions to pass, but still not cripple the verifier so that you accidentally include things built in different environments.
Any further verify blocks can be overridden with `-f` or `--force`. There are very few legit developer reasons why you would want to completely ignore `lal verify`, but maybe you have such a special case.
Release specific flags:
- *--release*: Generate a tarball and lockfile in `./ARTIFACT` folder after building
- *--with-version n*: Jenkins specific option which will specify lockfile version
- *--with-sha str*: Jenkins specific option which will set revision id
Typically jenkins would do:
```sh
lal build --release --with-version=$BUILD_NUMBER --with-sha=$(git rev-parse HEAD)
```
And publish that with `lal publish`.
Passing configuration flags:
- *--config=name*: Passes a named config to `BUILD` as `$2`.
This allows multiple blessed configurations of the same component, i.e. `lal build dme-unit-tests --config=asan` and `lal build dme-unit-tests --config=debug`. Both are valid provided `dme-unit-tests` provides those `configurations` in the `components` part of the manifest.
#### lal update [components..]
Find the latest available version of a component that is available in all currently `supportedEnvironments` from the manifest.
- *lal update component [--save]*: fetches the latest version of a component. The optional `--save` flag will also update the manifest file locally.
- *lal update component=version [--save]*: fetches a specific version. If the version is parsable as an integer, it is fetched from artifactory. Otherwise, it is assumed to be a stashed version.
Many `component` or `component=version` arguments can be used in one invocation.
#### lal fetch
- *lal fetch [--core]*: fetches all versions corresponding to the manifest from the registry and puts them into `INPUT`. The optional `--core` flag will disregard any `devDependencies`.
Any components already found in `INPUT` are reused if they are present at the right version and correct environment.
Any extraneous versions found in `INPUT` are removed.
#### lal shell
Enters an interactive shell in the container corresponding to the environment key in the manifest mounting the current directory.
Useful for experimental builds using internal scripts in a repo.
Assumes you have run `lal fetch` or equivalent so that `INPUT` is ready for this.
lal shell should basically run:
```sh
docker run \
-v $PWD:/home/lal/volume \
-w /home/lal/volume \
--user lal \
-it ${SOME_CONTAINER} \
/bin/bash
```
Note that the config can be edited to pass in extra mounts.
lal shell should allow passing in trailing arguments to run arbitrary commands:
- `lal shell ./BUILD something` # runs this command rather than `/bin/bash` and removes `-i`
- `lal shell -p ./BUILD something` # same, but adds --privileged to `docker`
- `lal shell bash -c "cmd1; cmd2"` # multiple commands in one go
- `lal shell --print-only` prints above command
- `lal shell --print-only ./BUILD something` # prints what would have been done
lal shell should also allow making it easy to forward the X11 socket:
- `lal shell -X` # mounts `/tmp/.X11-unix`, `~/.Xauthority` and forwards the `DISPLAY` evar
Host networking should be convenience flag:
- `lal shell -n` # passes `--net=host` to docker.
A combination of these two flags allows forwarding X through `ssh` and `lal`:
```sh
ssh -X somemachine # ssh with X11 forwarding
lal sh -X -n xcalc # run xcalc and forward X11 all the way through ssh
```
The `X11` forwarding setup requires `xhost` installed, and also `xauth` installed if you need it through `ssh` as well. You may need to run `xhost local:docker` to allow docker to access X.
Alias: `lal sh`
#### lal run [name]
Runs scripts in the local `.lal/scripts/` folder via `lal shell`. Because `lal shell` mounts `$PWD`, the scripts folder can contain parametrised scripts such as:
```sh
#!/bin/bash
# contents of .lal/scripts/subroutine
main() {
echo "hi $1 $2"
}
completer() {
echo "foo bar"
}
```
Which could be invoked with `lal run subroutine there mr`, which would `echo hi there mr` in the container. An optional `completer` function can be supplied for autocomplete of values.
Alias: `lal script`
#### lal stash [name]
Stashes the current `OUTPUT` folder to in `~/.lal/cache/stash/${component}/${NAME}` for future reuse. This can be put into another repository with `lal update component=name`
Alias: `lal save`
#### lal verify
Helper command used by `lal build` exposed for convenience/sanity. Verifies that:
- `manifest.json` exists in `$PWD` and is valid JSON
- dependencies in `INPUT` match `manifest.json`
- the dependency tree is flat
- dependencies in `INPUT` contains only published dependencies
- dependencies in `INPUT` were built using the correct environment
`lal build` normally guards on this command.
An optional `--simple` or `-s` can be passed to `lal verify` to not check for published dependencies and a flat dependency tree.
#### lal configure [defaults]
Sets up a default config with a set of pre-configured defaults from a seperately supplied file with default values:
```sh
lal configure configs/edonus.json
```
Will set up the docker environments, artifactory downnload settings, and common mounts to scan for for the edonus team.
To tweak different settings, edit `~/.lal/config` after the original `configure` call, then manage it yourself.
#### lal init [environment]
Creates a basic `manifest.json` in the current directory, assuming directory name as the name of the main component.
A `-f` flag can be supplied to force overwrite the manifest.
```sh
lal init -f centos
```
#### lal upgrade
Performs an upgrade check of `lal`. If new versions are found, it reports how to upgrade your lal tool. This is normally checked daily. But it can be done manually with this command.
This is currently disabled awaiting a redesign.
#### lal clean
Deletes artifacts in the cache directory older than 14 days. The day is configurable with `-d <days>`.
#### lal export [component]
Exports a build artifact from the storage backend in the current directory or a directory of choice.
The component can be either the name of the component for latest version, or suffixed with `=version` for a specific version:
```sh
lal -e xenial export gtest -o mystorage/
test -f mystorage/gtest.tar.gz
lal -e xenial export liblzma=6
test -f ./liblzma.tar.gz
```
NB: export does not read the manifest.json for environment overrides.
#### lal query [component]
Lists the availble versions in the storage backend that were built in a speific environent.
```sh
lal -e xenial query libwebsockets
```
NB: query does not read the manifest.json for environment overrides.
#### lal remove [components..]
Removes and optionally saves a removal of a component from `INPUT` and the manifest.
```sh
lal remove libwebsockets --save
lal remove gtest --save-dev
```
Note you can only use one of save or save-dev at a time. Without either save flag, this subcommand simply deletes the corresponding subdirectory of `INPUT`.
Alias: `lal rm`
#### lal publish
Publishes a release build in the local `ARTIFACT` subdirectory provided it is built with a correct version and proper credentials are presented.
```sh
lal env set xenial
lal fetch
lal build libldns --release --with-version=20 --with-sha=$(git rev-parse HEAD)
# specific builds should push immediately (even if there are more builds)
lal publish libldns
```
The publish command will upload to a bucket named after the environment used to build it (found in `./ARTIFACT/lockfile.json`). It will also verify that the version is set with `--with-version`.
The uploaded artifact will in this case end up the following location:
- `https://artifactory.host/artifactory/group/env/xenial/libldns/20/`
If you have more `supportedEnvironments` then `lal update` will look in all the buckets corresponing to your environments before finding a version that can be useg in all environments.
#### lal propagate [component]
Retraces a dependency tree in reverse to figure out steps needed to propagate a leaf dependency properly. This is useful for satisfying the full version strictness checks of `lal verify` in a large dependency tree (recall that we enforce a flat dependency tree).
Given a component with the following example dependency tree:
```sh
~ > mycomponent on master $ lal ls -f
mycomponent
├── openssl
├─┬ libcurl
│ ├── c-ares
│ └── openssl
├─┬ cucumber-cpp (dev)
│ └── gtest
├── gtest (dev)
├─┬ qt
│ ├── openssl
│ └── freetype
└── zlib
```
If we need to propagate a new version of `openssl`, we need to do it in two stages:
```sh
~ > mycomponent on master $ lal propagate openssl
Assuming openssl has been updated:
Stage 1:
- update [openssl] in libcurl
- update [openssl] in qt
Stage 2:
- update [libcurl, openssl, qt] in mycomponent
```
Every step in each stage is paralellizable, but every stage must wait for the previous stage. A simple web service to perform this scheduling and upgrade can be set up if you are willing to hook this up to your CI infrastructure.
### Universal Options
- `--help` or `-h`
- `-v`
- `--env` or `-e`
Note that `-v` is a global option that gradually increases verbosity (allows multiple uses), and goes before subcommands.
```sh
lal update zlib # update with only standard logging (info!, warn!, and error!)
lal -v fetch # fetch with debug! messages
lal -vv build # build with debug! and trace! messages
```
The `--env` flag will override the default environment in both `manifest.environment` and `.lal/opts` for the current command:
```sh
lal --env xenial fetch
lal -e xenial verify
lal -e xenial build
lal -e xenial shell
lal -e xenial script unit-test websocket_server_test 5 asan
```
Because these commands are often used together you can instead make it sticky with [`lal env`](#lal-env-environment).
For full autogenerated help of all flags of every subcommand help can be requested:
```sh
lal build -h
lal help build # equivalent
```