libpybuild 0.0.2

Python src+dependency bundler library
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87

<h1 style="font-size: 56px; margin: 0; padding: 0;" align="center">
  pybuild-install
</h1>
<p align="center">
  <a href="https://crates.io/crates/pybuild-install">
    <img src="https://img.shields.io/crates/v/pybuild-install.svg" alt="crates.io">
  </a>
  <a href="https://github.com/serverless-delivery/pybuild/blob/main/pybuild/pybuild-install/LICENSE">
    <img src="https://img.shields.io/badge/License-Apache_2.0-blue.svg" alt="License">
  </a>
  <a href="https://github.com/serverless-delivery/pybuild/actions/workflows/ci-py.yml">
    <img src="https://github.com/serverless-delivery/pybuild/actions/workflows/ci-py.yml/badge.svg" alt="CI">
  </a>
</p>

Install command is meant to provide a stopgap solution and is best suited to be handled by other tools. However, fitting those tools easily and natively into the workflows of tools/frameworks for packaging AWS Lambda functions is challenging for users:

## Compare & Contrast with NodeJS

- In NodeJS with the `esbuild` tool - due to the ability to tree-shake any excess code, users can install dependencies once in a central location. `esbuild` does not concern itself with installing dependencies.
- In NodeJS, it seems less common to have to install dependencies in a "Lambda-like" environment. This greatly simplifies the process of package management to standard tools like NPM, Yarn, etc.
- This leaves the typical CI/CD pipeline for NodeJS artifacts that are bundled to look like:
  - Checkout source code
  - Install dependencies using standard tooling (NPM, Yarn, etc.) within the CI environment
  - Build/bundle artifact using `esbuild` (this step can be spread out and parallelized for multiple artifacts)
  - Deploy artifacts
- If we contrast this with a Python project
  - Checkout source code

## TODOs

- ✅ For pipenv and poetry, how to export only production dependencies?
  - For `pipenv`, this seems to be the default <https://github.com/pypa/pipenv/issues/245>
  - For `poetry`, this also seems to be the default when inspecting `poetry export -h`
- [ ] `install`
      This is just a simple wrapper that shells out commands to the cli to ergonomically handle where deps are installed for bundling
  - ✅ --package-manager, -p: one of `pip`, `poetry`, or `pipfile`
    - `pip` will use dependencies listed in `requirements.txt`

      ```
      pip install -r requirements.txt --target ./.pybuild/site-packages
      ```

    - `pipenv`
      - <https://github.com/pypa/pipenv/issues/746>

      ```
      pipenv run pip install -r <(pipenv lock -r) --target ./.pybuild/site-packages
      ```

    - `poetry`
      - <https://github.com/python-poetry/poetry/issues/1937#issuecomment-612398471>

      ```
      poetry export -f requirements.txt --without-hashes > requirements.txt & \
      poetry run pip install -r requirements.txt -t ./.pybuild/site-packages
      ```

  - ✅ --extra-args, -e: extra arguments to pass to package manager install command
  - ✅ --target, -t: Install packages into <dir>. By default this will not replace existing files/folders in <dir>. Use `--extra-args='--upgrade'` to replace existing packages in <dir> with new versions.
    - By default we will use `.pybuild/site-packages` since this is where `pybuild bundle ...` will look
    - Clearly document that if users change `--target` tha they will also want to change `--packages` to match when bundling
  - ✅ --upgrade, -u: Upgrade all specified packages to the newest available version
  - [ ] --requirements-file, r: path to requirements file (requirements.txt, Pipfile, pyproject.toml, etc.)
- ✅ --log-level, -l: log level
- ✅ --version, -v: version

## Enhancements

- [ ] Add ability for users to `--install-in-docker` to ensure libraries are compatible for various runtime environments like AWS Lambda
- [ ] Allow users to select which container image to use for `--install-in-docker` using `--container-image`
- [ ] Add flag `--python-version` to aid in selecting default container image and archive path, etc.

### Package Install Location Challenges

#### Pip

- <https://www.python.org/dev/peps/pep-0582/>
  - <https://discuss.python.org/t/pep-582-python-local-packages-directory/963>

#### Pipenv

- <https://github.com/pypa/pipenv/issues/746>

#### Poetry

- <https://github.com/python-poetry/poetry/issues/1937>