Supply chain threats in open-source packaging ecosystems are numerous. Malicious packages of all sorts have been found on PyPI, some using clever typo squatting, others taking over names of deprecated packages, others used as part of fake-recruiter social engineering campaigns. Of course, individuals can take risks, but for organizations of 30 or 300 Python users running within corporate networks, the risks are greater. There are various controls that can be put in place to limit or deny access to PyPI, but many of those can be circumvented. These observations led me to consider building a tool that can do enterprise-wide package allow listing. There are maybe two ways to do this: one is low-level file-system event monitoring with (on Linux) something like fanotify; another is system-wide scans of all installed packages. While both approaches could be used by a persistent agent that monitors systems and sends data to a central server, the latter also provides convenient system auditing functions, similar to what our check_dependencies.py does, and is my present focus.
Fetter is a Rust-built tool for system-wide package auditing. The fetter CLI presently offers four main tools: scan (discover all packages), derive (create a requirements file reflecting system-wide packages), validate (determine which packages violate a requirements file), and purge (remove packages that violate a requirements file).
--
Packages with known vulns:
jinja2-3.1.3
certifi-2023.11.17
--
The problem with virtual environments
They proliferate
They support multiple Python versions
They let the same Python exe have different site packages
Accumulation of lint
fetter count display
Opportunity
A tool for system-wide package auditing
How do we find packages?
Must find Pythons first
Can look at PATH
Can look at typical bin locations
Can search user HOME
Using each Python, we can find site packages
There are three possible site packages:
site.getsitepackages()
site.getusersitepackages()
dist-packages
python -c "import site; print(site.getsitepackages()); print(site.getusersitepackages())"
/usr/bin/python3 -c "import site; print(site.getsitepackages()); print(site.getusersitepackages())"
Can identify virtual environs with "pyvenv.cfg"
Can find a Python in bin/python3
Once we find a site directory, how do we find packages
find ".dist-info" directories
can find direct_url.json
can find RECORD
Parsing JSON in direct_url.json
ScanFS:
mapping of exe to sites
mapping of package to sites
Constructors
DepSpec
Using partial grammar based on
https://packaging.python.org/en/latest/specifications/dependency-specifiers/
DepManifest
Command-line interface with CLAP
Build a Python wrapper with Maturin
The magic of project.scripts
[project.scripts]
fetter = "fetter:run_with_argv"
Next steps
Validating URL based packages
Implement purge
Support toml
Pulling in vulnerability information
Total serialization: for caching, communication to a server
Monitor mode, server implementation
Explore fanotify
--------------------------
Fetter: 9 commands
---
./target/release/fetter scan display
./target/release/fetter -e python3 scan display
./target/release/fetter -e python3 scan write -o /tmp/out.txt
cat /tmp/out.txt
./target/release/fetter --help
./target/release/fetter -e /home/ariza/.env311/bin/python3 audit display
./target/release/fetter -e python3 unpack display
./target/release/fetter -e python3 unpack --count display
---
Data model to tables
Data model might define more than row per record
Data model might need to have derived columns
Basic model of a Report { Vec<Records> }
---
Table traits:
Two traits:
Rowable
define a method that converts a record into a Vec of strings
Tableable
Generic to a type of Rowable
Implementer must define get_header, get_records as a Vec of Rowabel
Provides to_file, to_writer methods
Count Report
Audit Report
Antother application of traits: testing APIs
Ureq client
---------------------------------------------------
# new articles
Stop Running Python Blind: Ensure Package Alignment with Every Python Execution
Find Python Package Vulnerabilties in OSV Faster than pip-audit
The Fastest Way to Search the Open Source Vulnerability DB for Python Vulnerabilities.
A Faster Way to find Python Packages in the Open Source Vulnerability Database