scallion
scallion is a library and command line tool for identifying license texts.
This project is a fork of askalono, which is no longer getting developed and is now archived on GitHub.
Notice
This tool does not provide legal advice and it is not a lawyer. It endeavors to match your input to a database of similar license texts, and tell you what it thinks is a close match. But, it can't tell you that the given license is authoritative over a project. Nor can it tell you what to do with a license once it's identified. You are not entitled to rely on the accuracy of the output of this tool, and should seek independent legal advice for any licensing questions that may arise from using this tool.
Usage
On the command line
Basic usage:
scallion id <filename>
where <filename> is a file (not folder) containing license text to analyze. In many projects, this file is called LICENSE or COPYING. scallion will analyze the text and output what it thinks it is.
If scallion can't identify a file, it may simply be a license it just doesn't know. But, if it's actually source code with a file header (or footer, or anything in between) it may be able to dig deeper. To try this, pass the --optimize flag:
scallion id --optimize <filename>
If you'd like to discover license files within a directory tree, scallion offers a crawl action:
scallion crawl <directory>
As a library
At the moment, Store and LicenseContent are exposed for usage.
The best way to get an idea of how to use scallion as a library in its early state is to look at the example. Some examples are also available in the documentation.
Details
Implementation
tl;dr: Sørensen–Dice scoring, multi-threading, compressed cache file
At its core, scallion builds up bigrams (word pairs) of input text, and compares that with other license texts it knows about to see how similar they are. It scores each match with a Sørensen–Dice coefficient and looks for the highest result. There is some minimal preprocessing happening before matching, but there are no hand-maintained regular expressions or curations used to determine a match.
In detail, the matching process:
- Reads in input text
- Normalizes everything it reasonably can -- Unicode characters, whitespace, quoting styles, etc. are all whittled down to something common.
- Lines that tend to change a lot in licenses, like "Copyright 20XX Some Person", are additionally removed.
- Tokenizes normalized text into a set of bigrams.
- In parallel, the bigram set is compared with all of the other sets scallion knows about.
- The resulting list is sorted, the top match identified, and result returned.
To optimize startup, scallion builds up a database of license texts (applying the same normalization techniques described above), and persists this data to a MessagePack'd & zstd compressed cache file. This cache is loaded at startup, and is optionally embedded in the binary itself.
Name
Just sticking to the "onion" theme.
Where do the licenses come from?
License data is sourced directly from SPDX: https://github.com/spdx/license-list-data
scallion can parse the "json" format included in that repository to generate its cache.
At this time, scallion is not taking requests for additional licenses in its default dataset -- its dataset is SPDX's own.
License
This library is licensed under the Apache 2.0 License.