Skip to main content

Crate upstream_ontologist

Crate upstream_ontologist 

Source
Expand description

§Upstream Ontologist

The upstream ontologist provides a common interface for finding metadata about upstream software projects.

It will gather information from any sources available, prioritize data that it has higher confidence in as well as report the confidence for each of the bits of metadata.

The ontologist originated in Debian and the currently reported metadata fields are loosely based on DEP-12, but it is meant to be distribution-agnostic.

§Provided Fields

Standard fields:

  • Name: human name of the upstream project
  • Contact: contact address of some sort of the upstream (e-mail, mailing list URL)
  • Repository: VCS URL
  • Repository-Browse: Web URL for viewing the VCS
  • Bug-Database: Bug database URL (for web viewing, generally)
  • Bug-Submit: URL to use to submit new bugs (either on the web or an e-mail address)
  • Screenshots: List of URLs with screenshots
  • Archive: Archive used - e.g. SourceForge
  • Security-Contact: e-mail or URL with instructions for reporting security issues
  • Documentation: Link to documentation on the web
  • Changelog: URL to the changelog
  • FAQ: URL to the FAQ
  • Donation: URL to a donation page
  • Funding: List of sources of funding for the project

Extensions for upstream-ontologist, not defined in DEP-12:

  • SourceForge-Project: sourceforge project name
  • Wiki: Wiki URL
  • Summary: one-line description of the project
  • Description: longer description of the project
  • License: Single line license (e.g. “GPL 2.0”)
  • Copyright: List of copyright holders
  • Version: Current upstream version
  • Security-MD: URL to markdown file with security policy
  • Author: List of people who contributed to the project
  • Maintainer: The maintainer of the project
  • Homepage: homepage URL (present in debian/control in Debian packages)

§Supported Data Sources

At the moment, the ontologist can read metadata from the following upstream data sources:

It will also scan README and INSTALL for possible upstream repository URLs (and will attempt to verify that those match the local repository).

In addition to local files, it can also consult external directories using their APIs:

§Example Usage

The easiest way to use the upstream ontologist is by invoking the guess-upstream-metadata command in a software project:

$ guess-upstream-metadata ~/src/dulwich
Security-MD: https://github.com/dulwich/dulwich/tree/HEAD/SECURITY.md
Name: dulwich
Version: 0.20.15
Bug-Database: https://github.com/dulwich/dulwich/issues
Repository: https://www.dulwich.io/code/
Summary: Python Git Library
Bug-Submit: https://github.com/dulwich/dulwich/issues/new

Alternatively, there is a Python API as part of the upstream_ontologist Python package. There are also autocodemeta and autodoap commands that can generate output in the codemeta and DOAP formats, respectively.

§Reporting bugs

When reporting bugs, please include the observed output of the guess-upstream-metadata command, the version of the upstream-ontologist package you are using, what output you were expecting, and ideally the location of the upstream source code you are using (e.g. a URL to a Git repository).

If there are additional metadata fields you would like to see supported, please let us know - either with or without a patch.

Similarly, if you have a new data source you would like to see supported, please file a bug and we can discuss how to add it.

Modules§

extrapolate
Functionality for extrapolating upstream metadata from various sources
forges
Support for various code forges (GitHub, GitLab, etc.)
github
GitHub API and raw file access helpers Helpers for accessing the GitHub API and raw repository files.
homepage
Homepage URL detection and validation
http
HTTP utilities for fetching remote resources
providers
Various metadata providers for different programming languages and ecosystems
readme
README file parsing and metadata extraction
repology
Integration with Repology package repository aggregator
vcs
Version control system utilities and URL handling
vcs_command
Command-line interface for version control operations

Structs§

EnvironmentGuesser
Guesser that extracts metadata from environment variables
GitHub
GitHub forge implementation
GitLab
GitLab forge implementation
GuesserSettings
Settings for upstream metadata guessers
Launchpad
Launchpad forge implementation
ParseError
PathGuesser
Guesser that extracts metadata from a specific file path
PathSegmentError
Error when manipulating URL path segments
Person
Represents a person (author, maintainer, etc.) with optional contact information
SourceForge
SourceForge forge implementation
UpstreamDatumWithMetadata
Upstream datum with additional metadata about its origin and certainty
UpstreamMetadata
Collection of upstream metadata with convenience methods for accessing specific fields
UpstreamMetadataGuesser
A guesser that can extract upstream metadata from a specific file

Enums§

BugTracker
A bug tracker for an upstream project.
CanonicalizeError
Errors that can occur when canonicalizing URLs
Certainty
Certainty levels for the data
HTTPJSONError
Errors that can occur when loading JSON from HTTP
Origin
Origin of the data
ProviderError
Errors that can occur when fetching metadata from providers
UpstreamDatum
Represents various types of upstream metadata for a software project

Traits§

Forge
Trait for different code forges (GitHub, GitLab, etc.)
ThirdPartyRepository
Trait for third-party repositories that can provide upstream metadata
UpstreamDataProvider
Trait for providing upstream metadata

Functions§

bug_database_from_issue_url
Extracts the bug database URL from an issue URL
bug_database_url_from_bug_submit_url
Derives a bug database URL from a bug submission URL
bug_submit_url_from_bug_database_url
Derives a bug submission URL from a bug database URL
check_bug_database_canonical
Checks if a bug database URL is canonical
check_bug_submit_url_canonical
Checks if a bug submission URL is canonical
check_upstream_metadata
Check upstream metadata.
check_url_canonical
Checks if a URL is canonical by following redirects
extend_upstream_metadata
Extends upstream metadata with additional information from external sources
extract_hackage_package
Extracts the Hackage package name from a URL
extract_pecl_package_name
Extracts the PECL package name from a URL
find_forge
Determines which forge a URL belongs to
fix_upstream_metadata
Fix existing upstream metadata.
get_repology_metadata
Fetches metadata from the Repology API for a given source package
get_upstream_info
Gets upstream information for a project
guess_bug_database_url_from_repo_url
Guesses the bug database URL from a repository URL
guess_from_environment
Extracts upstream metadata from environment variables
guess_from_path
Guesses upstream metadata from a file or directory path
guess_from_travis_yml
Extracts upstream metadata from a Travis CI configuration file
guess_upstream_metadata
Guess the upstream metadata dictionary.
guess_upstream_metadata_items
Guess upstream metadata items, in no particular order.
load_json_url
Loads JSON data from a URL with optional timeout
metadata_from_url
Obtain metadata from a URL related to the project
repo_url_from_merge_request_url
Extracts the repository URL from a merge request URL
summarize_upstream_metadata
Summarize the upstream metadata into a dictionary.
update_from_guesses
Updates metadata collection with new guesses based on certainty levels
upstream_metadata_stream
Creates a stream of upstream metadata by running all applicable guessers
verify_screenshots
Verifies that screenshot URLs are accessible
with_path_segments
Creates a new URL with the specified path segments