Linkletter - URL replacement and analytics service

Linkletter is a demo web service for replacing URLs in documents with other managed URLs, which when visited collect information for analytics purposes before redirecting to the original resource.

Usage

To start the Linkletter server, simply execute the installed binary:

$ LL_RUN_MODE=<desired run mode> LL_CONFIG_DIR=<path to config directory> linkletter

If defined, LL_CONFIG_DIR should point to a directory containing a settings.json with settings as described below, and optionally a settings.<LL_RUN_MODE>.json containing settings to override based on the value of LL_RUN_MODE.

If not given, LL_RUN_MODE defaults to dev and LL_CONFIG_DIR defaults to ./config.

Endpoints

When running, Linkletter exposes a REST API with the following endpoints:

• POST /documents: Replaces URLs in message body with managed URLs. Returns JSON object with format:

  {
      "id": "<ID assigned to document>",
      "replacement": "<passed message body with URLs replaced>"
  }
  

• GET /documents/<ID>/urls: Gets URLs found in document with given ID, in order of appearance. Returns JSON list of objects with format:

  {
      "id": "<ID assigned to URL>",
      "url": "<URL as it appears in the original document>"
  }
  

• GET /documents/<ID>/visits: Gets all visits of URLs found in document with given ID, in order of time of visit. Returns JSON list of objects with format:

  {
      "id": "<ID assigned to URL>",
      "timestamp": "<date and time of visit in RFC 3339 format>",
      "ipAddr": "<connecting address of visitor>",
      "userAgent": "<value of `User-Agent` header in visit request>"
  }
  

• GET /documents/<ID>/analytics: Aggregates information of visits to URLs found in document with given ID. Returns JSON list of objects with format:

  {
      "id": "<ID assigned to URL>",
      "visitCount": "<total number of visits to URL>",
      "firstVisit": "<date and time of first visit in RFC 3339 format, or null>",
      "latestVisit": "<date and time of latest visit in RFC 3339 format, or null>"
  }
  

• GET /urls/<ID>: Return original URL with given ID as plain text in message body.
• GET /urls/<ID>/visits: Gets all visits of URL with given ID, in order of time of visit. Returns JSON list of objects with format:

  {
      "timestamp": "<date and time of visit in RFC 3339 format>",
      "ipAddr": "<connecting address of visitor>",
      "userAgent": "<value of `User-Agent` header in visit request>"
  }
  

• /visit/<ID>: Not meant to be called directly. Returns redirect to original URL with given ID.

Configuration

Linkletter looks for configuration in <LL_CONFIG_DIR>/settings.json, which should include the following:

{
    "address": {
        "ip": "<IP address to bind server on>",
        "port": <port to bind server on>
    },
    "database": {
        "url": "<URL to SQLite database>",
        "migrations": "<path to directory containing scripts for schema migrations>"
    },
    "api": {
        "host_url": "<base URL for host which logs and redirects visits>"
    }
}

Installation

This software can be installed using Cargo, from the crates.io registry.

$ cargo install a-puzzle-a-day

This will place the executable linkletter into $HOME/.cargo/bin (by default), which should be added to $PATH.

Alternatively, to build from source:

$ git clone https://github.com/mrbjarksen/linkletter
$ cd linkletter
$ cargo build --release
$ cp target/release/linkletter <desired location>

This repository also exposes a Nix flake:

$ nix build github:mrbjarksen/linkletter
$ cp result/bin/linkletter <desired location>

---

Design overview

Linkletter is implemented as a single-process asynchronous web server using Rust and SQLite. Various industry standard crates are used, such as:

• tokio as an async runtime,
• axum for HTTP routing and request-handling,
• sqlx for database communication via SQL.

Application flow

Application flow is generally catagorized into three parts: replacement, visit handling and analytics.

Replacement

Upon receiving a request to replace URLs in a document, Linkletter first generates a UUIDv7 for the document before persisting the document content in the database for safekeeping.

Then the content is searched for URLs and each one found is assigned a UUIDv4 and persisted in the database sequentially (keeping track of its position in the document), before it is added to the replaced content being constructed.

This was chosen for ease of implementation. Since Linkletter is an asynchronous service, this is not optimal, as database inserts could be performed concurrently.

Visit handling

When a managed URL is visited by an end-user, Linkletter fetches the original URL with ID embedded in replaced URL and logs information such as the date and time of the visit, connecting IP address of visitor and metadata found in the HTTP header.

This database operation is done in the background, concurrently to the user being redirected to the original URL, such that the user is not affected in case of failure.

The redirect is achieved using the HTTP status code 307 Temporary Redirect, which avoids the client caching the resulting URL and bypassing Linkletter on subsequent visits.

Analytics

The remaining analytics requests are all handled similarly: by routing the request to an appropriate SQL query whose result is processed and serialized into a response.

Database schema

Below is a diagram describing the expected database schema for Linkletter.

Document and URL IDs

Linkletter uses UUIDs for unique identifiers of both documents and URLs, formatted as hexadecimal strings without hyphens.

In terms of performance, these IDs should ideally all be UUIDv7 (i.e. seeded by timestamp), since they are used as database keys. This would improve performance of both read and write queries, as keys would be less fragmented throughout the database. For this reason, document IDs are always UUIDv7s.

Since URL IDs are exposed to end-users in the form of replaced URLs, however, it is generally more safe for them to avoid UUIDv7. Instead, Linkletter assigns UUIDv4s to URLs. This reduces attack surface, by way of minimizing the likelihood of being able to sniff out some ID for a URL intended for some other user.

Considerations

Some aspects of Linkletter need further consideration before the service can be applied in a production environment.

Authentication

As it stands, Linkletter does not implement any form of authentication, relying entirely on cryptographic randomness of generated IDs for security. This may be fine in cases where the host service is completely isolated from external parties, e.g. when used as an internal microservice.

If this is API should be more publicly available and be trusted to handle potentially sensitive information, this should be the first addition made.

Database engine

Among the first changes that should be made before using this service at scale is to change database providers from SQLite to another relational database, e.g. PostgreSQL or MySQL.

SQLite was chosen for its flexibility and ease of use in small-to-medium sized applications, but its limitations quickly become liabilities as soon as architecture needs to scale horizontally. When this happens, some client/server SQL database engine should be used instead (with appropriate load-balancing/caching/redundancy/partitioning based on needs).

This is especially true for Linkletter, since we should expect many concurrent writes as users visit URLs through the service, which may quickly overwhelm the system due to file locks.

NoSQL databases are not recommended as a primary datastore as flexibility in data querying is important for analytics. With that said, they may be useful as secondary datastores, e.g. using Redis as a caching layer.

If SQLite is opted to be used in production, WAL mode should at minimum be enabled to improve concurrent performance.

Database size

A likely use case of Linkletter is to track visits of URL over some relatively short period of time. In this case, some expiry age would be preferable to avoid disk usage ballooning over several years. After data is deemed expired (e.g. based on creation time or last time visited) it would then either be deleted or moved to some compacted archive.

Another method to reducing database size would be to move document content to some object storage service (e.g. Amazon S3). This is beneficial as document content is not currently used for any analytic needs. Document content might further be stored by hash in cases where one document needs to be sent to many users, with per-user analytics, for example in cases of email templates.

GraphQL for analytics

The current analytic capabilities of Linkletter are functional, but somewhat half-baked. If this service is intended to be used by potentially many parties with varied analytical needs, an endpoint accepting GraphQL queries should be considered.

This would allow for much greater flexibility than the current REST API (which does not even support filtering on timestamp).

Schema migrations

Linkletter leverages sqlx::migrations for the management of schema changes. On start-up, scripts found in db/migrations (relative to the working directory) are automatically applied if they have not already.

This is convenient for quick iteration and version control, but care must be taken to avoid applying scripts that could cause data loss or corruption. If this is a concern, manual control of database changes may be more appropriate.

Structured and event-driven logging

Currently, Linkletter only performs logging by printing errors to stderr. This means logs are not persisted, which makes diagnostics difficult. Therefore, some sort of logging framework should be introduced before this service is used in any serious capacity, e.g. using the log crate.

Since Linkletter is asynchronous, and may potentially be used by many concurrent users, a structured and event-driven logging framwork is appropriate. This allows to quickly search through logs to follow errors to their source. The tracing crate is recommended.

---

AI Notice

No generative AI has been used in any direct capacity at any part of the development of this project.