commonware_deployer

Module ec2

Source
Expand description

AWS EC2 deployer

Deploy a custom binary (and configuration) to any number of EC2 instances across multiple regions. View metrics and logs from all instances with Grafana.

§Features

  • Automated creation, update, and destruction of EC2 instances across multiple regions
  • Provide a unique name, instance type, region, binary, and configuration for each deployed instance
  • Collect metrics, profiles (when enabled), and logs from all deployed instances on a long-lived monitoring instance (accessible only to the deployer’s IP)

§Architecture

                   Deployer's Machine (Public IP)
                                 |
                                 |
                                 v
              +-----------------------------------+
              | Monitoring VPC (us-east-1)        |
              |  - Monitoring Instance            |
              |    - Prometheus                   |
              |    - Loki                         |
              |    - Pyroscope                    |
              |    - Tempo                        |
              |    - Grafana                      |
              |  - Security Group                 |
              |    - All: Deployer IP             |
              |    - 3100: Binary VPCs            |
              |    - 4040: Binary VPCs            |
              |    - 4318: Binary VPCs            |
              +-----------------------------------+
                    ^                       ^
               (Telemetry)             (Telemetry)
                    |                       |
                    |                       |
+------------------------------+  +------------------------------+
| Binary VPC 1                 |  | Binary VPC 2                 |
|  - Binary Instance           |  |  - Binary Instance           |
|    - Binary A                |  |    - Binary B                |
|    - Promtail                |  |    - Promtail                |
|    - Node Exporter           |  |    - Node Exporter           |
|    - Pyroscope Agent         |  |    - Pyroscope Agent         |
|    - Memleak Agent           |  |    - Memleak Agent           |
|  - Security Group            |  |  - Security Group            |
|    - All: Deployer IP        |  |    - All: Deployer IP        |
|    - 9090: Monitoring IP     |  |    - 9090: Monitoring IP     |
|    - 9100: Monitoring IP     |  |    - 9100: Monitoring IP     |
|    - 9200: Monitoring IP     |  |    - 9200: Monitoring IP     |
|    - 8012: 0.0.0.0/0         |  |    - 8765: 12.3.7.9/32       |
+------------------------------+  +------------------------------+

§Instances

§Monitoring

  • Deployed in us-east-1 with a configurable ARM64 instance type (e.g., t4g.small) and storage (e.g., 10GB gp2).
  • Runs:
    • Prometheus: Scrapes binary metrics from all instances at :9090 and system metrics from all instances at :9100.
    • Loki: Listens at :3100, storing logs in /loki/chunks with a TSDB index at /loki/index.
    • Pyroscope: Listens at :4040, storing profiles in /var/lib/pyroscope.
    • Tempo: Listens at :4318, storing traces in /var/lib/tempo.
    • Grafana: Hosted at :3000, provisioned with Prometheus and Loki datasources and a custom dashboard.
  • Ingress:
    • Allows deployer IP access (TCP 0-65535).
    • Binary instance traffic to Loki (TCP 3100), Pyroscope (TCP 4040), and Tempo (TCP 4318).

§Binary

  • Deployed in user-specified regions with configurable ARM64 instance types and storage.
  • Run:
    • Custom Binary: Executes with --hosts=/home/ubuntu/hosts.yaml --config=/home/ubuntu/config.conf, exposing metrics at :9090.
    • Promtail: Forwards /var/log/binary.log to Loki on the monitoring instance.
    • Node Exporter: Exposes system metrics at :9100.
    • Pyroscope Agent: Forwards perf profiles to Pyroscope on the monitoring instance.
    • Memleak Agent: Exposes memleak metrics at :9200.
  • Ingress:
    • Deployer IP access (TCP 0-65535).
    • Monitoring IP access to :9090, :9100, and :9200 for Prometheus.
    • User-defined ports from the configuration.

§Networking

§VPCs

One per region with CIDR 10.<region-index>.0.0/16 (e.g., 10.0.0.0/16 for us-east-1).

§Subnets

Single subnet per VPC (e.g., 10.<region-index>.1.0/24), linked to a route table with an internet gateway.

§VPC Peering

Connects the monitoring VPC to each binary VPC, with routes added to route tables for private communication.

§Security Groups

Separate for monitoring (tag) and binary instances ({tag}-binary), dynamically configured for deployer and inter-instance traffic.

§Workflow

§ec2 create

  1. Validates configuration and generates an SSH key pair, stored in /tmp/deployer-{tag}/id_rsa_{tag}.
  2. Creates VPCs, subnets, internet gateways, route tables, and security groups per region.
  3. Establishes VPC peering between the monitoring region and binary regions.
  4. Launches the monitoring instance, uploads service files, and installs Prometheus, Grafana, Loki, and Pyroscope.
  5. Launches binary instances, uploads binaries, configurations, and hosts.yaml, and installs Promtail and the binary.
  6. Configures BBR on all instances and updates the monitoring security group for Loki traffic.
  7. Marks completion with /tmp/deployer-{tag}/created.

§ec2 update

  1. Stops the binary service on each binary instance.
  2. Uploads the latest binary and configuration from the YAML config.
  3. Restarts the binary service, ensuring minimal downtime.

§ec2 authorize

  1. Obtains the deployer’s current public IP address (or parses the one provided).
  2. For each security group in the deployment, adds an ingress rule for the IP (if it doesn’t already exist).

§ec2 destroy

  1. Terminates all instances across regions.
  2. Deletes security groups, subnets, route tables, VPC peering connections, internet gateways, key pairs, and VPCs in dependency order.
  3. Marks destruction with /tmp/deployer-{tag}/destroyed, retaining the directory to prevent tag reuse.

§Persistence

  • A temporary directory /tmp/deployer-{tag} stores the SSH private key, service files, and status files (created, destroyed).
  • The deployment state is tracked via these files, ensuring operations respect prior create/destroy actions.

§Example Configuration

tag: ffa638a0-991c-442c-8ec4-aa4e418213a5
monitoring:
  instance_type: t4g.small
  storage_size: 10
  storage_class: gp2
  dashboard: /path/to/dashboard.json
instances:
  - name: node1
    region: us-east-1
    instance_type: t4g.small
    storage_size: 10
    storage_class: gp2
    binary: /path/to/binary
    config: /path/to/config.conf
    profiling: true
  - name: node2
    region: us-west-2
    instance_type: t4g.small
    storage_size: 10
    storage_class: gp2
    binary: /path/to/binary2
    config: /path/to/config2.conf
    profiling: false
ports:
  - protocol: tcp
    port: 4545
    cidr: 0.0.0.0/0

Modules§

aws
AWS EC2 SDK function wrappers
services
Service configuration for Prometheus, Loki, Grafana, Promtail, and a caller-provided binary
utils
Utility functions for interacting with EC2 instances

Structs§

Config
Deployer configuration
Host
Host deployment information
Hosts
List of hosts
InstanceConfig
Instance configuration
MonitoringConfig
Monitoring configuration
PortConfig
Port configuration

Enums§

Error
Errors that can occur when deploying infrastructure on AWS

Constants§

AUTHORIZE_CMD
Authorize subcommand name
CMD
Subcommand name
CREATE_CMD
Create subcommand name
DESTROY_CMD
Destroy subcommand name
METRICS_PORT
Port on binary where metrics are exposed
UPDATE_CMD
Update subcommand name

Functions§

authorize
Adds the deployer’s IP (or the one provided) to all security groups.
create
Sets up EC2 instances, deploys files, and configures monitoring and logging
destroy
Tears down all resources associated with the deployment tag
update
Updates the binary and configuration on all binary nodes