drasi-bootstrap-http 0.1.2

HTTP bootstrap plugin for Drasi - fetches initial state from REST APIs
Documentation

HTTP Bootstrap Plugin for Drasi

Fetches initial state from REST APIs to populate Drasi continuous queries. Supports multiple endpoints, various authentication methods, pagination strategies, and flexible response-to-graph-element mapping.

Features

  • Multiple endpoints – fetch nodes and relations from different APIs in a single bootstrap
  • 5 pagination strategies – offset/limit, page number, cursor-based, Link header, next-URL
  • 4 authentication methods – Bearer token, API key, Basic auth, OAuth2 client credentials
  • Flexible mapping – Handlebars templates transform any JSON/XML/YAML response into graph elements
  • Retry with backoff – configurable retries for transient failures
  • Streaming – emits elements as they're parsed, no full-dataset buffering

Configuration Reference

Top-Level Config

Field Type Default Description
endpoints array (required) List of endpoint configurations
timeoutSeconds integer 30 HTTP request timeout in seconds
maxRetries integer 3 Maximum retry attempts on failure
retryDelayMs integer 1000 Base delay between retries in milliseconds; grows exponentially (delay × 2^(attempt−1)), capped at 60 seconds

Endpoint Config

Field Type Default Description
url string (required) The URL to fetch data from
method string "GET" HTTP method (GET, POST, PUT)
headers object {} Additional HTTP headers
body object null Request body (for POST/PUT)
auth object null Authentication configuration
pagination object null Pagination configuration
response object (required) Response parsing and mapping

Authentication

Bearer Token

{
  "type": "bearer",
  "token_env": "MY_API_TOKEN"
}

API Key (Header or Query)

{
  "type": "api-key",
  "location": "header",
  "name": "X-API-Key",
  "value_env": "MY_API_KEY"
}

{
  "type": "api-key",
  "location": "query",
  "name": "api_key",
  "value_env": "MY_API_KEY"
}

Basic Auth

{
  "type": "basic",
  "username_env": "MY_USERNAME",
  "password_env": "MY_PASSWORD"
}

OAuth2 Client Credentials

{
  "type": "oauth2-client-credentials",
  "token_url": "https://auth.example.com/oauth/token",
  "client_id_env": "MY_CLIENT_ID",
  "client_secret_env": "MY_CLIENT_SECRET",
  "scopes": ["read"]
}

Pagination Strategies

Offset/Limit

Classic offset-based pagination. Increments offset by page_size each request.

{
  "type": "offset-limit",
  "offset_param": "offset",
  "limit_param": "limit",
  "page_size": 100,
  "total_path": "$.meta.total"
}

Stop conditions: total_path exceeded, or page returns fewer items than page_size.

Page Number

Simple page number pagination. Increments page number each request.

{
  "type": "page-number",
  "page_param": "page",
  "page_size_param": "per_page",
  "page_size": 100,
  "total_pages_path": "$.meta.total_pages"
}

Stop conditions: total_pages_path exceeded, or page returns fewer items than page_size.

Cursor (Stripe-style)

Extracts a cursor value from the response and passes it as a query parameter.

{
  "type": "cursor",
  "cursor_param": "starting_after",
  "cursor_path": "$.data[-1].id",
  "has_more_path": "$.has_more",
  "page_size_param": "limit",
  "page_size": 100
}

Stop conditions: has_more_path is false, cursor is null/empty, or page is empty.

Link Header (GitHub/Shopify-style)

Follows the rel="next" URL in the RFC 5988 Link response header.

{
  "type": "link-header",
  "page_size_param": "per_page",
  "page_size": 100
}

Stop conditions: no Link header with rel="next", or page is empty.

Next URL (Salesforce/Twilio-style)

Extracts the next page URL from the response body.

{
  "type": "next-url",
  "next_url_path": "$.nextRecordsUrl",
  "base_url": "https://instance.salesforce.com"
}

If the extracted URL is relative, base_url is prepended. Stop conditions: field is null/absent.

Response Config

Field Type Default Description
itemsPath string "$" JSONPath to the array of items in the response
contentType string auto-detect Override content type (json, xml, yaml)
mappings array (required) Element mapping configurations

Element Mapping

Each mapping produces a node or relation from each item:

{
  "elementType": "node",
  "template": {
    "id": "{{item.id}}",
    "labels": ["User"],
    "properties": {
      "name": "{{item.name}}",
      "email": "{{item.email}}"
    }
  }
}

For relations, include from and to:

{
  "elementType": "relation",
  "template": {
    "id": "{{item.id}}",
    "labels": ["FOLLOWS"],
    "from": "{{item.follower_id}}",
    "to": "{{item.following_id}}"
  }
}

Template values use Handlebars syntax. The item variable contains the current response item.

Real-World API Examples

GitHub REST API

Fetch all repositories for an organization. Uses Link header pagination and Bearer token auth.

{
  "endpoints": [
    {
      "url": "https://api.github.com/orgs/my-org/repos",
      "method": "GET",
      "headers": {
        "Accept": "application/vnd.github+json",
        "X-GitHub-Api-Version": "2022-11-28"
      },
      "auth": {
        "type": "bearer",
        "token_env": "GITHUB_TOKEN"
      },
      "pagination": {
        "type": "link-header",
        "page_size_param": "per_page",
        "page_size": 100
      },
      "response": {
        "itemsPath": "$",
        "mappings": [
          {
            "elementType": "node",
            "template": {
              "id": "{{item.id}}",
              "labels": ["Repository"],
              "properties": {
                "name": "{{item.name}}",
                "full_name": "{{item.full_name}}",
                "description": "{{item.description}}",
                "language": "{{item.language}}",
                "stars": "{{item.stargazers_count}}",
                "forks": "{{item.forks_count}}",
                "private": "{{item.private}}"
              }
            }
          }
        ]
      }
    }
  ],
  "timeoutSeconds": 30,
  "maxRetries": 3,
  "retryDelayMs": 2000
}

Shopify REST API

Fetch products from a Shopify store. Uses Link header pagination (with cursor in page_info) and API key header auth.

{
  "endpoints": [
    {
      "url": "https://mystore.myshopify.com/admin/api/2024-01/products.json",
      "method": "GET",
      "auth": {
        "type": "api-key",
        "location": "header",
        "name": "X-Shopify-Access-Token",
        "value_env": "SHOPIFY_ACCESS_TOKEN"
      },
      "pagination": {
        "type": "link-header",
        "page_size_param": "limit",
        "page_size": 50
      },
      "response": {
        "itemsPath": "$.products",
        "mappings": [
          {
            "elementType": "node",
            "template": {
              "id": "{{item.id}}",
              "labels": ["Product"],
              "properties": {
                "title": "{{item.title}}",
                "vendor": "{{item.vendor}}",
                "product_type": "{{item.product_type}}",
                "status": "{{item.status}}",
                "created_at": "{{item.created_at}}"
              }
            }
          }
        ]
      }
    }
  ],
  "timeoutSeconds": 30,
  "maxRetries": 3,
  "retryDelayMs": 1000
}

Stripe API

Fetch all customers. Uses cursor-based pagination (starting_after + has_more) and Basic auth (API key as username).

{
  "endpoints": [
    {
      "url": "https://api.stripe.com/v1/customers",
      "method": "GET",
      "auth": {
        "type": "basic",
        "username_env": "STRIPE_SECRET_KEY"
      },
      "pagination": {
        "type": "cursor",
        "cursor_param": "starting_after",
        "cursor_path": "$.data[-1].id",
        "has_more_path": "$.has_more",
        "page_size_param": "limit",
        "page_size": 100
      },
      "response": {
        "itemsPath": "$.data",
        "mappings": [
          {
            "elementType": "node",
            "template": {
              "id": "{{item.id}}",
              "labels": ["Customer"],
              "properties": {
                "name": "{{item.name}}",
                "email": "{{item.email}}",
                "created": "{{item.created}}",
                "currency": "{{item.currency}}"
              }
            }
          }
        ]
      }
    }
  ],
  "timeoutSeconds": 60,
  "maxRetries": 5,
  "retryDelayMs": 2000
}

Salesforce REST API

Fetch Accounts via SOQL query. Uses next-URL pagination (nextRecordsUrl) and OAuth2 client credentials.

{
  "endpoints": [
    {
      "url": "https://myinstance.salesforce.com/services/data/v59.0/query?q=SELECT+Id,Name,Industry,AnnualRevenue+FROM+Account",
      "method": "GET",
      "auth": {
        "type": "oauth2-client-credentials",
        "token_url": "https://login.salesforce.com/services/oauth2/token",
        "client_id_env": "SF_CLIENT_ID",
        "client_secret_env": "SF_CLIENT_SECRET",
        "scopes": []
      },
      "pagination": {
        "type": "next-url",
        "next_url_path": "$.nextRecordsUrl",
        "base_url": "https://myinstance.salesforce.com"
      },
      "response": {
        "itemsPath": "$.records",
        "mappings": [
          {
            "elementType": "node",
            "template": {
              "id": "{{item.Id}}",
              "labels": ["Account"],
              "properties": {
                "name": "{{item.Name}}",
                "industry": "{{item.Industry}}",
                "annual_revenue": "{{item.AnnualRevenue}}"
              }
            }
          }
        ]
      }
    }
  ],
  "timeoutSeconds": 120,
  "maxRetries": 3,
  "retryDelayMs": 5000
}

Twilio REST API

Fetch call records. Uses next-URL pagination (next_page_uri) and Basic auth (AccountSid:AuthToken).

{
  "endpoints": [
    {
      "url": "https://api.twilio.com/2010-04-01/Accounts/ACXXXXXXXXX/Calls.json",
      "method": "GET",
      "auth": {
        "type": "basic",
        "username_env": "TWILIO_ACCOUNT_SID",
        "password_env": "TWILIO_AUTH_TOKEN"
      },
      "pagination": {
        "type": "next-url",
        "next_url_path": "$.next_page_uri",
        "base_url": "https://api.twilio.com"
      },
      "response": {
        "itemsPath": "$.calls",
        "mappings": [
          {
            "elementType": "node",
            "template": {
              "id": "{{item.sid}}",
              "labels": ["Call"],
              "properties": {
                "from": "{{item.from}}",
                "to": "{{item.to}}",
                "status": "{{item.status}}",
                "duration": "{{item.duration}}",
                "start_time": "{{item.start_time}}"
              }
            }
          }
        ]
      }
    }
  ],
  "timeoutSeconds": 60,
  "maxRetries": 3,
  "retryDelayMs": 1000
}

Development

Building

cargo build -p drasi-bootstrap-http

Running Tests

# Unit tests
cargo test -p drasi-bootstrap-http --lib

# Integration tests (spins up real HTTP servers)
cargo test -p drasi-bootstrap-http --test integration_test

Linting

cargo clippy -p drasi-bootstrap-http