openapi: "3.0.3"
info:
title: SightingDB
version: "0.5.7"
description: |
A database for **sightings**: how often a value has been seen, where, and
when. A sighting is `<namespace, value>`; everything else — counts, first
and last seen, hourly statistics, consensus across namespaces — the
database keeps for you.
## Authentication
Every data and management endpoint takes the API key in an `Authorization`
header, as the key itself with no scheme:
Authorization: changeme
A key carries grants: `r`, `w` or `rw`, each optionally scoped to a
namespace prefix, plus `admin` for the management interface. `/health` and
`/i` need no key at all, and when `authenticate = false` the data endpoints
do not either — the management interface always does.
## Namespaces are paths
`feeds/misp/ips` is one namespace, not three, and it appears in the URL as
a path: `/r/feeds/misp/ips?val=1.2.3.4`. Storage is one file per *top-level*
namespace, which is the unit that is kept in memory or evicted.
servers:
- url: https://localhost:9999
description: A local instance with TLS on, as `--setup` configures it
- url: http://localhost:9999
description: A local instance with `ssl = false`, as the container ships
tags:
- name: sightings
description: Writing and reading sightings
- name: bulk
description: Many sightings in one request
- name: stix
description: STIX 2.1 export
- name: storage
description: What stays in memory
- name: service
description: Health and version, no key required
- name: management
description: The interface behind /_management/, for an `admin` key
security:
- apiKey: []
paths:
/w/{namespace}:
get:
tags: [sightings]
summary: Record a sighting
description: |
Counts one sighting of `val` in `namespace`, creating the namespace if
it is new, and answers with the running count.
A server that does not store this namespace forwards the write to every
live mirror that does, and answers with the first that accepted it. The
answer carries `X-SightingDB-Forwarded: 1`. A namespace nowhere in
reach is `421`; a galaxy wired in a loop is `508`.
A forwarded write is counted for the server it came from rather than
for each mirror that stores it, so fanning one write out to several
mirrors is one contribution on all of them. See `X-SightingDB-Origin`
in the README.
Namespaces whose first path segment begins with `_` — `_all`,
`_shadow/*`, `_config` — are written by the database about itself and
are not writable from outside, whatever the key: `403`. The same holds
for `/wb`, `/vwb` and `/d`. Reading them stays allowed apart from
`_config`, so `/r/_all?val=` is how a value's consensus is asked for.
parameters:
- $ref: "#/components/parameters/Namespace"
- name: val
in: query
required: true
schema: { type: string }
example: 1.2.3.4
- name: timestamp
in: query
description: Unix seconds. Absent records the sighting as now.
schema: { type: integer, format: int64 }
example: 1566624658
- name: ttl
in: query
description: |
Seconds from the last sighting until the value expires. Absent
leaves whatever the value already had; 0 clears it.
schema: { type: integer, format: int64, minimum: 0 }
example: 86400
- name: tags
in: query
description: |
Comma-separated tags, merged with whatever the value already
carried. The vocabulary the STIX export reads is in the README:
`stix-type:`, `tlp:`, `confidence:`, `identity:`, and others.
schema: { type: string }
example: stix-type:ipv4-addr,tlp:amber
responses:
"200":
description: Recorded
content:
application/json:
schema:
type: object
properties:
message: { type: string, example: ok }
count: { type: integer, format: int64, example: 2 }
"400": { $ref: "#/components/responses/BadRequest" }
"401": { $ref: "#/components/responses/Unauthorized" }
"403": { $ref: "#/components/responses/Forbidden" }
/r/{namespace}:
get:
tags: [sightings]
summary: Read a value, or list a whole namespace
description: |
With `val`, answers with that value. Without it, answers with every
value in the namespace.
A read is recorded as a *shadow sighting* under `_shadow/<namespace>`,
so you can see how often something was searched for — including things
that were never found. `noshadow` suppresses that.
parameters:
- $ref: "#/components/parameters/Namespace"
- name: val
in: query
schema: { type: string }
example: 1.2.3.4
- $ref: "#/components/parameters/NoShadow"
- name: count
in: query
allowEmptyValue: true
description: |
Present at any value to answer with how many values the namespace
holds instead of the values themselves. O(1): the value map already
knows its own length, so nothing is walked.
Read `exact` on the answer. It is false when the namespace has ever
held a TTL — a value stops being visible at expiry but is only
removed by the next sweep, so the stored count is an upper bound in
between. Nothing fires at the moment of expiry, so this is a
property of the design rather than a gap a tighter count would fix.
Raises no shadow sighting. Cannot be combined with `val`.
schema: { type: string }
- name: for_merge
in: query
allowEmptyValue: true
description: |
Answer in the shape `POST /_api/merge` takes, so a sync reads from
one server and posts to another unchanged.
With `val`, that one value. Without it, a page of every live value
in the namespace — which is what a catch-up walks — using `offset`
and `limit`.
`counts` is per server, not a total. Offering a total would make
the receiver attribute every server's sightings to the sender, and
two servers exchanging totals inflate each other without bound.
An expired value is not offered: a peer that took it would hold
something this server has already stopped showing.
schema: { type: string }
- name: offset
in: query
description: Where to start, when `for_merge` is asked of a namespace.
schema: { type: integer, minimum: 0 }
- name: limit
in: query
description: |
Values per page, when `for_merge` is asked of a namespace. Default
500, maximum 5000.
schema: { type: integer, minimum: 1, maximum: 5000 }
responses:
"200":
description: |
The value, the whole namespace, or — with `count` — how many values
it holds.
content:
application/json:
schema:
oneOf:
- $ref: "#/components/schemas/Attribute"
- $ref: "#/components/schemas/NamespaceView"
- $ref: "#/components/schemas/ValueCount"
"400": { $ref: "#/components/responses/BadRequest" }
"401": { $ref: "#/components/responses/Unauthorized" }
"403": { $ref: "#/components/responses/Forbidden" }
"404": { $ref: "#/components/responses/NotFound" }
/rs/{namespace}:
get:
tags: [sightings]
summary: Read a value with its hourly statistics
description: |
The same as `/r`, plus `stats`: a count per hour, keyed by the Unix
timestamp of the hour. `val` is required — a whole namespace has no
statistics of its own.
parameters:
- $ref: "#/components/parameters/Namespace"
- name: val
in: query
required: true
schema: { type: string }
- $ref: "#/components/parameters/NoShadow"
responses:
"200":
description: The value, with statistics
content:
application/json:
schema: { $ref: "#/components/schemas/Attribute" }
"400": { $ref: "#/components/responses/BadRequest" }
"404": { $ref: "#/components/responses/NotFound" }
/d/{namespace}:
get:
tags: [sightings]
summary: Delete a namespace
description: |
Removes the namespace and everything in it, giving back the consensus
its values were holding. Needs write access.
parameters:
- $ref: "#/components/parameters/Namespace"
responses:
"200": { $ref: "#/components/responses/Ok" }
"403": { $ref: "#/components/responses/Forbidden" }
"404": { $ref: "#/components/responses/NotFound" }
/wb:
post:
tags: [bulk]
summary: Record many sightings
description: |
Each item is authorized and recorded on its own, and `items` reports
the outcome of every one of them in request order. An item that fails
does not discard the items beside it: the sightings that were accepted
are kept, and a successful item carries the value's resulting `count`.
An item your key may not write fails like any other item: its `status`
is `error`, and the items beside it are still recorded. A refusal is
therefore reported in `items`, not in the status code, so read `items`
rather than the status to find out what happened.
The status describes the batch: `200` with `message: ok` when every
item landed, `200` with `message: partial` when some did, and — when
nothing did — `403` if every item was refused, or `400` otherwise.
requestBody:
required: true
content:
application/json:
schema: { $ref: "#/components/schemas/BulkRequest" }
example:
items:
- namespace: feeds/misp/ips
value: 1.2.3.4
tags: stix-type:ipv4-addr,tlp:amber
- namespace: feeds/misp/domains
value: evil.example
timestamp: 1566624658
ttl: 86400
responses:
"200":
description: Everything, or some of it, was recorded
content:
application/json:
schema: { $ref: "#/components/schemas/BulkWriteResponse" }
"400":
description: |
Nothing could be recorded, for reasons that were not all refusals.
content:
application/json:
schema: { $ref: "#/components/schemas/BulkWriteResponse" }
"403":
description: |
Nothing was recorded and every item was refused. A batch in which
only *some* items were refused answers `200`, with the refusals in
`items`.
content:
application/json:
schema: { $ref: "#/components/schemas/BulkWriteResponse" }
/vwb:
post:
tags: [bulk]
summary: Check a bulk write without recording it
description: |
A dry run of `/wb`. Takes the same body and answers what `/wb` would
have answered for it — the same status code, the same `message`, and
the same `status` and `error` on every item — without recording a
single sighting.
Use it to check a batch before committing to it: `message: ok` here
means the same batch sent to `/wb` is accepted in full. The response
field is `writable` rather than `written`, because nothing was.
Both routes decide each item with the same code, so the preview cannot
drift from the writer. It is a report, not a reservation: the ACL can
be rewritten between the two calls, so a batch that validates can still
be refused when it is written. Read the `items` that `/wb` itself
returns to learn what actually happened.
Items report no `count`, since nothing was counted.
requestBody:
required: true
content:
application/json:
schema: { $ref: "#/components/schemas/BulkRequest" }
example:
items:
- namespace: feeds/misp/ips
value: 1.2.3.4
tags: stix-type:ipv4-addr,tlp:amber
- namespace: feeds/misp/domains
value: evil.example
responses:
"200":
description: |
The batch was checked. Every item is writable (`message: ok`), or
only some are (`message: partial`).
content:
application/json:
schema: { $ref: "#/components/schemas/BulkValidateResponse" }
"400":
description: |
No item is writable, for reasons that were not all refusals.
content:
application/json:
schema: { $ref: "#/components/schemas/BulkValidateResponse" }
"403":
description: |
No item is writable and every one of them was refused.
content:
application/json:
schema: { $ref: "#/components/schemas/BulkValidateResponse" }
"401": { $ref: "#/components/responses/Unauthorized" }
/rb:
post:
tags: [bulk]
summary: Read many values
requestBody:
required: true
content:
application/json:
schema: { $ref: "#/components/schemas/BulkRequest" }
example:
items:
- namespace: feeds/misp/ips
value: 1.2.3.4
noshadow: true
responses:
"200":
description: One entry per item, in the order asked for
content:
application/json:
schema:
type: object
properties:
items:
type: array
description: Each item is an attribute, or an error object.
items: { type: object }
"403": { $ref: "#/components/responses/Forbidden" }
/rbs:
post:
tags: [bulk]
summary: Read many values, with statistics
requestBody:
required: true
content:
application/json:
schema: { $ref: "#/components/schemas/BulkRequest" }
responses:
"200":
description: One entry per item, each carrying its hourly statistics
content:
application/json:
schema:
type: object
properties:
items:
type: array
items: { type: object }
/stix/{namespace}:
get:
tags: [stix]
summary: Export one namespace as a STIX 2.1 bundle
description: |
Each value becomes an `indicator` carrying its pattern and a `sighting`
with the count and window, plus the identities and TLP markings they
refer to. Ids are derived, so the same data exports byte for byte the
same and one value in two namespaces yields one indicator.
A value whose observable type cannot be worked out is skipped, and the
response headers say how many that was.
parameters:
- $ref: "#/components/parameters/Namespace"
- name: recursive
in: query
allowEmptyValue: true
description: |
Present at any value to also export every namespace below this one,
as one bundle. "Below" matches whole path segments, so `feeds`
covers `feeds/misp/ips` and never `feeds-internal`.
The namespace named is authorized as always; one found underneath
that the key may not read is left out rather than refusing the
export. `X-SightingDB-Namespaces` reports how many contributed.
schema: { type: string }
- name: untyped
in: query
description: |
`include` also exports values whose observable type could not be
worked out, as `x-sightingdb-value` and flagged
`x_sightingdb_untyped`. Absent, or anything else, leaves them out
and counts them in `X-SightingDB-Skipped` — which is what this
route has always done.
schema: { type: string, enum: [skip, include] }
- name: limit
in: query
description: Values read, 10000 by default and 100000 at most.
schema: { type: integer, minimum: 1, maximum: 100000 }
responses:
"200":
description: A STIX 2.1 bundle
headers:
X-SightingDB-Exported: { $ref: "#/components/headers/Exported" }
X-SightingDB-Skipped: { $ref: "#/components/headers/Skipped" }
X-SightingDB-Untyped: { $ref: "#/components/headers/Untyped" }
X-SightingDB-Namespaces: { $ref: "#/components/headers/Namespaces" }
X-SightingDB-Truncated: { $ref: "#/components/headers/Truncated" }
content:
application/stix+json:
schema: { $ref: "#/components/schemas/StixBundle" }
"403": { $ref: "#/components/responses/Forbidden" }
"404": { $ref: "#/components/responses/NotFound" }
/_api/merge:
post:
tags: [bulk]
summary: Fold a peer's copy of values into ours
description: |
How a galaxy syncs. **Not a sighting**: nothing is counted here. The
sender states what each server has seen, and every field is combined by
a rule that does not care about order or repetition — so the same merge
may be sent twice, or two peers' copies may arrive either way round, and
the result is the same.
That is why this exists rather than reusing `/wb`. `/w` means "add one",
and replaying it doubles the count: measured on two instances, three
sightings became nine after two rounds of reading a peer and writing
back what was found.
The rules:
* `counts` — the greater of the two, per server. A server's own count
only rises, so taking the larger converges whichever copy arrives
first. Setting outright would let a stale copy undo a newer one.
* `stats` — the same, per bucket.
* `first_seen` — the earlier. `last_seen` — the later.
* `tags` — the union.
* `ttl` — the shortest non-zero one, zero meaning never. A value is
kept only as long as the most cautious server says.
An entry naming **this** server is ignored and counted in
`ignored_self`: a peer does not get to say what this server has seen.
Authorized as a write — the same ACL, the same refusal of internal
namespaces, the same `421` for a namespace this server does not store —
so a peer's key bounds what it may merge exactly as it bounds what it
may write.
requestBody:
required: true
content:
application/json:
schema:
type: object
properties:
items:
type: array
items: { $ref: "#/components/schemas/MergeItem" }
example:
items:
- namespace: feeds/misp/ips
value: 1.2.3.4
counts: { node-b: 3 }
stats: { node-b: { "1600000000": 3 } }
first_seen: 1600000000
last_seen: 1600003600
tags: tlp:amber
responses:
"200":
description: |
Every item merged, or only some. An item that changed nothing is a
success, not a failure: during catch-up it is how a caller learns it
has converged.
content:
application/json:
schema: { $ref: "#/components/schemas/MergeResponse" }
"400":
description: Nothing merged, for reasons that were not all refusals.
content:
application/json:
schema: { $ref: "#/components/schemas/MergeResponse" }
"403":
description: Nothing merged, and every item was refused.
content:
application/json:
schema: { $ref: "#/components/schemas/MergeResponse" }
"401": { $ref: "#/components/responses/Unauthorized" }
/_api/namespaces:
get:
tags: [bulk]
summary: Which namespaces exist here
description: |
What a catch-up needs that nothing else gave it: a server that was down
does not know about namespaces created while it was away, so it cannot
ask for their values.
Read-authorized per namespace, like browsing, so a name out of this
key's reach is simply absent rather than refused — the answer is "what
you may know about".
Deliberately not the management interface's namespace listing, which
needs an `admin` grant: a peer key should be able to be the narrowest
thing that does the job.
parameters:
- name: prefix
in: query
description: |
Only namespaces at or under this one, matched on whole path
segments. Absent means all of them.
schema: { type: string }
example: feeds
responses:
"200":
description: The namespaces this key may know about
content:
application/json:
schema:
type: object
properties:
namespaces:
type: array
items: { type: string }
"401": { $ref: "#/components/responses/Unauthorized" }
/_api/stix:
post:
tags: [stix]
summary: Export one or more namespaces as STIX 2.1
description: |
The same export for a script: a namespace is a path, so POST saves
encoding it into a URL, and several can be gathered into one bundle.
Every namespace is authorized on its own — naming one the key may not
read refuses the whole request.
requestBody:
required: true
content:
application/json:
schema: { $ref: "#/components/schemas/ExportRequest" }
example:
namespaces: [feeds/misp/ips, feeds/otx/ips]
q: "10.0."
limit: 5000
responses:
"200":
description: A STIX 2.1 bundle
headers:
X-SightingDB-Exported: { $ref: "#/components/headers/Exported" }
X-SightingDB-Skipped: { $ref: "#/components/headers/Skipped" }
X-SightingDB-Untyped: { $ref: "#/components/headers/Untyped" }
X-SightingDB-Namespaces: { $ref: "#/components/headers/Namespaces" }
X-SightingDB-Truncated: { $ref: "#/components/headers/Truncated" }
X-SightingDB-Missing:
description: Namespaces asked for that do not exist, comma separated.
schema: { type: string }
content:
application/stix+json:
schema: { $ref: "#/components/schemas/StixBundle" }
"400": { $ref: "#/components/responses/BadRequest" }
"403": { $ref: "#/components/responses/Forbidden" }
"404":
description: None of the namespaces asked for exist
/_api/tier:
post:
tags: [storage]
summary: Set what stays in memory, and for how long
description: |
Name any namespace and the setting lands on its top-level namespace,
which is one file paged in and out as a unit — so it covers everything
under it, and the reply says so.
Either field takes `"default"` to stop overriding. Needs write access
and a configured `tiers_file`, which is where the change is written so
it survives a restart.
requestBody:
required: true
content:
application/json:
schema: { $ref: "#/components/schemas/TierChange" }
example:
namespace: feeds/misp/ips
tier: warm
warm_idle: 86400
responses:
"200":
description: The settings now in force
content:
application/json:
schema: { $ref: "#/components/schemas/TierResult" }
"400": { $ref: "#/components/responses/BadRequest" }
"403": { $ref: "#/components/responses/Forbidden" }
"409":
description: No tiers_file is configured, so there is nowhere to write it
/health:
get:
tags: [service]
summary: Liveness and readiness
description: |
Needs no key whatever `authenticate` is set to. There is no separate
readiness path: the snapshot is restored before the listener is bound,
so an answer at all means the database is up.
security: []
responses:
"200":
description: The process, and what it holds in memory
content:
application/json:
schema: { $ref: "#/components/schemas/Health" }
/i:
get:
tags: [service]
summary: Implementation and version
security: []
responses:
"200":
description: What this is
content:
application/json:
schema:
type: object
properties:
implementation: { type: string, example: SightingDB }
version: { type: string, example: "0.5.7" }
vendor: { type: string }
author: { type: string }
/_api/openapi.yaml:
get:
tags: [service]
summary: This specification, as served by the running instance
security: []
responses:
"200":
description: The OpenAPI document
content:
application/yaml:
schema: { type: string }
/c/{namespace}:
get:
tags: [service]
summary: Not implemented
description: Reserved for per-namespace configuration; answers 501.
parameters:
- $ref: "#/components/parameters/Namespace"
responses:
"501":
description: Not implemented
content:
application/json:
schema: { $ref: "#/components/schemas/Message" }
/_management/api/session:
get:
tags: [management]
summary: Check a key holds the admin grant
responses:
"200": { $ref: "#/components/responses/Ok" }
"401": { $ref: "#/components/responses/Unauthorized" }
"403": { $ref: "#/components/responses/Forbidden" }
/_management/api/info:
get:
tags: [management]
summary: What this server is configured to do
description: |
Read from the configuration file at startup, so this reports rather
than edits.
responses:
"200":
description: The configuration in force
content:
application/json:
schema: { type: object }
/_management/api/namespaces:
get:
tags: [management]
summary: List namespaces
parameters:
- name: q
in: query
description: Substring filter over whole namespace names.
schema: { type: string }
- $ref: "#/components/parameters/Offset"
- $ref: "#/components/parameters/Limit"
responses:
"200":
description: A page of namespaces
content:
application/json:
schema:
allOf:
- $ref: "#/components/schemas/Page"
- type: object
properties:
items:
type: array
items: { $ref: "#/components/schemas/NamespaceEntry" }
post:
tags: [management]
summary: Create a namespace
description: |
Creates one before anything is written to it, nesting as deep as you
like: `feeds/misp/ips` creates the whole path. An empty namespace is a
real namespace — it is snapshotted and survives a restart.
requestBody:
required: true
content:
application/json:
schema:
type: object
required: [namespace]
properties:
namespace: { type: string, example: feeds/misp/ips }
responses:
"200":
description: Created
content:
application/json:
schema:
type: object
properties:
namespace: { type: string }
"400": { $ref: "#/components/responses/BadRequest" }
"403": { $ref: "#/components/responses/Forbidden" }
"409":
description: It already exists
/_management/api/tree:
get:
tags: [management]
summary: One level of the namespace tree
description: |
Namespaces are flat paths; this groups them by the segment after
`path`, so an interface can browse them like folders.
parameters:
- name: path
in: query
description: The folder to open. Empty is the root.
schema: { type: string }
example: feeds
- name: q
in: query
description: Substring filter over the names at this level.
schema: { type: string }
- $ref: "#/components/parameters/Offset"
- $ref: "#/components/parameters/Limit"
responses:
"200":
description: A page of children
content:
application/json:
schema:
allOf:
- $ref: "#/components/schemas/Page"
- type: object
properties:
items:
type: array
items: { $ref: "#/components/schemas/TreeEntry" }
/_management/api/galaxy:
get:
tags: [management]
summary: This server and the peers it knows
description: |
What a topology view reads: this server's own role, and each peer's
health as last probed. One request against whichever server the
operator is connected to.
Peer **keys are never included** — they are credentials, and this
response is rendered in a browser.
Health is probed, not assumed: each peer's `/health` is called every
`health_interval` seconds. `/health` needs no key, so a probe carries no
credential. What is reported is reachability *from this server*, which
is the thing that matters to it.
Nothing is forwarded to peers yet, so this describes the configured
galaxy and whether its members answer — not a traversal of it. A
cascade is reported one level deep: each peer appears as a peer, and
asking *it* for its own galaxy reaches the next level.
responses:
"200":
description: The galaxy as this server sees it
content:
application/json:
schema:
type: object
properties:
self:
type: object
description: The server answering, so a view has a root.
properties:
node_id: { type: string }
version: { type: string }
uptime_seconds: { type: integer }
role: { $ref: "#/components/schemas/Role" }
peers:
type: array
items: { $ref: "#/components/schemas/PeerHealth" }
below:
type: object
description: |
Each peer's own answer to this request, keyed by its url,
so one request describes a whole cascade. A peer that is
itself a router has peers of its own in here.
One level per hop, bounded by `max_hops`: a view of a
miswired galaxy terminates like everything else. A peer
that could not be reached is absent here and its health
in `peers` says why — a topology view that failed
because one server is down would be useless exactly when
it is needed.
additionalProperties: true
"401": { $ref: "#/components/responses/Unauthorized" }
"403": { $ref: "#/components/responses/Forbidden" }
/_management/api/rejections:
get:
tags: [management]
summary: Values that were not written
description: |
Every write path records what it turned away here — `/w`, `/wb`, the
management interface, and the ZMQ ingest, which has no caller of its
own to tell. It answers "which values errored?" after the response that
reported them has gone.
Newest first. Entries are filtered by what the key may *read*: a
rejection names a namespace and a value someone tried to put in it.
The record is bounded and held in memory: once `total` reaches
`capacity`, the oldest entries are being dropped to make room. It does
not survive a restart and is not written to snapshots — it is for
diagnosis, not evidence. The cap is `rejection_log` in `[daemon]`, and
0 switches it off.
`/vwb` records nothing here, since it writes nothing.
parameters:
- name: namespace
in: query
description: |
Only rejections under this namespace, matched as a subtree. Needs
read access to it, and answers `404` if the key has none.
schema: { type: string }
example: feeds/misp
- name: limit
in: query
schema: { type: integer, minimum: 1, maximum: 500, default: 50 }
responses:
"200":
description: The most recent rejections this key may see
content:
application/json:
schema:
type: object
properties:
rejections:
type: array
items: { $ref: "#/components/schemas/Rejection" }
total:
type: integer
description: How many are held in all, not just on this page.
capacity:
type: integer
description: |
The cap. Once `total` reaches it, the oldest rejections
are being lost.
"401": { $ref: "#/components/responses/Unauthorized" }
"403": { $ref: "#/components/responses/Forbidden" }
"404": { $ref: "#/components/responses/NotFound" }
delete:
tags: [management]
summary: Forget every rejection
description: |
Clears the record, which is how an operator marks a feed as dealt with.
All-or-nothing, so it needs a key with unscoped read access.
responses:
"200": { $ref: "#/components/responses/Ok" }
"401": { $ref: "#/components/responses/Unauthorized" }
"403": { $ref: "#/components/responses/Forbidden" }
/_management/api/values:
get:
tags: [management]
summary: The values in one namespace
parameters:
- name: namespace
in: query
required: true
schema: { type: string }
- name: q
in: query
schema: { type: string }
- $ref: "#/components/parameters/Offset"
- $ref: "#/components/parameters/Limit"
responses:
"200":
description: A page of values, without their statistics
content:
application/json:
schema:
allOf:
- $ref: "#/components/schemas/Page"
- type: object
properties:
items:
type: array
items: { $ref: "#/components/schemas/Attribute" }
"404": { $ref: "#/components/responses/NotFound" }
post:
tags: [management]
summary: Add one value, or a pasted list of them
requestBody:
required: true
content:
application/json:
schema:
type: object
required: [namespace, values]
properties:
namespace: { type: string, example: feeds/misp/ips }
values:
type: array
items: { type: string }
example: [1.2.3.4, 5.6.7.8]
tags: { type: string, example: stix-type:ipv4-addr }
ttl: { type: integer, format: int64 }
timestamp: { type: integer, format: int64 }
responses:
"200":
description: What was recorded, and what was rejected
content:
application/json:
schema:
type: object
properties:
namespace: { type: string }
written: { type: integer }
counts:
type: array
description: |
Each value that was recorded, with its running total
afterwards — the same number `/w` answers with. Omitted
when nothing was recorded.
items:
type: object
properties:
value: { type: string }
count: { type: integer, format: int64 }
errors:
type: array
description: |
The values that were rejected; omitted when there are
none. Blank lines are dropped before anything is tried,
so in practice a rejection here is a property of the
namespace and applies to every value in the request.
items:
type: object
properties:
value: { type: string }
error: { type: string }
"400": { $ref: "#/components/responses/BadRequest" }
"403": { $ref: "#/components/responses/Forbidden" }
/_management/api/value:
get:
tags: [management]
summary: One value, with its hourly statistics
parameters:
- name: namespace
in: query
required: true
schema: { type: string }
- name: value
in: query
required: true
schema: { type: string }
responses:
"200":
description: The value
content:
application/json:
schema: { $ref: "#/components/schemas/Attribute" }
"404": { $ref: "#/components/responses/NotFound" }
/_management/api/sightings:
get:
tags: [management]
summary: Every namespace holding one value
description: |
What the relationship graph draws. Only namespaces the key may read are
returned, while `consensus` counts them all — so a scoped key can tell
it is not seeing everything without being told the names.
parameters:
- name: value
in: query
required: true
schema: { type: string }
- name: limit
in: query
schema: { type: integer, minimum: 1, maximum: 500 }
responses:
"200":
description: Where the value has been seen
content:
application/json:
schema:
type: object
properties:
value: { type: string }
consensus: { type: integer, format: int64 }
truncated: { type: boolean }
paged_in:
type: boolean
description: Finding them all meant reading shards from disk.
items:
type: array
items:
type: object
properties:
namespace: { type: string }
shard: { type: string }
count: { type: integer, format: int64 }
first_seen: { type: integer, format: int64 }
last_seen: { type: integer, format: int64 }
"400": { $ref: "#/components/responses/BadRequest" }
/_management/api/tags:
post:
tags: [management]
summary: Replace a value's tags
description: |
Writes through `/w` merge tags; this replaces the set, which is the only
way a wrong tag comes off. It is not a sighting: nothing is counted and
no timestamp moves.
requestBody:
required: true
content:
application/json:
schema:
type: object
required: [namespace, value, tags]
properties:
namespace: { type: string }
value: { type: string }
tags:
type: string
description: The whole set, comma separated. Empty clears it.
example: stix-type:ipv4-addr,tlp:amber
responses:
"200":
description: The value as it now stands
content:
application/json:
schema: { $ref: "#/components/schemas/Attribute" }
"403": { $ref: "#/components/responses/Forbidden" }
"404": { $ref: "#/components/responses/NotFound" }
/_management/api/tier:
post:
tags: [management, storage]
summary: Set what stays in memory, and for how long
description: The same as `/_api/tier`, for the interface's own key.
requestBody:
required: true
content:
application/json:
schema: { $ref: "#/components/schemas/TierChange" }
responses:
"200":
description: The settings now in force
content:
application/json:
schema: { $ref: "#/components/schemas/TierResult" }
"400": { $ref: "#/components/responses/BadRequest" }
"409":
description: No tiers_file is configured
/_management/api/keys:
get:
tags: [management]
summary: List API keys and their grants
responses:
"200":
description: Every key
content:
application/json:
schema:
type: array
items: { $ref: "#/components/schemas/KeyEntry" }
post:
tags: [management]
summary: Create or replace a key
description: |
Written to the configured `acl_file` and adopted at once, with no
restart. Removing the admin grant from the last admin key is refused.
requestBody:
required: true
content:
application/json:
schema: { $ref: "#/components/schemas/KeyEntry" }
example:
key: analyst
admin: false
read: [feeds]
write: []
responses:
"200":
description: The key as saved
content:
application/json:
schema: { $ref: "#/components/schemas/KeyEntry" }
"400": { $ref: "#/components/responses/BadRequest" }
"409":
description: That would leave no admin key, or no acl_file is configured
/_management/api/keys/drift:
get:
tags: [management]
summary: Where this server's keys and its peers' disagree
description: |
Keys are gossiped to the peers a server administers, and the periodic
offer is deliberately **additive** — it never deletes a key a peer has
and this server does not, because this server is not necessarily the
only place keys are managed.
The consequence is that a key revoked while a peer was down stays live
on that peer. That is a security hole if it is invisible and merely a
chore if it is not, which is what this is for.
Asked through each peer's own management interface, so a peer this
server does not administer reports why rather than appearing to agree.
responses:
"200":
description: Each peer, and how its key list differs from this one
content:
application/json:
schema:
type: object
properties:
peers:
type: array
items:
type: object
properties:
url: { type: string }
readable:
type: boolean
description: |
Whether this server was allowed to ask at all.
`false` means it holds no `admin` key there, which
for a deliberately narrow peer key is expected.
error: { type: string }
revoked_but_present:
type: array
items: { type: string }
description: |
Keys this server **revoked** and the peer still
accepts — a revocation that did not land. **Revoke
them again now the peer is reachable.**
The record of revocations is held in memory, so
after a restart one of these moves into
`only_on_peer` and stops being flagged.
only_on_peer:
type: array
items: { type: string }
description: |
Keys the peer has that this server never knew
about. Ordinary — a peer has its own keys,
including the one this server authenticates with —
and kept separate so it does not drown the case
above.
missing:
type: array
items: { type: string }
description: |
Keys this server has and the peer does not. Usually
a peer that has not had the periodic offer yet.
agrees:
type: boolean
description: Readable, and nothing differs.
"401": { $ref: "#/components/responses/Unauthorized" }
"403": { $ref: "#/components/responses/Forbidden" }
/_management/api/keys/generate:
get:
tags: [management]
summary: Suggest a strong key
responses:
"200":
description: A key nobody has to invent
content:
application/json:
schema:
type: object
properties:
key: { type: string }
/_management/api/keys/{key}:
delete:
tags: [management]
summary: Revoke a key
parameters:
- name: key
in: path
required: true
schema: { type: string }
responses:
"200": { $ref: "#/components/responses/Ok" }
"404": { $ref: "#/components/responses/NotFound" }
"409":
description: That would revoke the last admin key
components:
securitySchemes:
apiKey:
type: apiKey
in: header
name: Authorization
description: |
The key on its own, with no `Bearer` or other scheme in front of it.
parameters:
Namespace:
name: namespace
in: path
required: true
description: |
A namespace path such as `feeds/misp/ips`. Slashes are part of the
name; do not encode them.
schema: { type: string }
example: feeds/misp/ips
NoShadow:
name: noshadow
in: query
description: |
Present at any value — including empty — to suppress the shadow
sighting this read would otherwise record.
allowEmptyValue: true
schema: { type: string }
Offset:
name: offset
in: query
schema: { type: integer, minimum: 0, default: 0 }
Limit:
name: limit
in: query
schema: { type: integer, minimum: 1, maximum: 500, default: 50 }
headers:
Exported:
description: Values written into the bundle.
schema: { type: integer }
Skipped:
description: |
Values left out because their observable type was unknown. Always 0
when `untyped` is `include`, which exports them instead.
schema: { type: integer }
Untyped:
description: |
How many of the exported values went out as `x-sightingdb-value`
because nothing identified them. A value carrying an explicit
`stix-type:` tag is never counted here.
schema: { type: integer }
Forwarded:
description: |
Present when this answer came from another server in the galaxy rather
than from the one asked. The body is that server's answer, unchanged.
schema: { type: string }
Namespaces:
description: |
How many namespaces contributed values to the bundle. Worth reading
after a recursive export, where one namespace was named and a subtree
may have been returned.
schema: { type: integer }
Truncated:
description: |
Whether there was more than the export read — either within a
namespace, or with namespaces still to go when the shared `limit`
budget ran out.
schema: { type: boolean }
responses:
Ok:
description: Done
content:
application/json:
schema: { $ref: "#/components/schemas/Message" }
BadRequest:
description: The request could not be read, or asked for something impossible
content:
application/json:
schema: { $ref: "#/components/schemas/Message" }
Unauthorized:
description: No API key was given
content:
application/json:
schema: { $ref: "#/components/schemas/Message" }
Forbidden:
description: |
The key may not do this. Deliberately the same answer whether the key
is unknown or merely unauthorised, so probing cannot tell them apart.
content:
application/json:
schema: { $ref: "#/components/schemas/Message" }
NotFound:
description: No such namespace, or no such value in it
content:
application/json:
schema: { $ref: "#/components/schemas/Message" }
schemas:
Message:
type: object
properties:
message: { type: string }
Attribute:
type: object
description: One value, and what is known about it.
properties:
value: { type: string, example: 1.2.3.4 }
first_seen:
type: integer
format: int64
description: Unix seconds.
example: 1566624658
last_seen: { type: integer, format: int64, example: 1566624689 }
count:
type: integer
format: int64
description: Sightings in this namespace.
example: 2
consensus:
type: integer
format: int64
description: How many namespaces have seen this value.
example: 3
ttl:
type: integer
format: int64
description: Seconds from the last sighting until it expires; 0 never.
tags:
type: string
description: Comma-separated set. See the tag vocabulary in the README.
example: stix-type:ipv4-addr,tlp:amber
stats:
type: object
description: |
Sightings per hour, keyed by the Unix timestamp of the hour. Only
present on `/rs`, `/rbs` and the management value endpoint.
additionalProperties: { type: integer, format: int64 }
NamespaceView:
type: object
description: Every value in a namespace, as `/r/<namespace>` answers.
properties:
attributes:
type: array
items: { $ref: "#/components/schemas/Attribute" }
Page:
type: object
properties:
total:
type: integer
description: Matches before paging, not the number returned.
offset: { type: integer }
NamespaceEntry:
type: object
properties:
namespace: { type: string, example: feeds/misp/ips }
shard:
type: string
description: The top-level namespace, which storage settings belong to.
example: feeds
tier:
type: string
enum: [hot, warm, cold]
resident:
type: boolean
description: Whether the shard is in memory rather than on disk.
warm_idle:
type: integer
format: int64
description: Seconds a warm shard may sit untouched.
own_tier:
type: boolean
description: The tier is set on the shard rather than the default.
own_warm_idle: { type: boolean }
TreeEntry:
type: object
properties:
name:
type: string
description: The path segment, which is what a row is labelled with.
example: misp
path: { type: string, example: feeds/misp }
is_namespace:
type: boolean
description: |
Whether the path is a namespace of its own and so may hold values.
A path can be both this and a folder with others underneath.
descendants: { type: integer }
shard: { type: string }
tier: { type: string, enum: [hot, warm, cold] }
resident: { type: boolean }
warm_idle: { type: integer, format: int64 }
own_tier: { type: boolean }
own_warm_idle: { type: boolean }
BulkRequest:
type: object
required: [items]
properties:
items:
type: array
items:
type: object
required: [namespace, value]
properties:
namespace: { type: string }
value: { type: string }
timestamp: { type: integer, format: int64, nullable: true }
ttl: { type: integer, format: int64, nullable: true }
tags: { type: string }
noshadow: { type: boolean }
BulkWriteResponse:
type: object
properties:
message:
type: string
enum: [ok, partial, failed]
written: { type: integer }
items:
type: array
description: |
One entry per request item, in request order, so an outcome can be
matched back onto what was sent even when a value repeats.
items: { $ref: "#/components/schemas/BulkWriteItem" }
errors:
type: array
description: |
The failures alone. Predates `items` and is kept for clients that
read it; every entry here also appears in `items`. Omitted when
there are none.
items:
type: object
properties:
namespace: { type: string }
value: { type: string }
error: { type: string }
ValueCount:
type: object
description: How many values a namespace holds, for `/r/{namespace}?count`.
properties:
namespace: { type: string }
values:
type: integer
description: How many values are stored.
exact:
type: boolean
description: |
Whether `values` is the number a reader would see. False when the
namespace has ever held a TTL, in which case `values` is an upper
bound: expired values stay stored until the next sweep. Sticky —
once a TTL has been seen this stays false, so it errs towards
claiming less.
paged_in:
type: boolean
description: |
Whether answering had to read an evicted shard back into memory.
Role:
type: object
description: |
What a server stores and who it knows about: the two things that decide
its place in a galaxy.
properties:
kind:
type: string
enum: [node, router, both]
description: |
`node` stores namespaces and forwards nothing; `router` stores none
of its own and exists to forward; `both` does each. A
configuration, not a type — any server can be any of them.
mirrors_everything:
type: boolean
description: Whether it stores every namespace — a full mirror.
namespaces:
type: array
items: { type: string }
description: |
The prefixes stored, when it is not everything. Empty on a router.
peers:
type: array
items: { type: string }
description: Peer urls. Keys are never included.
max_hops:
type: integer
description: |
How many hops a forwarded request may take. `0` when there is no
`[galaxy]`.
PeerHealth:
type: object
description: What the last probe of one peer found.
properties:
url: { type: string }
online:
type: boolean
description: Whether the last probe succeeded.
probed:
type: boolean
description: |
Whether this peer has been probed at all yet. `false` with
`online: false` means "not asked", which is not the same as down.
last_seen:
type: integer
format: int64
description: |
Unix seconds of the last *successful* probe, `0` if there has never
been one. Kept across failures, so it says how stale a peer is.
latency_ms:
type: integer
description: Round trip of the last successful probe.
version:
type: string
description: |
The peer's own version. A galaxy running mixed versions is worth
seeing before it misbehaves.
error:
type: string
description: Why the last probe failed.
failures:
type: integer
description: |
Consecutive failures, so a flapping peer reads differently from one
that has been gone all week.
MergeItem:
type: object
description: One value as a peer holds it.
required: [namespace, value, counts, first_seen, last_seen]
properties:
namespace: { type: string }
value: { type: string }
counts:
type: object
additionalProperties: { type: integer, format: int64 }
description: |
The peer's view of each server's contribution, keyed by node id. An
entry naming the receiving server is ignored.
stats:
type: object
description: |
The peer's view of each server's hourly buckets: node id, then the
Unix timestamp of the hour.
additionalProperties:
type: object
additionalProperties: { type: integer, format: int64 }
first_seen: { type: integer, format: int64 }
last_seen: { type: integer, format: int64 }
tags: { type: string }
ttl: { type: integer, format: int64 }
MergeResponse:
type: object
properties:
message:
type: string
enum: [ok, partial, failed]
changed:
type: integer
description: How many items changed the local copy.
items:
type: array
items: { $ref: "#/components/schemas/MergeResult" }
errors:
type: array
description: The failures alone; omitted when there are none.
items:
type: object
properties:
namespace: { type: string }
value: { type: string }
error: { type: string }
MergeResult:
type: object
properties:
index: { type: integer }
namespace: { type: string }
value: { type: string }
status:
type: string
enum: [ok, error]
changed:
type: boolean
description: |
Whether the local copy changed. `false` means it already held
everything offered, which during catch-up is how convergence is
recognised.
count:
type: integer
format: int64
description: The total after merging.
ignored_self:
type: integer
description: |
Entries naming the receiving server, which were ignored. Non-zero
means the sender is confused about who it is talking to.
error: { type: string }
Rejection:
type: object
description: One value that was not written, and why.
properties:
when:
type: integer
format: int64
description: Unix seconds.
namespace: { type: string }
value:
type: string
description: |
The value as it arrived — which, for the commonest rejection of
all, is the empty string.
reason:
type: string
description: Why it was turned away, in the words the caller was given.
source:
type: string
enum: [write, bulkwrite, management, ingest]
description: Which write path turned it away.
BulkValidateResponse:
type: object
description: |
What `/vwb` found. The same shape as `BulkWriteResponse`, except that
`written` is `writable` — a count of what *would* be recorded — and no
item carries a `count`, because nothing was counted.
properties:
message:
type: string
enum: [ok, partial, failed]
writable:
type: integer
description: How many items would be recorded. Nothing has been.
items:
type: array
items: { $ref: "#/components/schemas/BulkWriteItem" }
errors:
type: array
description: The unwritable items alone; omitted when there are none.
items:
type: object
properties:
namespace: { type: string }
value: { type: string }
error: { type: string }
BulkWriteItem:
type: object
properties:
index:
type: integer
description: Position of this item in the request.
namespace: { type: string }
value: { type: string }
status:
type: string
enum: [ok, error]
count:
type: integer
format: int64
description: |
The running total for this value after the sighting, taken under
the same lock that incremented it. Present only when `status` is
`ok`, and it removes the need for a follow-up read.
error:
type: string
description: Present only when `status` is `error`.
ExportRequest:
type: object
description: Name one namespace, or several to gather into one bundle.
properties:
namespace:
type: string
description: Shorthand for a single namespace.
namespaces:
type: array
items: { type: string }
recursive:
type: boolean
default: false
description: |
Also export every namespace below each one named, as one bundle.
"Below" matches whole path segments. Each namespace named is
authorized first; namespaces found underneath that the key may not
read are left out rather than refusing the export.
untyped:
type: string
enum: [skip, include]
default: skip
description: |
What to do with a value whose observable type cannot be worked out.
`skip` leaves it out and reports it; `include` exports it as
`x-sightingdb-value`, flagged `x_sightingdb_untyped` so a consumer
can filter. STIX 2.1 has no plain-text observable, so this is a
custom type rather than a standard one.
q:
type: string
description: Substring filter over values.
limit:
type: integer
minimum: 1
maximum: 100000
default: 10000
StixBundle:
type: object
description: A STIX 2.1 bundle. See https://oasis-open.github.io/cti-documentation/
properties:
type: { type: string, example: bundle }
id: { type: string, example: bundle--2ac7882f-76a3-4a9b-97b3-811b3af1c7c0 }
objects:
type: array
items: { type: object }
TierChange:
type: object
description: |
Both halves are optional, and both take `"default"` to stop overriding.
Sending neither clears the setting entirely.
properties:
namespace:
type: string
description: Any namespace; the setting lands on its top-level namespace.
example: feeds/misp/ips
tier:
type: string
description: hot, warm, cold, or default.
example: warm
warm_idle:
description: |
Seconds a warm shard may sit untouched. A number, or the string
`"default"` to go back to the configured window.
oneOf:
- type: integer
format: int64
minimum: 0
- type: string
example: 86400
TierResult:
type: object
properties:
shard: { type: string }
tier: { type: string, enum: [hot, warm, cold] }
warm_idle: { type: integer, format: int64 }
own_tier: { type: boolean }
own_warm_idle: { type: boolean }
effect:
type: string
description: What the shard will now do, in a sentence.
example: "'feeds' and everything under it is dropped after 86400s untouched"
KeyEntry:
type: object
required: [key, admin, read, write]
properties:
key: { type: string }
admin:
type: boolean
description: Whether the key may use the management interface.
read:
type: array
description: Namespace prefixes this key may read. An empty string is all.
items: { type: string }
write:
type: array
items: { type: string }
Health:
type: object
properties:
status: { type: string, example: ok }
version: { type: string }
uptime_seconds: { type: integer, format: int64 }
resident_shards:
type: integer
description: Shards in memory. A restored database reports 0 until used.
shards: { type: integer }
