<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" lang="" xml:lang="">
<head>
<meta charset="utf-8" />
<meta name="generator" content="pandoc" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
<title>README</title>
<style>
html {
line-height: 1.5;
font-family: Georgia, serif;
font-size: 20px;
color: #1a1a1a;
background-color: #fdfdfd;
}
body {
margin: 0 auto;
max-width: 36em;
padding-left: 50px;
padding-right: 50px;
padding-top: 50px;
padding-bottom: 50px;
hyphens: auto;
overflow-wrap: break-word;
text-rendering: optimizeLegibility;
font-kerning: normal;
}
@media (max-width: 600px) {
body {
font-size: 0.9em;
padding: 1em;
}
h1 {
font-size: 1.8em;
}
}
@media print {
body {
background-color: transparent;
color: black;
font-size: 12pt;
}
p, h2, h3 {
orphans: 3;
widows: 3;
}
h2, h3, h4 {
page-break-after: avoid;
}
}
p {
margin: 1em 0;
}
a {
color: #1a1a1a;
}
a:visited {
color: #1a1a1a;
}
img {
max-width: 100%;
}
h1, h2, h3, h4, h5, h6 {
margin-top: 1.4em;
}
h5, h6 {
font-size: 1em;
font-style: italic;
}
h6 {
font-weight: normal;
}
ol, ul {
padding-left: 1.7em;
margin-top: 1em;
}
li > ol, li > ul {
margin-top: 0;
}
blockquote {
margin: 1em 0 1em 1.7em;
padding-left: 1em;
border-left: 2px solid #e6e6e6;
color: #606060;
}
code {
font-family: Menlo, Monaco, 'Lucida Console', Consolas, monospace;
font-size: 85%;
margin: 0;
}
pre {
margin: 1em 0;
overflow: auto;
}
pre code {
padding: 0;
overflow: visible;
overflow-wrap: normal;
}
.sourceCode {
background-color: transparent;
overflow: visible;
}
hr {
background-color: #1a1a1a;
border: none;
height: 1px;
margin: 1em 0;
}
table {
margin: 1em 0;
border-collapse: collapse;
width: 100%;
overflow-x: auto;
display: block;
font-variant-numeric: lining-nums tabular-nums;
}
table caption {
margin-bottom: 0.75em;
}
tbody {
margin-top: 0.5em;
border-top: 1px solid #1a1a1a;
border-bottom: 1px solid #1a1a1a;
}
th {
border-top: 1px solid #1a1a1a;
padding: 0.25em 0.5em 0.25em 0.5em;
}
td {
padding: 0.125em 0.5em 0.25em 0.5em;
}
header {
margin-bottom: 4em;
text-align: center;
}
#TOC li {
list-style: none;
}
#TOC ul {
padding-left: 1.3em;
}
#TOC > ul {
padding-left: 0;
}
#TOC a:not(:hover) {
text-decoration: none;
}
code{white-space: pre-wrap;}
span.smallcaps{font-variant: small-caps;}
div.columns{display: flex; gap: min(4vw, 1.5em);}
div.column{flex: auto; overflow-x: auto;}
div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}
ul.task-list{list-style: none;}
ul.task-list li input[type="checkbox"] {
width: 0.8em;
margin: 0 0.8em 0.2em -1.6em;
vertical-align: middle;
}
pre > code.sourceCode { white-space: pre; position: relative; }
pre > code.sourceCode > span { display: inline-block; line-height: 1.25; }
pre > code.sourceCode > span:empty { height: 1.2em; }
.sourceCode { overflow: visible; }
code.sourceCode > span { color: inherit; text-decoration: inherit; }
div.sourceCode { margin: 1em 0; }
pre.sourceCode { margin: 0; }
@media screen {
div.sourceCode { overflow: auto; }
}
@media print {
pre > code.sourceCode { white-space: pre-wrap; }
pre > code.sourceCode > span { text-indent: -5em; padding-left: 5em; }
}
pre.numberSource code
{ counter-reset: source-line 0; }
pre.numberSource code > span
{ position: relative; left: -4em; counter-increment: source-line; }
pre.numberSource code > span > a:first-child::before
{ content: counter(source-line);
position: relative; left: -1em; text-align: right; vertical-align: baseline;
border: none; display: inline-block;
-webkit-touch-callout: none; -webkit-user-select: none;
-khtml-user-select: none; -moz-user-select: none;
-ms-user-select: none; user-select: none;
padding: 0 4px; width: 4em;
color: #aaaaaa;
}
pre.numberSource { margin-left: 3em; border-left: 1px solid #aaaaaa; padding-left: 4px; }
div.sourceCode
{ }
@media screen {
pre > code.sourceCode > span > a:first-child::before { text-decoration: underline; }
}
code span.al { color: #ff0000; font-weight: bold; } /* Alert */
code span.an { color: #60a0b0; font-weight: bold; font-style: italic; } /* Annotation */
code span.at { color: #7d9029; } /* Attribute */
code span.bn { color: #40a070; } /* BaseN */
code span.bu { color: #008000; } /* BuiltIn */
code span.cf { color: #007020; font-weight: bold; } /* ControlFlow */
code span.ch { color: #4070a0; } /* Char */
code span.cn { color: #880000; } /* Constant */
code span.co { color: #60a0b0; font-style: italic; } /* Comment */
code span.cv { color: #60a0b0; font-weight: bold; font-style: italic; } /* CommentVar */
code span.do { color: #ba2121; font-style: italic; } /* Documentation */
code span.dt { color: #902000; } /* DataType */
code span.dv { color: #40a070; } /* DecVal */
code span.er { color: #ff0000; font-weight: bold; } /* Error */
code span.ex { } /* Extension */
code span.fl { color: #40a070; } /* Float */
code span.fu { color: #06287e; } /* Function */
code span.im { color: #008000; font-weight: bold; } /* Import */
code span.in { color: #60a0b0; font-weight: bold; font-style: italic; } /* Information */
code span.kw { color: #007020; font-weight: bold; } /* Keyword */
code span.op { color: #666666; } /* Operator */
code span.ot { color: #007020; } /* Other */
code span.pp { color: #bc7a00; } /* Preprocessor */
code span.sc { color: #4070a0; } /* SpecialChar */
code span.ss { color: #bb6688; } /* SpecialString */
code span.st { color: #4070a0; } /* String */
code span.va { color: #19177c; } /* Variable */
code span.vs { color: #4070a0; } /* VerbatimString */
code span.wa { color: #60a0b0; font-weight: bold; font-style: italic; } /* Warning */
.display.math{display: block; text-align: center; margin: 0.5rem auto;}
</style>
<!--[if lt IE 9]>
<script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.3/html5shiv-printshiv.min.js"></script>
<![endif]-->
</head>
<body>
<h1
id="hyprstream-real-time-aggregation-windows-and-high-performance-cache-for-apache-arrow-flight-sql">Hyprstream:
Real-time Aggregation Windows and High-Performance Cache for Apache
Arrow Flight SQL 🚀</h1>
<p>Hyprstream is a next-generation application for real-time data
ingestion, windowed aggregation, caching, and serving. Built on Apache
Arrow Flight and DuckDB, and developed in Rust, Hyprstream dynamically
calculates metrics like running sums, counts, and averages, enabling
blazing-fast data workflows, intelligent caching, and seamless
integration with ADBC-compliant datastores. Its real-time aggregation
capabilities empower AI/ML pipelines and analytics with instant
insights. 💾✨</p>
<h2 id="key-features">Key Features 🎯</h2>
<h3 id="data-ingestion-via-apache-arrow-flight">🔄 Data Ingestion via
Apache Arrow Flight</h3>
<ul>
<li><strong>Streamlined Ingestion</strong>: Ingests data efficiently
using <strong>Arrow Flight</strong>, an advanced columnar data transport
protocol</li>
<li><strong>Real-Time Streaming</strong>: Supports real-time metrics,
datasets, and vectorized data for analytics and AI/ML workflows</li>
<li><strong>Write-Through to ADBC</strong>: Ensures data consistency
with immediate caching and write-through to backend datastores</li>
</ul>
<h3 id="intelligent-read-caching-with-duckdb">🧠 Intelligent Read
Caching with DuckDB</h3>
<ul>
<li><strong>In-Memory Performance</strong>: Uses <strong>DuckDB</strong>
for lightning-fast caching of frequently accessed data</li>
<li><strong>Optimized Querying</strong>: Stores query results and
intermediate computations for analytics workloads</li>
<li><strong>Automatic Management</strong>: Handles caching transparently
with configurable expiry policies</li>
</ul>
<h3 id="real-time-aggregation">⚡ Real-Time Aggregation</h3>
<ul>
<li><strong>Dynamic Metrics</strong>: Maintains running sums, counts,
and averages for real-time insights</li>
<li><strong>Time Window Partitioning</strong>: Supports fixed time
windows (e.g., 5m, 30m, hourly, daily) for granular analysis</li>
<li><strong>Lightweight State</strong>: Maintains only aggregate states
for efficient memory usage</li>
</ul>
<h3 id="data-serving-with-arrow-flight-sql">🌐 Data Serving with Arrow
Flight SQL</h3>
<ul>
<li><strong>High-Performance Queries</strong>: Serves cached data via
Arrow Flight SQL for minimal latency</li>
<li><strong>Vectorized Data</strong>: Optimized for AI/ML pipelines and
analytical queries</li>
<li><strong>Seamless Integration</strong>: Connects with analytics and
visualization tools</li>
</ul>
<h2 id="benefits">Benefits 🌟</h2>
<ul>
<li><strong>🚀 Low Latency</strong>: Millisecond-level query responses
for cached data</li>
<li><strong>⚙️ Scalable</strong>: Handles large-scale data workflows
with ease</li>
<li><strong>🔗 Flexible</strong>: Integrates with Postgres, Redis,
Snowflake, and other ADBC datastores</li>
<li><strong>🤖 AI/ML Ready</strong>: Optimized for vectorized data and
inference pipelines</li>
<li><strong>📈 Real-Time Metrics</strong>: Dynamic calculation of
statistical metrics</li>
<li><strong>⌛ Time Windows</strong>: Granular control of metrics with
configurable windows</li>
<li><strong>⛭ Rust-Powered</strong>: High-performance, memory-safe
implementation</li>
</ul>
<h2 id="getting-started">Getting Started 🚀</h2>
<ol type="1">
<li><p>Install Hyprstream:</p>
<div class="sourceCode" id="cb1"><pre
class="sourceCode bash"><code class="sourceCode bash"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a><span class="ex">cargo</span> install hyprstream</span></code></pre></div></li>
<li><p>Start the server with default configuration:</p>
<div class="sourceCode" id="cb2"><pre
class="sourceCode bash"><code class="sourceCode bash"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true" tabindex="-1"></a><span class="ex">hyprstream</span></span></code></pre></div></li>
<li><p>Use with PostgreSQL backend (requires PostgreSQL ADBC
driver):</p>
<div class="sourceCode" id="cb3"><pre
class="sourceCode bash"><code class="sourceCode bash"><span id="cb3-1"><a href="#cb3-1" aria-hidden="true" tabindex="-1"></a><span class="co"># Set backend-specific credentials securely via environment variables</span></span>
<span id="cb3-2"><a href="#cb3-2" aria-hidden="true" tabindex="-1"></a><span class="bu">export</span> <span class="va">HYPRSTREAM_ENGINE_USERNAME</span><span class="op">=</span>postgres</span>
<span id="cb3-3"><a href="#cb3-3" aria-hidden="true" tabindex="-1"></a><span class="bu">export</span> <span class="va">HYPRSTREAM_ENGINE_PASSWORD</span><span class="op">=</span>secret</span>
<span id="cb3-4"><a href="#cb3-4" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb3-5"><a href="#cb3-5" aria-hidden="true" tabindex="-1"></a><span class="co"># Start Hyprstream with connection details (but without credentials)</span></span>
<span id="cb3-6"><a href="#cb3-6" aria-hidden="true" tabindex="-1"></a><span class="ex">hyprstream</span> <span class="dt">\</span></span>
<span id="cb3-7"><a href="#cb3-7" aria-hidden="true" tabindex="-1"></a> <span class="at">--engine</span> adbc <span class="dt">\</span></span>
<span id="cb3-8"><a href="#cb3-8" aria-hidden="true" tabindex="-1"></a> <span class="at">--engine-connection</span> <span class="st">"postgresql://localhost:5432/metrics?pool_max=10&pool_min=1&connect_timeout=30"</span> <span class="dt">\</span></span>
<span id="cb3-9"><a href="#cb3-9" aria-hidden="true" tabindex="-1"></a> <span class="at">--engine-options</span> driver_path=/usr/local/lib/libadbc_driver_postgresql.so <span class="dt">\</span></span>
<span id="cb3-10"><a href="#cb3-10" aria-hidden="true" tabindex="-1"></a> <span class="at">--enable-cache</span> <span class="dt">\</span></span>
<span id="cb3-11"><a href="#cb3-11" aria-hidden="true" tabindex="-1"></a> <span class="at">--cache-engine</span> duckdb <span class="dt">\</span></span>
<span id="cb3-12"><a href="#cb3-12" aria-hidden="true" tabindex="-1"></a> <span class="at">--cache-connection</span> <span class="st">":memory:"</span></span></code></pre></div></li>
</ol>
<p>For configuration options and detailed documentation, run:</p>
<div class="sourceCode" id="cb4"><pre
class="sourceCode bash"><code class="sourceCode bash"><span id="cb4-1"><a href="#cb4-1" aria-hidden="true" tabindex="-1"></a><span class="ex">hyprstream</span> <span class="at">--help</span></span></code></pre></div>
<p>Or visit our <a href="https://docs.rs/hyprstream">API
Documentation</a> for comprehensive guides and examples.</p>
<h2 id="example-usage">Example Usage 💡</h2>
<h3 id="quick-start-with-adbc">Quick Start with ADBC</h3>
<p>Hyprstream implements the Arrow Flight SQL protocol, making it
compatible with any ADBC-compliant client:</p>
<div class="sourceCode" id="cb5"><pre
class="sourceCode python"><code class="sourceCode python"><span id="cb5-1"><a href="#cb5-1" aria-hidden="true" tabindex="-1"></a><span class="im">import</span> adbc_driver_flightsql.dbapi</span>
<span id="cb5-2"><a href="#cb5-2" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb5-3"><a href="#cb5-3" aria-hidden="true" tabindex="-1"></a><span class="co"># Connect to Hyprstream using standard ADBC</span></span>
<span id="cb5-4"><a href="#cb5-4" aria-hidden="true" tabindex="-1"></a>conn <span class="op">=</span> adbc_driver_flightsql.dbapi.<span class="ex">connect</span>(<span class="st">"grpc://localhost:50051"</span>)</span>
<span id="cb5-5"><a href="#cb5-5" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb5-6"><a href="#cb5-6" aria-hidden="true" tabindex="-1"></a><span class="cf">try</span>:</span>
<span id="cb5-7"><a href="#cb5-7" aria-hidden="true" tabindex="-1"></a> cursor <span class="op">=</span> conn.cursor()</span>
<span id="cb5-8"><a href="#cb5-8" aria-hidden="true" tabindex="-1"></a> </span>
<span id="cb5-9"><a href="#cb5-9" aria-hidden="true" tabindex="-1"></a> <span class="co"># Query metrics with time windows</span></span>
<span id="cb5-10"><a href="#cb5-10" aria-hidden="true" tabindex="-1"></a> cursor.execute(<span class="st">"""</span></span>
<span id="cb5-11"><a href="#cb5-11" aria-hidden="true" tabindex="-1"></a><span class="st"> SELECT </span></span>
<span id="cb5-12"><a href="#cb5-12" aria-hidden="true" tabindex="-1"></a><span class="st"> metric_id,</span></span>
<span id="cb5-13"><a href="#cb5-13" aria-hidden="true" tabindex="-1"></a><span class="st"> COUNT(*) as samples,</span></span>
<span id="cb5-14"><a href="#cb5-14" aria-hidden="true" tabindex="-1"></a><span class="st"> AVG(value_running_window_avg) as avg_value</span></span>
<span id="cb5-15"><a href="#cb5-15" aria-hidden="true" tabindex="-1"></a><span class="st"> FROM metrics</span></span>
<span id="cb5-16"><a href="#cb5-16" aria-hidden="true" tabindex="-1"></a><span class="st"> WHERE timestamp >= NOW() - INTERVAL '1 hour'</span></span>
<span id="cb5-17"><a href="#cb5-17" aria-hidden="true" tabindex="-1"></a><span class="st"> GROUP BY metric_id</span></span>
<span id="cb5-18"><a href="#cb5-18" aria-hidden="true" tabindex="-1"></a><span class="st"> ORDER BY avg_value DESC</span></span>
<span id="cb5-19"><a href="#cb5-19" aria-hidden="true" tabindex="-1"></a><span class="st"> """</span>)</span>
<span id="cb5-20"><a href="#cb5-20" aria-hidden="true" tabindex="-1"></a> </span>
<span id="cb5-21"><a href="#cb5-21" aria-hidden="true" tabindex="-1"></a> results <span class="op">=</span> cursor.fetch_arrow_table()</span>
<span id="cb5-22"><a href="#cb5-22" aria-hidden="true" tabindex="-1"></a> <span class="bu">print</span>(results.to_pandas())</span>
<span id="cb5-23"><a href="#cb5-23" aria-hidden="true" tabindex="-1"></a> </span>
<span id="cb5-24"><a href="#cb5-24" aria-hidden="true" tabindex="-1"></a><span class="cf">finally</span>:</span>
<span id="cb5-25"><a href="#cb5-25" aria-hidden="true" tabindex="-1"></a> cursor.close()</span>
<span id="cb5-26"><a href="#cb5-26" aria-hidden="true" tabindex="-1"></a> conn.close()</span></code></pre></div>
<h3 id="configuration">Configuration</h3>
<p>Hyprstream supports multiple storage backends and can be configured
through environment variables, command-line arguments, or configuration
files.</p>
<h3 id="adbc-backend">ADBC Backend</h3>
<p>The ADBC backend supports PostgreSQL and other ADBC-compatible
databases. Connection credentials can be provided in several ways,
listed in order of precedence:</p>
<ol type="1">
<li><p>Environment variables:</p>
<div class="sourceCode" id="cb6"><pre
class="sourceCode bash"><code class="sourceCode bash"><span id="cb6-1"><a href="#cb6-1" aria-hidden="true" tabindex="-1"></a><span class="bu">export</span> <span class="va">HYPRSTREAM_ENGINE_USERNAME</span><span class="op">=</span>postgres</span>
<span id="cb6-2"><a href="#cb6-2" aria-hidden="true" tabindex="-1"></a><span class="bu">export</span> <span class="va">HYPRSTREAM_ENGINE_PASSWORD</span><span class="op">=</span>secret</span></code></pre></div></li>
<li><p>Command-line arguments:</p>
<div class="sourceCode" id="cb7"><pre
class="sourceCode bash"><code class="sourceCode bash"><span id="cb7-1"><a href="#cb7-1" aria-hidden="true" tabindex="-1"></a><span class="ex">hyprstream</span> <span class="at">--engine-username</span> postgres <span class="at">--engine-password</span> secret</span></code></pre></div></li>
<li><p>Connection URI:</p>
<pre><code>postgresql://postgres:secret@localhost:5432/metrics</code></pre></li>
</ol>
<p>Example configuration:</p>
<div class="sourceCode" id="cb9"><pre
class="sourceCode toml"><code class="sourceCode toml"><span id="cb9-1"><a href="#cb9-1" aria-hidden="true" tabindex="-1"></a><span class="kw">[</span><span class="dt">storage</span><span class="kw">]</span></span>
<span id="cb9-2"><a href="#cb9-2" aria-hidden="true" tabindex="-1"></a><span class="dt">backend</span> <span class="op">=</span> <span class="st">"adbc"</span></span>
<span id="cb9-3"><a href="#cb9-3" aria-hidden="true" tabindex="-1"></a><span class="dt">connection</span> <span class="op">=</span> <span class="st">"postgresql://localhost:5432/metrics"</span></span>
<span id="cb9-4"><a href="#cb9-4" aria-hidden="true" tabindex="-1"></a><span class="dt">options</span> <span class="op">=</span> <span class="op">{ </span><span class="dt">pool_max</span><span class="op"> =</span> <span class="st">"10"</span><span class="op">, </span><span class="dt">pool_min</span><span class="op"> =</span> <span class="st">"1"</span><span class="op">, </span><span class="dt">connect_timeout</span><span class="op"> =</span> <span class="st">"30"</span><span class="op"> }</span></span></code></pre></div>
<p>For security best practices: - Use environment variables or
command-line arguments for credentials - Never commit credentials to
version control - Use connection pooling to manage database connections
efficiently - Set appropriate connection timeouts</p>
<h2 id="better-together-ecosystem-integration">Better Together:
Ecosystem Integration 🔄</h2>
<p>Hyprstream enhances modern data architectures by filling critical
gaps in the real-time data stack. While tools like Flink excel at
complex stream processing, Hyprstream adds the missing piece: instant,
SQL-based access to streaming data and real-time metrics. With support
for any ADBC-compliant database backend, Hyprstream enables
high-performance architectures that combine cloud-scale storage with
edge performance.</p>
<p>Additionally, through its DuckDB backend integration, Hyprstream can
serve as a high-performance aggregation and caching layer for cloud
analytics services like MotherDuck.</p>
<h3 id="comparison-with-stream-processing-analytics-tools">Comparison
with Stream Processing & Analytics Tools</h3>
<table>
<colgroup>
<col style="width: 19%" />
<col style="width: 25%" />
<col style="width: 29%" />
<col style="width: 25%" />
</colgroup>
<thead>
<tr class="header">
<th>Feature</th>
<th>Hyprstream</th>
<th>Apache Flink</th>
<th>MotherDuck</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><strong>Ingest-to-Query Latency</strong></td>
<td>1-10ms*</td>
<td>Seconds-minutes**</td>
<td>100ms-seconds</td>
</tr>
<tr class="even">
<td><strong>Query Interface</strong></td>
<td>Direct SQL</td>
<td>External sink required</td>
<td>Direct SQL</td>
</tr>
<tr class="odd">
<td><strong>Storage Model</strong></td>
<td>In-memory + ADBC</td>
<td>External systems</td>
<td>Cloud-native</td>
</tr>
<tr class="even">
<td><strong>Deployment</strong></td>
<td>Single binary</td>
<td>Cluster + job manager</td>
<td>Cloud service</td>
</tr>
<tr class="odd">
<td><strong>Scale Focus</strong></td>
<td>Hot data, edge</td>
<td>Stream processing</td>
<td>Cloud analytics</td>
</tr>
<tr class="even">
<td><strong>State Management</strong></td>
<td>Time windows, metrics</td>
<td>Full event state</td>
<td>Full dataset</td>
</tr>
<tr class="odd">
<td><strong>Data Access</strong></td>
<td>Arrow Flight SQL</td>
<td>Custom operators</td>
<td>DuckDB/SQL</td>
</tr>
<tr class="even">
<td><strong>Cost Model</strong></td>
<td>Compute-focused</td>
<td>Compute-focused</td>
<td>Storage-focused</td>
</tr>
</tbody>
</table>
<p>* <em>For cached data; backend queries add typical ADBC database
latency</em> ** <em>End-to-end latency including writing to external
storage and querying</em></p>
<h2 id="contributing">Contributing 🤝</h2>
<p>We welcome contributions! Please feel free to submit a Pull
Request.</p>
<h2 id="license">License 📄</h2>
<p>This project is licensed under the Apache 2.0 License - see the <a
href="LICENSE">LICENSE</a> file for details.</p>
<hr />
<p>For inquiries or support, contact us at <a
href="mailto:support@hyprstream.com">support@hyprstream.com</a> or visit
our GitHub repository to contribute! 🌐</p>
</body>
</html>