<!DOCTYPE HTML>
<html lang="en" class="light sidebar-visible" dir="ltr">
<head>
<meta charset="UTF-8">
<title>Cluster Example - RpcNet Guide</title>
<meta name="description" content="">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="theme-color" content="#ffffff">
<link rel="icon" href="favicon.svg">
<link rel="shortcut icon" href="favicon.png">
<link rel="stylesheet" href="css/variables.css">
<link rel="stylesheet" href="css/general.css">
<link rel="stylesheet" href="css/chrome.css">
<link rel="stylesheet" href="css/print.css" media="print">
<link rel="stylesheet" href="FontAwesome/css/font-awesome.css">
<link rel="stylesheet" href="fonts/fonts.css">
<link rel="stylesheet" id="highlight-css" href="highlight.css">
<link rel="stylesheet" id="tomorrow-night-css" href="tomorrow-night.css">
<link rel="stylesheet" id="ayu-highlight-css" href="ayu-highlight.css">
<script>
const path_to_root = "";
const default_light_theme = "light";
const default_dark_theme = "navy";
window.path_to_searchindex_js = "searchindex.js";
</script>
<script src="toc.js"></script>
</head>
<body>
<div id="mdbook-help-container">
<div id="mdbook-help-popup">
<h2 class="mdbook-help-title">Keyboard shortcuts</h2>
<div>
<p>Press <kbd>←</kbd> or <kbd>→</kbd> to navigate between chapters</p>
<p>Press <kbd>S</kbd> or <kbd>/</kbd> to search in the book</p>
<p>Press <kbd>?</kbd> to show this help</p>
<p>Press <kbd>Esc</kbd> to hide this help</p>
</div>
</div>
</div>
<div id="body-container">
<script>
try {
let theme = localStorage.getItem('mdbook-theme');
let sidebar = localStorage.getItem('mdbook-sidebar');
if (theme.startsWith('"') && theme.endsWith('"')) {
localStorage.setItem('mdbook-theme', theme.slice(1, theme.length - 1));
}
if (sidebar.startsWith('"') && sidebar.endsWith('"')) {
localStorage.setItem('mdbook-sidebar', sidebar.slice(1, sidebar.length - 1));
}
} catch (e) { }
</script>
<script>
const default_theme = window.matchMedia("(prefers-color-scheme: dark)").matches ? default_dark_theme : default_light_theme;
let theme;
try { theme = localStorage.getItem('mdbook-theme'); } catch(e) { }
if (theme === null || theme === undefined) { theme = default_theme; }
const html = document.documentElement;
html.classList.remove('light')
html.classList.add(theme);
html.classList.add("js");
</script>
<input type="checkbox" id="sidebar-toggle-anchor" class="hidden">
<script>
let sidebar = null;
const sidebar_toggle = document.getElementById("sidebar-toggle-anchor");
if (document.body.clientWidth >= 1080) {
try { sidebar = localStorage.getItem('mdbook-sidebar'); } catch(e) { }
sidebar = sidebar || 'visible';
} else {
sidebar = 'hidden';
sidebar_toggle.checked = false;
}
if (sidebar === 'visible') {
sidebar_toggle.checked = true;
} else {
html.classList.remove('sidebar-visible');
}
</script>
<nav id="sidebar" class="sidebar" aria-label="Table of contents">
<mdbook-sidebar-scrollbox class="sidebar-scrollbox"></mdbook-sidebar-scrollbox>
<noscript>
<iframe class="sidebar-iframe-outer" src="toc.html"></iframe>
</noscript>
<div id="sidebar-resize-handle" class="sidebar-resize-handle">
<div class="sidebar-resize-indicator"></div>
</div>
</nav>
<div id="page-wrapper" class="page-wrapper">
<div class="page">
<div id="menu-bar-hover-placeholder"></div>
<div id="menu-bar" class="menu-bar sticky">
<div class="left-buttons">
<label id="sidebar-toggle" class="icon-button" for="sidebar-toggle-anchor" title="Toggle Table of Contents" aria-label="Toggle Table of Contents" aria-controls="sidebar">
<i class="fa fa-bars"></i>
</label>
<button id="theme-toggle" class="icon-button" type="button" title="Change theme" aria-label="Change theme" aria-haspopup="true" aria-expanded="false" aria-controls="theme-list">
<i class="fa fa-paint-brush"></i>
</button>
<ul id="theme-list" class="theme-popup" aria-label="Themes" role="menu">
<li role="none"><button role="menuitem" class="theme" id="default_theme">Auto</button></li>
<li role="none"><button role="menuitem" class="theme" id="light">Light</button></li>
<li role="none"><button role="menuitem" class="theme" id="rust">Rust</button></li>
<li role="none"><button role="menuitem" class="theme" id="coal">Coal</button></li>
<li role="none"><button role="menuitem" class="theme" id="navy">Navy</button></li>
<li role="none"><button role="menuitem" class="theme" id="ayu">Ayu</button></li>
</ul>
<button id="search-toggle" class="icon-button" type="button" title="Search (`/`)" aria-label="Toggle Searchbar" aria-expanded="false" aria-keyshortcuts="/ s" aria-controls="searchbar">
<i class="fa fa-search"></i>
</button>
</div>
<h1 class="menu-title">RpcNet Guide</h1>
<div class="right-buttons">
<a href="print.html" title="Print this book" aria-label="Print this book">
<i id="print-button" class="fa fa-print"></i>
</a>
</div>
</div>
<div id="search-wrapper" class="hidden">
<form id="searchbar-outer" class="searchbar-outer">
<div class="search-wrapper">
<input type="search" id="searchbar" name="searchbar" placeholder="Search this book ..." aria-controls="searchresults-outer" aria-describedby="searchresults-header">
<div class="spinner-wrapper">
<i class="fa fa-spinner fa-spin"></i>
</div>
</div>
</form>
<div id="searchresults-outer" class="searchresults-outer hidden">
<div id="searchresults-header" class="searchresults-header"></div>
<ul id="searchresults">
</ul>
</div>
</div>
<script>
document.getElementById('sidebar-toggle').setAttribute('aria-expanded', sidebar === 'visible');
document.getElementById('sidebar').setAttribute('aria-hidden', sidebar !== 'visible');
Array.from(document.querySelectorAll('#sidebar a')).forEach(function(link) {
link.setAttribute('tabIndex', sidebar === 'visible' ? 0 : -1);
});
</script>
<div id="content" class="content">
<main>
<h1 id="cluster-example"><a class="header" href="#cluster-example">Cluster Example</a></h1>
<p>This chapter demonstrates building a distributed RPC cluster with automatic worker discovery, load balancing, and failure detection using RpcNet's built-in cluster features.</p>
<h2 id="architecture-overview"><a class="header" href="#architecture-overview">Architecture Overview</a></h2>
<p>The cluster example showcases three main components working together:</p>
<pre><code> ┌──────────────────────────┐
│ Director │
│ (Coordinator Node) │
│ │
│ - WorkerRegistry │
│ - ClusterClient │
│ - Load Balancing │
└────────┬─────────────────┘
│
Gossip Protocol (SWIM)
│
┌────────────────┼────────────────┐
│ │
┌───────▼────────┐ ┌────────▼───────┐
│ Worker A │ │ Worker B │
│ │ │ │
│ - Auto-join │ │ - Auto-join │
│ - Tag: worker │ │ - Tag: worker │
│ - Process tasks│ │ - Process tasks│
└─────────────────┘ └─────────────────┘
</code></pre>
<h3 id="components"><a class="header" href="#components">Components</a></h3>
<p><strong>1. Director</strong> - Coordinator node that:</p>
<ul>
<li>Uses <code>WorkerRegistry</code> for automatic worker discovery</li>
<li>Uses <code>ClusterClient</code> for load-balanced request routing</li>
<li>Employs <code>LeastConnections</code> strategy by default</li>
<li>Monitors worker pool status</li>
<li>Routes client requests to healthy workers</li>
</ul>
<p><strong>2. Workers</strong> - Processing nodes that:</p>
<ul>
<li>Join cluster automatically via gossip protocol</li>
<li>Tag themselves with <code>role=worker</code> for discovery</li>
<li>Process compute tasks from clients</li>
<li>Monitor cluster events (node joined/left/failed)</li>
<li>Support simulated failures for testing</li>
</ul>
<p><strong>3. Client</strong> - Application that:</p>
<ul>
<li>Connects to director</li>
<li>Gets worker assignment</li>
<li>Establishes direct connection to worker</li>
<li>Handles failover automatically</li>
</ul>
<h2 id="why-use-built-in-cluster-features"><a class="header" href="#why-use-built-in-cluster-features">Why Use Built-in Cluster Features?</a></h2>
<p>Compared to manual worker management patterns:</p>
<p><strong>Manual Approach</strong> ❌:</p>
<ul>
<li>Custom <code>HashMap<Uuid, WorkerInfo></code> for tracking</li>
<li>Manual round-robin selection logic</li>
<li>Explicit RPC calls for worker registration</li>
<li>Custom ping-based health checks</li>
<li>~200 lines of boilerplate code</li>
</ul>
<p><strong>Built-in Cluster</strong> ✅:</p>
<ul>
<li>Built-in <code>WorkerRegistry</code> + <code>ClusterClient</code></li>
<li>Multiple load balancing strategies (Round Robin, Random, Least Connections)</li>
<li>Automatic discovery via SWIM gossip protocol</li>
<li>Phi Accrual failure detection (accurate, adaptive)</li>
<li>~50 lines to set up</li>
<li><strong>75% code reduction!</strong></li>
</ul>
<h2 id="running-the-example"><a class="header" href="#running-the-example">Running the Example</a></h2>
<h3 id="prerequisites"><a class="header" href="#prerequisites">Prerequisites</a></h3>
<p>Ensure test certificates exist:</p>
<pre><code class="language-bash">ls certs/test_cert.pem certs/test_key.pem
</code></pre>
<p>All commands should be run from the <strong>project root directory</strong>.</p>
<h3 id="basic-setup"><a class="header" href="#basic-setup">Basic Setup</a></h3>
<p>Open four terminals and run each component:</p>
<p><strong>Terminal 1 - Director:</strong></p>
<pre><code class="language-bash">DIRECTOR_ADDR=127.0.0.1:61000 \
RUST_LOG=info \
cargo run --manifest-path examples/cluster/Cargo.toml --bin director
</code></pre>
<p><strong>Terminal 2 - Worker A:</strong></p>
<pre><code class="language-bash">WORKER_LABEL=worker-a \
WORKER_ADDR=127.0.0.1:62001 \
DIRECTOR_ADDR=127.0.0.1:61000 \
RUST_LOG=info \
cargo run --manifest-path examples/cluster/Cargo.toml --bin worker
</code></pre>
<p><strong>Terminal 3 - Worker B:</strong></p>
<pre><code class="language-bash">WORKER_LABEL=worker-b \
WORKER_ADDR=127.0.0.1:62002 \
DIRECTOR_ADDR=127.0.0.1:61000 \
RUST_LOG=info \
cargo run --manifest-path examples/cluster/Cargo.toml --bin worker
</code></pre>
<p><strong>Terminal 4 - Client:</strong></p>
<pre><code class="language-bash">DIRECTOR_ADDR=127.0.0.1:61000 \
RUST_LOG=info \
cargo run --manifest-path examples/cluster/Cargo.toml --bin client
</code></pre>
<h3 id="what-youll-see"><a class="header" href="#what-youll-see">What You'll See</a></h3>
<p><strong>Director Output:</strong></p>
<pre><code>🎯 Starting Director at 127.0.0.1:61000
📁 Loading certificates from "../../certs/test_cert.pem"
✅ Director registered itself in cluster
✅ Cluster enabled - Director is now discoverable
🔄 Load balancing strategy: LeastConnections
📊 Worker pool status: 2 workers available
- worker-a at 127.0.0.1:62001 (0 connections)
- worker-b at 127.0.0.1:62002 (0 connections)
🚀 Director ready - listening on 127.0.0.1:61000
</code></pre>
<p><strong>Worker Output:</strong></p>
<pre><code>👷 Starting Worker 'worker-a' at 127.0.0.1:62001
🔌 Binding server to 127.0.0.1:62001...
✅ Server bound successfully
🌐 Enabling cluster, connecting to director at 127.0.0.1:61000...
✅ Cluster enabled, connected to director
🏷️ Tagging worker with role=worker and label=worker-a...
✅ Worker 'worker-a' joined cluster with role=worker
🚀 Worker 'worker-a' is running and ready to handle requests
</code></pre>
<p><strong>Client Output:</strong></p>
<pre><code>📡 Starting Client - connecting to director at 127.0.0.1:61000
✅ connected to director
🔀 director assigned worker - establishing direct connection
✅ direct connection established to worker
📤 creating request stream
🌊 stream opened successfully, starting to consume responses
📦 received token (sequence=1, text="token-1", total=1)
📦 received token (sequence=2, text="token-2", total=2)
...
</code></pre>
<h2 id="testing-failure-scenarios"><a class="header" href="#testing-failure-scenarios">Testing Failure Scenarios</a></h2>
<h3 id="simulated-worker-failures"><a class="header" href="#simulated-worker-failures">Simulated Worker Failures</a></h3>
<p>Enable periodic failures to test automatic failover:</p>
<p><strong>Worker with Failures:</strong></p>
<pre><code class="language-bash">WORKER_LABEL=worker-a \
WORKER_ADDR=127.0.0.1:62001 \
DIRECTOR_ADDR=127.0.0.1:61000 \
WORKER_FAILURE_ENABLED=true \ # Enable failure simulation
RUST_LOG=info \
cargo run --manifest-path examples/cluster/Cargo.toml --bin worker
</code></pre>
<p><strong>Failure Cycle</strong> (~18 seconds):</p>
<ol>
<li><strong>Run</strong>: 10 seconds of normal operation</li>
<li><strong>Warning</strong>: "⚠️ Simulating worker failure in 3 seconds..."</li>
<li><strong>Failed</strong>: 5 seconds in failed state - "💥 Worker failed!"</li>
<li><strong>Recovery</strong>: "🔄 Worker recovering..."</li>
<li><strong>Ready</strong>: "✅ Worker recovered and ready to serve!"</li>
<li>Repeat</li>
</ol>
<p><strong>Client Behavior:</strong></p>
<ul>
<li>Detects failure via error response</li>
<li>Returns to director for new worker assignment</li>
<li>Switches to healthy worker seamlessly</li>
<li>Streaming continues with minimal interruption</li>
</ul>
<h3 id="hard-kill-test"><a class="header" href="#hard-kill-test">Hard Kill Test</a></h3>
<p>Test network-level failure detection:</p>
<pre><code class="language-bash"># In a worker terminal, press Ctrl+C
</code></pre>
<p><strong>Observe:</strong></p>
<ul>
<li>Director detects failure via gossip protocol</li>
<li><code>WorkerRegistry</code> removes worker from pool</li>
<li>Client requests automatically route to remaining workers</li>
<li>Zero downtime for ongoing operations</li>
</ul>
<h3 id="worker-restart-test"><a class="header" href="#worker-restart-test">Worker Restart Test</a></h3>
<p>After killing a worker, restart it to see re-discovery:</p>
<pre><code class="language-bash">WORKER_LABEL=worker-a \
WORKER_ADDR=127.0.0.1:62001 \
DIRECTOR_ADDR=127.0.0.1:61000 \
RUST_LOG=info \
cargo run --manifest-path examples/cluster/Cargo.toml --bin worker
</code></pre>
<p><strong>Observe:</strong></p>
<ul>
<li>Worker automatically rejoins cluster</li>
<li>Gossip spreads worker availability</li>
<li>Director adds worker back to registry</li>
<li>Client requests resume to all available workers</li>
</ul>
<h2 id="how-it-works"><a class="header" href="#how-it-works">How It Works</a></h2>
<h3 id="1-automatic-discovery"><a class="header" href="#1-automatic-discovery">1. Automatic Discovery</a></h3>
<p>Workers don't manually register - they just join the cluster:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>// Worker code (simplified)
let cluster = ClusterMembership::new(config).await?;
cluster.join(vec![director_addr]).await?;
// Tag for discovery
cluster.set_tag("role", "worker");
cluster.set_tag("label", worker_label);
// That's it! Director discovers automatically via gossip
<span class="boring">}</span></code></pre></pre>
<h3 id="2-load-balancing"><a class="header" href="#2-load-balancing">2. Load Balancing</a></h3>
<p>Director uses <code>WorkerRegistry</code> for automatic load balancing:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>// Director code
let registry = Arc::new(WorkerRegistry::new(
cluster,
LoadBalancingStrategy::LeastConnections
));
registry.start().await;
// Automatically tracks workers and balances load
<span class="boring">}</span></code></pre></pre>
<h3 id="3-failure-detection"><a class="header" href="#3-failure-detection">3. Failure Detection</a></h3>
<p>Phi Accrual algorithm provides accurate health monitoring:</p>
<ul>
<li>Adapts to network conditions</li>
<li>Distinguishes slow nodes from failed nodes</li>
<li>No false positives from temporary delays</li>
<li>Automatic recovery when nodes return</li>
</ul>
<h3 id="4-tag-based-routing"><a class="header" href="#4-tag-based-routing">4. Tag-Based Routing</a></h3>
<p>Filter workers by capabilities:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>// Get only GPU workers
let gpu_worker = registry.select_worker(Some("gpu=true")).await?;
// Get any worker
let any_worker = registry.select_worker(Some("role=worker")).await?;
<span class="boring">}</span></code></pre></pre>
<h2 id="key-cluster-features-demonstrated"><a class="header" href="#key-cluster-features-demonstrated">Key Cluster Features Demonstrated</a></h2>
<h3 id="-automatic-discovery"><a class="header" href="#-automatic-discovery">✅ Automatic Discovery</a></h3>
<p>No manual registration needed - gossip protocol handles everything</p>
<h3 id="-load-balancing"><a class="header" href="#-load-balancing">✅ Load Balancing</a></h3>
<p>Choose from:</p>
<ul>
<li><strong>Round Robin</strong>: Even distribution</li>
<li><strong>Random</strong>: Stateless workload distribution</li>
<li><strong>Least Connections</strong>: Balance based on current load (recommended)</li>
</ul>
<h3 id="-failure-detection"><a class="header" href="#-failure-detection">✅ Failure Detection</a></h3>
<p>Phi Accrual algorithm provides accurate, adaptive health monitoring</p>
<h3 id="-tag-based-routing"><a class="header" href="#-tag-based-routing">✅ Tag-Based Routing</a></h3>
<p>Route by worker capabilities (GPU, CPU, zone, etc.)</p>
<h3 id="-event-monitoring"><a class="header" href="#-event-monitoring">✅ Event Monitoring</a></h3>
<p>Subscribe to cluster events:</p>
<ul>
<li><code>NodeJoined</code> - New worker available</li>
<li><code>NodeLeft</code> - Worker gracefully departed</li>
<li><code>NodeFailed</code> - Worker detected as failed</li>
</ul>
<h2 id="configuration-options"><a class="header" href="#configuration-options">Configuration Options</a></h2>
<h3 id="environment-variables"><a class="header" href="#environment-variables">Environment Variables</a></h3>
<p><strong>Director:</strong></p>
<ul>
<li><code>DIRECTOR_ADDR</code> - Bind address (default: <code>127.0.0.1:61000</code>)</li>
<li><code>RUST_LOG</code> - Log level (e.g., <code>info</code>, <code>debug</code>)</li>
</ul>
<p><strong>Worker:</strong></p>
<ul>
<li><code>WORKER_LABEL</code> - Worker identifier (default: <code>worker-1</code>)</li>
<li><code>WORKER_ADDR</code> - Bind address (default: <code>127.0.0.1:62001</code>)</li>
<li><code>DIRECTOR_ADDR</code> - Director address (default: <code>127.0.0.1:61000</code>)</li>
<li><code>WORKER_FAILURE_ENABLED</code> - Enable failure simulation (default: <code>false</code>)</li>
<li><code>RUST_LOG</code> - Log level</li>
</ul>
<p><strong>Client:</strong></p>
<ul>
<li><code>DIRECTOR_ADDR</code> - Director address (default: <code>127.0.0.1:61000</code>)</li>
<li><code>RUST_LOG</code> - Log level</li>
</ul>
<h3 id="load-balancing-strategies"><a class="header" href="#load-balancing-strategies">Load Balancing Strategies</a></h3>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>use rpcnet::cluster::LoadBalancingStrategy;
// Options:
LoadBalancingStrategy::RoundRobin // Even distribution
LoadBalancingStrategy::Random // Random selection
LoadBalancingStrategy::LeastConnections // Pick least loaded (recommended)
<span class="boring">}</span></code></pre></pre>
<h3 id="cluster-configuration"><a class="header" href="#cluster-configuration">Cluster Configuration</a></h3>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>use rpcnet::cluster::ClusterConfig;
let config = ClusterConfig::default()
.with_gossip_interval(Duration::from_secs(1))
.with_health_check_interval(Duration::from_secs(2));
<span class="boring">}</span></code></pre></pre>
<h2 id="troubleshooting"><a class="header" href="#troubleshooting">Troubleshooting</a></h2>
<p><strong>Workers not discovered:</strong></p>
<ul>
<li>Ensure director starts first (it's the seed node)</li>
<li>Check firewall allows UDP for gossip</li>
<li>Verify workers connect to correct director address</li>
</ul>
<p><strong>Requests failing:</strong></p>
<ul>
<li>Check worker has <code>role=worker</code> tag</li>
<li>Verify compute handler is registered</li>
<li>Check logs for connection errors</li>
</ul>
<p><strong>Slow failover:</strong></p>
<ul>
<li>Adjust health check interval in config</li>
<li>Tune Phi Accrual threshold</li>
<li>Check network latency</li>
</ul>
<h2 id="production-considerations"><a class="header" href="#production-considerations">Production Considerations</a></h2>
<p>For production deployments:</p>
<ol>
<li><strong>TLS Certificates</strong>: Use proper certificates, not test certs</li>
<li><strong>Monitoring</strong>: Integrate cluster events with your monitoring system</li>
<li><strong>Scaling</strong>: Add more workers dynamically as needed</li>
<li><strong>Persistence</strong>: Consider persisting cluster state if needed</li>
<li><strong>Security</strong>: Add authentication and authorization</li>
<li><strong>Network</strong>: Plan for network partitions and split-brain scenarios</li>
</ol>
<h2 id="next-steps"><a class="header" href="#next-steps">Next Steps</a></h2>
<ul>
<li>Try different load balancing strategies</li>
<li>Add more workers dynamically</li>
<li>Test network partition scenarios</li>
<li>Add custom tags for routing (zone, GPU, etc.)</li>
<li>Integrate with your application logic</li>
</ul>
<p>For full source code, see <code>examples/cluster/</code> in the repository.</p>
</main>
<nav class="nav-wrapper" aria-label="Page navigation">
<a rel="prev" href="rpcnet-gen.html" class="mobile-nav-chapters previous" title="Previous chapter" aria-label="Previous chapter" aria-keyshortcuts="Left">
<i class="fa fa-angle-left"></i>
</a>
<a rel="next prefetch" href="cluster/overview.html" class="mobile-nav-chapters next" title="Next chapter" aria-label="Next chapter" aria-keyshortcuts="Right">
<i class="fa fa-angle-right"></i>
</a>
<div style="clear: both"></div>
</nav>
</div>
</div>
<nav class="nav-wide-wrapper" aria-label="Page navigation">
<a rel="prev" href="rpcnet-gen.html" class="nav-chapters previous" title="Previous chapter" aria-label="Previous chapter" aria-keyshortcuts="Left">
<i class="fa fa-angle-left"></i>
</a>
<a rel="next prefetch" href="cluster/overview.html" class="nav-chapters next" title="Next chapter" aria-label="Next chapter" aria-keyshortcuts="Right">
<i class="fa fa-angle-right"></i>
</a>
</nav>
</div>
<script>
const wsProtocol = location.protocol === 'https:' ? 'wss:' : 'ws:';
const wsAddress = wsProtocol + "//" + location.host + "/" + "__livereload";
const socket = new WebSocket(wsAddress);
socket.onmessage = function (event) {
if (event.data === "reload") {
socket.close();
location.reload();
}
};
window.onbeforeunload = function() {
socket.close();
}
</script>
<script>
window.playground_copyable = true;
</script>
<script src="elasticlunr.min.js"></script>
<script src="mark.min.js"></script>
<script src="searcher.js"></script>
<script src="clipboard.min.js"></script>
<script src="highlight.js"></script>
<script src="book.js"></script>
</div>
</body>
</html>
