Kotoba
Graph Processing System with Jsonnet Integration - A comprehensive graph processing platform featuring complete Jsonnet implementation, ISO GQL-compliant queries, and distributed execution.
📖 Overview
Kotoba is a powerful graph processing system built on graph theory foundations. It combines a complete Jsonnet implementation with GP2-based graph rewriting, providing ISO GQL-compliant queries, MVCC+Merkle persistence, and distributed execution capabilities.
🎯 Key Features
- Complete Jsonnet Implementation: Full support for arrays, objects, functions, string interpolation, and local variables
- DPO (Double Pushout) Graph Rewriting: Theoretical foundation for graph transformations
- ISO GQL-compliant Queries: Standardized graph query language
- MVCC + Merkle DAG Persistence: Consistent distributed data management
- Multi-format Support: JSON, YAML output capabilities
- Rust Native Architecture: Memory-safe, high-performance implementation
- Modular Crate Design: kotoba-jsonnet, kotoba-graph, kotoba-core, and more
🚀 Quick Start
Prerequisites
- Rust 1.70.0 or later
- Cargo package manager
🐳 Nix Development Environment (Recommended)
For a reproducible and stable development environment, use Nix with flakes:
# Install Nix (if not already installed)
|
# Enable flakes (add to ~/.config/nix/nix.conf)
# Clone and enter the project
# Run setup script
# Enter development environment
# Or use direnv for automatic activation
The Nix environment provides:
- ✅ Exact Rust version (1.82.0)
- ✅ All required dependencies
- ✅ Development tools (docker, kind, kubectl, helm)
- ✅ Reproducible builds
- ✅ Cross-platform support
Installation
# Clone the repository
# Install dependencies and build
# Run comprehensive test suite (38/38 tests passing)
# Build release version
Basic Usage Examples
Jsonnet Evaluation
Kotoba includes a complete Jsonnet implementation supporting arrays, objects, functions, and string interpolation:
example.jsonnet
// Local variables and functions
local version = "1.0.0";
local add = function(x, y) x + y;
// Object with computed values
{
app: {
name: "Kotoba Demo",
version: version,
features: ["jsonnet", "graph", "gql"],
},
// Array operations
numbers: [1, 2, 3, 4, 5],
doubled: [x * 2 for x in self.numbers],
// String interpolation
greeting: "Hello, %(name)s!" % { name: "World" },
// Function calls
sum: add(10, 20),
// Conditional logic
status: if self.sum > 25 then "high" else "low",
}
Run with Kotoba:
# Evaluate Jsonnet file
# Convert to JSON
Graph Processing
Users create .kotoba
files in Jsonnet format for graph processing:
graph.kotoba
{
// Graph data
graph: {
vertices: [
{ id: "alice", labels: ["Person"], properties: { name: "Alice", age: 30 } },
{ id: "bob", labels: ["Person"], properties: { name: "Bob", age: 25 } },
],
edges: [
{ id: "follows_1", src: "alice", dst: "bob", label: "FOLLOWS" },
],
},
// GQL queries
queries: [
{
name: "find_people",
gql: "MATCH (p:Person) RETURN p.name, p.age",
},
],
// 実行ロジック
handlers: [
{
name: "main",
function: "execute_queries",
metadata: { description: "Execute all defined queries" },
},
],
}
実行方法
# .kotobaファイルを実行
# またはサーバーモードで起動
🏗️ Architecture
Multi-Crate Architecture
Kotoba adopts a modular multi-crate architecture for maximum flexibility:
├── kotoba-core/ # Core types and IR definitions
├── kotoba-jsonnet/ # Complete Jsonnet implementation (38/38 tests passing)
├── kotoba-graph/ # Graph data structures and operations
├── kotoba-storage/ # High-performance RocksDB storage
├── kotoba-execution/ # Query execution and planner
├── kotoba-rewrite/ # Graph rewriting engine
├── kotoba-server/ # HTTP server and handlers
├── kotoba-kotobanet/ # Kotoba extensions for Jsonnet
├── kotoba2tsx/ # TypeScript/React code generation
└── kotoba/ # Main integration crate
Each crate can be used independently, allowing you to pick only the features you need.
Crate Highlights
kotoba-jsonnet
- Complete Jsonnet Implementation
- ✅ 38/38 tests passing - Full test coverage
- ✅ Arrays, Objects, Functions - Complete Jsonnet language support
- ✅ String Interpolation -
"%(name)s" % { name: "World" }
- ✅ Local Variables -
local x = 42; x + 1
- ✅ JSON/YAML Output - Multiple serialization formats
kotoba-graph
- Graph Processing Core
- ✅ Vertex/Edge Management - Full graph operations
- ✅ GP2 Rewriting - Theoretical graph transformations
- ✅ ISO GQL Queries - Standardized graph query language
Integration Features
- ✅ Workspace Testing -
cargo test --workspace
passes - ✅ Clean Codebase - Clippy warnings minimized
- ✅ Documentation - Comprehensive API docs
使用例
# .kotobaファイルで全て定義
# 開発時はウォッチモード
Rust API(内部使用)
// Rust APIは主に内部実装で使用
use *;
use *;
WASM対応
各crateは条件付きコンパイルによりWASMターゲットにも対応しています:
# WASMビルド
Process Network Graph Model
Kotoba is based on a Process Network Graph Model, where all components are centrally managed through dag.jsonnet
.
Main Components
┌─────────────────────────────────────────────────────────────┐
│ lib.rs │
│ (Main Library) │
├─────────────────────────────────────────────────────────────┤
│ execution/ │ rewrite/ │
│ (Query Executor) │ (DPO Rewriter) │
├─────────────────────────────────────────────────────────────┤
│ planner/ │ storage/ │
│ (Query Planner) │ (MVCC+Merkle) │
├─────────────────────────────────────────────────────────────┤
│ graph/ │ ir/ │
│ (Data Structures) │ (Core IR) │
├─────────────────────────────────────────────────────────────┤
│ types.rs │
│ (Common Types) │
└─────────────────────────────────────────────────────────────┘
Build Order (Topological Sort)
// Get build order from dag.jsonnet
$ jsonnet eval dag.jsonnet | jq .topological_order
[
"types",
"ir_catalog",
"ir_rule",
"ir_query",
"ir_patch",
"graph_vertex",
"graph_edge",
"ir_strategy",
"graph_core",
"storage_mvcc",
"storage_merkle",
"storage_lsm", // RocksDB-based high-performance storage
"planner_logical",
"planner_physical",
"execution_parser",
"rewrite_matcher",
"rewrite_applier",
"planner_optimizer",
"rewrite_engine",
"execution_engine",
"lib"
]
📋 Usage
1. Basic GQL Queries
queries.kotoba
{
config: {
type: "config",
name: "QueryExample",
},
// グラフデータ
graph: {
vertices: [
{ id: "alice", labels: ["Person"], properties: { name: "Alice", age: 30 } },
{ id: "bob", labels: ["Person"], properties: { name: "Bob", age: 25 } },
{ id: "charlie", labels: ["Person"], properties: { name: "Charlie", age: 35 } },
],
edges: [
{ id: "f1", src: "alice", dst: "bob", label: "FOLLOWS" },
{ id: "f2", src: "bob", dst: "charlie", label: "FOLLOWS" },
],
},
// GQLクエリ定義
queries: [
{
name: "follow_network",
gql: "MATCH (p:Person)-[:FOLLOWS]->(f:Person) WHERE p.age > 25 RETURN p.name, f.name",
description: "25歳以上の人がフォローしている人を取得",
},
],
handlers: [
{
name: "execute_query",
function: "run_gql_query",
parameters: { query_name: "follow_network" },
},
],
}
2. Graph Rewriting
rewrite.kotoba
{
config: {
type: "config",
name: "RewriteExample",
},
// グラフ書換えルール
rules: [
{
name: "triangle_collapse",
description: "三角形を折りたたむ",
lhs: {
nodes: [
{ id: "u", type: "Person" },
{ id: "v", type: "Person" },
{ id: "w", type: "Person" },
],
edges: [
{ id: "e1", src: "u", dst: "v", type: "FOLLOWS" },
{ id: "e2", src: "v", dst: "w", type: "FOLLOWS" },
],
},
rhs: {
nodes: [
{ id: "u", type: "Person" },
{ id: "w", type: "Person" },
],
edges: [
{ id: "e3", src: "u", dst: "w", type: "FOLLOWS" },
],
},
},
],
// 実行戦略
strategies: [
{
name: "exhaust_triangle_collapse",
rule: "triangle_collapse",
strategy: "exhaust",
order: "topdown",
},
],
handlers: [
{
name: "apply_rewrite",
function: "execute_rewrite",
parameters: { strategy_name: "exhaust_triangle_collapse" },
},
],
}
3. HTTP Server with Graph Operations
server.kotoba
{
config: {
type: "config",
name: "GraphServer",
server: { host: "127.0.0.1", port: 3000 },
},
// ルート定義
routes: [
{
method: "GET",
pattern: "/api/users",
handler: "list_users",
description: "ユーザー一覧を取得",
},
{
method: "POST",
pattern: "/api/users",
handler: "create_user",
description: "ユーザーを作成",
},
],
// グラフスキーマ
schema: {
node_types: ["User", "Post"],
edge_types: ["FOLLOWS", "LIKES"],
},
handlers: [
{
name: "list_users",
function: "execute_gql",
parameters: {
query: "MATCH (u:User) RETURN u.name, u.email",
format: "json",
},
},
{
name: "create_user",
function: "create_graph_node",
parameters: {
type: "User",
properties: ["name", "email", "age"],
},
},
],
}
📄 .kotoba File Format
Kotobaプロジェクトでは、設定ファイルやUI定義などに.kotoba
ファイル形式を使用します。これはJsonnet形式をベースとした構造化された設定フォーマットです。
概要
.kotoba
ファイルは以下の特徴を持ちます:
- Jsonnet形式: JSONのスーパーセットで、変数、関数、条件分岐などの機能を活用
- 構造化設定: オブジェクトと配列による階層的な設定構造
- ユーティリティ関数: 設定生成のための再利用可能な関数定義
- 計算プロパティ: 動的な設定生成とバリデーション
- 型安全: Jsonnetの型システムによる設定の整合性確保
ファイル形式仕様
基本構造
// 設定ファイルの基本構造
{
// 設定セクション
config: {
type: "config",
name: "MyApp",
version: "1.0.0",
metadata: {
description: "アプリケーション設定",
},
},
// コンポーネント定義
components: [
// コンポーネントオブジェクト
],
// ユーティリティ関数
makeComponent: function(name, type, props={}) {
// コンポーネント生成関数
},
}
主要なプロパティ
type
(必須): オブジェクトの種類を指定name
(推奨): オブジェクトの一意な識別子metadata
(オプション): 追加情報(説明、バージョンなど)local
変数: Jsonnetのローカル変数による設定の共通化関数
: 設定生成のための再利用可能な関数::
演算子: 計算プロパティによる動的設定生成
主要なセクション
1. config
- 設定オブジェクト
アプリケーション全体の設定を定義します。
local appVersion = "1.0.0";
config: {
type: "config",
name: "MyApp",
version: appVersion,
host: "127.0.0.1",
port: 8080,
theme: "light",
metadata: {
description: "アプリケーション設定",
environment: "development",
},
}
2. routes
/ middlewares
- HTTP設定
HTTPサーバーのルートとミドルウェアを構造化して定義します。
// ユーティリティ関数
local makeRoute = function(method, pattern, handler, desc) {
type: "route",
method: method,
pattern: pattern,
handler: handler,
metadata: { description: desc },
};
routes: [
makeRoute("GET", "/api/" + appVersion + "/users", "list_users", "List users"),
makeRoute("POST", "/api/" + appVersion + "/users", "create_user", "Create user"),
],
middlewares: [
{
type: "middleware",
name: "cors",
order: 10,
function: "cors_middleware",
metadata: {
description: "CORS handling middleware",
allowed_origins: ["*"],
},
},
],
3. components
- UIコンポーネント定義
Reactコンポーネントを構造化して定義します。
local styles = {
button: { primary: "button primary", secondary: "button secondary" },
layout: { header: "header", sidebar: "sidebar" },
};
local makeButton = function(name, text, style, onClick) {
type: "component",
name: name,
component_type: "button",
props: {
text: text,
className: style,
onClick: onClick,
},
metadata: { description: name + " button" },
};
components: [
makeButton("SaveButton", "Save", styles.button.primary, "handleSave"),
makeButton("CancelButton", "Cancel", styles.button.secondary, "handleCancel"),
],
4. handlers
/ states
- イベントと状態管理
イベントハンドラーと状態を定義します。
handlers: [
{
type: "handler",
name: "handleSave",
function: "handleSave",
metadata: { description: "Save form data" },
},
],
states: [
{
type: "state",
name: "user",
initial: null,
metadata: { description: "Current user state" },
},
{
type: "state",
name: "loading",
initial: false,
metadata: { description: "Loading state" },
},
],
5. 計算プロパティとバリデーション
Jsonnetの機能を活用した動的設定とバリデーション。
// 計算プロパティ
allRoutes:: [r.pattern for r in self.routes],
routeCount:: std.length(self.routes),
// バリデーション関数
validateRoutes:: function() {
local duplicates = [
pattern
for pattern in std.set([r.pattern for r in self.routes])
if std.count([r.pattern for r in self.routes], pattern) > 1
];
if std.length(duplicates) > 0 then
error "Duplicate route patterns: " + std.join(", ", duplicates)
else
"Routes validation passed";
},
使用例
HTTPサーバー設定例
// config.kotoba - HTTPサーバー設定
local apiVersion = "v1";
local defaultTimeout = 30000;
{
// サーバー設定
config: {
type: "config",
host: "127.0.0.1",
port: 8080,
max_connections: 1000,
timeout_ms: defaultTimeout,
metadata: {
description: "HTTP server configuration",
environment: "development",
},
},
// ユーティリティ関数
makeRoute: function(method, pattern, handler, desc) {
type: "route",
method: method,
pattern: pattern,
handler: handler,
metadata: { description: desc },
},
makeMiddleware: function(name, order, func, desc) {
type: "middleware",
name: name,
order: order,
function: func,
metadata: { description: desc },
},
// ルート定義
routes: [
$.makeRoute("GET", "/ping", "ping_handler", "Simple ping endpoint"),
$.makeRoute("GET", "/health", "health_check", "Health check endpoint"),
$.makeRoute("GET", "/api/" + apiVersion + "/users", "list_users", "List users"),
$.makeRoute("POST", "/api/" + apiVersion + "/users", "create_user", "Create user"),
],
// ミドルウェア定義
middlewares: [
$.makeMiddleware("cors", 10, "cors_middleware", "CORS handling"),
$.makeMiddleware("auth", 20, "auth_middleware", "Authentication"),
$.makeMiddleware("logger", 100, "request_logger", "Request logging"),
],
// 計算プロパティ
serverInfo:: {
host: $.config.host,
port: $.config.port,
routes_count: std.length($.routes),
middlewares_count: std.length($.middlewares),
},
}
React UI設定例
// app.kotoba - React UI設定
local appName = "MyApp";
local appVersion = "1.0.0";
{
// アプリケーション設定
config: {
type: "config",
name: appName,
version: appVersion,
theme: "light",
title: "My App",
metadata: {
framework: "React",
description: "Sample React application",
},
},
// スタイル定数
styles: {
button: {
primary: "button primary",
secondary: "button secondary",
},
layout: {
header: "header",
main: "main-content",
},
},
// ユーティリティ関数
makeComponent: function(name, componentType, props={}, children=[], desc="") {
type: "component",
name: name,
component_type: componentType,
props: props,
children: children,
metadata: { description: desc },
},
makeButton: function(name, text, style, onClick, desc) {
$.makeComponent(name, "button", {
text: text,
className: style,
onClick: onClick,
}, [], desc),
},
// コンポーネント定義
components: [
$.makeComponent("App", "div", {}, ["Header", "Main"], "Root application component"),
$.makeComponent("Header", "header", {
title: $.config.title,
className: $.styles.layout.header,
}, ["Nav"], "Application header"),
$.makeButton("SaveBtn", "Save", $.styles.button.primary, "handleSave", "Save button"),
$.makeButton("CancelBtn", "Cancel", $.styles.button.secondary, "handleCancel", "Cancel button"),
],
// イベントハンドラー
handlers: [
{
type: "handler",
name: "handleSave",
function: "handleSave",
metadata: { description: "Handle save action" },
},
{
type: "handler",
name: "handleCancel",
function: "handleCancel",
metadata: { description: "Handle cancel action" },
},
],
// 状態管理
states: [
{
type: "state",
name: "user",
initial: null,
metadata: { description: "Current user state" },
},
{
type: "state",
name: "theme",
initial: $.config.theme,
metadata: { description: "Current theme state" },
},
],
}
パースと使用方法
Jsonnetファイルはjsonnet
コマンドまたはプログラムによる評価が必要です:
# Jsonnetファイルを評価してJSONに変換
# またはプログラムで直接使用
|
// Rustでの使用例
use Command;
// Jsonnetファイルを評価
let output = new
.arg
.arg
.output?;
let config_json: Value = from_slice?;
// 設定を使用
if let Some = config_json.get
Jsonnet固有の機能活用
1. 変数と定数の使用
local appVersion = "v1";
local defaultPort = 8080;
{
config: {
version: appVersion,
port: defaultPort,
},
routes: [
{ pattern: "/api/" + appVersion + "/users" },
],
}
2. 関数による設定生成
local makeApiRoute = function(method, resource, action) {
type: "route",
method: method,
pattern: "/api/v1/" + resource + "/" + action,
handler: resource + "_" + action,
};
routes: [
makeApiRoute("GET", "users", "list"),
makeApiRoute("POST", "users", "create"),
],
3. 計算プロパティによる動的設定
{
components: [/* ... */],
// コンポーネント数の計算
componentCount:: std.length(self.components),
// コンポーネントタイプ別の集計
componentTypes:: std.set([c.component_type for c in self.components]),
}
4. 条件分岐とバリデーション
local environment = "production";
{
config: {
debug: if environment == "development" then true else false,
port: if environment == "production" then 80 else 3000,
},
// バリデーション
validate:: function() {
if std.length(self.config.name) == 0 then
error "Application name is required"
else
"Configuration is valid";
},
}
ベストプラクティス
- 変数の活用: 共通の値を
local
変数で定義してDRY原則を守る - 関数による抽象化: 設定生成パターンを関数化して再利用性を高める
- 計算プロパティの使用:
::
演算子で動的な設定値を生成 - 構造化: 設定を論理的なセクション(config, routes, components等)に分ける
- バリデーション: 設定の妥当性を検証する関数を定義
- コメント: Jsonnetの
//
コメントを活用して設定の意図を明確に - 再利用: 共通の関数やスタイルを別ファイルに分離してimport
拡張性
.kotoba
形式(Jsonnet)は非常に拡張性が高く、Jsonnetの全機能を活用できます:
カスタム関数ライブラリ
// utils.libsonnet
{
// 汎用ユーティリティ関数
makeCrudRoutes(resource):: [
{
type: "route",
method: "GET",
pattern: "/api/v1/" + resource,
handler: resource + "_list",
},
{
type: "route",
method: "POST",
pattern: "/api/v1/" + resource,
handler: resource + "_create",
},
],
// スタイル定数
themes: {
light: { bg: "#ffffff", fg: "#000000" },
dark: { bg: "#000000", fg: "#ffffff" },
},
}
設定の合成
// 複数の設定ファイルを合成
local base = import "base.libsonnet";
local api = import "api.libsonnet";
base + api + {
// 追加設定
customRoutes: [
{ pattern: "/health", handler: "health_check" },
],
}
環境別設定
// 環境に応じた設定切り替え
local environment = std.extVar("ENVIRONMENT");
{
config: {
debug: environment != "production",
port: if environment == "production" then 80 else 3000,
database: {
host: if environment == "production"
then "prod-db.example.com"
else "localhost",
},
},
}
開発ワークフロー
# 設定ファイルの検証
# 特定のセクションのみ取得
# バリデーション実行
# 設定をJSONとして保存
🛠️ Development
Using dag.jsonnet
1. Dependency Analysis
# Check dependencies of specific component
# Check components that depend on this component
2. Build Order Verification
# Get overall build order
|
# Check build order for specific node
3. Causality Identification During Troubleshooting
# Get investigation order when problems occur
|
Using lib.jsonnet
1. Build Configuration Verification
# Get configuration for specific target
# Resolve component dependencies
2. Packaging Configuration
# Get Docker image configuration
|
# Get Debian package configuration
|
Development Workflow
# 1. Make code changes
# 2. Check dependencies
# 3. Run tests
# 4. Check overall consistency
# 5. Validate DAG
# 6. Commit
🧪 Testing
Test Coverage: 95%
Kotoba maintains high test coverage across all components, with particular emphasis on the storage layer achieving 95% coverage.
# Run all tests
# Run storage tests (95% coverage)
# Run specific test
# Run documentation tests
# Generate coverage report (requires cargo-tarpaulin)
Coverage Highlights
- Storage Layer: 95% coverage with comprehensive LSM tree testing
- Core Types: Full coverage of Value, GraphRef, and IR types
- Graph Operations: Extensive testing of rewriting and query operations
- HTTP Server: Integration tests for API endpoints
Integration Tests
# Run integration tests
# Run benchmarks
LDBC-SNB Benchmark
# Run benchmark with LDBC-SNB dataset
📦 Packaging
Docker Image
# Build Docker image
# Run the image
Debian Package
# Create Debian package
# Install the package
Homebrew
# Install Homebrew Formula
🔧 CLI Tools
Kotoba CLI
Kotoba CLIはDeno CLIを参考にした使いやすいコマンドラインインターフェースを提供します。グラフ処理、クエリ実行、ファイル操作などをサポートしています。
インストール
# ビルドしてインストール
基本的な使用方法
# ヘルプ表示
# プロジェクト情報表示
# GQLクエリ実行
# ファイル実行
# ファイル検証
# ファイルフォーマット
# サーバー起動
# 新規プロジェクト初期化
# ドキュメント生成
# バージョン表示
主なコマンド
コマンド | 説明 |
---|---|
run <file.kotoba> |
.kotobaファイルを実行 |
server --config <file.kotoba> |
HTTPサーバーを起動 |
query "MATCH..." --graph <file> |
GQLクエリを直接実行 |
check <file.kotoba> |
.kotobaファイルを検証 |
fmt <file.kotoba> |
.kotobaファイルをフォーマット |
info |
プロジェクト情報を表示 |
repl |
インタラクティブGQL REPL |
init <project> |
新規.kotobaプロジェクトを初期化 |
version |
バージョン情報を表示 |
グローバルオプション
オプション | 説明 |
---|---|
-c, --config <CONFIG> |
設定ファイルパス |
-l, --log-level <LEVEL> |
ログレベル (info, debug, warn, error) |
-C, --cwd <DIR> |
作業ディレクトリ |
-h, --help |
ヘルプ表示 |
-V, --version |
バージョン表示 |
使用例
# .kotobaファイルを実行
# ウォッチモードで開発(ファイル変更時に自動再実行)
# サーバーモードで起動
# ファイルを検証
# ファイルをフォーマット
# インタラクティブREPLでクエリを実行
# 新規プロジェクトを作成
# プロジェクト情報を表示
📚 API Documentation
# Generate documentation
# Generate documentation including private items
🤝 Contributing
Contribution Guidelines
- Create Issue: Bug reports or feature requests
- Create Branch:
feature/your-feature-name
- Implement Changes:
- Add tests
- Update documentation
- Verify dag.jsonnet consistency
- Create Pull Request
Development Environment Setup
# Install development dependencies
# Set up pre-commit hooks
Coding Standards
- Rust: Use
rustfmt
andclippy
- Commit Messages: Conventional Commits
- Testing: Add tests for all changes
- Documentation: Add documentation for all public APIs
📄 License
This project is licensed under the MIT License. See the LICENSE file for details.
🙏 Acknowledgments
- GP2 Team: Theoretical foundation for graph rewriting systems
- ISO/IEC: GQL standard specification
- Rust Community: Excellent programming language
📞 Support
- Documentation: https://kotoba.jun784.dev
- Issues: GitHub Issues
- Discussions: GitHub Discussions
Kotoba - Exploring the world of graphs through words