Expand description
§🌳 Oak Core
Oak Core is the foundational library for the Oak language framework, providing the essential building blocks for high-performance, fault-tolerant language processing.
§🚀 Key Features
- Green/Red Tree Architecture: Implements an immutable, persistent syntax tree structure inspired by Roslyn and rust-analyzer, enabling efficient tree manipulation and sharing.
- Incremental Lexing & Parsing: Supports incremental updates to handle real-time editing in IDEs with minimal re-computation.
- Arena-based Memory Management: Utilizes custom arenas and bump allocators for high-performance memory allocation during tree construction.
- SIMD-accelerated Lexing: Leverages hardware acceleration for extremely fast tokenization of source text.
- Fault Tolerance: Designed to handle incomplete and syntactically incorrect source code, essential for IDE features like IntelliSense.
§🚦 Quick Start
§Basic Lexing and Parsing
use oak_core::{Language, Parser, Lexer, Source};
// 1. Define your language-specific types (TokenType, ElementType)
// 2. Implement the Language trait
// 3. Create a Lexer and Parser for your language
// 4. Parse source text:
let source = "let x = 42;";
let mut cache = ParseCache::new();
let output = parse::<MyLanguage>(&source, &[], &mut cache);
// 5. Access the resulting syntax tree
let green_tree = output.green();
let red_tree = RedTree::new(green_tree);§🏗️ Core Components
§1. Syntax Trees (Green & Red)
- Green Trees: Immutable, lossless, and position-independent trees. They are cheap to clone and easy to cache.
- Red Trees: State-ful, position-aware views over Green Trees. They provide parent pointers and absolute offsets.
§2. Incremental Builder
The Builder and BuilderCache allow for constructing syntax trees incrementally, reusing existing subtrees when the source code changes.
§3. Memory Management
Uses SyntaxArena for efficient allocation of tree nodes, reducing fragmentation and improving cache locality.
§4. Source & Range
Provides robust source text management with Source and precise location tracking using Range and SourceId.
§🔍 Module Overview
- builder: Incremental tree construction.
- errors: Diagnostic and error reporting.
- language: Language-specific definitions.
- lexer: High-performance tokenization.
- memory: Arena and bump allocation.
- parser: Recursive descent and Pratt parsing.
- tree: Green and Red tree implementations.
- source: Source text and location management.
Re-exports§
pub use crate::builder::Builder;pub use crate::builder::BuilderCache;pub use crate::errors::OakDiagnostics;pub use crate::errors::OakError;pub use crate::errors::OakErrorKind;pub use crate::language::ElementRole;pub use crate::language::ElementType;pub use crate::language::Language;pub use crate::language::LanguageCategory;pub use crate::language::TokenRole;pub use crate::language::TokenType;pub use crate::language::UniversalElementRole;pub use crate::language::UniversalTokenRole;pub use crate::lexer::LexOutput;pub use crate::lexer::Lexer;pub use crate::lexer::LexerCache;pub use crate::lexer::LexerState;pub use crate::lexer::NoLexerCache;pub use crate::lexer::Token;pub use crate::lexer::TokenStream;pub use crate::lexer::Tokens;pub use crate::memory::arena::SyntaxArena;pub use crate::parser::Associativity;pub use crate::parser::OperatorInfo;pub use crate::parser::ParseCache;pub use crate::parser::ParseOutput;pub use crate::parser::ParseSession;pub use crate::parser::Parser;pub use crate::parser::ParserState;pub use crate::parser::Pratt;pub use crate::parser::PrattParser;pub use crate::parser::binary;pub use crate::parser::parse;pub use crate::parser::parse_one_pass;pub use crate::parser::postfix;pub use crate::parser::state::TreeSink;pub use crate::parser::unary;pub use crate::source::Source;pub use crate::source::SourceText;pub use crate::source::TextEdit;pub use crate::tree::GreenNode;pub use crate::tree::GreenTree;pub use crate::tree::RedLeaf;pub use crate::tree::RedNode;pub use crate::tree::RedTree;
Modules§
- builder
- Incremental tree builder and cache management.
- errors
- Error handling and diagnostic reporting for the parsing system.
- helpers
- Helper utilities for common operations. Helper utilities for common operations in the Oak Core parsing framework.
- language
- Language definition trait for coordinating language-specific components.
- lexer
- Lexical analysis and tokenization functionality.
- memory
- Memory management utilities (Arena, Bump).
- parser
- Parsing functionality for converting tokens to kind trees.
- serde_
arc_ str - Serde support for
triomphe::Arc<str>. - serde_
range - Serde support for
core::range::Range. - source
- Source text management and location tracking. Source text management and location tracking for incremental parsing.
- tree
- Tree structures for representing kind trees (green and red trees).
- visitor
- Tree traversal and transformation utilities. Tree traversal and transformation utilities.