Skip to main content

Module loader

lex_core::lex

Module loader 

Source
Expand description

Document loading and transform execution

This module provides DocumentLoader - the universal API for loading and processing lex source text. It abstracts over input sources (file vs. string) and provides convenient shortcuts for common transform operations.

§Purpose

DocumentLoader serves as the primary entry point for most lex processing tasks:

  • Universal Input: Load from files or strings with the same API
  • Transform Shortcuts: Common operations (.parse(), .tokenize()) built-in
  • Custom Transforms: Execute any transform via .with()
  • Source Access: Retrieve original source text via .source()
  • Reusable: Create once, run multiple transforms on the same source

§Relationship to Transform System

DocumentLoader is a convenience layer on top of the transform system. It manages source text loading and delegates to the appropriate transform:

§Common Usage Patterns

§Parse a Document

use lex_parser::lex::loader::DocumentLoader;

let loader = DocumentLoader::from_string("Session:\n    Content\n");
let doc = loader.parse()?;

§Load from File

use lex_parser::lex::loader::DocumentLoader;

let loader = DocumentLoader::from_path("document.lex")?;
let doc = loader.parse()?;

§Get Tokens

use lex_parser::lex::loader::DocumentLoader;

let loader = DocumentLoader::from_string("Hello world\n");
let tokens = loader.tokenize()?;  // With semantic indentation
let base = loader.base_tokens()?; // Core tokens only

§Custom Transform

use lex_parser::lex::loader::DocumentLoader;
use lex_parser::lex::transforms::standard::TO_IR;

let loader = DocumentLoader::from_string("Hello\n");
let ir = loader.with(&*TO_IR)?;  // Get intermediate representation

§Multiple Operations on Same Source

use lex_parser::lex::loader::DocumentLoader;

let loader = DocumentLoader::from_string("Hello\n");
let source = loader.source();      // Get original text
let tokens = loader.tokenize()?;   // Get tokens
let doc = loader.parse()?;         // Get AST

§Use Cases

  • CLI Tools: Load files and apply stage+format transforms
  • Tests: Load test fixtures and verify different processing stages
  • Library Code: Process lex documents programmatically
  • REPL/Interactive: Parse user input on-the-fly

Structs§

DocumentLoader
Document loader with transform shortcuts

Enums§

LoaderError
Error that can occur when loading documents