Skip to main content

Module lexer

busbar_sf_agentscript

Module lexer 

Source
Expand description

Lexer for AgentScript source code.

This module provides tokenization of AgentScript source, handling:

  • Keywords (config, topic, reasoning, etc.)
  • Operators (==, !=, and, or, etc.)
  • Literals (strings, numbers, booleans)
  • References (@variables.name)
  • Indentation tracking (INDENT/DEDENT tokens)

§Indentation Handling

AgentScript uses significant whitespace like Python. The lexer tracks indentation levels and emits INDENT/DEDENT tokens when the level changes. This is handled by lex_with_indentation().

§Example

use busbar_sf_agentscript::lexer::{lexer, lex_with_indentation, Token};
use chumsky::prelude::*;  // For Parser trait

// Basic tokenization
let tokens = lexer().parse("config:").into_result().unwrap();
assert_eq!(tokens[0].0, Token::Config);

// With indentation tracking
let source = "config:\n   agent_name: \"Test\"";
let tokens = lex_with_indentation(source).unwrap();
// Contains INDENT token after the newline

§Token Types

CategoryExamples
Keywordsconfig, variables, topic, reasoning
Typesstring, number, boolean, list
Operators==, !=, and, or, not
Literals"text", 42, True, False, None
Punctuation:, ., @, |, ->
IndentationINDENT, DEDENT, Newline

Enums§

Token
A token in AgentScript.

Functions§

add_indentation_tokens
Process raw tokens to add INDENT/DEDENT tokens based on indentation.
lex_with_indentation
Full lexer that produces indentation-aware tokens.
lexer
Create the lexer parser.

Type Aliases§

Span
Span type for tokens.
Spanned
A token with its span.