state.rs defines all the mutable data that the evaluator carries through an expansion run: configuration, the scope stack of variables and macros, the source-file registry, and call-site records for the tracing infrastructure.

Design rationale

Scope stack instead of a hash-chain

EvaluatorState keeps a Vec<ScopeFrame>. Each macro call pushes a new ScopeFrame and pops it on return. Variable and macro lookups walk the stack from top (innermost scope) to bottom (global scope), returning the first match. This gives correct lexical shadowing without the overhead of cloning entire hash maps or maintaining parent pointers.

%export exploits the flat Vec: it copies a binding one index down (stack_len - 2) into the parent frame, making the binding survive the child’s pop.

TrackedValue: three span-density levels

TrackedValue wraps a String value with a Vec<SpanRange> that carries source attribution:

  • Empty spans — untracked (Rhai/Python script result). The value is pushed to the output sink as push_untracked, contributing no source information.

  • One coarse span covering [0, value.len()] — the fast path. The tracer knows the value came from a particular token but does not record per-character attribution.

  • Multiple spans — full per-token threading. Each element covers a sub-range of value and attributes it to a different source token. This is produced only on the PreciseTracingOutput path when argument evaluation returns a Vec<SpanRange>.

SourceManager: u32 indices into a byte registry

SourceManager stores raw Vec<u8> (not String) because the lexer operates on byte slices. Each source file gets a u32 index that fits in Token.src. Canonical paths are used as the deduplication key so that including the same file through two different relative paths resolves to the same index.

VarDefRaw / MacroDefRaw: call-site byte offsets

These records capture the exact byte position of every %set or %def call so that the MCP tracing tools can answer "where was this variable set?" or "where was this macro defined?" without a full re-parse.

State type overview

PlantUML diagram

File structure

// <<@file weaveback-macro/src/evaluator/state.rs>>=
// <<state preamble>>
// <<eval config>>
// <<script kind>>
// <<macro definition>>
// <<tracked value>>
// <<scope frame>>
// <<source manager>>
// <<var def raw>>
// <<macro def raw>>
// <<max recursion>>
// <<evaluator state>>
// @

Preamble

// <<state preamble>>=
// crates/weaveback-macro/src/evaluator/state.rs

use crate::evaluator::output::{SourceSpan, SpanRange};
use crate::types::ASTNode;
use std::collections::{HashMap, HashSet};
use std::path::PathBuf;
use std::sync::Arc;
// @

EvalConfig — per-run configuration

EvalConfig is created once per evaluation run (or shared across a batch of files via eval_files). It is immutable after construction; all mutable per-call state lives in EvaluatorState.

discovery_mode is a special flag used by the directory-scan driver: instead of recursively evaluating %include/%import files, the evaluator records the resolved paths in discovered_includes and returns. This allows collecting all input file dependencies without side effects.

allow_env gates access to environment variables via %env(NAME). It is disabled by default so templates cannot silently exfiltrate secrets without an explicit --allow-env flag.

// <<eval config>>=
#[derive(Debug, Clone)]
pub struct EvalConfig {
    pub special_char: char,
    pub include_paths: Vec<PathBuf>,
    /// When true, `%include`/`%import` evaluate their path argument but do not
    /// recurse into the file.  The resolved path is recorded in
    /// `EvaluatorState::discovered_includes` instead.  Used by the directory
    /// driver-discovery pass.
    pub discovery_mode: bool,
    /// When true, the `%env(NAME)` builtin is permitted to read environment
    /// variables.  Disabled by default so that templates cannot silently
    /// exfiltrate secrets without the user opting in via `--allow-env`.
    pub allow_env: bool,
}

impl Default for EvalConfig {
    fn default() -> Self {
        Self {
            special_char: '%',
            include_paths: vec![PathBuf::from(".")],
            discovery_mode: false,
            allow_env: false,
        }
    }
}
// @

ScriptKind — which engine evaluates a macro body

After the body text is expanded (variable substitution etc.), ScriptKind determines whether the result is returned as-is (None) or passed through a scripting engine (Rhai, Python).

// <<script kind>>=
#[derive(Debug, Clone, PartialEq)]
pub enum ScriptKind {
    None,
    Rhai,
    Python,
}
// @

MacroDefinition — stored macro

MacroDefinition is cloned out of the scope stack at every call site. body is an Arc<ASTNode> so cloning is cheap (no deep copy of the AST).

frozen_args is populated by Evaluator::freeze_macro_definition when a macro is exported via %export. It maps variable names that appear in the body (but are not declared parameters) to their values at export time, implementing a lexical closure without a reference to the enclosing scope.

// <<macro definition>>=
#[derive(Debug, Clone)]
pub struct MacroDefinition {
    pub name: String,
    pub params: Vec<String>,
    pub body: Arc<ASTNode>,
    pub script_kind: ScriptKind,
    pub frozen_args: HashMap<String, String>,
}
// @

TrackedValue — value with optional source attribution

// <<tracked value>>=
#[derive(Debug, Clone)]
pub struct TrackedValue {
    pub value: String,
    /// Per-token span ranges relative to `value[0]`.
    /// Empty means untracked (script/builtin result).
    /// Single entry covering `[0, value.len()]` is the coarse-span fast path.
    /// Multiple entries carry full per-token attribution threaded through argument evaluation.
    pub spans: Vec<SpanRange>,
}
// @

ScopeFrame — one lexical scope level

Each macro call creates a new ScopeFrame. A frame holds the variables bound by that call’s parameter list (and by %set calls within the macro body) and any macros defined inside the call body via %def.

// <<scope frame>>=
#[derive(Debug, Default, Clone)]
pub struct ScopeFrame {
    pub variables: HashMap<String, TrackedValue>,
    pub macros: HashMap<String, MacroDefinition>,
}
// @

SourceManager — source file registry

SourceManager stores the raw bytes of every source file that has been read during this evaluation run. Files are deduplicated by canonical path so that including the same file through two different relative paths yields the same u32 index.

// <<source manager>>=
pub struct SourceManager {
    source_files: Vec<Vec<u8>>,
    file_names: Vec<PathBuf>,
    sources_by_path: HashMap<PathBuf, usize>,
}

impl SourceManager {
    pub fn new() -> Self {
        Self {
            source_files: Vec::new(),
            file_names: Vec::new(),
            sources_by_path: HashMap::new(),
        }
    }

    pub fn add_source_if_not_present(&mut self, file_path: PathBuf) -> Result<u32, std::io::Error> {
        let file_path = file_path.canonicalize()?;
        if let Some(&src) = self.sources_by_path.get(&file_path) {
            return Ok(src as u32);
        }
        let content = std::fs::read(file_path.clone())?;
        let src = self.add_source_bytes(content, file_path.clone());
        Ok(src)
    }

    pub fn add_source_bytes(&mut self, content: Vec<u8>, path: PathBuf) -> u32 {
        let index = self.source_files.len();
        self.source_files.push(content);
        self.file_names.push(path.clone());
        self.sources_by_path.insert(path, index);
        index as u32
    }

    pub fn get_source(&self, src: u32) -> Option<&[u8]> {
        self.source_files.get(src as usize).map(|v| v.as_slice())
    }

    pub fn num_sources(&self) -> usize {
        self.source_files.len()
    }

    pub fn source_files(&self) -> &[PathBuf] {
        &self.file_names
    }
}
// @

VarDefRaw and MacroDefRaw — call-site records

These lightweight structs record the byte position of every %set and %def call encountered during evaluation. The MCP server drains them after each run to build the var_defs_map and macro_defs_map that answer "where was this symbol defined?".

// <<var def raw>>=
/// Raw record of a `%set(var_name, ...)` call site, captured during evaluation.
/// Positions are absolute byte offsets in the source file (same as Token.pos / Token.length).
#[derive(Debug, Clone)]
pub struct VarDefRaw {
    pub var_name: String,
    /// Source file index (same as Token.src).
    pub src: u32,
    /// Byte offset of the `set` keyword in the source.
    pub pos: u32,
    /// Byte length of the whole `set(...)` call.
    pub length: u32,
}
// @

// <<macro def raw>>=
/// Raw record of a `%def / %rhaidef / %pydef(name, ...)` call site.
#[derive(Debug, Clone)]
pub struct MacroDefRaw {
    pub macro_name: String,
    /// Source file index (same as Token.src).
    pub src: u32,
    /// Byte offset of the def keyword in the source.
    pub pos: u32,
    /// Byte length of the whole def(...) call.
    pub length: u32,
}
// @

Recursion depth guard

// <<max recursion>>=
pub use weaveback_core::MAX_RECURSION_DEPTH;
// @

EvaluatorState — full mutable evaluation state

EvaluatorState owns all mutable state. Evaluator (in core.rs) holds an EvaluatorState plus the stateless singletons (RhaiEvaluator, MontyEvaluator, builtins map).

The helper methods on EvaluatorState encapsulate common patterns: the three set_*_variable variants manage the span-density levels of TrackedValue, and get_variable / get_macro both walk the scope stack from top to bottom.

// <<evaluator state>>=
pub struct EvaluatorState {
    pub config: EvalConfig,
    pub scope_stack: Vec<ScopeFrame>,
    pub open_includes: HashSet<PathBuf>,
    pub current_file: PathBuf,
    pub source_manager: SourceManager,
    pub call_depth: usize,
    /// Set by `%here` to stop further evaluation cleanly (not an error).
    pub early_exit: bool,
    /// Populated during discovery mode: every path resolved by `%include`/`%import`.
    pub discovered_includes: Vec<PathBuf>,
    /// Accumulated `%set` call sites for the var_defs_map.
    pub var_defs: Vec<VarDefRaw>,
    /// Accumulated `%def/%rhaidef/%pydef` call sites for the macro_defs_map.
    pub macro_defs: Vec<MacroDefRaw>,
}

impl EvaluatorState {
    pub fn new(config: EvalConfig) -> Self {
        Self {
            config,
            scope_stack: vec![ScopeFrame::default()],
            open_includes: HashSet::new(),
            current_file: PathBuf::from(""),
            source_manager: SourceManager::new(),
            call_depth: 0,
            early_exit: false,
            discovered_includes: Vec::new(),
            var_defs: Vec::new(),
            macro_defs: Vec::new(),
        }
    }

    pub fn drain_var_defs(&mut self) -> Vec<VarDefRaw> {
        std::mem::take(&mut self.var_defs)
    }

    pub fn drain_macro_defs(&mut self) -> Vec<MacroDefRaw> {
        std::mem::take(&mut self.macro_defs)
    }

    pub fn push_scope(&mut self) {
        self.scope_stack.push(ScopeFrame::default());
    }

    pub fn pop_scope(&mut self) {
        if self.scope_stack.len() > 1 {
            self.scope_stack.pop();
        }
    }

    pub fn current_scope_mut(&mut self) -> &mut ScopeFrame {
        self.scope_stack.last_mut().unwrap()
    }

    /// Set a variable with no origin tracking (legacy/computed path).
    pub fn set_variable(&mut self, name: &str, value: &str) {
        self.current_scope_mut().variables.insert(
            name.into(),
            TrackedValue {
                value: value.into(),
                spans: vec![],
            },
        );
    }

    /// Set a variable with a single coarse origin span (fast path).
    pub fn set_tracked_variable(&mut self, name: &str, value: &str, span: Option<SourceSpan>) {
        let spans = if let Some(sp) = span {
            vec![SpanRange { start: 0, end: value.len(), span: sp }]
        } else {
            vec![]
        };
        self.current_scope_mut().variables.insert(
            name.into(),
            TrackedValue { value: value.into(), spans },
        );
    }

    /// Set a variable with full per-token span attribution (precise tracing path).
    pub fn set_traced_variable(&mut self, name: &str, value: String, spans: Vec<SpanRange>) {
        self.current_scope_mut().variables.insert(
            name.into(),
            TrackedValue { value, spans },
        );
    }

    /// Retrieve just the string value of a variable.
    pub fn get_variable(&self, name: &str) -> String {
        for frame in self.scope_stack.iter().rev() {
            if let Some(tv) = frame.variables.get(name) {
                return tv.value.clone();
            }
        }
        "".to_string()
    }

    /// Retrieve the tracked value of a variable.
    pub fn get_tracked_variable(&self, name: &str) -> Option<TrackedValue> {
        for frame in self.scope_stack.iter().rev() {
            if let Some(tv) = frame.variables.get(name) {
                return Some(tv.clone());
            }
        }
        None
    }

    pub fn define_macro(&mut self, mac: MacroDefinition) {
        self.current_scope_mut()
            .macros
            .insert(mac.name.clone(), mac);
    }

    pub fn get_macro(&self, name: &str) -> Option<MacroDefinition> {
        for frame in self.scope_stack.iter().rev() {
            if let Some(m) = frame.macros.get(name) {
                return Some(m.clone());
            }
        }
        None
    }

    pub fn get_special_char(&self) -> Vec<u8> {
        vec![self.config.special_char as u8]
    }
}
// @