weaveback-macro vs GNU m4

m4 was designed by Brian Kernighan and Dennis Ritchie in the early 1970s as a general-purpose text-substitution layer for Unix. Its canonical use today is as the expansion engine inside GNU Autoconf: it processes .m4 files at configure-generation time and never runs on the end user’s machine. The sendmail mail transfer agent also ships a large library of m4 macros for generating its notoriously complex configuration files. These two use cases — build-system glue and system-administration templating — define m4’s sweet spot. m4 does not know or care what the text around a macro means; it is a pure text-replacement machine.

weaveback-macro was designed for a narrower and more specific job: pre-processing the prose layer of a literate programming document before the tangle pass extracts code chunks. The text around a macro does matter — it is AsciiDoc or Markdown that a human will read and that Asciidoctor will render. This shapes every design decision: the special character is configurable so it does not collide with document markup, macros are defined inside ordinary prose, and the tool is meant to be transparent to a reader who does not know it exists.

In short: m4 is a universal macro language that happens to be used for literate-adjacent tasks. weaveback-macro is a literate-programming tool that happens to be implemented as a macro language.

m4 provides explicit scope manipulation via pushdef/popdef, which requires manual discipline: the programmer must balance every pushdef with a popdef.

weaveback-macro uses a scope stack that is managed automatically. %def inside a macro body creates a definition local to that invocation; it is discarded when the call returns. To promote a definition to the caller’s scope, %export is used explicitly. %include evaluates the included file in the caller’s current scope — equivalent to inlining the text at the call site. When called at the top level (the normal case) definitions land at the top level; when called inside a macro body they are local to that invocation. %export promotes a name one level up to the parent scope. This model gives the locality of pushdef/popdef without requiring the programmer to balance them manually.

m4 uses ifdef to test whether a name is defined. weaveback-macro uses %if(cond, then, else) where the condition is truthy if it expands to a non-empty string; Python handles any condition more complex than that.

m4 is written in C and processes tens of megabytes of macro-heavy input per second. Its --freeze-state / --reload-state mechanism lets you pre-compute a common initialization state and amortise it across many runs. Its changeword feature (which allows redefining word syntax) slows it down considerably and is rarely used. m4 is stateless between runs: no database, no carry-over cost.

weaveback-macro itself is fast (Rust, memchr single-byte scan). The macro pass and the tangle pass are not the bottleneck for large projects. The bottleneck is weaveback.db.

This is where the difference is most pronounced.

m4’s re-scanning model means that many syntactically valid programs produce subtly wrong output rather than an error. A missing or misplaced quote tier causes a macro argument to expand one pass too early or too late. The output looks almost right. The bug can propagate through several files of generated configuration before surfacing. The difficulty is structural: it follows directly from the design.

weaveback-macro’s no-rescan model eliminates that class of bug entirely. The remaining failure modes are:

All of these produce explicit errors with source locations. None of them produce subtly-wrong-but-plausible output.

GPL means that any work that incorporates m4 itself at runtime may have licensing implications depending on how courts interpret dynamic linking and tool invocation. In practice, Autoconf-generated configure scripts are explicitly exempted by the Free Software Foundation, and using m4 as a build tool does not affect the license of the generated output. But it is a question that legal teams sometimes ask.

weaveback’s triple license (0BSD / MIT / Apache-2.0) imposes no restrictions on the generated output and is compatible with any project, including proprietary ones.

m4 has no knowledge of literate programming. Using it as a pre-processor for noweb or similar requires wiring the two tools together manually, handling newline injection carefully (the dnl builtin exists largely to suppress the extra newlines that m4 leaves after macro definitions), and ensuring that m4’s special characters do not conflict with the noweb chunk delimiters (<< and >>). People have done this, but it is an integration problem you solve yourself.

weaveback-macro is designed as a first-class stage of the weaveback pipeline. It runs before the tangle pass; its output is the input to chunk extraction. The special character is configurable so it never conflicts with any chunk syntax you choose. Source locations from the macro expansion pass are stored in the database and surfaced by wb-query trace, so an agent or developer can trace an output line back through both the chunk expansion and the macro expansion to the exact token in the original document.

m4 is a POSIX-standard tool with fifty years of history. It is pre-installed on essentially every Unix-like system. There is a large body of documentation, Stack Overflow answers, and working code — though much of it is Autoconf-specific and not easily transferred to other contexts. The user base is large but passive: most people encounter m4 through Autoconf and never write m4 directly.

weaveback-macro is a new tool with a small user base and no separate community. Its documentation is the source itself (literate-programmed) and this docs directory. You get Rust-quality error messages and a maintainer who reads every issue, but you cannot paste your question into a forum and expect an answer within the hour.

m4 runs on every Unix-like system and Windows (via MinGW/MSYS or native binaries). It is available in every Linux distribution’s package manager.

weaveback is distributed as pre-built binaries for glibc Linux, musl Linux (fully static), Fedora/RPM, and Windows (MinGW and native), as well as an AUR package (weaveback-bin). The Rust toolchain can cross-compile it for any tier-1 or tier-2 Rust target. It is not yet in any Linux distribution’s official repositories.

m4’s system and esyscmd builtins can execute arbitrary shell commands. This is expected — m4 is a build tool, not a sandbox. In contexts where input is not fully trusted, this is a real concern.

weaveback-macro’s %env builtin (reads environment variables) is disabled by default and requires --allow-env to enable, to prevent templates from silently exfiltrating secrets.

%pydef runs inside a sandboxed interpreter with no filesystem access, no network access, and no subprocess spawning. %pydef uses monty, a minimal, secure Python interpreter written in Rust — not CPython. Neither can read files, make network calls, or escape the weaveback process in any way. This is a meaningful security advantage over m4, where system and esyscmd are available to any macro with no opt-in required.

The tangle pass offers a --dry-run flag that prints what files would be written without touching the filesystem. This makes it safe to inspect the effect of a template or chunk set on an unfamiliar codebase before committing to any writes.

m4 is limited to ASCII and 8-bit character sets. It has no notion of multi-byte characters; substr and index operate on bytes. Unicode text passes through only if macros do not touch it; any transformation through substr, translit, or len produces incorrect results for non-ASCII input.

weaveback-macro operates on UTF-8 strings throughout. The special character and macro names must be ASCII: the special character uses memchr for fast scanning, and macro names follow C-style identifier rules ([a-zA-Z_][a-zA-Z0-9_]*). Arguments and output text are Unicode-clean.

Use m4 when:

Use weaveback-macro when: