NAME

Text::Fab - A powerful, general-purpose document expansion framework (currently Control Flow, Loops, and Caching are unimplemented {???})

SYNOPSIS

# -- File: config.fab --   Example of merging multiple files (with almost no content!)
# Declare which configuration keys will be treated as lists
#append Fab/list_keys chapters css_files joiners/lines

# Set some configuration variables
#set site_title=My Awesome Book
#append css_files global.css theme.css
#append joiners/lines "\n"

# Define the inheritance graph for our namespaces
#set_parents ChapterLayout BaseLayout
#set_parents Book _main # The final book inherits from _main's structure

# -- File: main.fab --
# The main assembly script. The default output target is '_main:body'.
#append chapters chapter1 chapter2 chapter3 # Build the chapter list

#emb title
#emb table_of_contents
#emb all_chapters
#emb footer

# -- File: components.fab --
# Define reusable components in namespaces
#target_section title in ChapterLayout
    <h1>#emb _main:title</h1> # Note: Fully qualified name
#end_section

#target_section all_chapters in ChapterLayout
    #= FOREACH chapter IN C('chapters') ... # (Chunk Preprocessor syntax)
        #emb @chapter:body
    #= END #
#end_section

#target_section footer in BaseLayout
    <footer>Copyright 2025</footer>
#end_section

DESCRIPTION

Text::Fab is not a templating engine; it is a framework for building your own. At its core, it is more or less a minimal "dumb" state machine that exposes a stable API of primitive operations. Most of the "smart" logic for parsing syntax, processing text, and creating complex control structures is delegated to user-configurable components. (However, a certain minimal level of smartness is a — switchable off — default to handle the most common cases.) As a result, it knows nothing about the processed “language”'s syntax or semantics.

The main design goal is that all state of Fab is explicitly introspectable: there is no hidden magic. The entire state of the system is contained in the Configuration hash and the collection of already constructed Output Sections. What you see is what you get.

The Fab’s purpose is to factor out the common needs of all the configurable-on-the-fly document processing engines; more precisely, it focuses on the needs orthogonal to the particular syntax of a particular problem domain. (This architecture aims to overcome the experiences with designing “simple” configurable systems — which turn out to be not scalable, so all the approaches to make them operable crumble under their own weight.)

In short: this attempts to abstract out all the complexity of scalable configurability. The resulting “tools” support architecture with a top→down approach to recognizing the “reconfiguration directives”.

————————————————————————————————————

By default, Text::Fab reads input and filters it through a specified preprocessing engine (which defaults to pass-through-unchanged) to the standard output. Its power comes from executing the interleaved-in-the-input directives that are tools for controlling the processing engine, reshuffling content, and templating.

The process is straightforward: Text::Fab asks a De-Interleaver callback "Where is the next control directive?". It then (optionally, see below) passes the text chunk before that location to a Chunk Preprocessor (the "stomach"). Finally, it allows the Parser to execute the directive (typically, the Parser would convert the details of the directive into calls to zero or more of Text::Fab's primitive API methods to alter its internal state). This cycle repeats until the input is exhausted.

Furthermore, as an extra configurable step, the De-Interleaver may need to report that a particular directive “is a comment-to-ignore”. In this case, Fab would postpone processing the text chunk before that location, instead merging it with the following chunk(s) before passing to the Pre-Processor. So the process’ cycle is: the De-Interleaver finds the next directive. The Comment Recognizer may then optionally identify a comment, allowing for the fusion of the preceding and succeeding text chunks. Depending on this: now — or later — this final chunk is then passed to the Chunk Preprocessor. Finally, the Parser executes the directive found above, which may alter the system's state for the next cycle.

Finally, this “structured” (see below) pre-processed data created this way is “merged” into the final output text stream following flexible “hierarchical” rules.

Preview of architecture

To explain what this engine is supposed to simplify, consider a (somewhat contrived) example of document preparation. You have a prepared source of a document; it may consists of several files which should be joined by certain rules; each file is a mix of (recognizable) chunks of markup, LaTeX and HTML; etc. You want to produce a best-effort PDF file, several LaTeX files prepared for publication in various outlets, a readable plain-text version, and a web page.

To add insult to injury, many parts of the document should be displayed differently in different parts of the output. (For example, a section header may be quadriplicated: as a TOC entry, a running page header, as a PDF’s TOC entry, and its rendition in the main text.) And the details of prepocessing some embedded promise (say, MathChunk with the “content” x²) may need to be “massaged” into (the same) string x² for the PDF header, or to $x^2$ when handled inside TOC (or running headers), or to $$x^2$$ in the header rendered “as a part of main text” (well, I warned you that the example is contrived!).

On top of this, the rendition may depend on a mix of recipes inherited from different sources: the default solution “from the vendor”, the enhancements designed by your organization, enhancements found on the web, the requirements of the journal, the legal requirements of the country, your personal enhancements to these etc.

How to handle processing of all this mess through one pass over the initial document? The architecture of Text::Fab encodes the intermediate steps of processing into 6 components, One of them — The Configuration — is very special since all other components (except the content of output sections) are in fact encoded inside it. The aim of these components is to provide carpets under which one can conveniently sweep all the parts of complexity-dust which “require coding” — leaving only “the elementary core dependencies” exposed to both the user and the developer. The intent is that after this sweeping, all (or: better be all?) the necessary configuration/fine-tuning might be done not by writing/editing new code, but by issuing processing directives.

Configuration

A multi-level hash with all the control data needed to process the flow, as well as the temporary variables.

Undo engine

The (structured) list of undo actions to execute at the end of certain processing steps (possibly including undoing changes to the configuration).

Input chain

The inclusion-chain of input files currently in processing (but not yet finished).

Output sections

They are named, and encode chunks of semi-digested input. Each consists of plain text strings and of “promises” to be resolved later, all in one (1-dimensional) list. Each promise contains a name of recipe and the name of section to process (and possibly hints on the “name resolution” of recipes).

Namespaces

They form a DAG encoding “which namespace overrides what” (with the root overriding them all, etc.). Each one resolves certain recipe names to the code to implement these recipes. (Note that if A overrides B, this may be interpreted as “B contains the default actions for A”. — Following the metaphor of inheritance, we call B “a parent” for A.)

A chain of “lenses”

Each lens encodes “the aim of the currently active pipeline” of processing the promises in the sections. Each request to set a new aim is translated (see "Massaging of lens names") to just a particular vertex in the DAG of namespaces, a “virtual root” — so the name resolution is restricted to it and “its defaults”: whatever namespaces it overrides. The resulting vertex is appended the chain of lenses.

Example of usage

With our still contrived example, suppose that the initial pass over the document converted to an output section §base with 4 promises (and nothing else):

#emb secname §sec1name
#emb sectext §sec1text
#emb secname §sec2name
#emb sectext §sec2text

(with §sec1name etc. being the names of other sections), and 4 “referenced above” sections — as well as some other sections which may be referenced from those 4 — etc. (These 4 sections are minimally preprocessed — just enough to recognize what region in the input files goes where (but one can as well also run the preprocessing steps common for all output formats). When processing to LaTeX: one of the first calls would establish the outermost lens to be LaTeX. When one needs to produce TOC, one processes §base as

#emb[aim=as_TOC_lens] as_TOC §base

The installed “recipes for lenses” would (presumably) change the requested root as_TOC_lens to LaTeX/as_TOC_lens. The resulting stack of lenses is now LaTeX LaTeX/as_TOC_lens. And the active lens LaTeX/as_TOC_lens is (presumably) set up to inherit from as_TOC_lens — which sets the name resolution to process sectext promises to completely ignore the corresponding output sections (which are now read for input!), while processing sections of secname promises into their “maximally compact output format”. And on top of this, LaTeX/as_TOC_lens would also arrange to wrap the result into the necessary LaTeX macros.

Should not we list how exactly it is done ???  A promise C<MathChunk> inside §sec2name would be (presumably) resolved to a
C<LaTeX/as_TOC_lens/MathChunk> recipe.  This recipe would pass the content (HOW???  How can it delegate to
C<as_TOC_lens/MathChunk>?  By another C<asiffrom>?) into a promise C<compact/MathChunk>, and wrap the result into a suitable
LaTeX macro.

One processes to PDF_TOC likewise — however, here one needs to transform from the content of §sec2name not to LaTeX macros, but to Unicode plain text. — This may be arranged as above, through a certain installed sub-preprocessor used as a pipe in “an output section” → “an output section”.

When one processes “the main body” of the output, one calls

#emb[aim=as_BODY_lens] as_BODY §base

which would lead (likewise) to the output sections of both secnames and sectexts to be fully processed (with suitable macros wrapping section names — which are also processed into a compact form to be put into the running headers).

Note that how the fact “that the output is to LaTeX” is set up by one top-level call (inside a certain 1-liner output section — WHICH???)! If one wants conversion to PDF or HTML instead and the necessary translators of ”elementary subblocks” are already present and configured, all that one needs to do is to replace this 1-liner by another one. Due to a change to the toplevel lens, the names of the following lenses on the stack would change accordingly (although the specified “aims” when installing these lenses did not change!) — which would change the namespace resolution “as needed”, so the same promises have a chance to depend completely differently. In particular, they may insert “the same promises as for LaTeX” (but which may now depend differently), or they may replace/enhance them by absolutely different promises specialized for HTML processing.

When one need to reformat the LaTeX output for a particular journal, one may override how the names of promises are converted to their “implementation” by including the needed implementations/enhancements into a new namespace(s), inserting them accordingly into the DAG of namespaces.

Massaging of lens names

The translation aim → lens above defaults to id (then the names of lenses conincide with “the request”). More generally, it depends on the current chain of lenses — controlled on the registered list of “which lenses affect which”. If a lens Foo affects Bar, then the result is the lens Foo/Bar; additionally, in the longer chain A, B, Foo, C, D, Bar, if A affects Foo/Bar, then the result is¹⁾ A/(Foo/Bar).

¹⁾ Caveat usor: in the chain C<A, Foo, Foo/Baz> a request for C<Bar> may result in²⁾ C<(Foo/Baz)/Bar>.
   (May later propagate to C<((Foo/Baz)/Bar)/Gee>.)  Since it is only the user/developer who could declare such “affections”,
   it is their responsibility to insert
²⁾ May we ever need to clean C<Foo/(Foo/Baz)/Bar>???

TODO: DAG affecting affections — need inheritance on Foo above???

TODO: If a new aim PDF3.0 appears, and one declared that it affects PDFbookmarks — which has been affecting TeXmath: if the inheritance of PDF3.0/PDFbookmarks on PDFbookmarks is declared, then it is fine: the request to TeXmath inside such a chain would still create PDFbookmarks/TeXmath — which (hopefully) would be processing things correctly. (And if not, then it is the responsibility of whoever implemented this “new aim” to recode how TeXmath behaves in presence of PDF3.0/PDFbookmarks, and declare the corresponding “affects” declaration).

On the othe hand, if the inheritance of PDF3.0/PDFbookmarks on PDFbookmarks is not declared (this may happen for example when the recipes in this namespace are ), then to enable the transparent handling (as above) one should use the other mechanism for “affecting to PDFbookmarks/TeXmath” (now with TeXmath “being a joker”): declare ???

THE PROCESSING MODEL

First of all, the inclusion mechanism handles a stack of currently opened-for-processing files; these files are “combined as usual” into one input stream. The active file chain is reflected in the configuration key opened_includes. In addition to this, the Fab uses three more data collections.

The configuration directives may affect how the preprocessor handles the chunks of its input, whether “the comments interleaved into the stream” split these chunks etc., and may modify how the “directives interleaved into the input stream” are recognized (and even what these directives mean). The grouping directives control which changes to configuration are undone and when. The output directives may insert into the output section "an order to inline another section" (a promise, which is not executed until much later). Another kind of output directive allows switching the target section for the output of the preprocessor.

(It makes sense to also focus on specific parts of the Configuration Hash: the Control Stack which manages conditional handling and looping [Planned], Grouping Stack which handles undoing changes to configuration, and Caching Engine which helps to avoid the penalties of constantly recalculating data dependent only on the configuration stack [Planned].)

After all the sections have been constructed, they may be “joined arbitrarily” into the final document. This assembly process is controlled by the rules how to choose a named section if “multiple flavors” have been defined. This allows a flexible hierarchical system of “overridable templates”. Such templates together with the rules of “namespace resolution” build the logic of processing that ultimately generates the final document(s).

The Configuration Hash

The heart of Text::Fab is a single Configuration hash. It is a live data structure that directives can modify during processing. It controls everything, from user-defined variables to the very components (parsers, etc.) that define the language's behavior. By default, all internal configuration used by the framework itself is stored under the Fab/ top-level key. (Later we are going to discuss how to modify this default, and how this prefix changes during the input processing. However, in examples below we assume that this hasn't been changed.)

Configuration values can be scalars, lists, or hashes. Special hashes, Fab/list_keys and Fab/hash_keys, are used to declare which keys should be treated as lists or hashes by the primitive operations. Both register themselves in new().

The Output Sections

A section is a named buffer that holds already preprocessed content. A section's name is a pair of strings: a namespace and a basename (e.g., Chapter1:body). The default namespace is _main.

It makes sense to imagine that a section consists of two interleaved lists, each of its specific type of content:

  1. Plain text chunks, which have been processed by the Chunk Preprocessor.

  2. embed placeholders, which are promises to inline another section during the final Assembly Phase. This other section is described by its basename as well as some extra data (these data are used in the calculation of its namespace, which is going to be performed later).

Target Section Stacking and Scoping

#target_section <basename [in <namespace>]>

Switches the active output target to the specified section, pushing the previous target onto an internal target stack. If namespace is omitted, defaults to the current target's namespace (initially _main). Autovivifies the section buffer if it does not already exist.

#end_section

Pops the previous target section from the target stack, restoring the output stream to the enclosing section. If the stack is exhausted, falls back to _main:body.

#create_section <basename [in <namespace>]>

Initializes or clears the specified section buffer without redirecting current preprocessor output.

Grouping: Scoped/Undoable Configuration Changes

(Below, we assume a particular #-based format for directives. The actual operation is agnostic of this format.)

#start_group [flavor_name]

Begins a scope. A certain set of keys in the Configuration hash are snapshotted. The flavor_name is used to look up the list of Configuration keys to memorize/restore under Fab/group_flavors/$flavor_name/keys (an array reference). If this key list is not defined for the flavor, the default behavior is to snapshot the entire Configuration hash. Validates nesting constraints against the enclosing group flavor via Fab/group_nesting/$flavor/allowed_parents, halting with E_INVALID_GROUP_NESTING if violated. The active group flavor chain is mirrored in opened_groups.

#end_group [flavor_name]

If a flavor_name is provided, it must match the flavor of the currently open group; otherwise, processing halts with E_MISMATCHED_END_GROUP. All changes made to the observed configuration keys are undone. If the optional name is not given, closes the most recently opened scope. Calling #end_group with no open scope halts with E_DANGLING_END_GROUP. Executes all postponed undo actions and restores the configuration snapshot.

Note: The nesting behavior of groups (e.g., whether flavor A must be nested within flavor B) is itself controlled by the Configuration hash.

DEFAULT DIRECTIVES REFERENCE

The default parser (_default_directive_parser) recognizes the following directives interleaved into the input stream:

#set <key=<value>>

Sets a scalar configuration variable.

#append <key <values...>>

Appends values to a list key, appends string values to a scalar key, or merges an even-sized list of key-value pairs into a hash key.

#prepend <key <values...>>

Prepends values to a list key (preserving their order at the front of the list) or prepends string values to a scalar key. Prohibited on hash keys.

#pop <key [<count_or_subkeys...>]>

Pops elements from a list key by count (defaulting to 1), or deletes specified subkeys from a hash key.

#target_section <basename [in <namespace>]>

Redirects output to the specified section, pushing the previous target onto the target stack.

#end_section

Pops the target stack and restores the enclosing output target.

#create_section <basename [in <namespace>]>

Clears or creates a section buffer without redirecting output.

#emb[<options] <recipe> [<arg_sections...>]>

Inserts an unresolved embed placeholder into the current section. Options in [...] accept comma-separated key-value pairs (e.g. aim=..., in=..., with_blinder=..., callbacks=...). Operands after recipe are bound to $ARGV0, $ARGV1, etc. in the recipe body during assembly. Bare #emb recipe acts as a direct section transclusion.

#set_parents <namespace <parents...>>

Declares inheritance edges in the DAG. Validates cycle freedom (halting with E_CYCLIC_INHERITANCE).

#start_group [flavor] / #end_group [flavor]

Scoped configuration management and rollback boundaries. Supports optional bracketed options #start_group[callbacks=...].

#include [how] <data>

Slurps and processes input. For non-container modes (filename, string), strictly takes exactly 1 target argument. For container modes (section, configkey), strictly requires position 2 as the joiner key and position 3 as the target. Arity violations halt with E_SYNTAX_ERROR.

#include_scoped [how] [joiner] <data <flavor>>

Wraps inclusion in a group named flavor, guaranteeing complete rollback of configuration modifications upon EOF.

## [comment]

Comment line recognized by _default_comment_recognizer. Fuses preceding and succeeding text chunks across the comment boundary.

PRIMITIVE API REFERENCE

These are the low-level methods on the Text::Fab object that user-defined Parsers can call. The user-facing directives (like #set) are defined by the Parser, whose role is to convert these directives into calls to the Text::Fab methods listed below.

Output Control Primitives

out__target_section( $basename, $namespace )

Redirects subsequent output to the specified section. Creates the section if it does not exist.

out__create_section( $basename, $namespace )

Creates a section, clearing its content if it already exists.

out__embed( $recipe_basename, $recipe_ns, \@arg_sections, \%options )

Places an embed placeholder into the current target section. \@arg_sections are positional operands mapped to $ARGVn. The %options hash can contain features like aim, with_blinder, or callbacks.

out__get_joined( $spec, \@joiners, \%options )

A utility for converting a section (representable as a list of text-parts and embed-promises) into a single string by joining text chunks and filtering embeds.

Configuration Primitives

cfg__set( $key, @values )

Sets the value of a key. For a scalar key, accepts at most one value (empty @values sets undef, exactly one value sets the scalar). Passing more than one value halts with E_TYPE_MISMATCH. For a list key, @values sets the list elements (or accepts a single array reference). For a hash key, @values accepts an even-sized list of key-value pairs or a single hash reference; an odd number of items halts with E_TYPE_MISMATCH.

cfg__append( $key, @values )

Appends one or more items to a list key, appends a string value to a scalar key, or merges an even-sized list of key-value pairs (or a hash reference) into a hash key. Appending an odd number of items to a hash key halts with E_TYPE_MISMATCH.

cfg__prepend( $key, @values )

Prepends one or more items to a list key (preserving their order at the front of the list), or prepends a string value to a scalar key. Calling cfg__prepend on a hash key is prohibited and halts with E_TYPE_MISMATCH.

cfg__get( $key, [$offset] )

When $offset is omitted, returns the scalar value, array reference, or hash reference. When $offset is provided, acts as an array index for list keys, or as a sub-key for hash keys. Passing $offset on a scalar key halts with E_TYPE_MISMATCH.

cfg__get_joined( $key, \@joiners, \%options )

A powerful utility that retrieves a list-type or hash-type key and joins its elements into a single string. The @joiners array provides one or more separators that are cycled through when joining. For hashes, joiner 0 binds key to value while subsequent joiners cycle between pairs. The %options can control complex formatting, for example specify permitted numbers of joiners modulo scalar @joiners. (E.g., if “joiners come in pairs”, then the number of emitted joiners must be even!)

cfg__pop( $key, [@args] )

Pops count elements from a list key (defaulting to 1), or deletes one or more subkeys from a hash value when keys are passed.

cfg__prepend_elt( $key, $offset, $value )

Inserts $value at array index $offset for list keys, or inserts/updates a key-value pair [$k, $v] for hash keys. (The parser for directive may implement only the flavors with $offset in 0 and -1 only, e.g., “as if” in cfg__prepend_last( $key, $value ) etc. {???})

Working with Hashes

In addition to scalars and lists, the framework supports hashes as a first-class configuration data type. This is controlled by two special configuration keys:

Fab/hash_keys

Analogous to Fab/list_keys, this is a hash whose keys are the names of configuration variables that should be treated as hashes by the primitive operations.

# In a config file (registering a hash name):
#append Fab/hash_keys my_parameters 1
Fab/hash_key_sort_order

This key holds a subroutine reference that defines how the keys of a hash should be sorted when the hash needs to be treated as an ordered list (e.g., for cfg__get_joined). The subroutine receives two keys as arguments. The default is standard string comparison.

# In Perl code, to set numeric sorting:
$fab->cfg__set('Fab/hash_key_sort_order', sub { $_[0] <=> $_[1] });

Grouping and Postponement Primitives

group__start( [$flavor_name, \%options] )

Starts a scoped group.

group__end( [$flavor_name] )

Ends a scoped group.

group__postpone( $depth, $priority, $how, $method_name, @args )

Schedules a Text::Fab method call (or code reference) to run when the group at relative depth $depth is closed. All postponed actions have strict undo semantics: failure in one action does not abort the remaining undo actions or snapshot rollback. $priority is a real number: Priorities < 0 execute PRE-undo while the group scope is still active. Priority 0.0 is the snapshot rollback point where configuration is restored. Priorities >= 0 execute POST-undo in the restored enclosing scope. $how must be 'append' or 'prepend' to define execution order among actions with equal priority.

The closing frame remains active on the stack during unwinding, so actions can target the closing group at depth 0 or parent groups at depth > 0. Insertions into existing current/future priority buckets are consumed directly by the active loop without recalculation. Insertions into the past (time-machine actions) or into non-existing priorities trigger on-demand recalculation of the priority array.

(The parser for directives may allow more user-friendly UI, such as specifying “a path” to this group, as in “the last-but-one x group before the last two y groups.)

priority_for_file_close( [$dist_from_end] )

Calculates file-closing priority based on distance from the end: distance 0 (last closed file) is -1000.0, distance 999 is -1999.0, and distance >= 1000 is -2000.0 (overflow).

Input and Inclusion Primitives

include__text( $how, [$joiner_key], $source )

Processes included input. If $how is 'filename', $data is read from disk and processed. Recursion is tracked on an inclusion stack and mirrored in the configuration key opened_includes; attempting to include a file currently in progress halts with E_CYCLIC_INCLUDE. An unreadable file halts with E_FILE_READ_ERROR. If $how is 'string', $data is processed directly as an in-memory string.

If $how is 'section' or 'configkey', requires $joiner_key (unless reading from a scalar-valued key); the $source is interpreted as a section-name or a key, correspondingly. This retrieves the joiner list from configuration and joins the content using cyclic separators before processing. {??? requires ($joiner_key, $target, [$options])} {??? is reading from a scalar-valued key special-cased?}

Namespace and Inheritance Primitives

NS__set_parents( $namespace, @parents )

Registers the list of immediate parent namespaces for $namespace in the inheritance DAG (stored under Fab/inheritance_graph/$namespace/parents). Before writing, the inheritance graph is validated for cycles; adding a parent that would create a directed cycle halts with E_CYCLIC_INHERITANCE.

assemble( $root_spec )

Assembles and expands the section specified by $root_spec (e.g. Namespace:Basename or bare Basename defaulting to namespace _main). Coordinates lens affection massaging, positional argument binding ($ARGV0...), and blinder masks.

Configuration Prefix and Call Stack API

For recursive processing, the Configuration contains a stack of prefixes at Fab/call_stack_prefixes. Primitives are provided to safely interact with keys relative to this stack.

(We list only the most basic retrieval primitives; the rest follows the same principle.)

call_stack__push( $prefix, [$filter_name] )

Pushes a new configuration prefix onto the call stack at Fab/call_stack_prefixes. If $filter_name is provided, pushes it onto Fab/call_stack_filter_name.

call_stack__pop()

Pops the topmost prefix from Fab/call_stack_prefixes and pops Fab/call_stack_filter_name. Guards the root prefix 'Fab/' from being popped, halting with E_UPLEVEL_TOO_DEEP on underflow.

cfg__get_prefixed( $key )

Retrieves the value of $key using only the current, most recent prefix (i.e., CURRENT_PREFIX/$key).

cfg__get_prefixed_scan( $key )

Retrieves the value of $key by searching for it under each prefix in the call stack, from most recent to least recent.

cfg__get_uplevel_scan( $level, $key )

Retrieves the value of $key by first popping $level prefixes from the call stack and then performing a prefixed search.

cfg__set_prefixed( $key, @values )

Sets the value of $key using the current (most recent) prefix.

For setting things “uplevel on this stack”, there are two methods matching two types of access:

cfg__set_prefixed_uplevel( $level, $key, @values )

As cfg__set_prefixed(), but goes back $level steps on the stack of prefixes. This matches retrieval via cfg__get_prefixed().

cfg__set_prefixed_uplevel_fixup( $level, $key, @values )

As cfg__set_prefixed_uplevel(), but first runs “fixup”: copying the preceding value at $level to the levels between this one and the current level — stopping when value is found on a particular level. This matches retrieval via cfg__get_prefixed(): this call won’t change the results of these calls on the levels between $level and the current one.

ADVANCED TOPICS

Input Processing and Recursion

The primitive to support the #include directive is include__text($how, @args). If $how is filename, the file $data is searched for and slurped; if not, $data is the input string to be processed. A special form #include_scoped ensures that configuration changes made by the included file are temporary by wrapping the call in a group named by the last argument. Active inclusion paths are mirrored in opened_includes.

Recursive processing (e.g., using one Text::Fab “process” to filter data for another) is managed entirely within the Configuration hash. A stack of configuration prefixes is maintained in Fab/call_stack_prefixes (which usually starts as just Fab/). When a named recursive filter is invoked, its name is pushed to the filter name stack Fab/call_stack_filter_name. This also pushes a certain prefix onto Fab/call_stack_prefixes via call_stack__push(). Since it is the last prefix on this stack which is actually used by Text::Fab for its operation, this allows a special predefined configuration (e.g., Fab/myList2ScalarFilter) to take precedence without conflicting with the “normal” operation of the Fab.

The uplevel family of configuration directives operates with respect to both the group stack and this call stack, allowing for powerful, controlled communication between different layers of processing.

Namespace Resolution and Inheritance of output sections

The inheritance system is a layer of logic built on top of the primitive operations. It is controlled entirely by the Configuration hash. Here we illustrate the API by how the corresponding directives may look like. The API has an extra prefix NS__, and omits the UI verbiage.

#set_parents <namespace> <parent1> <parent2> ...

A user-facing directive that calls cfg__set on a key within the Configuration that stores the inheritance DAG. For example, it might set Fab/inheritance_graph/MyNamespace/parents to [Parent1, Parent2].

The Assembly Phase uses this graph to resolve embed placeholders in the output sections; below we denote them as #emb. The resolution can be controlled with extra data:

#emb[aim=<Aim>, ...] <recipe> [<arg_sections...>]

This performs lens-guided resolution. The requested aim is passed through _massage_lens across the active lens stack via static joining rules and registered affections in Fab/lens_affects, establishing the virtual root in the DAG. When the virtual root is not the root, this makes the name resolution "blind" to any “override sections” up the tree/DAG. (Recall that the root contains the most specific overrides.)

Operands are mapped to $ARGV0, $ARGV1, etc. in the recipe body.

#emb[with_blinder=<SubName>, ...] <recipe> ...

Here SubName provides a completely customizable analog of the previous call, looked up as a subroutine reference under Fab/blinders/$SubName. Instead of (or in addition to) obscuring the part of DAG which is not reachable from the aim, the output of this user-defined subroutine obscures an arbitrary subset of the DAG. Blinding a namespace masks that namespace's own section definitions from satisfying the embed, while allowing inheritance traversal to continue through its parents in the DAG. It receives the entire current embedding stack (each entry is the currently executed embedding promise) together with the start namespace, the used blinder name, and the output of the blinder for each level. It returns a dynamic list of namespaces to "blind" for this specific resolution and any sub-resolutions (which may happen during processing of this embedded section).

Support of Control-Flow-Like constructs (conditionals and/or loops) [Planned] {???}

To simplify support of matched nested control-constructs in the user-defined parser, Fab maintains 6 stacks of matched lengths. Because of this match, they should be modified only via the corresponding API. E.g., the last elements of these lists describe:

  • The type of the last encountered starter of the construct. (Set by control__start())

  • The (optional) text label for the construct. (Likewise. Defined for “loop-like” constructs only.)

  • Are we in a skipping or not (“live”) branch/flavor of the construct. (With the value 2 if the processing were skipping even before meeting the start of the construct. Set likewise at start, then may be changed.)

  • The offset of the start of the body (for “loop-like” constructs only). (This may be the file offset for seek()able files, otherwise offset in a suitable buffer.)

  • Other useful offsets (for support of various loop-related targets for analogues of goto) packed into a hash.

  • The loop_counter (defined for “loop-like” constructs only). Changed (in a parser-specified ways) on “jump” calls.

  • Which targetToSeek we are trying to find in the “skipping” mode now.

There is also the list of the indices Fab/control__loop_indices of “loop-like” constructs in this stack.

Essentially, when encountering the start of an analogue of if/elsif/else/endif, the parser may registers its type, as well as whether the start is in the “skipping” mode via ctrl__start(TYPE,IS_skipping,LABEL). Likewise, every flip of the skipping mode is registered with ctrl__skipping(IS_skipping) (this is going to be ignored if the current skipping state is 2; likewise, Fab knows when to set the state 2 automatically at start). When the end of the construct is encountered, these data may be popped by ctrl__end(TYPE) (with TYPE given for error-checking only).

The latest “skipping” state determines whether the preprocessor calls are skipped, and it is also given as an argument to the de-interlacer and parser callbacks. It also affects the state of loops nested inside “skipped” input.

These Control Stacks are reset when a new input file is included and restored when processing of that file completes.

The API to deal with these stacks is:

control__start( $type, $is_skipping, $label) [Planned] {???}

Pushes a new frame onto the Control Stack.

  • $type and $label are stored directly.

  • The initial skipping_state for the new frame is computed as follows: if the parent frame (if one exists) is already in any skipping state, the new frame's state is set to "skipping-on-entry" (state 2). Otherwise, the new frame's skipping state is set based on the boolean $is_skipping (true -> "skipping", false -> "live").

control__set_skipping( 'flip' | $boolean ) [Planned] {???}

Modifies the skipping_state of the current control block.

  • If the argument is 'flip', the state toggles between "live" and "skipping", but only if the current state is not "skipping-on-entry" (state 2).

  • If a boolean is provided, the state is set accordingly, again respecting the immutability of the "skipping-on-entry" state. This is called by the Parser for constructs like #elif.

control__end( $expected_type ) [Planned] {???}

Pops the top frame from the Control Stack. For error-checking, the Parser must provide the $expected_type. (Typically, this is used for loop-like constructs only when they are in “skipping mode”; see the next API.) Fab/control__loop_indices is changed accordingly.

control__define_invalidation_pointer($expected_type) [Planned] {???}

Mark the next position as “First postion not in the loop”. When this position “is activated”, or the jump is performed past this position (or before the starting position), the effect should be the same as control__end( $expected_type ).

control__set_pointer( $pointer_name, [$before], [$force] ) [Planned] {???}

Records the current position in the input stream (file or buffer) (if $before, this is the position before the directive) and stores it under $pointer_name in the "Custom Pointers" hash of the current control block. This is the sole mechanism for defining loop start points, continue targets, or any other labeled position. For all loop-like constructs, the Parser is expected to call this at the beginning of the loop body to define a 'body_start' pointer.

These pointers form the structural geometry of the loop. One must keep in mind that Fab may assume that this geometry is not modified (only enhanced by definition of new pointers) during “each pass” of a particular “instance of processing” this loop. (Here our terminology is a bit dense. If the current loop is nested inside another loop, it can be entered many times — leading to different instances of processing this loop. In each instance the body may be looped over in several passes.)

By default, this condition is auto-enforced: this call is ignored on the “replay passes” of “loop-like” constructs (i.e., if the current portion of the loop’s body “has already been read” — inside the current instance). This filtering may be disabled by the $force flag. (A similar flag $is_replay is passed to the parser — just in case.)

control__jump_to( $pointer_name, [$block_label] , [$incr_loop_counter]) [Planned] {???}

This is the primitive for all control flow transfers. It instructs the Fab core's main processing loop to change its read position.

  • If $block_label is given, it walks up the Control Stack to find the frame with that primary label. If not, it uses the current (top-most) frame.

  • It looks up $pointer_name in that frame's pointers hash.

  • If found, it commands the main loop to immediately stop scanning at the current location and resume reading from the retrieved pointer's location (either by seeking in a file or switching to a buffer at a specific offset).

  • If not found, it adds a new targetToSeek name, sets the current execution state to "skipping" (this essentially initiates the forward scan).

The loop_counter is incremented by the number in $incr_loop_counter (when the target is found).

Sometimes a callback may need a computationally expensive combination of individual entries from a configuration hash. (On may think of a regular expression depending on several values in this hash.) To avoid recalculating it every time, the callback may use the logic like this:

unless ($cache_at_count == (my $new_cnt = $fab->{config}{cnt__parser_REx})) {
  $cache = my_recalc_cache($Fab);
  $cache_at_count = $new_cnt;
}

This assumes that $fab->{config}{cnt__parser_REx} changes every time one of the variables used by my_recalc_cache() is updated (or restored). To support this, one register each of these variables with the “dependent counter” cnt__parser_REx using the API:

  • config__append_dependent_counter($counter_key, $KEY1, ...) [Planned] {???}

(Warning: the implementation is free to change the counter back when a group is undone if it knows all the relevant-and-changed variables were properly undone. So (theoretically) one cannot rely on the counter to increase-on-modification.

A tacit assumption — caveat usor

Summary: Before starting a design using Text::Fab, check that inside your preprocessor, the internal representation of the partially processed input can be easily enough kept in the Configuration Hash. We assume that this is going to hold as far as the preprocessor “proceeds with a well-defined processing pipeline”.

---

In many cases one needs to allow the configuration driving the logic of work of a preprocessor to be changeable by a directive at any time. (For example, a text in one programming language may inline a text in another programming language!)

The design of Fab’s API can make this (seem to be?) hard: (due it its up→down approach to the search of directives) the directive above would interrupt the input to the preprocessor; — but sometimes the preprocessor may need a non-local view on the input to decide how to deal with it. “Just seeing what comes before the directive”, then “Just seeing what comes after the directive” (in a separate invocation!) may be not enough for such a non-local view.

On the other hand, one should be ready for the situation when the input after this directive should be handled differently than the input before this directive. So if the target domain requires handling such situations, then any implementation of such a preprocessor must be ready to deal with such hiccups when processing the input (either by handling the update of its config, or bulking out with an error message). While doing this with a bottom→up design may seem easier, the experience shows that often it is prohibitively hard to implement. (E.g., when nested constructions are dealt with by recursive calls, the calls to reconfigure_me() would happen deep inside the stack of recursive calls!)

In short: to implement such a design through Text::Fab the only possible complication is: how to preserve the internal representation of the “preceding not-yet-fully-digested input” within the preprocessor “when the preprocessor is reentered” after handling such a directive?

With the bottom→up approach (when the preprocessor may “call some API” to handle the directive) this is easy: it is easy to make sure that such a call may update “the configuration” of the preprocessor while keeping “the internal representation of the already-read-input-to-preprocess” fully intact. However, when the preprocessor is controlled from a Fab, what happens instead of “calling the API” is: “exit the preprocessor”, then somebody else (the Fab!) calling the API, then “reenter the preprocessor again”. Therefore when the preprocessor can see that the chunk it obtained is a part of “some larger construct” (so it is not ready to be flush()'ed to the output section yet), it must preserve its internal state “somewhere”, then read it back when it is reentered (if it is ever reentered). This is the price to pay for (a lot of) work offloaded to Fab.

CONCLUSION:

  1. In such situations the preprocessor needs a guarantee that it is going to be reentered;

  2. Its internal data should be easily translatable to/from formats supported by the configuration hash.

(Indeed, this hash is the only supported storage provided by Fab’s API.)

In other words: the preprocessor cannot be stateless now, but as far as the state may be (de)serialized easily, it is not a big deal. So one possible tacit assumption about designs using Text::Fab may be:

  • The preprocessing is hierarchical, each level of hierarchy of not-yet-fully-processed-input may be put into a value suitable for the Configuration Hash, and the steps remaining to finish processing these data are not affected by the directive above.

————————————————————————————————————

Next, inspect how not-stateless but still “easy to implement” preprocessors may look like.

Example B: We have a text which may contain nested “blocks”, each block made of one or more paragraphs. The nesting is defined by the indent level (as in Python). The “class” of each block is determined by inspecting the immediate start of the its first paragraph, as well as inspecting the immediate end of the last paragraph of block (provided this last paragraph is “on the same indent level as the start” — so it is not in a nested subblock).

In addition to this we allow #define directives which take a whole line — but are not considered as interrupting the logic of indentation-which-defines-nested-depth.

Implementation: To implement “reenterancy”, the stored data consists of the stack of already-preprocessed text of non-yet-finished blocks. When preprocessing a chunk of input finishes a block, the preprocessor can inspect its start and end, determine its class, pop()'s the known “intermediate textural representation” from the stack, and converts it to the final representation in the way suitable for this class. When this was the outermost block, the result is put into the output section. Otherwise it is appended to the preceding entry on the stack (the containing block).

————————————————————————————————————

Example H: As above, but the rules for massaging a block depend not only on its class, but also on the classes of its parents. This requires “the most general” hierarchical model of the pipeline of not-yet-fully-processed input.

In this case again the handled data may be flush()'ed to an output section only when the outermost encountered block is terminated. Until then, one should maintain the stack of “live blocks” (not yet terminated), as well as a forest of “dead” (already terminated, but waiting the finite processing) blocks. Each terminated block has its class and the list of content, elements are either “a literal text” (preprocessed-except-the-final-step), or another (nested) terminated sub-block. The live blocks are likewise, but the class is not yet fully known, and the last nested sub-block may be live.

How to preserve this in the Configuration Hash? We have a planar rooted tree with vertices assigned the class (or undef for live vertices), and leaves containing “a plain text”. When flush()'ing to an output section, we essentially scan depth-first through the vertices, maintaining the stack of classes (they correspond to the path from the root to the current vertex). So the only bookkeeping data in addition to the stack of classes is the (parallel) stack of “where the currently-not-yet-finished sub-block ends”. (So we can pop() the stack of classes when this end is reached.)

So if we store the leaves in the depth-first order, we need to interlace this list with markers “start sub-block of the class CLASS ending at position POS” (or introduce nested start and end markers). To design an easy representation, suppose that the semantic of “massaging the blocks” allows introdution of empty "plain text leaves" between every two adjacent sub-vertices; then we may assume that “end-subblock” entries are represented by out-of-bound elements (e.g., undef), and the plain text leaves are alternating with “start subblock” entries in the stored list, as in flattened list of the following form:

0thLlvTxt1
  Blk1
    1stLvl1Txt11
      Blk11
        2ndLvl1Txt111
      undef
    1stLvl1Txt12
      Blk12
        2ndLvl1Txt121
      undef
    1stLvl1Txt13
  undef
0thLlvTxt2
  Blk2
    1stLvl1Txt21
      Blk21
        2ndLvl1Txt211
      undef
    1stLvl1Txt22
      Blk22
        2ndLvl1Txt221
      undef
    1stLvl1Txt23
  undef
0thLlvTxt3

Observe alternation Txt/BlkClass/Txt/BlkClass/Txt/… after every undef. It is trivial to process such a list when flush()'ing. When filling the list, one should additionally maintain a stack of positions in this list where live block starts, so one can change a placeholder-for-its-class to the calculated class when the end-of-block is found. (Other than this, all the results of pre-processing the input land at the end of the list.)

————————————————————————————————————

Example A: Consider a macroprocessor for macros with arguments; assume that the argument-separators and argument-terminators for a macro on a certain level of nesting may be contained in the output of nested-deeper macros. So one needs to expand the deeper macros before the handling of enclosing macros is finished. Can one support directives occuring deep inside the nested arguments — but we assume they cannot be contained in the macro-expansions?

Here the data may be even easier than in the preceding example. Since the output produced up to the start of the outermost no-yet-terminated macro may be flash()'ed, the stack of unprocessed data may contain the ID of the outermost no-yet-terminated macro, the list of its completed arguments, then the already expanded prefix of the current argument, then the ID of the next no-yet-terminated macro, the list of its completed arguments etc. In addition to this list, it is enough to have the list of offsets at which the IDs of the macros live.

When a terminator of a macro is found, one pops from the list above the macro ID and the completed arguments, and performs the macrosubstitution. There are two ways to proceed with this string: it is either appended to “the already expanded prefix of the current argument” (as above) of the enclosed macro (if the expansion should not be macro-re-expanded) — and optionally scanned for macro-argument-delimiter or -terminator, or it is prepended to the buffer containing the not-yet-processed input (otherwise). The desired choice is determined by the semantic of the macro-expansion.

————————————————————————————————————

Example E: As above, but the directives may appear as the result of macro-expansion.

This case is much more involved, since it seems it is too late for the Fab to inspect the internal state of the macro-processor. So the preprocessor should scan its output itself: it has the full access to the De-Interlacer/Parser/etc callbacks designed to detect the directives.

WARNING: However, the directives are not designed to work with not-yet-completed buffers. For best result, De-Interlacer should return “the high-water mark”: the offset before which the directives cannot appear — so there is no sense to rescan this part when the buffer is extended.

ERROR REFERENCE

The preprocessor will halt with one of the following error codes if a fatal condition is met.

Parsing and Input Errors (E_Input)

  • E_SYNTAX_ERROR: A directive does not conform to the grammar, or an unrecognized directive is encountered.

  • E_FILE_READ_ERROR: An #included file cannot be opened or read.

  • E_CYCLIC_INCLUDE: An #include chain references a file that is already being processed.

Grouping Errors (E_Group)

  • E_MISMATCHED_END_GROUP: An #end_group <flavor> does not match the currently open group.

  • E_DANGLING_END_GROUP: An #end_group is found with no matching #start_group.

  • E_INVALID_GROUP_NESTING: An attempt to nest groups in a way forbidden by the Configuration.

  • E_UPLEVEL_TOO_DEEP: An uplevel directive or call stack pop targets a level that does not exist.

Configuration Errors (E_Config)

  • E_TYPE_MISMATCH: A list-specific operation is attempted on a scalar key, or vice-versa.

  • E_LONG_POP: Popping too many elements.

Namespace and Assembly Errors (E_Assembly)

  • E_CYCLIC_INHERITANCE: An #set_parents directive creates a loop in the inheritance DAG.

  • E_UNDEFINED_PARENT: An #set_parents directive refers to a non-existent namespace. {??? Documented in error specification, but NS__set_parents currently validates cycle freedom without requiring candidate parents to pre-exist.}

  • E_ROOT_NOT_FOUND: A section specified via #set_root does not exist. {??? Planned root directive; currently assemble() accepts the target spec as an argument.}

  • E_NO_ROOTS_SPECIFIED: The assembly phase is triggered, but no roots were ever defined. {??? Planned root directive; currently assemble() accepts the target spec as an argument.}

  • E_EMBED_NOT_FOUND: A section referenced by an #emb placeholder cannot be resolved.

  • E_CIRCULAR_EMBED: An #emb chain results in a loop during the final assembly.

  • E_UNKNOWN_BUNDLE: A requested callback bundle is not registered under Fab/callbacks_bundle/.

  • E_ARGV_EMPTY: Referenced an $ARGVn section parameter outside of a recipe invocation.

  • E_ARGV_BOUNDS: Referenced an $ARGVn index that exceeds the number of supplied section arguments.

Control Flow Errors (E_Control) [Planned] {???}

  • E_MISMATCHED_CONTROL_END: An end directive (e.g., #endif) was encountered, but the currently open control block is of a different type (e.g., a #for loop).

  • E_DANGLING_CONTROL_END: An end directive was encountered when the Control Stack was empty.

  • E_UNCLOSED_CONTROL_BLOCK: The end of an input file was reached while one or more control blocks were still open.

  • E_POINTER_NOT_FOUND: A control__jump_to call referenced a pointer name that has not been defined in the target control block.

  • E_BLOCK_LABEL_NOT_FOUND: A control__jump_to call referenced a block label that does not exist on the Control Stack, and was not found before the end of the enclosing block.

    Payload: (target_pointer_name, block_label)

[PLACEHOLDERS]

Future sections will detail the full APIs and default implementations for:

  • The Parser: How it interacts with the main loop and calls primitives.

  • Chunk Preprocessors: The interface for the "stomach".

  • Blinder Subroutines: The data structures passed to them.

  • The default configuration schema under the Fab/ key:

    Fab/callbacks_bundle/dbg0

    A hash reference containing the canonical operational engine callbacks: interleaver, directive_parser, chunk_preprocessor, comment_recognizer, and error_handler. {??? The active callbacks bundle is referenced via Fab/callbacks.}

    The comment_recognizer callback receives ($fab, $buffer, $offset) and returns the offset after the comment line if recognized (defaulting to lines prefixed with ##), which triggers chunk fusion across the comment boundary. {??? Docs of the rest?}

EXPORT

None by default.

TODO

The default mode of process is equivalent to writing to a buffer __default__, and at the end executing the file

= = = = = filterbuffer(raw) __default__

Doing the same abstraction once more, the buffer with that contents may be also created dynamically, then loaded via filterbuffer(). On the gripping hand, this may be also

= = = = = filterbuffer(raw) __default__pre
= = = = = filterbuffer(raw) __default__
= = = = = filterbuffer(raw) __default__post

With (initially) empty buffers __default__pre and __default__post (to which one has a chance to write later!). And what one writes to these buffers may be triplicated likewise. Etc.

For postponed actions need an extra argument "ignore errors": with 2 would just put into eval, with 1 converted to a warning, negative values stop the remaining undo actions for that many frames (with -2 stopping for the current frame and one more).

One needs to prohibit __get()ing a reference. Use __getref() when you expect a reference (but no checks for reference are performed).

MISS: input__end($n) with $n==0 ending the current input stream, or also $n enclosing streams.

MISS: need to specify what happens with a group on input__end(): survives / should be closed before / close-if-still-open.

MISS: input__end_to($group_type): end input of the files opened up to the the opened-latest-group of the given type.

Likewise for “closing enclosed groups”??? Join together into 1 API by introducing suitable groups???

  May run forever??? Even the language if group__start, group_end, group__postpone (with the arguments to the last command
  restricted to these 3 calls) is probably Turing complete???

MISS: _-prefixed versions of config__*() which check the prefix _ __ ___ of the key and make it read/write only.

MISS: pipelining preprocessor with “filters”: filters consists of 3 components: preprocessor, massaging of promises, and the concatenator of the resulting candidates-for-sections.

Uplevel w.r.t. the recursion stack should be recalculated into the usual uplevels; a special do-nothing group may be used.

allow not 1 active preprocessor engine, but a stack of them, chained, and have a cfg__pop() primitive method.

Macros are templates marked as OK-to-be-ignored.

Hash to mark macros as “types”, with per-type-warning configurable? Which way to hash, types to macros, or macros to types, or pairs???

An extra parameter $is_replay for the syntax callbacks for the non-first pass. (Probably, the boundary of the preceding pass cannot be part of this???

hash keys. Setting/getting them employs a serialization method, by default JSON???

     !!!!!!!!!!!!!!!!!!!!!!!!!!!!	THIS IS TOO DANGEROUS.
					Access control to Fab/* keys should be guarded by ungettable/settable types of values ???

#unset — but maybe it is going to make undo harder???

Recursive invocation is needed for massaging data between different levels:

• The hash
• Input file names
• Input strings
• Output sections

A top→down approach to recognizing the “reconfiguration directives”: but cannot we support a bottom→up approach too, when the preprocessor recognizes interlaced directives? How to deal with recursion: return an indicator that the state changed? But it is changing permanently, one needs to filter out changes — but how the internal calls know the filters needed for the enclosing calls???

Resolving to _main namespace is broken (too early); it should be left undefined, so the resolver knows it should do the late resolution — at runtime, when the lenses may be set up differently.

When piping one output section to another, how to decide which promises are “executed”, and which are postponed? Compare a real “watermark” set in the promise and in the piping stage?

the preprocessor needs a guarantee that it is going to be reentered

SEE ALSO

Mention other useful documentation such as the documentation of related modules or operating system documentation (such as man pages in UNIX), or any relevant external documentation such as RFCs or standards.

If you have a mailing list set up for your module, mention it here.

If you have a web site set up for your module, mention it here.

AUTHOR

Ilya Zakharevich, <ilyaz@cpan.com>

COPYRIGHT AND LICENSE

Copyright (C) 2025, 2026 by Ilya Zakharevich

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself, either Perl version 5.22.2 or, at your option, any later version of Perl 5 you may have available.