Glob-aware Tera dependencies

The problem

Some Tera templates depend not on the contents of any one file, but on the set of files that satisfy a pattern. The motivating example is from teaching-slides:

- Currently there are {{ shell_output(command="find marp/courses -type f -name '*.md' -printf '%h\n' | sort -u | wc -l") }} courses in this repo.
- Currently there are {{ shell_output(command="find marp/lectures -type f -name '*.md' | wc -l") }} lectures in this repo.
- Currently there are {{ git_count_files(pattern="marp/**/*.md") }} marp files in this repo.
- Currently there are {{ shell_output(command="python3 scripts/count_slides.py") }} marp slides in this repo.
- Currently there are {{ git_count_files(pattern="svg/**/*.svg") }} SVG diagrams in this repo.
- Currently there are {{ shell_output(command="grep -r '^|.*---.*|' marp/ --include='*.md' | wc -l") }} tables in this repo.

The rendered output changes when files under marp/ and svg/ are added, removed, or renamed, even though no specific file has been edited and no path is listed as a dependency. rsconstruct today tracks content-addressed dependencies: a product is rebuilt when the contents of its declared inputs change. Add or remove a .md file under marp/ and the template product is happily served from cache, even though its rendered output is now stale.

This is a fundamental “the build action depends on the directory listing, not on a fixed set of files” problem. We need a way to express that.

How other tools solve it

Tool	Approach
make	Doesn’t solve it. Globs in Makefiles are evaluated once, when the Makefile is parsed. Re-running `make` after adding a file picks up the new file because the Makefile is re-parsed. Make’s dependency model = file content + mtime, never directory contents.
ninja	Same as make — relies on the build-graph generator (cmake, gn) to re-run and produce new `build.ninja`.
bazel / buck	Solve it cleanly by forbidding implicit globs inside actions. Globs are evaluated by the build tool itself at graph-construction time, producing a fixed list of files baked into the action. Add a file → BUILD-file evaluation produces a different action with a different cache key. The action itself never reads the directory. This works because Bazel owns the `glob()` function — it’s not a side effect inside the user’s tool.
ninja + depfiles	The tool emits a `.d` file at build time listing every input it actually read. Doesn’t help here because pandoc/tera/sh don’t know they’re “reading the directory”.
tup	Hooks the build with FUSE/ptrace and observes every `open()` and `readdir()` call. Captures directory listings as true dependencies. Heavyweight; cross-platform pain.
directory mtime	Cheap and approximate. A directory’s mtime updates when entries are added or removed. Catches add/remove. Does not catch rename within a directory if the new name still hashes the same number of files; does not catch content-dependent queries (`grep -r`).

Chosen approach: Bazel-style, ported to Tera

Stop letting templates make uncontrolled directory queries. Promote directory queries to first-class rsconstruct constructs that participate in the dependency graph.

Three layers:

1. New Tera function: `glob(pattern=...)`

Templates that today say:

{{ shell_output(command="find marp/courses -type f -name '*.md' | wc -l") }}
{{ git_count_files(pattern="marp/**/*.md") }}

are migrated to:

{{ glob(pattern="marp/courses/**/*.md") | length }}
{{ glob(pattern="marp/**/*.md") | length }}

glob() is implemented inside rsconstruct’s Tera engine. It does two things:

Expands the glob using rsconstruct’s FileIndex — the same mechanism the rest of the build uses, so it respects .gitignore and .rsconstructignore and behaves consistently with src_dirs.
Records the pattern as a graph-level dependency on the calling product. Specifically:
- Every matched file is added to product.inputs (its content checksum contributes to the product’s input checksum).
- The sorted list of resolved paths is hashed into a glob-set fingerprint which is mixed into the product’s config_hash.

The fingerprint matters because rsconstruct’s cache key is content-addressed (Cache System) — renaming a file with identical content otherwise wouldn’t bust the cache. Mixing the sorted-path fingerprint into the config hash makes add / remove / rename all flip the cache key deterministically.

Returns a list of file paths as strings. Templates can use Tera’s built-in length, sort, iteration, etc. on the result.

2. Tightened `shell_output(command=..., depends_on=[...])`

For users who genuinely need to shell out for content-dependent results (grep -r '...' marp/), require an explicit depends_on list of glob patterns:

{{ shell_output(
    command="grep -r '^|.*---.*|' marp/ --include='*.md' | wc -l",
    depends_on=["marp/**/*.md"]
) }}

shell_output then:

Resolves every pattern in depends_on, adding matched files to product.inputs.
Mixes the sorted union of resolved paths into the product’s config hash.
Mixes the literal command string into the product’s config hash so command edits also rebuild.
Runs the command and returns its trimmed stdout (current behavior).

A shell_output call with no depends_on is rejected at template-render time:

shell_output() requires a depends_on=[...] list of glob patterns. rsconstruct cannot otherwise tell when its output should be invalidated. If your command genuinely has no file dependencies, pass depends_on=[] explicitly to acknowledge that.

The empty-list escape hatch is for commands like date or whoami whose output rsconstruct can never track — the user takes responsibility.

3. The Tera analyzer extends to scan for these calls

src/analyzers/tera.rs already scans templates for {% include %}, {% import %}, {% extends %}, and load_lua/load_data/... directives. Extend it to also recognize:

glob(pattern="...") — captures pattern, expands to file list.
shell_output(..., depends_on=[...]) — captures the literal pattern list, expands each to file list. The command string itself is captured into config_hash (not inputs).

Captured patterns and their resolved file lists become per-product dependency contributions, fed through the same machinery used today for include/import dependencies. The analyzer cache (.rsconstruct/deps.redb) gets one entry per template per pattern, so re-runs on unchanged templates are O(1).

Why this is the right shape

Composes with existing machinery. No new caching layer; everything flows through product.inputs, config_hash, and the existing content-addressed cache key. The only new piece is the glob-set fingerprint, which is a one-liner sha256(sorted_paths.join("\n")).
Catches all three failure modes. Add → new path in glob set → new fingerprint. Remove → path missing → new fingerprint. Rename → both fingerprint and inputs change.
User opt-in is local. The user adds a function call in one place; rsconstruct does the rest. Compare with dep_inputs = [...] in rsconstruct.toml, which scales poorly because every template that uses globbing would need a config update.
Failure of the old idiom is loud. shell_output without depends_on becomes an error, not a silent stale build. Existing templates surface immediately when first re-rendered.
No filesystem instrumentation. No FUSE, no ptrace, no eBPF. Stays cross-platform and inside the “simple, cross-platform” philosophy in CLAUDE.md.

Migration

The change is backwards-incompatible for shell_output calls without depends_on. Two options:

Hard cut-over. Bump the processor version on tera. First build after upgrade fails fast on every legacy shell_output call with the error above. User migrates to glob(...) (preferred) or shell_output(..., depends_on=[...]).
Soft transition. Add a new function tracked_shell_output(..., depends_on=...) and leave shell_output working but emit a deprecation warning at render time. Remove shell_output after one release cycle.

We pick option 1. The library is pre-1.0 and the existing shell_output is the source of incremental-build bugs we’re fixing — making it loud is the point. Per feedback_no_backcompat in our memory, on-disk and config back-compat aren’t a concern at this stage.

Lower-effort fallback (not chosen)

Track the mtime of each src_dir as an automatic input for every product whose processor scans that directory. Catches add/remove and most rename cases, zero user-facing API change. Doesn’t catch content-dependent queries like grep -r '...'. Roughly 30 lines of code, no template-engine changes. Could ship as a stop-gap if the full design proves too disruptive.

We chose against it because:

mtime is unreliable on some filesystems and tar/git restores set it arbitrarily.
It would silently miss the grep -r case from the motivating example.
It papers over the real bug (shell_output is unsafe) without forcing users to think about what their templates actually depend on.

Implementation

Status: shipped. Implemented as two pieces:

Static analysis (the cache-key path) — src/analyzers/tera.rs reads each template file as text and regex-matches:

{% include "..." %}, {% import "..." %}, {% extends "..." %} (existing)
load_lua/load_data/...(path="...") (existing)
glob(pattern="...") (new)
shell_output(command="...", depends_on=[...]) (new)

For each match, the analyzer:

Resolves any glob patterns via the glob crate.
Adds matched files to the product’s inputs (so file-content changes are caught by the existing input-checksum mechanism).
Mixes a deterministic string into the product’s config_hash via Product::extend_config_hash(). The string is built from the literal pattern plus the sorted list of resolved paths plus, for shell_output, the literal command text. Adding/removing/renaming a matching file flips the path list; editing the command text flips the command piece.

The analyzer rejects shell_output(...) without depends_on at graph-construction time. The error message includes the source file path, the offending command, and migration advice.

A new helper analyzers::analyze_with_full_scanner was added to support analyzers whose scanner returns a [ScanResult] (deps + config-hash piece) rather than just deps. The existing C/C++/Python/markdown analyzers continued to use the simpler analyze_with_scanner. (Both helpers have since been folded into analyzers::Analysis: every analyzer returns a ScanResult, and one that contributes hash pieces sets always_rescan.)

Runtime evaluation (the rendered-output path) — src/processor/generator/tera.rs registers two functions on the Tera engine:

glob(pattern="...") — runs the same expansion the analyzer did and returns the sorted list as a Tera string array. Templates can use length, iterate, or join it.
shell_output(command="...", depends_on=[...]) — runs the command and returns its trimmed stdout. Validates that depends_on is present and is an array. Empty list is permitted (explicit user opt-out of file-level dependency tracking).

The runtime validation in shell_output is a defense-in-depth check: the analyzer enforces depends_on at graph-construction time, but if for any reason a template were rendered without going through the analyzer (e.g., during early prototyping or a misconfigured project) the render itself would still surface the missing argument with a clear error.

Caching note. The path list returned by scan_template is not cached in .rsconstruct/deps.redb. Tera analysis is cheap (regex + glob + path hashing) and the config-hash piece needs to be recomputed on every build anyway — its value depends on filesystem state, not on the template file’s content. A richer cache could be added later if profiling demands it.

Files changed

src/graph.rs — added Product::extend_config_hash().
src/analyzers/mod.rs — added ScanResult and analyze_with_full_scanner.
src/analyzers/tera.rs — replaced scan_includes with scan_template, which handles all five constructs; added expand_glob helper.
src/processor/generator/tera.rs — added GlobFunction; tightened ShellOutputFunction to require depends_on.
tests/processor/tera.rs — 8 new tests covering count, add/remove/rename invalidation, missing depends_on rejection, content-change invalidation, command-edit invalidation, and empty-glob handling.

Known limitations

The shell_output(...) body regex stops at the first ). Commands or argument values that contain a literal ) could confuse the analyzer. None of the existing teaching-slides commands hit this, but a quote-and-paren-aware parser would be a small follow-up if the limitation becomes painful.
Globs are evaluated with the glob crate’s syntax (POSIX-ish with ** for recursive) and do not currently honor .gitignore / .rsconstructignore. For most users this is fine because the patterns are scoped to a known directory. If unwanted matches appear, exclude them with a more specific pattern.

Open questions

What about git_count_files? The existing function uses git ls-files, which is essentially a glob restricted to git-tracked files. We can keep it for backwards compatibility and have it record the same kind of dependency under the hood (the resolved file list from git ls-files), or we can just deprecate it in favor of glob(). The former is friendlier; do the former.
Caching the glob expansion itself. FileIndex is already cached per-build. For repeated glob() calls inside one render the resolution is cheap. Across builds, the analyzer cache keys deps by source file, so if the .tera file hasn’t changed and the source tree hasn’t changed, the cached dep list is reused. No extra caching needed.
What if glob() matches zero files? Return an empty list. Don’t error — empty results are a legitimate state, and an error would force awkward if-glob-empty-or template logic.

Keyboard shortcuts

RSConstruct - Rust Build Tool