Mass Generator Processor

Runs a tool that enumerates its outputs in advance. The tool’s plan command prints a manifest; rsconstruct turns every entry into a product and runs the tool’s build command at most once per build. The design rationale is in Output Prediction.

Why “mass generator”?

Existing processor types cover a matrix of “how many outputs” and “are they known in advance”:

A mass generator is the transparent creator: it produces many output files (like a creator), but the tool itself answers “what will you produce?” before running. Each predicted file becomes a declared product with its own inputs, cache entry, and dependency edges.

Names considered:

• mass_generator — chosen. Says what it does: “generator” (per-file outputs like the generator type), “mass” (many products from one tool invocation).
• transparent_creator — accurate but awkward.
• predicting_creator — describes the mechanism, not the result.
• site_generator — too narrow; the type is useful beyond static sites.

Purpose

Wraps a tool that:

1. Produces many output files from a set of source files (a static site generator, say).
2. Can enumerate its outputs in advance via a separate plan command.
3. Builds all its outputs in a single invocation.

Once wired as a mass generator, the tool gets a cache entry per file, shares its output directory safely with other processors, and lets downstream processors depend on individual outputs.

Configuration

[processor.mass_generator.generic.site]
command         = "rssite"
args            = ["build"]
predict_command = "rssite"
predict_args    = ["plan"]
output_dirs     = ["_site"]
# Files the plan is computed from; while none changes, the plan is reused.
src_dirs        = ["docs", "templates"]
src_extensions  = [".md", ".html"]
src_files       = ["mysite.toml"]
# loose_manifest = true     # report plan/build mismatches as warnings; default false
# dep_inputs = ["mysite.toml"]

Commands are executed directly, never through a shell (see No-Shell Policy), so a command and its arguments are separate fields: command + args for the build, predict_command + predict_args for the plan. They may be the same binary with different arguments or two separate scripts.

Fields

The scan fields discover no products — the manifest’s sources decide every product’s inputs — but they declare which files the plan is computed from; see Plan cache. src_dirs without src_extensions is rejected, because it would match no file and silently key the cache on nothing.

Plan cache

predict_command runs at every graph build — build, status, graph, clean outputs. When the config declares the plan’s inputs through src_dirs (with src_extensions) and/or src_files, rsconstruct caches the manifest in .rsconstruct/mass_generator/<instance>.json, keyed by the rsconstruct version, the instance’s whole config, and the path and content of every declared input file. A later build whose key matches reuses the cached manifest and does not run predict_command; a new, removed or edited input file — or any config change — runs it again. Without declared inputs, nothing is cached and the plan runs every time.

The declaration is a promise: the plan must be a function of those files alone. A file the plan reads that is not declared (a template directory, a config file) can change the plan without invalidating the cache, so declare everything the tool’s plan mode reads. A cache file that does not parse is an error naming the file; delete it to re-plan.

Manifest format

{
  "version": 1,
  "outputs": [
    {
      "path": "_site/index.html",
      "sources": ["docs/index.md", "templates/default.html", "mysite.toml"]
    },
    {
      "path": "_site/about/index.html",
      "sources": ["docs/about.md", "templates/default.html", "mysite.toml"]
    }
  ]
}

• version — integer. Schema version; only 1 is accepted.
• outputs[].path — plain relative path (no leading /, no . or .. components). Must fall within one of the processor’s output_dirs. Listed once.
• outputs[].sources — the files whose content determines this output. At least one; the tool’s own config file counts. These become the product’s inputs, so keep the list minimal: an output that names every source rebuilds whenever any source changes.

Unknown keys are rejected. rsconstruct sorts entries by path, so the tool’s output order does not matter.

Several outputs may list exactly the same sources — a tag’s index page, its page/1/ redirect and its feed all depend on the same posts. Each is still cached under its own path: the predicted path is part of the product’s cache key, where for other processor types the output path follows from the input and needs no such entry.

How it works

Plan phase (at graph-build time)

rsconstruct runs predict_command once per instance and parses its stdout. A non-zero exit, malformed JSON, an unsupported version, or a path outside output_dirs fails the graph build with the tool’s own message. For each manifest entry a product is added:

• inputs = the entry’s sources (plus dep_inputs)
• outputs = [entry.path]
• processor = this instance

Predicted outputs join the file index as virtual files, so a downstream processor can discover them even before they exist on disk.

Build phase

Products are classified individually, like any generator’s: unchanged inputs skip, a known input checksum restores from the blob cache, anything else builds. The first product that needs building runs command; every later one in the same build finds the tool already run and does nothing but cache its file. If the tool fails, every remaining product of the instance fails with the same message rather than re-running a build that just failed.

Before the tool runs, rsconstruct unlinks every predicted path, including the ones whose products are clean. A clean file may be a read-only hardlink left by an earlier cache restore, which the tool could not overwrite. The tool then regenerates everything; the clean products keep their cache entries, so the next build matches them again as long as the tool is deterministic (same inputs, same bytes).

Verification

After the tool exits, rsconstruct compares what it wrote against the plan:

• missing — a predicted path that does not exist afterwards.
• unexpected — a file under output_dirs that is new or was rewritten during the run, is not in the plan, and is not another product’s declared output.

Either is a build error. Files the tool left untouched are not its business and are not reported: a neighbor processor’s output, a CNAME committed into _site/, or a page from an older plan that nobody cleans up. A file another processor declares (an explicit writing _site/about.html, say) is never reported even if the tool rewrote it — that is the shared-directory contract of Shared Output Directory.

loose_manifest = true turns both findings into a warning on stderr. Even then a product whose own predicted file is absent fails: there is nothing to cache.

Restore phase

When every product of an instance is cache-clean, each is restored from its blob independently and the tool does not run at all. rsconstruct clean outputs followed by rsconstruct build therefore never invokes the tool.

Clean behavior

This processor is a mass generator — rsconstruct clean outputs removes each predicted output file individually with no directory recursion, exactly as for a generator. After all per-product cleans complete, the orchestrator removes any parent directories that are now empty. Pass --no-empty-dirs to keep them. See Clean behavior and rsconstruct clean.

A page the tool no longer predicts has no product and is neither rebuilt nor cleaned; delete it by hand.

Cross-processor dependencies

Every output file is a declared product, so downstream processors wire up through the normal graph:

[processor.mass_generator.generic.site]
command         = "rssite"
args            = ["build"]
predict_command = "rssite"
predict_args    = ["plan"]
output_dirs     = ["_site"]

[processor.explicit.generic.sitemap]
command      = "./sitemap.sh"
inputs       = ["_site/index.html", "_site/about/index.html"]
output_files = ["_site/sitemap.xml"]

The sitemap runs after the site is built and rebuilds when either page changes. No ordering knobs are involved; the topological sort handles it.

Tool author contract

For a tool to work as a mass generator, its plan command must uphold these invariants:

1. Pure function of config + source tree. Same inputs → same manifest, bit for bit. No network, no timestamps, no environment peeking (unless the file read is declared as a source).
2. Cheap, or cacheable. rsconstruct runs it at every graph build — build, status, clean outputs, graph — unless the config declares its inputs, in which case it runs only when one of them changed (see Plan cache).
3. Exact match with the build. Predicted paths equal the paths command writes, no more and no fewer. Violations are errors.
4. Deterministic build. Same inputs → same bytes, or clean products will mismatch their cache after every rebuild of a neighbor.
5. Deterministic variable outputs. Content-derived pages (tag indexes, archives, feeds) must be enumerable from the same parsing pass that plan does.

Both modes should be driven by the same internal function that enumerates outputs — otherwise the plan and the build drift apart, and verification turns every build red.

See rssite for a tool built to this contract.

Comparison with other processor types

Migration from a creator

If a tool exists first as a creator and later grows a plan mode, the migration is config-only:

# Before
[processor.creator.generic.mysite]
command        = "mysite"
args           = ["build"]
src_extensions = ["mysite.toml"]
src_dirs       = ["."]
output_dirs    = ["_site"]

# After
[processor.mass_generator.generic.mysite]
command         = "mysite"
args            = ["build"]
predict_command = "mysite"
predict_args    = ["plan"]
output_dirs     = ["_site"]

The anchor-file scan goes away: the manifest names the product inputs (declare the plan’s own inputs with the scan fields if you want the plan cache). Downstream processors start getting per-file dependencies without changes on their side.

See also

• Output Prediction — design rationale, invariants, and what the implementation left for later
• Shared Output Directory — the ownership rules for opaque creators, which mass generators inherit
• Processor Ordering — sibling discussion about explicit ordering knobs
• rssite — a static site generator built to the manifest contract
• zola plan patch — adds a plan mode to zola v0.23.3 so a Zola site can run as a mass generator; verified on an 800-page site