Architecture

This page describes RSConstruct’s internal design for contributors and those interested in how the tool works.

Core concepts

Processors

Processors implement the Processor trait (src/processor/mod.rs). Each processor:

1. Auto-detects whether it is relevant for the current project
2. Scans the project for source files matching its conventions
3. Creates products describing what to build
4. Executes the build for each product

Run rsconstruct processor list to see all available processors and their auto-detection results.

Auto-detection

Every processor implements auto_detect(), which returns true if the processor appears relevant for the current project based on filesystem heuristics. This allows RSConstruct to guess which processors a project needs without requiring manual configuration.

Only two Processor methods are required — scan_config() and execute(ctx, product). Everything else has a default implementation driven by the processor’s StandardConfig. The main methods:

Batch execution (execute_batch) and tool declaration (required_tools, tool_version_commands) round out the trait.

Both auto_detect and discover receive a &FileIndex — a pre-built index of all non-ignored files in the project (see File indexing below).

Detection heuristics are per-processor filesystem checks (e.g. ruff detects when the project contains .py files, cargo when it contains Cargo.toml). The authoritative, always-current list is rsconstruct processor list, which shows every processor together with its auto-detection result for the current project.

Products

A product represents a single build unit with:

• Inputs — source files that the product depends on
• Outputs — files that the product generates
• Output directory (optional) — for creators, the directory whose entire contents are cached and restored as a unit

BuildGraph

The BuildGraph manages dependencies between products. It performs a topological sort to determine the correct build order, ensuring that dependencies are built before the products that depend on them.

Executor

The executor runs products in dependency order. It supports:

• Sequential execution (default)
• Parallel execution of independent products (with -j flag)
• Dry-run mode (show what would be built)
• Keep-going mode (continue after errors)
• Batch execution (group multiple products into one tool invocation)

Incremental rebuild after partial failure

Each product is cached independently after successful execution. If a build is interrupted or fails partway through, the next run only rebuilds products that don’t have valid cache entries:

• Non-batch mode: Each product executes and is cached individually. If the build stops after 400 of 800 products, the next run skips the 400 cached successes and rebuilds the remaining 400. Products go down this path when batching is disabled (--batch-size -1), when the processor doesn’t support batching, or when fewer than two of its products need rebuilding.

• Batch mode with external tools (the default for batch-capable processors): The external tool receives all files in the batch in one invocation — with no --batch-size limit set, that is every rebuilding product of the processor in the level. If the tool exits with an error, all products in that batch are marked failed — there is no way to determine which outputs are valid from a single exit code. On the next run, all products from the failed batch are rebuilt. Use --batch-size N to bound the blast radius, or --batch-size -1 for per-product execution and caching.

• Batch mode with internal processors (e.g., imarkdown2html, isass, ipdfunite): These process files sequentially in-process and return per-file results, so partial failure is handled correctly even in batch mode — only the failed products are rebuilt.

Interrupt handling

External subprocess execution goes through the runner family in src/processor/mod.rs — run_command(), run_command_capture(), run_command_with_timeout(), run_command_with_stdin() — which share one inner implementation. It spawns children via tokio::process::Command with kill_on_drop(true) and then awaits a biased tokio::select! racing the child’s exit against a tokio::sync::watch interrupt receiver obtained from BuildContext::interrupt_receiver() (plus an optional timeout arm).

Routing through it is also what gives a call site log_command() (--show-child-processes) and, in debug builds, the declared-tools assertion that a processor only runs tools its required_tools() names. Hand-rolled spawns silently opt out of all four, which is why they are held to exactly two exceptions:

• Tool installation (tools::run, the binary installer) inherits the terminal so sudo can prompt for a password and apt/dnf can render progress. Capturing would hang on the prompt.
• Opening a viewer (Builder::open_file) launches a detached process that must outlive rsconstruct — kill_on_drop and waiting are both exactly wrong.

Both are commented as such at the call site. Anything else that spawns a child is a bug.

A command that needs to feed stdin uses run_command_with_stdin(), which pumps the write concurrently with draining stdout/stderr. Writing all of stdin first deadlocks as soon as the child fills its output pipe — this is why the aspell checker used to carry a hand-rolled spawn with a writer thread.

The Ctrl+C handler — a dedicated thread in main.rs awaiting tokio::signal::ctrl_c() — calls BuildContext::interrupt(), which sets an AtomicBool and broadcasts on the watch channel. Every waiting subprocess wakes immediately (there is no polling interval) and its child is killed on drop. The executor also consults is_interrupted() between products and levels to stop scheduling new work. A second Ctrl+C force-exits with status 130.

File indexing

RSConstruct walks the project tree once at startup and builds a FileIndex — a sorted list of all non-ignored files. The walk is performed by the ignore crate (ignore::WalkBuilder), which natively handles:

• .gitignore — standard git ignore rules, including nested .gitignore files and negation patterns
• .rsconstructignore — project-specific ignore patterns using the same glob syntax as .gitignore

Two kinds of directory are never walked, regardless of ignore files:

• .rsconstruct/ — the tool’s own state directory (cache descriptors, objects, databases).
• Configured output roots — the global [build] output_dir plus every instance’s output_dir/output/output_dirs values (Config::file_index_walk_dirs). Generated files must not be discovered as project source: otherwise build 1 generates a file and build 2 lints it, and fresh vs. incremental builds discover different graphs. Downstream processors still see upstream outputs — as virtual files injected by declaration during the fixed-point discovery loop, which works identically whether or not the files exist on disk.

The opt-in: naming a directory under an output root in a processor’s src_dirs (e.g. src_dirs = ["out/processor.generator.generic"]) force-walks it unconditionally — ignoring gitignore and the output-root exclusion. That is the deliberate way to scan generated files. output_dir = "" (in-place output at the project root) cannot be excluded; in-place outputs remain indexed.

Processors never walk the filesystem themselves. Instead, auto_detect and discover receive a &FileIndex and query it with their scan configuration (src_extensions, exclude directories, exclude files). This replaces the previous design where each processor performed its own recursive walk.

Build pipeline

This is the core algorithm — every rsconstruct build follows these phases in order. Use --phases to see timing for each phase.

Phase 1: File indexing

The project tree is walked once to build the FileIndex — a sorted list of all non-ignored files. This is the only filesystem walk; all subsequent file lookups go through the index. See File indexing below.

Phase 2: Discovery (fixed-point loop)

Each enabled processor queries the file index and adds products to the BuildGraph. Discovery runs in a fixed-point loop to handle cross-processor dependencies:

file_index = walk filesystem
loop (max 10 passes):
    for each processor:
        processor.discover(graph, file_index)
    if no new products were added → break
    collect outputs from new products
    inject them as virtual files into file_index
    if no new virtual files were injected → break

On each pass, processors may re-declare existing products (silently deduplicated) or discover new products whose inputs are virtual files from upstream generators. The loop converges when a full pass adds nothing new. Most projects converge in 1 pass; projects with generator → checker/generator chains converge in 2.

See Cross-Processor Dependencies for details on deduplication and the virtual file mechanism.

Phase 3: Dependency analysis

Dependency analyzers (e.g., the C/C++ header scanner) run against the graph to add additional input edges. For example, if main.c includes util.h, the analyzer adds util.h as an input to the main.c product. Results are cached in deps.redb for incremental builds.

Phase 4: Tool version hashing

For each processor with a tool lock entry (rsconstruct tool lock), the locked tool version hash is appended to the product’s config hash. This ensures that upgrading a tool (e.g., ruff 0.4 → 0.5) triggers rebuilds even if source files haven’t changed.

Tool detection

The TOOLS registry in src/tools.rs lists every external tool rsconstruct knows about. Each entry’s name is the detection key: it is passed directly to which::which by builder/tools.rs and tool_lock.rs, so it decides whether rsconstruct tool list reports installed or missing, and which binary rsconstruct tool lock records a version for.

Registry names must be bare binary names, never paths. which switches behaviour on the presence of a path separator:

A path-shaped entry therefore resolves only when rsconstruct is invoked from the one directory that has that subtree beneath it, and reports missing everywhere else — including for a user who has the tool installed and on $PATH at, say, ~/install/gems/bin/mdl. The registry once carried gems/bin/mdl and node_modules/.bin/markdownlint alongside correct bare mdl / markdownlint entries; both path forms were removed.

In both modes which also requires the executable bit — a file that exists but is not executable reads as missing.

To pin a project-local binary, set the per-processor command config field (src/config/processor_configs.rs), which is the value actually executed:

[markdownlint]
command = "node_modules/.bin/markdownlint"

The registry name stays bare so detection works from any working directory; command is where a vendored path belongs.

Phase 5: Dependency resolution

resolve_dependencies() scans the graph for products whose inputs match other products’ outputs. When found, it creates a dependency edge — the producer must complete before the consumer can start. This is how cross-processor ordering works automatically (e.g., pandoc runs before the explicit site generator because pandoc’s HTML outputs are the site generator’s inputs).

After resolution, the graph is topologically sorted to produce the execution order.

Phase 6: Classify

Each product is classified as one of:

• Skip (up-to-date) — input checksum matches the cache entry and all outputs exist on disk. No work needed.
• Restore — input checksum matches a cache entry but outputs are missing (e.g., after rsconstruct clean). Outputs are restored from cache via hardlink or copy.
• Build (stale) — input checksum doesn’t match any cache entry. The product must be rebuilt.

Input checksums are computed by hashing all input files (SHA-256). The mtime pre-check (mtime_check = true, default) skips rehashing files whose mtime hasn’t changed since the last build.

Phase 7: Execute

Products are executed in topological order, respecting dependency edges. Independent products at the same dependency level run in parallel (controlled by -j / RSCONSTRUCT_THREADS). Batch-capable processors group their products into a single tool invocation.

Batch chunk sizing: By default a batch group is sent as one chunk — all of a batch-capable processor’s rebuilding products that became ready together go to the tool in a single invocation. With --batch-size N, chunks are limited to N products. --batch-size -1 disables batching entirely, giving per-product execution and caching — the best incremental recovery after partial failure, at the cost of one tool invocation per file. In fail-fast mode (no --keep-going), a failing chunk stops later chunks from being dispatched.

Held checkers: products become ready as their own dependencies finish, so a batching checker whose files partly depend on generated files would form several batches, one per wave, running at the same time. That breaks tools that assume one process over the whole set: pytest given only __init__.py files collects no tests and exits 5, and two mypy processes sharing .mypy_cache crash with INTERNAL ERROR. The ReadyTracker therefore holds a batching checker’s ready products back until none of its products still waits on a dependency, and releases them all in one wave. Only checkers are held: nothing depends on a checker’s products, so the hold delays nothing else, while a generator’s products may feed one another.

For each product:

1. Compute input checksum (if not already done in classify)
2. Check cache — skip or restore if possible
3. Execute the processor’s command
4. On success: store outputs in the cache (content-addressed under .rsconstruct/objects/)
5. On failure: report error (or continue if --keep-going)

Processor source layout

All processor code lives under src/processor/. The folder structure mirrors processor type:

src/processor/
├── mod.rs          # Processor trait, shared helpers (run_command, run_checker,
│                   # SimpleChecker, SimpleGenerator, ProcessorBase, …)
├── checker/        # One file per checker (ruff.rs, pylint.rs, cppcheck.rs, …)
│   └── mod.rs      # mod declarations
├── generator/      # One file per generator (generator.rs, marp.rs, sass.rs, tags.rs, …)
│   └── mod.rs      # Shared helpers: find_templates, output_path, discover_single_format, …
├── creator/        # One file per creator (cargo.rs, cc.rs, gem.rs, jekyll.rs,
│   │               # mdbook.rs, npm.rs, pip.rs)
│   ├── mod.rs      # mod declarations
│   └── creator.rs  # Generic creator processor
├── explicit/       # Explicit processor (user-defined command with declared outputs)
│   ├── mod.rs
│   └── explicit.rs
└── mass_generator/ # Mass generator (tool that enumerates its outputs via predict_command)
    ├── mod.rs
    └── mass_generator.rs

Category directories are singular (checker/, not checkers/), like every other module in the crate; a category that holds a processor named after itself (generator::generator, creator::creator, explicit::explicit, mass_generator::mass_generator) carries a #[allow(clippy::module_inception)] on that one mod line.

Conventions

• Every file in src/processor/ is a real single processor — no utility-only files anywhere in the tree. Shared helpers live in mod.rs or generator/mod.rs; processor-specific data tables (e.g. the requirements generator’s stdlib list) live in the processor’s own file. The tool registry (src/tools.rs) and build statistics (src/stats.rs) live at the crate root for this reason.
• Checkers use SimpleChecker (data-driven, no boilerplate) or implement Processor directly for checkers with custom discovery logic (e.g., clippy, script).
• Generators use SimpleGenerator (data-driven with a custom execute_fn) or GeneratorProcessor for the generic pass-through generator.
• Creators use CreatorProcessor for the generic case, or their own struct for creators with special discovery (cargo profiles, npm siblings, etc.).
• Explicit is a singleton processor type with its own folder because it is neither a checker nor a generator.
• Lua is the only processor type that hosts external scripts rather than wrapping a fixed external tool. It has its own folder because it carries significant runtime state (the Lua VM).
• A processor is one file. Everything that defines a processor lives in its file: the Processor impl (or SimpleChecker/SimpleGenerator params), its config struct, and one inventory::submit! plugin entry carrying the whole schema — fields (a FieldSpec list from which known/checksum/must fields, descriptions, and expected types all derive), scan_defaults, defaults, and omit_standard_fields. Tools missing from the central registry are submitted as ToolInfo from the processor file. There are no central per-processor tables: the former 92/71/~120-arm match tables in config/mod.rs are deleted, and every default value is encoded exactly once. The remaining out-of-file touch-points (the category mod declaration, the docs page, the test file) are enforced by completeness tests in config/tests.rs.

Determinism

Build order is deterministic:

• File discovery is sorted
• Processor iteration order is sorted
• Topological sort produces a stable ordering

This ensures that the same project always builds in the same order, regardless of filesystem ordering.

Caching

See Cache System for full details on cache keys, storage format, rebuild classification, and per-processor caching behavior.

Subprocess execution

RSConstruct uses two internal functions to run external commands:

• run_command() — by default captures stdout/stderr via OS pipes and only prints output on failure (quiet mode). Use --show-output flag to show all tool output. Use for compilers, linters, and any command where errors should be shown.

• run_command_capture() — always captures stdout/stderr via pipes. Use only when you need to parse the output (dependency analysis, version checks, Python config loading). Returns the output for processing.

Parallel safety

When running with -j, each thread spawns its own subprocess. Each subprocess gets its own OS-level pipes for stdout/stderr, so there is no interleaving of output between concurrent tools. On failure, the captured output for that specific tool is printed atomically. This design requires no shared buffers or cross-thread output coordination.

Path handling

All paths are relative to project root. RSConstruct assumes it is run from the project root directory (where rsconstruct.toml lives).

Internal paths (always relative)

• Product.inputs and Product.outputs — stored as relative paths
• FileIndex — returns relative paths from scan() and query()
• Cache keys (Product::descriptor_key()) — contain no machine-specific paths (the key hashes processor identity, config, and input content), enabling cache sharing across different checkout locations
• Cache entries (CacheEntry.outputs[].path) — stored as relative paths

Processor execution

• Processors pass relative paths directly to external tools
• Processors set cmd.current_dir(project_root) to ensure tools resolve paths correctly
• fs::read(), fs::write(), etc. work directly with relative paths since cwd is project root

Exception: Processors requiring absolute paths

If a processor absolutely must use absolute paths (e.g., for a tool that doesn’t respect current directory), it should:

1. Store the project_root in the processor struct
2. Join paths with project_root only at execution time
3. Never store absolute paths in Product.inputs or Product.outputs

Why relative paths?

• Cache portability — cache keys don’t include machine-specific absolute paths
• Remote cache sharing — same project checked out to different paths can share cache
• Simpler code — no need to strip prefixes for display or storage