Cache System

RSConstruct uses a content-addressed cache to enable fast incremental builds. This page describes the cache architecture, storage format, and rebuild logic.

Overview

The cache lives in .rsconstruct/ and consists of:

• objects/ — content-addressed blob store, sharded by the first two hex characters of the content hash
• descriptors/ — cache descriptors (marker/blob/tree JSON), sharded the same way but addressed by the descriptor key rather than by content
• db.redb — redb database holding the processor_configs table used for config-change diffing and the product_last_tree table used to unlink stale outputs (see Stale outputs before a rebuild)
• deps.redb — source file dependency cache (see Dependency Caching)
• mtime.redb — mtime→checksum pre-check cache that lets unchanged files skip re-hashing
• webcache.redb — HTTP request cache for processors that fetch remote resources

Data model

The object store contains three kinds of objects, inspired by git:

Blobs

A blob is a file’s raw content, addressed by its SHA-256 content hash. Blobs are optionally zstd-compressed and made read-only to prevent corruption when restored via hardlinks.

Blobs are stored content-addressed — two products producing identical output share the same blob. This enables deduplication and hardlink-based restoration.

Why blobs don’t store output paths

A blob is pure content — it has no knowledge of where it will be restored. This is critical for two reasons:

1. Rename survival. If you rename foo.md to bar.md without changing its content, the cache key (which is content-addressed) is the same. The blob is reused and restored to the new output path (bar.txt instead of foo.txt). If the blob stored its output path, this wouldn’t work.

2. Deduplication across trees. Multiple tree entries can point to the same blob under different paths. For example, if two files in a creator’s output have identical content, they share the same blob object in the store. The tree records the path; the blob just holds the content.

Trees

A tree is a serialized list of (path, mode, blob_checksum) entries describing a set of output files. Trees are stored as descriptors under descriptors/, addressed by the descriptor key (not by content hash). A tree maps relative file paths to content-addressed blobs. Multiple trees can point to the same blobs — deduplication happens at the blob level.

Markers

A marker is a minimal descriptor indicating that a check passed. Markers are stored under descriptors/, addressed by the descriptor key, and reference no blobs.

Cache entries

A cache entry is a small JSON descriptor stored under .rsconstruct/descriptors/ at the path derived from the descriptor key. It contains:

{"type": "blob", "checksum": "abc123...", "mode": 493}

Note: the blob descriptor has no path — the product knows where its output goes.

Or:

{"type": "tree", "entries": [{"path": "dir/file.txt", "checksum": "def456...", "mode": 493}]}

or:

{"type": "marker"}

The actual file content lives in separate content-addressed blob objects. The cache entry is just a pointer (for generators) or a manifest (for creators).

On-disk layout

.rsconstruct/objects/
  a1/b2c3d4...    # blob (raw file content), addressed by content hash
  cd/ef5678...    # another blob (".zst" suffix when compressed)
.rsconstruct/descriptors/
  ff/0011aa...    # descriptor (JSON), addressed by descriptor key

Descriptors and blobs live in separate directories with the same two-hex-char sharding. Keeping them separate is what lets cache trim delete unreferenced blobs by scanning the descriptors.

Cache keys

The descriptor key identifies a product build. It is computed as (CacheKey::descriptor_key in src/cache_key.rs):

hash(processor_name, processor_version, cache_key_digest, input_content_hash)

Where:

• processor_name — the processor type (e.g., pandoc, ruff)
• processor_version — the processor’s cache version from the plugin registry; bumping it invalidates every entry the processor ever produced (see Processor versioning)
• cache_key_digest — a digest of every piece of non-input state that affects the output (below)
• input_content_hash — combined SHA-256 hash of all input file contents

The key is content-addressed: it depends on what the inputs contain, not what they’re named. Renaming a file without changing its content produces the same cache key.

Cache key components

CacheKey is the single owner of key composition. Key material used to be assembled in five uncoordinated places, all splicing into one opaque config_hash: Option<String> with three different separator conventions — so a component that silently failed to be mixed in was invisible, and a changed key could not be attributed to a cause. Contributors now append a named, typed component and one function folds them into the digest:

The component kind is part of the digest, so a config value and a tool value that happen to be equal do not collide. Order is significant and deterministic (discovery, then analyzers, then tool versions).

rsconstruct product show <path> prints the components individually, so any descriptor-key change is attributable to exactly one of them (or to input_checksum).

Multi-format processors

For processors that produce multiple output formats from the same input (e.g., pandoc producing PDF, HTML, and DOCX), each format is a separate product with a separate cache key. The output format is mixed into the key as the variant component, so each format gets its own key naturally.

Output depends on input name

Most processors produce output that depends only on input content, so the descriptor key hashes input content, not input path. A rename with identical content is a cache hit by design: the blob is path-free and is restored to the product’s current output path (see the cache_survives_input_rename test), and two identical files at different paths share one entry.

That is wrong for a processor whose result depends on the path: one that embeds the input filename in its output (a // Generated from foo.c header) would restore output still naming the old file, and a checker with per-path rules (ruff’s per-file-ignores) could reuse another path’s verdict. Such a processor sets output_depends_on_input_name = true, a universal field like enabled. ruff, pylint and mypy default it to true through their plugin’s ProcessorDefaults (their verdicts depend on module names, per-file rules and the presence of __init__.py); a user’s explicit value wins. The builder reads it from the declared config after discovery and marks the instance’s products (BuildGraph::key_by_input_paths); Product::descriptor_key then hashes the input paths together with the input checksum. The paths are taken from the product’s current inputs at key time, so inputs an analyzer adds mid-build are covered too. Turning the field on yields new keys, so the first build after it rebuilds. Tests: renamed_input_restores_by_default and output_depends_on_input_name_rebuilds_renamed_input (tests/tests_mod/incremental.rs).

Flows

Lookup

1. Compute the cache key from processor name + config + input contents
2. Look up the object at that key in the object store
3. If not found: cache miss, product must be built
4. If found: read the descriptor, act based on type

Cache (after successful build)

Checker:

1. Store a {"type": "marker"} entry at the cache key

Generator (single output):

1. Store the output file content as a content-addressed blob
2. Store a {"type": "blob", "checksum": "..."} entry at the cache key

Creator (multiple outputs):

1. Walk all output directories and files
2. Store each file as a content-addressed blob
3. Build the tree entries: [{"path": "...", "checksum": "...", "mode": ...}, ...]
4. Store a {"type": "tree", "entries": [...]} entry at the cache key

Restore

Checker: Nothing to restore. Cache entry exists = check passed.

Generator:

1. Read the cache entry, get the blob checksum
2. Hardlink or copy the blob to the output path

Creator:

1. Read the cache entry, get the tree entries
2. For each (path, checksum, mode): restore the blob to the path, set permissions

Skip

If the cache entry exists AND all output files are present on disk, no work is needed.

Rebuild classification

Because the cache key incorporates input content, a changed input produces a different key. There’s no “stale entry” — either the key exists or it doesn’t.

Stale outputs before a rebuild

Cache objects are read-only and a hardlink restore shares their inode, so a restored output cannot be overwritten in place by the tool that regenerates it. Before a product is built or restored, the files of its previous tree are unlinked. That tree cannot be found through the current key (the input changed, so the key did too): the product_last_tree table in db.redb maps a product’s owner key — processor instance, primary input and output paths, nothing that changes with content or config — to the descriptor it last built or restored. Outputs left by an rsconstruct older than that table are recognised by the restore’s own signature instead: read-only files with a link count above one.

Config-aware caching

Processor configuration is hashed into cache keys. Changing a config value triggers rebuilds even if source files haven’t changed.

Cache restoration methods

Hardlinks work because blob objects contain raw file content (not wrapped in a descriptor). Only cache entries (which point to blobs) contain JSON metadata.

Cache commands

Clean vs Clear

rsconstruct clean removes build outputs but preserves the cache:

• Generators: Output files deleted. Next build restores via hardlink/copy.
• Checkers: Nothing to delete. Next build skips.
• Creators: Output directories deleted. Next build restores from tree.

rsconstruct cache clear wipes everything — descriptors and blobs. A cleared cache means “forget everything, rebuild from scratch.” The entire .rsconstruct/ directory is removed, which also takes the auxiliary caches (mtime.redb, deps.redb, webcache.redb) with it. If only blobs were cleared but descriptors survived, the cache would think outputs are available but fail to restore them. Clearing both together avoids this inconsistency.

Incremental rebuild after partial failure

Each product is cached independently after successful execution. If a build fails partway through, the next run only rebuilds products without valid cache entries.

Remote caching

See Remote Caching for sharing cache between machines and CI.