feature: implement Relink, Don't Build (RDR) #151265
Draft
+4,444
−221
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Part of the RDR (Relink, Don't Rebuild) work to separate span data from .rmeta files. This commit converts Span fields to SpanRef in two areas: 1. ExpnData (hygiene.rs): Convert call_site and def_site from Span to SpanRef. The constructors convert incoming Spans internally, keeping the API stable. 2. rmeta tables (mod.rs, encoder.rs, decoder.rs, cstore_impl.rs): - Predicate tables: (Clause/PolyTraitRef, Span) → SpanRef - Direct span tables: def_span, def_ident_span, proc_macro_quoted_spans - Type-span table: assumed_wf_types_for_rpitit - Added ProcessQueryValue impls and macro variants for SpanRef decoding
Collaborator
|
cc @davidtwco, @TaKO8Ki Some changes occurred in compiler/rustc_codegen_cranelift cc @bjorn3 |
rustbot
added
A-query-system
Collaborator
|
r? @nnethercote rustbot has assigned @nnethercote. Use |
rustbot
added
S-waiting-on-author
davidbarsky
changed the title
This comment has been minimized.
This comment has been minimized.
bjorn3
reviewed
Jan 17, 2026
| /// | ||
| /// The `spans_hash` parameter stores a hash of the spans at the time this work product was | ||
| /// created. For metadata work products compiled with `-Z separate-spans`, this hash is used | ||
| /// to determine whether diagnostics should be replayed when reusing the metadata. |
Member
There was a problem hiding this comment.
Why specifically metadata work products? Codegen can also result in diagnostics. And why does reusing the metadata have any effect on diagnostics?
Contributor
Author
There was a problem hiding this comment.
Sorry, I accidentally published this as a pull request. this is not yet ready for review!
Add IdentRef type that stores identifiers with SpanRef instead of Span, preserving SyntaxContext for hygiene while avoiding absolute byte positions in metadata. Changes: - rustc_span/symbol.rs: New IdentRef struct with name/span fields, bidirectional conversion methods (to_ident_ref/ident), and proper PartialEq/Hash using SyntaxContext - rustc_span/lib.rs: Re-export IdentRef - rmeta: Change fn_arg_idents table from LazyArray<Option<Ident>> to LazyArray<Option<IdentRef>>, with encoding/decoding support
Update diagnostic types to serialize Span fields as SpanRef to avoid storing absolute byte positions in metadata. Unlike other span-containing types which store SpanRef internally, diagnostic types keep Span internally for performance (diagnostics are created/modified frequently; serialization is rare). Custom Encodable/Decodable impls convert at serialization boundaries: - MultiSpan: primary_spans and span_labels - SubstitutionPart: span field - TrimmedSubstitutionPart: original_span and span - DiagInner: sort_span field
When compiling with -Zseparate-spans, the source_map is now written to a separate .spans file instead of being embedded in .rmeta. This allows .rmeta to remain stable when only span positions change (e.g., adding whitespace or comments), enabling Relink, Don't Rebuild (RDR). Changes: - SpanFileRoot: new struct with source_map field for .spans files - CrateRoot: conditionally excludes source_map when separate_spans enabled - Encoder: writes source_map to .spans file with required_source_files - Decoder: loads source_map from .spans file when present - locator.rs: get_span_metadata_section() to load .spans files - CrateMetadata: stores span_blob and span_source_map for decoding
Load .spans files on-demand during span resolution instead of eagerly when crates are loaded. This avoids I/O overhead for crates where spans are never actually resolved. Changes: - CrateMetadata: stores span_file_path instead of loaded blob - SpanFileData: new struct for lazily-loaded span blob and source_map - span_file_data: OnceLock for lazy initialization - CrateMetadata::span_file_data(): loads span file on first access - creader.rs: only checks if span file exists, does not load it
Handle missing .spans files gracefully by returning dummy spans with preserved SyntaxContext, allowing diagnostics to proceed without precise location info. Changes: - ImportedSourceFileState enum: Loaded/Unavailable variants to track three states (not yet loaded, loaded, unavailable) - imported_source_file(): returns Option<ImportedSourceFile> - Returns DUMMY_SP with preserved SyntaxContext when source files are unavailable - Caches unavailable state to avoid repeated load attempts
Replace the derived HashStable_Generic on SpanRef with a custom impl that respects hash_spans(). When hash_spans() returns false (e.g., with -Zincremental-ignore-spans), SpanRef contributes nothing to the hash. This is essential for RDR where span-only changes should not trigger downstream rebuilds. The implementation mirrors Span's HashStable behavior: - Early return when !ctx.hash_spans() - When hashing, include discriminant tag and all fields - SyntaxContext is hashed for hygiene correctness Also adds -Zseparate-spans flag and comprehensive incremental tests: - rdr_add_println_private: adding println to private fn - rdr_add_private_fn: adding a private function - rdr_metadata_spans: metadata span stability - rdr_private_span_changes: private span changes - rdr_separate_spans: separate span file handling - rdr_span_hash_cross_crate: cross-crate span hashing
Add the core infrastructure for RDR span resolution across crates: SpanBlob/SpanBlobDecodeContext: Wrapper for span file data with header validation and root decoding, parallel to BlobDecodeContext. CrateMetadata additions: - span_file_data(): lazily loads and caches span file - load_span_file(): loads span file and validates against rmeta hash TyCtxtSpanResolver: Implements SpanResolver trait - resolve_span_ref(): converts SpanRef to Span via source file lookups - span_ref_from_span(): converts Span to SpanRef with file-relative offsets find_and_import_source_file_by_stable_id(): Searches external crates for source files needed to resolve spans. Also includes tests for include_str!/include_bytes! handling with separate spans.
Add safety mechanism to prevent diagnostics from being emitted after codegen begins, when span files may be unavailable for relinking. Changes: - Session::diagnostics_allowed(): tracks whether diagnostics can be emitted - Session::enter_codegen_phase(): marks transition to codegen - Dep graph integration: force diagnostics before codegen starts - Codegen backends: call enter_codegen_phase() at appropriate points - SpanRef resolution: check diagnostics_allowed() before resolving - Work product handling: track span file dependencies Run-make tests: - rdr-diagnostic-gating: verify gating works correctly - rdr-hygiene-hash-collision: test hygiene context handling - rdr-missing-spans-fallback: test graceful degradation
Implement "Relink, Don't Rebuild" (RDR) by modifying the Strict Version Hash (SVH) computation to only include publicly-visible API elements, rather than the entire HIR. Previously, any change to a crate (including private function bodies) would change the SVH, causing all downstream crates to rebuild. Now, the SVH only changes when the public API changes. The new `public_api_hash` query hashes: - Public item signatures (types, bounds, generics, predicates) - Inlinable function bodies (#[inline] and generic functions) - Trait implementations Private function bodies are excluded from the hash, so modifying them no longer triggers downstream rebuilds. Internal incremental compilation continues to work correctly because it uses the query system's fine-grained dependency tracking, which is separate from the SVH.
Add run-make tests to verify RDR works correctly with: 1. Panic locations (rdr-panic-location): - Verifies panic messages show correct file:line:col - Tests public functions, private functions, and format strings - Ensures span resolution works for panic messages 2. Debug info (rdr-debuginfo): - Verifies DWARF debug info contains correct source references - Tests with -Cdebuginfo=2 and -Zseparate-spans - Checks reproducibility of debug line tables 3. Coverage instrumentation (rdr-coverage): - Verifies coverage profiling works with -Zseparate-spans - Tests generation of profraw data - Exercises various control flow patterns (branches, loops)
The RDR changes now correctly hash syntax context even with -Zincremental-ignore-spans. This is a correctness fix: hygiene information must always be considered to avoid false cache hits when macro expansion contexts differ. For change_return_impl_trait, the trait bounds (Clone vs Copy) are semantic differences that should invalidate typeck regardless of span handling settings. The previous test expectation was incorrect - it relied on the old behavior where spans contributed nothing to the hash with -Zincremental-ignore-spans.
davidbarsky
force-pushed
the
push-ukyuqwxxwmxr
branch
from
8be6eb3 to
6836b37
Compare
davidbarsky
force-pushed
the
push-ukyuqwxxwmxr
branch
from
6836b37 to
f445e3d
Compare
Collaborator
|
The job Click to see the possible cause of the failure (guessed by this bot) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hello! I didn't meant to convert this to a PR (let alone a draft PR), but this branch implements "Relink, Don't Rebuild", which was approved this major change proposal. The idea behind RDR is to treat changes to non-public items as not a factor in considering whether dependent crates also need to be rebuild. The trick, of course, if defining what "non-public" items are. This changes considers two classes:
.spansfile. This approach was suggested by David Lattimore in this Zulip post.This feature is currently exposed as "
-Zseparate-spans", but it would probably make sense to add a separate feature flag for the stable version hash changes. I didn't do that before opening this as a draft pull request because, well, I accidentally published this pull request.On my work project, this brought down incremental
cargo checktimes from 4 seconds to:For rust-analyzer, whose compile times have been optimized to a ridiculous degree, RDR reduced
cargo checktimes (after changing a non-public item inhir-ty) from 3 seconds to 1.5 seconds.AI Disclosure: I used Claude Code (Opus 4.5) and Codex (GPT-5.2-codex) to implement this. This is not ready for review; but I've been using this change for the last day or so.