rustdoc: follow-up work for `--emit=toolchain-shared-resources`

[jyn514]

This is a tracking issue for the unstable feature --emit=toolchain-shared-resources, which requires -Z unstable-options. This feature is not meant to ever be stabilized; it's used by docs.rs to avoid copying the same files for each crate. Users other than docs.rs are not supported.

Note

Tracking issues are used to record the overall progress of implementation.
They are also used as hubs connecting to other relevant issues, e.g., bugs or open design questions.

A tracking issue is however not meant for large scale discussion, questions, or bug reports.
Instead, open a dedicated issue for the specific matter or post in the #t-rustdoc Zulip channel.

Discussion comments will get marked as off-topic or deleted.
Repeated discussions on the tracking issue may lead to the tracking issue getting locked.

Unresolved Questions

• Can we get rid of unversioned-shared-resources completely? It's unclear what the difference between a versioned and unversioned resource is; I don't know if rustdoc can promise its files will never change even across toolchains.
• main.js is currently marked as a toolchain resource, which is wrong if the user passes --theme in RUSTDOCFLAGS.

Implementation history

• Initial implementation: rustdoc: Add unstable option to only emit shared/crate-specific files #83478
• Use in docs.rs Upload exactly the shared files needed and give precedence to global files over local ones docs.rs#1340
• Remove --print=unversioned-files: Remove --print unversioned-files from rustdoc  #85770

Why does this feature exist?

Docs.rs has been having quite a lot of trouble with essential files. Here are the recent developments:

1. rustdoc added a new crates.js file a few weeks ago, breaking search for all newly built crates Search within crate is broken: "Uncaught TypeError: crates is undefined" docs.rs#1300
2. dtolnay added crates.js and also some other essential files that were missing Add crates.js and woff2 fonts to required files docs.rs#1301
3. It turned out the other essential files had a typo, breaking all builds Static resources corresponding to 20210305-1.52.0-nightly-caca2121f are missing docs.rs#1305. the typo was fixed by Fix spelling of FiraSans-*.woff2 docs.rs#1308
4. rustdoc wanted to start adding more essential files (Remove theme.js file #82732, Update Source Code Pro and include italics #80705) and I got fed up of dealing with it so I opened Don't hard-code essential rustdoc files docs.rs#1312
5. Don't hard-code essential rustdoc files docs.rs#1312 had a bug (Don't hard-code essential rustdoc files docs.rs#1312 (comment)) if we copy more files than necessary. I opened Give precedence to local shared files over global ones docs.rs#1324 giving precedence to local shared files over global ones which avoids the bug.
6. Give precedence to local shared files over global ones docs.rs#1324 broke docs that weren't using --extern-html-root-url (Docs for releases during 2018-{10-12} appear unstyled docs.rs#1327 (comment)). The tracking issue for rebuilding those is Docs built during 2018-{10-12} link to the wrong url for static resources docs.rs#1181.

In summary, docs.rs can't copy fewer shared files than rustdoc generates, and due to 2 months of broken builds about 3 years ago, it also can't copy more. So there has to be a way to know exactly which files should be shared and which shouldn't.