reduce the amount of panics in `{TokenStream, Literal}::from_str` calls by cyrgani · Pull Request #147859 · rust-lang/rust

cyrgani · 2025-10-18T20:13:31Z

Before this PR, calling TokenStream::from_str or Literal::from_str with an invalid argument would always cause a compile error, even if the TokenStream is not used afterwards at all.
This PR changes this so it returns a LexError instead in some cases.

This is very theoretically a breaking change, but the doc comment on the impl already says

/// NOTE: some errors may cause panics instead of returning `LexError`. We reserve the right to
/// change these errors into `LexError`s later.

Fixes some cases of #58736.

rustbot · 2025-10-18T20:13:34Z

rust-analyzer is developed in its own repository. If possible, consider making this change to rust-lang/rust-analyzer instead.

cc @rust-lang/rust-analyzer

rustbot · 2025-10-18T20:13:36Z

r? @davidtwco

rustbot has assigned @davidtwco.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

cyrgani · 2025-10-18T21:47:52Z

@rustbot author

rustbot · 2025-10-18T21:47:56Z

Reminder, once the PR becomes ready for a review, use @rustbot ready.

cyrgani · 2025-10-19T10:12:43Z

@rustbot ready

tests/ui/proc-macro/invalid-punct-ident-4.rs

cyrgani · 2025-10-23T14:03:39Z

This does not fix #58736 entirely yet:

TokenStream::from_str("0b2");

is still a compile error because it internally calls literal_from_str, which then emits the diagnostic.

davidtwco · 2025-10-25T11:33:55Z

I'm not familiar enough with this part of the compiler

r? compiler

add larger test for `proc_macro` `FromStr` implementations Currently, there are only few tests that check the output of `TokenStream::from_str` and `Literal::from_str` (which is somewhat understandable as the rustc implementation just delegates these calls to the parser). In preparation for both the standalone backend (rust-lang#130856) which will probably need to reimplement this logic as well as for removing panics from these functions (rust-lang#58736), this PR adds a test which shows the various messy ways of how these functions report errors and the return values for successful parses. Followup PRs such as rust-lang#147859 will change more and more of these "diagnostic + error"s into `LexErrors`. The test structure with the extra module is used to allow reusing it later easily for the standalone backend.

joshtriplett · 2026-02-17T16:50:33Z

We reviewed this in today's @rust-lang/libs-api meeting. This change seems reasonable. We'd like to see it run through crater, though.

Separate from that, in a different PR, we would also like to see a change to remove the case that does rustc fixups and returns a valid token stream but sets a compile error in the background, and make that produce an error all the time.

JonathanBrouwer · 2026-02-17T17:30:46Z

@bors try

reduce the amount of panics in `{TokenStream, Literal}::from_str` calls

rust-bors · 2026-02-17T19:41:38Z

☀️ Try build successful (CI)
Build commit: 0269aec (0269aeccce4fd29ce9ad0ebd47a62e584d7e25ef, parent: 3f6250a7bb79e600d9036bbc7c8f65af43933643)

JonathanBrouwer · 2026-02-17T20:24:51Z

@craterbot check

craterbot · 2026-02-17T20:25:16Z

👌 Experiment pr-147859 created and queued.
🤖 Automatically detected try build 0269aec
🔍 You can check out the queue and this experiment's details.

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

craterbot · 2026-02-19T16:16:01Z

🚧 Experiment pr-147859 is now running

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

craterbot · 2026-02-21T11:54:41Z

🎉 Experiment pr-147859 is completed!
📊 1 regressed and 4 fixed (818761 total)
📊 2549 spurious results on the retry-regressed-list.txt, consider a retry¹ if this is a significant amount.
📰 Open the summary report.

⚠️ If you notice any spurious failure please add them to the denylist!
ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

re-run the experiment with crates=https://crater-reports.s3.amazonaws.com/pr-147859/retry-regressed-list.txt ↩

JonathanBrouwer · 2026-02-21T14:09:23Z

@joshtriplett Crater run has only spurious results

petrochenkov · 2026-02-22T08:03:29Z

Great.
@bors r+

rust-bors · 2026-02-22T08:03:32Z

📌 Commit f1c80f5 has been approved by petrochenkov

It is now in the queue for this repository.

JonathanBrouwer · 2026-02-22T08:07:15Z

@bors r=petrochenkov,JonathanBrouwer rollup

rust-bors · 2026-02-22T08:07:17Z

📌 Commit f1c80f5 has been approved by petrochenkov,JonathanBrouwer

It is now in the queue for this repository.

…, r=petrochenkov,JonathanBrouwer reduce the amount of panics in `{TokenStream, Literal}::from_str` calls *[View all comments](https://triagebot.infra.rust-lang.org/gh-comments/rust-lang/rust/pull/147859)* Before this PR, calling `TokenStream::from_str` or `Literal::from_str` with an invalid argument would always cause a compile error, even if the `TokenStream` is not used afterwards at all. This PR changes this so it returns a `LexError` instead in some cases. This is very theoretically a breaking change, but the doc comment on the impl already says ``` /// NOTE: some errors may cause panics instead of returning `LexError`. We reserve the right to /// change these errors into `LexError`s later. ``` Fixes some cases of rust-lang#58736.

JonathanBrouwer · 2026-02-25T12:12:00Z

@rust-timer build 62f2e76

rust-timer · 2026-02-25T13:28:20Z

Finished benchmarking commit (62f2e76): comparison URL.

Overall result: no relevant changes - no action needed

Benchmarking this pull request means it may be perf-sensitive – we'll automatically label it not fit for rolling up. You can override this, but we strongly advise not to, due to possible changes in compiler perf.

@bors rollup=never
@rustbot label: -S-waiting-on-perf -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

Results (primary 1.0%, secondary 2.0%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	2.7%	[2.3%, 3.2%]	2
Regressions ❌ (secondary)	6.3%	[6.3%, 6.3%]	1
Improvements ✅ (primary)	-2.5%	[-2.5%, -2.5%]	1
Improvements ✅ (secondary)	-2.4%	[-2.4%, -2.4%]	1
All ❌✅ (primary)	1.0%	[-2.5%, 3.2%]	3

Cycles

Results (secondary 4.2%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	10.6%	[10.6%, 10.6%]	1
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-2.3%	[-2.3%, -2.3%]	1
All ❌✅ (primary)	-	-	0

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 481.835s -> 480.795s (-0.22%)
Artifact size: 397.84 MiB -> 395.91 MiB (-0.49%)

rustbot assigned davidtwco Oct 18, 2025

This comment has been minimized.

Sign in to view

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 18, 2025

cyrgani force-pushed the nonfatal-tokenstream-parse branch from e2c0009 to 80b67bb Compare October 19, 2025 09:03

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Oct 19, 2025

bjorn3 reviewed Oct 20, 2025

View reviewed changes

tests/ui/proc-macro/invalid-punct-ident-4.rs Show resolved Hide resolved

cyrgani force-pushed the nonfatal-tokenstream-parse branch from ba098bc to 0274d78 Compare October 21, 2025 14:28

rustbot assigned wesleywiser and unassigned davidtwco Oct 25, 2025

cyrgani force-pushed the nonfatal-tokenstream-parse branch from 0274d78 to 13d3d06 Compare October 29, 2025 22:19

This comment has been minimized.

Sign in to view

cyrgani changed the title ~~replace TokenStream::from_str panics with LexErrors~~ reduce the amount of panics in {TokenStream, Literal}::from_str calls Oct 29, 2025

cyrgani mentioned this pull request Nov 4, 2025

add larger test for proc_macro FromStr implementations #148505

Merged

petrochenkov added the I-libs-api-nominated Nominated for discussion during a libs-api team meeting. label Feb 12, 2026

dtolnay linked an issue Feb 17, 2026 that may be closed by this pull request

Panic in proc_macro::TokenStream::from_str #58736

Closed

rust-bors bot pushed a commit that referenced this pull request Feb 17, 2026

Auto merge of #147859 - cyrgani:nonfatal-tokenstream-parse, r=<try>

0269aec

reduce the amount of panics in `{TokenStream, Literal}::from_str` calls

This comment has been minimized.

Sign in to view

craterbot added S-waiting-on-crater Status: Waiting on a crater run to be completed. and removed S-waiting-on-t-libs-api Status: Awaiting decision from T-libs-api labels Feb 17, 2026

craterbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-crater Status: Waiting on a crater run to be completed. labels Feb 21, 2026

petrochenkov removed the I-libs-api-nominated Nominated for discussion during a libs-api team meeting. label Feb 22, 2026

rust-bors bot added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Feb 22, 2026

jhpratt mentioned this pull request Feb 22, 2026

Rollup of 15 pull requests #152960

Closed

This was referenced Feb 22, 2026

Rollup of 13 pull requests #152964

Closed

Rollup of 14 pull requests #152965

Merged

This comment has been minimized.

Sign in to view

Uh oh!

Conversation

cyrgani commented Oct 18, 2025 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rustbot commented Oct 18, 2025

Uh oh!

rustbot commented Oct 18, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

cyrgani commented Oct 18, 2025

Uh oh!

rustbot commented Oct 18, 2025

Uh oh!

cyrgani commented Oct 19, 2025

Uh oh!

Uh oh!

cyrgani commented Oct 23, 2025

Uh oh!

davidtwco commented Oct 25, 2025

Uh oh!

This comment has been minimized.

joshtriplett commented Feb 17, 2026

Uh oh!

JonathanBrouwer commented Feb 17, 2026

Uh oh!

This comment has been minimized.

rust-bors bot commented Feb 17, 2026

Uh oh!

JonathanBrouwer commented Feb 17, 2026

Uh oh!

craterbot commented Feb 17, 2026

Uh oh!

craterbot commented Feb 19, 2026

Uh oh!

craterbot commented Feb 21, 2026

Footnotes

Uh oh!

JonathanBrouwer commented Feb 21, 2026

Uh oh!

petrochenkov commented Feb 22, 2026

Uh oh!

rust-bors bot commented Feb 22, 2026

Uh oh!

JonathanBrouwer commented Feb 22, 2026

Uh oh!

rust-bors bot commented Feb 22, 2026

Uh oh!

JonathanBrouwer commented Feb 25, 2026

Uh oh!

This comment has been minimized.

rust-timer commented Feb 25, 2026

Overall result: no relevant changes - no action needed

Instruction count

Max RSS (memory usage)

Cycles

Binary size

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

15 participants

cyrgani commented Oct 18, 2025 •

edited by rustbot

Loading