Follow-up on benchmarks

[Robbepop]

During the call I mentioned that we might not only want to look at the file size improvements when judging whether compact imports are worth it. I just tested with for a small (~5kB) Rust compiled Wasm file that is typical for certain users of Wasmi with Wasmi's lazy translation mode. I did the comparison by emptying all the module name identifiers of the import section to "simulate" the effect of this proposal.

This benchmarks the validation performance of this small Wasm file original and "stripped".
The file size of the stripped Wasm file only decreased by ~0.8% compared to the original.

	Original	Stripped	Improvement (%)
Parse + Validate	~9.6us	~9.1us	~5.3%
Only Parse*	~4.7us	~4.45us	~5.3%

*: This is useful if a user knows that the input has already been validated prior.

So all in all more modest improvements that I'd had hoped but still the effect is larger than those ~0.8% file size reduction. Also the effect might be amplified slightly with a proper implementation.

The reason for this is that most of a Wasm file size is comprised of function bodies. However, when deferring parsing, validation and translation of those, the remaining parts of the Wasm binary become way more influential concerning performance.

For larger Wasm binaries this effect is less noticable though.

[bvisness]

I just published the tool I used to generate the stats spreadsheet: https://github.com/bvisness/wasm-import-compactor

Perhaps you could give that a run and try again? The numbers already seem promising though.

EDIT: Oh, except you'd have to implement the proposal too. Well, it's phase 3 and not too hard in my opinion, so perhaps worth a shot!

[Robbepop]

@bvisness exactly, Wasmi does not yet have an implementation for the new binary format.

Wasmi uses an old version of the Bytecode Alliance's wasmparser crate. I am currently not having capacity to fork wasmparser, implement this functionsliaty there, then implement it in Wasmi.

The above benchmarks are just a pointer that there are use cases for which compact import section improves performance slightly on top of just file sizes.

[bvisness]

I just did a bit of testing with our test corpus, and found that on average compact imports seem to speed up WebAssembly.validate by 0.6%. And many modules saw speedups close to 2%, including photoshop, acrobat, and sqlite. (spreadsheet) Not earth-shattering, but good confirmation that it does something.

Of course, for the average to be 0.6%, there are definitely a few regressions in this dataset, but this is only measuring SpiderMonkey and it's possible that we have a small performance regression in e.g. our handling of encoding 2. So I plan to investigate, but in any case I think there are still pretty clear signs of improvement here.

[Robbepop]

I want to emphasize that the significant speed-ups reported with Wasmi above were only possible due to Wasmi's lazy translation feature that skips all of the function bodies. Without this feature, performance improvements likely are very similar to their respective file size improvements.

[rossberg]

Isn't what we are seeing an improvement in pure decoding time? No doubt that decoding will be faster. @Robbepop, the way I understood your conjecture was a possible improvement in validation itself, beyond just decoding. Or did you simply mean the total time until validation is complete?

[Robbepop]

@rossberg the "Only Parse" row in the table above shows time it took for decoding without validation.

Interestingly decoding + validation (the row above it) also improved by the same relative amount, thus indicating to me that also validation performance improved. Note that all this is without a proper implementation of compact import section, thus numbers could be different with one.

The effect is larger than the file size improvements due to the lazy translation scheme that skips function bodies, thus making the (relatively large) type and import sections sizes of the Wasm and their respective decoding/validation performance more signfiicant overall.

This is a common use case for Wasmi users that have a limited control over the Wasm inputs.

[bvisness]

Isn't what we are seeing an improvement in pure decoding time?

Not in my case; my timings are from running WebAssembly.validate.

[rossberg]

@Robbepop, ah, thanks for the clarification. But if your validation minus parsing times already decrease merely by changing the names of imports then something is off. Did you perhaps measure UTF-8 checking as part of validation? (Semantically, it is part of parsing.) I just want to understand what's going on.

@bvisness, I understand, but that only means that those numbers can't distinguish parse vs validate.

[bvisness]

I am looking at WebAssembly.validate because I am primarily interested in how quickly compilation can start. (Although I grant that I am also validating function bodies here, but still.) I'm happy to gather other timings if anyone is curious.

[rossberg]

@bvisness, that's totally fine. Primarily I was curious about the details of the OP. I suspect we're all in violent agreement modulo some terminology/measurement diff about what constitutes parsing versus validation. No need to spend more engineering time on it.

[Robbepop]

@rossberg Wasmi is using the Bytecode Alliance's wasmparser crate for parsing (decoding) and validation. Assuming that wasmparser works correctly I'd assume it also validates UTF-8. Did that answer your question?

[rossberg]

@Robbepop, sorry for sounding so pedantic. I suppose my question boils down to what "Only Parse" measures, and whether or not that includes UTF-8 checks (I don't know how wasmparser does it). I can only guess it doesn't, but if parse indeed includes UTF-8 decoding then I'd be really mystified by your numbers. What would be a plausible explanation for name changes affecting validation time, given that Wasm validation rules don't even look at import names?

[Robbepop]

@rossberg ah i am sorry about the confusion on my part!

I had to check myself but here we go:
https://github.com/bytecodealliance/wasm-tools/blob/main/crates/wasmparser/src/binary_reader.rs#L678-L702

tldr; the wasmparser crate performs an UTF8 check during decoding.

[tlively]

I know for V8, at least, validation happens concurrently with decoding the module bytes. There's no separate decoded AST that can be validated. So I would absolutely expect improving decompressed module size to improve validation times by a corresponding amount. Perhaps other implementations are similar.

[rossberg]

@Robbepop, thanks for checking. So I'm still at a loss of explanation.

@tlively, yes, many or most engines fuse the two phases, so the difference isn't observable for those. That's fine. My only intent is understanding the numbers in the OP, which appear to say that wasmparser can separate both phases, but with completely mysterious results for this experiment (to me anyway).