-
Notifications
You must be signed in to change notification settings - Fork 40
Update Emscripten jobs to add llvm 21 #776
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update Emscripten jobs to add llvm 21 #776
Conversation
|
clang-tidy review says "All clean, LGTM! 👍" |
dc36556 to
5d9fc77
Compare
|
clang-tidy review says "All clean, LGTM! 👍" |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #776 +/- ##
=======================================
Coverage 79.32% 79.32%
=======================================
Files 11 11
Lines 3966 3966
=======================================
Hits 3146 3146
Misses 820 820 🚀 New features to boost your workflow:
|
patches/llvm/emscripten-clang21-2-enable_exception_handling.patch
Outdated
Show resolved
Hide resolved
718d5cb to
746beec
Compare
|
clang-tidy review says "All clean, LGTM! 👍" |
746beec to
bf68cf8
Compare
|
clang-tidy review says "All clean, LGTM! 👍" |
bf68cf8 to
f22dd9a
Compare
|
clang-tidy review says "All clean, LGTM! 👍" |
anutosh491
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @vgvassilev @mcbarton,
All the changes we want have been applied as patches here and I just expect everything to work and the tests to pass here.
But something that concerns me heavily is providing an option to switch between EH model (wasm exceptions vs emscripten exceptions) that is being done through
CPPINTEROP_ENABLE_WASM_EXCEPTIONS
It is just not worth it. The reward we get out of it vs the maintanance cost doesn't match at all !
A) Firstly as a product, we shouldn't be giving our users options for anything & everything. It's just an exception handling model, we can just go with 1
B) We've switched to emscripten 4 on cppinterop and using packages from https://prefix.dev/emscripten-forge-4x for building the deployment. All the packages on this channel (around 400) have been built using wasm exceptions, just cause we realized that emscripten 4 works well with wasm exceptions (this was mainly done for scipy and other R packages but has its effect on everything on the channel like C++/Rust packages)
C) The reward for all this maintaince is just 1 single cell (cell 3) in our demo notebook.
We can obviously solve this cell through different EH models but that should not be the goto. We should have more confidence in one Exception model we pick and report issues on llvm as and when we find them in the future.
Sure, we can do it for both ... but then both model might have different errors in the future that we might have to fix on LLVM's side which will be painful and introduces lot of waiting.
I suggest let's go ahead with one model and build our confidence in it. That should be enough !
|
clang-tidy review says "All clean, LGTM! 👍" |
|
Wait, I don't get why the wasm exceptions related flags/changes have been dropped in the previous commit. We are moving away from the emscripten EH model and towards the wasm EH model !! That's what the patches are about and all the discussion & effort Vassil and I put in with the Prs for the past 1.5 weeks ! |
20098ea to
247ff63
Compare
|
clang-tidy review says "All clean, LGTM! 👍" |
.github/workflows/deploy-pages.yml
Outdated
| -DCMAKE_INSTALL_PREFIX=$PREFIX \ | ||
| -DCMAKE_FIND_ROOT_PATH_MODE_PACKAGE=ON \ | ||
| -DSYSROOT_PATH=$SYSROOT_PATH \ | ||
| -DCPPINTEROP_ENABLE_WASM_EXCEPTIONS=ON \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| -DCPPINTEROP_ENABLE_WASM_EXCEPTIONS=ON \ | |
| -DCPPINTEROP_ENABLE_WASM_EXCEPTIONS=ON \ |
We make CPPINTEROP_ENABLE_WASM_EXCEPTIONS=ON default and we implement a new option CPPINTEROP_EXTRA_WASM_FLAGS=<...> where people can specify -sSUPPORT_LONGJMP=wasm -fwasm-exceptions which will be appended and overrule the default wasm exception model...
|
clang-tidy review says "All clean, LGTM! 👍" |
| if(EMSCRIPTEN) | ||
| set( | ||
| CPPINTEROP_EXCEPTION_HANDLING_FLAGS | ||
| "-sSUPPORT_LONGJMP=wasm -fwasm-exceptions" | ||
| CACHE STRING | ||
| "Compiler flags for CPP interop exception handling" | ||
| ) | ||
| endif() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| if(EMSCRIPTEN) | |
| set( | |
| CPPINTEROP_EXCEPTION_HANDLING_FLAGS | |
| "-sSUPPORT_LONGJMP=wasm -fwasm-exceptions" | |
| CACHE STRING | |
| "Compiler flags for CPP interop exception handling" | |
| ) | |
| endif() | |
| option(CPPINTEROP_EXTRA_WASM_FLAGS "Enable custom flags for wasm" "-sSUPPORT_LONGJMP=wasm -fwasm-exceptions") |
| EMCC_CFLAGS="-sSUPPORT_LONGJMP=wasm -fwasm-exceptions" emmake make libclang -j $(nproc --all) | ||
| EMCC_CFLAGS="-sSUPPORT_LONGJMP=wasm -fwasm-exceptions" emmake make clangInterpreter clangStaticAnalyzerCore -j $(nproc --all) | ||
| EMCC_CFLAGS="-sSUPPORT_LONGJMP=wasm -fwasm-exceptions" emmake make lldWasm -j $(nproc --all) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can revert back these things as now we have them in the variables by default.
|
clang-tidy review says "All clean, LGTM! 👍" |
|
clang-tidy review says "All clean, LGTM! 👍" |
| option(CPPINTEROP_USE_REPL "Use clang-repl as backend" ON) | ||
| option(CPPINTEROP_ENABLE_TESTING "Enable the CppInterOp testing infrastructure." ON) | ||
| if(EMSCRIPTEN) | ||
| set(CPPINTEROP_EXTRA_WASM_FLAGS "-sSUPPORT_LONGJMP=wasm -fwasm-exceptions" CACHE STRING "Extra flags for wasm") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
option(CPPINTEROP_EXTRA_WASM_FLAGS "Enable custom flags for wasm" "-sSUPPORT_LONGJMP=wasm -fwasm-exceptions") which was suggested didn't work. This works though.
The error you get using option can be found here https://github.com/compiler-research/CppInterOp/actions/runs/21318907457/job/61366057216?pr=776#step:9:324
|
clang-tidy review says "All clean, LGTM! 👍" |
|
clang-tidy review says "All clean, LGTM! 👍" |
Emscripten-build-instructions.md
Outdated
| emmake make lldWasm -j $(nproc --all) | ||
| EMCC_CFLAGS="-sSUPPORT_LONGJMP=wasm -fwasm-exceptions" emmake make libclang -j $(nproc --all) | ||
| EMCC_CFLAGS="-sSUPPORT_LONGJMP=wasm -fwasm-exceptions" emmake make clangInterpreter clangStaticAnalyzerCore -j $(nproc --all) | ||
| EMCC_CFLAGS="-sSUPPORT_LONGJMP=wasm -fwasm-exceptions" emmake make lldWasm -j $(nproc --all) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is probably not needed right ?
Building clangInterpreter above this line would need building lldwasm anyways.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also once set, there's no need to repeat I'm guessing
export EMCC_CFLAGS="-sSUPPORT_LONGJMP=wasm -fwasm-exceptions"
And then we can just have emmake make ...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, lldWasm is not a target we need and was planning to remove it at some point but haven't got around to it.
I don't plan to apply your export EMCC_CFLAGS suggestion though as that makes the environment variable persistent and if not unset then it would override cppinterops new config flag making the flag pointless. I will put all the make targets on a single line to reduce the diff, but keep the EMCC_CFLAGS to a single command rather than using export
| GTEST_SKIP() << "Test fails with Cling on Windows on ARM"; | ||
| #endif | ||
| #ifdef __EMSCRIPTEN__ | ||
| GTEST_SKIP() << "This test crashes for Emscripten builds of CppInterOp"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm, could you paste the crash. Let's see if we can fix !
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The ci doesn'f give any useful output for the error since we have a minimum size build. I don't have time to debug this locally in the time we have left. If you have time to debug locally though and find a solution i'll try it out in the PR.
CMakeLists.txt
Outdated
| if (APPLE OR EMSCRIPTEN) | ||
| set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fno-common -Woverloaded-virtual -Wcast-qual -fno-strict-aliasing -Wno-long-long -Wall -W -Wno-unused-parameter -Wwrite-strings") | ||
| if (EMSCRIPTEN) | ||
| set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fwasm-exceptions") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn’t this go with in the option default value?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes if should, and is what I had originally. The reason it is like this at the moment is because -sSUPPORT_LONGJMP=wasm causes googletest to have a build warning, and we treat building warnings as errors. I planned on fixing is soon after post release since I don't really have any time go debug the googletest build error in the time we have before release. Would a fixme suffice for now?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we filter that one warning to error flag out from gtest? I think we have the logic in clad to copy paste from.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changed the logic so that -sSUPPORT_LONGJMP=wasm is just filtered out of CMAKE_CXX_FLAGS just for the Emscripten Googletest build. We will see if my quick attempt at a filter works when the ci runs.
|
clang-tidy review says "All clean, LGTM! 👍" |
|
clang-tidy review says "All clean, LGTM! 👍" |
vgvassilev
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks good to me. Let's wait for @anutosh491 to review it.
anutosh491
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this is good.
Let's see if the deployment notebook works well after we merge this (the raytracer and mamba magic install should be a good test to confirm wasm exceptions)
Also falling short on time to check the error on this
#776 (comment) but going through the test, I can somewhat sense what the error is going to be (and I know off the fix already 😄 ) ... but I don't mind getting it in through another PR if someone can help me with the error log.
|
clang-tidy review says "All clean, LGTM! 👍" |
During the llvm 21 upgrade PR we couldn't upgrade the Emscripten jobs due to an issue where using wasm exceptions broke between llvm 20 and 21. This PR https://github.com/llvm/llvm-project/issues/175717contains the changes which will enable an llvm 21 Emscripten build with wasm exceptions, once the following llvm issue llvm/llvm-project#175717 opened by Anutosh has been resolved .