Introduce coverage script #3119
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Generating test coverage reports for the SnarkVM repo is very CPU/RAM heavy and time taking (days). Because of that I am introducing a new script that mirrors the CI (by dividing the tests in groups/workflows) and coverage reports can be sone by groups.
The script has the ability to merge these reports and to cover all tests like that.
The idea is to run this script for every group on a very powerful machine/VM from time to time and then generate a report.
I do not recommend this to be done on the CI, but the CI may create some VM that does it and can gather results after it is done on specific branch, or something similar to the stress tests manager can do that.
All depends on the costs. I spent some time trying to make it work and came with this idea. I have an idea for code analysis for a very loose coverage report, but won't be including it here, as it is very experimental.
Test Plan
I successfully ran two groups locally and merged them. Couldn't run heavier groups like the
ledgeron my machine, tried to make it lighter and do it but was not really possible. Can be done with the optional custom filters I left in the script as logic and test tagging, but the idea is not to see what my machine can run, so with thecircuitandmiscgroups I had success.Documentation
This script has a README in the PR.
Backwards compatibility