Functional Test Updates and CI Support Scripts #67
+1,135
−65
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add GitHub Actions workflow for automated hardware testing on AST1060. The workflow runs in two stages: 1. Precommit checks on ubuntu-22.04: - Verify Cargo.lock - Run cargo xtask precommit (format/lint/build) - Upload bloat reports 2. Hardware functional tests on self-hosted runner with AST1060: - Build firmware with test features (hmac, hash, rsa, ecdsa) - Generate UART boot image with proper 4-byte size header - Upload firmware via UART following AST1060 boot protocol - Monitor test execution and parse PASS/FAIL/SKIP results - Upload test logs and artifacts The workflow supports manual triggering with test suite selection and only runs hardware tests if precommit checks pass.
- Added instructions on how to configure the test host environment. - Added scripts required to package the binary and kick off the test.
This fully encapsulates the test flow into Python. Use of tio has been removed and replaced entirely by pyserial, removing the complication of an external tool. TODO: The test firmware itself needs a clear delineator when it is complete that we can look for. As it stands, tests just sort of run and complete with inconsistent syntax, and eventually we run out of tests and get stuck looking at timer ISR test leftovers. As it stands the script will time out after 600 seconds. All test output is logged to a file within the working directory. The script also serves as a tool to manipulate the GPIOs controlling boot functionality, and can be configured to target different GPIOs if needed.
Made tests outputs more consistent, but further refinements are necessary. Also added a test end print and adjusted the script to key off of it.
Move i2c to i2c2
The --upload-only parameter lets us target devices that we have only manual SRST and UART control over.
wmaroneAMD
requested review from
lxuxx,
rusty1968,
stevenlee7189 and
yvettewu1
| return False | ||
|
|
||
| # Keep only last few lines in buffer | ||
| buffer = '\n'.join(lines[-10:]) |
Collaborator
There was a problem hiding this comment.
The line buffer = '\n'.join(lines[-10:]) retains already-processed lines, causing the loop to re-scan them in the next iteration. This leads to double-counting of PASS/FAIL results.
Test execution completed!
Results: {'passed': 119, 'failed': 6, 'skipped': 0}
❌ Test execution failed!
Collaborator
There was a problem hiding this comment.
I verified that applying the following patch fixes the issue by only retaining the last incomplete fragment:
diff --git a/scripts/uart-test-exec.py b/scripts/uart-test-exec.py
index 9530f94..05ba45c 100644
--- a/scripts/uart-test-exec.py
+++ b/scripts/uart-test-exec.py
@@ -309,7 +309,7 @@ class UartTestExecutor:
# Parse test results
lines = buffer.split('\n')
- for line in lines:
+ for line in lines[:-1]:
if 'PASS' in line:
test_results['passed'] += 1
elif 'FAIL' in line:
@@ -328,7 +328,7 @@ class UartTestExecutor:
return False
# Keep only last few lines in buffer
- buffer = '\n'.join(lines[-10:])
+ buffer = lines[-1]With this fix, the counts are correct (showing 3 fails instead of 6):
Test execution completed!
Results: {'passed': 43, 'failed': 3, 'skipped': 0}
❌ Test execution failed!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Basic help is in the root, detailed documentation of the CI support script and configuration is in docs/ci-aspeed.md
This also refines the functional test structure to make it more machine parseable. This is effectively a manual convention due to lack of a test infrastructure to enforce it, but it ensures the script only has to key on 4 tokens.