Enhance CI/CD with new workflows and reusable setup #906
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…e CI/CD workflows Introduce a reusable Python environment setup workflow to standardize dependency management, caching, and environment configuration across all workflows. This reduces duplication and maintenance overhead. Enhance the CI workflow with smart file change detection, parallel execution, and comprehensive static analysis. Add infrastructure validation, markdown linting, and efficient caching strategies. Implement a deployment workflow triggered by releases and manual dispatch, supporting staging and production environments. Add a Docker workflow for multi-platform builds, security scanning, and registry management. Include a maintenance workflow for automated housekeeping tasks like TODO conversion and image cleanup. Introduce a notifications workflow to handle CI failures by creating GitHub issues and closing them upon success. Finally, add a release workflow for automated version management, changelog generation, and release deployment, ensuring quality assurance with test suite integration. These changes aim to improve the project's CI/CD processes by providing a consistent, efficient, and secure environment for development, testing, and deployment.
Contributor
Reviewer's GuideThis PR refactors and extends the project’s CI/CD by extracting a reusable Python setup action and introducing dedicated GitHub Actions workflows for testing, Docker builds/deployments, security scanning, maintenance, releases, and deployments—each enhanced with smart change detection, parallel execution, conditional steps, and multi-level caching. Sequence diagram for Docker build and deploy workflowsequenceDiagram
participant GitHub as actor GitHub Event
participant DockerWorkflow as Docker Build & Deploy Workflow
participant Buildx as Docker Buildx
participant Registry as Container Registry (ghcr.io)
participant Trivy as Trivy Security Scan
participant DeployWorkflow as Deploy Workflow
GitHub->>DockerWorkflow: Trigger (push/tag/PR/schedule)
DockerWorkflow->>Buildx: Set up builder
DockerWorkflow->>Buildx: Build image (multi-platform if release)
Buildx->>Registry: Push image (if not PR)
DockerWorkflow->>Trivy: Scan built image (if not PR)
DockerWorkflow->>DeployWorkflow: Trigger deploy on release
DeployWorkflow->>Registry: Pull image for deployment
DeployWorkflow->>Environment: Deploy to staging/production
Class diagram for reusable setup-python actionclassDiagram
class SetupPythonAction {
+python-version: string = '3.13'
+install-groups: string = 'dev,types'
+cache-suffix: string = 'default'
+generate-prisma: string = 'true'
+Install Poetry()
+Set up Python()
+Cache Poetry dependencies()
+Install dependencies()
+Generate Prisma client()
}
Class diagram for setup-nodejs-markdown reusable actionclassDiagram
class SetupNodejsMarkdownAction {
+node-version: string = '20'
+Setup Node.js()
+Cache node modules()
+Install markdownlint()
}
Class diagram for upload-coverage reusable actionclassDiagram
class UploadCoverageAction {
+coverage-file: string
+junit-file: string = ''
+flags: string
+name: string
+codecov-token: string
+slug: string = 'allthingslinux/tux'
+Upload coverage to Codecov()
+Upload test results to Codecov()
}
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
Contributor
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.OpenSSF Scorecard
Scanned Files
|
Contributor
Author
|
@sourcery-ai title |
sourcery-ai
bot
changed the title
Update job names in GitHub Actions workflows to improve clarity and consistency. This includes renaming jobs to better reflect their purpose, such as changing "Python Code Quality" to "Python Type Checking" and "Markdown Documentation" to "Markdown Linting". These changes help in quickly identifying the purpose of each job, making the CI/CD pipeline more understandable and maintainable. chore(pre-commit): add auto commit messages for pre-commit hooks Add default commit messages for auto-fixes and updates from pre-commit hooks. This ensures that changes made by pre-commit hooks are clearly documented in the commit history, improving traceability and understanding of automated changes.
Deploying tux with
|
| Latest commit: |
a015db3
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://1c8b410d.tux-afh.pages.dev |
| Branch Preview URL: | https://fix-gh-actions.tux-afh.pages.dev |
Refactor GitHub Actions workflows to use composite actions for reusability and maintainability. Introduce new composite actions for setting up Python, Node.js, and test environments, as well as detecting file changes and uploading coverage reports. This change reduces duplication, simplifies workflow files, and centralizes common logic, making it easier to manage and update CI processes. The removal of the reusable workflow `_setup-python.yml` in favor of a composite action further enhances flexibility and reduces maintenance overhead.
…tions/changed-files Remove the custom GitHub Action for detecting file changes and replace it with the widely-used `tj-actions/changed-files` action. This change simplifies the maintenance of the CI configuration by leveraging a well-supported third-party action. It also ensures more reliable and up-to-date functionality for detecting file changes, which is crucial for optimizing CI workflows by skipping unnecessary jobs.
Update the job names in the CI and security workflows for clarity. Change "matrix.type Linting" to "Infrastructure Linting" and "CodeQL (language)" to "CodeQL Analysis" to provide more descriptive names. Remove the notifications workflow to streamline the CI process and reduce unnecessary overhead. The notifications workflow was responsible for creating GitHub issues on workflow failures and closing them on success, which is now deemed unnecessary for the current project requirements.
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Hey @kzndotsh - I've reviewed your changes and they look great!
Blocking issues:
- Using variable interpolation
${{...}}withgithubcontext data in arun:step could allow an attacker to inject their own code into the runner. This would allow them to steal secrets and code.githubcontext data can have arbitrary user input and should be treated as untrusted. Instead, use an intermediate environment variable withenv:to store the data and use the environment variable in therun:script. Be sure to use double-quotes the environment variable, like this: "$ENVVAR". (link) - Using variable interpolation
${{...}}withgithubcontext data in arun:step could allow an attacker to inject their own code into the runner. This would allow them to steal secrets and code.githubcontext data can have arbitrary user input and should be treated as untrusted. Instead, use an intermediate environment variable withenv:to store the data and use the environment variable in therun:script. Be sure to use double-quotes the environment variable, like this: "$ENVVAR". (link) - Using variable interpolation
${{...}}withgithubcontext data in arun:step could allow an attacker to inject their own code into the runner. This would allow them to steal secrets and code.githubcontext data can have arbitrary user input and should be treated as untrusted. Instead, use an intermediate environment variable withenv:to store the data and use the environment variable in therun:script. Be sure to use double-quotes the environment variable, like this: "$ENVVAR". (link)
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location> `.github/workflows/docker.yml:166` </location>
<code_context>
+ # Skips pull requests to prevent unnecessary deployments
+ # Waits for validation to complete before proceeding
if: github.event_name != 'pull_request'
+ needs: # Always wait for validation
+ - validate
runs-on: ubuntu-latest
</code_context>
<issue_to_address>
The build job always waits for validation, but validation only runs on pull requests.
Consider making the 'build' job's dependency on 'validate' conditional or restructuring the workflow to prevent unnecessary waiting or confusion for non-pull request events.
</issue_to_address>
### Comment 2
<location> `.github/workflows/docker.yml:273` </location>
<code_context>
+ cache-to: type=gha,mode=max # Update cache comprehensively
+ # CONDITIONAL MULTI-PLATFORM BUILDS
+ # ARM64 builds only for tagged releases to save resources
+ platforms: ${{ (github.event_name == 'push' && startsWith(github.ref, 'refs/tags/') && contains(github.ref, 'v')) && 'linux/amd64,linux/arm64' || 'linux/amd64' }}
+ # SECURITY ATTESTATIONS
+ # SLSA provenance and SBOM only for releases
</code_context>
<issue_to_address>
The conditional for multi-platform builds may not match all intended release tags.
'contains(github.ref, 'v')' may be unnecessary or could exclude valid tags. Please review and clarify the tag matching logic to ensure all intended release tags are handled.
</issue_to_address>
<suggested_fix>
<<<<<<< SEARCH
platforms: ${{ (github.event_name == 'push' && startsWith(github.ref, 'refs/tags/') && contains(github.ref, 'v')) && 'linux/amd64,linux/arm64' || 'linux/amd64' }}
=======
platforms: ${{ (github.event_name == 'push' && startsWith(github.ref, 'refs/tags/')) && 'linux/amd64,linux/arm64' || 'linux/amd64' }}
>>>>>>> REPLACE
</suggested_fix>
### Comment 3
<location> `.github/workflows/maintenance.yml:165` </location>
<code_context>
+ package-type: container # Container images only
+ # CONFIGURABLE RETENTION POLICY
+ # Default 10 images, override via manual trigger
min-versions-to-keep: ${{ github.event.inputs.keep_amount || '10' }}
+ # UNTAGGED IMAGE HANDLING
+ # Configurable untagged image cleanup (typically safe to remove)
</code_context>
<issue_to_address>
Using a string default for min-versions-to-keep may cause type issues.
Validate or cast the input to an integer to prevent unexpected behavior if a string is provided.
</issue_to_address>
### Comment 4
<location> `.github/workflows/release.yml:75` </location>
<code_context>
+ - name: Determine version
+ id: version
+ run: |
+ if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
+ VERSION="${{ github.event.inputs.version }}"
+ else
</code_context>
<issue_to_address>
Manual release version input is not validated for semantic versioning.
Add a validation step to ensure manual version inputs follow semantic versioning (e.g., v1.2.3) to prevent inconsistent tags.
</issue_to_address>
<suggested_fix>
<<<<<<< SEARCH
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
VERSION="${{ github.event.inputs.version }}"
else
VERSION="${GITHUB_REF#refs/tags/}"
fi
echo "version=$VERSION" >> "$GITHUB_OUTPUT"
=======
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
VERSION="${{ github.event.inputs.version }}"
# Validate semantic versioning: vMAJOR.MINOR.PATCH (e.g., v1.2.3)
if ! [[ "$VERSION" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
echo "Error: Manual version input '$VERSION' does not match semantic versioning (vMAJOR.MINOR.PATCH, e.g., v1.2.3)."
exit 1
fi
else
VERSION="${GITHUB_REF#refs/tags/}"
fi
echo "version=$VERSION" >> "$GITHUB_OUTPUT"
>>>>>>> REPLACE
</suggested_fix>
## Security Issues
### Issue 1
<location> `.github/workflows/deploy.yml:48` </location>
<issue_to_address>
**security (opengrep-rules.yaml.github-actions.security.run-shell-injection):** Using variable interpolation `${{...}}` with `github` context data in a `run:` step could allow an attacker to inject their own code into the runner. This would allow them to steal secrets and code. `github` context data can have arbitrary user input and should be treated as untrusted. Instead, use an intermediate environment variable with `env:` to store the data and use the environment variable in the `run:` script. Be sure to use double-quotes the environment variable, like this: "$ENVVAR".
*Source: opengrep*
</issue_to_address>
### Issue 2
<location> `.github/workflows/deploy.yml:71` </location>
<issue_to_address>
**security (opengrep-rules.yaml.github-actions.security.run-shell-injection):** Using variable interpolation `${{...}}` with `github` context data in a `run:` step could allow an attacker to inject their own code into the runner. This would allow them to steal secrets and code. `github` context data can have arbitrary user input and should be treated as untrusted. Instead, use an intermediate environment variable with `env:` to store the data and use the environment variable in the `run:` script. Be sure to use double-quotes the environment variable, like this: "$ENVVAR".
*Source: opengrep*
</issue_to_address>
### Issue 3
<location> `.github/workflows/release.yml:74` </location>
<issue_to_address>
**security (opengrep-rules.yaml.github-actions.security.run-shell-injection):** Using variable interpolation `${{...}}` with `github` context data in a `run:` step could allow an attacker to inject their own code into the runner. This would allow them to steal secrets and code. `github` context data can have arbitrary user input and should be treated as untrusted. Instead, use an intermediate environment variable with `env:` to store the data and use the environment variable in the `run:` script. Be sure to use double-quotes the environment variable, like this: "$ENVVAR".
*Source: opengrep*
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
| # Skips pull requests to prevent unnecessary deployments | ||
| # Waits for validation to complete before proceeding | ||
| if: github.event_name != 'pull_request' | ||
| needs: # Always wait for validation |
Contributor
There was a problem hiding this comment.
suggestion: The build job always waits for validation, but validation only runs on pull requests.
Consider making the 'build' job's dependency on 'validate' conditional or restructuring the workflow to prevent unnecessary waiting or confusion for non-pull request events.
| # CONDITIONAL MULTI-PLATFORM BUILDS | ||
| # ARM64 builds only for tagged releases to save resources | ||
| platforms: ${{ (github.event_name == 'push' && startsWith(github.ref, 'refs/tags/') && contains(github.ref, 'v')) && 'linux/amd64,linux/arm64' || 'linux/amd64' }} | ||
| # SECURITY ATTESTATIONS |
Contributor
There was a problem hiding this comment.
suggestion: The conditional for multi-platform builds may not match all intended release tags.
'contains(github.ref, 'v')' may be unnecessary or could exclude valid tags. Please review and clarify the tag matching logic to ensure all intended release tags are handled.
Suggested change
| platforms: ${{ (github.event_name == 'push' && startsWith(github.ref, 'refs/tags/') && contains(github.ref, 'v')) && 'linux/amd64,linux/arm64' || 'linux/amd64' }} | |
| platforms: ${{ (github.event_name == 'push' && startsWith(github.ref, 'refs/tags/')) && 'linux/amd64,linux/arm64' || 'linux/amd64' }} |
| package-type: container # Container images only | ||
| # CONFIGURABLE RETENTION POLICY | ||
| # Default 10 images, override via manual trigger | ||
| min-versions-to-keep: ${{ github.event.inputs.keep_amount || '10' }} |
Contributor
There was a problem hiding this comment.
issue (bug_risk): Using a string default for min-versions-to-keep may cause type issues.
Validate or cast the input to an integer to prevent unexpected behavior if a string is provided.
Comment on lines
+75
to
+80
| if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then | ||
| VERSION="${{ github.event.inputs.version }}" | ||
| else | ||
| VERSION="${GITHUB_REF#refs/tags/}" | ||
| fi | ||
| echo "version=$VERSION" >> "$GITHUB_OUTPUT" |
Contributor
There was a problem hiding this comment.
suggestion (bug_risk): Manual release version input is not validated for semantic versioning.
Add a validation step to ensure manual version inputs follow semantic versioning (e.g., v1.2.3) to prevent inconsistent tags.
Suggested change
| if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then | |
| VERSION="${{ github.event.inputs.version }}" | |
| else | |
| VERSION="${GITHUB_REF#refs/tags/}" | |
| fi | |
| echo "version=$VERSION" >> "$GITHUB_OUTPUT" | |
| if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then | |
| VERSION="${{ github.event.inputs.version }}" | |
| # Validate semantic versioning: vMAJOR.MINOR.PATCH (e.g., v1.2.3) | |
| if ! [[ "$VERSION" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then | |
| echo "Error: Manual version input '$VERSION' does not match semantic versioning (vMAJOR.MINOR.PATCH, e.g., v1.2.3)." | |
| exit 1 | |
| fi | |
| else | |
| VERSION="${GITHUB_REF#refs/tags/}" | |
| fi | |
| echo "version=$VERSION" >> "$GITHUB_OUTPUT" |
Comment on lines
+48
to
+68
| run: | | ||
| ENV="${{ github.event.inputs.environment || 'production' }}" | ||
| IMAGE="${{ steps.image.outputs.image }}" | ||
| echo "🚀 Deploying $IMAGE to $ENV environment" | ||
|
|
||
| # This is where you'd integrate with your deployment system | ||
| # Examples: | ||
| # - Update Kubernetes manifests | ||
| # - Deploy to cloud platforms | ||
| # - Update docker-compose files | ||
| # - Trigger external deployment systems | ||
|
|
||
| # For now, just simulate deployment | ||
| echo "✅ Deployment completed successfully" | ||
|
|
||
| # Set deployment URL (customize for your infrastructure) | ||
| if [ "$ENV" = "production" ]; then | ||
| echo "url=https://your-app.com" >> "$GITHUB_OUTPUT" | ||
| else | ||
| echo "url=https://staging.your-app.com" >> "$GITHUB_OUTPUT" | ||
| fi |
Contributor
There was a problem hiding this comment.
security (opengrep-rules.yaml.github-actions.security.run-shell-injection): Using variable interpolation ${{...}} with github context data in a run: step could allow an attacker to inject their own code into the runner. This would allow them to steal secrets and code. github context data can have arbitrary user input and should be treated as untrusted. Instead, use an intermediate environment variable with env: to store the data and use the environment variable in the run: script. Be sure to use double-quotes the environment variable, like this: "$ENVVAR".
Source: opengrep
Comment on lines
+71
to
+78
| run: |- | ||
| ENV="${{ github.event.inputs.environment || 'production' }}" | ||
| if [ "${{ steps.deploy.outcome }}" = "success" ]; then | ||
| echo "✅ Successfully deployed to $ENV" | ||
| echo "🔗 URL: ${{ steps.deploy.outputs.url }}" | ||
| else | ||
| echo "❌ Deployment to $ENV failed" | ||
| fi |
Contributor
There was a problem hiding this comment.
security (opengrep-rules.yaml.github-actions.security.run-shell-injection): Using variable interpolation ${{...}} with github context data in a run: step could allow an attacker to inject their own code into the runner. This would allow them to steal secrets and code. github context data can have arbitrary user input and should be treated as untrusted. Instead, use an intermediate environment variable with env: to store the data and use the environment variable in the run: script. Be sure to use double-quotes the environment variable, like this: "$ENVVAR".
Source: opengrep
Comment on lines
+74
to
+90
| run: | | ||
| if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then | ||
| VERSION="${{ github.event.inputs.version }}" | ||
| else | ||
| VERSION="${GITHUB_REF#refs/tags/}" | ||
| fi | ||
| echo "version=$VERSION" >> "$GITHUB_OUTPUT" | ||
|
|
||
| # Check if this is a prerelease (contains alpha, beta, rc) | ||
| if [[ "$VERSION" =~ (alpha|beta|rc) ]]; then | ||
| echo "is_prerelease=true" >> "$GITHUB_OUTPUT" | ||
| else | ||
| echo "is_prerelease=false" >> "$GITHUB_OUTPUT" | ||
| fi | ||
| echo "Release version: $VERSION" | ||
| echo "Is prerelease: $([ "$VERSION" != "${VERSION/alpha/}" ] || [ "$VERSION" != "${VERSION/beta/}" ] || [ "$VERSION" != "${VERSION/rc/}" ] && echo "true" || echo "false")" | ||
|
|
Contributor
There was a problem hiding this comment.
security (opengrep-rules.yaml.github-actions.security.run-shell-injection): Using variable interpolation ${{...}} with github context data in a run: step could allow an attacker to inject their own code into the runner. This would allow them to steal secrets and code. github context data can have arbitrary user input and should be treated as untrusted. Instead, use an intermediate environment variable with env: to store the data and use the environment variable in the run: script. Be sure to use double-quotes the environment variable, like this: "$ENVVAR".
Source: opengrep
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary by Sourcery
Standardize and extend the project’s CI/CD processes by introducing a reusable Python setup and adding comprehensive workflows for testing, Docker builds, security scanning, maintenance, releases, notifications, and deployments
New Features:
Enhancements:
CI:
Deployment:
Tests:
Summary by Sourcery
Overhaul CI/CD by introducing reusable setup actions and comprehensive GitHub Actions workflows for testing, building, security, maintenance, releases, and deployments.
New Features:
Enhancements:
Build:
CI:
Deployment:
Tests: