Migrate from docker save/load to registry push/pull #533
Conversation
…cesses - Removed caching steps for requirements and test requirements from multiple workflows. - Updated build steps to directly invoke requirements and test requirements builds. - Simplified conditions for authentication and Docker configuration based on repository ownership. - Ensured that the self-hosted build job only runs for the appropriate repository owner.
… processes - Removed caching steps for app and test app images across multiple workflows. - Introduced new steps to push and pull images with temporary tags for better CI job management. - Updated Docker commands to streamline image loading and authentication processes. - Consolidated image build steps in the Makefile for improved clarity and efficiency.
![seer-by-sentry[bot]](https://avatars.githubusercontent.com/in/801464?s=60&v=4){kind=small}
CodSpeed Performance ReportMerging #533 will not alter performanceComparing Summary
|
…cesses - Simplified build commands in the GitHub Actions workflow by removing the use of input variables for make targets. - Enhanced the Makefile by streamlining the build requirements and test requirements image creation steps. - Updated Docker commands to ensure consistent tagging and pulling of images, improving clarity and efficiency in CI processes.
.github/workflows/_push-env.yml
Outdated
| - name: Pull built image | ||
| run: | | ||
| docker load --input ${{ inputs.output_directory }}/app.tar | ||
| make ${{ inputs.make_target_prefix }}pull.app-temp |
There was a problem hiding this comment.
Potential bug: The _push-env.yml workflow attempts to pull a Docker image from a private registry before authenticating, and some jobs lack authentication, causing build failures.
-
Description: In the
_push-env.ymlworkflow, thepush-environmentjob attempts to pull a Docker image viamake ... pull.app-tempbefore the Google Cloud authentication step. For internal builds whereAR_REPOpoints to a private Google Artifact Registry, this will cause thedocker pullto fail with an authentication error. Therollingandreleasejobs within the same workflow are also affected as they lack the necessary Google Cloud authentication step entirely, which will also lead to pull failures. -
Suggested fix: Move the Google Cloud authentication steps to occur before any
docker pullcommands in thepush-environment,rolling, andreleasejobs. Add the necessary Google Cloud authentication steps to therollingandreleasejobs.
severity: 0.85, confidence: 0.98
Did we get this right? 👍 / 👎 to inform future reviews.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #533 +/- ##
=======================================
Coverage 93.83% 93.83%
=======================================
Files 1280 1280
Lines 46229 46229
Branches 1523 1523
=======================================
Hits 43378 43378
Misses 2543 2543
Partials 308 308
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
…thentication - Updated conditions for pushing app and test app images to exclude shared targets. - Added a new step for authenticating to Google Cloud using workload identity and service account secrets. - Introduced a step to retrieve the short SHA for better traceability in the workflow.
…zation - Adjusted the order of steps for pulling built images and Docker configuration for better clarity. - Removed redundant authentication step to streamline the workflow. - Ensured consistent formatting and spacing for improved readability.
- Changed the runner from 'ubuntu-large' to 'ubuntu' for the build-test-app job, optimizing resource usage.
- Changed the runner from 'ubuntu' to 'ubuntu-latest' for the build-test-app job to ensure access to the latest features and updates.
Problem
GitHub Actions runners were running out of disk space due to storing Docker images as tar files in the GitHub Actions cache. The workflow architecture used
docker saveto export images to tar files, cached them, and then useddocker loadto restore them in subsequent jobs.Solution
Migrated from tar-based image caching to direct registry pushes/pulls. This eliminates tar files from disk, leverages efficient registry layer storage, and simplifies the workflow.
Behavior
Internal PRs (codecov org):
Fork PRs:
docker pull || docker buildpatternImpact
Disk space savings: ~23.5 GB per full CI run