full ebuild preservation archive for full build reproducibility

[magiruuvelvet]

I want to implement an full ebuild preservation system that

• copies the ebuild file itself (portage already does this: /var/db/pkg/${CATEGORY}/${PN}-${PVR}/${PN}-${PVR}.ebuild)
• copies the ebuild Manifest (contains file hashes; can be technically reproduced, but copying the manifest ensures historic accuracy of the stored distfiles)
• copies all corresponding ebuild files/ (can contain patches and other mandatory files to reproduce builds later on)
• copies all corresponding user patches from /etc/portage/patches (snapshot for long-term historic accuracy)
• copies all required eclass files from the belonging repository (ensure the ebuild can be reproduced after required eclasses were modified or deleted)
• creates a build info manifest with some environment data (use flags, features, profile, timestamps, etc.; this is for documentation purposes)

The purpose of this system is to allow for full package build reproducibility at any point in time, even after the ebuild is long gone from the repository due to EOL or other reasons.

Real-world use cases and problems:

• Last week I updated Qt5 from 5.15.14 to 5.15.17. Some required components were deleted from the repository because they have no revdeps anymore, but I still have deleted (as in the ebuild files are gone) packages installed on my system. This caused a concrete problem with the dev-qt/qtsensors package having a non-matching version - Qt refuses to mix different versioned Qt libraries. I managed to rebuild this particular package at 5.15.17, but other ebuilds aren't that forgiving. Especially when they require files/ and eclasses that might be long gone from the repository.
• Upgrading icu and boost caused some troubles with deleted packages. Those issues can be worked around with by using --nodeps, but portage will keep old libraries around for a long time until all offending packages are rebuilt - immediate upgrades are not always possible on my system due to some delayed and carefully planned upgrades.
• ditto for ffmpeg and any other rebuild trigger heavy package

To account for all possible scenarios in the future where I rely on deleted ebuilds for @preserved-rebuild, or other rebuild and upgrade scenarios, I will be preserving the entire stack that is required to reproduce a package at any given point in time for every newly installed and upgraded package. The preservation archive also serves as a documentation and source of truth what is actually installed on my system, with the exact build definitions.

Ideas:

• (low prio) design the ebuild preservation archive as a functioning portage overlay and add it with low priority. never need to worry again about ebuilds vanishing from the repository I still rely on.

References:

• https://devmanual.gentoo.org/ebuild-writing/variables/

Tasks:

• portage overlay with low priority containing only the currently installed packages; make the overlay a child overlay of gentoo
  • think about eclass compatibility if necessary
• mechanism to clean up uninstalled packages from the preservation archive
• mechanism to limit the per-version history of packages; currently just indefinitely growing

[magiruuvelvet]

initial version → 99a33b6
patches:

• ad79087 bug fix, manifest update
• db97bf9 don't pre-create unnecessary directories
• 9216eb8 rename preservation directory, delete empty patches directory
• b8807b7 strip SLOT variable for compatibility with portage behavior, also required for overlay generation later on
• ff3ce67 add slot of build manifest and ensure all timestamps are consistent and ISO formatted