WIP: add native rust-based initramfs generator, init and qemu wrapper#258
Open
ddiss wants to merge 279 commits intorapido-linux:masterfrom
Open
WIP: add native rust-based initramfs generator, init and qemu wrapper#258ddiss wants to merge 279 commits intorapido-linux:masterfrom
ddiss wants to merge 279 commits intorapido-linux:masterfrom
Conversation
Likely only useful for rapido.rs, but I've split it out as a separate project at https://github.com/ddiss/kv-conf . This source corresponds to commit 1f8d1448c613f87ed681a928c61b686914273b6e. Signed-off-by: David Disseldorp <ddiss@suse.de>
Vaguely similar to vm_autorun.env. Uses the new kv-conf parsing library for rapido.conf . The kernel command line is parsed crudely to obtain rapido vm-specific parameters. Signed-off-by: David Disseldorp <ddiss@suse.de>
Signed-off-by: David Disseldorp <ddiss@suse.de>
Signed-off-by: David Disseldorp <ddiss@suse.de>
rapido-init now attempts to do as much work as possible before handing off to execute any autorun payloads. Signed-off-by: David Disseldorp <ddiss@suse.de>
There's no real point, and it makes rapido-init parsing more difficult (without kv-conf). Signed-off-by: David Disseldorp <ddiss@suse.de>
This new fn allows for the kv HashMap to be pre-filled with defaults prior to config processing. Additive multi-file configuration processing also becomes much simpler with this functionality. Signed-off-by: David Disseldorp <ddiss@suse.de>
Determine CPU, memory and network resources based on cpio rapido-rsc paths. Generate kernel command line parameters for rapido-init and systemd-networkd. Signed-off-by: David Disseldorp <ddiss@suse.de>
Signed-off-by: David Disseldorp <ddiss@suse.de>
Parses the text-based modules.dep file and prints recursive dependencies
for a given set of modules. Only supports ".ko" or ".ko.zst" extensions.
E.g.
$ tools/parse_depmod.awk -- btrfs ext4 xfs \
< /lib/modules/6.4.0-150600.23.70-default/modules.dep
kernel/lib/raid6/raid6_pq.ko.zst
kernel/fs/jbd2/jbd2.ko.zst
kernel/fs/mbcache.ko.zst
kernel/crypto/xor.ko.zst
kernel/fs/xfs/xfs.ko.zst
kernel/fs/ext4/ext4.ko.zst
kernel/lib/crc16.ko.zst
kernel/lib/libcrc32c.ko.zst
kernel/fs/btrfs/btrfs.ko.zst
Signed-off-by: David Disseldorp <ddiss@suse.de>
This significantly simplifies archiving. Any hardlink will be treated as a normal file: assigned a unique inode number and associated data segment. Data segments may be duplicated if multiple hardlinks with the same source inode are archived (easy to avoid). The main benefit here is that deferred I/O for hardlink data segments is removed, so src-path = archive-path behaviour can finally be dropped. Signed-off-by: David Disseldorp <ddiss@suse.de>
Following removal of dracut-cpio hardlink support, remove GNU cpio vs dracut-cpio output comparison tests. Leave a single remaining hardlink tests, which simply checks that the extracted output data remains the same. Signed-off-by: David Disseldorp <ddiss@suse.de>
This is the final step in removing src-path-is-dest-path archiving behaviour for the core cpio library; the archive_symlink caller is responsible for providing the symlink target path. Signed-off-by: David Disseldorp <ddiss@suse.de>
Make state.off private. Also, use consts for the trailer name string. Signed-off-by: David Disseldorp <ddiss@suse.de>
It's only used by dracut-cpio based on a command line parameter, so change it to a local variable. Signed-off-by: David Disseldorp <ddiss@suse.de>
Move list_separator out of the core cpio library and into dracut-cpio. Don't allow it to be changed at runtime, as doing so would mean reworking too many callers. Drop the one test which checks for nul-separators. Signed-off-by: David Disseldorp <ddiss@suse.de>
Passing both of these around everywhere doesn't make much sense, as the properties should be static. Signed-off-by: David Disseldorp <ddiss@suse.de>
We can pass in an &str. Signed-off-by: David Disseldorp <ddiss@suse.de>
I'm finally getting my head around the String->PathBuf &str->&Path mappings. Signed-off-by: David Disseldorp <ddiss@suse.de>
Add parent directories to initramfs as they are encountered for files. Use a new paths_seen map to filter duplicates. Signed-off-by: David Disseldorp <ddiss@suse.de>
Parameter parsing only for now. Signed-off-by: David Disseldorp <ddiss@suse.de>
The fs::Metadata type is private, which makes it difficult to create cpio archive entries for items which don't exist locally. Create a new public ArchiveMd type for this purpose, which can easily be instantiated from fs::Metadata . This also has the benefit of moving some of the duplicate 64-bit -> 32-bit (cpio) overflow checks into a single function. There's no backward compatibility offered, so this change will break existing callers. Signed-off-by: David Disseldorp <ddiss@suse.de>
Signed-off-by: David Disseldorp <ddiss@suse.de>
Signed-off-by: David Disseldorp <ddiss@suse.de>
We now have the ability to mock up ArchiveMd metadata for cpio entries, so src→dest install can be properly implemented. Support is still a bit ugly in that symlinks are not handled - it gets a bit messy / complex if we need to modify target paths, so I'd prefer to skip it if we can get away with it. Signed-off-by: David Disseldorp <ddiss@suse.de>
Without a workspace, Cargo appears to only supports a single lib. cpio is selfcontained, so just split it out as a separate (internal) crate. Signed-off-by: David Disseldorp <ddiss@suse.de>
Implement a rust-style iterator API which walks through each entry in the Reader-provided cpio archive. For simplicity, only metadata (including filename) is available to the caller. Rapido VM CPU, network and memory resources are specified in images via special file paths, so data segments can be ignored for now. Signed-off-by: David Disseldorp <ddiss@suse.de>
namesize must include a zero terminator. Not sure how other archivers behave here. Signed-off-by: David Disseldorp <ddiss@suse.de>
GNU cpio adds zeros after the trailer, which causes a header parsing failure if we continue. Signed-off-by: David Disseldorp <ddiss@suse.de>
Avoid calling GNU cpio --list and instead use the new cpio iter API. Further optimizations can be made to stop traversing after all rapido-rsc entries have been covered. Signed-off-by: David Disseldorp <ddiss@suse.de>
Signed-off-by: David Disseldorp <ddiss@suse.de>
The paths should probably be based on rustc-env vars. Signed-off-by: David Disseldorp <ddiss@suse.de>
upload-artifact zips the file, so don't pre-compress it. Signed-off-by: David Disseldorp <ddiss@suse.de>
Rapido relies on kvm for aarch64, which ubuntu-24.04-arm Github workers don't provide. Signed-off-by: David Disseldorp <ddiss@suse.de>
Some widely used Linux distributions don't include the zram module, which may mean that a user's first impression is a rapido-cut error. Ubuntu 24.04 is one such distro. Signed-off-by: David Disseldorp <ddiss@suse.de>
Dracut is now no longer needed. Many of the old cut scripts still use it, but I think most people will be writing their own with simple-example/network as a reference. Signed-off-by: David Disseldorp <ddiss@suse.de>
ddiss
added a commit
to ddiss/linux
that referenced
this pull request
Feb 26, 2026
Github hosted "ubuntu-latest" x86-64 runners have enough resources (4 CPU, 16G RAM, 14 GB SSD) to build and run mainline kernel + xfstests in a nested VM. This script uses rapido rapido-linux/rapido#258 as a minimal initramfs generator and thin wrapper around QEMU. There are some test failures due to the minimal test environment (e.g. no udev yet). TEST and SCRATCH devices are back by 8G zstd compressed qcow2 images. Signed-off-by: David Disseldorp <ddiss@suse.de>
Rust uses hardlinks for compiled release binaries, which means that this error message (incorrectly) warns of rapido-init duplication every time. Switch it to a debug message. Signed-off-by: David Disseldorp <ddiss@suse.de>
modules.weakdep is often not present so remove the error message. Signed-off-by: David Disseldorp <ddiss@suse.de>
ddiss
added a commit
to ddiss/linux
that referenced
this pull request
Feb 26, 2026
Github hosted "ubuntu-latest" x86-64 runners have enough resources (4 core, 16G RAM, 14 GB SSD) to build and run mainline kernel + xfstests in a nested VM. This script uses rapido rapido-linux/rapido#258 as a minimal initramfs generator and thin wrapper around QEMU. There are some test failures due to the minimal test environment (e.g. no udev yet). TEST and SCRATCH devices are backed by 8G zstd compressed qcow2 images. Signed-off-by: David Disseldorp <ddiss@suse.de>
This omits the old dracut-based fstests runners for now, so failure to start udevd isn't fatal. Signed-off-by: David Disseldorp <ddiss@suse.de>
Needed for generic/478. Signed-off-by: David Disseldorp <ddiss@suse.de>
Dracut no longer does this for us. Signed-off-by: David Disseldorp <ddiss@suse.de>
Needed for generic/492 and generic/598 respectively. Signed-off-by: David Disseldorp <ddiss@suse.de>
Signed-off-by: David Disseldorp <ddiss@suse.de>
This allows us to run a bunch of tests which are otherwise skipped via _require_mknod(). Signed-off-by: David Disseldorp <ddiss@suse.de>
We might be able to avoid the open() by using something like:
enum BufIn {
File(io::BufReader<fs::File>),
Stdin(io::BufReader<io::Stdin>),
}
struct ArgsState {
manifests: Vec<BufIn>,
I don't think it's worth the extra complexity though.
Signed-off-by: David Disseldorp <ddiss@suse.de>
This isn't a public API, so there's no need to worry. Signed-off-by: David Disseldorp <ddiss@suse.de>
In preparation for supporting explicit path filters, we should ensure that pretty much everything gets put in paths_seen. I'd be interested in measuring the perf / resource effects of this change, but don't have time atm. Signed-off-by: David Disseldorp <ddiss@suse.de>
filter can be used to stop path traversal at specific points within a tree. This is particularly useful for source directories with .git/ artifacts. Link: rapido-linux#256 Signed-off-by: David Disseldorp <ddiss@suse.de>
Use the new "filter" directive to avoid archiving fstests git artifacts during $FSTESTS_SRC traversal. This reduces the initramfs size considerably. I.e. before 161835528, after 134994440 = ~25M saved. Link: rapido-linux#256 Signed-off-by: David Disseldorp <ddiss@suse.de>
This is helpful for rapido-rsc paths. Signed-off-by: David Disseldorp <ddiss@suse.de>
The "file" manifest directive no longer requires a local source for archiving, so drop the silly "dracut.conf.d/.empty" uses. Signed-off-by: David Disseldorp <ddiss@suse.de>
A few currently-skipped tests need these to run. flock is part of util-linux so can be listed as mandatory. Signed-off-by: David Disseldorp <ddiss@suse.de>
ddiss
added a commit
to ddiss/linux
that referenced
this pull request
Feb 27, 2026
Github hosted "ubuntu-latest" x86-64 runners have enough resources (KVM, 4 cores, 16G RAM, 14 GB SSD) to build and run mainline kernel + xfstests in a nested VM. This script uses rapido rapido-linux/rapido#258 as a minimal initramfs generator and thin wrapper around QEMU. For simplicity it'd likely make sense to branch it under the btrfs namespace. The test VM currently uses btrfs-progs from the ubuntu-24.04 host system. This could also be changed to a source-compiled version. TEST and SCRATCH devices are backed by 8G zstd compressed qcow2 images. Signed-off-by: David Disseldorp <ddiss@suse.de>
Collaborator
Author
|
I think these changes are pretty much ready for inclusion now, with systemd init support the only blocker. Existing dracut scripts will continue to work here, but use rapido-init and rapido-vm (as qemu wrapper) for boot. I'll branch before merge so that people can continue to use the unmodified Dracut boot path. |
btrfs/058 calls xfs_io interactively. If the termcap file is missing then libedit prints an error: Cannot read termcap database; using dumb terminal settings. Add the linux termcap file, corresponding to rapido-init's TERM env. It may make sense to use "ncurses6-config --terminfo-dirs" based paths, but this should be fine for now. Signed-off-by: David Disseldorp <ddiss@suse.de>
This allows us to run a few more skipped tests. unshare is from util-linux, so make it a hard dependency. Use hyphens instead of underscores in dm kmod names to match actual module name. Signed-off-by: David Disseldorp <ddiss@suse.de>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sangeetha and I have been working on some changes to replace Dracut with
a rust-based initramfs / cpio image generator. The main reasons for this
are:
locally available
modules, and kernel / user dependency gathering for cpio
and is mostly written in bash
The rewrite builds on my previous dracut-cpio implementation and adds:
parsing (thanks @thackara !)
scripts
It's otherwise kept as minimal as possible, with rust-elf and the std
library the only major external dependencies. One single-file crosvm
argument parser is also bundled.
The preliminary benchmark results look good, particularly for initramfs
image generation (cut):
To try these changes yourself, check out this branch and run:
You may need to change your rapido.conf file a little if you
use env variables or shell callouts.
I'm flagging this as WIP, as there are still a few things to do:
in the working directory
I don't expect to convert remaining cut scripts before merge. Dracut and
rust based functionality should be able to live side by side, although
rapido.conf parsing is much less flexible in rust: no invocations,
currently no env var expansion, variables must be wrapped in {}.