[WIP] storage: add splitfdstream for efficient layer transfer with reflink support #651

giuseppe · 2026-02-10T12:32:04Z

Add a new splitfdstream package that enables efficient container layer transfer between storage instances by separating tar metadata from file content. File content is passed as file descriptors.

Signed-off-by: Giuseppe Scrivano <[email protected]>

mtrmac

6% of the code base for an extremely niche use case :/

I’m afraid I can’t spend time on this now; and anyway some other work on pkg/archive needs to happen first, and expect the rebase to be somewhat non-trivial.

giuseppe · 2026-02-10T15:55:34Z

6% of the code base for an extremely niche use case :/

I’m afraid I can’t spend time on this now; and anyway some other work on pkg/archive needs to happen first, and expect the rebase to be somewhat non-trivial.

what other work would block this feature?

I don't love either the amount of extra code, but it is almost all in new APIs/packages that are experimental and not leaking too much to the rest of the library.

We went around this problem for a long time, and in the end this seems like an ~ok solution to not expose too much of the containers storage internals augmenting the tar format that is already plugged everywhere.

giuseppe · 2026-02-10T17:43:19Z

to add more context: the goal of this feature is to be able to copy a layer, similarly to what we would do with Diff, but having the possibility to deduplicate files using reflinks. We need two separate copies because OS files can't be kept on the containers storage for different reasons, a podman system reset -f should not break the system, and OS files need to be relabeled.

There are other interesting use cases enabled with a RPC like facilitate usage from different languages (Rust in our case), inject into a build container to grab R/O access to layers/images, but this is out of scope now.

I realize complexity is not trivial just to get access to the raw files, but what alternatives do we have?

From the store API PoV we could just extend it to return a map $LAYER_FILE -> $PATH_TO_THE_FILE, but how would we use it from Rust?

Any suggestions?

mtrmac · 2026-02-10T18:24:55Z

(Just to avoid a misunderstanding, I didn’t really read the PR. I have, at this point, ~no opinion on RPC vs. Go API.)

cgwalters · 2026-02-10T19:18:18Z

storage/pkg/archive/archive.go

+	return &tarReaderIterator{
+		tr:     tr,
+		trBuf:  trBuf,
+		buffer: make([]byte, 1<<20),


Seems like 1<<20 should be in a const somewhere with a rationale?

cgwalters · 2026-02-10T19:19:02Z

storage/pkg/chrootarchive/splitfdstream_linux.go

+
+// splitFDStreamSocketDescriptor is the fd for the Unix socket used to
+// receive file descriptors via SCM_RIGHTS in the re-exec child.
+const splitFDStreamSocketDescriptor = 5


Hardcoding this seems a bit fragile, can't we pass it as an env var or a CLI arg?

cgwalters · 2026-02-10T19:20:11Z

storage/pkg/chrootarchive/splitfdstream_linux.go

+// FDs are streamed to the child process one-at-a-time over a Unix socket
+// using SCM_RIGHTS, avoiding EMFILE from inheriting too many FDs at exec.


It doesn't have to be truly one at a time, see also https://github.com/cgwalters/jsonrpc-fdpass?tab=readme-ov-file#41-fd-batching

In my initial PoC work here I handled this by just sending the splitfdstream inline data itself as a fd (could be O_TMPFILE or memfd or pipe), and then sending all the associated fds over jsonrpc-fdpass which automatically handles buffering.

Ensure we processing just 1 recvmsg() at a time (only ~200 fds on Linux) seems really reasonable and unlikely to get anywhere close to modern fd limits.

So if we do choose jsonrpc-fdpass, I think we also need to standardize this "splitfdstream serialization".

cgwalters · 2026-02-10T19:30:29Z

storage/pkg/fileutils/reflink_linux.go

+		return err
+	}
+
+	// Try copy_file_range - kernel-level copy, more efficient than userspace


Surprising we weren't doing this before.

cgwalters · 2026-02-10T19:33:57Z

storage/pkg/splitfdstream/fdpass.go

+
+// ReadLine reads bytes until a newline is encountered, using recvmsg.
+// Any FDs received are closed since line-reading doesn't expect them.
+func (p *FDPasser) ReadLine() ([]byte, error) {


Why would one want this?

cgwalters · 2026-02-10T19:35:02Z

storage/pkg/splitfdstream/protocol.go

+
+// JSON-RPC 2.0 protocol constants
+const (
+	JSONRPCVersion = "2.0"


BTW see also https://github.com/cgwalters/jsonrpc-fdpass-go

Signed-off-by: Giuseppe Scrivano <[email protected]>

Extend the store with splitfdstream capabilities exposed via a UNIX socket for JSON-RPC communication. Signed-off-by: Giuseppe Scrivano <[email protected]>

giuseppe · 2026-02-11T16:13:53Z

still PoC, so you can avoid a review as it is not yet ready but we can use to analyze the complexity.

Moved to use https://github.com/cgwalters/jsonrpc-fdpass-go

Now the code size is almost 50% than the last revision

Signed-off-by: Giuseppe Scrivano <[email protected]>

Implement the SplitFDStreamDriver interface for the overlay driver, enabling efficient layer operations with reflink support. Signed-off-by: Giuseppe Scrivano <[email protected]>

Signed-off-by: Giuseppe Scrivano <[email protected]>

packit-as-a-service · 2026-02-11T16:46:06Z

Packit jobs failed. @containers/packit-build please check.

github-actions bot added the storage Related to "storage" package label Feb 10, 2026

This was referenced Feb 10, 2026

[WIP] composefs-splitfdstream: new crate containers/composefs-rs#228

Draft

reflink by default (if possible) when copying between containers-storage: instances #144

Open

pkg/fileutils: use CopyFileRange if possible

829c910

Signed-off-by: Giuseppe Scrivano <[email protected]>

giuseppe force-pushed the splitfdstream branch 2 times, most recently from 4cca2a3 to 3b6f591 Compare February 10, 2026 14:59

mtrmac reviewed Feb 10, 2026

View reviewed changes

giuseppe force-pushed the splitfdstream branch from 3b6f591 to 26cd4b6 Compare February 10, 2026 16:01

cgwalters reviewed Feb 10, 2026

View reviewed changes

storage/vendor: add github.com/cgwalters/jsonrpc-fdpass-go

7ea722a

Signed-off-by: Giuseppe Scrivano <[email protected]>

giuseppe force-pushed the splitfdstream branch from 26cd4b6 to 23ed544 Compare February 11, 2026 15:50

giuseppe added 2 commits February 11, 2026 17:07

storage/pkg/splitfdstream: new package

e513d9b

Signed-off-by: Giuseppe Scrivano <[email protected]>

storage: add SplitFDStreamStore interface

d1b5faa

Extend the store with splitfdstream capabilities exposed via a UNIX socket for JSON-RPC communication. Signed-off-by: Giuseppe Scrivano <[email protected]>

giuseppe force-pushed the splitfdstream branch from 23ed544 to 8d74f84 Compare February 11, 2026 16:07

giuseppe added 4 commits February 11, 2026 17:25

storage/archive,chrootarchive: add support for splitfdstream

cc6bfee

Signed-off-by: Giuseppe Scrivano <[email protected]>

storage/overlay: implement SplitFDStreamDriver

440830c

Implement the SplitFDStreamDriver interface for the overlay driver, enabling efficient layer operations with reflink support. Signed-off-by: Giuseppe Scrivano <[email protected]>

storage/cmd: new commands json-rpc-server and apply-splitfdstream

0aaa4d5

Signed-off-by: Giuseppe Scrivano <[email protected]>

storage/tests: add tests for splitfdstream

51a6c01

Signed-off-by: Giuseppe Scrivano <[email protected]>

giuseppe force-pushed the splitfdstream branch from 8d74f84 to 51a6c01 Compare February 11, 2026 16:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] storage: add splitfdstream for efficient layer transfer with reflink support #651

[WIP] storage: add splitfdstream for efficient layer transfer with reflink support #651

Uh oh!

giuseppe commented Feb 10, 2026

Uh oh!

mtrmac left a comment

Uh oh!

giuseppe commented Feb 10, 2026

Uh oh!

giuseppe commented Feb 10, 2026

Uh oh!

mtrmac commented Feb 10, 2026

Uh oh!

cgwalters Feb 10, 2026

Uh oh!

cgwalters Feb 10, 2026

Uh oh!

cgwalters Feb 10, 2026

Uh oh!

cgwalters Feb 10, 2026

Uh oh!

cgwalters Feb 10, 2026

Uh oh!

cgwalters Feb 10, 2026

Uh oh!

giuseppe commented Feb 11, 2026

Uh oh!

packit-as-a-service bot commented Feb 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		// FDs are streamed to the child process one-at-a-time over a Unix socket
		// using SCM_RIGHTS, avoiding EMFILE from inheriting too many FDs at exec.

[WIP] storage: add splitfdstream for efficient layer transfer with reflink support #651

Are you sure you want to change the base?

[WIP] storage: add splitfdstream for efficient layer transfer with reflink support #651

Uh oh!

Conversation

giuseppe commented Feb 10, 2026

Uh oh!

mtrmac left a comment

Choose a reason for hiding this comment

Uh oh!

giuseppe commented Feb 10, 2026

Uh oh!

giuseppe commented Feb 10, 2026

Uh oh!

mtrmac commented Feb 10, 2026

Uh oh!

cgwalters Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

cgwalters Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

cgwalters Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

cgwalters Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

cgwalters Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

cgwalters Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

giuseppe commented Feb 11, 2026

Uh oh!

packit-as-a-service bot commented Feb 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants