Support extracting remote tarballs when used as named contexts with Bake

[dvdksn]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

When using a remote tarball (e.g. GitHub release archives) as a context or contexts entry in bake, the tarball is downloaded but not extracted.

Expected behaviour

Remote tarballs should be extracted and the contents should be used as a context. 1
Have Bake detect and extract tarball archives so that tarballs can be used as contexts directly. Similar to how build handles it:

• buildx/build/opt.go

  Lines 399 to 422 in a6e198a

  	rc := inp.InStream.NewReadCloser()
  	magic, err := inp.InStream.Peek(archiveHeaderSize * 2)
  	if err != nil && err != io.EOF {
  	return nil, errors.Wrap(err, "failed to peek context header from STDIN")
  	}
  	if err != io.EOF || len(magic) != 0 {
  	if isArchive(magic) {
  	// stdin is context
  	up := uploadprovider.New()
  	target.FrontendAttrs["context"] = up.Add(rc)
  	target.Session = append(target.Session, up)
  	} else {
  	if inp.DockerfilePath != "" {
  	return nil, errors.Errorf("ambiguous Dockerfile source: both stdin and flag correspond to Dockerfiles")
  	}
  	// stdin is dockerfile
  	dockerfileReader = rc
  	inp.ContextPath, _ = os.MkdirTemp("", "empty-dir")
  	toRemove = append(toRemove, inp.ContextPath)
  	if err := setLocalMount("context", inp.ContextPath, target); err != nil {
  	return nil, err
  	}
  	}
  	}

Actual behaviour

The tarball is downloaded but not extracted.

Buildx version

github.com/docker/buildx v0.26.1-desktop.1 532a478c2ea39e2d0eb40ad2e3f6bec57df4c8af

Docker info

Builders list

docker buildx ls
NAME/NODE           DRIVER/ENDPOINT     STATUS     BUILDKIT   PLATFORMS
container           docker-container                          
 \_ container0       \_ desktop-linux   inactive              
default             docker                                    
 \_ default          \_ default         running    v0.23.2    linux/amd64 (+2), linux/arm64, linux/ppc64le, linux/s390x, (2 more)
desktop-linux*      docker                                    
 \_ desktop-linux    \_ desktop-linux   running    v0.23.2    linux/amd64 (+2), linux/arm64, linux/ppc64le, linux/s390x, (2 more)

Configuration

target "default" {
  contexts = {
    buildx = "https://github.com/docker/buildx/archive/refs/tags/v0.27.0.tar.gz"
  }
  dockerfile-inline = <<EOT
FROM alpine:3.22
COPY --from=buildx . .
RUN ls -l && stop
EOT
}

Build logs

Additional info

• Note: GitHub release tarballs 302-redirect to a codeload.github.com URL.

• Even using the codeload URL directly still results in just the compressed tarball being available.

• It looks like Bake just sets the context state to the file rather than extracting it:

  • buildx/bake/bake.go

    Lines 1291 to 1292 in a6e198a

    	st := llb.Scratch().File(llb.Copy(*inp.State, v.Path, "/"), llb.WithCustomNamef("set context %s to %s", k, v.Path))
    	t.NamedContexts[k] = build.NamedContext{State: &st}

  • buildx/bake/bake.go

    Lines 1305 to 1310 in a6e198a

    	st := llb.Scratch().File(
    	llb.Copy(*inp.State, t.ContextPath, "/", &llb.CopyInfo{
    	CopyDirContentsOnly: true,
    	}),
    	llb.WithCustomNamef("set context to %s", t.ContextPath),
    	)

[tonistiigi]

The http(s) URLs in named contexts should have probably only supported tar.gz like the build command arg1 allows. Instead this was modeled after the ADD command. I'm not sure how we could change it now though after we have already shipped it with blob semantics.

I don't think there is anything bake-specific in here, though. It just seems to be how named contexts are handled in BuildKit/Dockerfile.

[crazy-max]

I don't think there is anything bake-specific in here, though. It just seems to be how named contexts are handled in BuildKit/Dockerfile.

Yes indeed looks handled in https://github.com/moby/buildkit/blob/070d993324ee4f90c0517ca70229dcffa37c270a/frontend/dockerui/namedcontext.go#L149-L150

The http(s) URLs in named contexts should have probably only supported tar.gz like the build command arg1 allows. Instead this was modeled after the ADD command. I'm not sure how we could change it now though after we have already shipped it with blob semantics.

Yes it might break users that don't expect tarball to be unpacked. Wonder if we could handle this through hash tag maybe?:

target "default" {
  contexts = {
    buildx = "https://github.com/docker/buildx/archive/refs/tags/v0.27.0.tar.gz#buildkit-unpack=true"
  }
  dockerfile-inline = <<EOT
FROM alpine:3.22
COPY --from=buildx . .
RUN ls -l && stop
EOT
}

[dvdksn]

I see! I only tried with Bake which lead me to believe it was something specific to this. Configurability via hash tag makes sense since there is precedence in git urls. Maybe key-value could also work: --build-context "buildx=https://github.com/docker/buildx/archive/refs/tags/v0.27.0.tar.gz,unpack=true"