From e41bbef81757bb30702be2f31211fe64069e0e5f Mon Sep 17 00:00:00 2001 From: Pranab Das <31024886+pranabdas@users.noreply.github.com> Date: Wed, 17 Dec 2025 16:39:43 +0800 Subject: [PATCH 1/4] SOF-7764: how to build containerized applications with gpu support --- lang/en/docs/cli/actions/add-software.md | 79 +++++++++++++++++++----- 1 file changed, 65 insertions(+), 14 deletions(-) diff --git a/lang/en/docs/cli/actions/add-software.md b/lang/en/docs/cli/actions/add-software.md index 0860d23a..82d03aa3 100644 --- a/lang/en/docs/cli/actions/add-software.md +++ b/lang/en/docs/cli/actions/add-software.md @@ -28,7 +28,7 @@ from the AlmaLinux 9 Docker image to a subdirectory named `qe_sandbox`. Now, to install packages and save them to the sandbox folder, we can enter into the container in shell (interactive) mode with write permission (use `--writable` or `-w` flag). We will also need `--fakeroot` or `-f` flag to -Install software as root inside the container: +install software as root inside the container: ```bash apptainer shell --writable --fakeroot qe_sandbox/ @@ -54,8 +54,8 @@ We may either package the sandbox directory into a final image: apptainer build -f espresso.sif qe_sandbox/ ``` -After the container is built and saved as an SIF image, we may delete our -sandbox folder. We need to set appropriate permissions to be able to delete: +After the container is built and saved as SIF image, we may delete our sandbox +folder. We need to set appropriate permissions to be able to delete: ```bash chmod -R u+rwX qe_sandbox @@ -134,9 +134,14 @@ along with its dependencies. 4. Set runtime environment variables 5. Build routine, under the `post` section +Now we are ready to build the container with: +```bash +apptainer build espresso.sif espresso.def +``` + ### Build Considerations -#### Running resource-intensive builds in batch mode +#### **Running resource-intensive builds in batch mode** Prototyping the build is convenient using sandbox mode, but when the routines are clear and the `.def` file is ready, we suggest that users submit a @@ -160,14 +165,66 @@ cd $PBS_O_WORKDIR apptainer build espresso.sif espresso.def ``` -#### Porting large libraries from the host +#### **Porting large libraries from the host** -Large libraries such as the Intel OneAPI suite and NVIDIA HPC SDK, which are +Large libraries such as the Intel OneAPI suite, NVIDIA HPC SDK, which are several gigabytes in size, can be mapped from our cluster host instead of bundling together with the application. However, this is not applicable if one needs a different version of these libraries than the one provided. +### How to build containers with GPU support + +In order to run applications with GPU acceleration, first we need to compile the +GPU code with appropriate GPU libraries, which is done during the container +build phase. Here we will describe how we can compile our application code using +NVIDIA HPC SDK (which includes CUDA libraries) and package the compiled code as +a containerized application. The process works even on systems without GPU +devices or drivers, thanks to the availability of dummy shared objects (e.g., +`libcuda.so`) in recent versions of the NVHPC SDK and CUDA Toolkit. These dummy +libraries allow the linker to complete compilation without requiring an actual +GPU. + +As we mentioned in the above paragraph, NVIDIA HPC SDK (or CUDA Toolkit) is a +large package, typically several gigabytes in size. Unless a specific version of +CUDA is required, it’s more efficient to map the NVHPC installation available on +the host cluster. Currently, NVHPC 25.3 with CUDA 12.8 is installed in the +Mat3ra clusters. This version matches the NVIDIA driver version on the cluster's +compute nodes. + +We build our GPU containers in two stages: + +1. **Base Image and Compilation Stage**: Start from a base image, install +NVHPC, and other dependencies. Then, compile the application code. +2. **Minimized Production Image**: Create a slim container by copying the +compiled code and the smaller required dependencies into a new base image, +omitting the NVHPC SDK. + +To run such container, we must `--bind` the NVHPC paths from the host and set +appropriate `PATH` and `LD_LIBRARY_PATH` for apptainer. Specialized software +libraries are installed under `/export/compute/software` in Mat3ra clusters. +Also, to map the NVIDIA GPU drivers from compute node, we must use `--nv` flag. +Now, to set `PATH` inside apptainer, we can set `APPTAINERENV_PREPEND_PATH` (or +`APPTAINERENV_APPEND_PATH`) on the host. However, for other ENV variables such +special Apptainer variable is not present, so we can use `APPTAINERENV_` prefix +for them. So a typical job script would look like: +```bash +export APPTAINERENV_PREPEND_PATH="/export/compute/software/libraries/nvhpc-25.3-cuda-12.8/Linux_x86_64/25.3/comm_libs/12.8/hpcx/hpcx-2.22.1/hcoll/bin:/export/compute/software/libraries/nvhpc-25.3-cuda-12.8/Linux_x86_64/25.3/comm_libs/12.8/hpcx/hpcx-2.22.1/ompi/bin:/export/compute/software/libraries/nvhpc-25.3-cuda-12.8/Linux_x86_64/25.3/comm_libs/12.8/hpcx/hpcx-2.22.1/ucx/mt/bin:/export/compute/software/compilers/gcc/11.2.0/bin" + +export APPTAINERENV_LD_LIBRARY_PATH="/export/compute/software/libraries/nvhpc-25.3-cuda-12.8/Linux_x86_64/25.3/comm_libs/12.8/hpcx/hpcx-2.22.1/hcoll/lib:/export/compute/software/libraries/nvhpc-25.3-cuda-12.8/Linux_x86_64/25.3/comm_libs/12.8/hpcx/hpcx-2.22.1/ompi/lib:/export/compute/software/libraries/nvhpc-25.3-cuda-12.8/Linux_x86_64/25.3/comm_libs/12.8/hpcx/hpcx-2.22.1/nccl_rdma_sharp_plugin/lib:/export/compute/software/libraries/nvhpc-25.3-cuda-12.8/Linux_x86_64/25.3/comm_libs/12.8/hpcx/hpcx-2.22.1/sharp/lib:/export/compute/software/libraries/nvhpc-25.3-cuda-12.8/Linux_x86_64/25.3/comm_libs/12.8/hpcx/hpcx-2.22.1/ucx/mt/lib:/export/compute/software/libraries/nvhpc-25.3-cuda-12.8/Linux_x86_64/25.3/comm_libs/12.8/hpcx/hpcx-2.22.1/ucx/mt/lib/ucx:/export/compute/software/libraries/nvhpc-25.3-cuda-12.8/Linux_x86_64/25.3/comm_libs/12.8/nccl/lib:/export/compute/software/libraries/nvhpc-25.3-cuda-12.8/Linux_x86_64/25.3/compilers/lib:/export/compute/software/libraries/nvhpc-25.3-cuda-12.8/Linux_x86_64/25.3/cuda/12.8/lib64:/export/compute/software/libraries/nvhpc-25.3-cuda-12.8/Linux_x86_64/25.3/math_libs/12.8/lib64:/export/compute/software/compilers/gcc/11.2.0/lib64:\${LD_LIBRARY_PATH}" + +apptainer exec --nv --bind /export,/cluster-001-share pw.x -in pw.in > pw.out +``` + +You may inspect modulefiles (e.g., +`/cluster-001-share/compute/modulefiles/applications/espresso/7.4.1-cuda-12.8 `) +available in Mat3ra clusters and [job scripts]( +https://github.com/Exabyte-io/cli-job-examples/blob/main/espresso/gpu/job.gpu.pbs) +to see how it is implemented in Mat3ra provided containers. Do not forget to use +a GPU enabled queue, such as [GOF](../../infrastructure/clusters/google.md) to +submit your GPU jobs. + + ## Run jobs using Apptainer Once the container is built, we are ready to run applications packaged in it. A @@ -214,14 +271,8 @@ You can build containers on your local machine or use pull pre-built ones from sources such as [NVIDIA GPU Cloud]( https://catalog.ngc.nvidia.com/orgs/hpc/containers/quantum_espresso). -If Apptainer is installed locally, build the container using: - -```bash -apptainer build espresso.sif espresso.def -``` - -Once built, you can push the image to a container registry such as the -[GitHub Container Registry]( +If the container is build locally, you can push the image to a container +registry such as the [GitHub Container Registry]( https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry). ```bash From ee141cbc0f5de75cfc75a6b2deab9919d9738581 Mon Sep 17 00:00:00 2001 From: Pranab Das <31024886+pranabdas@users.noreply.github.com> Date: Wed, 17 Dec 2025 16:54:58 +0800 Subject: [PATCH 2/4] chore: wording --- lang/en/docs/cli/actions/add-software.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/lang/en/docs/cli/actions/add-software.md b/lang/en/docs/cli/actions/add-software.md index 82d03aa3..2826574f 100644 --- a/lang/en/docs/cli/actions/add-software.md +++ b/lang/en/docs/cli/actions/add-software.md @@ -185,20 +185,20 @@ devices or drivers, thanks to the availability of dummy shared objects (e.g., libraries allow the linker to complete compilation without requiring an actual GPU. -As we mentioned in the above paragraph, NVIDIA HPC SDK (or CUDA Toolkit) is a -large package, typically several gigabytes in size. Unless a specific version of -CUDA is required, it’s more efficient to map the NVHPC installation available on +As mentioned the above, NVIDIA HPC SDK (or CUDA Toolkit) is a large package, +typically several gigabytes in size. Unless a specific version of CUDA is +required, it’s more efficient to map the NVHPC installation available on the host cluster. Currently, NVHPC 25.3 with CUDA 12.8 is installed in the Mat3ra clusters. This version matches the NVIDIA driver version on the cluster's compute nodes. We build our GPU containers in two stages: -1. **Base Image and Compilation Stage**: Start from a base image, install -NVHPC, and other dependencies. Then, compile the application code. -2. **Minimized Production Image**: Create a slim container by copying the -compiled code and the smaller required dependencies into a new base image, -omitting the NVHPC SDK. +1. **Base Image and Compilation Stage**: Install NVHPC and all other +dependencies, and compile the application code. +2. **Slim Production Image**: Create a final production container by copying +only the compiled application and smaller dependencies (if any) into a new base +image, omitting the NVHPC SDK. To run such container, we must `--bind` the NVHPC paths from the host and set appropriate `PATH` and `LD_LIBRARY_PATH` for apptainer. Specialized software From 669cba77829f3bcab2b017ebad4f176615f0d79c Mon Sep 17 00:00:00 2001 From: Timur Bazhirov Date: Wed, 17 Dec 2025 16:56:46 -0800 Subject: [PATCH 3/4] Update add-software.md --- lang/en/docs/cli/actions/add-software.md | 70 +++++++++++++----------- 1 file changed, 39 insertions(+), 31 deletions(-) diff --git a/lang/en/docs/cli/actions/add-software.md b/lang/en/docs/cli/actions/add-software.md index 2826574f..3d66e4eb 100644 --- a/lang/en/docs/cli/actions/add-software.md +++ b/lang/en/docs/cli/actions/add-software.md @@ -2,22 +2,23 @@ ## Overview Users can compile their own software via the -[Command Line Interface](../overview.md) (CLI). This is helpful, for example, -after introducing some changes or patches to the source code, or if users need +[Command Line Interface](../overview.md) (CLI). This is helpful if users need to run a specific version of an application that is not installed "globally". -Most of the globally installed applications are currently distributed as +The globally installed applications are currently distributed as Apptainer[^1] (Singularity[^2]) containers, bundled with all required dependencies. This ensures that each application is isolated and avoids -dependency conflicts. If you plan to run an application that is not installed in -our cluster, we encourage you to package your code and its dependencies as an -Apptainer/Singularity container. If you already have a Docker image, it -can be converted into an Apptainer/Singularity image. +dependency conflicts. -## Experiment in Sandbox mode +When planning to run an application that is not installed in +our cluster, we encourage packaging code and its dependencies as an +Apptainer/Singularity container. Existing Docker images +can be converted into an Apptainer/Singularity images. +## Using Sandbox mode Apptainer's sandbox mode is helpful for testing and fine-tuning the build steps interactively. To start it, first initialize a sandbox with `--sandbox` or `-s` flag: + ```bash apptainer build --sandbox qe_sandbox/ docker://almalinux:9 ``` @@ -45,16 +46,16 @@ Once you are happy with the sandbox, have tested the build steps, and installed everything you need, `exit` from the Apptainer shell mode. -## Build container +## Building containers -### Build from a sandbox folder +### Build from a Sandbox folder We may either package the sandbox directory into a final image: ```bash apptainer build -f espresso.sif qe_sandbox/ ``` -After the container is built and saved as SIF image, we may delete our sandbox +After the container is built and saved as an SIF image, we may delete our sandbox folder. We need to set appropriate permissions to be able to delete: ```bash @@ -141,7 +142,7 @@ apptainer build espresso.sif espresso.def ### Build Considerations -#### **Running resource-intensive builds in batch mode** +#### Running resource-intensive builds in batch mode Prototyping the build is convenient using sandbox mode, but when the routines are clear and the `.def` file is ready, we suggest that users submit a @@ -165,27 +166,34 @@ cd $PBS_O_WORKDIR apptainer build espresso.sif espresso.def ``` -#### **Porting large libraries from the host** +#### Porting large libraries from the host -Large libraries such as the Intel OneAPI suite, NVIDIA HPC SDK, which are -several gigabytes in size, can be mapped from our cluster host instead of +Large libraries such as the Intel OneAPI suite and NVIDIA HPC SDK, which are +several gigabytes in size, can be mapped from the cluster host instead of bundling together with the application. However, this is not applicable if one needs a different version of these libraries than the one provided. +This can be done by using the `--bind` directives and passing the appropriate +library location from the host, e.g., from `/cluster-001-share/compute/software/libraries` +or `/export/compute/software/libraries/`. + +See the GPU example below for more details. -### How to build containers with GPU support +#### Building containers with GPU support -In order to run applications with GPU acceleration, first we need to compile the -GPU code with appropriate GPU libraries, which is done during the container -build phase. Here we will describe how we can compile our application code using +To run applications with GPU acceleration, first, we need to compile the +GPU code with appropriate GPU libraries used, which is done during the container +build phase. Here, we will describe how we can compile our application code using NVIDIA HPC SDK (which includes CUDA libraries) and package the compiled code as -a containerized application. The process works even on systems without GPU -devices or drivers, thanks to the availability of dummy shared objects (e.g., +a containerized application. + +The process works even on systems without GPU devices or drivers, +thanks to the availability of dummy shared objects (e.g., `libcuda.so`) in recent versions of the NVHPC SDK and CUDA Toolkit. These dummy libraries allow the linker to complete compilation without requiring an actual GPU. -As mentioned the above, NVIDIA HPC SDK (or CUDA Toolkit) is a large package, +NVIDIA HPC SDK (or CUDA Toolkit) is a large package, typically several gigabytes in size. Unless a specific version of CUDA is required, it’s more efficient to map the NVHPC installation available on the host cluster. Currently, NVHPC 25.3 with CUDA 12.8 is installed in the @@ -200,14 +208,15 @@ dependencies, and compile the application code. only the compiled application and smaller dependencies (if any) into a new base image, omitting the NVHPC SDK. -To run such container, we must `--bind` the NVHPC paths from the host and set +To run such a container, we must `--bind` the NVHPC paths from the host and set appropriate `PATH` and `LD_LIBRARY_PATH` for apptainer. Specialized software libraries are installed under `/export/compute/software` in Mat3ra clusters. -Also, to map the NVIDIA GPU drivers from compute node, we must use `--nv` flag. +Also, to map the NVIDIA GPU drivers from the compute node, we must use the `--nv` flag. Now, to set `PATH` inside apptainer, we can set `APPTAINERENV_PREPEND_PATH` (or -`APPTAINERENV_APPEND_PATH`) on the host. However, for other ENV variables such -special Apptainer variable is not present, so we can use `APPTAINERENV_` prefix +`APPTAINERENV_APPEND_PATH`) on the host. However, for other ENV variables, such +special Apptainer variables, are not present, so we can use the `APPTAINERENV_` prefix for them. So a typical job script would look like: + ```bash export APPTAINERENV_PREPEND_PATH="/export/compute/software/libraries/nvhpc-25.3-cuda-12.8/Linux_x86_64/25.3/comm_libs/12.8/hpcx/hpcx-2.22.1/hcoll/bin:/export/compute/software/libraries/nvhpc-25.3-cuda-12.8/Linux_x86_64/25.3/comm_libs/12.8/hpcx/hpcx-2.22.1/ompi/bin:/export/compute/software/libraries/nvhpc-25.3-cuda-12.8/Linux_x86_64/25.3/comm_libs/12.8/hpcx/hpcx-2.22.1/ucx/mt/bin:/export/compute/software/compilers/gcc/11.2.0/bin" @@ -216,13 +225,12 @@ export APPTAINERENV_LD_LIBRARY_PATH="/export/compute/software/libraries/nvhpc-25 apptainer exec --nv --bind /export,/cluster-001-share pw.x -in pw.in > pw.out ``` -You may inspect modulefiles (e.g., +To understand the details about library paths, one may inspect modulefiles (e.g., `/cluster-001-share/compute/modulefiles/applications/espresso/7.4.1-cuda-12.8 `) -available in Mat3ra clusters and [job scripts]( +available in our clusters and [job scripts]( https://github.com/Exabyte-io/cli-job-examples/blob/main/espresso/gpu/job.gpu.pbs) -to see how it is implemented in Mat3ra provided containers. Do not forget to use -a GPU enabled queue, such as [GOF](../../infrastructure/clusters/google.md) to -submit your GPU jobs. +to see how it is implemented. Do not forget to use a GPU-enabled queue, +such as [GOF](../../infrastructure/clusters/google.md) to submit your GPU jobs. ## Run jobs using Apptainer From 89972188f567744b8c82307b35965561c22be9d1 Mon Sep 17 00:00:00 2001 From: Pranab Das <31024886+pranabdas@users.noreply.github.com> Date: Thu, 18 Dec 2025 09:06:51 +0800 Subject: [PATCH 4/4] chore: formatting --- lang/en/docs/cli/actions/add-software.md | 36 +++++++++++++----------- 1 file changed, 19 insertions(+), 17 deletions(-) diff --git a/lang/en/docs/cli/actions/add-software.md b/lang/en/docs/cli/actions/add-software.md index 3d66e4eb..5594d67c 100644 --- a/lang/en/docs/cli/actions/add-software.md +++ b/lang/en/docs/cli/actions/add-software.md @@ -7,7 +7,7 @@ to run a specific version of an application that is not installed "globally". The globally installed applications are currently distributed as Apptainer[^1] (Singularity[^2]) containers, bundled with all required dependencies. This ensures that each application is isolated and avoids -dependency conflicts. +dependency conflicts. When planning to run an application that is not installed in our cluster, we encourage packaging code and its dependencies as an @@ -55,8 +55,8 @@ We may either package the sandbox directory into a final image: apptainer build -f espresso.sif qe_sandbox/ ``` -After the container is built and saved as an SIF image, we may delete our sandbox -folder. We need to set appropriate permissions to be able to delete: +After the container is built and saved as an SIF image, we may delete our +sandbox folder. We need to set appropriate permissions to be able to delete: ```bash chmod -R u+rwX qe_sandbox @@ -174,8 +174,9 @@ bundling together with the application. However, this is not applicable if one needs a different version of these libraries than the one provided. This can be done by using the `--bind` directives and passing the appropriate -library location from the host, e.g., from `/cluster-001-share/compute/software/libraries` -or `/export/compute/software/libraries/`. +library location from the host, e.g., from +`/cluster-001-share/compute/software/libraries` or +`/export/compute/software/libraries/`. See the GPU example below for more details. @@ -183,11 +184,11 @@ See the GPU example below for more details. To run applications with GPU acceleration, first, we need to compile the GPU code with appropriate GPU libraries used, which is done during the container -build phase. Here, we will describe how we can compile our application code using -NVIDIA HPC SDK (which includes CUDA libraries) and package the compiled code as -a containerized application. +build phase. Here, we will describe how we can compile our application code +using NVIDIA HPC SDK (which includes CUDA libraries) and package the compiled +code as a containerized application. -The process works even on systems without GPU devices or drivers, +The process works even on systems without GPU devices or drivers, thanks to the availability of dummy shared objects (e.g., `libcuda.so`) in recent versions of the NVHPC SDK and CUDA Toolkit. These dummy libraries allow the linker to complete compilation without requiring an actual @@ -211,11 +212,12 @@ image, omitting the NVHPC SDK. To run such a container, we must `--bind` the NVHPC paths from the host and set appropriate `PATH` and `LD_LIBRARY_PATH` for apptainer. Specialized software libraries are installed under `/export/compute/software` in Mat3ra clusters. -Also, to map the NVIDIA GPU drivers from the compute node, we must use the `--nv` flag. -Now, to set `PATH` inside apptainer, we can set `APPTAINERENV_PREPEND_PATH` (or -`APPTAINERENV_APPEND_PATH`) on the host. However, for other ENV variables, such -special Apptainer variables, are not present, so we can use the `APPTAINERENV_` prefix -for them. So a typical job script would look like: +Also, to map the NVIDIA GPU drivers from the compute node, we must use the +`--nv` flag. Now, to set `PATH` inside apptainer, we can set +`APPTAINERENV_PREPEND_PATH` (or `APPTAINERENV_APPEND_PATH`) on the host. +However, for other ENV variables, such special Apptainer variables are not +present, so we can use the `APPTAINERENV_` prefix for them. So a typical job +script would look like: ```bash export APPTAINERENV_PREPEND_PATH="/export/compute/software/libraries/nvhpc-25.3-cuda-12.8/Linux_x86_64/25.3/comm_libs/12.8/hpcx/hpcx-2.22.1/hcoll/bin:/export/compute/software/libraries/nvhpc-25.3-cuda-12.8/Linux_x86_64/25.3/comm_libs/12.8/hpcx/hpcx-2.22.1/ompi/bin:/export/compute/software/libraries/nvhpc-25.3-cuda-12.8/Linux_x86_64/25.3/comm_libs/12.8/hpcx/hpcx-2.22.1/ucx/mt/bin:/export/compute/software/compilers/gcc/11.2.0/bin" @@ -229,7 +231,7 @@ To understand the details about library paths, one may inspect modulefiles (e.g. `/cluster-001-share/compute/modulefiles/applications/espresso/7.4.1-cuda-12.8 `) available in our clusters and [job scripts]( https://github.com/Exabyte-io/cli-job-examples/blob/main/espresso/gpu/job.gpu.pbs) -to see how it is implemented. Do not forget to use a GPU-enabled queue, +to see how it is implemented. Do not forget to use a GPU-enabled queue, such as [GOF](../../infrastructure/clusters/google.md) to submit your GPU jobs. @@ -295,8 +297,8 @@ apptainer pull oras://ghcr.io///: