QUICK25.03 AmberTools25 HIP compatibility issue and performance regression #433
Description
Update: I was able to compile QUICK-25.03 together with sander by replacing the bundled QUICK in AmberTools23 with the latest release from GitHub. The build completed successfully. However, I observed a significant performance drop: ~3 ps/day compared to ~15 ps/day with the previous QUICK version included in AmberTools23. This slowdown persists both with and without MAGMA enabled. (Due to HPC update I have no access to old versions of ROCM anymore)
Dear Amber Team,
I’m trying to build AmberTools 25 with QUICK (HIP backend) on the Setonix supercomputer (AMD MI250 / gfx90a, ROCm 6.2.1). Below is a concise summary of what we’ve run into and what we’ve tried.
System
• GPU: AMD MI250 (gfx90a)
• ROCm: 6.2.1 (also briefly tried 6.4.1)
• Compilers: GCC 14.3.x and hipcc from ROCm
• Target: AmberTools 25 with QUICK-25.03, HIP enabled
Issues and workarounds so far
1. HIP/CUDA path mixing
Even with HIP=ON and CUDA=FALSE, the build still tried to compile CUDA-only code (e.g., cpptraj/src/cuda_kernels using cudaMalloc/dim3). This subsided only after we passed all HIP-related paths explicitly and guarded QUICK’s CUDA logic. We now configure with:
-DMPI=FALSE -DCUDA=OFF -DINSTALL_TESTS=FALSE
-DHIP=ON
-DROCM_PATH=/software/setonix/rocm/6.2.1
-DHIP_TOOLKIT_ROOT_DIR=/software/setonix/rocm/6.2.1
-DHIP_HIPCC_EXECUTABLE=/software/setonix/rocm/6.2.1/bin/hipcc
-DCMAKE_HIP_COMPILER=/software/setonix/rocm/6.2.1/bin/hipcc
-DHIP_ROOT_DIR=/software/setonix/rocm/6.2.1
-DBUILD_QUICK=TRUE
In QUICK’s CMake we also added an explicit guard “if(CUDA AND NOT HIP)” and a small safety check to avoid re-creating imported targets (“if(TARGET ) return()”).
2. HIP/ROCm version detection failure (root cause, including rocBLAS module path)
On our system the ROCm install path is purely numeric (…/6.2.1). In this layout the scripts resolved HIP_VERSION as “..”, so version comparisons failed and downstream logic misfired. This also led to a wrong assumption about the rocBLAS Fortran module location (expecting ${ROCM_PATH}/rocblas/include/), whereas on ROCm 6.x it is actually under ${ROCM_PATH}/include/rocblas/rocblas_module.f90.
Workaround: we explicitly set HIP_VERSION=6.2.1 and adjusted the module path to the ROCm 6.x layout; with these fixes the configuration proceeds further.
3. ROCm version guard in QUICK
QUICKCudaConfig.cmake still aborts for HIP_VERSION ≥ 5.4.3 due to a historical compiler bug. Since ROCm 6.2.1 is expected to be supported again, we temporarily disabled the FATAL_ERROR to continue.
4. nvcc-only flags under HIP
We hit “clang++: error: unknown argument: –compiler-options” because several places still inject “–compiler-options -fPIC”. We replaced those with plain “-fPIC” and/or bypassed the CUDA blocks when HIP is on. We also excluded cpptraj’s CUDA kernels in the HIP build to avoid compiling CUDA-only APIs yet this lead to no makefiles generated.
Current status
• CMake now completes (with warnings) and proceeds much further.
• The build succeeds only if CUDA-specific pieces (e.g., cpptraj CUDA) are excluded or guarded under NOT HIP.
• It is unclear whether HIP builds are fully supported when QUICK is built inside AmberTools 25 (as opposed to standalone QUICK).
Questions
• Is HIP on ROCm 6.2.1 (gfx90a) expected to work for QUICK when built within AmberTools 25?
• Are there updated CMake patches or instructions (covering robust HIP version detection and the ROCm 6.x rocBLAS Fortran module path)?
• For HIP builds, should CUDA-related subcomponents (e.g., cpptraj CUDA) be explicitly disabled?
Thank you very much for your help and guidance.
Sincerely yours
Yuchen