Compilation on LUMI

[whornsby82]

Hello,
ive now got access to LUMI in Finland, which is a small Frontier. Ive used the Frontier build settings and the modules are the same on LUMI, the loaded modules are:

`Currently Loaded Modules:

1. libfabric/1.15.2.0 6) ModuleLabel/label (S) 11) cray-libsci/23.09.1.1 16) cray-mpich/8.1.27
2. craype-network-ofi 7) lumi-tools/23.11 (S) 12) PrgEnv-cray/8.4.0 17) cray-parallel-netcdf/1.12.3.7
3. perftools-base/23.09.0 8) init-lumi/0.2 (S) 13) craype-accel-amd-gfx90a 18) rocm/5.2.3
4. xpmem/2.5.2-2.4_3.50__gd0f7936.shasta 9) craype/2.7.23 14) craype-x86-milan
5. cce/16.0.1 10) cray-dsmml/0.2.2 15) cray-python/3.10.10

Where:
S: Module is Sticky, requires --force to unload or purge`

And i've set the parameter HIPFORT_DIR=/opt/rocm/hipfort/

But I get the following error when compiling (on the front end)

`ftn -J /users/wihornsby/gacode/modules -homp -hacc -DHIPGPU -DGACODE_GPU_AMD -I/opt/rocm/hipfort///include/hipfort/amdgcn -hacc_model=auto_async_none:no_fast_addr:no_deep_copy -s real64 -Ofast -c cgyro.f90

use cgyro_globals
^
ftn-894 ftn: ERROR CGYRO, File = cgyro.f90, Line = 4, Column = 7
Module "CGYRO_GLOBALS" has compile errors, therefore declarations obtained from the module via the USE statement may be incomplete.

use cgyro_io
^
ftn-894 ftn: ERROR CGYRO, File = cgyro.f90, Line = 5, Column = 7
Module "CGYRO_IO" has compile errors, therefore declarations obtained from the module via the USE statement may be incomplete.`

Do you have any hints? Does one need to compile on a compute node? When I try this there is no rocm module.

Many thanks

[fusionby2030]

Nice you are using Lumi ;)

This may or may not help, but I vaguely recall when compiling a different code on a different CSC cluster (Mahti not Lumi) that the login nodes and the compute nodes had different instruction sets, so I had to set the -march=cpu-type for the compute nodes, here on LUMI the AMD EPYC (https://docs.lumi-supercomputer.eu/hardware/lumig/), but I can't find the type at the moment (https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html), maybe you have more luck.

[whornsby82]

Is it not Trento? Or is that something else?

[whornsby82]

A sorry, I didn't paste the first error that comes up, it is:

`ftn -J /users/wihornsby/gacode/modules -homp -hacc -DHIPGPU -DGACODE_GPU_AMD -I/opt/rocm/hipfort///include/hipfort/amdgcn -hacc_model=auto_async_none:no_fast_addr:no_deep_copy -s real64 -Ofast -c cgyro_globals.F90

module cgyro_globals
^
ftn-855 ftn: ERROR CGYRO_GLOBALS, File = cgyro_globals.F90, Line = 9, Column = 8
The compiler has detected errors in module "CGYRO_GLOBALS". No module information file will be created for this module.

use hipfort_hipfft
^
ftn-1644 ftn: ERROR CGYRO_GLOBALS, File = cgyro_globals.F90, Line = 19, Column = 7
File "/opt/rocm/hipfort///include/hipfort/amdgcn/hipfort_hipfft.mod" is not compatible with this compiler or is corrupt. The compiler expected to find information about module "HIPFORT_HIPFFT".
^
ftn-292 ftn: ERROR CGYRO_GLOBALS, File = cgyro_globals.F90, Line = 19, Column = 7
"HIPFORT_HIPFFT" is specified as the module name on a USE statement, but the compiler cannot find it.

Cray Fortran : Version 16.0.1 (20230807191643_6d4824324d375100ba18ca639dfc956fe6546d06)
Cray Fortran : Compile time: 0.0471 seconds
Cray Fortran : 526 source lines
Cray Fortran : 3 errors, 0 warnings, 0 other messages, 0 ansi
Cray Fortran : "explain ftn-message number" gives more information about each message.

[jcandy]

@sfiligoi Any thoughts?

@whornsby82 We use a "private" build of hipfort on Frontier

[sfiligoi]

Yeah... the hipfort libraries and header files should come from your local system setup.
Just like you would expect fftw to come from the local system on CPU-only setups.

On Frontier it was provided by their sysadmins.
(And it is "private" in the sense that not many other users seem to care about it)

[whornsby82]

You mean compile my own? The rocm install in the modules has the hipfort modules there.

[sfiligoi]

@whornsby82
You need the Fortran bindings (compiled with the same compiler used by CGYRO, i.e. the Cray Compilers).
I suspect those are not built by default.
(If they are, maybe you just need a proper include?)

As said above, we did not do that by ourselves.
The OLCF staff did it for us.
(and we are pointing the includes on Frontier to where they told us to find them)

[jcandy]

@reubendb would you have some advice here?

[reubendb]

I can provide build instruction or build script so that you can build Hipfort on Lumi. It should be fairly similar to Frontier (I built the one on Frontier).

Please don't use the hipfort module that comes with ROCm (e.g. in /opt/rocm). They were built with a different compiler and would not be compatible.

Is there newer ROCm in Lumi by any chance? 5.2.3 is pretty old (the hipfort version to build will have to match the rocm version.)

[jcandy]

Thanks very much for the help @reubendb

[whornsby82]

Thanks for the help everyone.

@reubendb Yes, 5.2.3 seems to be the only version on there

[whornsby82]

Ah, sorry I'm incorrect there, we have available versions:
5.2.5
5.3.3
5.4.6
5.6.1

[reubendb]

@whornsby82 Sorry for the delay. I actually already have Hipfort build that should work for CCE-16.0.1 with ROCm 5.6.1 on Frontier. So maybe that's the easiest to try. You can download a tarball from this link (Google Drive):
https://drive.google.com/file/d/1PhB1P-eNG14UCXqoeASWSDle3PPNnkoz/view?usp=drive_link

Untar it and point the HIPFORT_DIR to the directory. Could you see if that would work, and if not, copy-paste the build error here?
And just to be sure, you have to module swap to use ROCM 5.6.1 on Lumi.

Thanks,
Reuben

[whornsby82]

Hello Reuben,

many thanks for the help. Ive uploaded it and pointed the build to it, it seems to have fixed the previous issue. Now I get:

make[1]: Entering directory '/pfs/lustrep3/users/wihornsby/gacode/cgyro/src'
ftn -J /users/wihornsby/gacode/modules -homp -hacc -DHIPGPU -DGACODE_GPU_AMD -I/scratch/project_462000507/cce_16.0.1-rocm_5
.6.0/include/hipfort/amdgcn -hacc_model=auto_async_none:no_fast_addr:no_deep_copy -s real64 -Ofast -c cgyro_nl_fftw.f90
include 'fftw3.f03'
ftn-63 ftn: ERROR CGYRO_NL_FFTW_STEPR, File = cgyro_nl_fftw.f90, Line = 27
The compiler cannot open INCLUDE file "fftw3.f03".

The modules that ive got loaded are:

Currently Loaded Modules:

1. perftools-base/23.09.0 8) cray-libsci/23.09.1.1 15) craype-accel-amd-gfx90a
2. cce/16.0.1 9) PrgEnv-cray/8.4.0 16) libfabric/1.15.2.0
3. ModuleLabel/label (S) 10) cray-python/3.10.10 17) craype-network-ofi
4. lumi-tools/23.11 (S) 11) cray-mpich/8.1.27 18) xpmem/2.5.2-2.4_3.50__gd0f7936.shasta
5. init-lumi/0.2 (S) 12) cray-parallel-netcdf/1.12.3.7 19) partition/G (S)
6. craype/2.7.23 13) LUMI/23.09 (S) 20) rocm/5.6.1
7. cray-dsmml/0.2.2 14) craype-x86-trento

[reubendb]

Hi William, FFTW should be provided by the system, could you try module load cray-fftw?