Add checkpoint file functionality (native, HDF5 formats)

[vtripath65]

Calculations can be restarted from checkpoint files containing coordinates and/or density.

Keywords can be read from multiple lines.

Closes #387.

[vtripath65]

This branch is not ready to merge yet.

[vtripath65]

git workflow needs to install HDF5 for this branch to pass tests (I think)

[ohearnk]

@vtripath65: I believe that my previous commits should address the missing HDF5 install dependencies for Github CI tests. The legacy build system failures still need to be addressed. Please review the test artifacts for CMake builds once the updated code in my previous commits is run.

Second times a charm. New commits should hopefully fix the CMake build issues.

Also, we should probably make HDF5 an optional install dependency for users. This may require a bit of code tweaking with conditional compilation -- i.e., if HDF5 is found at configure time, then a preprocessor directive gets set which enables HDF5-specific code to be compiled. Else, code with errors messages indicating HDF5 is not installed gets compiled.

[ohearnk]

At this point, I'm convinced that there's something wrong with the HDF5 libraries built for Ubuntu v24.04 (specifically for the Fortran bindings), as all failing MPI tests error out on boilerplate HDF5 code to initialize the Fortran interface.

@agoetz, @vtripath65 -- if either of you have any ideas here, I'd appreciate it. If not, I'm inclined to change the Github workflows to not test HDF5-based checkpoint functionality on Ubuntu v24.04 -- this is easy to do given how the workflow YAML files are written (and is currently done for tests with Intel oneAPI and NVHPC SDK due to Fortran module version mismatches with HDF5 module files).

Otherwise, feel free to test this further on additional platforms and report any issues. In particular, GPU-based tests still need to be done.

[vtripath65]

There is no dependency of HDF5 on GPU code. It's also strange why MPI should have any impact on HDF5 part of the code. We are only using restart capability through the master branch!

…

On Mon, Nov 10, 2025 at 10:57 AM ohearnk ***@***.***> wrote: *ohearnk* left a comment (merzlab/QUICK#396) <https://urldefense.com/v3/__https://github.com/merzlab/QUICK/pull/396*issuecomment-3513433351__;Iw!!Mih3wA!Eb2iq45mYLwWxtzgYuA0HSFqUS-6YRZu7T-A0ygenyt_q1e_PBgMf-W7FBW6Jv2OfkuSlcvEZr6V91oA5bFeytusuA$> At this point, I'm convinced that there's something wrong with the HDF5 libraries built for Ubuntu v24.04 (specifically for the Fortran bindings), as all failing MPI tests error out on boilerplate HDF5 code to initialize the Fortran interface. @agoetz <https://urldefense.com/v3/__https://github.com/agoetz__;!!Mih3wA!Eb2iq45mYLwWxtzgYuA0HSFqUS-6YRZu7T-A0ygenyt_q1e_PBgMf-W7FBW6Jv2OfkuSlcvEZr6V91oA5bEx8vaKWw$>, @vtripath65 <https://urldefense.com/v3/__https://github.com/vtripath65__;!!Mih3wA!Eb2iq45mYLwWxtzgYuA0HSFqUS-6YRZu7T-A0ygenyt_q1e_PBgMf-W7FBW6Jv2OfkuSlcvEZr6V91oA5bGmHakFrQ$> -- if either of you have any ideas here, I'd appreciate it. If not, I'm inclined to change the Github workflows to not test HDF5-based checkpoint functionality on Ubuntu v24.04 -- this is easy to do given how the workflow YAML files are written (and is currently done for tests with Intel oneAPI and NVHPC SDK due to Fortran module version mismatches with HDF5 module files). Otherwise, feel free to test this further on additional platforms and report any issues. In particular, GPU-based tests still need to be done. — Reply to this email directly, view it on GitHub <https://urldefense.com/v3/__https://github.com/merzlab/QUICK/pull/396*issuecomment-3513433351__;Iw!!Mih3wA!Eb2iq45mYLwWxtzgYuA0HSFqUS-6YRZu7T-A0ygenyt_q1e_PBgMf-W7FBW6Jv2OfkuSlcvEZr6V91oA5bFeytusuA$>, or unsubscribe <https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AM6PFQQOAY3T5DDR4EVD2TT34DNYLAVCNFSM6AAAAABXPWVILWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTKMJTGQZTGMZVGE__;!!Mih3wA!Eb2iq45mYLwWxtzgYuA0HSFqUS-6YRZu7T-A0ygenyt_q1e_PBgMf-W7FBW6Jv2OfkuSlcvEZr6V91oA5bHAZsUoiQ$> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Vikrant Tripathy Postdoctoral Research Scientist San Diego Supercomputer Center University of California, San Diego

[vtripath65]

I do see that I am using the module in all the processes including master however, calling it only from master. That probably is the issue. We really do not need to use the module from all the processes, just master should suffice. On Mon, Nov 10, 2025 at 12:06 PM Vikrant Tripathy ***@***.***> wrote:

…

There is no dependency of HDF5 on GPU code. It's also strange why MPI should have any impact on HDF5 part of the code. We are only using restart capability through the master branch! On Mon, Nov 10, 2025 at 10:57 AM ohearnk ***@***.***> wrote: > *ohearnk* left a comment (merzlab/QUICK#396) > <https://urldefense.com/v3/__https://github.com/merzlab/QUICK/pull/396*issuecomment-3513433351__;Iw!!Mih3wA!Eb2iq45mYLwWxtzgYuA0HSFqUS-6YRZu7T-A0ygenyt_q1e_PBgMf-W7FBW6Jv2OfkuSlcvEZr6V91oA5bFeytusuA$> > > At this point, I'm convinced that there's something wrong with the HDF5 > libraries built for Ubuntu v24.04 (specifically for the Fortran bindings), > as all failing MPI tests error out on boilerplate HDF5 code to initialize > the Fortran interface. > > @agoetz > <https://urldefense.com/v3/__https://github.com/agoetz__;!!Mih3wA!Eb2iq45mYLwWxtzgYuA0HSFqUS-6YRZu7T-A0ygenyt_q1e_PBgMf-W7FBW6Jv2OfkuSlcvEZr6V91oA5bEx8vaKWw$>, > @vtripath65 > <https://urldefense.com/v3/__https://github.com/vtripath65__;!!Mih3wA!Eb2iq45mYLwWxtzgYuA0HSFqUS-6YRZu7T-A0ygenyt_q1e_PBgMf-W7FBW6Jv2OfkuSlcvEZr6V91oA5bGmHakFrQ$> > -- if either of you have any ideas here, I'd appreciate it. If not, I'm > inclined to change the Github workflows to not test HDF5-based checkpoint > functionality on Ubuntu v24.04 -- this is easy to do given how the workflow > YAML files are written (and is currently done for tests with Intel oneAPI > and NVHPC SDK due to Fortran module version mismatches with HDF5 module > files). > > Otherwise, feel free to test this further on additional platforms and > report any issues. In particular, GPU-based tests still need to be done. > > — > Reply to this email directly, view it on GitHub > <https://urldefense.com/v3/__https://github.com/merzlab/QUICK/pull/396*issuecomment-3513433351__;Iw!!Mih3wA!Eb2iq45mYLwWxtzgYuA0HSFqUS-6YRZu7T-A0ygenyt_q1e_PBgMf-W7FBW6Jv2OfkuSlcvEZr6V91oA5bFeytusuA$>, > or unsubscribe > <https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AM6PFQQOAY3T5DDR4EVD2TT34DNYLAVCNFSM6AAAAABXPWVILWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTKMJTGQZTGMZVGE__;!!Mih3wA!Eb2iq45mYLwWxtzgYuA0HSFqUS-6YRZu7T-A0ygenyt_q1e_PBgMf-W7FBW6Jv2OfkuSlcvEZr6V91oA5bHAZsUoiQ$> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> > -- Vikrant Tripathy Postdoctoral Research Scientist San Diego Supercomputer Center University of California, San Diego

-- Vikrant Tripathy Postdoctoral Research Scientist San Diego Supercomputer Center University of California, San Diego

[vtripath65]

On second thought that may not be possible. On Mon, Nov 10, 2025 at 12:09 PM Vikrant Tripathy ***@***.***> wrote:

…

I do see that I am using the module in all the processes including master however, calling it only from master. That probably is the issue. We really do not need to use the module from all the processes, just master should suffice. On Mon, Nov 10, 2025 at 12:06 PM Vikrant Tripathy ***@***.***> wrote: > There is no dependency of HDF5 on GPU code. It's also strange why MPI > should have any impact on HDF5 part of the code. We are only using restart > capability through the master branch! > > On Mon, Nov 10, 2025 at 10:57 AM ohearnk ***@***.***> > wrote: > >> *ohearnk* left a comment (merzlab/QUICK#396) >> <https://urldefense.com/v3/__https://github.com/merzlab/QUICK/pull/396*issuecomment-3513433351__;Iw!!Mih3wA!Eb2iq45mYLwWxtzgYuA0HSFqUS-6YRZu7T-A0ygenyt_q1e_PBgMf-W7FBW6Jv2OfkuSlcvEZr6V91oA5bFeytusuA$> >> >> At this point, I'm convinced that there's something wrong with the HDF5 >> libraries built for Ubuntu v24.04 (specifically for the Fortran bindings), >> as all failing MPI tests error out on boilerplate HDF5 code to initialize >> the Fortran interface. >> >> @agoetz >> <https://urldefense.com/v3/__https://github.com/agoetz__;!!Mih3wA!Eb2iq45mYLwWxtzgYuA0HSFqUS-6YRZu7T-A0ygenyt_q1e_PBgMf-W7FBW6Jv2OfkuSlcvEZr6V91oA5bEx8vaKWw$>, >> @vtripath65 >> <https://urldefense.com/v3/__https://github.com/vtripath65__;!!Mih3wA!Eb2iq45mYLwWxtzgYuA0HSFqUS-6YRZu7T-A0ygenyt_q1e_PBgMf-W7FBW6Jv2OfkuSlcvEZr6V91oA5bGmHakFrQ$> >> -- if either of you have any ideas here, I'd appreciate it. If not, I'm >> inclined to change the Github workflows to not test HDF5-based checkpoint >> functionality on Ubuntu v24.04 -- this is easy to do given how the workflow >> YAML files are written (and is currently done for tests with Intel oneAPI >> and NVHPC SDK due to Fortran module version mismatches with HDF5 module >> files). >> >> Otherwise, feel free to test this further on additional platforms and >> report any issues. In particular, GPU-based tests still need to be done. >> >> — >> Reply to this email directly, view it on GitHub >> <https://urldefense.com/v3/__https://github.com/merzlab/QUICK/pull/396*issuecomment-3513433351__;Iw!!Mih3wA!Eb2iq45mYLwWxtzgYuA0HSFqUS-6YRZu7T-A0ygenyt_q1e_PBgMf-W7FBW6Jv2OfkuSlcvEZr6V91oA5bFeytusuA$>, >> or unsubscribe >> <https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AM6PFQQOAY3T5DDR4EVD2TT34DNYLAVCNFSM6AAAAABXPWVILWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTKMJTGQZTGMZVGE__;!!Mih3wA!Eb2iq45mYLwWxtzgYuA0HSFqUS-6YRZu7T-A0ygenyt_q1e_PBgMf-W7FBW6Jv2OfkuSlcvEZr6V91oA5bHAZsUoiQ$> >> . >> You are receiving this because you were mentioned.Message ID: >> ***@***.***> >> > > > -- > Vikrant Tripathy > Postdoctoral Research Scientist > San Diego Supercomputer Center > University of California, San Diego > -- Vikrant Tripathy Postdoctoral Research Scientist San Diego Supercomputer Center University of California, San Diego

-- Vikrant Tripathy Postdoctoral Research Scientist San Diego Supercomputer Center University of California, San Diego

[ohearnk]

@vtripath65, @agoetz

There is no dependency of HDF5 on GPU code. It's also strange why MPI should have any impact on HDF5 part of the code. We are only using restart capability through the master branch!

A few notes:

1. Github CI tests have been modified such that all tests except those with Intel oneAPI and NVIDIA HPC SDK are currently testing HDF5-based checkpoint functionality. For oneAPI and NVHPC SDK, they are falling back to the custom built-in checkpoint format (currently the default in the HEAD of the master branch). This applies for both serial and MPI versions.
2. Just because the HDF5 code does not depend on GPU functionality does not mean that it does not need to be tested. Changes in the CPU code can still inadvertantly introduce bugs in GPU versions. Examples would include forgetting to copy data in CPU memory to GPU memory.
3. I opened up a few of the serial test artifacts, and tests are actually failing for HDF5 but being incorrectly marked as passing. As such, the runtest script needs to be fixed for checkpoint / restart tests to actually correctly mark failures (I'll look into this).
4. The serial test failures follow the same pattern as the MPI failures, namely all failures are localized to Ubuntu v24.04. Annoyingly, (3) partially obfuscated this pattern of failures. Again, I stand by my statement that something appears wrong with the HDF5 libraries being built & installed for Ubuntu v24.04. As a further datapoint, tests on my local system pass on Fedora 41 with GCC v14.3.1, OpenMPI v5.0.5 OR MPICH v4.2.2, HDF5 v1.12.1. Please test on additional systems and report back any failures with version info.

[ohearnk]

I believe the Ubuntu v24.04 failures are related to issues reported with gfortran due to recent supported added for half-precision (bfloat16) datatypes. See, e.g., this recent issue.

The way to get a definitive answer would be to build HDF5 ourselves on an Ubuntu v24.04 platform to see if that resolves the failures. Thoughts?

[agoetz]

Agreed. The Intel server petra has ubuntu 24.04.2 LTS installed. I don't have the bandwidth to test this at the moment but both of you do have an account on petra.