accelerator_cuda_check_addr expensive in CUDA-aware MPI

[NamjaeChoi]

What version of Open MPI are you using? (e.g., v4.1.6, v5.0.1, git branch name and hash, etc.)

OpenMPI 5.0.5

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

From source

Please describe the system on which you are running

• Operating system/version: Linux
• Computer hardware: AMD EPYC 9654
• Network type: Single node

---

Details of the problem

We are profiling our code with gperf and seeing a big overhead caused by accelerator_cuda_check_addr. cuPointerGetAttribute and cuMemRetainAllocationHandle calls cudbgMain, cuEGLApiInit, and cuVDPAUCtxCreate under the hood. They seem to be related with debugging, event logging, and profiling which we are not interested in. Is there a way to turn them off?

[lindsayad]

I'm using version 5.0.8. I'll add that we were wondering if using UCX would remove/mitigate this cost with memtype caching but it doesn't appear to do so

[lindsayad]

I never enjoy doing code-to-code comparisons but it might be helpful to know that mpich calls cudaPointerGetAttributes but we do not see the slow down and seeming debugging checks that we see with openmpi. The performance of this function in openmpi also gets worse with number of processes. mpich stays flat

[lindsayad]

I went back to verify that I did not configure openmpi with something like --enable-debug