Intel(R) SHMEM Library (ISHMEM) 1.2.0 doc pages

Author: sys-shmem

README.md

@@ -22,10 +22,32 @@ Intel® SHMEM provides an efficient implementation of GPU-initiated communicatio
 ### SYCL support <!-- omit in toc -->
 Intel® oneAPI DPC++/C++ Compiler with Level Zero support.
 
-To install Level Zero, refer to the instructions in [Intel® Graphics Compute Runtime for oneAPI Level Zero and OpenCL™ Driver repository](https://github.com/intel/compute-runtime/releases) or to the [installation guide](https://dgpu-docs.intel.com/installation-guides/index.html) for oneAPI users.
 
 ## Installation
-Intel® SHMEM requires a host OpenSHMEM back-end to be used for host-sided operations support. In particular, it relies on a collection of extension APIs (`shmemx_heap_create`, `shmemx_heap_preinit`, and `shmemx_heap_postinit`) to coordinate the Intel® SHMEM and OpenSHMEM heaps. We recommend [Sandia OpenSHMEM v1.5.3rc1](https://github.com/Sandia-OpenSHMEM/SOS/releases/tag/v1.5.3rc1) or newer for this purpose.
+
+### Building Level Zero
+For detailed information on Level Zero, refer to the [Intel® Graphics Compute Runtime for oneAPI Level Zero and OpenCL™ Driver repository](https://github.com/intel/compute-runtime/releases) or to the [installation guide](https://dgpu-docs.intel.com/installation-guides/index.html) for oneAPI users.
+
+To install, download the oneAPI Level Zero from the repository.
+
+```
+git clone https://github.com/oneapi-src/level-zero.git
+```
+
+Build Level Zero following instructions below. 
+
+```
+cd level-zero
+mkdir build
+cd build
+cmake -DCMAKE_INSTALL_PREFIX=<level_zero_dir> ..
+make -j
+make install
+```
+### The Host Back-End Library
+Intel® SHMEM requires a host OpenSHMEM or MPI back-end to be used for host-sided operations support. In particular, the OpenSHMEM back-end relies on a collection of extension APIs (`shmemx_heap_create`, `shmemx_heap_preinit`, and `shmemx_heap_postinit`) to coordinate the Intel® SHMEM and OpenSHMEM heaps. We recommend [Sandia OpenSHMEM v1.5.3rc1](https://github.com/Sandia-OpenSHMEM/SOS/releases/tag/v1.5.3rc1) or newer for this purpose. A [work-in-progress branch](https://github.com/davidozog/oshmpi/tree/wip/ishmem) of [OSHMPI](https://github.com/pmodels/oshmpi.git) is also supported but is currently considered experimental.  See the [Building OSHMPI](#building-oshmpi-optional-and-experimental) section before for more details.
+
+We recommend the Intel® MPI Library as the MPI back-end option for the current version of Intel® SHMEM. See the [Building Intel® SHMEM](#building-intel-shmem) section below for more details.
 
 ### Building Sandia OpenSHMEM (SOS)
 Download the SOS repo to be configured as a back-end for Intel® SHMEM.
@@ -39,67 +61,105 @@ Build SOS following instructions below. `FI_HMEM` support in the provider is req
 ```
 cd SOS
 ./autogen.sh
-./configure --prefix=<sos_dir> --with-ofi=<ofi_installation> --enable-pmi-simple --enable-ofi-mr=basic --disable-ofi-inject --enable-ofi-hmem --disable-bounce-buffers --enable-hard-polling
+CC=icx CXX=icpx ./configure --prefix=<shmem_dir> --with-ofi=<ofi_installation> --enable-pmi-simple --enable-ofi-mr=basic --disable-ofi-inject --enable-ofi-hmem --disable-bounce-buffers --enable-hard-polling
 make -j
 make install
 ```
 To configure SOS with the HPE Slingshot provider `cxi`, please use the following instructions:
 ```
 cd SOS
 ./autogen.sh
-./configure --prefix=<sos_dir> --with-ofi=<ofi_installation> --enable-pmi-simple --enable-ofi-mr=basic --disable-ofi-inject --enable-ofi-hmem --disable-bounce-buffers --enable-ofi-manual-progress --enable-mr-endpoint --disable-nonfetch-amo --enable-manual-progress
+CC=icx CXX=icpx ./configure --prefix=<shmem_dir> --with-ofi=<ofi_installation> --enable-pmi-simple --enable-ofi-mr=basic --disable-ofi-inject --enable-ofi-hmem --disable-bounce-buffers --enable-ofi-manual-progress --enable-mr-endpoint --disable-nonfetch-amo --enable-manual-progress
 make -j
 make install
 ```
 To configure SOS with the `psm3` provider, please use the following instructions:
 ```
 cd SOS
 ./autogen.sh
-./configure --prefix=<sos_dir> --with-ofi=<ofi_installation> --enable-pmi-simple --enable-manual-progress --enable-ofi-hmem --disable-bounce-buffers --enable-ofi-mr=basic --enable-mr-endpoint
+CC=icx CXX=icpx ./configure --prefix=<shmem_dir> --with-ofi=<ofi_installation> --enable-pmi-simple --enable-manual-progress --enable-ofi-hmem --disable-bounce-buffers --enable-ofi-mr=basic --enable-mr-endpoint
 make -j
 make install
 ```
- 
+
 Please choose an appropriate PMI configure flag based on the available PMI client library in the system. Please check for further instructions on [SOS Wiki pages](https://github.com/Sandia-OpenSHMEM/SOS/wiki). Optionally, users may also choose to add `--disable-fortran` since fortran interfaces will not be used.
 
+### Building OSHMPI (Optional and experimental)
+Intel® SHMEM has experimental support for OSHMPI when built using the Intel® MPI Library.
+Here is information on how to [Get Started with Intel® MPI Library on Linux](https://www.intel.com/content/www/us/en/docs/mpi-library/get-started-guide-linux/2021-11/overview.html).
+
+To download the OSHMPI repository:
+
+```
+git clone -b wip/ishmem --recurse-submodules https://github.com/davidozog/oshmpi.git oshmpi
+```
+After ensuring Intel® MPI Library is enabled (for example, by sourcing the `/opt/intel/oneapi/setvars.sh` script),
+please build OSHMPI following the instructions below.
+
+```
+cd oshmpi
+./autogen.sh
+CC=mpiicx CXX=mpiicpx ./configure --prefix=<shmem_dir> --disable-fortran --enable-rma=direct --enable-amo=direct --enable-async-thread=yes
+make -j
+make install
+```
 
 ### Building Intel® SHMEM
-Check that the SOS build process has successfully created an `<sos_dir>` directory with `include` and `lib` as subdirectories. Please find `shmem.h` and `shmemx.h` in `include`. 
+Check that the SOS build process has successfully created a `<shmem_dir>` directory with `include` and `lib` as subdirectories. Please find `shmem.h` and `shmemx.h` in `include`.
 
-Build Intel® SHMEM using the following instructions:
+Build Intel® SHMEM with an OpenSHMEM back-end using the following instructions:
 
 ```
 cd ishmem
 mkdir build
 cd build
-cmake .. -DSHMEM_INSTALL_PREFIX=<sos_dir> -DCMAKE_INSTALL_PREFIX=<ishmem_install_dir>
+CC=icx CXX=icpx cmake .. -DENABLE_OPENSHMEM=ON -DSHMEM_DIR=<shmem_dir> -DCMAKE_INSTALL_PREFIX=<ishmem_install_dir>
 make -j
 ```
+Alternatively, Intel® SHMEM can be built by enabling an Intel® MPI Library back-end.
+Here is information on how to [Get Started with Intel® MPI Library on Linux](https://www.intel.com/content/www/us/en/docs/mpi-library/get-started-guide-linux/2021-11/overview.html).
+
+```
+CC=icx CXX=icpx cmake .. -DENABLE_OPENSHMEM=OFF -DENABLE_MPI=ON -DMPI_DIR=<impi_dir> -DCMAKE_INSTALL_PREFIX=<ishmem_install_dir>
+```
+where `<impi_dir>` is the path to the Intel® MPI Library installation.
+
+Enabling both the OpenSHMEM and MPI back-ends is also supported.  In this case,
+the desired backend can be selected via the environment variable,
+`ISHMEM_RUNTIME`, which can be set to either "OpenSHMEM" or "MPI".
+The default value for `ISHMEM_RUNTIME` is "OpenSHMEM".
 
 ## Usage
 
 ### Launching Example Application
 
 Validate that Intel® SHMEM was built correctly by running an example program.
 
-1. Add the library path for SOS to the environment:
+1. Add the path for the back-end library to the environment, for example:
 
 ```
-export LD_LIBRARY_PATH=<sos_dir>/lib:$LD_LIBRARY_PATH
+export LD_LIBRARY_PATH=<shmem_dir>/lib:$LD_LIBRARY_PATH
 ```
 
+When enabling only the Intel® MPI Library back-end, simply source the appropriate
+`setvars.sh` script. When enabling both OpenSHMEM and MPI back-ends, first
+source the `setvars.sh` script, then configure the dynamic linker to load the
+OpenSHMEM library (for example by prepending `<shmem_dir>/lib` to
+`LD_LIBRARY_PATH`).
+
 2. Run the example program or test on an allocated node using a process launcher:
 
 ```
-mpiexec.hydra -n 2 -hosts <allocated_node_id> ./scripts/ishmrun ./test/unit/SHMEM/int_get_device
+ISHMEM_RUNTIME=<back-end> mpiexec.hydra -n 2 -hosts <allocated_node_id> ./scripts/ishmrun ./test/unit/int_get_device
 ```
+where `<back-end>` is the selected host back-end library.
 
 - *Note:* Current supported launchers include: MPI process launchers (i.e. `mpiexec`, `mpiexec.hydra`, `mpirun`, etc.), Slurm (i.e. `srun`, `salloc`, etc.), and PBS (i.e. `qsub`).
 
 - *Note:* Intel® SHMEM execution model requires applications to use a 1:1 mapping between PEs and GPU devices. Attempting to run an application without the ishmrun launch script may result in undefined behavior if this mapping is not maintained.
   - For further details on the device selection, please see [the ONEAPI_DEVICE_SELECTOR](https://github.com/intel/llvm/blob/sycl/sycl/doc/EnvironmentVariables.md#oneapi_device_selector).
 
-3. Validate the application ran succesfully; example output:
+3. Validate the application ran successfully; example output:
 
 ```
 Selected device: Intel(R) Data Center GPU Max 1550
@@ -124,7 +184,7 @@ To launch a single test, execute:
 ctest -R <test_name>
 ```
 
-Alternatively, all the tests in a directory (such as `test/unit/SHMEM/`) can be run with the following command:
+Alternatively, all the tests in a directory (such as `test/unit/`) can be run with the following command:
 
 ```
 ctest --test-dir <directory_name>
@@ -141,7 +201,7 @@ By default, a passed or failed test can be detected by the output:
 To have a test's output printed to the console, add either the `--verbose` or `--output-on-failure` flag to the `ctest` command
 
 ### Available Scheduler Wrappers for Jobs Run via CTest
-The following values may be assigned to `CTEST_SCHEDULER` at configure-time (ex. `-DCTEST_SCHEDULER=mpi`) to set which scheduler will be used to run tests launched through a call to `ctest`:
+The following values may be assigned to `CTEST_LAUNCHER` at configure-time (ex. `-DCTEST_LAUNCHER=mpi`) to set which scheduler will be used to run tests launched through a call to `ctest`:
  - srun (default)
    - Launches CTest jobs on a single node using Slurm's `srun`.
  - mpi

docs/_sources/building_ishmem.rst

@@ -0,0 +1,161 @@
+.. _building_ishmem:
+
+=====================
+Building Intel® SHMEM
+=====================
+
+To download Intel® SHMEM, either clone the repository::
+
+    git clone https://github.com/oneapi-src/ishmem.git
+
+or download a release tarball from: https://github.com/oneapi-src/ishmem/releases.
+
+The ``README.md`` file describes how to build the dependencies of Intel® SHMEM.
+
+The :ref:`CMake Build Options<cmake_options>` section lists all the available
+CMake build options for customizing Intel® SHMEM.
+
+Intel® SHMEM requires enabling a host back-end that is either:
+
+#. a suitable OpenSHMEM v1.5 library
+#. Intel® MPI Library (Message Passing Interface)
+
+.. #. PMI (Process Manager Interface)
+
+These options are covered in sections :ref:`Enabling
+OpenSHMEM<enabling_openshmem>` and :ref:`Enabling MPI<enabling_mpi>`,
+respectively.
+
+.. and :ref:`Enabling PMI<enabling_pmi>`, respectively.
+
+.. _enabling_openshmem:
+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Enabling the OpenSHMEM back-end
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+As of version |release|, Intel® SHMEM supports the following OpenSHMEM
+libraries:
+
+- |sos_url| - please use the ``v1.5.3`` release.
+- |oshmpi_url| - please use this |oshmpi_branch|.
+
+The OpenSHMEM back-end is enabled by default and is controlled via the
+``ENABLE_OPENSHMEM`` CMake option, and the installation path is set via the
+``SHMEM_DIR`` option, for example::
+
+    CC=icx CXX=icpx cmake .. -DSHMEM_DIR=<shmem_dir> -DCMAKE_INSTALL_PREFIX=<ishmem_install_dir>
+
+where ``<shmem_dir>`` is the path to the Sandia OpenSHMEM installation
+directory, and ``<ishmem_install_dir>`` is the path to the desired Intel®
+SHMEM installation directory.
+
+After successfully running the ``cmake`` command, run ``make`` to build the
+library and ``make install`` to install the library to
+``<ishmem_install_dir>``.
+
+To enable OSHMPI, the build process is similar::
+
+    CC=icx CXX=icpx cmake .. -DSHMEM_DIR=<oshmpi_dir> -DCMAKE_INSTALL_PREFIX=<ishmem_install_dir>
+
+where ``<oshmpi_dir>`` is the path to the OSHMPI installation. Additionally,
+setting the following environment variables are required for execution::
+
+    export ISHMEM_SHMEM_LIB_NAME=liboshmpi.so
+    export ISHMEM_RUNTIME_USE_OSHMPI=true
+
+.. note:: Enabling the OSHMPI back-end in Intel® SHMEM version |release| may
+   require setting environment variable ``OSHMPI_TEAM_SHARED_ONLY_SELF=1`` if the
+   ``ISHMEM_TEAM_SHARED`` team does not properly match ``SHMEM_TEAM_SHARED``. This
+   is likely the case if the following warning is encountered:
+   ``ishmemi_runtime_team_predefined_set SHARED failed. Runtime ISHMEM_TEAM_SHARED
+   unable to use SHMEMX_TEAM_NODE``
+
+.. |sos_url| raw:: html
+
+   <a href="https://github.com/Sandia-OpenSHMEM/SOS.git" target="_blank">Sandia OpenSHMEM</a>
+
+.. |oshmpi_url| raw:: html
+
+   <a href="https://github.com/pmodels/oshmpi.git" target="_blank">OSHMPI</a>
+
+.. |oshmpi_branch| raw:: html
+
+   <a href="https://github.com/davidozog/oshmpi/tree/wip/ishmem" target="_blank">experimental branch</a>
+
+.. _enabling_mpi:
+
+^^^^^^^^^^^^^^^^^^^^^^^^^
+Enabling the MPI back-end
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+As of version |release|, Intel® SHMEM supports the following MPI libraries:
+
+- |impi_url| - please use the ``2021.14.0`` release.
+
+.. |impi_url| raw:: html
+
+   <a href="https://www.intel.com/content/www/us/en/developer/tools/oneapi/mpi-library-download.html" target="_blank">Intel® MPI Library</a>
+
+To enable the MPI back-end and disable the OpenSHMEM back-end::
+
+    CC=icx CXX=icpx cmake .. -DENABLE_OPENSHMEM=OFF -DENABLE_MPI=ON -DMPI_DIR=<impi_install_dir> -DCMAKE_INSTALL_PREFIX=<ishmem_install_dir>
+
+where ``<impi_install_dir>`` is the path to the Intel® MPI Library installation.
+
+After successfully running the ``cmake`` command, run ``make`` to build the
+library and ``make install`` to install the library to
+``<ishmem_install_dir>``.
+
+Note that enabling *both* the OpenSHMEM and MPI back-ends is also supported.
+In this case, the desired backend can be selected via the environment variable,
+``ISHMEM_RUNTIME``, which can be set to either "OpenSHMEM" or "MPI". For
+example building with both runtimes enabled::
+
+    CC=icx CXX=icpx cmake .. -DSHMEM_DIR=<shmem_dir> -DENABLE_MPI=ON -DMPI_DIR=<impi_install_dir>
+
+And running with either ``ISHMEM_RUNTIME=OPENSHMEM`` or ``ISHMEM_RUNTIME=MPI``.
+
+.. TODO: add "PMI" as a possible ISHMEM_RUNTIME option above
+.. .. _enabling_pmi:
+.. 
+.. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. Enabling the PMI back-end (experimental)
+.. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. 
+.. The Process Management Interface (PMI) is planned to be supported as an Intel®
+.. SHMEM back-end in a future release.
+
+.. _cmake_options:
+
+^^^^^^^^^^^^^^^^^^^
+CMake Build Options
+^^^^^^^^^^^^^^^^^^^
+
++---------------------------------+------------------------------------------------------------+---------+
+| **CMake Variable**              | Description                                                | Default |
++=================================+============================================================+=========+
+| ``ENABLE_OPENSHMEM``            | Enable OpenSHMEM back-end support                          | ON      |
++---------------------------------+------------------------------------------------------------+---------+
+| ``ENABLE_MPI``                  | Enable MPI back-end support                                | OFF     |
++---------------------------------+------------------------------------------------------------+---------+
+| ``BUILD_UNIT_TESTS``            | Build unit tests                                           | OFF     |
++---------------------------------+------------------------------------------------------------+---------+
+| ``BUILD_PERF_TESTS``            | Build performance tests                                    | OFF     |
++---------------------------------+------------------------------------------------------------+---------+
+| ``BUILD_EXAMPLES``              | Build examples                                             | OFF     |
++---------------------------------+------------------------------------------------------------+---------+
+| ``BUILD_APPS``                  | Build apps                                                 | OFF     |
++---------------------------------+------------------------------------------------------------+---------+
+| ``BUILD_CMAKE_CONFIG``          | Build CMake config files                                   | ON      |
++---------------------------------+------------------------------------------------------------+---------+
+| ``ENABLE_ERROR_CHECKING``       | Validate API inputs                                        | OFF     |
++---------------------------------+------------------------------------------------------------+---------+
+| ``ENABLE_DLMALLOC``             | Enable dlmalloc for shared heap                            | ON      |
++---------------------------------+------------------------------------------------------------+---------+
+| ``ENABLE_REDUCED_LINK_ENGINES`` | Enable reduced link engines (i.e. for single tile devices) | OFF     |
++---------------------------------+------------------------------------------------------------+---------+
+| ``ENABLE_AOT_COMPILATION``      | Enables Ahead-Of-Time compilation for GPU kernels          | ON      |
++---------------------------------+------------------------------------------------------------+---------+
+| ``SKIP_COMPILER_CHECK``         | Skips compiler validation (NOT RECOMMENDED)                | OFF     |
++---------------------------------+------------------------------------------------------------+---------+