Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extend container support to all Tier 1 platforms #85

Merged
merged 3 commits into from
Apr 1, 2024
Merged

Extend container support to all Tier 1 platforms #85

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a91059a
update ufs land exe path
EdwardSnyder-NOAA Mar 28, 2024
3608ca6
update docs to include the expansion of container support
EdwardSnyder-NOAA Mar 29, 2024
49ba528
Merge branch 'expand-container-support' of https://github.com/EdwardS…
EdwardSnyder-NOAA Mar 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion do_submit_cycle.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ fi
if [[ -e ${CYCLEDIR}/exec/ufsLand.exe ]]; then
export LSMexec=${CYCLEDIR}/exec/ufsLand.exe
else
export LSMexec=${CYCLEDIR}/sorc/ufs-land-driver-emc-dev/run/ufsLand.exe
export LSMexec=${CYCLEDIR}/sorc/ufsLand.fd/run/ufsLand.exe
fi

export DADIR=${CYCLEDIR}/sorc/DA_update/
Expand Down
53 changes: 38 additions & 15 deletions doc/source/BuildingRunningTesting/Container.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,19 +99,21 @@ On many NOAA :term:`RDHPCS` systems, a container named ``ubuntu20.04-intel-landd

.. table:: Locations of Pre-Built Containers

+--------------+--------------------------------------------------------+
| Machine | File location |
+==============+========================================================+
| Derecho | /glade/work/epicufsrt/contrib/containers |
+--------------+--------------------------------------------------------+
| Gaea | /lustre/f2/dev/role.epic/contrib/containers |
+--------------+--------------------------------------------------------+
| Hera | /scratch1/NCEPDEV/nems/role.epic/containers |
+--------------+--------------------------------------------------------+
| Jet | /mnt/lfs4/HFIP/hfv3gfs/role.epic/containers |
+--------------+--------------------------------------------------------+
| Orion | /work/noaa/epic/role-epic/contrib/containers |
+--------------+--------------------------------------------------------+
+-----------------+--------------------------------------------------------+
| Machine | File location |
+=================+========================================================+
| Derecho | /glade/work/epicufsrt/contrib/containers |
+-----------------+--------------------------------------------------------+
| Gaea | /gpfs/f5/epic/world-shared/containers |
+-----------------+--------------------------------------------------------+
| Hera | /scratch1/NCEPDEV/nems/role.epic/containers |
+-----------------+--------------------------------------------------------+
| Jet | /mnt/lfs4/HFIP/hfv3gfs/role.epic/containers |
+-----------------+--------------------------------------------------------+
| NOAA Cloud | /contrib/EPIC/containers |
+-----------------+--------------------------------------------------------+
| Orion/Hercules | /work/noaa/epic/role-epic/contrib/containers |
+-----------------+--------------------------------------------------------+

Users can simply set an environment variable to point to the container:

Expand Down Expand Up @@ -244,7 +246,7 @@ Navigate to the ``land-DA_workflow`` directory after it has been successfully co

cd land-DA_workflow

When using a Singularity container, Intel compilers and Intel :term:`MPI` (preferably 2020 versions or newer) need to be available on the host system to properly launch MPI jobs. Generally, this is accomplished by loading a module with a recent Intel compiler and then loading the corresponding Intel MPI. For example, users can modify the following commands to load their system's compiler/MPI combination:
When using a Singularity container, Intel compilers and Intel :term:`MPI` (preferably 2020 versions or newer) need to be available on the host system to properly launch MPI jobs. The Level 1 systems that have Intel compilers and Intel MPI available are: Hera, Jet, NOAA Cloud, and Orion. Generally, this is accomplished by loading a module with a recent Intel compiler and then loading the corresponding Intel MPI. For example, users can modify the following commands to load their system's compiler/MPI combination:

.. code-block:: console

Expand All @@ -260,6 +262,27 @@ When using a Singularity container, Intel compilers and Intel :term:`MPI` (prefe

Then they should be able to load the appropriate modules.

The remaining Level 1 systems that do not have Intel MPI available will need to load a different Intel compiler and MPI combination. Refer to :numref:`Table %s <NonIMPICompilers>` for which Intel compiler and MPI to load for these systems.

.. _NonIMPICompilers:

.. table:: Intel compilers and MPIs for non-Intel MPI Level 1 systems

+-----------------+-------------------------------------------------------------------------+
| Machine | Intel compiler and MPI combinations |
+=================+=========================================================================+
| Derecho | module load intel-oneapi/2023.2.1 cray-mpich/8.1.25 |
+-----------------+-------------------------------------------------------------------------+
| Gaea | module load intel-classic/2023.1.0 cray-mpich/8.1.25 |
+-----------------+-------------------------------------------------------------------------+
| Hercules | module load intel-oneapi-compilers/2022.2.1 intel-oneapi-mpi/2021.7.1 |
+-----------------+-------------------------------------------------------------------------+

For Derecho and Gaea, an additional script is needed to help set up the land-DA workflow scripts so that the container can run there.

.. code-block:: console
./setup_container.sh -p=<derecho|gaea>

.. _ConfigureExptC:

Configure the Experiment
Expand Down Expand Up @@ -320,4 +343,4 @@ The ``do_submit_cycle.sh`` script will read the ``settings_DA_cycle_*`` file and

Depending on the experiment, either the ``vector`` or the ``tile`` directory will have data, but not both.

Users can check experiment progress/success according to the instructions in :numref:`Section %s <VerifySuccess>`, which apply to both containerized and non-containerized versions of the Land DA System.
Users can check experiment progress/success according to the instructions in :numref:`Section %s <VerifySuccess>`, which apply to both containerized and non-containerized versions of the Land DA System.
Loading