Add cycled-run capability

doc/source/BuildingRunningTesting/BuildRunLandDA.rst

@@ -111,20 +111,18 @@ Copy the experiment settings into ``land_analysis.yaml``:
 .. code-block:: console
 
    cd $LANDDAROOT/land-DA_workflow/parm
-   cp land_analysis_<forcing>_<platform>.yaml land_analysis.yaml
+   cp land_analysis_<platform>.yaml land_analysis.yaml
 
-where: 
-
-   * ``<forcing>`` is either ``gswp3`` or ``era5`` forcing data.
-   * ``<platform>`` is ``hera`` or ``orion``.
+where ``<platform>`` is ``hera`` or ``orion``.
 
 Users will need to configure certain elements of their experiment in ``land_analysis.yaml``: 
 
    * ``MACHINE:`` A valid machine name (i.e., ``hera`` or ``orion``)
    * ``ACCOUNT:`` A valid account name. Hera, Orion, and most NOAA RDHPCS systems require a valid account name; other systems may not
    * ``EXP_BASEDIR:`` The full path to the directory where land-DA_workflow was cloned (i.e., ``$LANDDAROOT``)
    * ``JEDI_INSTALL:`` The full path to the system's ``jedi-bundle`` installation
-   * ``LANDDA_INPUTS:`` The full path to the experiment data. See :ref:`Data <GetData>` below for information on prestaged data on Level 1 platforms. 
+   * ``FORCING:`` Forcing options; ``gswp3`` or ``era5``
+   * ``cycledef/spec:`` Cycle specification
 
 .. note::
 
@@ -151,7 +149,7 @@ Data
    | Orion     | /work/noaa/epic/UFS_Land-DA/inputs               |
    +-----------+--------------------------------------------------+
 
-Users who have difficulty accessing the data on Hera or Orion may download it according to the instructions in :numref:`Section %s <GetDataC>` and set ``LANDDA_INPUTS`` to point to the location of the downloaded data. Similarly, users with access to data for additional experiments may set the path to that data in ``LANDDA_INPUTS``. 
+Users who have difficulty accessing the data on Hera or Orion may download it according to the instructions in :numref:`Section %s <GetDataC>`. Its sub-directories are soft-linked to the ``fix`` directory of the land-DA workflow by the build script ``sorc/app_build.sh``.
 
 .. _generate-wflow:
 
@@ -186,16 +184,16 @@ Each Land DA experiment includes multiple tasks that must be run in order to sat
 
    * - J-job Task
      - Description
-   * - JLANDDA_PREP_EXP
-     - Sets up the experiment
    * - JLANDDA_PREP_OBS
-     - Sets up the observation files
-   * - JLANDDA_PREP_BMAT
-     - Sets up the :term:`JEDI` run
+     - Sets up the observation data files
+   * - JLANDDA_PRE_ANAL
+     - Transfers the snow data from the restart files to the surface data files
    * - JLANDDA_ANALYSIS
-     - Runs JEDI
+     - Runs :term:`JEDI` and adds the increment to the surface data files
+   * - JLANDDA_POST_ANAL
+     - Transfers the JEDI result from the surface data files to the restart files
    * - JLANDDA_FORECAST
-     - Runs forecast
+     - Runs the forecast model
 
 Users may run these tasks :ref:`using the Rocoto workflow manager <run-w-rocoto>` or :ref:`using a batch script <run-batch-script>`. 
 
@@ -246,7 +244,7 @@ The experiment has successfully completed when all tasks say SUCCEEDED under STA
 Run Without Rocoto
 --------------------
 
-Users may choose not to run the workflow with uwtools and Rocoto. To run the :term:`J-jobs` scripts in the ``jobs`` directory, navigate to the ``parm`` directory and edit ``run_without_rocoto.sh`` (e.g., using vim or preferred command line editor). Users will likely need to change the ``MACHINE``, ``ACCOUNT``, and ``EXP_BASEDIR`` variables to match their system. Then, run ``run_without_rocoto.sh``:
+Users may choose not to run the workflow with uwtools and Rocoto for a non-cycled run. To run the :term:`J-jobs` scripts in the ``jobs`` directory, navigate to the ``parm`` directory and edit ``run_without_rocoto.sh`` (e.g., using vim or preferred command line editor). Users will likely need to change the ``MACHINE``, ``ACCOUNT``, and ``EXP_BASEDIR`` variables to match their system. Then, run ``run_without_rocoto.sh``:
 
 .. code-block:: console
 
@@ -265,32 +263,24 @@ As the experiment progresses, it will generate a number of directories to hold i
     ├── ptmp (<PTMP>)
     │     └── test (<envir>)
     │           └── com
-    │                 ├── landda (<NET>)
-    │                 │     └── vX.Y.Z (<model_ver>)
-    │                 │           └── landda.YYYYMMDD (<RUN>.<PDY>)
-    │                 │                 └── HH (<cyc>)
-    │                 │                       ├── DA: Directory containing the output files of JEDI run
-    │                 │                       │     ├── hofx
-    │                 │                       │     └── jedi_incr
-    │                 │                       └── mem000: Directory containing the output files
-    │                 └── output
-    │                       └── logs
-    │                             └── run_<forcing> (<LOGDIR>): Directory containing the log file of the Rocoto workflow
-    └── workdir(<WORKDIR>)
-          └── run_<forcing>
-                └── mem000: Working directory
+    │                ├── landda (<NET>)
+    │                │     └── vX.Y.Z (<model_ver>)
+    │                │           └── landda.YYYYMMDD (<RUN>.<PDY>): Directory containing the output files
+    │                └── output
+    │                      └── logs
+    │                            └── run_<forcing> (<LOGDIR>): Directory containing the log file of the Rocoto workflow
+    └── tmp (<DATAROOT>)
+         ├── <jobid> (<DATA>): Working directory
+         └── DATA_SHARE
+               ├── YYYYMMDD (<PDY>): Directory containing the intermediate or temporary files
+               └── DATA_RESTART: Directory containing the soft-links to the restart files for the next cycles
 
 ``<forcing>`` refers to the type of forcing data used (``gswp3`` or ``era5``). Each variable in parentheses and angle brackets (e.g., ``(<VAR>)``) is the name for the directory defined in the file ``land_analysis.yaml``. In the future, this directory structure will be further modified to meet the :nco:`NCO Implementation Standards<>`.
 
-Check for the background and analysis files in the experiment directory:
+Check for the output files for each cycle in the experiment directory:
 
 .. code-block:: console
 
-   ls -l $LANDDAROOT/ptmp/test/com/landda/v1.2.1/landda.<PDY>/<cyc>/run_<forcing>/mem000/restarts/<vector_or_tile>
-
-where: 
-
-   * ``<forcing>`` is either ``era5`` or ``gswp3``, and
-   * ``<vector_or_tile>`` is either ``vector`` or ``tile`` depending on whether ERA5 or GSWP3 forcing data was used, respectively. 
+   ls -l $LANDDAROOT/ptmp/test/com/landda/v1.2.1/landda.YYYYMMDD
 
-The experiment should generate several restart files. 
+where ``YYYYMMDD`` is the cycle date. The experiment should generate several restart files. 

doc/source/BuildingRunningTesting/TestingLandDA.rst

@@ -18,12 +18,15 @@ From the working directory (``$LANDDAROOT``), navigate to ``build``. Then run:
 .. code-block:: console
 
    salloc --ntasks 8 --exclusive --qos=debug --partition=debug --time=00:30:00 --account=<account_name>
-   module use modulefiles && module load landda_<platform>.intel 
+   cd land-DA_workflow/sorc/build
+   source ../../versions/build.ver_<platform>
+   module use ../../modulefiles
+   module load build_<platform>_intel 
    ctest
 
 where ``<account_name>`` corresponds to the user's actual account name and ``<platform>`` is ``hera`` or ``orion``.
 
-This will allocate a compute node, load the appropriate modulefiles, and run the CTests. 
+This will submit an interactive job, load the appropriate modulefiles, and run the CTests. 
 
 Tests
 *******

jobs/JLANDDA_ANALYSIS

@@ -56,25 +56,25 @@ cd $DATA
 export NET="${NET:-landda}"
 export RUN="${RUN:-landda}"
 
+# Run setpdy to initialize PDYm and PDYp variables
+export cycle="${cycle:-t${cyc}z}"
+setpdy.sh
+. ./PDY
+
 [[ "$WORKFLOW_MANAGER" = "rocoto" ]] && export COMROOT=$COMROOT
 if [ "${MACHINE}" = "WCOSS2" ]; then
-  export COMIN="${COMIN:-$(compath.py -o ${NET}/${model_ver}/${RUN}.${PDY}/${cyc}${SLASH_ENSMEM_SUBDIR})}"
-  export COMOUT="${COMOUT:-$(compath.py -o ${NET}/${model_ver}/${RUN}.${PDY}/${cyc}${SLASH_ENSMEM_SUBDIR})}"
+  export COMIN="${COMIN:-$(compath.py -o ${NET}/${model_ver}/${RUN}.${PDY})}"
+  export COMOUT="${COMOUT:-$(compath.py -o ${NET}/${model_ver}/${RUN}.${PDY})}"
 else
-  export COMIN="${COMIN:-${COMROOT}/${NET}/${model_ver}/${RUN}.${PDY}/${cyc}${SLASH_ENSMEM_SUBDIR}}"
-  export COMOUT="${COMOUT:-${COMROOT}/${NET}/${model_ver}/${RUN}.${PDY}/${cyc}${SLASH_ENSMEM_SUBDIR}}"
+  export COMIN="${COMIN:-${COMROOT}/${NET}/${model_ver}/${RUN}.${PDY}}"
+  export COMOUT="${COMOUT:-${COMROOT}/${NET}/${model_ver}/${RUN}.${PDY}}"
 fi
 
 mkdir -p ${COMOUT}
 
 # Create a teomporary share directory
-export DATA_SHARE="${DATA_SHARE:-${DATAROOT}/DATA_SHARE/${PDY}${cyc}}"
+export DATA_SHARE="${DATA_SHARE:-${DATAROOT}/DATA_SHARE/${PDY}}"
 mkdir -p ${DATA_SHARE}
-
-# Run setpdy to initialize PDYm and PDYp variables
-export cycle="${cycle:-t${cyc}z}"
-setpdy.sh
-. ./PDY
 #
 #-----------------------------------------------------------------------
 #

jobs/JLANDDA_FORECAST

@@ -56,25 +56,29 @@ cd $DATA
 export NET="${NET:-landda}"
 export RUN="${RUN:-landda}"
 
+# Run setpdy to initialize PDYm and PDYp variables
+export cycle="${cycle:-t${cyc}z}"
+setpdy.sh
+. ./PDY
+
 [[ "$WORKFLOW_MANAGER" = "rocoto" ]] && export COMROOT=$COMROOT
 if [ "${MACHINE}" = "WCOSS2" ]; then
-  export COMIN="${COMIN:-$(compath.py -o ${NET}/${model_ver}/${RUN}.${PDY}/${cyc}${SLASH_ENSMEM_SUBDIR})}"
-  export COMOUT="${COMOUT:-$(compath.py -o ${NET}/${model_ver}/${RUN}.${PDY}/${cyc}${SLASH_ENSMEM_SUBDIR})}"
+  export COMIN="${COMIN:-$(compath.py -o ${NET}/${model_ver}/${RUN}.${PDY})}"
+  export COMOUT="${COMOUT:-$(compath.py -o ${NET}/${model_ver}/${RUN}.${PDY})}"
+  export COMINm1="${COMINm1:-$(compath.py -o ${NET}/${model_ver}/${RUN}.${PDYm1})}"
 else
-  export COMIN="${COMIN:-${COMROOT}/${NET}/${model_ver}/${RUN}.${PDY}/${cyc}${SLASH_ENSMEM_SUBDIR}}"
-  export COMOUT="${COMOUT:-${COMROOT}/${NET}/${model_ver}/${RUN}.${PDY}/${cyc}${SLASH_ENSMEM_SUBDIR}}"
+  export COMIN="${COMIN:-${COMROOT}/${NET}/${model_ver}/${RUN}.${PDY}}"
+  export COMOUT="${COMOUT:-${COMROOT}/${NET}/${model_ver}/${RUN}.${PDY}}"
+  export COMINm1="${COMINm1:-${COMROOT}/${NET}/${model_ver}/${RUN}.${PDYm1}}"
 fi
 
 mkdir -p ${COMOUT}
 
 # Create a teomporary share directory
-export DATA_SHARE="${DATA_SHARE:-${DATAROOT}/DATA_SHARE/${PDY}${cyc}}"
+export DATA_SHARE="${DATA_SHARE:-${DATAROOT}/DATA_SHARE/${PDY}}"
 mkdir -p ${DATA_SHARE}
-
-# Run setpdy to initialize PDYm and PDYp variables
-export cycle="${cycle:-t${cyc}z}"
-setpdy.sh
-. ./PDY
+export DATA_RESTART="${DATA_RESTART:-${DATAROOT}/DATA_SHARE/RESTART}"
+mkdir -p ${DATA_RESTART}
 #
 #-----------------------------------------------------------------------
 #

jobs/JLANDDA_PREP_EXP → jobs/JLANDDA_POST_ANAL

@@ -56,25 +56,27 @@ cd $DATA
 export NET="${NET:-landda}"
 export RUN="${RUN:-landda}"
 
+# Run setpdy to initialize PDYm and PDYp variables
+export cycle="${cycle:-t${cyc}z}"
+setpdy.sh
+. ./PDY
+
 [[ "$WORKFLOW_MANAGER" = "rocoto" ]] && export COMROOT=$COMROOT
 if [ "${MACHINE}" = "WCOSS2" ]; then
-  export COMIN="${COMIN:-$(compath.py -o ${NET}/${model_ver}/${RUN}.${PDY}/${cyc}${SLASH_ENSMEM_SUBDIR})}"
-  export COMOUT="${COMOUT:-$(compath.py -o ${NET}/${model_ver}/${RUN}.${PDY}/${cyc}${SLASH_ENSMEM_SUBDIR})}"
+  export COMIN="${COMIN:-$(compath.py -o ${NET}/${model_ver}/${RUN}.${PDY})}"
+  export COMOUT="${COMOUT:-$(compath.py -o ${NET}/${model_ver}/${RUN}.${PDY})}"
 else
-  export COMIN="${COMIN:-${COMROOT}/${NET}/${model_ver}/${RUN}.${PDY}/${cyc}${SLASH_ENSMEM_SUBDIR}}"
-  export COMOUT="${COMOUT:-${COMROOT}/${NET}/${model_ver}/${RUN}.${PDY}/${cyc}${SLASH_ENSMEM_SUBDIR}}"
+  export COMIN="${COMIN:-${COMROOT}/${NET}/${model_ver}/${RUN}.${PDY}}"
+  export COMOUT="${COMOUT:-${COMROOT}/${NET}/${model_ver}/${RUN}.${PDY}}"
 fi
 
 mkdir -p ${COMOUT}
 
 # Create a teomporary share directory
-export DATA_SHARE="${DATA_SHARE:-${DATAROOT}/DATA_SHARE/${PDY}${cyc}}"
+export DATA_SHARE="${DATA_SHARE:-${DATAROOT}/DATA_SHARE/${PDY}}"
 mkdir -p ${DATA_SHARE}
-
-# Run setpdy to initialize PDYm and PDYp variables
-export cycle="${cycle:-t${cyc}z}"
-setpdy.sh
-. ./PDY
+export DATA_RESTART="${DATA_RESTART:-${DATAROOT}/DATA_SHARE/RESTART}"
+mkdir -p ${DATA_RESTART}
 #
 #-----------------------------------------------------------------------
 #
@@ -85,7 +87,7 @@ setpdy.sh
 export pgmout="${DATA}/OUTPUT.$$"
 env
 
-${SCRIPTSlandda}/exlandda_prep_exp.sh
+${SCRIPTSlandda}/exlandda_post_anal.sh
 export err=$?; err_chk
 
 if [ -e "$pgmout" ]; then

jobs/JLANDDA_PREP_OBS

@@ -56,25 +56,25 @@ cd $DATA
 export NET="${NET:-landda}"
 export RUN="${RUN:-landda}"
 
+# Run setpdy to initialize PDYm and PDYp variables
+export cycle="${cycle:-t${cyc}z}"
+setpdy.sh
+. ./PDY
+
 [[ "$WORKFLOW_MANAGER" = "rocoto" ]] && export COMROOT=$COMROOT
 if [ "${MACHINE}" = "WCOSS2" ]; then
-  export COMIN="${COMIN:-$(compath.py -o ${NET}/${model_ver}/${RUN}.${PDY}/${cyc}${SLASH_ENSMEM_SUBDIR})}"
-  export COMOUT="${COMOUT:-$(compath.py -o ${NET}/${model_ver}/${RUN}.${PDY}/${cyc}${SLASH_ENSMEM_SUBDIR})}"
+  export COMIN="${COMIN:-$(compath.py -o ${NET}/${model_ver}/${RUN}.${PDY})}"
+  export COMOUT="${COMOUT:-$(compath.py -o ${NET}/${model_ver}/${RUN}.${PDY})}"
 else
-  export COMIN="${COMIN:-${COMROOT}/${NET}/${model_ver}/${RUN}.${PDY}/${cyc}${SLASH_ENSMEM_SUBDIR}}"
-  export COMOUT="${COMOUT:-${COMROOT}/${NET}/${model_ver}/${RUN}.${PDY}/${cyc}${SLASH_ENSMEM_SUBDIR}}"
+  export COMIN="${COMIN:-${COMROOT}/${NET}/${model_ver}/${RUN}.${PDY}}"
+  export COMOUT="${COMOUT:-${COMROOT}/${NET}/${model_ver}/${RUN}.${PDY}}"
 fi
 
 mkdir -p ${COMOUT}
 
 # Create a teomporary share directory
-export DATA_SHARE="${DATA_SHARE:-${DATAROOT}/DATA_SHARE/${PDY}${cyc}}"
+export DATA_SHARE="${DATA_SHARE:-${DATAROOT}/DATA_SHARE/${PDY}}"
 mkdir -p ${DATA_SHARE}
-
-# Run setpdy to initialize PDYm and PDYp variables
-export cycle="${cycle:-t${cyc}z}"
-setpdy.sh
-. ./PDY
 #
 #-----------------------------------------------------------------------
 #

jobs/JLANDDA_PREP_BMAT → jobs/JLANDDA_PRE_ANAL

@@ -56,25 +56,27 @@ cd $DATA
 export NET="${NET:-landda}"
 export RUN="${RUN:-landda}"
 
+# Run setpdy to initialize PDYm and PDYp variables
+export cycle="${cycle:-t${cyc}z}"
+setpdy.sh
+. ./PDY
+
 [[ "$WORKFLOW_MANAGER" = "rocoto" ]] && export COMROOT=$COMROOT
 if [ "${MACHINE}" = "WCOSS2" ]; then
-  export COMIN="${COMIN:-$(compath.py -o ${NET}/${model_ver}/${RUN}.${PDY}/${cyc}${SLASH_ENSMEM_SUBDIR})}"
-  export COMOUT="${COMOUT:-$(compath.py -o ${NET}/${model_ver}/${RUN}.${PDY}/${cyc}${SLASH_ENSMEM_SUBDIR})}"
+  export COMIN="${COMIN:-$(compath.py -o ${NET}/${model_ver}/${RUN}.${PDY})}"
+  export COMOUT="${COMOUT:-$(compath.py -o ${NET}/${model_ver}/${RUN}.${PDY})}"
 else
-  export COMIN="${COMIN:-${COMROOT}/${NET}/${model_ver}/${RUN}.${PDY}/${cyc}${SLASH_ENSMEM_SUBDIR}}"
-  export COMOUT="${COMOUT:-${COMROOT}/${NET}/${model_ver}/${RUN}.${PDY}/${cyc}${SLASH_ENSMEM_SUBDIR}}"
+  export COMIN="${COMIN:-${COMROOT}/${NET}/${model_ver}/${RUN}.${PDY}}"
+  export COMOUT="${COMOUT:-${COMROOT}/${NET}/${model_ver}/${RUN}.${PDY}}"
 fi
 
 mkdir -p ${COMOUT}
 
 # Create a teomporary share directory
-export DATA_SHARE="${DATA_SHARE:-${DATAROOT}/DATA_SHARE/${PDY}${cyc}}"
+export DATA_SHARE="${DATA_SHARE:-${DATAROOT}/DATA_SHARE/${PDY}}"
 mkdir -p ${DATA_SHARE}
-
-# Run setpdy to initialize PDYm and PDYp variables
-export cycle="${cycle:-t${cyc}z}"
-setpdy.sh
-. ./PDY
+export DATA_RESTART="${DATA_RESTART:-${DATAROOT}/DATA_SHARE/RESTART}"
+mkdir -p ${DATA_RESTART}
 #
 #-----------------------------------------------------------------------
 #
@@ -85,7 +87,7 @@ setpdy.sh
 export pgmout="${DATA}/OUTPUT.$$"
 env
 
-${SCRIPTSlandda}/exlandda_prep_bmat.sh
+${SCRIPTSlandda}/exlandda_pre_anal.sh
 export err=$?; err_chk
 
 if [ -e "$pgmout" ]; then

modulefiles/tasks/hera/task.analysis.lua

@@ -2,7 +2,7 @@ prepend_path("MODULEPATH", os.getenv("modulepath_spack_stack"))
 prepend_path("MODULEPATH", os.getenv("modulepath_spack_stack_jedi"))
 
 load(pathJoin("stack-intel", stack_intel_ver))
-load(pathJoin("stack-intel-oneapi-mpi", stack_impi_ver))
+load(pathJoin("stack-intel-oneapi-mpi", stack_intel_oneapi_mpi_ver))
 load(pathJoin("stack-python", stack_python_ver))
 
 load(pathJoin("prod_util", prod_util_ver))

modulefiles/tasks/hera/task.forecast.lua

@@ -2,7 +2,7 @@ prepend_path("MODULEPATH", os.getenv("modulepath_spack_stack"))
 prepend_path("MODULEPATH", os.getenv("modulepath_spack_stack_jedi"))
 
 load(pathJoin("stack-intel", stack_intel_ver))
-load(pathJoin("stack-intel-oneapi-mpi", stack_impi_ver))
+load(pathJoin("stack-intel-oneapi-mpi", stack_intel_oneapi_mpi_ver))
 load(pathJoin("stack-python", stack_python_ver))
 
 load(pathJoin("prod_util", prod_util_ver))

modulefiles/tasks/orion/task.prep_bmat.lua → modulefiles/tasks/hera/task.post_anal.lua

@@ -2,7 +2,7 @@ prepend_path("MODULEPATH", os.getenv("modulepath_spack_stack"))
 prepend_path("MODULEPATH", os.getenv("modulepath_spack_stack_jedi"))
 
 load(pathJoin("stack-intel", stack_intel_ver))
-load(pathJoin("stack-intel-oneapi-mpi", stack_impi_ver))
+load(pathJoin("stack-intel-oneapi-mpi", stack_intel_oneapi_mpi_ver))
 load(pathJoin("stack-python", stack_python_ver))
 
 load(pathJoin("prod_util", prod_util_ver))

modulefiles/tasks/orion/task.prep_exp.lua → modulefiles/tasks/hera/task.pre_anal.lua

@@ -2,7 +2,7 @@ prepend_path("MODULEPATH", os.getenv("modulepath_spack_stack"))
 prepend_path("MODULEPATH", os.getenv("modulepath_spack_stack_jedi"))
 
 load(pathJoin("stack-intel", stack_intel_ver))
-load(pathJoin("stack-intel-oneapi-mpi", stack_impi_ver))
+load(pathJoin("stack-intel-oneapi-mpi", stack_intel_oneapi_mpi_ver))
 load(pathJoin("stack-python", stack_python_ver))
 
 load(pathJoin("prod_util", prod_util_ver))

modulefiles/tasks/hera/task.prep_obs.lua

@@ -2,7 +2,7 @@ prepend_path("MODULEPATH", os.getenv("modulepath_spack_stack"))
 prepend_path("MODULEPATH", os.getenv("modulepath_spack_stack_jedi"))
 
 load(pathJoin("stack-intel", stack_intel_ver))
-load(pathJoin("stack-intel-oneapi-mpi", stack_impi_ver))
+load(pathJoin("stack-intel-oneapi-mpi", stack_intel_oneapi_mpi_ver))
 load(pathJoin("stack-python", stack_python_ver))
 
 load(pathJoin("prod_util", prod_util_ver))

modulefiles/tasks/orion/task.analysis.lua

@@ -2,7 +2,7 @@ prepend_path("MODULEPATH", os.getenv("modulepath_spack_stack"))
 prepend_path("MODULEPATH", os.getenv("modulepath_spack_stack_jedi"))
 
 load(pathJoin("stack-intel", stack_intel_ver))
-load(pathJoin("stack-intel-oneapi-mpi", stack_impi_ver))
+load(pathJoin("stack-intel-oneapi-mpi", stack_intel_oneapi_mpi_ver))
 load(pathJoin("stack-python", stack_python_ver))
 
 load(pathJoin("prod_util", prod_util_ver))