JEDI bundle update with Spack Stack 1.5.1

[jkbk2004]

Need to update JEDI bundle following Spack Stack release

• Hera: /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.5.1/envs/unified-env/install/modulefiles/Core
• Orion: /work/noaa/epic/role-epic/spack-stack/orion/spack-stack-1.5.1/envs/unified-env/install/modulefiles/Core
• Test and validate with land DA ctests

[jkbk2004]

@rhaesung sounds like you are re-installing /scratch2/NAGAPE/epic/UFS_Land-DA/jedi. This directory is currently used in ctest. So ctest is broken. Can you restore December jedi version and add new version in different name? same thing on orion as well: /work/noaa/epic/UFS_Land-DA/jedi. Please, add ChangeLog file to record what you have done. All build steps, etc.

[jkbk2004]

@rhaesung I tried jedi_skylabv4.0 in those directories. But it still fails. Please, re-store December jedi version.

[rhaesung]

@rhaesung I tried jedi_skylabv4.0 in those directories. But it still fails. Please, re-store December jedi version.

@jkbk2004 JEDI-Skylabv4.0 is restored and ctest is working fine on Orion. Here are the steps:

$git clone -b develop --recursive https://github.com/ufs-community/land-DA_workflow.git
$ln -fs /work/noaa/epic/UFS_Land-DA/inputs
$cd land-DA_workflow
$module use modulefiles && module load landda_orion.intel
$mkdir build && cd build
$ecbuild ../ && make -j4
$salloc --qos=debug --tasks-per-node=6 --account=epic --time=00:30:00
$module use ../modulefiles && module load landda_orion.intel
$ctest

Checking on Hera now.

[rhaesung]

@rhaesung I tried jedi_skylabv4.0 in those directories. But it still fails. Please, re-store December jedi version.

@jkbk2004 JEDI-Skylabv4.0 is restored and ctest is working fine on Orion. Here are the steps:

$git clone -b develop --recursive https://github.com/ufs-community/land-DA_workflow.git $ln -fs /work/noaa/epic/UFS_Land-DA/inputs $cd land-DA_workflow $module use modulefiles && module load landda_orion.intel $mkdir build && cd build $ecbuild ../ && make -j4 $salloc --qos=debug --ntasks=6 --account=epic --time=00:30:00 $module use ../modulefiles && module load landda_orion.intel $ctest

Checking on Hera now.

@jkbk2004 I reconfirm that the ctest on Orion is working fine. Here is my testing directory: /work/noaa/epic/rkim/test2/land-DA_workflow/build/Testing.

[jkbk2004]

@rhaesung Ok! orion ctest works on my side. Sounds like some env variables are screwed up. Please, confirm DA cycling experiment as well on orion and leave a note for steps here. JEDI installation steps for both orion and hera as well.

[rhaesung]

@jkbk2004 I confirmed DA cycling experiments on Orion. Here is the testing directory: /work/noaa/epic/rkim/test2/landda_expts

To reproduce after building,
$ cd land-DA_workflow
$ ./do_submit_cycle.sh settings_DA_cycle_era5 or
./do_submit_cycle.sh settings_DA_cycle_gswp3

[rhaesung]

@jkbk2004 I also confirmed that the ctest and DA cycling experiments work fine on Hera. Here is the testing directory: /scratch2/NCEPDEV/stmp/Rhaesung.Kim/test1/land-DA_workflow/build/Testing and /scratch2/NCEPDEV/stmp/Rhaesung.Kim/test1/landda_expts

How to run the land DA on Hera:
$ git clone -b develop --recursive https://github.com/ufs-community/land-DA_workflow.git
$ ln -fs /scratch2/NAGAPE/epic/UFS_Land-DA/inputs
$ cd land-DA_workflow
$ module use modulefiles && module load landda_hera.intel
$ mkdir build && cd build
$ ecbuild ../ && make -j4
$ salloc --qos=debug --ntasks=6 --account=epic --time=00:30:00
$ module use ../modulefiles && module load landda_hera.intel
$ ctest
$ cd ..
$ ./do_submit_cycle.sh settings_DA_cycle_era5 or ./do_submit_cycle.sh settings_DA_cycle_gswp3

[rhaesung]

JEDI-SKYLAB 7.0 installation steps

1. Hera

The following is required for building new spack environments and for using spack to build and run software.

$module purge
$module use /scratch1/NCEPDEV/jcsda/jedipara/spack-stack/modulefiles
$module load miniconda/3.9.12
$module load ecflow/5.5.3
$module load mysql/8.0.31

For spack-stack-1.5.1 with Intel, load the following modules after loading miniconda and ecflow:

$module use /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.5.1/envs/unified-env/install/modulefiles/Core
$module load stack-intel/2021.5.0
$module load stack-intel-oneapi-mpi/2021.5.1
$module load stack-python/3.10.8

2. Orion

The following is required for building new spack environments and for using spack to build and run software.

$module purge
$module use /work/noaa/epic/role-epic/spack-stack/orion/modulefiles
$module load python/3.9.2
$module load ecflow/5.8.4
$module load mysql/8.0.31

For spack-stack-1.5.1 with Intel, load the following modules after loading miniconda and ecflow:

$module use /work/noaa/epic/role-epic/spack-stack/orion/spack-stack-1.5.1/envs/unified-env/install/modulefiles/Core
$module load stack-intel/2022.0.2
$module load stack-intel-oneapi-mpi/2021.5.1
$module load stack-python/3.10.8

3. For both Hera and Orion after the above steps:

$module load jedi-fv3-env
$module load ewok-env
$module load soca-env

$mkdir $JEDI_ROOT
$cd $JEDI_ROOT
$git clone -b release/skylab-v7 https://github.com/jcsda/jedi-bundle

$mkdir $JEDI_BUILD
$cd $JEDI_BUILD
$ecbuild $JEDI_SRC
$make -j8