Feature/href MPMD and other fixes #647

BinbinZhou-NOAA · 2025-01-08T20:10:08Z

Note to developers: You must use this PR template!

Description of Changes

The MPMD is applied in both stat and plot generation jobs.
Most of the issues for cam/ens (href) listed in the EVSv2 Fixes and Additions document have been addressed
Some other issues that are not listed in the EVSv2 Fixes and Additions document

Please include a summary of the changes and the related GitHub issue(s). Please also include relevant motivation and context.
The detailed changes are listed as below:
1: Run all jobs in MET/METplus 6.0.0-rc1

Applied MPMD in both stats and plots jobs ( EVSv2 Fixes and Additions No.3)
Changed subprocess.Popen to subprocess.run in EVS v2.0 (general in EVSv2 Fixes and Additions)
This change also requires ps.communicated()[0] to be changed to ps.stdout to work correctly
Defined $NET prior to $HOMEevs; apply this change across all J-jobs (general in EVSv2 Fixes and Additions)
Removed old concatenating log file in stats job, but keep it in plotting job (general)
Removed LOOP_ORDER, LOG_LINE_FORMAT, etc from conf file (general)
Updated CAPE setting in the conf files (No. 14)
Changed "past" to "last" for the plot file names (in dev, and ecf) (general)
Followed the EVSv2 file naming standards (No. 13)
Separated valid times for snowfall plots job (old one combined all validation times together)
Required actual cpu resources used for number of the jobs. (general)
Reduced the subscribed memory in plotting jobs to match it is actually used. (general)
Fixed unnecessary gather processes in StatAnalysis (No. 6)
Fixed the "empty CAPE data" issue in SPCoutlook job in case of extreme cold weather and extreme small risk spcoutlook areas (listed in EVSv2 Fixes and Additions No. 7)
Updated lead_average.py file to address the empty data set in cape plotting ( No. 2)
Reduced the walltime for the grid2obs stats jobs, from 4hr 20min to 2hr
Replaced mater_metplus.py with run_metplus.py in ush/cam/evs_href_prepare.sh ( No. 8)
Removed all *.txt file in the href's parm directory (No.12)
Deleted file evs_href_spcoutlook_cape.sh from/ush/cam directory ( script is not used)
Added checking input forecast files before running metplus in the precip and grid2obs stat scripts (No. 4)
All scripts have "set -x" (EVSv2 Fixes and Additions No.18)
Changed [email protected] to [email protected] in all jobs ( EVSv2 Fixes and Additions No.11)
Increased memory used in 90days plot jobs for precip, cape, ctc and snowfall (EVSv2 Fixes and Additions No.16)
Added checking file existence before copy netCDF files to href.YYYYMMDD/precip_mean24 (No.17)

Developer Questions and Checklist

Is this a high priorty PR? If so, why and is there a date it needs to be merged by? No
Do you have any planned upcoming annual leave/PTO? No
Are there any changes needed for when the jobs are supposed to run? No

[y ] The code changes follow [NCO's EE2 Standards] (https://www.nco.ncep.noaa.gov/idsb/implementation_standards/ImplementationStandards.v11.0.0.pdf).
[ y] Developer's name is removed throughout the code and have used ${USER} where necessary throughout the code.
[ y] References the feature branch for HOMEevs are removed from the code.
[ y] J-Job environment variables, COMIN and COMOUT directories, and output follow what has been defined for EVS.
[ y] Jobs over 15 minutes in runtime have restart capability.
[ y] If applicable, changes in the dev/drivers/scripts or dev/modulefiles have been made in the corresponding ecf/scripts and ecf/defs/evs-nco.def?
[ y] Jobs contain the approriate file checking and don't run METplus for any missing data.
[ y] Code is using METplus wrappers structure and not calling MET executables directly.
[ y] Log is free of any ERRORs or WARNINGs.

Testing Instructions

Please include testing instructions for the PR assignee. Include all relevant input datasets needed to run the tests.
Since many scripts in bot stats and plots jobs have been updated, bot stats and plots jobs needed to test:\

For the stats, in the dev script directory, run following 3 jobs
jevs_cam_href_grid2obs_stats.sh
jevs_cam_href_precip_stats.sh
jevs_cam_href_spcoutlook_stats.sh
For the plots, run following 31day and 90 day jobs
Before testing, set COMIN=/lfs/h2/emc/vpppg/noscrub/emc.vpppg/evs/v2.0

jevs_cam_href_grid2obs_cape_last31days_plots.sh
jevs_cam_href_grid2obs_cape_last90days_plots.sh
jevs_cam_href_grid2obs_ctc_last31days_plots.sh
jevs_cam_href_grid2obs_ctc_last90days_plots.sh
jevs_cam_href_grid2obs_ecnt_last31days_plots.sh
jevs_cam_href_grid2obs_ecnt_last90days_plots.sh
jevs_cam_href_precip_last31days_plots.sh
jevs_cam_href_precip_last90days_plots.sh
jevs_cam_href_precip_spatial_plots.sh
jevs_cam_href_profile_last31days_plots.sh
jevs_cam_href_profile_last90days_plots.sh
jevs_cam_href_snowfall_last31days_plots.sh
jevs_cam_href_snowfall_last90days_plots.sh
jevs_cam_href_spcoutlook_last31days_plots.sh
jevs_cam_href_spcoutlook_last90days_plots.sh

malloryprow · 2025-01-08T20:30:50Z

Lots of great updates here @BinbinZhou-NOAA! I'll start reviewing and testing on Friday.

…A/EVS into feature/href_MPMD * 'feature/href_MPMD' of https://github.com/BinbinZhou-NOAA/EVS: Fixed the plots codes to prevent potential failure. (NOAA-EMC#645) Fixed the plots codes to prevent potential failure. (NOAA-EMC#642)

ecf/scripts/plots/cam/jevs_cam_href_grid2obs_cape_last31days_plots.ecf

ecf/scripts/plots/cam/jevs_cam_href_grid2obs_cape_last90days_plots.ecf

ecf/scripts/plots/cam/jevs_cam_href_grid2obs_ctc_last31days_plots.ecf

ecf/scripts/plots/cam/jevs_cam_href_grid2obs_ctc_last90days_plots.ecf

ecf/scripts/plots/cam/jevs_cam_href_grid2obs_ecnt_last31days_plots.ecf

ecf/scripts/plots/cam/jevs_cam_href_grid2obs_ecnt_last90days_plots.ecf

ecf/scripts/plots/cam/jevs_cam_href_precip_last31days_plots.ecf

ecf/scripts/plots/cam/jevs_cam_href_precip_last90days_plots.ecf

ecf/scripts/plots/cam/jevs_cam_href_profile_last31days_plots.ecf

ecf/scripts/plots/cam/jevs_cam_href_profile_last90days_plots.ecf

ecf/scripts/plots/cam/jevs_cam_href_snowfall_last31days_plots.ecf

ecf/scripts/plots/cam/jevs_cam_href_snowfall_last90days_plots.ecf

ecf/scripts/plots/cam/jevs_cam_href_spcoutlook_last31days_plots.ecf

ecf/scripts/plots/cam/jevs_cam_href_spcoutlook_last90days_plots.ecf

malloryprow · 2025-01-10T13:05:52Z

@BinbinZhou-NOAA I noticed in the HREF METplus config files that there are settings that start with POINT_STAT_NC_PAIRS_FLAG_. This is not a real METplus setting (search METplus Configuration Glossary). Can these be removed?

scripts/plots/cam/exevs_href_grid2obs_ecnt_plots.sh

malloryprow · 2025-01-10T14:22:33Z

One thing I noticed in the plots ex-scripts is the to the job scripts is writing for run_all_poe.sh, it checks for the file existing before copying to restart, which is good. However, restart is in COMOUT and there is no check that SENDCOM = YES.

Example: scripts/plots/cam/exevs_href_spcoutlook_plots.sh
Line 18: restart=$COMOUT/restart/$last_days/href_spcoutlook_plots
Lines 175-178: Checking for images, copy to restart, but no check for SENDCOM = YES

I also see that the code does cd $restart and then renames files, so it is working from COMOUT to rename the files. Can the files be renamed in DATA and then copied to COMOUT?

Example: scripts/plots/cam/exevs_href_spcoutlook_plots.sh
Line 213: cd $restart
Line 227: Renaming files in COMOUT

scripts/stats/cam/exevs_href_grid2obs_stats.sh

scripts/stats/cam/exevs_href_precip_stats.sh

ush/cam/evs_href_grid2obs_product.sh

ush/cam/evs_href_grid2obs_profile.sh

BinbinZhou-NOAA · 2025-01-10T14:48:53Z

Mallory, These 2 txt files have been removed days ago. Please double check. Binbin

…

On Fri, Jan 10, 2025 at 7:16 AM Mallory Row ***@***.***> wrote: Hi @BinbinZhou-NOAA <https://github.com/BinbinZhou-NOAA>! THere were two other files I have caught in recent reveiws of EVS: - parm/metplus_config/stats/cam/precip/note.txt - parm/metplus_config/stats/cam/precip/grib2_mrms.txt Are these needed? — Reply to this email directly, view it on GitHub <#647 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQBMPFPRPJAAMPHJF4TLKK32J62YTAVCNFSM6AAAAABU2WAKNKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKOBSGU3TOMBRGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Binbin Zhou Physical Scientist Lynker at NOAA/NWS/NCEP/EMC 5830 University Research Ct. College Park, MD 20740 ***@***.*** 301-683-3683

ush/cam/evs_href_grid2obs_system.sh

BinbinZhou-NOAA · 2025-01-10T15:00:56Z

Mallory, You suggest removing "exclhost" from all of the plotting drive scripts. But it has been removed. No such an option in those plotting driver scripts. Binbin

…

On Fri, Jan 10, 2025 at 7:31 AM Mallory Row ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In ecf/scripts/plots/cam/jevs_cam_href_grid2obs_cape_last31days_plots.ecf <#647 (comment)>: > #PBS -j oe #PBS -S /bin/bash #PBS -q %QUEUE% #PBS -A %PROJ%-%PROJENVIR% #PBS -l walltime=00:15:00 -#PBS -l place=vscatter:exclhost,select=10:ncpus=84:mem=100GB +#PBS -l place=vscatter:exclhost,select=9:ncpus=85:mem=20GB Remove exclhost — Reply to this email directly, view it on GitHub <#647 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQBMPFKNMX6EPAVJCZEYK3T2J64SNAVCNFSM6AAAAABU2WAKNKVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDKNBSGMZTCNZQHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Binbin Zhou Physical Scientist Lynker at NOAA/NWS/NCEP/EMC 5830 University Research Ct. College Park, MD 20740 ***@***.*** 301-683-3683

malloryprow · 2025-01-10T15:03:54Z

The suggestions are for the ecf scripts to match the dev driver scripts.

BinbinZhou-NOAA · 2025-01-10T15:05:11Z

I did not update the log history of all of the changed files. Sure I'll do it after the testing is completed. Thanks!

…

On Fri, Jan 10, 2025 at 10:01 AM Mallory Row ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In ush/cam/evs_href_grid2obs_system.sh <#647 (comment)>: > @@ -2,17 +2,16 @@ #************************************************************************* # Purpose: Generate href grid2obs ecnt poe and sub-jobs files # Last update: -# 04/25/2024, add restart, by Binbin Zhou ***@***.***/NCEP -# 10/30/2023, by Binbin Zhou ***@***.***/NCEP +# 10/30/2024, by Binbin Zhou ***@***.***/NCEP Does this need to be a more recent date? — Reply to this email directly, view it on GitHub <#647 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQBMPFLRQEKCCOAAYTVIKDT2J7ODLAVCNFSM6AAAAABU2WAKNKVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDKNBSGY2TQMRRG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Binbin Zhou Physical Scientist Lynker at NOAA/NWS/NCEP/EMC 5830 University Research Ct. College Park, MD 20740 ***@***.*** 301-683-3683

ush/cam/evs_href_precip.sh

ush/cam/evs_href_snowfall.sh

ush/cam/evs_href_spcoutlook.sh

malloryprow · 2025-01-10T15:08:49Z

For some ush scripts:

ush/cam/evs_href_grid2obs_product.sh
ush/cam/evs_href_grid2obs_profile.sh
ush/cam/evs_href_grid2obs_system.sh
ush/cam/evs_href_precip.sh
ush/cam/evs_href_spcoutlook.sh
ush/cam/evs_href_prepare.sh
ush/cam/evs_href_snowfall.sh

Change of previous to precious; change it back to previous
Example: ush/cam/evs_href_grid2obs_product.sh, line 44
Add checks for SENDCOM=YES before copying to COMOUT
Example: ush/cam/evs_href_grid2obs_product.sh, line 114

BinbinZhou-NOAA · 2025-01-10T15:09:07Z

For this one, let me test it again. Thanks!

…

On Fri, Jan 10, 2025 at 8:57 AM Mallory Row ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In scripts/plots/cam/exevs_href_grid2obs_ecnt_plots.sh <#647 (comment)>: > @@ -222,17 +216,17 @@ chmod +x run_all_poe.sh # Run the POE script in parallel or in sequence order to generate png files #************************************************************************** if [ $run_mpi = yes ] ; then - mpiexec -np 66 -ppn 33 -depth 2 --cpu-bind verbose,depth cfp ${DATA}/run_all_poe.sh + mpiexec -np 66 -ppn 33 -depth 2 --cpu-bind verbose,depth cfp ${DATA}/scripts/run_all_poe.sh I believe -depth 2 needs to be removed here. — Reply to this email directly, view it on GitHub <#647 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQBMPFJVSWADJ4UBXJEF2YD2J7GUVAVCNFSM6AAAAABU2WAKNKVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDKNBSGUYDANRTGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Binbin Zhou Physical Scientist Lynker at NOAA/NWS/NCEP/EMC 5830 University Research Ct. College Park, MD 20740 ***@***.*** 301-683-3683

malloryprow · 2025-01-10T15:09:55Z

@BinbinZhou-NOAA I finished my initial review. It may be easiest to see and review everything looking at PR conversionsation on a web browser. (#647)

BinbinZhou-NOAA · 2025-01-10T15:20:16Z

Mallory, Thanks for your code review and catching up with several issues. I'll update them one by one accordingly. Binbin

…

On Fri, Jan 10, 2025 at 10:10 AM Mallory Row ***@***.***> wrote: @BinbinZhou-NOAA <https://github.com/BinbinZhou-NOAA> I finished my initial review. It may be easiest to see and review everything looking at PR conversionsation on a web browser. (#647 <#647>) — Reply to this email directly, view it on GitHub <#647 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQBMPFOZQBWSNCUD5IKUQ732J7PFVAVCNFSM6AAAAABU2WAKNKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKOBSHEZDGNZSHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Binbin Zhou Physical Scientist Lynker at NOAA/NWS/NCEP/EMC 5830 University Research Ct. College Park, MD 20740 ***@***.*** 301-683-3683

…A/EVS into feature/href_MPMD * 'feature/href_MPMD' of https://github.com/BinbinZhou-NOAA/EVS: feature/rm_cpreq (NOAA-EMC#643)

malloryprow · 2025-01-13T12:58:03Z

scripts/plots/cam/exevs_href_profile_plots.sh

@@ -244,16 +254,16 @@ chmod +x run_all_poe.sh
 # Run the POE script in parallel or in sequence order to generate png files
 #**************************************************************************
 if [ $run_mpi = yes ] ; then
-   mpiexec -np 60 -depth 1 --cpu-bind verbose,depth cfp ${DATA}/run_all_poe.sh
+   mpiexec -np 60 -depth 1 --cpu-bind verbose,depth cfp ${DATA}/scripts/run_all_poe.sh


I believe this should be mpiexec np 60 -ppn 60

malloryprow · 2025-01-13T13:18:37Z

The new changes are looking good! A few more things (a few were noted previously but not sure if they are still being worked on so sorry if that is the case!)...

In the scripts being created to run in run_all_poe.sh, after echo "#!/bin/ksh" >> [script name], can there be a echo "set -x" >> [script name] be added?
I believe the *.completed files for restart are being written directly to COMOUT. They need to be written to DATA first and then copied to COMOUT (check SENDCOM=YES).
Change of previous to precious; change it back to previous
Example: ush/cam/evs_href_grid2obs_product.sh, line 44
Add checks for SENDCOM=YES before copying to COMOUT for stats
Example: ush/cam/evs_href_grid2obs_product.sh, line 115; ush/cam/evs_href_grid2obs_profile.sh, line 478

BinbinZhou-NOAA · 2025-01-13T13:41:32Z

Mallory, Sure, I'll update the script accordingly. Thanks! Binbin

…

On Mon, Jan 13, 2025 at 8:19 AM Mallory Row ***@***.***> wrote: The new changes are looking good! A few more things (a few were noted previously but not sure if they are still being worked on so sorry if that is the case!)... 1. In the scripts being created to run in run_all_poe.sh, after echo "#!/bin/ksh" >> [script name], can there be a echo "set -x" >> [script name] be added? 2. I believe the *.completed files for restart are being written directly to COMOUT. They need to be written to DATA first and then copied to COMOUT (check SENDCOM=YES). 3. Change of previous to precious; change it back to previous Example: ush/cam/evs_href_grid2obs_product.sh, line 44 4. Add checks for SENDCOM=YES before copying to COMOUT for stats Example: ush/cam/evs_href_grid2obs_product.sh, line 115; ush/cam/evs_href_grid2obs_profile.sh, line 478 — Reply to this email directly, view it on GitHub <#647 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQBMPFPVXISPXOQFKE6GTUT2KO4MHAVCNFSM6AAAAABU2WAKNKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKOBXGA4DINZXHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Binbin Zhou Physical Scientist Lynker at NOAA/NWS/NCEP/EMC 5830 University Research Ct. College Park, MD 20740 ***@***.*** 301-683-3683

…A/EVS into feature/href_MPMD * 'feature/href_MPMD' of https://github.com/BinbinZhou-NOAA/EVS: Address Bugzilla 1606 - counting of models in (NOAA-EMC#648)

BinbinZhou-NOAA · 2025-01-15T00:56:52Z

Mallory, I have pushed a new commit for this PR. Please double check. Thanks! Binbin On Mon, Jan 13, 2025 at 8:41 AM Binbin Zhou - NOAA Affiliate < ***@***.***> wrote:

…

Mallory, Sure, I'll update the script accordingly. Thanks! Binbin On Mon, Jan 13, 2025 at 8:19 AM Mallory Row ***@***.***> wrote: > The new changes are looking good! A few more things (a few were noted > previously but not sure if they are still being worked on so sorry if that > is the case!)... > > 1. In the scripts being created to run in run_all_poe.sh, after echo > "#!/bin/ksh" >> [script name], can there be a echo "set -x" >> > [script name] be added? > 2. I believe the *.completed files for restart are being written > directly to COMOUT. They need to be written to DATA first and then copied > to COMOUT (check SENDCOM=YES). > 3. Change of previous to precious; change it back to previous > Example: ush/cam/evs_href_grid2obs_product.sh, line 44 > 4. Add checks for SENDCOM=YES before copying to COMOUT for stats > Example: ush/cam/evs_href_grid2obs_product.sh, line 115; > ush/cam/evs_href_grid2obs_profile.sh, line 478 > > — > Reply to this email directly, view it on GitHub > <#647 (comment)>, or > unsubscribe > <https://github.com/notifications/unsubscribe-auth/AQBMPFPVXISPXOQFKE6GTUT2KO4MHAVCNFSM6AAAAABU2WAKNKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKOBXGA4DINZXHE> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> > -- Binbin Zhou Physical Scientist Lynker at NOAA/NWS/NCEP/EMC 5830 University Research Ct. College Park, MD 20740 ***@***.*** 301-683-3683

-- Binbin Zhou Physical Scientist Lynker at NOAA/NWS/NCEP/EMC 5830 University Research Ct. College Park, MD 20740 ***@***.*** 301-683-3683

BinbinZhou-NOAA and others added 3 commits January 7, 2025 15:40

Add MPMD and update other scripts

4b21fc5

Adjust walltime and fix ush/cam/evs_href_prepare.sh

e15049e

Merge branch 'NOAA-EMC:develop' into feature/href_MPMD

43ce358

malloryprow requested review from malloryprow and AndrewBenjamin-NOAA January 8, 2025 20:28

malloryprow self-assigned this Jan 8, 2025

malloryprow added the NCO Bugzilla label Jan 8, 2025

malloryprow added this to the EVS v2.0.0 milestone Jan 8, 2025

malloryprow linked an issue Jan 8, 2025 that may be closed by this pull request

cam - ens: Address Bugzilla 1547 - MPMD processes share the same working directory #552

Open

20 tasks

BinbinZhou-NOAA added 4 commits January 9, 2025 14:12

Adjust memory for some plot jobs

2e4d214

delete strange file parm/metplus_config/stats/cam/precip/1

62a6450

delete parm/metplus_config/stats/cam/snowfall/note.txt

8123caf

malloryprow added the cleanup label Jan 10, 2025