Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Modifies how 15 minute output is processed #592

Merged
merged 13 commits into from
Jan 14, 2025
Merged

Modifies how 15 minute output is processed #592

merged 13 commits into from
Jan 14, 2025

Conversation

MatthewPyle-NOAA
Copy link
Contributor

@MatthewPyle-NOAA MatthewPyle-NOAA commented Jan 10, 2025 •
edited
Loading

Makes changes to generate a single top of the hour "subh" file that has 15 minute output for the preceding hour.

DESCRIPTION OF CHANGES:

  • Removes two column integrated condensate fields from subh UPP files, as they no longer are generated by the RRFS.
  • Adds a subh_fields.txt field to pull the records needed to create a 15 minute look alike file from the full top of hour file.
  • Adds logic to the post ex-script to concatenate the sub-hourly files into a single "subh" file. This likely will need modification with the coming umbrella directory changes.
  • Adds a line in the prdgen ex-script to copy the "subh" file to COMOUT.
  • Removes need to run prdgen job for sub-hourly times.

TESTS CONDUCTED:

Machines/Platforms:

  • WCOSS2
    • Cactus/Dogwood
    • Acorn
  • RDHPCS
    • Hera
    • Jet
    • Orion
    • Hercules

Test cases:

  • Engineering tests
    • Non-DA engineering test - Made forecast to 24 h, with 15 minute output for first 18 h to confirm that processing was only done for the period with 15 minute output.
    • DA engineering test
      • Retro
      • Ensemble
      • Parallel
  • RRFS fire weather
  • RRFS_A:
  • RRFS_B:
  • RTMA:
  • Others:

ISSUE:

CONTRIBUTORS (optional):

MatthewPyle-NOAA and others added 11 commits November 7, 2024 13:39
@MatthewPyle-NOAA
Trivial change to get things started
5ad9aef
@MatthewPyle-NOAA
Adds new subh_fields.txt file to pull needed records from full hour
efe3ccf 
output for 15 minute products.

Strips out TCOLW and TCOLI from the subh output (got removed from
product list long ago).
@MatthewPyle-NOAA
Updates subsetting file for top of hour 15 minute products, and adds
83fdfb5 
comment in post script of command needed to use it.
@MatthewPyle-NOAA
Merge branch 'NOAA-EMC:main' into feature/subh_mod
98d4448
@MatthewPyle-NOAA
Merge branch 'NOAA-EMC:main' into feature/subh_mod
9770ccd
@MatthewPyle-NOAA
Adds changes that seem to have the post side working properly to subset
0d6e9d9 
from the full hourly file.  Hasn't yet been tested for a full forecast
(to see if there are issues beyond f018) or with prdgen script (which
will need work).
@MatthewPyle-NOAA
Adds item to combine the 15 minute files at the top of the hour.
997b7cd 
Details may need to change, but believe this removes a need for
subhourly prdgen jobs.
@MatthewPyle-NOAA
Merge branch 'NOAA-EMC:main' into feature/subh_mod
910bca2
@MatthewPyle-NOAA
Adds updated prdgen script
570d636
@MatthewPyle-NOAA
Eliminates spurious "fi" statement that was creating an error.
47b8f0b
@MatthewPyle-NOAA
Cleans up some added blank lines and comments.
3e0004e
@MatthewPyle-NOAA MatthewPyle-NOAA marked this pull request as ready for review January 13, 2025 19:10
BenjaminBlake-NOAA
BenjaminBlake-NOAA previously approved these changes Jan 13, 2025
View reviewed changes
Copy link
Contributor

@BenjaminBlake-NOAA BenjaminBlake-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MatthewPyle-NOAA I left two comments but the changes look good to me. It will be great to not have to run the prdgen task at subhourly times anymore.

fix/upp/postxconfig-NT-rrfs_subh.txt Show resolved Hide resolved
scripts/exrrfs_post.sh Outdated
# expect this will need to be changed due to future umbrella directory changes
tm15=${DATA}/../rrfs_post_${envir}_${cyc}_f0${fhrm1}-45-00/PRSLEV.GrbF${fhrm1}.45
tm30=${DATA}/../rrfs_post_${envir}_${cyc}_f0${fhrm1}-30-00/PRSLEV.GrbF${fhrm1}.30
tm45=${DATA}/../rrfs_post_${envir}_${cyc}_f0${fhrm1}-15-00/PRSLEV.GrbF${fhrm1}.15
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes this directory structure will likely need to be updated but this is fine for now.

@MatthewPyle-NOAA
Adds some error checking to ensure that the needed 15 minute output
c1fb1a8 
is available, and to exit if it isn't after ~10 minutes.  Also cleaned
up the formatting for the new material to make the nested if blocks
clearer.
@MatthewPyle-NOAA
Attempt at adding POSTPROC_SUBH_LEN_HRS variable to control how long to
e5e71b7 
apply the 15 minute aggregating within exrrfs_post.sh
BenjaminBlake-NOAA
BenjaminBlake-NOAA reviewed Jan 14, 2025
View reviewed changes
Copy link
Contributor

@BenjaminBlake-NOAA BenjaminBlake-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MatthewPyle-NOAA Thanks for the additional changes. If you were able to complete a successful test with POSTPROC_SUBH_LEN_HRS then I can go ahead and re-approve this PR.

@MatthewPyle-NOAA
Copy link
Contributor Author

Thanks @BenjaminBlake-NOAA - I made a fresh test that worked well. So if you think I've defined it where it is needed, I think we're good.

@MatthewPyle-NOAA MatthewPyle-NOAA merged commit 95d388e into NOAA-EMC:main Jan 14, 2025
2 checks passed
@MatthewPyle-NOAA MatthewPyle-NOAA deleted the feature/subh_mod branch January 14, 2025 18:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@BenjaminBlake-NOAA BenjaminBlake-NOAA BenjaminBlake-NOAA approved these changes

@hu5970 hu5970 hu5970 approved these changes

@ShunLiu-NOAA ShunLiu-NOAA Awaiting requested review from ShunLiu-NOAA ShunLiu-NOAA is a code owner

@JacobCarley-NOAA JacobCarley-NOAA Awaiting requested review from JacobCarley-NOAA JacobCarley-NOAA is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@MatthewPyle-NOAA @hu5970 @BenjaminBlake-NOAA