Remove the GTS BUFR2IODA part of the snow obs prep job #2900

jiaruidong2017 · 2024-09-09T18:37:55Z

Description

This PR removed the GTS BUFR2IODA part of the snow obs prep job, and replaced it with a direct read from BUFR snow data at runtime by the JEDI executable.

This PR depends on NOAA-EMC/GDASApp#1276

Type of change

Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO
Does this change require an update to any of the following submodules? NO

How has this been tested?

Cycled test on Hera

Checklist

Any dependent changes have been merged and published
My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
My changes generate no new warnings
New and existing tests pass with my changes
I have made corresponding changes to the documentation if necessary

Replace it with a direct read from BUFR at runtime.

aerorahul · 2024-09-09T18:55:39Z

@jiaruidong2017
I left a comment in the GDASapp PR that this PR depends on. It might have have an impact on this PR. Let me know if what I commented is confusing.

jiaruidong2017 · 2024-09-09T19:10:13Z

@jiaruidong2017 I left a comment in the GDASapp PR that this PR depends on. It might have have an impact on this PR. Let me know if what I commented is confusing.

Thanks @aerorahul for your suggestions.

CoryMartin-NOAA · 2024-09-11T17:50:51Z

This should partially solve #2902

CoryMartin-NOAA · 2024-09-11T19:53:24Z

ush/python/pygfs/task/snow_analysis.py

+        gts_config = parse_j2yaml(self.task_config.GTS_LIST, localconf)
+        logger.debug(f"{self.task_config.GTS_LIST}:\n{pformat(gts_config)}")
+
+        # Generate bufr2ioda mapping YAML files


why do we need to generate this? is it not static?

Yes, you are right. I made the changes to both PRs. Thanks @CoryMartin-NOAA

CoryMartin-NOAA · 2024-09-12T13:00:38Z

parm/config/gfs/config.snowanl

@@ -9,6 +9,7 @@ echo "BEGIN: config.snowanl"
 source "${EXPDIR}/config.resources" snowanl

 export OBS_LIST="${PARMgfs}/gdas/snow/obs/lists/gdas_snow.yaml.j2"
+export GTS_LIST="${PARMgfs}/gdas/snow/obs/config/bufr2ioda_mapping.yaml.j2"


I think I would be more descriptive here and name this GTS_SNOW_STAGE_YAML unless other reviewers find this too verbose

Made changes as suggested.

CoryMartin-NOAA · 2024-09-12T13:40:34Z

Will merge GDASApp PR whenever others are ready, then that hash can be updated here before CI testing

WalterKolczynski-NOAA

Approved pending update to GDAS hash.

This PR removed the GTS BUFR2IODA part of the snow obs prep job, and replaced it with a direct read from BUFR snow data at runtime by the JEDI executable. This PR contributes to NOAA-EMC/global-workflow#2900 --------- Co-authored-by: Cory Martin <[email protected]>

jiaruidong2017 · 2024-09-17T20:07:07Z

@WalterKolczynski-NOAA GDAS hash has been updated. Thanks.

emcbot · 2024-09-17T20:39:35Z

Build FAILED on Hera in Build# 1 with error logs:

/scratch1/NCEPDEV/global/CI/2900/gefs/sorc/logs/build_ufs_utils.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2024-09-19T00:58:50Z

CI Passed on Hera in Build# 2
Built and ran in directory /scratch1/NCEPDEV/global/CI/2900


Experiment C48mx500_3DVarAOWCDA_7de13607 Completed 2 Cycles: *SUCCESS* at Wed Sep 18 22:44:14 UTC 2024
Experiment C48_ATM_7de13607 Completed 1 Cycles: *SUCCESS* at Wed Sep 18 22:50:02 UTC 2024
Experiment C96_atm3DVar_7de13607 Completed 3 Cycles: *SUCCESS* at Wed Sep 18 23:57:25 UTC 2024
Experiment C96C48_hybatmDA_7de13607 Completed 3 Cycles: *SUCCESS* at Wed Sep 18 23:57:26 UTC 2024
Experiment C48_S2SWA_gefs_7de13607 Completed 1 Cycles: *SUCCESS* at Thu Sep 19 00:28:46 UTC 2024
Experiment C48_S2SW_7de13607 Completed 1 Cycles: *SUCCESS* at Thu Sep 19 00:34:21 UTC 2024
Experiment C96C48_ufs_hybatmDA_7de13607 Completed 2 Cycles: *SUCCESS* at Thu Sep 19 00:34:21 UTC 2024
Experiment C96C48_hybatmaerosnowDA_7de13607 Completed 3 Cycles: *SUCCESS* at Thu Sep 19 00:58:43 UTC 2024

emcbot · 2024-09-19T03:40:21Z

CI Update on Wcoss2 at 09/19/24 03:40:19 AM
============================================
Cloning and Building global-workflow PR: 2900
with PID: 118191 on host: clogin03

emcbot · 2024-09-19T04:32:41Z

Automated global-workflow Testing Results:

Machine: Wcoss2
Start: Thu Sep 19 03:50:04 UTC 2024 on clogin03
---------------------------------------------------
Build: Completed at 09/19/24 04:32:11 AM
Case setup: Completed for experiment C48_ATM_7de13607
Case setup: Skipped for experiment C48mx500_3DVarAOWCDA_7de13607
Case setup: Skipped for experiment C48_S2SWA_gefs_7de13607
Case setup: Completed for experiment C48_S2SW_7de13607
Case setup: Completed for experiment C96_atm3DVar_extended_7de13607
Case setup: Skipped for experiment C96_atm3DVar_7de13607
Case setup: Completed for experiment C96C48_hybatmaerosnowDA_7de13607
Case setup: Completed for experiment C96C48_hybatmDA_7de13607
Case setup: Completed for experiment C96C48_ufs_hybatmDA_7de13607

emcbot · 2024-09-19T07:39:24Z

CI Passed on Hercules in Build# 3
Built and ran in directory /work2/noaa/stmp/CI/HERCULES/2900


Experiment C48_ATM_7de13607 Completed 1 Cycles: *SUCCESS* at Thu Sep 19 01:02:25 CDT 2024
Experiment C96_atm3DVar_7de13607 Completed 3 Cycles: *SUCCESS* at Thu Sep 19 02:15:03 CDT 2024
Experiment C48_S2SWA_gefs_7de13607 Completed 1 Cycles: *SUCCESS* at Thu Sep 19 02:15:51 CDT 2024
Experiment C96C48_hybatmDA_7de13607 Completed 3 Cycles: *SUCCESS* at Thu Sep 19 02:21:10 CDT 2024
Experiment C48_S2SW_7de13607 Completed 1 Cycles: *SUCCESS* at Thu Sep 19 02:39:12 CDT 2024

emcbot · 2024-09-19T16:14:14Z

All CI Test Cases Passed on Wcoss2:

Experiment C48_ATM_7de13607 *** SUCCESS *** at 09/19/24 05:56:10 AM
Experiment C48_S2SW_7de13607 *** SUCCESS *** at 09/19/24 06:14:11 AM
Experiment C96C48_hybatmDA_7de13607 *** SUCCESS *** at 09/19/24 06:49:21 AM
Experiment C96C48_hybatmaerosnowDA_7de13607 *** SUCCESS *** at 09/19/24 07:49:22 AM
Experiment C96C48_ufs_hybatmDA_7de13607 *** SUCCESS *** at 09/19/24 08:42:17 AM
Experiment C96_atm3DVar_extended_7de13607 *** SUCCESS *** at 09/19/24 04:07:32 PM

Remove the GTS BUFR2IODA part of the snow obs prep job and

4098f10

Replace it with a direct read from BUFR at runtime.

jiaruidong2017 mentioned this pull request Sep 9, 2024

Remove the GTS BUFR2IODA part of the snow obs prep job NOAA-EMC/GDASApp#1276

Merged

jiaruidong2017 added 2 commits September 9, 2024 20:38

Change HOMEgfs to PARMgfs.

8d1bc87

Merge branch 'develop' into feature/snow_gtsobs

6ff65bc

Merge branch 'develop' into feature/snow_gtsobs

2295cab

CoryMartin-NOAA reviewed Sep 11, 2024

View reviewed changes

Made changes to stage the static GTS mapping files.

5074d11

CoryMartin-NOAA reviewed Sep 12, 2024

View reviewed changes

Change the name to be more descriptive.

f122edf

CoryMartin-NOAA previously approved these changes Sep 12, 2024

View reviewed changes

WalterKolczynski-NOAA added the blocked Issue is currently being blocked by another issue label Sep 13, 2024

Merge branch 'develop' into feature/snow_gtsobs

73b4ec0

WalterKolczynski-NOAA approved these changes Sep 17, 2024

View reviewed changes

WalterKolczynski-NOAA previously approved these changes Sep 17, 2024

View reviewed changes

jiaruidong2017 added 2 commits September 17, 2024 16:02

Merge branch 'develop' into feature/snow_gtsobs

be337ec

Update GDASApp's commit hash.

7de1360

jiaruidong2017 dismissed stale reviews from WalterKolczynski-NOAA and CoryMartin-NOAA via 7de1360 September 17, 2024 20:04

WalterKolczynski-NOAA added CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera and removed blocked Issue is currently being blocked by another issue labels Sep 17, 2024

emcbot added CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera and removed CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera labels Sep 17, 2024

emcbot removed the CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera label Sep 17, 2024

WalterKolczynski-NOAA added the CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera label Sep 18, 2024

jiaruidong2017 mentioned this pull request Sep 18, 2024

Replace separate JEDI radiance bias correction files with tarball #2875

Merged

10 tasks

emcbot added CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully and removed CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Sep 19, 2024

WalterKolczynski-NOAA added CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules CI-Wcoss2-Ready **CM use only** PR is ready for CI testing on WCOSS labels Sep 19, 2024

emcbot added CI-Hercules-Passed **Bot use only** CI testing on Hercules for this PR has completed successfully and removed CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress labels Sep 19, 2024

emcbot added CI-Wcoss2-Passed **Bot use only** CI testing on WCOSS for this PR has completed successfully and removed CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress labels Sep 19, 2024

WalterKolczynski-NOAA approved these changes Sep 19, 2024

View reviewed changes

WalterKolczynski-NOAA merged commit c6e3262 into NOAA-EMC:develop Sep 19, 2024
10 of 11 checks passed

jiaruidong2017 deleted the feature/snow_gtsobs branch September 19, 2024 17:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove the GTS BUFR2IODA part of the snow obs prep job #2900

Remove the GTS BUFR2IODA part of the snow obs prep job #2900

jiaruidong2017 commented Sep 9, 2024 •

edited

Loading

aerorahul commented Sep 9, 2024

jiaruidong2017 commented Sep 9, 2024

CoryMartin-NOAA commented Sep 11, 2024

CoryMartin-NOAA Sep 11, 2024

jiaruidong2017 Sep 12, 2024

CoryMartin-NOAA Sep 12, 2024

jiaruidong2017 Sep 12, 2024

CoryMartin-NOAA commented Sep 12, 2024

WalterKolczynski-NOAA left a comment

jiaruidong2017 commented Sep 17, 2024

emcbot commented Sep 17, 2024

emcbot commented Sep 19, 2024

emcbot commented Sep 19, 2024

emcbot commented Sep 19, 2024

emcbot commented Sep 19, 2024

emcbot commented Sep 19, 2024

Remove the GTS BUFR2IODA part of the snow obs prep job #2900

Remove the GTS BUFR2IODA part of the snow obs prep job #2900

Conversation

jiaruidong2017 commented Sep 9, 2024 • edited Loading

Description

Type of change

Change characteristics

How has this been tested?

Checklist

aerorahul commented Sep 9, 2024

jiaruidong2017 commented Sep 9, 2024

CoryMartin-NOAA commented Sep 11, 2024

CoryMartin-NOAA Sep 11, 2024

Choose a reason for hiding this comment

jiaruidong2017 Sep 12, 2024

Choose a reason for hiding this comment

CoryMartin-NOAA Sep 12, 2024

Choose a reason for hiding this comment

jiaruidong2017 Sep 12, 2024

Choose a reason for hiding this comment

CoryMartin-NOAA commented Sep 12, 2024

WalterKolczynski-NOAA left a comment

Choose a reason for hiding this comment

jiaruidong2017 commented Sep 17, 2024

emcbot commented Sep 17, 2024

emcbot commented Sep 19, 2024

emcbot commented Sep 19, 2024

emcbot commented Sep 19, 2024

emcbot commented Sep 19, 2024

emcbot commented Sep 19, 2024

jiaruidong2017 commented Sep 9, 2024 •

edited

Loading