Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python example of grid coordinates and updates to GitHub links #146

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion index.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ CMOR output has the following characteristics:

* For metadata, different MIPs may have different requirements, but these are accommodated by CMOR, within the constraints of the CF convention and as specified in the MIP tables (e.g. [CMIP6 MIP tables](https://github.com/PCMDI/cmip6-cmor-tables)).

* CMOR can rely on NetCDF4 [See unidata web page](http://www.unidata.ucar.edu/software/netcdf) to write the output files and can take advantage of its compression and chunking capabilities. In that case, compression is controlled with the MIP tables using the shuffle, deflate and deflate_level attributes, default values are respectively 0, 0 and 0(disable). It is worth noting that even when using NetCDF4, CMOR3 still produces NETCDF4 CLASSIC formatted output. This allows the file generated to be readable by any application that can read NetCDF3 provided they are re-linked against NetCDF4. When using the NetCDF4 library it is also still possible to write files that can be read through the NetCDF3 library by adding "_3" to the appropriate cmor_setup argument (see below). Note: CMOR3 **NOW** output NetCDF4 files by default. For CMIP6, the NetCDF4/NC_CLASSIC_Model mode is used (and chunking is **NOW** invoked... shuffle and deflation can be invoke on-demand by setting flags in the table. [example](https://github.com/PCMDI/cmor/blob/master/Test/speed_test_table_A#L1691-L1693)).
* CMOR can rely on NetCDF4 [See unidata web page](http://www.unidata.ucar.edu/software/netcdf) to write the output files and can take advantage of its compression and chunking capabilities. In that case, compression is controlled with the MIP tables using the shuffle, deflate and deflate_level attributes, default values are respectively 0, 0 and 0(disable). It is worth noting that even when using NetCDF4, CMOR3 still produces NETCDF4 CLASSIC formatted output. This allows the file generated to be readable by any application that can read NetCDF3 provided they are re-linked against NetCDF4. When using the NetCDF4 library it is also still possible to write files that can be read through the NetCDF3 library by adding "_3" to the appropriate cmor_setup argument (see below). Note: CMOR3 **NOW** output NetCDF4 files by default. For CMIP6, the NetCDF4/NC_CLASSIC_Model mode is used (and chunking is **NOW** invoked... shuffle and deflation can be invoke on-demand by setting flags in the table. [example](https://github.com/PCMDI/cmor/blob/main/Test/speed_test_table_A#L1691-L1693)).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Compression isn't controlled in the MIP tables - rather this would be the user input, no?

Note: CMOR3 Now output s NetCDF4 file by default (missing s)

..invoke d on demand.. (missing d)


* CMOR also must be linked against the udunits2 library [see http://www.unidata.ucar.edu/software/udunits/](http://www.unidata.ucar.edu/software/udunits/), which enables CMOR to check that the units attribute is correct[\[6\]](#6). Finally CMOR3 must also be linked against the uuid library [see http://www.ossp.org/pkg/lib/uuid](http://www.ossp.org/pkg/lib/uuid) in order to produce a unique tracking number for each file.

Expand Down
78 changes: 78 additions & 0 deletions mydoc/examples/example7.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
import cmor
import numpy
import os


var_data = numpy.random.random((3, 4, 2)) + 34.

x = numpy.arange(4, dtype=float)
x_bnds = numpy.array([[x_ - 0.5, x_ + 0.5] for x_ in x])
y = numpy.arange(3, dtype=float)
y_bnds = numpy.array([[y_ - 0.5, y_ + 0.5] for y_ in y])

lat = numpy.array([10, 20, 30]).reshape((3, 1))
lat = numpy.tile(lat, (1, 4))
lon = numpy.array([0, 90, 180, 270])
lon = numpy.tile(lon, (3, 1))
lon_bnds = numpy.zeros((3, 4, 4))
lat_bnds = numpy.zeros((3, 4, 4))

for j in range(3):
for i in range(4):
lon_bnds[j, i, 0] = (lon[j, i] - 45) % 360
lon_bnds[j, i, 1] = lon[j, i]
lon_bnds[j, i, 2] = (lon[j, i] + 45) % 360
lon_bnds[j, i, 3] = lon[j, i]
lat_bnds[j, i, 0] = lat[j, i]
lat_bnds[j, i, 1] = lat[j, i] - 5
lat_bnds[j, i, 2] = lat[j, i]
lat_bnds[j, i, 3] = lat[j, i] + 5

time = numpy.array([0, 1])
time_bnds = numpy.array([0, 1, 2])

ipth = opth = 'Test'
cmor.setup(inpath=ipth,
set_verbosity=cmor.CMOR_NORMAL,
netcdf_file_action=cmor.CMOR_REPLACE,
exit_control=cmor.CMOR_EXIT_ON_MAJOR)
cmor.dataset_json('CMOR_input_example.json')

# First, load the grids table to set up x and y axes and the lat-long grid
grid_table_id = cmor.load_table('CMIP6_grids.json')
cmor.set_table(grid_table_id)

y_axis_id = cmor.axis(table_entry='y_deg',
units='degrees',
coord_vals=y,
cell_bounds=y_bnds)
x_axis_id = cmor.axis(table_entry='x_deg',
units='degrees',
coord_vals=x,
cell_bounds=x_bnds)

grid_id = cmor.grid(axis_ids=[y_axis_id, x_axis_id],
latitude=lat,
longitude=lon,
latitude_vertices=lat_bnds,
longitude_vertices=lon_bnds)

# Now, load the Omon table to set up the time axis and variable
omon_table_id = cmor.load_table('CMIP6_Omon.json')
cmor.set_table(omon_table_id)

time_axis_id = cmor.axis(table_entry='time',
units='months since 1980',
coord_vals=time,
cell_bounds=time_bnds)

var_id = cmor.variable(table_entry='sos',
units='0.001',
axis_ids=[grid_id, time_axis_id])

cmor.write(var_id, var_data, 2)

filename = cmor.close(var_id, file_name=True)
print("Stored in:", filename)
cmor.close()
os.system("ncdump {}".format(filename))
10 changes: 5 additions & 5 deletions mydoc/mydoc_cmip6_validator.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,14 +54,14 @@ where:

PrePARE will verify that all attributes in the input file are present and conform to CMIP6 for publication into ESGF. We also recommand running the python program [cfchecker](https://pypi.python.org/pypi/cfchecker) created by the University of Reading in the UK to confirm that your file is CF-1 compliant.

* In order to validate all CMIP6 required attributes by PrePARE, a [Controlled Vocabulary file](https://github.com/PCMDI/cmip6-cmor-tables/blob/master/Tables/CMIP6_CV.json) is read by the program where a JSON dictionnary called ["required_global_attributes"](https://github.com/PCMDI/cmip6-cmor-tables/blob/master/Tables/CMIP6_CV.json#L3) point to a list of strings. Each element of that list corresponds to a global attribute.
* PrePARE can also use regular expressions to validate the value of the some global attributes. Here is an [example](https://github.com/PCMDI/cmip6-cmor-tables/blob/master/Tables/CMIP6_CV.json#L6343-L6344) used for variant_label.
* In order to validate all CMIP6 required attributes by PrePARE, a [Controlled Vocabulary file](https://github.com/PCMDI/cmip6-cmor-tables/blob/main/Tables/CMIP6_CV.json) is read by the program where a JSON dictionnary called ["required_global_attributes"](https://github.com/PCMDI/cmip6-cmor-tables/blob/a46dcbf17ec5e11af23dc2d55107f5e52afbcade/Tables/CMIP6_CV.json#L3) point to a list of strings. Each element of that list corresponds to a global attribute.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

, a (extra space)
dictionnary (typo extra n)
point s to a list of strings (missing s)

* PrePARE can also use regular expressions to validate the value of the some global attributes. Here is an [example](https://github.com/PCMDI/cmip6-cmor-tables/blob/a46dcbf17ec5e11af23dc2d55107f5e52afbcade/Tables/CMIP6_CV.json#L11554-L11555) used for variant_label.

* Institutions and institution_ids need to be registered into a list. PrePARE will only accept institutions which have been pre-registered for CMIP6 publication into ESGF. Click [here](https://github.com/PCMDI/cmip6-cmor-tables/blob/master/Tables/CMIP6_CV.json#L65) for the list of institutions. If you wish to register your institution write to the [cmor mailing list](mailto:[email protected]).
* Institutions and institution_ids need to be registered into a list. PrePARE will only accept institutions which have been pre-registered for CMIP6 publication into ESGF. Click [here](https://github.com/PCMDI/cmip6-cmor-tables/blob/a46dcbf17ec5e11af23dc2d55107f5e52afbcade/Tables/CMIP6_CV.json#L74) for the list of institutions. If you wish to register your institution write to the [cmor mailing list](mailto:[email protected]).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Registering an institution will require engagement with the appropriate project_CVs repo, e.g. CMIP6Plus_CVs, CMIP7_CVs, input4MIPs_CVs, and obs4MIPs-cmor-tables (which will migrate to obs4MIPs_CVs once we have all the tables etc in place)


* Source and Source ID also need to be registered for CMIP6 publication. Here is the [list](https://github.com/PCMDI/cmip6-cmor-tables/blob/master/Tables/CMIP6_CV.json#L93) of registered sources.
* Source and Source ID also need to be registered for CMIP6 publication. Here is the [list](https://github.com/PCMDI/cmip6-cmor-tables/blob/a46dcbf17ec5e11af23dc2d55107f5e52afbcade/Tables/CMIP6_CV.json#L125) of registered sources.

* Only experiments found in the Controlled Vocabulary files are accepted for CMIP6 publication. A list of [experiment_ids](https://github.com/PCMDI/cmip6-cmor-tables/blob/master/Tables/CMIP6_CV.json#L548) have been pre-defined including mandatory attributes. A warning will be displayed if one experiment attribute is missing or is not properly set by your program.
* Only experiments found in the Controlled Vocabulary files are accepted for CMIP6 publication. A list of [experiment_ids](https://github.com/PCMDI/cmip6-cmor-tables/blob/a46dcbf17ec5e11af23dc2d55107f5e52afbcade/Tables/CMIP6_CV.json#L3708) have been pre-defined including mandatory attributes. A warning will be displayed if one experiment attribute is missing or is not properly set by your program.

* grid and nominal_resolution are mandatory global attributes in CMIP6. PrePARE will make sure that these attributes are conformed to one of the following syntax:

Expand Down
2 changes: 1 addition & 1 deletion mydoc/mydoc_cmor3_CV.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ permalink: /mydoc_cmor3_CV/

### CMIP6 required global attributes

* [CMIP6_CV.json](https://github.com/PCMDI/cmor/blob/master/TestTables/CMIP6_CV.json)
* [CMIP6_CV.json](https://github.com/PCMDI/cmor/blob/main/TestTables/CMIP6_CV.json)

<details><summary markdown="span"><b>Click to expand example JSON section</b></summary>

Expand Down
2 changes: 1 addition & 1 deletion mydoc/mydoc_cmor3_api.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ Python: dataset_json(name)

*Arguments*:

* **name**: JSON file which contains all information needed by CMOR in the form of key:value. Here is an example: [CMIP6_input_example.json](https://github.com/PCMDI/cmip6-cmor-tables/blob/master/Tables/CMIP6_input_example.json)
* **name**: JSON file which contains all information needed by CMOR in the form of key:value. Here is an example: [CMIP6_input_example.json](https://github.com/PCMDI/cmip6-cmor-tables/blob/main/Tables/CMIP6_input_example.json)

Returns upon success:

Expand Down
6 changes: 3 additions & 3 deletions mydoc/mydoc_cmor3_c.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ permalink: /mydoc_cmor3_c/

### CMOR user input

* [CMOR_input_example.json](https://github.com/PCMDI/cmor/blob/master/Test/CMOR_input_example.json)
* [CMOR_input_example.json](https://github.com/PCMDI/cmor/blob/main/Test/CMOR_input_example.json)

<details><summary markdown="span"><b>Click to expand JSON file</b></summary>

Expand Down Expand Up @@ -96,8 +96,8 @@ permalink: /mydoc_cmor3_c/

### C source code

* [ipcc_test_code.c](https://github.com/PCMDI/cmor/blob/master/Test/ipcc_test_code.c)
* [reader_2D_3D.h](https://github.com/PCMDI/cmor/blob/master/Test/reader_2D_3D.h)
* [ipcc_test_code.c](https://github.com/PCMDI/cmor/blob/main/Test/ipcc_test_code.c)
* [reader_2D_3D.h](https://github.com/PCMDI/cmor/blob/main/Test/reader_2D_3D.h)

<details><summary markdown="span"><b>Click to expand C code</b></summary>

Expand Down
6 changes: 3 additions & 3 deletions mydoc/mydoc_cmor3_fortran.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ permalink: /mydoc_cmor3_fortran/

### CMOR user input

* [CMOR_input_example.json](https://github.com/PCMDI/cmor/blob/master/Test/CMOR_input_example.json)
* [CMOR_input_example.json](https://github.com/PCMDI/cmor/blob/main/Test/CMOR_input_example.json)

<details><summary markdown="span"><b>Click to expand json file</b></summary>

Expand Down Expand Up @@ -93,8 +93,8 @@ permalink: /mydoc_cmor3_fortran/

### Fortran source code

* [ipcc_test_code.f90](https://github.com/PCMDI/cmor/blob/master/Test/ipcc_test_code.f90)
* [reader_2D_3D.f90](https://github.com/PCMDI/cmor/blob/master/Test/reader_2D_3D.f90)
* [ipcc_test_code.f90](https://github.com/PCMDI/cmor/blob/main/Test/ipcc_test_code.f90)
* [reader_2D_3D.f90](https://github.com/PCMDI/cmor/blob/main/Test/reader_2D_3D.f90)

<details><summary markdown="span"><b>Click to expand Fortran code</b></summary>

Expand Down
Loading