Add handling of "Average" and "Alt_Average" data types to Signature #229
Conversation
stglib/sig/cdf2nc.py
Outdated
| print( | ||
| "make time encoding to dtype double because no sample dimension, round to milliseconds first" | ||
| f"make time encoding to dtype double because tstep {tstep} seconds is < 1 minute, round to milliseconds first" |
There was a problem hiding this comment.
Rather than printing this to the screen, please add it to the netCDF history using utils.insert_history (that will also print it to the screen).
There was a problem hiding this comment.
changed as suggested.
stglib/sig/cdf2nc.py
Outdated
| ) | ||
| ds["time"] = ds["time"].dt.round( | ||
| "s" | ||
| ) # round time to seconds if time interval >= 1 minute |
There was a problem hiding this comment.
Please put comments like this on their own line so black doesn't do this kind of unnecessary line wrapping
There was a problem hiding this comment.
changed as suggested.
| ) | ||
| ds["time"] = ds["time"].dt.round( | ||
| "s" | ||
| ) # round time to seconds if time interval >= 1 minute | ||
| ds["time"].encoding["dtype"] = "i4" |
There was a problem hiding this comment.
We should make sure this doesn't cause an overflow. Please add a check using utils.check_fits_in_int32
There was a problem hiding this comment.
Add time handling to utils.check_fits_in_int32 and added check for dtype 'i4' time encoding.
.gitattributes
Outdated
| @@ -4,3 +4,5 @@ stglib/_version.py export-subst | |||
| stglib/tests/data/055109_20220808_1605.zip filter=lfs diff=lfs merge=lfs -text | |||
| stglib/tests/data/055109_20220808_1605_config.yaml filter=lfs diff=lfs merge=lfs -text | |||
| stglib/tests/data/gatts_055109_20220808_1605.txt filter=lfs diff=lfs merge=lfs -text | |||
| stglib/tests/data/*.cdf filter=lfs diff=lfs merge=lfs -text | |||
There was a problem hiding this comment.
Ideally the test files should be listed in the .gitattributes file in stglib/tests/data, not this .gitattributes file.
There was a problem hiding this comment.
moved these lines to .gitattributes file in stglib/tests/data folder
stglib/core/utils.py
Outdated
| f"32-bit integer overflow on {var}; setting encoding to i4 will fail" | ||
| ) | ||
| return False | ||
| def check_fits_in_int32(ds, var, units="s"): |
There was a problem hiding this comment.
Whoops, sorry, I could have been more clear. There is already a check_time_fits_in_int32 function, defined below. Please use that one for time rather than changing this func def (unless there is something else I am missing).
.gitattributes
Outdated
| stglib/tests/data/055109_20220808_1605.zip filter=lfs diff=lfs merge=lfs -text | ||
| stglib/tests/data/055109_20220808_1605_config.yaml filter=lfs diff=lfs merge=lfs -text | ||
| stglib/tests/data/gatts_055109_20220808_1605.txt filter=lfs diff=lfs merge=lfs -text | ||
| stglib/_version.py export-subst |
There was a problem hiding this comment.
Please add a newline at the end of this file.
dnowacki-usgs
changed the title
Add handling of "Average" and "Alt_Average" data types to Signature processing. Additionally, had creation of instrument transformation matrix and use time step to determine time encoding instead of non-presences of "sample" dimension in data set. Test for new data type(s) added to test scripts.