Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement basic coarse filtering for ALL variables #17

Open
gremau opened this issue Jun 5, 2015 · 0 comments
Open

Implement basic coarse filtering for ALL variables #17

gremau opened this issue Jun 5, 2015 · 0 comments

Comments

@gremau
Copy link
Owner

gremau commented Jun 5, 2015

Lots of variables have problematic outliers or "spikes" that are very temporally limited ( < 1 day ), despite the qc process we have in place. Often I think this is just because they are relatively rarely used variables, or these outlier/spikes are uncommon and not explainable. Whatever the case, it would be nice to remove many of them with a simple coarse filter. For example:

Wind direction: Remove data < 0 and > 360
RH: Remove outliers > 100 and < 0
Other: Remove outliers greater than 3-4 standard deviations from the 1 or 2 week rolling mean.

This should probably occur as part of the "RemoveBadData" pipeline.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant