Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Irene dev2 #39

Open
wants to merge 27 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
ab16004
add being able to load data from AGOL
Mar 19, 2024
d90c9cb
add dict for agol database urls
Mar 19, 2024
3072477
add filtering for data pulled from agol
Mar 20, 2024
ad8cc5f
adding new bmi functions
Mar 21, 2024
7bb031d
adding loading and writing bmi functions
Mar 21, 2024
06ea4c6
updating man files
Mar 21, 2024
4f0c83b
remove BMI stuff from SQL functions
Mar 21, 2024
5a1a296
Added package prefixes to functions; last commit before radical chang…
Apr 10, 2024
35b08b3
Created LoadStreamsAndLakes, ReadAGOL, ReadSqlDatabase, ReadAquarius,…
Apr 13, 2024
3595c06
Added Roxygen info to utils functions
Apr 17, 2024
33a435f
Updated all function arguments to accommodate LoadStreamsAndLakes
Apr 17, 2024
b07568a
Created new BMI data wrangling functions, updated BMI summary and vis…
Apr 22, 2024
7d044e2
Changed the way Aquarius database connection is stored in the environ…
May 9, 2024
91950b1
Added formatting function for plotting chem data; added new lake and …
May 10, 2024
50325c1
Added function to compare deprecated to recent data from BugLab
May 10, 2024
622787b
Minor change to plot formatting
May 10, 2024
31f1f6b
Updates to function names
May 10, 2024
594c608
Modified lake levels function to integrate string method authoritativ…
May 10, 2024
449a5b3
Minor updates to Roxygen
May 10, 2024
615548b
Added file to ignore
May 10, 2024
8925f0d
Fixed minor typos
May 10, 2024
b78c3ad
Added habit metrics plot function
May 10, 2024
447b71b
New function
May 10, 2024
a3dda91
Fixed minor errors
May 10, 2024
68e24f5
Minor updates to BMI metrics function based on updates to AGOL database
May 17, 2024
57172c8
Added back deleted columns in LakeSurfaceElevation function for calcu…
May 30, 2024
f3ef1ff
Updated end year from MDL and ML lookup
Jun 14, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .Rbuildignore
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
^.*\.Rproj$
^\.Rproj\.user$
test-space.R
2 changes: 1 addition & 1 deletion DESCRIPTION
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ Imports:
measurements,
statip,
stringr
RoxygenNote: 7.2.0
RoxygenNote: 7.3.1
Suggests:
testthat,
roxygen2
24 changes: 19 additions & 5 deletions NAMESPACE
Original file line number Diff line number Diff line change
Expand Up @@ -4,32 +4,44 @@ export(BMIDiversityMetricsPlot)
export(BMIFormatted)
export(BMIFunctionalMetricsPlot)
export(BMIGeneralMetricsPlot)
export(BMIHabitMetricsPlot)
export(BMILong)
export(BMIMetricsLong)
export(BMISensitivityMetricsPlot)
export(BMISpecies)
export(BMITaxonomicMetricsPlot)
export(BMIToleranceMetricsPlot)
export(BenchmarkConsistencyPlot)
export(ChannelCharacteristics)
export(ChannelFLow)
export(ChannelFlow)
export(ChannelSubstrate)
export(ChemANC)
export(ChemFormatted)
export(ChemLakeANCPlot)
export(ChemLakeIonPlot)
export(ChemLakeIonSplitPlot)
export(ChemLakeNutrientBarPlot)
export(ChemLakeNutrientPlot)
export(ChemLakeNutrientSplitPlot)
export(ChemStreamANCPlot)
export(ChemStreamIonPlot)
export(ChemStreamIonSplitPlot)
export(ChemStreamNutrientBarPlot)
export(ChemStreamNutrientPlot)
export(ChemStreamNutrientSplitPlot)
export(CloseDatabaseConnection)
export(FormatPlot)
export(GetRawData)
export(GetSiteName)
export(LakeSurfaceElevation)
export(LakeWqMedian)
export(OpenDatabaseConnection)
export(LoadStreamsAndLakes)
export(PlotBenchmarkElevation)
export(PlotLakeSurfaceElevation)
export(PlotStringComparisons)
export(ReadAndFilterData)
export(ReadCSV)
export(SaveDataToCsv)
export(StreamWqMedian)
export(StringSurveyElevation)
export(SurveyPointElevation)
export(WqDailyMean)
export(WqDailyMeanLong)
Expand All @@ -38,9 +50,10 @@ export(WqPlotPHDepthProfile)
export(WqPlotSpCondDepthProfile)
export(WqPlotTemperatureDepthProfile)
export(expect_dataframe_equal)
export(fetchAndWrangleAGOL)
export(getMDLLookup)
export(qcBMIDiscrepancies)
export(qcBenchmarkElevation)
export(qcBenchmarkConsistency)
export(qcChemFieldBlanks)
export(qcChemFieldDupes)
export(qcChemFlags)
Expand Down Expand Up @@ -72,6 +85,7 @@ export(qcWqCompletenessPlot)
export(qcWqGrades)
export(qcWqGradesLong)
export(qcWqGradesPlot)
export(writeBMI)
importFrom(magrittr,"%<>%")
importFrom(magrittr,"%>%")
importFrom(stats,median)
665 changes: 385 additions & 280 deletions R/bmi-qc.R

Large diffs are not rendered by default.

96 changes: 45 additions & 51 deletions R/calculations.R
Original file line number Diff line number Diff line change
Expand Up @@ -25,99 +25,93 @@ MaxDQF <- function(flags) {

#' Calculate median values for each water quality parameter for each lake visit.
#'
#' @param conn Database connection generated from call to \code{OpenDatabaseConnection()}. Ignored if \code{data.source} is \code{"local"}.
#' @param path.to.data The directory containing the csv data exports generated from \code{SaveDataToCsv()}. Ignored if \code{data.source} is \code{"database"}.
#' @param park Optional. Four-letter park code to filter on, e.g. "GRBA".
#' @param site Optional. Site code to filter on, e.g. "GRBA_L_DEAD0".
#' @param field.season Optional. Field season name to filter on, e.g. "2019".
#' @param data.source Character string indicating whether to access data in the live database (\code{"database"}, default) or to use data saved locally (\code{"local"}). In order to access the most up-to-date data, it is recommended that you select \code{"database"} unless you are working offline or your code will be shared with someone who doesn't have access to the database.
#'
#' @return A tibble with columns for park, field season, site code, visit date, and the median values, flags, and counts for temperature, specific conductance, pH, and dissolved oxygen.
#' @return A tibble
#' @export
#'
LakeWqMedian <- function(conn, path.to.data, park, site, field.season, data.source = "database") {
temp <- ReadAndFilterData(conn = conn, path.to.data = path.to.data, park = park, site = site, field.season = field.season, data.source = data.source, data.name = "WaterQualityTemperature")
spcond <- ReadAndFilterData(conn = conn, path.to.data = path.to.data, park = park, site = site, field.season = field.season, data.source = data.source, data.name = "WaterQualitySpCond")
ph <- ReadAndFilterData(conn = conn, path.to.data = path.to.data, park = park, site = site, field.season = field.season, data.source = data.source, data.name = "WaterQualitypH")
do <- ReadAndFilterData(conn = conn, path.to.data = path.to.data, park = park, site = site, field.season = field.season, data.source = data.source, data.name = "WaterQualityDO")
LakeWqMedian <- function(park, site, field.season) {
temp <- ReadAndFilterData(park = park, site = site, field.season = field.season, data.name = "WaterQualityTemperature")
spcond <- ReadAndFilterData(park = park, site = site, field.season = field.season, data.name = "WaterQualitySpCond")
ph <- ReadAndFilterData(park = park, site = site, field.season = field.season, data.name = "WaterQualitypH")
do <- ReadAndFilterData(park = park, site = site, field.season = field.season, data.name = "WaterQualityDO")

wq.visits <- ReadAndFilterData(conn = conn, path.to.data = path.to.data, park = park, site = site, field.season = field.season, data.source = data.source, data.name = "Visit")
wq.visits <- ReadAndFilterData(park = park, site = site, field.season = field.season, data.name = "Visit")

temp.med <- temp %>%
dplyr::left_join(dplyr::select(wq.visits, SampleFrame, Park, FieldSeason, SiteCode, VisitDate), by = c("Park", "FieldSeason", "SiteCode", "VisitDate")) %>%
dplyr::filter(MonitoringStatus == "Sampled") %>%
dplyr::group_by(Park, FieldSeason, SiteCode, VisitDate, VisitType, SampleFrame, MeasurementDepth_m, Flag, FlagNote, DPL) %>%
temp.med <- temp |>
dplyr::left_join(dplyr::select(wq.visits, SampleFrame, Park, FieldSeason, SiteCode, VisitDate), by = c("Park", "FieldSeason", "SiteCode", "VisitDate")) |>
dplyr::filter(MonitoringStatus == "Sampled") |>
dplyr::group_by(Park, FieldSeason, SiteCode, VisitDate, VisitType, SampleFrame, MeasurementDepth_m, Flag, FlagNote, DPL) |>
dplyr::summarise(TemperatureMedian_C = median(WaterTemperature_C, na.rm = TRUE),
TemperatureCount = sum(!is.na(WaterTemperature_C))) %>%
dplyr::rename(TemperatureFlag = Flag) %>%
TemperatureCount = sum(!is.na(WaterTemperature_C))) |>
dplyr::rename(TemperatureFlag = Flag) |>
dplyr::arrange(SiteCode, VisitDate)

spcond.med <- spcond %>%
dplyr::left_join(dplyr::select(wq.visits, SampleFrame, Park, FieldSeason, SiteCode, VisitDate), by = c("Park", "FieldSeason", "SiteCode", "VisitDate")) %>%
dplyr::filter(MonitoringStatus == "Sampled") %>%
dplyr::group_by(Park, FieldSeason, SiteCode, VisitDate, VisitType, SampleFrame, MeasurementDepth_m, Flag, FlagNote, DPL) %>%
spcond.med <- spcond |>
dplyr::left_join(dplyr::select(wq.visits, SampleFrame, Park, FieldSeason, SiteCode, VisitDate), by = c("Park", "FieldSeason", "SiteCode", "VisitDate")) |>
dplyr::filter(MonitoringStatus == "Sampled") |>
dplyr::group_by(Park, FieldSeason, SiteCode, VisitDate, VisitType, SampleFrame, MeasurementDepth_m, Flag, FlagNote, DPL) |>
dplyr::summarise(SpCondMedian_microS_per_cm = median(SpecificConductance_microS_per_cm, na.rm = TRUE),
SpCondCount = sum(!is.na(SpecificConductance_microS_per_cm))) %>%
dplyr::rename(SpCondFlag = Flag) %>%
SpCondCount = sum(!is.na(SpecificConductance_microS_per_cm))) |>
dplyr::rename(SpCondFlag = Flag) |>
dplyr::arrange(SiteCode, VisitDate)

ph.med <- ph %>%
dplyr::left_join(dplyr::select(wq.visits, SampleFrame, Park, FieldSeason, SiteCode, VisitDate), by = c("Park", "FieldSeason", "SiteCode", "VisitDate")) %>%
dplyr::filter(MonitoringStatus == "Sampled") %>%
dplyr::group_by(Park, FieldSeason, SiteCode, VisitDate, VisitType, SampleFrame, MeasurementDepth_m, Flag, FlagNote, DPL) %>%
ph.med <- ph |>
dplyr::left_join(dplyr::select(wq.visits, SampleFrame, Park, FieldSeason, SiteCode, VisitDate), by = c("Park", "FieldSeason", "SiteCode", "VisitDate")) |>
dplyr::filter(MonitoringStatus == "Sampled") |>
dplyr::group_by(Park, FieldSeason, SiteCode, VisitDate, VisitType, SampleFrame, MeasurementDepth_m, Flag, FlagNote, DPL) |>
dplyr::summarise(pHMedian = median(pH, na.rm = TRUE),
pHCount = sum(!is.na(pH))) %>%
dplyr::rename(pHFlag = Flag) %>%
pHCount = sum(!is.na(pH))) |>
dplyr::rename(pHFlag = Flag) |>
dplyr::arrange(SiteCode, VisitDate)

do.med <- do %>%
dplyr::left_join(dplyr::select(wq.visits, SampleFrame, Park, FieldSeason, SiteCode, VisitDate), by = c("Park", "FieldSeason", "SiteCode", "VisitDate")) %>%
dplyr::filter(MonitoringStatus == "Sampled") %>%
dplyr::group_by(Park, FieldSeason, SiteCode, VisitDate, VisitType, SampleFrame, MeasurementDepth_m, Flag, FlagNote, DPL) %>%
do.med <- do |>
dplyr::left_join(dplyr::select(wq.visits, SampleFrame, Park, FieldSeason, SiteCode, VisitDate), by = c("Park", "FieldSeason", "SiteCode", "VisitDate")) |>
dplyr::filter(MonitoringStatus == "Sampled") |>
dplyr::group_by(Park, FieldSeason, SiteCode, VisitDate, VisitType, SampleFrame, MeasurementDepth_m, Flag, FlagNote, DPL) |>
dplyr::summarise(DOMedian_percent = median(DissolvedOxygen_percent), DOMedian_mg_per_L = median(DissolvedOxygen_mg_per_L),
DOPercentCount = sum(!is.na(DissolvedOxygen_percent)),
DOmgLCount = sum(!is.na(DissolvedOxygen_mg_per_L))) %>%
dplyr::rename(DOFlag = Flag) %>%
DOmgLCount = sum(!is.na(DissolvedOxygen_mg_per_L))) |>
dplyr::rename(DOFlag = Flag) |>
dplyr::arrange(SiteCode, VisitDate)

wq.med <- temp.med %>%
dplyr::full_join(spcond.med, by = c("Park", "FieldSeason", "SiteCode", "VisitDate", "VisitType", "SampleFrame", "MeasurementDepth_m", "FlagNote", "DPL")) %>%
dplyr::full_join(ph.med, by = c("Park", "FieldSeason", "SiteCode", "VisitDate", "VisitType", "SampleFrame", "MeasurementDepth_m", "FlagNote", "DPL")) %>%
dplyr::full_join(do.med, by = c("Park", "FieldSeason", "SiteCode", "VisitDate", "VisitType", "SampleFrame", "MeasurementDepth_m", "FlagNote", "DPL")) %>%
wq.med <- temp.med |>
dplyr::full_join(spcond.med, by = c("Park", "FieldSeason", "SiteCode", "VisitDate", "VisitType", "SampleFrame", "MeasurementDepth_m", "FlagNote", "DPL")) |>
dplyr::full_join(ph.med, by = c("Park", "FieldSeason", "SiteCode", "VisitDate", "VisitType", "SampleFrame", "MeasurementDepth_m", "FlagNote", "DPL")) |>
dplyr::full_join(do.med, by = c("Park", "FieldSeason", "SiteCode", "VisitDate", "VisitType", "SampleFrame", "MeasurementDepth_m", "FlagNote", "DPL")) |>
dplyr::ungroup()

return(wq.med)
}

#' Calculate median values for each water quality parameter for each stream visit.
#'
#' @param conn Database connection generated from call to \code{OpenDatabaseConnection()}. Ignored if \code{data.source} is \code{"local"}.
#' @param path.to.data The directory containing the csv data exports generated from \code{SaveDataToCsv()}. Ignored if \code{data.source} is \code{"database"}.
#' @param park Optional. Four-letter park code to filter on, e.g. "GRBA".
#' @param site Optional. Site code to filter on, e.g. "GRBA_L_DEAD0".
#' @param field.season Optional. Field season name to filter on, e.g. "2019".
#' @param data.source Character string indicating whether to access data in the live database (\code{"database"}, default) or to use data saved locally (\code{"local"}). In order to access the most up-to-date data, it is recommended that you select \code{"database"} unless you are working offline or your code will be shared with someone who doesn't have access to the database.
#'
#' @return A tibble with columns for park, field season, site code, visit date, and the median values, flags, and counts for temperature, specific conductance, pH, and dissolved oxygen.
#' @return A tibble
#' @export
#'
StreamWqMedian <- function(conn, path.to.data, park, site, field.season, data.source = "database") {
stream_wq <- ReadAndFilterData(conn = conn, path.to.data = path.to.data, park = park, site = site, field.season = field.season, data.source = data.source, data.name = "WQStreamXSection")
wq.visits <- ReadAndFilterData(conn = conn, path.to.data = path.to.data, park = park, site = site, field.season = field.season, data.source = data.source, data.name = "Visit")
StreamWqMedian <- function(park, site, field.season) {
stream_wq <- ReadAndFilterData(park = park, site = site, field.season = field.season, data.name = "WQStreamXSection")
wq.visits <- ReadAndFilterData(park = park, site = site, field.season = field.season, data.name = "Visit")

stream_wq_med <- stream_wq %>%
dplyr::left_join(dplyr::select(wq.visits, SampleFrame, Park, FieldSeason, SiteCode, VisitDate, MonitoringStatus), by = c("Park", "FieldSeason", "SiteCode", "VisitDate")) %>%
dplyr::filter(MonitoringStatus == "Sampled") %>%
dplyr::group_by(Park, FieldSeason, SiteCode, VisitDate, VisitType, SampleFrame, pHFlag, DOFlag, SpCondFlag, TemperatureFlag, FlagNote, DPL) %>%
stream_wq_med <- stream_wq |>
dplyr::left_join(dplyr::select(wq.visits, SampleFrame, Park, FieldSeason, SiteCode, VisitDate, MonitoringStatus), by = c("Park", "FieldSeason", "SiteCode", "VisitDate")) |>
dplyr::filter(MonitoringStatus == "Sampled") |>
dplyr::group_by(Park, FieldSeason, SiteCode, VisitDate, VisitType, SampleFrame, pHFlag, DOFlag, SpCondFlag, TemperatureFlag, FlagNote, DPL) |>
dplyr::summarise(TemperatureMedian_C = median(WaterTemperature_C),
TemperatureCount = sum(!is.na(WaterTemperature_C)),
pHMedian = median(pH),
pHCount = sum(!is.na(pH)),
DOMedian_mg_per_L = median(DissolvedOxygen_mg_per_L),
DOmgLCount = sum(!is.na(DissolvedOxygen_mg_per_L)),
SpCondMedian_microS_per_cm = median(SpecificConductance_microS_per_cm),
SpCondCount = sum(!is.na(SpecificConductance_microS_per_cm))) %>%
dplyr::arrange(SiteCode, VisitDate) %>%
SpCondCount = sum(!is.na(SpecificConductance_microS_per_cm))) |>
dplyr::arrange(SiteCode, VisitDate) |>
dplyr::ungroup()


Expand Down
Loading