Implemented Glue ETL spreadsheet Google LIMS processing #19
+317
−0
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Story: Let Glue the Google LIMS! Following the warehouse framework methodology that is being built, let glue the Google LIMS sheet as the second spreadsheet importing use case. The target is OrcaVault database staging data area tsa schema table. Since it is for data warehouse purpose, the ETL approach retain all factual information; without reshaping much or dropping any but column renaming and harmonisation. Light-weight data clean up tasks. * Think of it as the Google LIMS spreadsheet now become a table in database; as-is all columns and values are being retained. * Change history tracking and records archival will be further implemented by downstream warehouse layers in psa and vault schema. * Technical steps are now mainly inherited by the framework implemented in PRs #13 and #14. Hence, this Glue data import job becomes pretty straight forward task and, cookiecutter template code with only need to focus on transformation.
victorskl
added
documentation
victorskl
added a commit
that referenced
this pull request
Jan 12, 2025
* Story: Let Glue the Google LIMS! (continue) Now that we have `tsa.spreadsheet_library_tracking_metadata` in staging data area by Glue ETL job in #19, we can source this data table with dbt to further feed into the downstream warehouse layers in psa and vault schema. * Technical steps are now mainly inherited by the framework implemented in PR #15. Hence, this step becomes pretty straight forward task and template code.
victorskl
added a commit
that referenced
this pull request
Jan 12, 2025
* Story: Let Glue the Google LIMS! (continue) Now that we have `tsa.spreadsheet_google_lims` in staging data area by Glue ETL job in #19, we can source this data table with dbt to further feed into the downstream warehouse layers in psa and vault schema. * Technical steps are now mainly inherited by the framework implemented in PR #15. Hence, this step becomes pretty straight forward task and template code.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Following the warehouse framework methodology that is being built, let
glue the Google LIMS sheet as the second spreadsheet importing use case.
The target is OrcaVault database staging data area tsa schema table.
Since it is for data warehouse purpose, the ETL approach retain all
factual information; without reshaping much or dropping any but column
renaming and harmonisation. Light-weight data clean up tasks.
as-is all columns and values are being retained.
by downstream warehouse layers in psa and vault schema.
Implemented Glue ETL spreadsheet processing #13 and Implemented Glue ETL job script deployment using terraform #14. Hence, this Glue data import job becomes pretty straight forward
task and, cookiecutter template code with only need to focus on transformation.