Handle TA upload error #998
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
these 2 new tasks are meant to replace the test results processor and test results finisher tasks the difference between the tasks is that the new tasks: - use the new parse_raw_upload function provided by the test results parser library - instead of writing to the database in the processor, the finisher takes care of writing to the database - the processor writes the results of its parsing and the finisher pulls from that
the upload task will check the new ta tasks feature flag to determine whether to use the newly introduced ta processor and ta finisher tasks we also call the new ta processor and ta finisher tasks via a chord since we removed the concurrent db writes from the processors
also update test results parser version
we don't need to chunk them anymore and each upload can get its own processing task
i want to introduce the new finished state in the test results upload pipeline to do so safely i'm adding the new v2_processed and v2_finished states the reason for this is that the meaning of processed and v2_processed are different processed means that test instances are in the db and persisted but in the new pipeline v2_finished has that meaning and v2_processed just means that the intermediate results are in redis for now
this commit changes the TA upload states to have the v2_persisted state which represents the test run data being persisted to the db and the v2_finished state representing that the upload was taken into account when making the latest commment on the PR this also removes the v2_failed state since i want an upload to both be able to have valid test runs and have some failed parsing, the v2_failed state made it seem like it was either processed or failed when it could be both the failures related to an upload will instead be represented by upload errors
this is an abstraction to help us use bigquery for test analytics data this abstraction allows us to write data to bigquery using the Storage Write API and query it using SQL statements as strings, it also handles creating the connection to bigquery for us the use protocol buffers is a consequence of using the Storage Write API feat: create TADriver abstraction this commit creates the TADriver abstraction which (for now) exposes a write_testruns method, which will take care of persisting a list of testruns to some specific storage this commit implements this interface for Postgres and for Bigquery the idea is that we will start by persisting testruns to both, then eventually we add a method to the interface for retrieving the aggregates at which point it will be safe for our cloud offering to switch over to only bigquery for storage of TA testruns.
remove msgpack add pytest-insta update test-results-parser
update type hints usage from test_results_parser don't update upload state in pg.py
gist of it is we're keeping the new upload states but we're reverting the msgpack changes we're also using the ta_storage utilities for persisting data in the processor now and we're using pytest-insta for snapshots in the tests
we need to update the test results tasks' use of the test results parser library
|
This PR includes changes to |
Codecov ReportAttention: Patch coverage is ✅ All tests successful. No failed tests found.
📢 Thoughts on this report? Let us know! |
|
✅ All tests successful. No failed tests were found. 📣 Thoughts on this report? Let Codecov know! | Powered by Codecov |
Codecov ReportAttention: Patch coverage is
✅ All tests successful. No failed tests found.
Additional details and impacted files@@ Coverage Diff @@
## main #998 +/- ##
==========================================
- Coverage 97.74% 97.72% -0.03%
==========================================
Files 454 452 -2
Lines 36368 36768 +400
==========================================
+ Hits 35548 35930 +382
- Misses 820 838 +18
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.