Store workflow execution trace in user's storage account #173
Labels
enhancement
New feature or request
future planning
To be prioritized in future
needs discussion
Team discussion is needed: for example: discuss technical directions, alternatives etc.
TES Priority: P2
Groomed to a Priority 2 issue
Troubleshooting
Enable users to identify and debug errors
Problem:
We collect information about TesTask and store it as json in PostgreSQL database. This information might be useful for debugging, workflow performance tuning, etc, but it is not accessible for user when it's stored in database.
Solution:
Store TesTask information in user's storage account.
At this moment for each workflow run we generate 3 files in container 'outputs': metadata.json, outputs.data, timing.html. We might create one more file trace.json that will contain all TesTasks for that workflow run.
In Scheduler.cs (OrchestrateTesTasksOnBatch) when tesTask changes state to terminal state (completed, failed) we append tesTask to trace.json as JSONL.
Additional context
This work overlaps with the work required for having TES as a self-contained executable. Requires discussion of implementation details to fit into future work on self-contained executable.
The text was updated successfully, but these errors were encountered: