Version 1.1 Major changes to Error Logging #12

dwest77a · 2024-03-08T15:56:19Z

New Software Features:

Per Project Code (PPC) Error tracking
Individual log files for each dataset, updated automatically with filehandler updates built into the pipeline - for upcoming job allocation improvements.
Scanning improvements and identification of types of dimensions.
Support for virtual dimension additions (file_number)
BypassSwitch option and default changes.

New Documentation:

Documentation Updates
Example CCI Water Vapour files and tutorial
Kerchunk Powerpoints

…ptions

Dev update

… utilises dimension checks

…ing within scanning phase

…s once created, minor variable changes

Changes from CCI branch to main

…e scripts

dwest77a and others added 30 commits February 20, 2024 11:46

Updated inspection method

51613b2

Aded backtrack and xkshape warning

cec7630

Added backtrack

ab5772d

Added XKShape tolerance and get_concat_dims for multiple validation o…

934f093

…ptions

Updated version

df6fddb

Updated setup and added cci env variables

d6edc55

Changed variable difference message between files

36cad7e

Merge branch 'dev' into dev_update

1d7f8ac

Merge pull request #10 from cedadev/dev_update

91f2ccb

Dev update

Integration of blacklist into progress tracker

b8de738

Added dryrun carrythrough

c1a6ae6

Overhauled dimension handling and added extra functions, scanning now…

5b18988

… utilises dimension checks

Added functionality for dimension highlighting and concatenation test…

1688b01

…ing within scanning phase

Added multiple nan-type checks

66ef1d4

Added concat fatal error for l2 data

22a74d7

Integration with other pipeline functions

aa96b57

Added blacklist/virtual support, reading log and status files

1abd97c

Minor changes experimenting with streamline/dependency - abandoned

b964318

Removed error and output log options per job

d8790b2

Set up a dedicated place for config files

6334ec4

Organised multi-purpose functions to a single script within the pipeline

f5ea0df

Initialised allocator script

84139a8

Moved scripts

7fc1387

Added requirements that were misplaced

c4bbc23

Added new features; loading and creating refs as required, saving ref…

512b26a

…s once created, minor variable changes

Added file handling and status logging

1d66435

Updated import locations

2f3b154

Removed scanfile skip option

89e7f37

Added file handlers and status logging, also pass/fail counters

ab9890d

Added test/example notebooks

6355ed8

dwest77a and others added 19 commits March 6, 2024 16:02

Updated with comments and rearranged sections

32f7c3f

Merge pull request #11 from cedadev/cci_cases

7b71b88

Changes from CCI branch to main

Removed old examples

834e867

Removed unnecessary or outdated files

b004939

Updated for move to cedaproc

2a9669b

Added template for cedaproc use

905291e

Removed testing notebooks and updated showcase tools

775a640

Updated assessor documentation

5aefe8e

updated gitignore

42335f2

Added example input file for tutorial and startup:

2721234

Added introductory powerpoints

8d4fc67

Updated docs with example

e4e7dac

Added template setup file

ff9d068

Fixed syntax issue

43181e1

Renamed file

8e9f3ec

Removed deploy function for dependent jobs

85ea009

Updated utils, added filehandler continuation support for all pipelin…

363d953

…e scripts

Added job log system support and traceback within error logs

c03aeb3

Merge branch 'main' into dev

affa3e4

dwest77a merged commit 80232c3 into main Mar 8, 2024
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Version 1.1 Major changes to Error Logging #12

Version 1.1 Major changes to Error Logging #12

dwest77a commented Mar 8, 2024

Version 1.1 Major changes to Error Logging #12

Version 1.1 Major changes to Error Logging #12

Conversation

dwest77a commented Mar 8, 2024