Processing Provenance

[AdamTheisen]

I have been thinking about how we could trace the processing of a data product using ACT in such a manner that we could recreate it easily in the future, essentially the processing provenance. This would be important for any data products that are created using ACT. It would be a fair undertaking but one option is to add a "act_history" attribute (or whatever we would want to call it) such that whenever a function is called, it appends the function and parameter values to that history. As a short example:

-ACT Version: 1.3.5.post0.dev0+g368a9269b
-act.retrievals.aeri.aeri2irt(wnum_name=wnum, rad_name=mean_rad, hatch_name=hatchOpen, tolerance=0.1, temp_low=150.0, temp_high=320.0, maxiter=200)

But essentially, we could use this to trace the whole history of how ACT was used for each product. Again, this would be a rather large undertaking and put some added burden on future contributions so I wanted to just throw it out there for discussion.

[zssherman]

I think this would definitely be useful. We add a sorta cmac history to cmac generated files, which might be similar.

[AdamTheisen]

@mgrover1 @kenkehoe @scollis curious to get your thoughts here as well.

[kenkehoe]

Would be a nice addition but I'm curious how exact we will want this to be. Every sub-function will need to add to this global attribute and we would need to include keywords and more to have the true full history. That will end up being a lot of information. Also, the version of other libraries could affect the calculations. We should decide on a level of documentation that is good enough, or else we will end up like ARM storing a stupid amount of stuff.

[AdamTheisen]

It would be a lot to add in that's for sure. I agree that we don't want to overdo it but at the same time, if we want someone to be able to fully recreate a data product that was produced using ACT, we need all that information technically. I was not even thinking about the library versions until you mentioned it though but more to think about...

There are other ways around this as well which could be having users submit an example if they created a data product that they wanted to stamp as the official version. We could do this through ACT or on ARM-Notebooks. Another option would be to give the option to write all this information out to a configuration file instead of housing in an attribute.

In the end, we do want to get away from sharing data and towards sharing code so maybe there is not a need for this long-term....

[AdamTheisen]

I've been testing some things out on this and there could be a relatively easy solution by using the python logging library. If we could apply it in the init.py file. For example the below code will save the function call and keywords as

sum() called with args 10, b=20
sum() called with args 30, b=20

I am just unsure if we could easily do this in the init.py file and how that could be saved to the object... It is an interesting option though that could save a lot of hardcoding this in like I was originally thinking.

import functools
import logging
import io
import tempfile

logfile = tempfile.mkstemp(suffix='.log', dir='.')[1]
logging.basicConfig(level=logging.INFO, format='%(message)s', filename=logfile)
logger = logging.getLogger()

### Setup the console handler with a StringIO object
log_capture_string = io.StringIO()
ch = logging.StreamHandler(log_capture_string)
ch.setLevel(logging.INFO)
### Add the console handler to the logger
logger.addHandler(ch)

def log(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        args_repr = [repr(a) for a in args]
        kwargs_repr = [f"{k}={v!r}" for k, v in kwargs.items()]
        signature = ", ".join(args_repr + kwargs_repr)
        logger.info(f"{func.__name__}() called with args {signature}")
        try:
            result = func(*args, **kwargs)
            return result
        except Exception as e:
            logger.exception(f"Exception raised in {func.__name__}. exception: {str(e)}")
            raise e
    return wrapper

@log
def sum(a, b=10):
    return a+b

if __name__ == "__main__":
    a = sum(10, b=20)
    a = sum(a, b=20)

    print('test')
    ### Pull the contents back into a string and close the stream
    log_contents = log_capture_string.getvalue()
    log_capture_string.close()

    ### Output as lower case to prove it worked. 
    print(log_contents.lower())

[mgrover1]

I think this gets tricky to deal with at scale/generically... for example if a user passes in some calculation (ex. calculate_temperature)

temp = act.calc.(temperature, dewpoint)

This would store those variable names... the trick is what does this mean? It is dependent on what variables people use in their scripts. Also, we would run into issues when storing arrays in that form, such as multidimensional array that becomes very large...

Dask has some nice functions for building the task group, which could help, but this is very tricky problem.

There are some docs here https://carpentries-lab.github.io/python-aos-lesson/09-provenance/index.html from carpentries that might help, so maybe just showing people how to do this would help?

I think this might be something we could contribute upstream as well, as prototyped here xarray-contrib/cf-xarray#259 , since ACT uses an xarray dataset as the core data object and this is what we would be adding the provenance variables to.