Redesign system statistics

[leo-schick]

Mara collects system statistic information (e.g. CPU, Disk, Memory) when a pipeline is executed.

This is a collection ticket of things which should be redesigned in this logic.

• add option to disable the collection of system statistic information. This should disable the cards in the web UI as well.
• redesign system statistics, remove it from the standard EventHandler
• add extras_require statistics with package psutil, remove psutil from install_requires (default package requirements)

[jankatins]

Alternatives to complete removal: PG has pg_stats_statements and similar views which one could use to get/derive statistics from which serve a similar usecase.

This sounds like a nice recipe for doing this: https://www.timescale.com/blog/point-in-time/

[leo-schick]

@jankatins I think a generic interface for statistical data provider would make sence here e.g. by adding a global MARA_STATISTIC_PROVIDER to the __init__.py file of a module.

Example:
I build a mara_singer module which synchronizes singer taps to postgres. In mara you just have to add a single task SingerTapToDB, but the tap sends Metric Messages which could be made visible in the UI. Currently they are just detected but interpreted:

These metric data could be saved and made visible in the UI somewhere.

I do not have much of an idea how such a generic "statistic provider" interface might look like so I think I will more focus on disabling the current statistic implementation. Maybe then with a deprecation mark that the current statistic collection feature will be removed at the next major version.

[jankatins]

Would be interesting to have a basic (run_id, ts, metric_name) -> float-value interface which some provider could add to and then let the UI just dynamically create graphs from it (would be interesting: e.g. start from singer ones but then have pg ones. So maybe something like

METRIC_NAME = str
def MARA_STATISTIC_PROVIDER() -> DICT[METRIC_NAME, float]
    ...

Given that mara also works with multiple DBs, you can't really expect that the metrics names are predefined, so the UI would need to dynamically react to new metrics showing up...