Add DAG to remove old entries from metadata database

[lucaszanotelli]

This PR adds a maintenance DAG to prune the content of Airflow’s metadata database.

This DAG removes old entries from job, dag_run, task_instance, log, xcom, sla_miss, dags, task_reschedule, task_fail and import_error tables by default. In the DAG, review the list of tables and decide whether old entries must be removed from them. In general, most space savings are provided by cleaning log, task_instance, dag_run and xcom tables. To exclude a table from cleanup, modify the DAG and comment corresponding items in the DATABASE_OBJECTS list.

The default retention period is 30 days. To modify that, create and set the max_db_entry_age_in_days variable with an integer value that corresponds to the amount of days to retain the logs.

The DAG has already ran successfully in the test-hubble environment: https://he02f27a661269b05p-tp.appspot.com/dags/cleanup_metadata_dag/grid

This workflow was created mainly following the guide: https://cloud.google.com/composer/docs/cleanup-airflow-database

[sydneynotthecity]

This looks great! Thanks for putting together to keep our Airflow environment healthy.

When I looked at the DAG definition in airflow, I noticed the cleanup_sessions task instance does not have an upstream dependency on print_configuration. I assume this is ok? The dependency tree just doesn't match the rest of the tasks.

Do you think it's also worthwhile to enable Sentry -> Slack alerting by adding the alert_after_max_retries parm to the PythonOperators?

Neither of these suggestions block the merge.

[lucaszanotelli]

This looks great! Thanks for putting together to keep our Airflow environment healthy.

When I looked at the DAG definition in airflow, I noticed the cleanup_sessions task instance does not have an upstream dependency on print_configuration. I assume this is ok? The dependency tree just doesn't match the rest of the tasks.

Do you think it's also worthwhile to enable Sentry -> Slack alerting by adding the alert_after_max_retries parm to the PythonOperators?

Neither of these suggestions block the merge.

Regarding the cleanup_sessions dependency, it's not an issue since this task doesn't need any configuration loaded from the print_configuration task.

About the alert_after_max_retries, nice catch! I'd add it right way.