Support prometheus multiprocess collector

[victorsamun]

Current prometheus integration does not support custom metrics reporting.
Let's we have following code in sample.py:

import celery
import prometheus_client


app = celery.Celery(
    'app',
    broker='redis://localhost:6379/0',
    result_backend='redis://localhost:6379/0',
)


gauge = prometheus_client.Gauge('my_gauge', '')


@app.task
def my_task(x):
    gauge.set(int(x))
    return x

Then we run celery:

(venv) $ celery -A sample worker

And flower:

(venv) $ FLOWER_UNAUTHENTICATED_API=1 celery -A sample flower --port=5000

After send task my_task gauge doesn't change:

$ curl -sH 'Content-Type: application/json' -o /dev/null -d '{"args": [3]}' 'http://localhost:5000/api/task/apply/sample.my_task'
$ curl -s http://localhost:5000/metrics | grep my_gauge
# HELP my_gauge 
# TYPE my_gauge gauge
my_gauge 0.0

If we set PROMETHEUS_MULTIPROC_DIR=/tmp/pmp nothing changes:

$ curl -sH 'Content-Type: application/json' -o /dev/null -d '{"args": [3]}' 'http://localhost:5000/api/task/apply/sample.my_task'
$ curl -s http://localhost:5000/metrics | grep my_gauge
# HELP my_gauge 
# TYPE my_gauge gauge
my_gauge{pid="1256424"} 0.0
my_gauge{pid="1256439"} 0.0

After applying fix from this PR gauge changed (we must run celery and flower with PROMETHEUS_MULTIPROC_DIR environment variable):

$ curl -sH 'Content-Type: application/json' -o /dev/null -d '{"args": [3]}' 'http://localhost:5000/api/task/apply/sample.my_task'
$ curl -s http://localhost:5000/metrics | grep my_gauge
# HELP my_gauge 
# TYPE my_gauge gauge
my_gauge{pid="1256426"} 3.0
my_gauge{pid="1256424"} 0.0
my_gauge{pid="1256439"} 0.0

[victorsamun]

@mher @auvipy PTAL

[mher]

This will break the offline worker purge