Upstream HTTP request error on gunicorn worker auto-restart

[lihopentium]

We use gunicorn on fprocess command ENV fprocess="gunicorn -c gconfig.py index:app"
The of-watchdog appear the Upstream HTTP request error while the gunicon worker restart by the max_requests setting.

I use ab to stress test on of-watchdog and gunicorn endpoint, and only happened on the POST method.
The of-watchdog endpoint had some failed requests while the gunicorn worker restart.
But direct sending requests to gunicorn endpoint seems no problem.

Expected Behaviour

Current Behaviour

Possible Solution

Steps to Reproduce (for bugs)

code:

gunicorn config

bind = "0.0.0.0:5000"
workers = 4
worker_class = "gthread"
threads = 1
timeout = 300
graceful_timeout = 30
loglevel = "debug"
max_requests = 10

• gunicorn config
• code diff with template

Steps

1. run the docker container

docker run -it -p 8080:8080 -p 5000:5000 lihodocker/faas-python3-gunicorn:20200710182115

2. create the body file

echo "{}" > /tmp/body.txt

3. stress test on gunicorn endpoint, increase the concurrent requests or total requests as you want

ab -c 20 -n 100 -p /tmp/body.txt http://localhost:8080/

you may see some failed requests due to the gunicorn worker auto-restart.

failed log

2020/07/10 10:36:03 Upstream HTTP request error: Post http://127.0.0.1:5000/: read tcp 127.0.0.1:57600->127.0.0.1:5000: read: connection reset by peer

4. stress test on gunicorn endpoint, increase the concurrent requests or total requests as you want

ab -c 20 -n 100 -p /tmp/body.txt http://localhost:5000/

There are no failed requests on gunicorn endpoint.

5. Remove the max_requests = 10 setting from gconfig.py, run step 3 again.
   No restarts on gunicorn workers, requests are all success.

Context

of-watchdog have the retry or fix the connection close as well as other HTTP clients do.

Your Environment

• Docker version docker version (e.g. Docker 17.0.05 ):
  19.03.8

• Are you using Docker Swarm or Kubernetes (FaaS-netes)?
  No

• Operating System and version (e.g. Linux, Windows, MacOS):
  MacOS

• Link to your project or a code example to reproduce issue:

[lihopentium]

I have a solution proposal, could we make the keep-alive to be an optional argument.
Let the user decide to turn on or turn off for their use cases.
@alexellis Does it make sense to you?