Getting functional server error

[llmwesee]

Description

Hi, I'm encountering an issue while attempting to use a functional server to upload files through an API. Specifically, when I run the server with the following command, I receive an error related to connection establishment. However, if I run the server without enabling the functional server, it works as expected.

Please help me to solve the issue.

Thank You

Command

 python generate.py --base_model=meta-llama/Meta-Llama-3.1-8B-Instruct  --score_model=None --langchain_mode='UserData' --user_path=user_path --use_auth_token=True --visible_visible_models=False --max_seq_len=8192 --max_max_new_tokens=4096 --max_new_tokens=4096 --min_new_tokens=256 --api_open=True --allow_api=True --max_quality=True  --function_server=True --function_server_workers=5 --multiple_workers_gunicorn=True --function_server_port=5002 --function_api_key=API_KEY

Issue Details

Traceback (most recent call last):
  File "/home/versha/miniconda3/envs/narad_050824_pmdemomobile_lake/lib/python3.10/site-packages/urllib3/connection.py", line 196, in _new_conn
    sock = connection.create_connection(
  File "/home/versha/miniconda3/envs/narad_050824_pmdemomobile_lake/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
    raise err
  File "/home/versha/miniconda3/envs/narad_050824_pmdemomobile_lake/lib/python3.10/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/versha/miniconda3/envs/narad_050824_pmdemomobile_lake/lib/python3.10/site-packages/urllib3/connectionpool.py", line 789, in urlopen
    response = self._make_request(
  File "/home/versha/miniconda3/envs/narad_050824_pmdemomobile_lake/lib/python3.10/site-packages/urllib3/connectionpool.py", line 495, in _make_request
    conn.request(
  File "/home/versha/miniconda3/envs/narad_050824_pmdemomobile_lake/lib/python3.10/site-packages/urllib3/connection.py", line 398, in request
    self.endheaders()
  File "/home/versha/miniconda3/envs/narad_050824_pmdemomobile_lake/lib/python3.10/http/client.py", line 1278, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/home/versha/miniconda3/envs/narad_050824_pmdemomobile_lake/lib/python3.10/http/client.py", line 1038, in _send_output
    self.send(msg)
  File "/home/versha/miniconda3/envs/narad_050824_pmdemomobile_lake/lib/python3.10/http/client.py", line 976, in send
    self.connect()
  File "/home/versha/miniconda3/envs/narad_050824_pmdemomobile_lake/lib/python3.10/site-packages/urllib3/connection.py", line 236, in connect
    self.sock = self._new_conn()
  File "/home/versha/miniconda3/envs/narad_050824_pmdemomobile_lake/lib/python3.10/site-packages/urllib3/connection.py", line 211, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7de388ea7010>: Failed to establish a new connection: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/versha/miniconda3/envs/narad_050824_pmdemomobile_lake/lib/python3.10/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
  File "/home/versha/miniconda3/envs/narad_050824_pmdemomobile_lake/lib/python3.10/site-packages/urllib3/connectionpool.py", line 843, in urlopen
    retries = retries.increment(
  File "/home/versha/miniconda3/envs/narad_050824_pmdemomobile_lake/lib/python3.10/site-packages/urllib3/util/retry.py", line 519, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='0.0.0.0', port=5002): Max retries exceeded with url: /execute_function/ (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7de388ea7010>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/versha/Documents/narad050824_pmdemo_mobile_lake_apitest/weseecore/src/gpt_langchain.py", line 9383, in update_user_db
    return _update_user_db(file, db1s=db1s,
  File "/home/versha/Documents/narad050824_pmdemo_mobile_lake_apitest/weseecore/src/gpt_langchain.py", line 9664, in _update_user_db
    sources = call_function_server('0.0.0.0', function_server_port, 'path_to_docs', (file,), simple_kwargs,
  File "/home/versha/Documents/narad050824_pmdemo_mobile_lake_apitest/weseecore/src/function_client.py", line 50, in call_function_server
    execute_result = execute_function_on_server(host, port, function_name, args, kwargs, use_disk, use_pickle,
  File "/home/versha/Documents/narad050824_pmdemo_mobile_lake_apitest/weseecore/src/function_client.py", line 21, in execute_function_on_server
    response = requests.post(url, json=payload, headers=headers)
  File "/home/versha/miniconda3/envs/narad_050824_pmdemomobile_lake/lib/python3.10/site-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
  File "/home/versha/miniconda3/envs/narad_050824_pmdemomobile_lake/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/versha/miniconda3/envs/narad_050824_pmdemomobile_lake/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/versha/miniconda3/envs/narad_050824_pmdemomobile_lake/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/home/versha/miniconda3/envs/narad_050824_pmdemomobile_lake/lib/python3.10/site-packages/requests/adapters.py", line 700, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='0.0.0.0', port=5002): Max retries exceeded with url: /execute_function/ (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7de388ea7010>: Failed to establish a new connection: [Errno 111] Connection refused'))

[pseudotensor]

That means the client side can't reach the function server. You can check the start-up shows the 5002 port started, and you can look at the openai_logs/* function server related files to see if present

i.e. they may look like:

(h2ogpt) jon@pseudotensor:~/h2ogpt$ ls -alrt openai_logs/
total 224
-rw-------  1 jon jon     0 Oct 20 10:00 gunicorn_OpenAI_d9ffa2da-638d-4d78-a66b-2ddf28748f4a_stdout.log
-rw-------  1 jon jon     0 Oct 20 10:00 gunicorn_Function_c8db57d7-aa31-494f-b4a6-01cc810663ed_stdout.log
-rw-------  1 jon jon  1154 Oct 20 10:00 gunicorn_OpenAI_d9ffa2da-638d-4d78-a66b-2ddf28748f4a_stderr.log
-rw-------  1 jon jon  1154 Oct 20 10:00 gunicorn_Function_c8db57d7-aa31-494f-b4a6-01cc810663ed_stderr.log
-rw-------  1 jon jon   299 Oct 20 15:48 gunicorn_OpenAI_110a8b60-9f67-4045-ad93-4fa75b21ff44_stdout.log
-rw-------  1 jon jon 11486 Oct 20 15:48 gunicorn_Agent_f900440d-2f0f-4b81-a2b0-0b44dfd6231c_stdout.log
-rw-------  1 jon jon 21009 Oct 20 15:56 gunicorn_OpenAI_110a8b60-9f67-4045-ad93-4fa75b21ff44_stderr.log
-rw-------  1 jon jon 20392 Oct 20 15:56 gunicorn_Agent_f900440d-2f0f-4b81-a2b0-0b44dfd6231c_stderr.log
-rw-------  1 jon jon   167 Oct 20 15:58 gunicorn_OpenAI_2a853fe3-bc2b-4069-a3a0-7ab4f3346568_stdout.log
-rw-------  1 jon jon 11486 Oct 20 15:58 gunicorn_Agent_a3da0ca9-6eef-4f01-b5d6-0cff2d36c92c_stdout.log
-rw-------  1 jon jon 19283 Oct 20 16:00 gunicorn_OpenAI_2a853fe3-bc2b-4069-a3a0-7ab4f3346568_stderr.log
-rw-------  1 jon jon 20073 Oct 20 16:00 gunicorn_Agent_a3da0ca9-6eef-4f01-b5d6-0cff2d36c92c_stderr.log
-rw-------  1 jon jon     0 Oct 20 16:01 gunicorn_OpenAI_96b3c872-136c-45f8-a0f2-c9a228887872_stdout.log
-rw-------  1 jon jon     0 Oct 20 16:01 gunicorn_Agent_5e81accd-e122-4890-902f-dddccbcc6edb_stdout.log
drwx------  2 jon jon  4096 Oct 20 16:01 ./
-rw-------  1 jon jon 15287 Oct 20 16:02 gunicorn_OpenAI_96b3c872-136c-45f8-a0f2-c9a228887872_stderr.log
-rw-------  1 jon jon 15923 Oct 20 16:02 gunicorn_Agent_5e81accd-e122-4890-902f-dddccbcc6edb_stderr.log
drwxrwxr-x 83 jon jon 61440 Oct 23 12:51 ../
(h2ogpt) jon@pseudotensor:~/h2ogpt$ 

over multiple runs (new file each run).

e.g. this would be a bad startup:

(h2ogpt) jon@pseudotensor:~/h2ogpt$ cat openai_logs/gunicorn_Function_c8db57d7-aa31-494f-b4a6-01cc810663ed_stderr.log
[2024-10-20 10:00:37 -0700] [907130] [INFO] Starting gunicorn 23.0.0
[2024-10-20 10:00:37 -0700] [907130] [ERROR] Connection in use: ('0.0.0.0', 5002)
[2024-10-20 10:00:37 -0700] [907130] [ERROR] connection to ('0.0.0.0', 5002) failed: [Errno 98] Address already in use
[2024-10-20 10:00:38 -0700] [907130] [ERROR] Connection in use: ('0.0.0.0', 5002)
[2024-10-20 10:00:38 -0700] [907130] [ERROR] connection to ('0.0.0.0', 5002) failed: [Errno 98] Address already in use
[2024-10-20 10:00:39 -0700] [907130] [ERROR] Connection in use: ('0.0.0.0', 5002)
[2024-10-20 10:00:39 -0700] [907130] [ERROR] connection to ('0.0.0.0', 5002) failed: [Errno 98] Address already in use
[2024-10-20 10:00:40 -0700] [907130] [ERROR] Connection in use: ('0.0.0.0', 5002)
[2024-10-20 10:00:40 -0700] [907130] [ERROR] connection to ('0.0.0.0', 5002) failed: [Errno 98] Address already in use
[2024-10-20 10:00:41 -0700] [907130] [ERROR] Connection in use: ('0.0.0.0', 5002)
[2024-10-20 10:00:41 -0700] [907130] [ERROR] connection to ('0.0.0.0', 5002) failed: [Errno 98] Address already in use
[2024-10-20 10:00:42 -0700] [907130] [ERROR] Can't connect to ('0.0.0.0', 5002)
(h2ogpt) jon@pseudotensor:~/h2ogpt$ 

While a good output would be:

(h2ogpte) jon@pseudotensor:~/h2ogpte/docker_data/h2ogpt/docker_logs/h2ogpt_openai$ tail -10 gunicorn_Function_296f27f4-e7ae-4284-85e2-5fd7a741b25c_stdout.log
git_hash: cf74d576ecfca0e24cac27588b52a4701dd7cb1d
visible_models: ['meta-llama/Meta-Llama-3.1-8B-Instruct']
visible_vision_models: ['mistralai/Pixtral-12B-2409']
Command: /usr/bin/gunicorn -w 5 -k uvicorn.workers.UvicornWorker --timeout 60 -b 0.0.0.0:5002 openai_server.function_server:app
Hash: cf74d576ecfca0e24cac27588b52a4701dd7cb1d
Prep: persist_directory=db_dir_UserData exists, using
Prep: persist_directory=db_dir_UserData exists, using
Prep: persist_directory=db_dir_UserData exists, using
Prep: persist_directory=db_dir_UserData exists, using
Prep: persist_directory=db_dir_UserData exists, using
(h2ogpte) jon@pseudotensor:~/h2ogpte/docker_data/h2ogpt/docker_logs/h2ogpt_openai$ tail -10 gunicorn_Function_296f27f4-e7ae-4284-85e2-5fd7a741b25c_stderr.log 
[2024-10-23 22:37:58 +0000] [198] [INFO] Application startup complete.
[2024-10-23 22:37:58 +0000] [200] [INFO] Started server process [200]
[2024-10-23 22:37:58 +0000] [200] [INFO] Waiting for application startup.
[2024-10-23 22:37:58 +0000] [200] [INFO] Application startup complete.
[2024-10-23 22:37:58 +0000] [201] [INFO] Started server process [201]
[2024-10-23 22:37:58 +0000] [201] [INFO] Waiting for application startup.
[2024-10-23 22:37:58 +0000] [201] [INFO] Application startup complete.
[2024-10-23 22:37:58 +0000] [197] [INFO] Started server process [197]
[2024-10-23 22:37:58 +0000] [197] [INFO] Waiting for application startup.
[2024-10-23 22:37:58 +0000] [197] [INFO] Application startup complete.
(h2ogpte) jon@pseudotensor:~/h2ogpte/docker_data/h2ogpt/docker_logs/h2ogpt_openai$