You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
PaddleOCR-VL runtime error: image_features is None in FastDeploy 2.4.0 #6179
Hello, I am trying to serve PaddleOCR-VL using the latest version of FastDeploy. However, I encounter the following error during runtime. I would like to understand what might be causing this issue and how it can be resolved.
Environment
fastdeploy-gpu 2.4.0
paddlepaddle-gpu 3.3.0
Error Message
Traceback (most recent call last):
File "/path/to/env/lib/python3.12/site-packages/fastdeploy/worker/worker_process.py", line 1089, in <module>
run_worker_proc()
File "/path/to/env/lib/python3.12/site-packages/fastdeploy/worker/worker_process.py", line 1085, in run_worker_proc
worker_proc.event_loop_normal()
File "/path/to/env/lib/python3.12/site-packages/fastdeploy/worker/worker_process.py", line 522, in event_loop_normal
self.worker.execute_model(req_dicts, max_occupied_batch_index)
File "/path/to/env/lib/python3.12/site-packages/fastdeploy/worker/gpu_worker.py", line 193, in execute_model
output = self.model_runner.execute_model(model_forward_batch, num_running_request)
File "/path/to/env/lib/python3.12/site-packages/fastdeploy/worker/gpu_model_runner.py", line 2238, in execute_model
model_output = self.model(
File "/path/to/env/lib/python3.12/site-packages/paddle/nn/layer/layers.py", line 1833, in __call__
return self.forward(*inputs, **kwargs)
File "/path/to/env/lib/python3.12/site-packages/fastdeploy/model_executor/models/paddleocr_vl/paddleocr_vl.py", line 240, in forward
input_embeddings = self.get_input_embeddings(
File "/path/to/env/lib/python3.12/site-packages/fastdeploy/model_executor/models/paddleocr_vl/paddleocr_vl.py", line 231, in get_input_embeddings
input_embeddings[image_mask] = image_features.cast(self._dtype)
AttributeError: 'NoneType' object has no attribute 'cast'
@Wanglongzhi2001
Hi,
When running PaddleOCR-VL with FastDeploy under very high concurrent requests (PDF parsing workload), the serving process sometimes crashes due to a runtime error.
In most cases it works well, but the process occasionally dies unexpectedly, requiring a manual restart.
Is there any guidance, recommended configuration, or known limitations when running FastDeploy in production or high-concurrency environments?
Send requests using PaddleOCR-VL pipeline (sample)
from paddleocr import PaddleOCRVL
pipeline = PaddleOCRVL()
output = pipeline.predict("https://paddle-model-ecology.bj.bcebos.com/paddlex/imgs/demo_image/paddleocr_vl_demo.png")
for res in output:
res.print()
Hello,
This issue appears to be caused by excessive GPU memory usage.
You can increase the gpu-memory-utilization to 0.9 to allocate more KV cache.
You can also reduce the max-num-batched-tokens to 4096 to use a smaller number of chunks in the chunked prefill.
We also recommend setting the graph optimization configuration to: --graph-optimization-config '{"use_cudagraph":true, "use_unique_memory_pool":true, "cudagraph_capture_sizes":[1, 4, 16, 64, 128, 256]}. This will reduce the number of graphs in cudagraph and enable use_unique_memory_pool to reduce cudagraph's memory usage.
I tried the suggested options, but I encountered another error at the same line of code.
When running inference with paddleocr_vl using FastDeploy, the worker process crashes due to a shape mismatch during input embedding construction.
The error occurs in get_input_embeddings() when assigning image_features to input_embeddings[image_mask].
Traceback (most recent call last):
File ".../site-packages/fastdeploy/worker/worker_process.py", line 1089, in <module>
run_worker_proc()
File ".../site-packages/fastdeploy/worker/worker_process.py", line 1085, in run_worker_proc
worker_proc.event_loop_normal()
File ".../site-packages/fastdeploy/worker/worker_process.py", line 522, in event_loop_normal
self.worker.execute_model(req_dicts, max_occupied_batch_index)
File ".../site-packages/fastdeploy/worker/gpu_worker.py", line 193, in execute_model
output = self.model_runner.execute_model(model_forward_batch, num_running_request)
File ".../site-packages/fastdeploy/worker/gpu_model_runner.py", line 2238, in execute_model
model_output = self.model(
File ".../site-packages/paddle/nn/layer/layers.py", line 1833, in __call__
return self.forward(*inputs, **kwargs)
File ".../fastdeploy/model_executor/models/paddleocr_vl/paddleocr_vl.py", line 240, in forward
input_embeddings = self.get_input_embeddings(
File ".../fastdeploy/model_executor/models/paddleocr_vl/paddleocr_vl.py", line 231, in get_input_embeddings
input_embeddings[image_mask] = image_features.cast(self._dtype)
ValueError: (InvalidArgument) shape mismatch: value tensor of shape [944, 1024]
cannot be broadcast to indexing result of shape [945, 1024].
We recommend using Docker to install FastDeploy according to install document, and then start the service following parctice document.
We have routine testing and monitoring for this usage scenario.
We recommend using Docker to install FastDeploy according to install document, and then start the service following parctice document. We have routine testing and monitoring for this usage scenario.
Unfortunately, the same issue occurs even when applying the suggested options or using the official Docker image.
Additionally, during long-running operation, another issue was observed where the following error occurs and incoming requests remain stuck in the waiting queue without transitioning to the running state.
ERROR:legacy.api_server:request[chatcmpl-50a6b82b-1777-4eec-93aa-977bd62a6698]full generator error: Engine is not healthy: Worker Service Not Healthy, Traceback (most recent call last):
File "/usr/lib/python3.10/asyncio/queues.py", line 159, in get
await getter
asyncio.exceptions.CancelledError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.10/asyncio/tasks.py", line 456, in wait_for
return fut.result()
asyncio.exceptions.CancelledError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/fastdeploy/entrypoints/openai/serving_chat.py", line 556, in chat_completion_full_generator
response = await asyncio.wait_for(response_queue.get(), timeout=10)
File "/usr/lib/python3.10/asyncio/tasks.py", line 458, in wait_for
raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/fastdeploy/entrypoints/openai/serving_chat.py", line 172, in create_chat_completion
return await self.chat_completion_full_generator(
File "/usr/local/lib/python3.10/dist-packages/fastdeploy/entrypoints/openai/serving_chat.py", line 563, in chat_completion_full_generator
raise ValueError(f"Engine is not healthy: {msg}")
ValueError: Engine is not healthy: Worker Service Not Healthy
Activity
Wanglongzhi2001 commentedon Jan 23, 2026
Could you please provide a minimal code example for reproduction?
byeongjaeSon commentedon Jan 23, 2026
@Wanglongzhi2001
Hi,
When running PaddleOCR-VL with FastDeploy under very high concurrent requests (PDF parsing workload), the serving process sometimes crashes due to a runtime error.
In most cases it works well, but the process occasionally dies unexpectedly, requiring a manual restart.
Is there any guidance, recommended configuration, or known limitations when running FastDeploy in production or high-concurrency environments?
Reproduction
Wanglongzhi2001 commentedon Jan 26, 2026
Hello,
This issue appears to be caused by excessive GPU memory usage.
gpu-memory-utilizationto 0.9 to allocate more KV cache.max-num-batched-tokensto 4096 to use a smaller number of chunks in the chunked prefill.--graph-optimization-config '{"use_cudagraph":true, "use_unique_memory_pool":true, "cudagraph_capture_sizes":[1, 4, 16, 64, 128, 256]}. This will reduce the number of graphs in cudagraph and enable use_unique_memory_pool to reduce cudagraph's memory usage.byeongjaeSon commentedon Jan 26, 2026
@Wanglongzhi2001
Hi, thank you for response.
I tried the suggested options, but I encountered another error at the same line of code.
When running inference with paddleocr_vl using FastDeploy, the worker process crashes due to a shape mismatch during input embedding construction.
The error occurs in get_input_embeddings() when assigning image_features to input_embeddings[image_mask].
How can I handle this issue?
Reproduction
Error log
Environment
fastdeploy-gpu : 2.4.0
paddlepaddle-gpu : 3.3.0
GPU / CUDA : A100 80G, CUDA Version: 12.2, Driver Version: 535.230.02
Python : 3.12
ming1753 commentedon Jan 26, 2026
We recommend using Docker to install FastDeploy according to install document, and then start the service following parctice document.
We have routine testing and monitoring for this usage scenario.
Wanglongzhi2001 commentedon Jan 30, 2026
@byeongjaeSon Hi, has this resolved your issue?
byeongjaeSon commentedon Jan 31, 2026
@Wanglongzhi2001 @ming1753
Unfortunately, the same issue occurs even when applying the suggested options or using the official Docker image.
Additionally, during long-running operation, another issue was observed where the following error occurs and incoming requests remain stuck in the waiting queue without transitioning to the running state.