Skip to content

PaddleOCR-VL runtime error: image_features is None in FastDeploy 2.4.0 #6179

Open

Description

@byeongjaeSon

Bug Description

Hello, I am trying to serve PaddleOCR-VL using the latest version of FastDeploy. However, I encounter the following error during runtime. I would like to understand what might be causing this issue and how it can be resolved.

Environment

fastdeploy-gpu 2.4.0
paddlepaddle-gpu 3.3.0

Error Message

Traceback (most recent call last):
  File "/path/to/env/lib/python3.12/site-packages/fastdeploy/worker/worker_process.py", line 1089, in <module>
    run_worker_proc()
  File "/path/to/env/lib/python3.12/site-packages/fastdeploy/worker/worker_process.py", line 1085, in run_worker_proc
    worker_proc.event_loop_normal()
  File "/path/to/env/lib/python3.12/site-packages/fastdeploy/worker/worker_process.py", line 522, in event_loop_normal
    self.worker.execute_model(req_dicts, max_occupied_batch_index)
  File "/path/to/env/lib/python3.12/site-packages/fastdeploy/worker/gpu_worker.py", line 193, in execute_model
    output = self.model_runner.execute_model(model_forward_batch, num_running_request)
  File "/path/to/env/lib/python3.12/site-packages/fastdeploy/worker/gpu_model_runner.py", line 2238, in execute_model
    model_output = self.model(
  File "/path/to/env/lib/python3.12/site-packages/paddle/nn/layer/layers.py", line 1833, in __call__
    return self.forward(*inputs, **kwargs)
  File "/path/to/env/lib/python3.12/site-packages/fastdeploy/model_executor/models/paddleocr_vl/paddleocr_vl.py", line 240, in forward
    input_embeddings = self.get_input_embeddings(
  File "/path/to/env/lib/python3.12/site-packages/fastdeploy/model_executor/models/paddleocr_vl/paddleocr_vl.py", line 231, in get_input_embeddings
    input_embeddings[image_mask] = image_features.cast(self._dtype)
AttributeError: 'NoneType' object has no attribute 'cast'


Activity

  1. Wanglongzhi2001 commented on Jan 23, 2026

    @Wanglongzhi2001
    Collaborator

    Could you please provide a minimal code example for reproduction?

  2. byeongjaeSon commented on Jan 23, 2026

    @byeongjaeSon
    Author

    @Wanglongzhi2001
    Hi,
    When running PaddleOCR-VL with FastDeploy under very high concurrent requests (PDF parsing workload), the serving process sometimes crashes due to a runtime error.

    In most cases it works well, but the process occasionally dies unexpectedly, requiring a manual restart.

    Is there any guidance, recommended configuration, or known limitations when running FastDeploy in production or high-concurrency environments?

    Reproduction

    1. Launch FastDeploy server
    python -m fastdeploy.entrypoints.openai.api_server \
      --model PaddlePaddle/PaddleOCR-VL \
      --workers 4 \
    	--port 8000 \
    	--metrics-port 3000 \
    	--host 0.0.0.0 \
      --gpu-memory-utilization 0.45 \
      --max-model-len 16384 \
      --max-num-batched-tokens 131072 \
      --max-num-seqs 256 \
      --enable-mm \
      --max-concurrency 8192 \
      --graph-optimization-config '{"graph_opt_level":0, "use_cudagraph":true}'
    
    1. Send requests using PaddleOCR-VL pipeline (sample)
    from paddleocr import PaddleOCRVL
    pipeline = PaddleOCRVL()
    output = pipeline.predict("https://paddle-model-ecology.bj.bcebos.com/paddlex/imgs/demo_image/paddleocr_vl_demo.png")
    for res in output:
        res.print()
    
  3. Wanglongzhi2001 commented on Jan 26, 2026

    @Wanglongzhi2001
    Collaborator

    Hello,
    This issue appears to be caused by excessive GPU memory usage.

    1. You can increase the gpu-memory-utilization to 0.9 to allocate more KV cache.
    2. You can also reduce the max-num-batched-tokens to 4096 to use a smaller number of chunks in the chunked prefill.
    3. We also recommend setting the graph optimization configuration to: --graph-optimization-config '{"use_cudagraph":true, "use_unique_memory_pool":true, "cudagraph_capture_sizes":[1, 4, 16, 64, 128, 256]}. This will reduce the number of graphs in cudagraph and enable use_unique_memory_pool to reduce cudagraph's memory usage.
  4. byeongjaeSon commented on Jan 26, 2026

    @byeongjaeSon
    Author

    @Wanglongzhi2001

    Hi, thank you for response.

    I tried the suggested options, but I encountered another error at the same line of code.

    When running inference with paddleocr_vl using FastDeploy, the worker process crashes due to a shape mismatch during input embedding construction.
    The error occurs in get_input_embeddings() when assigning image_features to input_embeddings[image_mask].

    How can I handle this issue?

    Reproduction

     python -m fastdeploy.entrypoints.openai.api_server \
      --model PaddlePaddle/PaddleOCR-VL \
      --workers 4 \
    	--port 8002 \
    	--metrics-port 3002 \
    	--host 0.0.0.0 \
      --gpu-memory-utilization 0.9 \
    	--max-model-len 16384 \
      --max-num-batched-tokens 4096 \
      --max-num-seqs 256 \
      --enable-mm \
      --max-concurrency 8192 \
      --graph-optimization-config '{"use_cudagraph":true, "use_unique_memory_pool":true, "cudagraph_capture_sizes":[1, 4, 16, 64, 128, 256]}'
    
    

    Error log

    Traceback (most recent call last):
      File ".../site-packages/fastdeploy/worker/worker_process.py", line 1089, in <module>
        run_worker_proc()
      File ".../site-packages/fastdeploy/worker/worker_process.py", line 1085, in run_worker_proc
        worker_proc.event_loop_normal()
      File ".../site-packages/fastdeploy/worker/worker_process.py", line 522, in event_loop_normal
        self.worker.execute_model(req_dicts, max_occupied_batch_index)
      File ".../site-packages/fastdeploy/worker/gpu_worker.py", line 193, in execute_model
        output = self.model_runner.execute_model(model_forward_batch, num_running_request)
      File ".../site-packages/fastdeploy/worker/gpu_model_runner.py", line 2238, in execute_model
        model_output = self.model(
      File ".../site-packages/paddle/nn/layer/layers.py", line 1833, in __call__
        return self.forward(*inputs, **kwargs)
      File ".../fastdeploy/model_executor/models/paddleocr_vl/paddleocr_vl.py", line 240, in forward
        input_embeddings = self.get_input_embeddings(
      File ".../fastdeploy/model_executor/models/paddleocr_vl/paddleocr_vl.py", line 231, in get_input_embeddings
        input_embeddings[image_mask] = image_features.cast(self._dtype)
    ValueError: (InvalidArgument) shape mismatch: value tensor of shape [944, 1024]
    cannot be broadcast to indexing result of shape [945, 1024].
    
    

    Environment

    fastdeploy-gpu : 2.4.0
    paddlepaddle-gpu : 3.3.0
    GPU / CUDA : A100 80G, CUDA Version: 12.2, Driver Version: 535.230.02
    Python : 3.12

  5. ming1753 commented on Jan 26, 2026

    @ming1753
    Collaborator

    We recommend using Docker to install FastDeploy according to install document, and then start the service following parctice document.
    We have routine testing and monitoring for this usage scenario.

  6. Wanglongzhi2001 commented on Jan 30, 2026

    @Wanglongzhi2001
    Collaborator

    We recommend using Docker to install FastDeploy according to install document, and then start the service following parctice document. We have routine testing and monitoring for this usage scenario.

    @byeongjaeSon Hi, has this resolved your issue?

  7. byeongjaeSon commented on Jan 31, 2026

    @byeongjaeSon
    Author

    @Wanglongzhi2001 @ming1753

    Unfortunately, the same issue occurs even when applying the suggested options or using the official Docker image.

    Additionally, during long-running operation, another issue was observed where the following error occurs and incoming requests remain stuck in the waiting queue without transitioning to the running state.

    ERROR:legacy.api_server:request[chatcmpl-50a6b82b-1777-4eec-93aa-977bd62a6698]full generator error: Engine is not healthy: Worker Service Not Healthy, Traceback (most recent call last):
      File "/usr/lib/python3.10/asyncio/queues.py", line 159, in get
        await getter
    asyncio.exceptions.CancelledError
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/lib/python3.10/asyncio/tasks.py", line 456, in wait_for
        return fut.result()
    asyncio.exceptions.CancelledError
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "/usr/local/lib/python3.10/dist-packages/fastdeploy/entrypoints/openai/serving_chat.py", line 556, in chat_completion_full_generator
        response = await asyncio.wait_for(response_queue.get(), timeout=10)
      File "/usr/lib/python3.10/asyncio/tasks.py", line 458, in wait_for
        raise exceptions.TimeoutError() from exc
    asyncio.exceptions.TimeoutError
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/local/lib/python3.10/dist-packages/fastdeploy/entrypoints/openai/serving_chat.py", line 172, in create_chat_completion
        return await self.chat_completion_full_generator(
      File "/usr/local/lib/python3.10/dist-packages/fastdeploy/entrypoints/openai/serving_chat.py", line 563, in chat_completion_full_generator
        raise ValueError(f"Engine is not healthy: {msg}")
    ValueError: Engine is not healthy: Worker Service Not Healthy
    
    
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions