Skip to content
This repository was archived by the owner on Nov 20, 2026. It is now read-only.

Improve the performance of multimodal large models. Input parameters pass URL links to large models. Require structure to run code will report an error #7005

Description

@leididi

What happened?

Describe the bug
Improve the performance of multimodal large models. Input parameters pass URL links to large models. Require structure to run code will report an error

Content usage is required
[{"image_url":"url_adress"}]

import asyncio
import mimetypes

from autogen_agentchat.agents import AssistantAgent
from autogen_ext.models.openai import OpenAIChatCompletionClient
from autogen_agentchat.messages import MultiModalMessage, ModelClientStreamingChunkEvent
from autogen_agentchat.base import TaskResult


model_client = OpenAIChatCompletionClient(
    model="qwen-vl-plus",
    api_key="api-key",
    base_url= "https://dashscope.aliyuncs.com/compatible-mode/v1",
    model_info={
        "vision": True,
        "function_calling": True,
        "json_output": True,
        "structured_output": True,  # 添加 structured_output 字段
        "family": "unknown",
        "multiple_system_messages": True
    }
)
system_message = """
你是专业的图片分析专家,能够分析各种类型的图片并提取测试相关信息。

请分析图片内容并返回JSON格式的结果。
"""

async def main():
    agent = AssistantAgent(
        name="assistant",
        model_client=model_client,
        system_message=system_message,
        model_client_stream=True)

    image_path = 'https://img.lycheer.net/xupload/398/68bfe4c56efxx04c62d.png'

    multimodal_message = MultiModalMessage(
        content=["图片里面有什么文字信息。", image_path],
        source="user"
    )

    stream = agent.run_stream(task=multimodal_message)
    async for event in stream:  # type: ignore
        if isinstance(event, ModelClientStreamingChunkEvent):
            print(event.content, end="", flush=True)
            continue
        if isinstance(event, TaskResult):
            print("\n--- 完整结果 ---")
            if event.messages and hasattr(event.messages[-1], "content"):
                print(event.messages[-1].content)
            else:
                print(event)
if __name__ == "__main__":
    asyncio.run(main())

Please help me. What should I do?

Which packages was the bug in?

Python Extensions (autogen-ext)

AutoGen library version.

Python 0.6.4

Other library version.

No response

Model used

No response

Model provider

None

Other model provider

No response

Python version

None

.NET version

None

Operating system

None

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions