suggest adding support for the local large model Qwen.

[lixy0420]

Hopefully it can be included in this project.

[reatang]

In fact, you can initialize the client directly for the local large model

[lixy0420]

In fact, you can initialize the client directly for the local large model

The locally deployed Qwen model with vllm will throw an error if used directly, as it does not support bind_tools, and there are still some details that need to be modified.

[JoshuaC215]

Running a local model with vllm is outside the scope of what I can support with this repo since it isn’t a workflow I do myself very much.

One suggestion would be to try running it on FastChat instead of vllm directly? I believe it’s a bit simpler and maybe easier to integrate.

I would be open to add some simple instructions on how to run that and very minimal modifications to the repo code to support the use case.

Let me know what you think.

[lixy0420]

Thank you for your quick response! I know how to use vllm for deployment. The issue I am encountering is an error when using research assistant agent . By calling the Qwen local model to build the agent, which likely requires modifications in the tools section. I hope it can support calling the lcoal Qwen large model to implement the function call feature.

[JoshuaC215]

Did you try running vllm with the —enable-auto-tool-choice and —tool-call-parser flags mentioned in the error message you posted?

The tool calling is pretty fundamental to the research assistant and a fairly commodity thing at this point, so I don’t think I’ll be making any changes to make it work with an LLM API that doesn’t support it. I would guess the other built in agents or any custom agent you build without tool calling on the repo would work fine.