create_tool_calling_agent with .with_structured_output() causes error

[rmalde]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from langchain_core.pydantic_v1 import BaseModel, Field
from langchain_openai import ChatOpenAI
from langchain.tools import tool
from langchain.agents import AgentExecutor, create_tool_calling_agent
from langchain_core.prompts import ChatPromptTemplate

OPENAI_API_KEY="XXX"

class Output(BaseModel):
    new_price: float = Field("New price to offer", ge=0.0)
    strategy: str = Field("The negotiation strategy that was used",
                          min_length=1)
    output_message: str = Field("Output message to send to the buyer", min_length=1)

@tool
def get_new_price(old_price: float) -> float:
    """
    Returns a new negotiated price based on the old price.
    """
    return old_price * 0.9

llm = ChatOpenAI(model="gpt-3.5-turbo-0125", temperature=0, api_key=OPENAI_API_KEY)

tools = [get_new_price]
structured_llm = llm.with_structured_output(Output)

prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a negotiation agent negotiating with a buyer. Given the tools that you have, negotiate with the buyer."),
    ("human", "{input}"),
    ("placeholder", "{agent_scratchpad}")
])

agent = create_tool_calling_agent(structured_llm, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)
out = agent_executor.invoke({"input": "I want to buy the item for $50"})

print(out)

Description

Hi! I'm trying to use Langchain to create a negotiation agent that can call tools, and then output a final json of the negotiation decision.

I'm using create_tool_calling_agent, and I want the output to be structured like the Output object, and so I bind the llm using with_structured_output.

I get the following error:
ValueError: This function requires a .bind_tools method be implemented on the LLM.

The code works if I just use .with_structured_output but not the tool_calling_agent, and it also works if I just use tool_calling_agent but not .with_structured_output. I can't get both to work.

Please let me know how I can make this possible, or if there's another way to make this usage work!

In this thread #20343 (reply in thread), @baskaryan seems to say you shouldn't need to call both tools and structured output, but I don't know how to otherwise get the json output I want and also let the agent call tools.

System Info

System Information

OS: Linux
OS Version: #1 SMP Thu Jan 11 04:09:03 UTC 2024
Python Version: 3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0]

Package Information

langchain_core: 0.1.52
langchain: 0.1.20
langchain_community: 0.0.38
langsmith: 0.1.40
langchain_openai: 0.1.1
langchain_text_splitters: 0.0.1

[dosubot[bot]]

The issue you're facing stems from the create_tool_calling_agent function requiring the language model (LLM) to have a .bind_tools method. When you wrap your ChatOpenAI instance with .with_structured_output, the resulting object likely doesn't expose a .bind_tools method, leading to the error you encountered.

To resolve this, you'll need to ensure that the LLM you pass to create_tool_calling_agent retains a .bind_tools method after any modifications, such as applying .with_structured_output. This might involve extending the ChatOpenAI class to include a custom implementation that supports both structured output and tool binding. Here's a simplified approach:

from langchain_openai import ChatOpenAI

class CustomChatOpenAI(ChatOpenAI):
    def bind_tools(self, tools):
        # Custom logic to bind tools
        self.tools = tools
        # Ensure other necessary setups are also handled

Use this custom class instead of the direct ChatOpenAI instance when setting up your agent. This ensures that when create_tool_calling_agent checks for the .bind_tools method, it is present, thus avoiding the error.

Remember, the key is to maintain the .bind_tools method's availability in your LLM object throughout any modifications you apply, such as .with_structured_output. This approach should help you achieve the structured output you desire while also allowing the agent to call tools.

Details

Sources

• libs/langchain/langchain/agents/tool_calling_agent/base.py
• libs/partners/groq/langchain_groq/chat_models.py
• libs/partners/mistralai/langchain_mistralai/chat_models.py
• libs/partners/anthropic/langchain_anthropic/chat_models.py
• libs/partners/openai/langchain_openai/chat_models/base.py

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[pedrojrv]

Same issue here