How to get structured output from a conversational agent?

[th4ruka]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Description

I'm currently working on how to best handle structured output in a conversational chat-bot (conversational agent).
This chat-bot works as a text-to-image generator, it get details of the image that user wants to generate. Then it calls a tool to generate the image according to the developed prompt, which returns the URL of the generated image.

• My problem is that the agent returns the URL embedded inside a text message as follows,

"assistant": "Here is the image of a black BMW car riding on a country road. \n
[Click here to view the image]
(https://storage.googleapis.com/text-to-image-bot-657bb.appspot.com/20240507_041456.jpg)\n\n
Is there anything specific you would like to add or modify in the image?"

• But what I want the agent to return is a structured output as follows,

{
  "assistant": "Here is the image of a black BMW car riding on a country road.\n Is there anything specific you would like to add or modify in the image?",
  "url": "https://storage.googleapis.com/text-to-image-bot-657bb.appspot.com/20240507_041456.jpg"
}

• url should be null during a normal conversation. It should have a value only after an image is generated.

I would really appreciate any support or insights anyone have on this!

Example Code

I have implemented the following code to get structured output. But it does not always follow the structured output declared in Response BaseModel class.

llm = ChatOpenAI(model="gpt-3.5-turbo-1106", temperature=0)
prompt = ChatPromptTemplate.from_messages(
    [
        ("system", SYSTEM_MESSAGE),
        MessagesPlaceholder(variable_name="chat_history"),
        ("human", "{input}"),
        MessagesPlaceholder(variable_name="agent_scratchpad")
    ]
)


class Response(BaseModel):
    """Final response to the input message"""
    assistant: str = Field(description="Your Response to the message asked")
    url: str = Field(description="URL returned by `generate_image` tool")


def parse(output):
    # If no function was invoked, return to user
    if "function_call" not in output.additional_kwargs:
        return AgentFinish(return_values={"assistant": output.content}, log=output.content)

    # Parse out the function call
    function_call = output.additional_kwargs["function_call"]
    name = function_call["name"]
    inputs = json.loads(function_call["arguments"])

    # If the Response function was invoked, return to the user with the function inputs
    if name == "Response":
        return AgentFinish(return_values=inputs, log=str(function_call))
    # Otherwise, return an agent action
    else:
        return AgentActionMessageLog(
            tool=name, tool_input=inputs, log="", message_log=[output]
        )


if __name__ == "__main__":
    chat_history = ChatMessageHistory()
    llm_with_tools = llm.bind_functions([generate_image, Response]) # generate_image is a tool
    agent = (
            {
                "input": lambda x: x["input"],
                "chat_history": lambda x: x["chat_history"],
                # Format agent scratchpad from intermediate steps
                "agent_scratchpad": lambda x: format_to_openai_function_messages(
                    x["intermediate_steps"]
                ),
            }
            | prompt
            | llm_with_tools
            | parse
    )
    agent_executor = AgentExecutor(tools=[generate_image], agent=agent, verbose=True)

    while True:
        user_msg = input("[user]>>> ")
        if user_msg == "/exit":
            break
        # chat_history.add_user_message(user_msg)
        result = agent_executor.invoke(
            {"input": [HumanMessage(user_msg)], "chat_history": chat_history.messages},
            return_only_outputs=True,
        )
        chat_history.add_user_message(user_msg)

        # URL is returned by generate_image tool
        # following try-except block handles adding url to chat_history.
        try:
            chat_history.add_ai_message(result["assistant"]
                                        + "| url: " + result["url"])
        except KeyError as key_err:
            chat_history.add_ai_message(result["assistant"])
        except Exception as e:
            print("Error adding ai message: ", e)

        print(result)
        # print("[assistant]>>> ", result["assistant"])

[klurpicolo]

@th4ruka I run into the similar issue when try to follow the document. Did you manage to make it work?

[th4ruka]

I haven't correctly figured it out yet. But this might help: Returning structured outputs from agents