withStructuredOutput does not accept tools

[TiagoGouvea]

When using withStructuredOutput, the tools bind to the llm are ignored.

If we add a tools to the structured schema as a workaround it almost never works as expected.

On the invoke method, if it's a withStructuredOutput it will declare just one tool, to force the model to call a parser method to the response.

  const llm = new ChatOpenAI({ modelName: 'gpt-4o-mini' });
  llm.bindTools(tools}; // these tools will be ignored
  const llmResult = await llm
    .withStructuredOutput(outputSchema, { name: 'Response' })
    .invoke(messages);

Could be a way to use withStructuredOutput with custom tools as well.

[jacoblee93]

Thanks for the issue!

What would be the expected behavior here?

[TiagoGouvea]

Using withStructuredOutput, it never add the tools. It add only one tool extract (that enforces the structured output) other than the tools bind before.

  withStructuredOutput(outputSchema, config) {
   ....
               llm = this.bind({
                    tools: [
                        {
                            type: "function",
                            function: {
                                name: functionName,
                                description: asJsonSchema.description,
                                parameters: asJsonSchema,
                            },
                        },
                    ],
                    tool_choice: {
                        type: "function",
                        function: {
                            name: functionName,
                        },
                    },
                    // Do not pass `strict` argument to OpenAI if `config.strict` is undefined
                    ...(config?.strict !== undefined ? { strict: config.strict } : {}),
                });
   ...
  }

The expected behavior is allow the llm to call the bind tools.

[jacoblee93]

allow the llm to call the bind tools.

So it would either return a tool message or structured output?

[TiagoGouvea]

My ideia is to have the structured output, but, if I need to call some tool receive the tool_calls as well.

[TiagoGouvea]

Let me explain better. I'm implementing a multi agent experiment, using the Command and the goto structure like showed on lanchain youtube video.

I loved the Command idea with the json object as return, made the life easier for me. Thank you for that!

On my project each agent will have access to some tools.

On my tests, when I use withStructuredOutput with my desired schema to return, it doesn't include the tools.

I work arround calling OpenAi library directly, with response_format parameter and it worked like expected, but on my project each agent could use a different model, so I would need implement the call for each provider and so.

What do you think?

[jasontjahjono]

allow the llm to call the bind tools.

So it would either return a tool message or structured output?

Hi Jacob, I really need this functionality! LLM should return a tool call (AI Message calling a tool) or Structured Output. Do you know if this is in progress, or planned to be implemented?

[jacoblee93]

It's not super straightforward/generalizable - best bet at the moment is to attach a response or default tool in addition to other tools you want the model to be able to call