Structured Output with Groq: Error code: 400 - {'error': {'message': 'Failed to call a function. Please adjust your prompt. ...'

[ptrem]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

class Code(BaseModel):
    """Code output"""

    prefix: str = Field(description="Description of the code snip. Describes the detailed function usage, the arguments, outputs and gives an example if needed.")
    imports: str = Field(description="Code block import statements")
    function_name: str = Field(description="Name of the function")
    code: str = Field(description="Executable code block using pep 8 code style. Not including import statements and not including applying function")

structured_llm = llm.with_structured_output(Code, include_raw=True)

Description

I'm trying to implement a coding assistant using an LLM with structured outuput (llm.with_structured_output ) similar to the LangChain tutorial. Using the OpenAI API everything works fine, but using the Groq API the model struggles with generating a structured output. Especially with longer context examples.
For me the model output looks quite okay and i don't know why the tool function caller struggles with the output.
I use llama3-70b-8192, using a model with larger context length results in the same error.
Any idea what the problem is or how to implement a fallback for more robust pipeline.
Thanks for your help

The error message:

Error code: 400 -
    { 
        'error': 
            { 
                'message': 
                    "Failed to call a function. Please adjust your prompt. See 'failed_generation' for more details.", 
                'type': 'invalid_request_error', 
                'code': 'tool_use_failed', 
                'failed_generation': 
                    '<tool-use>\n 
                        {\n 
                            "tool_calls": 
                                [\n 
                                    {\n 
                                        "id": "pending",\n 
                                        "type": "function",\n 
                                        "function": 
                                            {\n
                                                "name": "Code"\n
                                            },\n
                                        "parameters": 
                                            {\n 
                                                "code": 
                                                 """\n \
                                                    def my_function(args):\n
                                                        \'\'\'\n 
                                                        Function description
                                      
                                                        Args:\n
                                                            arg1 (str): argument 1\n 
                                                            arg2 (dict): argument 2.\n 
                                                        Returns:\n 
                                                            None\n 
                                                        '\'\'\n\n
                                                        # Code comes here\n ...
                                                        
                                                    """,\n \
                                                    "function_name": "my_function",\n
                                                    "prefix": "Description of the function.",\n
                                                    "imports": "import numpy"\n
                                            }\n
                                    }\n
                                ]\n
                        }\n
                    </tool-use>' 
            } 
    }

System Info

Platform: Linux
Python: 3.11.9

langchain 0.2.6
langchain-community 0.2.6
langchain-core 0.2.11
langchain-experimental 0.0.62
langchain-groq 0.1.6
langchain-nomic 0.1.2
langchain-openai 0.1.14
langchain-text-splitters 0.2.2
langgraph 0.1.5
langsmith 0.1.83

[dosubot[bot]]

To resolve the issue with the Groq API struggling to generate structured output for longer context examples in your coding assistant implementation, you can use the .with_structured_output() method to coerce ChatGroq into returning a structured output. Here is an example:

from langchain_core.pydantic_v1 import BaseModel, Field
from langchain_groq import ChatGroq

class Code(BaseModel):
    """Code output"""
    prefix: str = Field(description="Description of the code snip. Describes the detailed function usage, the arguments, outputs and gives an example if needed.")
    imports: str = Field(description="Code block import statements")
    function_name: str = Field(description="Name of the function")
    code: str = Field(description="Executable code block using pep 8 code style. Not including import statements and not including applying function")

chat = ChatGroq(
    temperature=0,
    model="llama3-70b-8192",
    # api_key="" # Optional if not set as an environment variable
)

structured_llm = chat.with_structured_output(Code, include_raw=True)

response = structured_llm.invoke("Generate a Python function with detailed description, imports, and code.")
print(response)

This method leverages the tool-calling functionality to ensure the output matches the specified schema, which can help manage longer context examples more effectively. Additionally, Groq supports JSON mode, which can be specified if needed:

structured_llm = chat.with_structured_output(Code, method="json_mode", include_raw=True)

response = structured_llm.invoke(
    "Generate a Python function with detailed description, imports, and code. Respond in JSON with `prefix`, `imports`, `function_name`, and `code` keys."
)
print(response)

However, be aware that there are known issues with the Groq API's structured output functionality, as indicated by the xfail markers in the integration tests. These tests show that the structured output and tool message histories may fail with errors such as "Failed to call a function. Please adjust your prompt." Therefore, it might be necessary to adjust your prompts or wait for updates from the Groq team to fully resolve these issues ^[1][2][3][4].

To continue talking to Dosu, mention @dosu.

[falmanna]

Any solution here?

[ptrem]

It seemed to be a problem of the Pydantic parser. Even if the LLM produces an output that looks structured to me. With the required keys and format, the parser isn't possible to extract the information. It seems not to be optimized for the Groq API or Llama3 model as it works much better with the OpenAI API. Especially in long context examples the parser fails. I therefor use now a custom parser. With the exact output structure given in the prompt Llama3 and Gemini-1.5 pro (where the Pydantic structured output doesn't work either) it is possible to structure generate a structured output and it works quite stable.

[sopfi]

I face the same issue. Could you provide your fixed code with your custom parser here? Thanks!

[vjrngn]

Could you please provide the code that fixed this issue? Looking forward. Thanks!

[ptrem]

Yes sure. The custom parser is really depending on your problem and also on the way you prompt the LLM. I advice you to clearly prompt the LLM to generate an output with the structure you need/want. And then write a parser. My way to go was to adjust the prompt to get a reproducible output structure. And then I used chatGPT to write a parser for my outputs given some examples. Works fine for me so far, i hope this helps you.

The following example shows the instruction prompt, query and parser for my problem.

Instruction Prompt

    prompt_instruct_gen = '''You are a coding assistant with expertise to write specific python functions.
        Here is a python code example with a specific task \n ------- \n  {code_example_context} \n ------- \n
        Write the python code for a new set of parameters given in the user question.
        Ensure the following JSON structure
            \{{
                "imports": "...", # of external tools
                "code": """...""", # code block
            \}}
        \n Customize the code answer the following setup and parameters:'''

Query

    query = f'''Write the python code to <<YOUR TASK>>
    Given is the following set of parameters: \n ------- \n  {params_input_file} \n ------- \n
    Insert the generated function code in the <<<FUNCTION CONTENT>>> section. 
    Add a list of the variable parameter names in <<<VARIABLE PARAMETERS>>>.
    Return the entire code block, including the function definition and the content, in executable Python format following PEP 8 style.
    List the function arguments and the required dictionary keys in the function documentation header. 
    IMPORTANT: make sure you use the correct name for the custom import file. Depending on the project name defined in the parameters file
    
    code:"""
    imports ... # make sure of the correct import names depending on the project_name

    def generate_geometry(argument1):
        """
        Description of function.
        Args:
            argument1: Interactive instance ....
                keys: <<<VARIABLE PARAMETERS>>>
        Returns:
            none
        """

        <<<FUNCTION CONTENT>>>
    return
    """
    Respond in JSON with string keys: "imports": "...", "code": """...""".
    '''

Parser

def custom_json_code_ouput_parser(input_string):
    """
    Parser for the code output into JSON format with the keys `imports` and `code`.
    The function searches for the defined keys and the defined JSON file pattern in the input string
    and parses it into a JSON file.

    Arguments:
        input_string (str): code string generated by the LLM
    
    Return:
        parsed_dict (dict): parsed dictionary
    """

    # Define a regex pattern to extract the components
    pattern = re.compile(
        r'"imports":\s*"([^"]+)",\s*'
        r'"code":\s*"""(.*?)"""\s*}', re.DOTALL
    )
    
    try:
        match = pattern.search(input_string)
        if not match:
            raise ValueError(f"Input string does not match the expected format! {input_string}")
    except:
        pass

        pattern = re.compile(
        r'"imports":\s*"([^"]+)",\s*'
        r'"code":\s*"(.*?)"\s*}', re.DOTALL
    )
    
    try:
        match = pattern.search(input_string)
        if not match:
            raise ValueError(f"Input string does not match the expected format! {input_string}")
    except:
        pass

    imports_raw, code_raw = match.groups()
    imports = imports_raw.replace('\\n', '\n')
    code = (code_raw.replace('\\n', '\n')
            .replace('\\"', '\"'))
    
    # Create the dictionary
    parsed_dict = ParsedOutput({
        "imports": imports,
        "code": code
    })
    
    return parsed_dict

[Ansumanbhujabal]

I am facing the same problem but with JsonOutputToolsParser, PydanticToolsParser , Using
llm = ChatGroq(model="llama3-8b-8192")
Is there any other solution available other than writing custom parser ?

[ayush-0411]

I am facing same problem but it shows API error, inspite I have set api in environment as a variable.