Skip to content

Commit

Permalink
adjusted prompt to reduce halucinations: nested filter/context calls …
Browse files Browse the repository at this point in the history
…and putting filter args in quotation marks
  • Loading branch information
ds-jakub-cierocki committed Jul 23, 2024
1 parent 5d4ff64 commit e7e8826
Show file tree
Hide file tree
Showing 2 changed files with 33 additions and 13 deletions.
8 changes: 7 additions & 1 deletion src/dbally/iql_generator/prompt.py
Original file line number Diff line number Diff line change
Expand Up @@ -76,10 +76,16 @@ def __init__(
"It is VERY IMPORTANT not to use methods other than those listed above.\n"
"Finally, if a called function argument value is not directly specified in the query but instead requires "
"some additional execution context, than substitute that argument value with: AskerContext().\n"
'The typical input phrase suggesting that the additional execution context need to be referenced contains words like: "I", "my", "mine", "current", "the" etc..\n'
"The typical input phrase suggesting that the additional execution context need to be referenced \n"
'contains words like: "I", "my", "mine", "current", "the" etc..\n'
'For example: "my position name", "my company valuation", "current day", "the ongoing project".\n'
"In that case, the part of the output will look like this:\n"
"filter4(AskerContext())\n"
"Outside this situation DO NOT combine filters like this:\n"
"filter4(filter2())\n"
"And NEVER quote the filter argument unless you're sure it represents the string/literal datatype, \n"
"Especially do not quote AskerContext() calls like this:\n"
"filter2('AskerContext()')\n"
"""If you DON'T KNOW HOW TO ANSWER DON'T SAY \"\", SAY: `UNSUPPORTED QUERY` INSTEAD! """
"This is CRUCIAL, otherwise the system will crash. "
),
Expand Down
38 changes: 26 additions & 12 deletions tests/unit/test_iql_format.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,12 +22,19 @@ async def test_iql_prompt_format_default() -> None:
"DO NOT INCLUDE arguments names in your response. Only the values.\n"
"You MUST use only these methods:\n"
"\n\n"
"It is VERY IMPORTANT not to use methods other than those listed above."
"Finally, if a called function argument value is not directly specified in the query but instead requires some additional execution context, than substitute that argument value with: AskerContext()."
'The typical input phrase suggesting that the additional execution context need to be referenced contains words like: "I", "my", "mine", "current", "the" etc..'
'For example: "my position name", "my company valuation", "current day", "the ongoing project".'
"In that case, the part of the output will look like this:"
"filter4(AskerContext())"
"It is VERY IMPORTANT not to use methods other than those listed above.\n"
"Finally, if a called function argument value is not directly specified in the query but instead requires "
"some additional execution context, than substitute that argument value with: AskerContext().\n"
"The typical input phrase suggesting that the additional execution context need to be referenced \n"
'contains words like: "I", "my", "mine", "current", "the" etc..\n'
'For example: "my position name", "my company valuation", "current day", "the ongoing project".\n'
"In that case, the part of the output will look like this:\n"
"filter4(AskerContext())\n"
"Outside this situation DO NOT combine filters like this:\n"
"filter4(filter2())\n"
"And NEVER quote the filter argument unless you're sure it represents the string/literal datatype, \n"
"Especially do not quote AskerContext() calls like this:\n"
"filter2('AskerContext()')\n"
"""If you DON'T KNOW HOW TO ANSWER DON'T SAY \"\", SAY: `UNSUPPORTED QUERY` INSTEAD! """
"This is CRUCIAL, otherwise the system will crash. ",
"is_example": False,
Expand Down Expand Up @@ -57,12 +64,19 @@ async def test_iql_prompt_format_few_shots_injected() -> None:
"DO NOT INCLUDE arguments names in your response. Only the values.\n"
"You MUST use only these methods:\n"
"\n\n"
"It is VERY IMPORTANT not to use methods other than those listed above."
"Finally, if a called function argument value is not directly specified in the query but instead requires some additional execution context, than substitute that argument value with: AskerContext()."
'The typical input phrase suggesting that the additional execution context need to be referenced contains words like: "I", "my", "mine", "current", "the" etc..'
'For example: "my position name", "my company valuation", "current day", "the ongoing project".'
"In that case, the part of the output will look like this:"
"filter4(AskerContext())"
"It is VERY IMPORTANT not to use methods other than those listed above.\n"
"Finally, if a called function argument value is not directly specified in the query but instead requires "
"some additional execution context, than substitute that argument value with: AskerContext().\n"
"The typical input phrase suggesting that the additional execution context need to be referenced \n"
'contains words like: "I", "my", "mine", "current", "the" etc..\n'
'For example: "my position name", "my company valuation", "current day", "the ongoing project".\n'
"In that case, the part of the output will look like this:\n"
"filter4(AskerContext())\n"
"Outside this situation DO NOT combine filters like this:\n"
"filter4(filter2())\n"
"And NEVER quote the filter argument unless you're sure it represents the string/literal datatype, \n"
"Especially do not quote AskerContext() calls like this:\n"
"filter2('AskerContext()')\n"
"""If you DON'T KNOW HOW TO ANSWER DON'T SAY \"\", SAY: `UNSUPPORTED QUERY` INSTEAD! """
"This is CRUCIAL, otherwise the system will crash. ",
"is_example": False,
Expand Down

0 comments on commit e7e8826

Please sign in to comment.