Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Automatically apply chat template in non-chat scenarios #1533

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Automatically apply chat template in non-chat scenarios #1533

wants to merge 3 commits into from
+138 −22

Conversation

sbalandi
Copy link
Contributor

@sbalandi sbalandi commented Jan 13, 2025 •
edited
Loading

@github-actions github-actions bot added category: visual language Visual language pipeline category: LLM LLM pipeline (stateful, static) no-match-files labels Jan 13, 2025
ilya-lavrenov
ilya-lavrenov reviewed Jan 13, 2025
View reviewed changes
src/README.md Outdated
@@ -73,6 +73,8 @@ output:
'it is made up of carbon atoms. The carbon atoms are arranged in a linear pattern, which gives the yellow color. The arrangement of carbon atoms in'
```

>**Note**: The chat_template from tokenizer_config.json will be automatically applied to the prompt at the generation stage. If you want to disable it, you can do it by calling pipe.get_tokenizer().set_chat_template("").
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we can also add to generate() method docs? for overloads which operate with strings for both VLM and LLM

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean like that

openvino.genai/src/cpp/include/openvino/genai/llm_pipeline.hpp

Line 188 in 863f556

* chat_template will be applied to the prompt, run pipe.get_tokenizer().set_chat_template(custom_chat_template) to update it.
?

src/cpp/src/icontinuous_batching.cpp
try {
ChatHistory history({{{"role", "user"}, {"content", prompt}}});
constexpr bool add_generation_prompt = true;
auto templated_prompt = m_tokenizer.apply_chat_template(history, add_generation_prompt);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like you need to fix our test framework to do the same as chat template is applied only for chat cases

The same for tests in .github folder.

@ilya-lavrenov ilya-lavrenov added this to the 2025.0 milestone Jan 13, 2025
Wovchena
Wovchena reviewed Jan 13, 2025
View reviewed changes
src/cpp/src/icontinuous_batching.cpp Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
@github-actions github-actions bot added category: GHA CI based on Github actions category: tokenizers Tokenizer class or submodule update category: GenAI C++ API Changes in GenAI C++ public headers labels Jan 13, 2025
@sbalandi sbalandi force-pushed the chat_templ branch 2 times, most recently from f1ece12 to e5fa889 Compare January 13, 2025 21:42
@github-actions github-actions bot added the category: samples GenAI samples label Jan 13, 2025
@AlexKoff88
Copy link
Collaborator

If using a chat template is supposed to be a default behavior for .generate() method it is not aligned with HF Transformers lib. We should turn it off in the tools at least (both WWB and LLM-Bench).

@ilya-lavrenov
Copy link
Contributor

ilya-lavrenov commented Jan 14, 2025 •
edited
Loading

If using a chat template is supposed to be a default behavior for .generate() method it is not aligned with HF Transformers lib. We should turn it off in the tools at least (both WWB and LLM-Bench).

what's about HF e2e pipeline?
do they apply chat_template by default?

@eaidova

@AlexKoff88
Copy link
Collaborator

If using a chat template is supposed to be a default behavior for .generate() method it is not aligned with HF Transformers lib. We should turn it off in the tools at least (both WWB and LLM-Bench).

what's about HF e2e pipeline? do they apply chat_template by default?

@eaidova

text2text-generation pipeline does not use a chat template by default from what I know.

@ilya-lavrenov
Copy link
Contributor

If using a chat template is supposed to be a default behavior for .generate() method it is not aligned with HF Transformers lib. We should turn it off in the tools at least (both WWB and LLM-Bench).

what's about HF e2e pipeline? do they apply chat_template by default?
@eaidova

text2text-generation pipeline does not use a chat template by default from what I know.

what if it's instruction model?

@AlexKoff88
Copy link
Collaborator

If using a chat template is supposed to be a default behavior for .generate() method it is not aligned with HF Transformers lib. We should turn it off in the tools at least (both WWB and LLM-Bench).

what's about HF e2e pipeline? do they apply chat_template by default?
@eaidova

text2text-generation pipeline does not use a chat template by default from what I know.

what if it's instruction model?

Double-checked, and it seems like HF changed the behaviour at some point for text-generation pipeline. Details. But the input should be formatted appropreatelly to trigger chat template usage. If the user just passes a string data, no chat template is applied.

@ilya-lavrenov
Copy link
Contributor

If using a chat template is supposed to be a default behavior for .generate() method it is not aligned with HF Transformers lib. We should turn it off in the tools at least (both WWB and LLM-Bench).

what's about HF e2e pipeline? do they apply chat_template by default?
@eaidova

text2text-generation pipeline does not use a chat template by default from what I know.

what if it's instruction model?

Double-checked, and it seems like HF changed the behaviour at some point for text-generation pipeline. Details. But the input should be formatted appropreatelly to trigger chat template usage. If the user just passes a string data, no chat template is applied.

do you think it's better to add explicit flag, then?

pipe.generate(prompt, apply_chat_template=True, max_new_tokens=40)

@AlexKoff88
Copy link
Collaborator

If using a chat template is supposed to be a default behavior for .generate() method it is not aligned with HF Transformers lib. We should turn it off in the tools at least (both WWB and LLM-Bench).

what's about HF e2e pipeline? do they apply chat_template by default?
@eaidova

text2text-generation pipeline does not use a chat template by default from what I know.

what if it's instruction model?

Double-checked, and it seems like HF changed the behaviour at some point for text-generation pipeline. Details. But the input should be formatted appropreatelly to trigger chat template usage. If the user just passes a string data, no chat template is applied.

do you think it's better to add explicit flag, then?

pipe.generate(prompt, apply_chat_template=True, max_new_tokens=40)

This option looks good to me but for drop-in replacement of HF API to OV GenAI it is better to follow HF approach with message format. Anyway, they should have more experience and user's feedback.

@sbalandi
Copy link
Contributor Author

If using a chat template is supposed to be a default behavior for .generate() method it is not aligned with HF Transformers lib. We should turn it off in the tools at least (both WWB and LLM-Bench).

what's about HF e2e pipeline? do they apply chat_template by default?
@eaidova

text2text-generation pipeline does not use a chat template by default from what I know.

what if it's instruction model?

Double-checked, and it seems like HF changed the behaviour at some point for text-generation pipeline. Details. But the input should be formatted appropreatelly to trigger chat template usage. If the user just passes a string data, no chat template is applied.

do you think it's better to add explicit flag, then?

pipe.generate(prompt, apply_chat_template=True, max_new_tokens=40)

This option looks good to me but for drop-in replacement of HF API to OV GenAI it is better to follow HF approach with message format. Anyway, they should have more experience and user's feedback.

Should both ways be added - possibility to put messages to the generate() function, apply chat_template if input value is messages and leave as is if it's string and add apply_chat_template as input parameter for generate() ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Wovchena Wovchena Wovchena left review comments

@ilya-lavrenov ilya-lavrenov ilya-lavrenov left review comments

@AlexKoff88 AlexKoff88 Awaiting requested review from AlexKoff88

At least 1 approving review is required to merge this pull request.

Assignees

@ilya-lavrenov ilya-lavrenov

@Wovchena Wovchena

Labels
category: GenAI C++ API Changes in GenAI C++ public headers category: GHA CI based on Github actions category: LLM LLM pipeline (stateful, static) category: samples GenAI samples category: tokenizers Tokenizer class or submodule update category: visual language Visual language pipeline no-match-files
Projects
None yet
Milestone
2025.0
Development

Successfully merging this pull request may close these issues.
4 participants
@sbalandi @AlexKoff88 @ilya-lavrenov @Wovchena