fix crash in warmup for xpu

[sywangyi]

What does this PR do?

Fixes # (issue)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

[sywangyi]

@jiqing-feng @kaixuanliu

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[sywangyi]

@echarlaix @IlyasMoutawwakil please check and help merge

[echarlaix]

Thanks for the fix @sywangyi !