【训练问题】SimPO 训练完，推理结果重复（复读机问题）

[empty2enrich]

Reminder

• I have read the README and searched the existing issues.

System Info

Linux , llama-factory 最新版

Reproduction

Dpo 训练正常，SimPO 出现复读机问题。

示例：

"output_dpo": "您好。<mark>意外事故的发生往往不可预测。<mark/>。<mark>无忧保·成人综合意外。<mark/>。<mark。<mark。<mark。<mark。<mark。<mark。<mark。<mark。<mark。<mark。<mark。<mark。<mark。<mark。<mark。<mark。<mark。<mark。<mark。<mark。<mark。

训练命令

python -m torch.distributed.run \
    --nproc_per_node $NUM_PROCESSES \
    --nnodes $WORLD_SIZE \
    --node_rank $NODE_RANK \
    --master_addr $MASTER_ADDR \
    --master_port $MASTER_PORT \
    src/train.py \
    --deepspeed ${DS_CONFIG_PATH} \
    --stage dpo \
    --model_name_or_path ${model_name_or_path} \
    --do_train \
    --dataset ${dataset} \
    --preprocessing_num_workers 32 \
    --template empty \
    --finetuning_type full \
    --pref_beta 2.0 \
    --simpo_gamma 1.0 \
    --pref_loss simpo \
    --output_dir ${output_dir} \
    --overwrite_cache \
    --per_device_train_batch_size 1 \
    --gradient_accumulation_steps 1 \
    --lr_scheduler_type cosine \
    --logging_steps 1 \
    --flash_attn fa2 \
    --save_only_model True \
    --save_strategy epoch \
    --save_total_limit 2 \
    --learning_rate 1.0e-6 \
    --cutoff_len ${cutoff_len} \
    --num_train_epochs 2.0 \
    --warmup_ratio 0.1 \
    --plot_loss \
    --gradient_checkpointing True \
    --enable_liger_kernel True \
    --use_unsloth_gc True \
    --ddp_timeout 1800000000 \
    --bf16 \

Others

No response

[Maydaytyh]

方便看一下推理的命令吗

[empty2enrich]

方便看一下推理的命令吗

import json
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "/mnt4/ckpt/chonghan.ll/dpo/qwen2.5_72b_fanzhou/full/20250105_212352/checkpoint-148"

model = AutoModelForCausalLM.from_pretrained(
model_name,
torch_dtype="auto",
device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

text = '''
<|im_start|>system\n xxxxx。<|im_end|>\n<|im_start|>user\n什么是保险<|im_end|>\n<|im_start|>assistant\n

'''

model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
generated_ids = model.generate(
**model_inputs,
max_new_tokens=1024,
do_sample = False,
temperature = 0.4,
top_k = 5,
top_p = 0.1
)
generated_ids = [
output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]
response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)