Embodied-Planning-PostTrain

A post-training project for embodied task planning with SFT, DPO, hard-case refinement, and reproducible evaluation.

简体中文说明

Embodied-Planning-PostTrain is a reproducible post-training project for embodied task planning. It covers project-level SFT data construction, DPO preference learning, hard-case refinement, structured planning evaluation, and released LoRA adapters built on Qwen2.5-7B-Instruct.

Highlights

• Structured embodied task planning with a fixed output schema
• Project-level SFT and DPO training data
• Hard-test benchmark and behavior-oriented evaluation
• Released LoRA adapters for DPO and the best-current hard-focused SFT model

Overview

This repository contains the data, scripts, configs, evaluation artifacts, and released adapter checkpoints for an embodied task planning post-training project.

The project studies how to make an LLM generate structured robot task plans from:

• scene information
• task goals
• constraints
• available actions

The target output schema includes:

• goal
• subtasks
• action_sequence
• risks
• uncertainty
• efficiency_notes

Current Status

This project has completed:

• project-level SFT data construction and validation
• LoRA SFT training with LLaMA-Factory
• base vs SFT evaluation on normal and hard test sets
• DPO data construction, training, and evaluation
• multiple rounds of hard-focused SFT refinement
• best-current checkpoint selection

Current best adapter:

• artifacts/checkpoints/embodied_qwen/lora_sft_hard_v5

Key result on the hard test:

• base: 15/20 unnecessary clarifications
• SFT: 15/20
• DPO: 13/20
• best_current (sft_hard_v5): 4/20

Repository Structure

.
├── artifacts/
│   └── checkpoints/
│       └── embodied_qwen/
│           ├── lora_dpo/
│           └── lora_sft_hard_v5/
├── configs/
│   └── llamafactory/
├── data/
│   ├── dpo/
│   ├── eval/
│   └── sft/
├── docs/
├── scripts/
├── LICENSE
├── README.md
├── README_zh.md
└── requirements.txt

What Is Released

This repository releases:

• SFT training data
• DPO preference data
• hard-test evaluation artifacts
• training and evaluation scripts
• LLaMA-Factory training configs
• released LoRA adapters:
  • lora_dpo
  • lora_sft_hard_v5

This repository does not redistribute the base model weights. You must download the base model separately.

Environment Setup

1. Clone this repository

git clone https://github.com/Young-Loser/Embodied-Planning-PostTrain.git
cd Embodied-Planning-PostTrain

2. Create a Python environment

You can either use your existing LLaMA-Factory environment or create a fresh one:

conda create -n embodied-plan python=3.11 -y
conda activate embodied-plan
pip install -r requirements.txt

3. Install LLaMA-Factory

This project assumes you use LLaMA-Factory for training.

Example:

git clone https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e .
cd ..

Required External Downloads

You need to download a supported base model yourself.

Recommended base model used in this project:

• Qwen/Qwen2.5-7B-Instruct

Example with ModelScope:

pip install modelscope
modelscope download --model Qwen/Qwen2.5-7B-Instruct --local_dir /path/to/models/Qwen/Qwen2___5-7B-Instruct

Then update the model path in the config files if needed.

Register Datasets into LLaMA-Factory

LLaMA-Factory requires dataset entries in data/dataset_info.json.

This repository provides an automatic registration script:

python scripts/register_dataset_info.py --llamafactory_root /path/to/LLaMA-Factory

Quick Start

1. Train SFT

cd /path/to/LLaMA-Factory
DISABLE_VERSION_CHECK=1 llamafactory-cli train /path/to/Embodied-Planning-PostTrain/configs/llamafactory/embodied_qwen_lora_sft.yaml

2. Train DPO

cd /path/to/LLaMA-Factory
DISABLE_VERSION_CHECK=1 llamafactory-cli train /path/to/Embodied-Planning-PostTrain/configs/llamafactory/embodied_qwen_lora_dpo.yaml

3. Evaluate on the hard test

python -u /path/to/Embodied-Planning-PostTrain/scripts/run_dpo_compare.py

To evaluate only the best-current SFT adapter:

RUN_BASE=0 RUN_DPO=0 \
SFT_ADAPTER_PATH=/path/to/Embodied-Planning-PostTrain/artifacts/checkpoints/embodied_qwen/lora_sft_hard_v5 \
SFT_OUTPUT_NAME=sft_hard_v5_outputs.json \
python -u /path/to/Embodied-Planning-PostTrain/scripts/run_dpo_compare.py

Use the Released Adapters

Released adapters:

• artifacts/checkpoints/embodied_qwen/lora_dpo
• artifacts/checkpoints/embodied_qwen/lora_sft_hard_v5

These are LoRA adapters and should be loaded on top of the base model.

Prepare Your Own Embodied-Planning Data

SFT format

Each SFT sample follows Alpaca-style fields:

{
  "instruction": "请根据场景信息、任务目标、约束条件和可用动作，生成结构化机器人任务规划。",
  "input": "[Scene] ... [Task] ... [Constraints] ... [Available Actions] ...",
  "output": "{ ... structured planning output ... }"
}

You can refer to:

• data/sft/train.json
• data/sft/val.json
• data/sft/test.json

DPO format

Each DPO sample uses:

{
  "instruction": "...",
  "input": "...",
  "chosen": "{ ... preferred output ... }",
  "rejected": "{ ... worse output ... }"
}

You can refer to:

• data/dpo/train.json
• data/dpo/val.json

Evaluation Artifacts

Key documents:

• docs/experiment_results_summary_en.md
• docs/experiment_results_summary_zh.md
• docs/evaluation_protocol.md

Key hard-test outputs:

• data/eval/hard/base_outputs.json
• data/eval/hard/sft_outputs.json
• data/eval/hard/dpo_outputs.json
• data/eval/hard/sft_hard_v5_outputs.json

Notes

• The released base-model path in the configs may need to be changed to match your local environment.
• Training was originally run in a dedicated conda environment with GPU support.
• The best-current route in this repository is hard-focused incremental SFT, not DPO alone.

License

This project is released under the MIT License.