Repository navigation
[BUG]: some questions about sequence parallel聽#3866
Description
馃悰 Describe the bug
I am very confused about the implementation of sequence parallelism in your code. I found that there are only some examples of applying sequence parallelism with pipe parallelism. Therefore, Have you applied the sequence parallelism in your main code? and is the tensor and sequence parallelism implemented together?
Environment
No response
Activity
JazyJiang commentedon Oct 19, 2026
Sorry for the late reply. Sequence parallelism is now part of
HybridParallelPluginand can be combined with tensor parallelism:sequence_parallelism_mode="split_gather"(or"ring") runs SP over the TP group, and"all_to_all"(DeepSpeed-Ulysses) and"ring_attn"use a separatesp_size. See https://colossalai.org/docs/features/sequence_parallelism. Closing as answered.