You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[DOC]: XFormer and FlashAttention which is faster? How to choose? #4625
Colossal AI LLaMA supports xformer and flash-attention both, but I can NOT find any doc about how to make a choice? Which one is faster? Which one has better accuracy?
Hi @butterluo , basically flash attention would be the first choice. However it may fail in some circumstances, where we have to use other versions of memory_efficient_attention from xformers. Fortunately, ColossalAI provides a ColoAttention API that is able to deal with the most cases. You could find its details at https://github.com/hpcaitech/ColossalAI/blob/main/colossalai/kernel/cuda_native/mha/mha.py
Closing as answered, see above. Flash attention is the default choice; through Shardformer it is enabled with enable_flash_attention=True, and ColoAttention picks a suitable kernel for the inputs.
Activity
kurisusnowdeng commentedon Sep 6, 2023
Hi @butterluo , basically flash attention would be the first choice. However it may fail in some circumstances, where we have to use other versions of
memory_efficient_attentionfrom xformers. Fortunately, ColossalAI provides aColoAttentionAPI that is able to deal with the most cases. You could find its details at https://github.com/hpcaitech/ColossalAI/blob/main/colossalai/kernel/cuda_native/mha/mha.pyJazyJiang commentedon Oct 19, 2026
Closing as answered, see above. Flash attention is the default choice; through Shardformer it is enabled with
enable_flash_attention=True, andColoAttentionpicks a suitable kernel for the inputs.