TL/UCP: Allow self copy in allgather using network loopback #1021
+224
−95
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Can one of the admins verify this patch? |
yaeliyac
force-pushed
the
origin/for_plugin
branch
from
5e852a7 to
7e6c55d
Compare
yaeliyac
force-pushed
the
origin/for_plugin
branch
from
d13b885 to
b3c2e80
Compare
yaeliyac
changed the title
|
ok to test |
nsarka
approved these changes
Dec 12, 2024
yaeliyac
force-pushed
the
origin/for_plugin
branch
3 times, most recently
from
482006f to
eecdebf
Compare
yaeliyac
force-pushed
the
origin/for_plugin
branch
from
1d88c9b to
820794d
Compare
|
@Sergei-Lebedev why do we have to keep approving runs when force pushing?? Seems an outlier here |
yaeliyac
force-pushed
the
origin/for_plugin
branch
3 times, most recently
from
32b3ccd to
b1c875b
Compare
yaeliyac
changed the title
yaeliyac
force-pushed
the
origin/for_plugin
branch
3 times, most recently
from
2dc74f3 to
a2c5fee
Compare
yaeliyac
force-pushed
the
origin/for_plugin
branch
from
a2c5fee to
5940100
Compare
samnordmann
reviewed
Dec 16, 2024
|
@yaeliyac btw, for knomial inplace, we don't expect any change, right? Do we have an explanation for the degradation in the graph? |
yaeliyac
force-pushed
the
origin/for_plugin
branch
2 times, most recently
from
6cb2e0e to
5ba0039
Compare
ikryukov
reviewed
Dec 18, 2024
yaeliyac
force-pushed
the
origin/for_plugin
branch
2 times, most recently
from
5619791 to
caa90f8
Compare
yaeliyac
force-pushed
the
origin/for_plugin
branch
from
caa90f8 to
ee808ea
Compare
ikryukov
reviewed
Jan 6, 2025
| ucc_coll_args_t *args = &TASK_ARGS(task); | ||
| ucc_tl_ucp_team_t *team = TASK_TEAM(task); | ||
| ucc_kn_radix_t radix = task->allgather_kn.p.radix; | ||
| ucc_tl_ucp_task_t * task = ucc_derived_of(coll_task, ucc_tl_ucp_task_t); |
There was a problem hiding this comment.
suggestion is not to reformat existing code (* symbol), because clang format formats different depending on version, so it is better to leave as before. @samnordmann @Sergei-Lebedev what do you think?
There was a problem hiding this comment.
indeed, clang format often does this undesirable reformatting... We should have a long-term fix -- in the meantime I suggest to manually place the "*" where they should
yaeliyac
force-pushed
the
origin/for_plugin
branch
5 times, most recently
from
a353504 to
b6d4812
Compare
yaeliyac
force-pushed
the
origin/for_plugin
branch
from
b6d4812 to
54e86f2
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
Add option for self copy loopback in out of place start for allgather algorithms: knomial, ring, neighbor, sparbit
Why ?
When using UCC plugin for NCCL, using cuda_memcpy might cause deadlock
How ?
Add UCC_TL_UCP_ALLGATHER_USE_LOOPBACK flag to control this
Tested on ISRAEL-1 with multiple test cases - showed good results and same performance with and without loopback