This repository has been archived by the owner on Dec 16, 2024. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathosu_allreduce_validation.log
152 lines (145 loc) · 7.14 KB
/
osu_allreduce_validation.log
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
# mpirun --allow-run-as-root -np 1 -H <cuda_ip> -mca pml ucx -mca coll_ucc_enable 1 -mca coll_ucc_priority 100 /osu-micro-benchmarks-7.4/c/mpi/collective/blocking/osu_allreduce -d cuda -T mpi_int --validation='log:/osu_validation' : -np 1 -H <rocm_ip> -mca pml ucx -mca coll_ucc_enable 1 -mca coll_ucc_priority 100 -x UCX_ROCM_COPY_D2H_THRESH=0 -x UCX_ROCM_COPY_H2D_THRESH=0 -x UCC_EC_ROCM_REDUCE_HOST_LIMIT=0 -x UCC_EC_ROCM_COPY_HOST_LIMIT=0 -x OMPI_MCA_mpi_accelerator_rocm_memcpyD2H_limit=0 -x OMPI_MCA_mpi_accelerator_rocm_memcpyH2D_limit=0 /osu-micro-benchmarks-7.4/c/mpi/collective/blocking/osu_allreduce -d rocm -T mpi_int --validation='log:/osu_validation'
# OSU MPI-CUDA Allreduce Latency Test v7.4
# Datatype: MPI_INT.
# Size Avg Latency(us) Validation
4 168.88 Fail
DATA VALIDATION ERROR: /osu-micro-benchmarks-7.4/c/mpi/collective/blocking/osu_allreduce exited with status 1 on message size 4.
__________________________________________________
### MPI_FLOAT CUDA RING VALIDATION LOG SNIPPET ###
Size: 4, Iteration:0, Datatype: MPI_FLOAT
Position Expected Actual
0 4.000000 2.000000
Size: 4, Iteration:1, Datatype: MPI_FLOAT
Position Expected Actual
0 8.000000 4.000000
Size: 4, Iteration:2, Datatype: MPI_FLOAT
Position Expected Actual
0 12.000000 6.000000
Size: 4, Iteration:3, Datatype: MPI_FLOAT
Position Expected Actual
0 16.000000 8.000000
Size: 4, Iteration:4, Datatype: MPI_FLOAT
Position Expected Actual
0 20.000000 10.000000
Size: 4, Iteration:5, Datatype: MPI_FLOAT
Position Expected Actual
0 24.000000 12.000000
Size: 4, Iteration:6, Datatype: MPI_FLOAT
Position Expected Actual
0 28.000000 14.000000
Size: 4, Iteration:7, Datatype: MPI_FLOAT
Position Expected Actual
0 32.000000 16.000000
Size: 4, Iteration:8, Datatype: MPI_FLOAT
Position Expected Actual
0 36.000000 18.000000
Size: 4, Iteration:9, Datatype: MPI_FLOAT
Position Expected Actual
0 40.000000 20.000000
Size: 4, Iteration:10, Datatype: MPI_FLOAT
Position Expected Actual
0 44.000000 22.000000
__________________________________________________
### MPI_FLOAT ROCM RING VALIDATION LOG SNIPPET ###
Size: 4, Iteration:0, Datatype: MPI_FLOAT
Position Expected Actual
0 4.000000 -0.000000
Size: 4, Iteration:1, Datatype: MPI_FLOAT
Position Expected Actual
0 8.000000 0.000000
Size: 4, Iteration:2, Datatype: MPI_FLOAT
Position Expected Actual
0 12.000000-466227832407371639270740615757824.000000
Size: 4, Iteration:3, Datatype: MPI_FLOAT
Position Expected Actual
0 16.000000 0.000000
Size: 4, Iteration:4, Datatype: MPI_FLOAT
Position Expected Actual
0 20.000000 0.000000
Size: 4, Iteration:5, Datatype: MPI_FLOAT
Position Expected Actual
0 24.000000 0.000000
Size: 4, Iteration:6, Datatype: MPI_FLOAT
Position Expected Actual
0 28.000000 0.000000
Size: 4, Iteration:7, Datatype: MPI_FLOAT
Position Expected Actual
0 32.000000 0.000000
Size: 4, Iteration:8, Datatype: MPI_FLOAT
Position Expected Actual
0 36.000000 0.000000
Size: 4, Iteration:9, Datatype: MPI_FLOAT
Position Expected Actual
0 40.000000-466227871092997866938874206355456.000000
Size: 4, Iteration:10, Datatype: MPI_FLOAT
Position Expected Actual
0 44.000000 0.000000
__________________________________________________
### MPI_INT CUDA RING VALIDATION LOG SNIPPET ###
Position Expected Actual
0 20 16843019
Size: 4, Iteration:1, Datatype: MPI_INT
Position Expected Actual
0 40 16843029
Size: 4, Iteration:2, Datatype: MPI_INT
Position Expected Actual
0 60 16843039
Size: 4, Iteration:3, Datatype: MPI_INT
Position Expected Actual
0 80 16843049
Size: 4, Iteration:4, Datatype: MPI_INT
Position Expected Actual
0 100 16843059
Size: 4, Iteration:5, Datatype: MPI_INT
Position Expected Actual
0 120 16843069
Size: 4, Iteration:6, Datatype: MPI_INT
Position Expected Actual
0 140 16843079
Size: 4, Iteration:7, Datatype: MPI_INT
Position Expected Actual
0 160 16843089
Size: 4, Iteration:8, Datatype: MPI_INT
Position Expected Actual
0 180 16843099
Size: 4, Iteration:9, Datatype: MPI_INT
Position Expected Actual
0 200 16843109
Size: 4, Iteration:10, Datatype: MPI_INT
Position Expected Actual
0 220 16843119
__________________________________________________
### MPI_INT ROCM RING VALIDATION LOG SNIPPET ###
Size: 4, Iteration:0, Datatype: MPI_INT
Position Expected Actual
0 20 -1903603473
Size: 4, Iteration:1, Datatype: MPI_INT
Position Expected Actual
0 40 1413327695
Size: 4, Iteration:2, Datatype: MPI_INT
Position Expected Actual
0 60 1289428272
Size: 4, Iteration:3, Datatype: MPI_INT
Position Expected Actual
0 80 1289428192
Size: 4, Iteration:4, Datatype: MPI_INT
Position Expected Actual
0 100 1289428192
Size: 4, Iteration:5, Datatype: MPI_INT
Position Expected Actual
0 120 1289428336
Size: 4, Iteration:6, Datatype: MPI_INT
Position Expected Actual
0 140 1413327695
Size: 4, Iteration:7, Datatype: MPI_INT
Position Expected Actual
0 160 1289428240
Size: 4, Iteration:8, Datatype: MPI_INT
Position Expected Actual
0 180 1289428192
Size: 4, Iteration:9, Datatype: MPI_INT
Position Expected Actual
0 200 1289428192
Size: 4, Iteration:10, Datatype: MPI_INT
Position Expected Actual
0 220 1289428336