benchdnn: graph: extend --dt to support specifying tensor id #2331
Conversation
github-actions
bot
added
documentation
TaoLv
force-pushed
the
lvtao/main/benchdnn-dt-rewrite
branch
from
086c6dd to
ab80ed6
Compare
TaoLv
force-pushed
the
lvtao/main/benchdnn-dt-rewrite
branch
from
ab80ed6 to
539767a
Compare
TaoLv
force-pushed
the
lvtao/main/benchdnn-dt-rewrite
branch
from
539767a to
143ce80
Compare
Thank you @ranukund . Suggestions are incorporated now. |
|
make test |
| // format like --dt=f32,bf16,f16 | ||
| const std::vector<dnnl_data_type_t> def_dt = {dnnl_data_type_undef}; | ||
| parser::parse_dt(dts, def_dt, str, option_name); | ||
| } else { |
There was a problem hiding this comment.
Based on this logic it looks like the following will work:
--dt=f16 --dt=0:f32+... --case=some_case
if branch will update dt, then else branch will update dt_map, then dt_rewrite will update the whole graph, and then dt_map_rewrite will update specific IDs.
Is it an expected flow?
If yes, should it be properly documented?
If no, then it probably should be restricted here by additional checking whether the other object was updated or not.
There was a problem hiding this comment.
Good catch! I did not expect the two --dt flavors would be specified in a single command line. I just added a check in the bench() function to error out.
TaoLv
force-pushed
the
lvtao/main/benchdnn-dt-rewrite
branch
from
3e42fa0 to
7d3f6c3
Compare
tests/benchdnn/graph/bench_graph.cpp
Outdated
| BENCHDNN_PRINT(0, "%s\n", | ||
| "Error: --dt is specified twice with different styles."); | ||
| SAFE_V(FAIL); | ||
| } |
There was a problem hiding this comment.
I would move it into parse_dt since it has both objects and can check their state to figure out if both options were passed. Afterall, it's a parser obligation to make sure the input is valid (when it can).
There was a problem hiding this comment.
Sure, it's moved to the parser function now.
rewrite sdpa-compressed-kv-int4-gs32.json for it.
TaoLv
force-pushed
the
lvtao/main/benchdnn-dt-rewrite
branch
from
7d3f6c3 to
2fd06e0
Compare
We introduced
--dtin 461388e to support floating-point data type rewrite for the whole graph.In this PR, the knob is further extended to support data type rewrite for specified tensor IDs, in a format of
--dt=ID0:DT0+ID1:DT1.... It provides more flexibility to test a json file with different data types, especially for the various zps/scales data types for quantizations which were excluded previously in whole graph data type rewrite. Due to the flexibility of this knob and the restriction of op definition in the library, users need to be cautious when rewriting the data type of a specific tensor in a graph as it may lead to graph construction failures in the library during testing (eg. incompatible input and output tensor data types).An example as below:
benchdnn --graph --dt=0:s8+2:s8+6:s8+8:s8 --case=complex_fusion/mha/sdpa-compressed-kv-int4-gs32.jsonIt can be used to test int8 compressed KV graph by rewriting rewrite the int4 compressed KV graph (0 is the key tensor, 2 is the zps pf key tensor, 6 is the value tensor, and 8 is the zps of values tensor).