Learn out_scale

[bus6479]

I am trying to do Hardware aware training.
I am interested in out_scaling, so I setout_scaling_alpha=Trueand out_scale_columnwise=True.
However, after many epochs, parameters out_scaling_alpha still stay at 1. I checked require_grad=True and it has gradient. Why out_scaling_alpha value is not changed? and please give me some tips about out_scaling alpha. Thank you!

[maljoras]

Hi @bus6479, many thanks for the question.

It is advised to not directly access the "out_scaling_alpha" value, but instead use the method analog_tile.get_scales() to get the scales. This is because there are multiple scales, such as the mapping scaling, that is used for the re-mapping processes. If you use the get_scales method, the overall scale is computed.

Besides this, there is also a weight remapping that multiplies the overall scales and re-sets the weights according to the weight remapping scales. This will re-set the output scales to 1 as the overall scale is applied to the mapping scales. To see how the out_scaling_alpha is changed, you need to turnoff the weight remapping, or watch the overall scale instead (with get_scales())

[bus6479]

I got it. Thank you!