HaloAttention掩码问题

[L-wwww]

mask out padding (in the paper, they claim to not need masks, but what about padding?)

    mask = torch.ones(1, 1, h, w, device = device)
    mask = F.unfold(mask, kernel_size = block + (halo * 2), stride = block, padding = halo)
    mask = repeat(mask, '() j i -> (b i h) () j', b = b, h = heads)
    mask = mask.bool()

    max_neg_value = -torch.finfo(sim.dtype).max
    sim.masked_fill_(mask, max_neg_value)

代码似乎将所有非0值置为无穷小，根据注释似乎是想将padding位置 置为无穷小