vulkan: support copy from f32 to q4_0/q4_1/q5_0/q5_1/q8_0/iq4_nl

[jeffbolznv]

Shaders are based on cpy.cu.

For #11127.

This supports the same set of quants to be converted from f32 as CUDA. Looks like CUDA also supports OP_CPY for Q8_0 to F32, and for any quant to itself. I don't know if those are required, but they wouldn't be hard to add if so.

I haven't done any perf testing of these. CUDA is also using one thread per CTA, which sounds kind of slow but maybe it's not a perf-critical operation. In fact, the only testing I've done is test-backend-ops. I'll try to pull this into stable-diffusion.cpp to test.

CC @stduhpf

[jeffbolznv]

stable_diffusion.cpp "works" with this change, but loading the models is way slower and seems to be doing the quantization on the CPU. @stduhpf any idea?

[stduhpf]

Thanks a lot! It seems to work very well for supported types.

loading the models is way slower and seems to be doing the quantization on the CPU

Are you talking about quantization of the LoRA or of the base model? Because if you're talking about the base model, I think this is the expected behaviour even without those changes, even though it would be nice to quantize using GPU now that the shaders exist for it. It's even single-threaded, so it takes forever with larger models.

I didn't notice any slowness loading the LoRA, and it looks like it was using the GPU as expected.

[jeffbolznv]

Yeah, I think it was the base model. Thanks for clarifying.

[slaren]

In llama.cpp, F32 -> Quant is needed for KV quantization, and Quant -> F32 conversion is used for context shifts when quantizing the K cache.

[jeffbolznv]

We also have GET_ROWS supporting dequant. Does context shifting use CPY or GET_ROWS?

[slaren]

It uses CPY.

[jeffbolznv]

I started implementing q->f32, noticed there weren't backend tests, and when I added them they error out because ggml_compute_forward_dup doesn't support q->f32. Am I missing something?

  CPY(type_src=q4_0,type_dst=f32,ne=[256,4,4,4],permute=[0,0,0,0]): C:\github\jeffbolznv\llama.cpp\ggml\src\ggml-cpu\ggml-cpu.c:3996: fatal error

[jeffbolznv]

I've gone ahead and implemented the missing ggml-cpu function.