Selecting GPUs and compute backends

[farooqkz]

Is your feature request related to a problem? Please describe.

These days a good number of computers come with multiple GPUs, and each GPU usually has got multiple APIs to offload computation to. For instance, Most GPUs support both OpenCL and Vulkan for compute. Intel Arc GPUs also have got XMX cores specifically created for matrix operations. Furthermore, many laptops and workstations come with more than a single GPU.

Describe the solution you'd like

It would be nice to have a selection mechanism for selecting GPU or CPU. And if it's a GPU, which GPU if there are multiple GPUs. There should be a global setting and users should be able to override the global one per model. Sensible defaults should be there. Such as always trying to use a GPU if one is available.

Describe alternatives you've considered

Not much of an alternative comes to mind.

Additional context
Additional to GPUs, some PCs, even cheap SBCs come with an NPU or TPU. This should be kept in mind.

[Jeffser]

Hi, the backend for Alpaca is Ollama, as far as I know Ollama is only compatible with ROCm and Cuda (no Vulkan / OpenCL)

Alpaca does provide some options in the preferences dialog to modify which GPU is being used, for example CUDA_VISIBLE_DEVICES and HIP_VISIBLE_DEVICES

[Jeffser]

I just noticed Ollama doesn't use HIP_VISIBLE_DEVICES anymore, I'm going to fix that

[Jeffser]

4d50b7d

Alright that fixed,

I could make a GPU / CPU selector in the preferences dialog but it could take a while to get it working correctly

For now, using those overrides is the best way of configuring Ollama

[ossenthi]

I haven't use the program for while, but I started using it again about a week ago and noticed it doesn't use my Nvidia RTX mobile Max-Q Gpu anymore. I Have tried everything guide told me to. I have cuda installed and works for other ollama-cuda setup. we need better way to diagnose problems when it comes to fails to utilize gpu.

[farooqkz]

Hi, the backend for Alpaca is Ollama, as far as I know Ollama is only compatible with ROCm and Cuda (no Vulkan / OpenCL)

Alpaca does provide some options in the preferences dialog to modify which GPU is being used, for example CUDA_VISIBLE_DEVICES and HIP_VISIBLE_DEVICES

There is a PR to add Vulkan support to Ollama. Then with that, in most multi GPU setups, there are also multiple backends as most GPUs, even ARM ones, support Vulkan.

Edit:

I just looked at the PR more thorough. And it seems the ollama team members are simply ignoring the PR...