[Question] What is the status of Vulkan backend?

[DanielMazurkiewicz]

Vulkan may be not the the best/fastest/easiest or so solution for inference, but is probably most portable GPU acceleration approach.

Is anyone working actively to add support for it? And if so what is status/progress? If not, is it planned?

[Green-Sky]

there are ggml-org/llama.cpp#2059 and ggml-org/llama.cpp#2039

[0cc4m]

Yeah, I'm working on it. Let me know if you have any questions. It's a big project, but I'm making progress.

[Calandiel]

Yeah, I'm working on it. Let me know if you have any questions. It's a big project, but I'm making progress.

Is there any low hanging fruit a newcomer to the project could help with?

[0cc4m]

Yeah, I'm working on it. Let me know if you have any questions. It's a big project, but I'm making progress.

Is there any low hanging fruit a newcomer to the project could help with?

@Calandiel If you have experience with Vulkan, maybe. Otherwise probably not.

[Calandiel]

Yeah, I'm working on it. Let me know if you have any questions. It's a big project, but I'm making progress.

Is there any low hanging fruit a newcomer to the project could help with?

@Calandiel If you have experience with Vulkan, maybe. Otherwise probably not.

I have. I've written vulkan based render pipelines professionally and made toy neural networks in vulkan trained with sgd. Been working with it at least in some capacity for the last 4 years or so.

[0cc4m]

Yeah, I'm working on it. Let me know if you have any questions. It's a big project, but I'm making progress.

Is there any low hanging fruit a newcomer to the project could help with?

@Calandiel If you have experience with Vulkan, maybe. Otherwise probably not.

I have. I've written vulkan based render pipelines professionally and made toy neural networks in vulkan trained with sgd. Been working with it at least in some capacity for the last 4 years or so.

Oh cool, I'd be glad to work something out. If you have Discord, send me a message (_occam), otherwise send me an Email and we'll find another way.

[Calandiel]

Yeah, I'm working on it. Let me know if you have any questions. It's a big project, but I'm making progress.

Is there any low hanging fruit a newcomer to the project could help with?

@Calandiel If you have experience with Vulkan, maybe. Otherwise probably not.

I have. I've written vulkan based render pipelines professionally and made toy neural networks in vulkan trained with sgd. Been working with it at least in some capacity for the last 4 years or so.

Oh cool, I'd be glad to work something out. If you have Discord, send me a message (_occam), otherwise send me an Email and we'll find another way.

Will do, see you on Discord!

[sorasoras]

I think nomic-ai have functional kompute of llama.cpp right now
https://github.com/nomic-ai/llama.cpp
and, GPT4ALL is plenty fast on my 7900XTX via vulkan.
but I am not sure how to integrate this on ggml as i am not a programmer.

[Green-Sky]

@sorasoras ggml-org/llama.cpp#4456

[slaren]

The vulkan and kompute backends have been merged in llama.cpp, all that is left is update the cmake build files to be able to use them in other ggml projects.

[Kreijstal]

The vulkan and kompute backends have been merged in llama.cpp, all that is left is update the cmake build files to be able to use them in other ggml projects.

does this affect anything using ggml? whisper.cpp stabledifussion.cpp? etc?

[Green-Sky]

@Kreijstal backends are usually upstreamed to ggml, but ggml-api consumers need to use them explicitly.

eg new-ish backend code here in ggml:
https://github.com/ggerganov/ggml/blob/master/src/ggml-kompute.h
https://github.com/ggerganov/ggml/blob/master/src/ggml-sycl.h
https://github.com/ggerganov/ggml/blob/master/src/ggml-vulkan.h

[codingWiz-rick]

I had a problem with the whispercpp while using with vulkan can you please tell me how to solve this ?
Would be great help to me

whisper_init_from_file_with_params_no_state: loading model from '/data/data/com.codewiz.ailyrics/files/home/whisper.cpp/models/ggml-tiny.bin' whisper_init_with_params_no_state: use gpu = 1 whisper_init_with_params_no_state: flash attn = 0 whisper_init_with_params_no_state: gpu_device = 0 whisper_init_with_params_no_state: dtw = 0 ggml_vulkan: Found 1 Vulkan devices: ggml_vulkan: 0 = Adreno (TM) 610 (Qualcomm Technologies Inc. Adreno Vulkan Driver) | uma: 1 | fp16: 0 | warp size: 64 | shared memory: 16384 | int dot: 0 | matrix cores: none whisper_init_with_params_no_state: devices = 2 whisper_init_with_params_no_state: backends = 2 whisper_model_load: loading model whisper_model_load: n_vocab = 51865 whisper_model_load: n_audio_ctx = 1500 whisper_model_load: n_audio_state = 384 whisper_model_load: n_audio_head = 6 whisper_model_load: n_audio_layer = 4 whisper_model_load: n_text_ctx = 448 whisper_model_load: n_text_state = 384 whisper_model_load: n_text_head = 6 whisper_model_load: n_text_layer = 4 whisper_model_load: n_mels = 80 whisper_model_load: ftype = 1 whisper_model_load: qntvr = 0 whisper_model_load: type = 1 (tiny) whisper_model_load: adding 1608 extra tokens whisper_model_load: n_langs = 99 ggml_vulkan: device Vulkan0 does not support 16-bit storage. libc++abi: terminating due to uncaught exception of type std::runtime_error: Unsupported device.

Can anyone provide me the solution for the error it would be great help to me