ggml_vulkan: No devices found using Vulkan docker image.

[git2212]

My starting assumption is that I missed something although I tried following the instructions for compiling Vulkan support for a docker image at https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md#for-docker-users.

Image compiles and the container runs using:

docker run -d --name llama-cpp-tmp -p 8010:8080 -v "/mnt/ai-models-llama:/models" --rm -ti --device /dev/dri/renderD128:/dev/dri/renderD128 --device /dev/dri/renderD129:/dev/dri/renderD129 llama-cpp-vulkan:b6503 ...
... but it loads everything onto the CPU...

running the command inside the docker container llama-server --list-devices returns

ggml_vulkan: No devices found.
load_backend: loaded Vulkan backend from /app/libggml-vulkan.so
load_backend: loaded CPU backend from /app/libggml-cpu-haswell.so
Available devices:

rather odd considering that on the host vulkaninfo reports multiple gpus and ls -la /dev/dri:

crw-rw---- 1 nobody nogroup 226, 128 Sep 20 00:37 renderD128
crw-rw---- 1 nobody nogroup 226, 129 Sep 20 00:37 renderD129
crw-rw---- 1 nobody nogroup 226, 130 Sep 20 00:37 renderD130
crw-rw---- 1 nobody nogroup 226, 131 Sep 20 00:37 renderD131
crw-rw---- 1 nobody nogroup 226, 132 Sep 20 00:37 renderD132
crw-rw---- 1 nobody nogroup 226, 133 Sep 20 00:37 renderD133

ls -la /dev/dri inside the docker container reports:

crw-rw---- 1 nobody nogroup 226, 128 Sep 20 00:37 renderD128
crw-rw---- 1 nobody nogroup 226, 129 Sep 20 00:37 renderD129

which matches the two devices the container was started with...

The host has libvulkan1, mesa-vulkan-drivers, vulkan-tools, mesa-utils installed, as well as cuda-toolkit for docker... this host works fine with CUDA based llama.cpp container... ?!?

the host's uname -a :
Linux gpu-lxc-vllm-1 6.8.12-15-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-15 (2025-09-12T11:02Z) x86_64 GNU/Linux

Any suggestions are welcome.

[EnderHane]

I meet this problem and find a solution.

1. Create a Dockerfile with:

FROM ghcr.io/ggml-org/llama.cpp:server-vulkan

RUN apt-get update && apt-get install -y --no-install-recommends libglvnd0 libgl1 libglx0 libegl1 libgles2 && rm -rf /var/lib/apt/lists/*

(saved as llama.cpp-vulkan-nvidia.ContainerFile)

2. Build it:

podman build -t localhost/llama.cpp:server-vulkan -f llama.cpp-vulkan-nvidia.ContainerFile .

3. Run:

podman run -it --rm --gpus all localhost/llama.cpp:server-vulkan --list-devices

outputs

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 4060 Ti (NVIDIA) | (...)

Story:

When llama.cpp-vulkan (either built by myself or pulled from ghcr.io) couldn't find my 4060Ti, docker.io/nvidia/vulkan:1.3-470 can find the GPU. The solution is to install some of the packages present in nvidia/vulkan but absent from llama.cpp-vulkan.