Cuda Support #290
Description
Issue description
Build fails when trying to use it with cuda
Expected Behavior
npx --no node-llama-cpp download --release latest --cuda should build the tool successfully.
Actual Behavior
npx --no node-llama-cpp download --release latest --cuda
Repo: ggerganov/llama.cpp
Release: latest
CUDA: enabled
✔ Fetched llama.cpp info
✔ Removed existing llama.cpp directory
Cloning llama.cpp
Clone ggerganov/llama.cpp (GitHub) 100% ████████████████████████████████████████ 0s
◷ Compiling llama.cpp
Not searching for unused variables given on the command line.
-- Selecting Windows SDK version 10.0.22621.0 to target Windows 6.2.9200.
-- The C compiler identification is MSVC 19.40.33812.0
-- The CXX compiler identification is MSVC 19.40.33812.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.40.33807/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.40.33807/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Git: C:/Users/ShakeebAftab/AppData/Local/Programs/Git/cmd/git.exe (found version "2.45.2.windows.1")
CMake Error at llama.cpp/CMakeLists.txt:95 (message):
LLAMA_CUBLAS is deprecated and will be removed in the future.
Use GGML_CUDA instead
Call Stack (most recent call first):
llama.cpp/CMakeLists.txt:100 (llama_option_depr)
-- Configuring incomplete, errors occurred!
Not searching for unused variables given on the command line.
-- Selecting Windows SDK version 10.0.22621.0 to target Windows 6.2.9200.
-- The C compiler identification is MSVC 19.40.33812.0
-- The CXX compiler identification is MSVC 19.40.33812.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.40.33807/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.40.33807/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Git: C:/Users/ShakeebAftab/AppData/Local/Programs/Git/cmd/git.exe (found version "2.45.2.windows.1")
CMake Error at llama.cpp/CMakeLists.txt:95 (message):
LLAMA_CUBLAS is deprecated and will be removed in the future.
Use GGML_CUDA instead
Call Stack (most recent call first):
llama.cpp/CMakeLists.txt:100 (llama_option_depr)
-- Configuring incomplete, errors occurred!
ERR! OMG Process terminated: 1
[node-llama-cpp] To resolve errors related to CUDA compilation, see the CUDA guide: https://withcatai.github.io/node-llama-cpp/guide/CUDA
✖ Failed to compile llama.cpp
node-llama-cpp download
Download a release of llama.cpp and compile it
Options:
-h, --help Show help [boolean]
--repo The GitHub repository to download a release of llama.cpp from. Can also be
set via the NODE_LLAMA_CPP_REPO environment variable
[string] [default: "ggerganov/llama.cpp"]
--release The tag of the llama.cpp release to download. Set to "latest" to download t
he latest release. Can also be set via the NODE_LLAMA_CPP_REPO_RELEASE envi
ronment variable [string] [default: "b3465"]
-a, --arch The architecture to compile llama.cpp for [string]
-t, --nodeTarget The Node.js version to compile llama.cpp for. Example: v18.0.0 [string]
--cuda Compile llama.cpp with CUDA support. Can also be set via the NODE_LLAMA_CPP
_CUDA environment variable [boolean] [default: false]
--skipBuild, --sb Skip building llama.cpp after downloading it [boolean] [default: false]
--noBundle, --nb Download a llama.cpp release only from GitHub, even if a local git bundle e
xists for the release [boolean] [default: false]
-v, --version Show version number [boolean]
Error: Command npm run -s cmake-js-llama -- compile --log-level warn --arch=x64 --runtime-version=20.15.1 --cmake-path C:\Users\ShakeebAftab\Videos\custom_classification\node_modules\node-llama-cpp\llama\xpack\store\@xpack-dev-tools\cmake\3.27.9-1.2\.content\bin\cmake.exe --CDLLAMA_METAL=OFF --CDLLAMA_CUBLAS=1 exited with code 1
at ChildProcess.<anonymous> (file:///C:/Users/ShakeebAftab/Videos/custom_classification/node_modules/node-llama-cpp/dist/utils/spawnCommand.js:27:24)
at ChildProcess.emit (node:events:519:28)
at cp.emit (C:\Users\ShakeebAftab\Videos\custom_classification\node_modules\cross-spawn\lib\enoent.js:34:29)
at ChildProcess._handle.onexit (node:internal/child_process:294:12)Steps to reproduce
Run the following command after installing the node-llama-cpp package
npx --no node-llama-cpp download --release latest --cuda
My Environment
| Dependency | Version |
|---|---|
| Operating System | Windows 11 |
| CPU | Intel Ultra i9 |
| Node.js version | 20.15.1 |
| Typescript version | 5.5.4 |
node-llama-cpp version |
2.8.14 |
Additional Context
Everything works fine without CUDA. I suspect that LLAMA_CUBLAS should be changed to GGML_CUDA in the compileLLamaCPP.ts file and the issue will be fixed.
Relevant Features Used
- Metal support
- CUDA support
- Grammar
Are you willing to resolve this issue by submitting a Pull Request?
Yes, I have the time, but I don't know how to start. I would need guidance.