WebAssembly and emscripten headers

[loretoparisi]

Hello I have tried a minimal Emscripten support to Makefile adding

# WASM
EMCXX = em++
EMCC = emcc
EMCXXFLAGS = --bind --std=c++11 -s WASM=1 -s ALLOW_MEMORY_GROWTH=1 -s "EXPORTED_RUNTIME_METHODS=['addOnPostRun','FS']" -s "DISABLE_EXCEPTION_CATCHING=0" -s "EXCEPTION_DEBUG=1" -s "FORCE_FILESYSTEM=1" -s "MODULARIZE=1" -s "EXPORT_ES6=0" -s 'EXPORT_NAME="LLAMAModule"' -s "USE_ES6_IMPORT_META=0" -I./
EMCCFLAGS = --bind -s WASM=1 -s ALLOW_MEMORY_GROWTH=1 -s "EXPORTED_RUNTIME_METHODS=['addOnPostRun','FS']" -s "DISABLE_EXCEPTION_CATCHING=0" -s "EXCEPTION_DEBUG=1" -s "FORCE_FILESYSTEM=1" -s "MODULARIZE=1" -s "EXPORT_ES6=0" -s 'EXPORT_NAME="LLAMAModule"' -s "USE_ES6_IMPORT_META=0" -I./ 

EMOBJS = utils.bc ggml.bc

wasm: llama_wasm.js quantize_wasm.js
wasmdebug: export EMCC_DEBUG=1
wasmdebug: llama_wasm.js quantize_wasm.js

#
# WASM lib
#

ggml.bc: ggml.c ggml.h
	$(EMCC) -c $(EMCCFLAGS) ggml.c -o ggml.bc
utils.bc: utils.cpp utils.h
	$(EMCXX) -c $(EMCXXFLAGS) utils.cpp -o utils.bc

$(info I EMOBJS:      $(EMOBJS))

#
# WASM executable
#
llama_wasm.js: $(EMOBJS) main.cpp Makefile
	$(EMCXX) $(EMCXXFLAGS) $(EMOBJS) -o llama_wasm.js
quantize_wasm.js: $(EMOBJS) quantize.cpp Makefile
	$(EMCXX) $(EMCXXFLAGS) $(EMOBJS) quantize.cpp -o quantize_wasm.js

It complies ok with both em++ and emcc. At this stage the problem is that main.cpp and quantize.cpp does not expose a proper header file, and I cannot call main as a module, or a function export using Emscripten EMSCRIPTEN_KEEPALIVE to main by example.

In fact a simple C++ headers could be compiled as a node module and then called like

/** file:llama.js */
const llamaModularized = require('./llama_wasm.js');
var llamaModule = null
const _initLLAMAModule = async function () {
    llamaModule = await llamaModularized();
    return true
}
let postRunFunc = null;
const addOnPostRun = function (func) {
    postRunFunc = func;
};
_initLLAMAModule().then((res) => {
    if (postRunFunc) {
        postRunFunc();
    }
});

class LLaMa {
    constructor() {
        this.f = new llamaModule.LLaMa();
    }
    // here modules fun impl
}

module.exports = { LLaMa, addOnPostRun };

and then executed in node scripts like

/** file:run.js */
(async () => {
    const LLaMa = require('./llama.js');
    const loadWASM = function () {
        var self = this;
        return new Promise(function (resolve, reject) {
            LLaMa.addOnPostRun(() => {
                let model = new LLaMa.LLaMa();
                /** use model functions */
            });
        });
    }//loadWASM
    await loadWASM();

}).call(this);

[MarkSchmidty]

Without https://github.com/WebAssembly/memory64 implemented in web assembly you are going to run into show stopping memory issues with the current 4GB limit due to 32bit addressing.

Do you have a plan for getting around that?

[Dicklesworthstone]

If you quantized the 7B model to a mixture of 3-bit and 4-bit quantization using https://github.com/qwopqwop200/GPTQ-for-LLaMa then you could stay within that memory envelope.

[MarkSchmidty]

I think that's a reasonable proposal @Dicklesworthstone.

A purely 3-bit implementation of llama.cpp using GPTQ could retain acceptable performance and solve the same memory issues. There's an open issue for implementing GPTQ quantization in 3-bit and 4-bit. GPTQ Quantization (3-bit and 4-bit) #9.

Other use cases could benefit from this same enhancement, such as getting 65B under 32GB and 30B under 16GB to further extend access to (perhaps slightly weaker versions of) the larger models.

[thypon]

https://twitter.com/nJoyneer/status/1637863946383155220

I was able to run llama.cpp in the browser with a minimal patchset and some *FLAGS

Following the Emscripten version used:

$ emcc --version
emcc (Emscripten gcc/clang-like replacement + linker emulating GNU ld) 3.1.33-git
Copyright (C) 2014 the Emscripten authors (see AUTHORS.txt)
This is free and open source software under the MIT license.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Following the compile flags:

make CC=emcc CXX=em++ LLAMA_NO_ACCELERATE=1 CFLAGS=" -DNDEBUG -s MEMORY64" CXXFLAGS=" -DNDEBUG -s MEMORY64" LDFLAGS="-s MEMORY64 -s TOTAL_MEMORY=8589934592 -s STACK_SIZE=2097152 --preload-file models " main.html

Following the minimal patch:

diff --git a/Makefile b/Makefile
index 1601079..12a1a80 100644
--- a/Makefile
+++ b/Makefile
@@ -189,12 +189,16 @@ utils.o: utils.cpp utils.h
 	$(CXX) $(CXXFLAGS) -c utils.cpp -o utils.o
 
 clean:
-	rm -f *.o main quantize
+	rm -f *.o main.{html,wasm,js,data,worker.js} main quantize
 
 main: main.cpp ggml.o utils.o
 	$(CXX) $(CXXFLAGS) main.cpp ggml.o utils.o -o main $(LDFLAGS)
 	./main -h
 
+main.html: main.cpp ggml.o utils.o
+	$(CXX) $(CXXFLAGS) main.cpp ggml.o utils.o -o main.html $(LDFLAGS)
+	go run server.go
+
 quantize: quantize.cpp ggml.o utils.o
 	$(CXX) $(CXXFLAGS) quantize.cpp ggml.o utils.o -o quantize $(LDFLAGS)
 
diff --git a/ggml.c b/ggml.c
index 4813f74..3dc2cbc 100644
--- a/ggml.c
+++ b/ggml.c
@@ -6,6 +6,8 @@
 #include <alloca.h>
 #endif
 
+#define _POSIX_C_SOURCE 200809L
+
 #include <assert.h>
 #include <time.h>
 #include <math.h>
@@ -107,7 +109,7 @@ typedef void* thread_ret_t;
     do { \
         if (!(x)) { \
             fprintf(stderr, "GGML_ASSERT: %s:%d: %s\n", __FILE__, __LINE__, #x); \
-            abort(); \
+            /*abort();*/ \
         } \
     } while (0)
 
diff --git a/main.cpp b/main.cpp
index e181056..afb0c53 100644
--- a/main.cpp
+++ b/main.cpp
@@ -785,7 +785,7 @@ int main(int argc, char ** argv) {
     const int64_t t_main_start_us = ggml_time_us();
 
     gpt_params params;
-    params.model = "models/llama-7B/ggml-model.bin";
+    params.model = "models/7B/ggml-model-q4_0.bin";
 
     if (gpt_params_parse(argc, argv, params) == false) {
         return 1;

[loretoparisi]

I was able to run llama.cpp in the browser with a minimal patchset and some *FLAGS

Following the Emscripten version used:

$ emcc --version
emcc (Emscripten gcc/clang-like replacement + linker emulating GNU ld) 3.1.33-git
Copyright (C) 2014 the Emscripten authors (see AUTHORS.txt)
This is free and open source software under the MIT license.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Following the compile flags:

make CC=emcc CXX=em++ LLAMA_NO_ACCELERATE=1 CFLAGS=" -DNDEBUG -s MEMORY64" CXXFLAGS=" -DNDEBUG -s MEMORY64" LDFLAGS="-s MEMORY64 -s TOTAL_MEMORY=8589934592 -s STACK_SIZE=2097152 --preload-file models " main.html

Following the minimal patch:

diff --git a/Makefile b/Makefile
index 1601079..12a1a80 100644
--- a/Makefile
+++ b/Makefile
@@ -189,12 +189,16 @@ utils.o: utils.cpp utils.h
 	$(CXX) $(CXXFLAGS) -c utils.cpp -o utils.o
 
 clean:
-	rm -f *.o main quantize
+	rm -f *.o main.{html,wasm,js,data,worker.js} main quantize
 
 main: main.cpp ggml.o utils.o
 	$(CXX) $(CXXFLAGS) main.cpp ggml.o utils.o -o main $(LDFLAGS)
 	./main -h
 
+main.html: main.cpp ggml.o utils.o
+	$(CXX) $(CXXFLAGS) main.cpp ggml.o utils.o -o main.html $(LDFLAGS)
+	go run server.go
+
 quantize: quantize.cpp ggml.o utils.o
 	$(CXX) $(CXXFLAGS) quantize.cpp ggml.o utils.o -o quantize $(LDFLAGS)
 
diff --git a/ggml.c b/ggml.c
index 4813f74..3dc2cbc 100644
--- a/ggml.c
+++ b/ggml.c
@@ -6,6 +6,8 @@
 #include <alloca.h>
 #endif
 
+#define _POSIX_C_SOURCE 200809L
+
 #include <assert.h>
 #include <time.h>
 #include <math.h>
@@ -107,7 +109,7 @@ typedef void* thread_ret_t;
     do { \
         if (!(x)) { \
             fprintf(stderr, "GGML_ASSERT: %s:%d: %s\n", __FILE__, __LINE__, #x); \
-            abort(); \
+            /*abort();*/ \
         } \
     } while (0)
 
diff --git a/main.cpp b/main.cpp
index e181056..afb0c53 100644
--- a/main.cpp
+++ b/main.cpp
@@ -785,7 +785,7 @@ int main(int argc, char ** argv) {
     const int64_t t_main_start_us = ggml_time_us();
 
     gpt_params params;
-    params.model = "models/llama-7B/ggml-model.bin";
+    params.model = "models/7B/ggml-model-q4_0.bin";
 
     if (gpt_params_parse(argc, argv, params) == false) {
         return 1;

Wow well done! Why did you had to remove abort(); from ggml?

[thypon]

The abort(); case was hit when out of memory, and before outputting the partial LLM output, so no string was shown.
Unfortunately, I did not find any way of using Memory64 WASM extension without incurring in some bugs on both Firefox and Chrome.

[loretoparisi]

The abort(); case was hit when out of memory, and before outputting the partial LLM output, so no string was shown. Unfortunately, I did not find any way of using Memory64 WASM extension without incurring in some bugs on both Firefox and Chrome.

So given the limit of WASM 64, you have to go for 3 and 4 bit quantization using GPTQ I think.

[thypon]

The abort(); case was hit when out of memory, and before outputting the partial LLM output, so no string was shown. Unfortunately, I did not find any way of using Memory64 WASM extension without incurring in some bugs on both Firefox and Chrome.

So given the limit of WASM 64, you have to go for 3 and 4 bit quantization using GPTQ I think.

It's already quantized 4bits when converting. 7b overflows 8GB of allocated WASM64 memory though.
Besides that it's quite slow since there is not both (pthread OR simd) AND memory64. Whenever I try to mix any of the two, the compiler or the linter fails to create or run the output.

[okpatil4u]

@thypon apparently memory64 is available in firefox nightly, did you check it ?

https://webassembly.org/roadmap/#feature-note-2

[lapo-luchini]

The new RedPajama-3B seems like a nice tiny model that could probably fit without memory64.

[IsaacRe]

@thypon @loretoparisi I'm curious, what sort of performance drop did you notice running in browser from running natively? How many toks/sec were you getting?

[thypon]

@IsaacRe Did not make a performance comparison since it was not 100% stable and needed to be refined. As mentioned it was single core since multithreaded + memory64 on firefox nightly was not working properly together, and crashing the experiment.

@okpatil4u already running with experimental memory64