llama : improve BPE pre-processing + LLaMA 3 and Deepseek support#6920

Merged

ggerganov merged 61 commits intomasterfrom

gg/bpe-preprocess

Apr 29, 2024

Commits on Apr 26, 2024

merged the changes from deepseeker models to main branch

jaggzh
authored and
ggerganov
committed
Moved regex patterns to unicode.cpp and updated unicode.h

dragnil1
authored and
ggerganov
committed
Moved header files

dragnil1
authored and
ggerganov
committed
Resolved issues

dragnil1
authored and
ggerganov
committed
added and refactored unicode_regex_split and related functions

dragnil1
authored and
ggerganov
committed
Updated/merged the deepseek coder pr

jaggzh
authored and
ggerganov
committed
Refactored code

dragnil1
authored and
ggerganov
committed
Adding unicode regex mappings

dragnil1
authored and
ggerganov
committed
Adding unicode regex function

dragnil1
authored and
ggerganov
committed
Added needed functionality, testing remains

dragnil1
authored and
ggerganov
committed
Fixed issues

dragnil1
authored and
ggerganov
committed
Fixed issue with gpt2 regex custom preprocessor

dragnil1
authored and
ggerganov
committed
unicode : fix? unicode_wstring_to_utf8
ggerganov
committed
lint : fix whitespaces
ggerganov
committed
tests : add tokenizer tests for numbers
ggerganov
committed
unicode : remove redundant headers
ggerganov
committed
tests : remove and rename tokenizer test scripts
ggerganov
committed
tests : add sample usage
ggerganov
committed
gguf-py : reader prints warnings on duplicate keys
ggerganov
committed
llama : towards llama3 tokenization support (wip)
ggerganov
committed
unicode : shot in the dark to fix tests on Windows
ggerganov
committed
unicode : first try custom implementations
ggerganov
committed
Merge branch 'master' into gg/bpe-preprocess
ggerganov
committed
convert : add "tokenizer.ggml.pre" GGUF KV (wip)
ggerganov
committed
llama : use new pre-tokenizer type
ggerganov
committed
convert : fix pre-tokenizer type writing
ggerganov
committed
lint : fix
ggerganov
committed
make : add test-tokenizer-0-llama-v3
ggerganov
committed
wip
ggerganov
committed

Commits on Apr 28, 2024

Commits on Apr 29, 2024