Support for LLaMA models

[ye7iaserag]

Meta just released there LLaMA model family
https://github.com/facebookresearch/llama
Can we got support for that?
They calim that the 13B model is better than GPT 3 175B model

---

This is how to use the model in the web UI:

LLaMA TUTORIAL

-- oobabooga, March 9th, 2023

[oobabooga]

The models are not public yet, unfortunately. You have to request access.

[MetaIX]

Psst. Somebody leaked them https://twitter.com/Teknium1/status/1631322496388722689

[Sumanai]

Of course, the weights themselves are closed. But the code from the repository should be enough to add support. And where the end users will download the weights from is their problem.

[catboxanon]

meta-llama/llama#73

[oobabooga]

Done ea5c5eb

1. Install LLaMa as in their README:

conda activate textgen
git clone https://github.com/facebookresearch/llama
cd llama
pip install -r requirements.txt
pip install -e .

2. Put the model that you downloaded using your academic credentials on models/LLaMA-7B (the folder name must start with llama)

3. Put a copy of the files inside that folder too: tokenizer.model and tokenizer_checklist.chk

4. Start the web ui. I have tested with

python server.py --no-stream --model LLaMA-7B

[moorehousew]

Getting a CUDA out-of-memory error- I assume lowmem support isn't included yet?

[oobabooga]

This isn't part of Hugging Face yet, so it doesn't have access to 8bit and CPU offloading.

The 7B model uses 14963MiB VRAM on my machine. Reducing the max_seq_len parameter from 2048 to 512 makes this go down to 13843MiB.

[musicurgy]

I get a bunch of dependency errors when launching despite setting up LLaMa beforehand (definitely my own fault and probably because of a messed up conda environment)

ModuleNotFoundError: No module named 'fire'
ModuleNotFoundError: No module named 'fairscale'

etc. Any chance you could include these in the default webui requirements assuming they aren't too heavy?

[oobabooga]

@musicurgy did you try pip install -r requirements.txt as in #147 (comment)?

[musicurgy]

Yeah, after a bit of a struggle I ended up getting it working by just copying all the dependencies into the webui folder. So far the model is really interesting. Thanks for supporting it.

[generic-username0718]

Awesome stuff. I'm able to load LLaMA-7b but trying to load LLaMA-13b crashes with the error:

Traceback (most recent call last):
  File "/home/user/Documents/oobabooga/text-generation-webui/server.py", line 189, in <module>
    shared.model, shared.tokenizer = load_model(shared.model_name)
  File "/home/user/Documents/oobabooga/text-generation-webui/modules/models.py", line 94, in load_model
    model = LLaMAModel.from_pretrained(Path(f'models/{model_name}'))
  File "/home/user/Documents/oobabooga/text-generation-webui/modules/LLaMA.py", line 82, in from_pretrained
    generator = load(
  File "/home/user/Documents/oobabooga/text-generation-webui/modules/LLaMA.py", line 44, in load
    assert world_size == len(
AssertionError: Loading a checkpoint for MP=2 but world size is 1

[generic-username0718]

Anyone reading this you can get past the issue above by changing the world_size variable found in modules/LLaMA.py like this:

def setup_model_parallel() -> Tuple[int, int]:
local_rank = int(os.environ.get("LOCAL_RANK", -1))
world_size = 2

My issue now is I'm running out of VRAM. I'm running dual 3090s and should be able to load the model if it's split among the cards...

[generic-username0718]

Is there a parameter I need to pass to oobabooga to tell it to split the model among my two 3090 gpus?

[Morb0]

Is there a parameter I need to pass to oobabooga to tell it to split the model among my two 3090 gpus?

Try --gpu-memory 10 5, at least that's what the README says.