ggml_new_tensor_impl: not enough space in the context's memory pool

[Ph0rk0z]

I'm using textgen with the llama.cpp python bindings. Built fresh from source today.

Loading camel-platypus2-70b. q4-k-m
Inference goes normally for small context. Speed is about the same as exllama_hf.

Unfortunately when using a prompt of 1848 on the 4096 model, I am presented with this error and a segfault.

System is 2x3090, 1xP40, xeon-v4 16 core with 256g of ram.

I have done tensor split of 40/40, correctly set GQA, and offloaded all layers to GPU. I have tried to use the new kernel and mlock or not. Batch size is set to the max but lowering has not made much difference.

About all I can do is turn off mmap or split across all 3 gpu but I think this might be a bug. Same inference is no problem for GPTQ.

[Ph0rk0z]

Seems like it goes away if n_batch is 512, cannot go any higher.

[klosax]

Seems like it goes away if n_batch is 512, cannot go any higher.

I think this will be solved when PR #2684 is merged.

[Ph0rk0z]

I can now do 768 but higher still fails. Perhaps I am truly out of memory?

[slaren]

Something that is not very well documented is that llama_eval cannot be called with more tokens than the value of n_batch used to initialize the context. If you do that, then out of memory errors are expected. Similarly, n_past + n_tokens cannot exceed n_ctx.

We should probably change llama_eval to automatically split the evaluation into multiple batches when needed, or at the very least add checks to tell the user what they are doing wrong.

[Ph0rk0z]

I was hitting regenerate on 2k/4k context but I did re-load the model. This is through the python bindings so not sure exactly what's going on in the background. Wanted to see if larger batches will complete faster.

[Ph0rk0z]

Ok.. I figured it out. I have to kill llama.cpp completely for the batches to work. But I don't see much difference for GPU.