Why max 64 parallel requests?

[Nico3012]

Hey,

why does llama.cpp only allow 64 parallel requests?
If i set --parallel to e.g. 256, i get the error:

"llama_init_from_model: failed to initialize the context: n_seq_max must be <= 64"

What is the reason for this limit?

[adhusch]

This was actually changed in 4f81b33#commitcomment-159972791 :)

[Nico3012]

Thank you!
For me, it makes no sense to have this upper limit. Do you know why we need it? And is it possible to change LLAMA_MAX_PARALLEL_SEQUENCES e.g., with an environment variable?

Would this bring any downside (apart from the slower generation speed and context split) to allow many more parallel sequences?

I'm using the prebuilt Docker container llama-server