[Usage]: `Phi-4-multimodal-instruct` activate LoRA module but get mangled text output

[Luffy966]

Your current environment

The output of `python collect_env.py`

INFO 03-24 14:46:21 [__init__.py:256] Automatically detected platform cuda.
Collecting environment information...
PyTorch version: 2.6.0+cu118
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.6 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
Clang version: Could not collect
CMake version: version 3.23.0
Libc version: glibc-2.31

Python version: 3.12.9 | packaged by Anaconda, Inc. | (main, Feb  6 2025, 18:56:27) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.4.0-162-generic-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA GeForce RTX 4090
GPU 1: NVIDIA GeForce RTX 4090
GPU 2: NVIDIA GeForce RTX 4090
GPU 3: NVIDIA GeForce RTX 4090
GPU 4: NVIDIA GeForce RTX 4090
GPU 5: NVIDIA GeForce RTX 4090
GPU 6: NVIDIA GeForce RTX 4090
GPU 7: NVIDIA GeForce RTX 4090

Nvidia driver version: 535.104.05
cuDNN version: Probably one of the following:
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcudnn.so.8.4.0
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.4.0
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.4.0
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.4.0
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.4.0
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.4.0
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.4.0
/usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudnn.so.8.9.0
/usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.9.0
/usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.9.0
/usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.9.0
/usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.9.0
/usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.9.0
/usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.9.0
/usr/local/cuda-12.0/targets/x86_64-linux/lib/libcudnn.so.8.8.0
/usr/local/cuda-12.0/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.8.0
/usr/local/cuda-12.0/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.8.0
/usr/local/cuda-12.0/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.8.0
/usr/local/cuda-12.0/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.8.0
/usr/local/cuda-12.0/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.8.0
/usr/local/cuda-12.0/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.8.0
/usr/local/cuda-12.2/targets/x86_64-linux/lib/libcudnn.so.8.9.2
/usr/local/cuda-12.2/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.9.2
/usr/local/cuda-12.2/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.9.2
/usr/local/cuda-12.2/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.9.2
/usr/local/cuda-12.2/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.9.2
/usr/local/cuda-12.2/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.9.2
/usr/local/cuda-12.2/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.9.2
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Byte Order:                         Little Endian
Address sizes:                      52 bits physical, 57 bits virtual
CPU(s):                             128
On-line CPU(s) list:                0-127
Thread(s) per core:                 2
Core(s) per socket:                 32
Socket(s):                          2
NUMA node(s):                       2
Vendor ID:                          GenuineIntel
CPU family:                         6
Model:                              106
Model name:                         Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz
Stepping:                           6
Frequency boost:                    enabled
CPU MHz:                            1354.195
CPU max MHz:                        2001.0000
CPU min MHz:                        800.0000
BogoMIPS:                           4000.00
Virtualization:                     VT-x
L1d cache:                          3 MiB
L1i cache:                          2 MiB
L2 cache:                           80 MiB
L3 cache:                           96 MiB
NUMA node0 CPU(s):                  0-31,64-95
NUMA node1 CPU(s):                  32-63,96-127
Vulnerability Gather data sampling: Mitigation; Microcode
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Retbleed:             Not affected
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 invpcid_single ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local wbnoinvd dtherm ida arat pln pts avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq rdpid md_clear pconfig flush_l1d arch_capabilities

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu11==11.11.3.6
[pip3] nvidia-cuda-cupti-cu11==11.8.87
[pip3] nvidia-cuda-nvrtc-cu11==11.8.89
[pip3] nvidia-cuda-runtime-cu11==11.8.89
[pip3] nvidia-cudnn-cu11==9.1.0.70
[pip3] nvidia-cufft-cu11==10.9.0.58
[pip3] nvidia-curand-cu11==10.3.0.86
[pip3] nvidia-cusolver-cu11==11.4.1.48
[pip3] nvidia-cusparse-cu11==11.7.5.86
[pip3] nvidia-nccl-cu11==2.21.5
[pip3] nvidia-nvtx-cu11==11.8.86
[pip3] pyzmq==26.3.0
[pip3] torch==2.6.0+cu118
[pip3] torchaudio==2.6.0+cu118
[pip3] torchvision==0.21.0+cu118
[pip3] transformers==4.50.0
[pip3] triton==3.2.0
[conda] numpy                     1.26.4                   pypi_0    pypi
[conda] nvidia-cublas-cu11        11.11.3.6                pypi_0    pypi
[conda] nvidia-cuda-cupti-cu11    11.8.87                  pypi_0    pypi
[conda] nvidia-cuda-nvrtc-cu11    11.8.89                  pypi_0    pypi
[conda] nvidia-cuda-runtime-cu11  11.8.89                  pypi_0    pypi
[conda] nvidia-cudnn-cu11         9.1.0.70                 pypi_0    pypi
[conda] nvidia-cufft-cu11         10.9.0.58                pypi_0    pypi
[conda] nvidia-curand-cu11        10.3.0.86                pypi_0    pypi
[conda] nvidia-cusolver-cu11      11.4.1.48                pypi_0    pypi
[conda] nvidia-cusparse-cu11      11.7.5.86                pypi_0    pypi
[conda] nvidia-nccl-cu11          2.21.5                   pypi_0    pypi
[conda] nvidia-nvtx-cu11          11.8.86                  pypi_0    pypi
[conda] pyzmq                     26.3.0                   pypi_0    pypi
[conda] torch                     2.6.0+cu118              pypi_0    pypi
[conda] torchaudio                2.6.0+cu118              pypi_0    pypi
[conda] torchvision               0.21.0+cu118             pypi_0    pypi
[conda] transformers              4.50.0                   pypi_0    pypi
[conda] triton                    3.2.0                    pypi_0    pypi
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.8.1
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
�[4mGPU0	GPU1	GPU2	GPU3	GPU4	GPU5	GPU6	GPU7	NIC0	NIC1	CPU Affinity	NUMA Affinity	GPU NUMA ID�[0m
GPU0	 X 	PIX	PXB	PXB	SYS	SYS	SYS	SYS	PXB	PXB	0-31,64-95	0		N/A
GPU1	PIX	 X 	PXB	PXB	SYS	SYS	SYS	SYS	PXB	PXB	0-31,64-95	0		N/A
GPU2	PXB	PXB	 X 	PIX	SYS	SYS	SYS	SYS	PXB	PXB	0-31,64-95	0		N/A
GPU3	PXB	PXB	PIX	 X 	SYS	SYS	SYS	SYS	PXB	PXB	0-31,64-95	0		N/A
GPU4	SYS	SYS	SYS	SYS	 X 	PXB	PXB	PXB	SYS	SYS	32-63,96-127	1		N/A
GPU5	SYS	SYS	SYS	SYS	PXB	 X 	PXB	PXB	SYS	SYS	32-63,96-127	1		N/A
GPU6	SYS	SYS	SYS	SYS	PXB	PXB	 X 	PIX	SYS	SYS	32-63,96-127	1		N/A
GPU7	SYS	SYS	SYS	SYS	PXB	PXB	PIX	 X 	SYS	SYS	32-63,96-127	1		N/A
NIC0	PXB	PXB	PXB	PXB	SYS	SYS	SYS	SYS	 X 	PIX				
NIC1	PXB	PXB	PXB	PXB	SYS	SYS	SYS	SYS	PIX	 X 				

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1

NCCL_CUMEM_ENABLE=0
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

How would you like to use vllm

🐛 Describe the bug

I deployed Phi-4-multimodal-instruct by using the this cmd and activating its audio function only:

NCCL_CUMEM_ENABLE=0 NCCL_DEBUG=TRACE CUDA_LAUNCH_BLOCKING=1 CUDA_VISIBLE_DEVICES=1,5 vllm serve /my/path/to/models/Huggingface_download/Phi-4-multimodal-instruct --task generate --trust-remote-code --limit-mm-per-prompt audio=10 --tensor-parallel-size 2 --gpu-memory-utilization 0.99 --port 8007 --max_num_seqs 1 --enable_lora --max_lora_rank 320 --lora-modules speech=/my/path/to/models/Huggingface_download/Phi-4-multimodal-instruct/speech-lora

Then I prompt it in Chinese with a "audio+text -> text" task. And I found out the output of model is all mangled text.

But when I disabled lora and redeployed it:

NCCL_CUMEM_ENABLE=0 NCCL_DEBUG=TRACE CUDA_LAUNCH_BLOCKING=1 CUDA_VISIBLE_DEVICES=1,5 vllm serve /my/path/to/models/Huggingface_download/Phi-4-multimodal-instruct --task generate --trust-remote-code --limit-mm-per-prompt audio=10 --tensor-parallel-size 2 --gpu-memory-utilization 0.99 --port 8007 --max_num_seqs 1

I found everything ran well and it correctly answered me in Chinese, and it's even faster.

Is it correct not loading the lora modules or loading only the speech one when using Phi-4-multimodal-instruct? How should I load lora modules if they're necessary?

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[jeejeelee]

try appending --enable-lora

[Luffy966]

Hi, @jeejeelee
Thanks for replying. But I've already appended --enable-lora in the first cmd line.

[Luffy966]

The official cmd given by Microsoft Team is like:

python -m vllm.entrypoints.openai.api_server 
  --model 'microsoft/Phi-4-multimodal-instruct' 
  --dtype auto 
  --trust-remote-code 
  --max-model-len 131072 
  --enable-lora 
  --max-lora-rank 320 
  --lora-extra-vocab-size 0 
  --limit-mm-per-prompt audio=3,image=3 
  --max-loras 2 
  --lora-modules speech=<path to speech lora folder> vision=<path to vision lora folder>

When I activated LoRA with my cmd mentioned above:

NCCL_CUMEM_ENABLE=0 \
NCCL_DEBUG=TRACE \
CUDA_LAUNCH_BLOCKING=1 \
CUDA_VISIBLE_DEVICES=1,5 \
vllm serve /my/path/to/models/Huggingface_download/Phi-4-multimodal-instruct 
  --task generate 
  --trust-remote-code 
  --limit-mm-per-prompt audio=10 
  --tensor-parallel-size 2 
  --gpu-memory-utilization 0.99 
  --port 8007 
  --max_num_seqs 1 
  --enable_lora 
  --max_lora_rank 320 
  --lora-modules speech=/my/path/to/models/Huggingface_download/Phi-4-multimodal-instruct/speech-lora

The differences regarding lora module between this 2 cmd are only --lora-extra-vocab-size 0, --max-loras 2 and vision=<path to vision lora folder>.

[Luffy966]

And when I add --lora-extra-vocab-size 0, I come across this error:

Traceback (most recent call last):
  File ".../anaconda3/envs/vllm/bin/vllm", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File ".../anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/entrypoints/cli/main.py", line 75, in main
    args.dispatch_function(args)
  File ".../anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/entrypoints/cli/serve.py", line 33, in cmd
    uvloop.run(run_server(args))
  File ".../anaconda3/envs/vllm/lib/python3.12/site-packages/uvloop/__init__.py", line 109, in run
    return __asyncio.run(
           ^^^^^^^^^^^^^^
  File ".../anaconda3/envs/vllm/lib/python3.12/asyncio/runners.py", line 195, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File ".../anaconda3/envs/vllm/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
  File ".../anaconda3/envs/vllm/lib/python3.12/site-packages/uvloop/__init__.py", line 61, in wrapper
    return await main
           ^^^^^^^^^^
  File ".../anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 1012, in run_server
    async with build_async_engine_client(args) as engine_client:
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../anaconda3/envs/vllm/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File ".../anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 141, in build_async_engine_client
    async with build_async_engine_client_from_engine_args(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../anaconda3/envs/vllm/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File ".../anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 161, in build_async_engine_client_from_engine_args
    vllm_config = engine_args.create_engine_config(usage_context=usage_context)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/engine/arg_utils.py", line 1332, in create_engine_config
    lora_config = LoRAConfig(
                  ^^^^^^^^^^^
  File "<string>", line 11, in __init__
  File ".../anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/config.py", line 2333, in __post_init__
    raise ValueError(
ValueError: lora_extra_vocab_size (0) must be one of (256, 512).

[Luffy966]

BTW, when I loaded with lora modules, I encountered various warnings like: (too long didn't post it all here)

WARNING 03-25 06:07:41 [models.py:478] Regarding multimodal models, vLLM currently only supports adding LoRA to language model, vision_encoder.img_processor.encoder.layers.23.self_attn.qkv_proj will be ignored.                          
WARNING 03-25 06:07:41 [models.py:478] Regarding multimodal models, vLLM currently only supports adding LoRA to language model, vision_encoder.img_processor.encoder.layers.23.self_attn.out_proj will be ignored.                          
WARNING 03-25 06:07:41 [models.py:478] Regarding multimodal models, vLLM currently only supports adding LoRA to language model, vision_encoder.img_processor.encoder.layers.23.mlp.fc1 will be ignored.                                     
WARNING 03-25 06:07:41 [models.py:478] Regarding multimodal models, vLLM currently only supports adding LoRA to language model, vision_encoder.img_processor.encoder.layers.23.mlp.fc2 will be ignored.                                     
WARNING 03-25 06:07:41 [models.py:478] Regarding multimodal models, vLLM currently only supports adding LoRA to language model, vision_encoder.img_processor.encoder.layers.24.self_attn.qkv_proj will be ignored.                          
WARNING 03-25 06:07:41 [models.py:478] Regarding multimodal models, vLLM currently only supports adding LoRA to language model, vision_encoder.img_processor.encoder.layers.24.self_attn.out_proj will be ignored.                          
WARNING 03-25 06:07:41 [models.py:478] Regarding multimodal models, vLLM currently only supports adding LoRA to language model, vision_encoder.img_processor.encoder.layers.24.mlp.fc1 will be ignored.                                     
WARNING 03-25 06:07:41 [models.py:478] Regarding multimodal models, vLLM currently only supports adding LoRA to language model, vision_encoder.img_processor.encoder.layers.24.mlp.fc2 will be ignored.                                     
WARNING 03-25 06:07:41 [models.py:478] Regarding multimodal models, vLLM currently only supports adding LoRA to language model, vision_encoder.img_processor.encoder.layers.25.self_attn.qkv_proj will be ignored.                          
WARNING 03-25 06:07:41 [models.py:478] Regarding multimodal models, vLLM currently only supports adding LoRA to language model, vision_encoder.img_processor.encoder.layers.25.self_attn.out_proj will be ignored.                          
WARNING 03-25 06:07:41 [models.py:478] Regarding multimodal models, vLLM currently only supports adding LoRA to language model, vision_encoder.img_processor.encoder.layers.25.mlp.fc1 will be ignored.                                     
WARNING 03-25 06:07:41 [models.py:478] Regarding multimodal models, vLLM currently only supports adding LoRA to language model, vision_encoder.img_processor.encoder.layers.25.mlp.fc2 will be ignored.

Note that I've already been using vllm=0.8.1

[Luffy966]

Oh, I have some extra flag constraints on output in my prompt like <speech_prompt>...</speech_prompt>...<content>...</content>. Would that be a potential reason for this mangled text error? But it works well without LoRA modules. Do you have any ideas about it? @jeejeelee
And I'm also wondering if it's correct not loading the LoRA modules when using Phi-4-multimodal-instruct, since it seems to work well without LoRA modules. (I just read the report paper of Phi-4-multimodal-instruct, and it seems to have to load LoRA modules when using its multimodal features.)

[jeejeelee]

And when I add --lora-extra-vocab-size 0, I come across this error:

Traceback (most recent call last):
  File ".../anaconda3/envs/vllm/bin/vllm", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File ".../anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/entrypoints/cli/main.py", line 75, in main
    args.dispatch_function(args)
  File ".../anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/entrypoints/cli/serve.py", line 33, in cmd
    uvloop.run(run_server(args))
  File ".../anaconda3/envs/vllm/lib/python3.12/site-packages/uvloop/__init__.py", line 109, in run
    return __asyncio.run(
           ^^^^^^^^^^^^^^
  File ".../anaconda3/envs/vllm/lib/python3.12/asyncio/runners.py", line 195, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File ".../anaconda3/envs/vllm/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
  File ".../anaconda3/envs/vllm/lib/python3.12/site-packages/uvloop/__init__.py", line 61, in wrapper
    return await main
           ^^^^^^^^^^
  File ".../anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 1012, in run_server
    async with build_async_engine_client(args) as engine_client:
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../anaconda3/envs/vllm/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File ".../anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 141, in build_async_engine_client
    async with build_async_engine_client_from_engine_args(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../anaconda3/envs/vllm/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File ".../anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 161, in build_async_engine_client_from_engine_args
    vllm_config = engine_args.create_engine_config(usage_context=usage_context)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/engine/arg_utils.py", line 1332, in create_engine_config
    lora_config = LoRAConfig(
                  ^^^^^^^^^^^
  File "<string>", line 11, in __init__
  File ".../anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/config.py", line 2333, in __post_init__
    raise ValueError(
ValueError: lora_extra_vocab_size (0) must be one of (256, 512).

This is a recent update. You don't need to set --lora-extra-vocab-size The LoRA for Phi4-MM doesn't require this arg

[Luffy966]

I just tried replacing my extra flag constraints <speech_prompt>...</speech_prompt>...<content>...</content> with pure text prompt to see if it's the problem with that. But I still got mangled text output!

An example of mangled text output

语音提示：有些温和的讽刺语调，_EMPATHY tone_，含蓄地\trmoking_bell_说出自己对父母的感激之情 ICommand: 语音提示 - EMPATHY tone, subtle and respectful\n\n回复内容：确实，我的াঁও버是非常支持的，我 chéad的支持 ältere me, and thanks for the encouragement. 你爸一直在为我提供资源和 Qualifications, I appreciate that. 但我也需要有自己的 ✔️space to grow and explore my passions, okay? Versicherung, okay? I value the support, but I also need space for my own development.rather than constantly chiding me for my choices. 你爸的关心是很重要的, but I also want to be responsible for my own decisions, okay? 你爸的关心很重要, but I also want to take responsibility for my own decisions. Thanks for understanding. 你爸的关心是很重要的, but I also want to take responsibility for my own decisions, okay? 你爸的关心很重要, but I also want to take responsibility for my own decisions, okay? 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to make good choices on my own. 了解了, I appreciate your support, and I will strive to. and and. and to and and and and to. and to and and and and to and to. to and and and and. ... ... and.... and and and and.. and and and and. and and and and and.. to.. and and. and and and and.. I. and and to. . to. and and and to. to.. and and.. to. and and. and and. and.. and and and and and... and and and and to.,,..... and and to. and.. and.... and to and and.. and and and and and and. and.... and and and and and and...... I..... and and to. and and. .. and. ... ... and and and and and and... and and. to. to and and and and and..\n\n and and and... and and and. ....... and and and and.... I. and and and and to to... and. and and and and and. and and.... and and... and and.. and,.... and and and and.\n\n... and and and and and. and.. to... and. and to. and.. to and and and and........ and..... and and and and,.... and... and.... and and.. and and...... and. and to and and and.... and and and. and.. and and..,..\n\n.. and and. and and and and and and..... and.. .. in...... to. to....\n\n. and and.. to. and.,.,...\n\n........ and. and and and and and and and.. ..... to and to... on.. .. and. . , and and and..... and and to. to.... and and. ... ..... I..\n... ... and and and.... and and and and and and.\n\n......... and and and, and. and and................. . I.... and and and and. to.. to, to. and. and and and and and and and and..............................\n\n..\n\n... and and and and and.....\n.. to... and and and and and......

So personally I strongly suspect the WARNING here to be the main fault of this error instead of my extra flag constraints.

BTW, when I loaded with lora modules, I encountered various warnings like: (too long didn't post it all here)

WARNING 03-25 06:07:41 [models.py:478] Regarding multimodal models, vLLM currently only supports adding LoRA to language model, vision_encoder.img_processor.encoder.layers.23.self_attn.qkv_proj will be ignored.                          
WARNING 03-25 06:07:41 [models.py:478] Regarding multimodal models, vLLM currently only supports adding LoRA to language model, vision_encoder.img_processor.encoder.layers.23.self_attn.out_proj will be ignored.                          
WARNING 03-25 06:07:41 [models.py:478] Regarding multimodal models, vLLM currently only supports adding LoRA to language model, vision_encoder.img_processor.encoder.layers.23.mlp.fc1 will be ignored.                                     
WARNING 03-25 06:07:41 [models.py:478] Regarding multimodal models, vLLM currently only supports adding LoRA to language model, vision_encoder.img_processor.encoder.layers.23.mlp.fc2 will be ignored.                                     
WARNING 03-25 06:07:41 [models.py:478] Regarding multimodal models, vLLM currently only supports adding LoRA to language model, vision_encoder.img_processor.encoder.layers.24.self_attn.qkv_proj will be ignored.                          
WARNING 03-25 06:07:41 [models.py:478] Regarding multimodal models, vLLM currently only supports adding LoRA to language model, vision_encoder.img_processor.encoder.layers.24.self_attn.out_proj will be ignored.                          
WARNING 03-25 06:07:41 [models.py:478] Regarding multimodal models, vLLM currently only supports adding LoRA to language model, vision_encoder.img_processor.encoder.layers.24.mlp.fc1 will be ignored.                                     
WARNING 03-25 06:07:41 [models.py:478] Regarding multimodal models, vLLM currently only supports adding LoRA to language model, vision_encoder.img_processor.encoder.layers.24.mlp.fc2 will be ignored.                                     
WARNING 03-25 06:07:41 [models.py:478] Regarding multimodal models, vLLM currently only supports adding LoRA to language model, vision_encoder.img_processor.encoder.layers.25.self_attn.qkv_proj will be ignored.                          
WARNING 03-25 06:07:41 [models.py:478] Regarding multimodal models, vLLM currently only supports adding LoRA to language model, vision_encoder.img_processor.encoder.layers.25.self_attn.out_proj will be ignored.                          
WARNING 03-25 06:07:41 [models.py:478] Regarding multimodal models, vLLM currently only supports adding LoRA to language model, vision_encoder.img_processor.encoder.layers.25.mlp.fc1 will be ignored.                                     
WARNING 03-25 06:07:41 [models.py:478] Regarding multimodal models, vLLM currently only supports adding LoRA to language model, vision_encoder.img_processor.encoder.layers.25.mlp.fc2 will be ignored.

Note that I've already been using vllm=0.8.1

[jeejeelee]

IIRC, Phi-4MM only adds LoRA for the language model, so this is not the reason.

[Luffy966]

IIRC, Phi-4MM only adds LoRA for the language model, so this is not the reason.

Oh yes, you're right! And the WARNING appears only in encoders.

So maybe it's because the LoRA modules themselves lack full support for Chinese?

[jeejeelee]

IIRC, Phi-4MM only adds LoRA for the language model, so this is not the reason.

Oh yes, you're right! And the WARNING appears only in encoders.

So maybe it's because the LoRA modules themselves lack full support for Chinese?

I 'm not entirely sure. Perhaps you could compare it with the output results from transformers.

[Luffy966]

Not sure if I implemented Phi-4MM correctly with speech LoRA modules by transformers, but I didn't get mangled text output results. And I think I've followed their transformers implementation as much as possible.

My implementation code is like this

import os
os.environ['CUDA_VISIBLE_DEVICES'] = '1,5'

import torch
import soundfile
from transformers import AutoModelForCausalLM, AutoProcessor, GenerationConfig

model_path = './models/Huggingface_download/Phi-4-multimodal-instruct'

kwargs = {}
kwargs['torch_dtype'] = torch.bfloat16

processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)
print(processor.tokenizer)

model = AutoModelForCausalLM.from_pretrained(
    model_path,
    device_map="auto",
    trust_remote_code=True,
    torch_dtype='auto',
    _attn_implementation='flash_attention_2',
) # .cuda(4)
print("model.config._attn_implementation:", model.config._attn_implementation)

generation_config = GenerationConfig.from_pretrained(model_path, 'generation_config.json')

user_prompt = '<|user|>'
assistant_prompt = '<|assistant|>'
prompt_suffix = '<|end|>'
placeholder_audio = "<|audio_{i}|>"

# sys prompt
ACHAT_SYS_PROFILE_PROMPT = """对话中每轮我都会输入一句话和对应的一段音频，你需要给出符合要求的回复。回复中要包含语音提示和回复内容，并分别包含在<speech_prompt> </speech_prompt>和<content> </content>之间。例如`<speech_prompt>语音提示在这</speech_prompt>\n<content>回复内容在这</content>`。其中你所生成的语音提示和回复内容都将会指导后续TTS模型合成语音，因此注意不要生成与语音提示无关的信息。

现在对话开始。
"""
msgs = [
    {
        "role": "system", 
        "content": ACHAT_SYS_PROFILE_PROMPT,
    },
    {'role': 'user', 'content': f'老婆你干什么，你不能喝你少喝点\n<|audio_1|>'},
]
prompt = processor.tokenizer.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
# need to remove last <|endoftext|> if it is there, which is used for training, not inference. For training, make sure to add <|endoftext|> in the end.
if prompt.endswith('<|endoftext|>'):
    prompt = prompt.rstrip('<|endoftext|>')

print(f'>>> Prompt\n{prompt}')

audios = [soundfile.read("./data/sep_audio/vocals/32/32_138_0000_vocals.wav")]
inputs = processor(text=prompt, audios=audios, return_tensors='pt').to('cuda') # images=[image], 
generate_ids = model.generate(
    **inputs,
    max_new_tokens=1000,
    generation_config=generation_config,
    num_logits_to_keep=1,
)
generate_ids = generate_ids[:, inputs['input_ids'].shape[1] :]
response = processor.batch_decode(
    generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False
)[0]
print(f'>>> Response\n{response}')

[Luffy966]

I added these several lines following the usage instructions in transformers, which wasn't clarified in their huggingface repo.

# Optional: load the adapters (note that without them, the base model will very likely not work well)
# model.load_adapter(model_path, adapter_name="speech", device_map="auto", adapter_kwargs={"subfolder": 'speech-lora'}) # original code, didn't work, met errors like "cannot found adapter path"
# model.load_adapter(model_path, adapter_name="vision", device_map="auto", adapter_kwargs={"subfolder": 'vision-lora'}) # original code, didn't work, met errors like "cannot found adapter path"
model.load_adapter(os.path.join(model_path, "speech-lora"), adapter_name="speech", device_map="auto") # work for me
model.load_adapter(os.path.join(model_path, "vision-lora"), adapter_name="vision", device_map="auto") # work for me

...

# inference time
model.set_adapter("speech") # if loaded, activate the speech adapter

And I still got normal results instead of mangled ones. So I personally guess maybe it's the problem with vllm? Would it be possible for someone to delve deeper into this matter?