[Usage]: Error when loading TIGER-Lab/VLM2Vec-Qwen2VL-2B

[yuafng]

Your current environment

The output of `python collect_env.py`

How would you like to use vllm

I want to run inference for TIGER-Lab/VLM2Vec-Qwen2VL-2b. I don't know how to initialize it with vllm.

I am doing like:

llm = LLM(
model="TIGER-Lab/VLM2Vec-Qwen2VL-2b",
runner="pooling",
trust_remote_code=True,
)

But get failure "There is no module or parameter named 'base_model' in Qwen2VLForConditionalGeneration" when load weights. It looks there are some config setting issues which cause weight key name mismatch.

Thanks in advance for any advice to debug this!

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

LoRA adapters aren't supported for multimodal models. You can merge the LoRA weights and save it to a separate HF repo before loading the model in vLLM. You also have to modify the chat template because it is different from the default one for Qwen2-VL. For example:

from peft import PeftConfig, PeftModel
from transformers import AutoModelForImageTextToText, AutoProcessor

MODEL_ID = "TIGER-Lab/VLM2Vec-Qwen2VL-7B"
IMAGE_TOKEN = "<|image_pad|>"

# Modify `examples/template_vlm2vec.jinja` to use tokens defined by Qwen2-VL instead of Phi-3V
CHAT_TEMPLATE = """{%- if messages | length > 1 -%}
    {{ raise_exception('Embedding models should only embed one message at a time') }}
{%- endif -%}

{% set vars = namespace(parts=[], next_image_id=1) %}
{%- for message in messages -%}
    {%- for content in message['content'] -%}
        {%- if content['type'] == 'text' -%}
            {%- set vars.parts = vars.parts + [content['text']] %}
        {%- elif content['type'] == 'image' -%}
            {%- set vars.parts = vars.parts + ["{IMAGE_TOKEN}"] %}
            {%- set vars.next_image_id = vars.next_image_id + 1 %}
        {%- endif -%}
    {%- endfor -%}
{%- endfor -%}
{{ vars.parts | join(' ') }}
""".replace("{IMAGE_TOKEN}", IMAGE_TOKEN)

def save_to_vllm_format(output_dir, torch_dtype=torch.bfloat16, use_fast=None):
    base_model = AutoModelForImageTextToText.from_pretrained(MODEL_ID, torch_dtype=torch_dtype)
    lora_model = PeftModel.from_pretrained(base_model, MODEL_ID, config=PeftConfig.from_pretrained(MODEL_ID))
    model = lora_model.merge_and_unload().to(dtype=torch_dtype).eval()
    model._hf_peft_config_loaded = False  # Needed to save the merged model

    processor = AutoProcessor.from_pretrained(
        MODEL_ID,
         # The current transformers version prefers this instead of `min_pixels` and `max_pixels`
        size={"shortest_edge": 3136, "longest_edge": 12845056},
        use_fast=use_fast,
    )
    processor.chat_template = CHAT_TEMPLATE  # The original chat template is wrong

    model.save_pretrained(output_dir)
    processor.save_pretrained(output_dir)

After saving the model:

$ vllm serve <output_dir> --runner pooling --convert embed

Personally, I found these additional arguments worked well with 4x A800 GPUs on #22070 branch:

-tp 4 --mm-encoder-tp-mode data --api-server-count 8 --mm-processor-kwargs '{"use_fast": true, "device": "cuda"}'

I haven't tried --mm-processor-cache-type shm yet but it should also help.