Pasting images works for vision models but not for image-to-text tools

[gmanskibiditoilet]

Describe your feature request

Hi there!
We wonder if it's a great idea to add the ability to paste images directly from our devices' clipboards to HuggingChat on ALL models available (not just the llama3.2-11b vision instruct model), since it's pretty much of a nuisance for us to download images (especially screenshots) to our computer and re-upload to HuggingChat, isn't it inconvenient?

Screenshots (if relevant)

This feature works pretty well on llama3.2-11b-vision instruct model but we want this feature to work seamlessly on other models as well.

[gary149]

I think it makes sense to support this feature for vision model only as today, or how would that work?

[nsarrazin]

@gary149 I think they mean that pasting image files works on vision models but doesn't work when you have a tool that accepts image inputs, which is a bug: pasting a file should always work if the MIME type is supported (either because its a vision model or bc you have a tool that accepts images)

Marking as good first issue, happy to review a PR for this :)

[dagoatzmcclusky]

we hope a new PR is needed to fix this issue, since copying and pasting images work seamlessly fine on llama3.2-11b-vision-instruct but not for other models.

[yousefelkilany]

Hey, I tested on Llama-3.3-70B-Instruct with Image Editor and Image Description using Florence 2 tools and worked as expected, chatbox attachment icon is updated with valid types after activating the tool and accepts images from clipboard, but didn't work with Vision (Images and Videos), though pasting from clipboard is still active if at least a single tool supports images, while drag-n-drop works with either. Though last tool also behaves as expected and isn't supported with non vision models as per its description, I mean that the root cause could be from the tools in question rather ui interaction with the tool.

@gmanskibiditoilet @nsarrazin This seems easy to fix either way, so I was gonna take it if available, and is still valid. If you have a specific scenario or a tool in question, I'll look into it.

[nsarrazin]

Hi! I think then the issue is that we're not handling cases where the tool file inputs are */* like for Vision.

The logic that needs changing is in here iirc. let me know if you need any help!

[dagoatzmcclusky]

ok, this might work for now, although florence 2 completely disappeared from community tools and we now had to rely on another community tool called "joy-caption" to see the images.