Skip to content

Update VLGuard plugin to use the original VLGuard dataset #5805

@blueCicada

Description

@blueCicada

Is your feature request related to a problem? Please describe.
The new VLGuard plugin added in #5231 / #5243 uses the dataset at https://huggingface.co/datasets/kirito011024/vlguard_unsafes

This dataset is derived from the original at https://huggingface.co/datasets/ys-zong/VLGuard by extracting only unsafe records, and transforming the data format.

Problems with the derivative dataset:

  1. Unlike the original which is licensed under MIT, the derivative dataset has no licence attached to it
  2. The absence of safe records (which are present in the original set) reduces its utility for benchmarking evaluations

Describe the solution you'd like
If it's not too much to ask, we'd like the VLGuard plugin for promptfoo to be updated to use the original https://huggingface.co/datasets/ys-zong/VLGuard dataset

Describe alternatives you've considered
A more general plugin that can flexibly take a link to an arbitrary dataset and other information about the dataset's structure. However, there don't appear to be many other datasets of this nature that are publicly available, and the few that are have more restrictive or dubious licensing that would not work for our use case (but could be suitable for others). For example, https://huggingface.co/datasets/AIML-TUDA/LlavaGuard / https://huggingface.co/datasets/AIML-TUDA/smid says "All reasonable efforts have been made to ensure that the images in the SMID are in the public domain or licensed under Creative Commons or similar open licensing schemes", but this makes no guarantees about whether any of those are licensed under CC-NC (which would prohibit commercial use).

That's a long-winded way to say "we don't need the ability to pull arbitrary datasets right now, but other folks might find it helpful in the future" 🙂

Additional context
I work with @sudosammy-atlassian , on behalf of our team I wanted to say thank you very much for your receptiveness and responsiveness to the previous feature request (#5231) 🙏

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions