Openai compatiblity for Rerank models

[Yiqian-Liu]

Feature request

Will TEI support openai compatible route and request body schema / response body schema for Rerank models?

Motivation

For better compatibility.

Your contribution

Looking forward to this new feature.

[alvarobartt]

Hey @Yiqian-Liu thanks for opening the issue and the interest! Could you point us to the OpenAI Documentation for such API? I could only see the ranking features mentioned around the Vector Stores, which might not be suitable in this case, so asking just in case.

Also note that Text Embeddings Inference (TEI) already supports the v1/embeddings route for creating embeddings with embedding models as per https://huggingface.github.io/text-embeddings-inference/#/Text%20Embeddings%20Inference/openai_embed.

Thanks in advance 🤗

[vrdn-23]

It seems both JinaAI and Cohere follow almost similar input and output schemas (with slight difference in the output metadata) when it comes to re-ranking

Input parameters:
- model
- query
- documents
- top_n

Output parameters:
{
  "results": [
    {
      "index": 3,
      "relevance_score": 0.999071
    },
    {
      "index": 4,
      "relevance_score": 0.7867867
    },
    {
      "index": 0,
      "relevance_score": 0.32713068
    }
  ],
<different metadata> 

Cohere's API seems to talk about compatibility with the OpenAI client, but they don't have an example for rerank so not sure if that is supported.

I think it would be nice if we could have a /v1/rerank endpoint that atleast takes the following arguments in common and returns results
cc @alvarobartt

[alvarobartt]

Hey @vrdn-23 thanks for sharing those, really insightful! Indeed I'm afraid that those are not natively compatible with OpenAI, but rather specific for both JinaAI and Cohere, respectively, which means that those might not be "standard".

This being said, which are the flaws or what you feel like is missing in the current /rerank endpoint that could be improved by any of those interfaces? If those were standard it would make sense to add those routes, but given that's unclear, I don't see any real benefit here, so if you happen to have more information to share that'd be helpful, thanks 🤗

P.S. Also it seems that these are also supported within vLLM as per https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html#re-rank-api, so it might make sense to implement those if used on other open-source solutions such as vLLM, given that even if not OpenAI-standards those do follow similar conventions (?)

[vrdn-23]

Hey @alvarobartt,
Apologies but I think I missed your reply to this. I would love to help contribute to implement these API endpoints (both /v1/rerank and /v2/rerank) if this something that the maintainer team is still interested in supporting!

[alvarobartt]

Hey @vrdn-23 thanks for flagging, instead still discussing, I'll let you know about our plans, as adding and maintaining those can be "expensive", but let me validate with the team and I'll come back to you with a plan if we intend to support those! 🤗