Using CPU Image without ONNX

[ramipellumbi]

Feature request

TEI 1.5 introduced feat(onnx): add onnx runtime for better CPU perf #328. Request to not use onnx runtime on CPU. It seems there is no way to opt out of the ONNX backend. Requesting ability to not use the ONNX backend.

pub async fn download_weights(api: &ApiRepo) -> Result<Vec<PathBuf>, ApiError> {
    let model_files = if cfg!(feature = "python") || cfg!(feature = "candle") {
        match download_safetensors(api).await {
            Ok(p) => p,
            Err(_) => {
                tracing::warn!("safetensors weights not found. Using `pytorch_model.bin` instead. Model loading will be significantly slower.");
                tracing::info!("Downloading `pytorch_model.bin`");
                let p = api.get("pytorch_model.bin").await?;
                vec![p]
            }
        }
    } else if cfg!(feature = "ort") {
        tracing::info!("Downloading `model.onnx`");
        match api.get("model.onnx").await {
            Ok(p) => vec![p],
            Err(err) => {
                tracing::warn!("Could not download `model.onnx`: {err}");
                tracing::info!("Downloading `onnx/model.onnx`");
                let p = api.get("onnx/model.onnx").await?;
                vec![p.parent().unwrap().to_path_buf()]
            }
        }
    } else {
        unreachable!()
    };
    Ok(model_files)
}

Motivation

Not all models have .onnx files posted. There doesn't seem to be a reason to lock the cpu runtime to onnx models only for people not concerned about performance.

Your contribution

Happy to open a PR adding an option to opt out of the onnx runtime and build the old cpu image as it was

[ycouble]

What's the status on this ? Is there a known workaround ?

[ramipellumbi]

What's the status on this ? Is there a known workaround ?

Using an earlier version is the workaround. It seems @OlivierDehaene is a maintainer and also the contributor to this change. Likely needs to be persuaded to revert his work—there is very little interaction with issues here.

[zanemcca]

We have millions of embeddings used in production with legacy older models and I assume many other people are in the same boat. Reverse compatibility definitely feels like the best course of action. We are just going to deploy using the 1.4.0 container for the time being but these kinds of breaking changes are one of the things that make us most nervous when using HuggingFace in production.

To make matters worse. In the huggingface endpoint settings page when you change the hardware configuration it automatically tries to upgrade the text-embeddings-inference to the latest version. We also got an email from HuggingFace stating that our old hardware is deprecated and that we have to upgrade. We will need to spin up a new endpoint and migrate traffic over to work around this issue. This does not give us a lot of confidence when managing long term production environments that cannot be adapted overnight.

[Narsil]

Hey, I can understand the frustration, sorry for that.

The fallback exists, but for some reasons, there's a hardcrash when fallback executes on this particular model.

But we should definitely improve the error message, providing both a very simple way to create a ONNX optimized variant, and how to use it:

• Run https://gist.github.com/tomaarsen/4b00b0e3be8884efa64cfab9230b161f
• Use --revision refs/pr/XX if you create a PR with the script (so everyone can benefit).

For instance here, I've enabled the model you want using:

 model=BAAI/llm-embedder
volume=/mnt

docker run --device=nvidia.com/gpu=all -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-embeddings-inference:cpu-1.7 --model-id $model --revision refs/pr/19

Add another option to force using a slower alternative that doesn't require modifying the model/creating PRs, since it's opt-in, you know it's going to be slower and you're OK with that.

[zanemcca]

Thanks for contributing and pushing that to the model @Narsil! I really appreciate it.

In this case though my point is more that making changes to the model should not be necessary to use the happy path through HuggingFace. We are a small startup so our whole team is software engineers without in-depth ML knowledge. We are not really keeping up to date on the latest in depth technical knowledge are are mostly just looking for plug-and-play type solutions using pre-trained models. With how advanced most modern models are these days I suspect this will be true for more and more users going forward.

I think things like embeddings are particularly tricky since the produced embeddings can be persisted and re-used for years in a production environment. This makes support for legacy modes very important. From a users perspective we want the exact same thing to be running for many years without any change unless we put the time into educating ourselves, testing and validating a new approach. I hope this all makes sense and helps clarify where we are coming from.