Could auto-truncate also deal with character limits? #364
Description
Feature request
Today, truncation only deals with the number of tokens. It doesn't manage the hard character limit (250*n_tokens). This limit seems high but it does fail on some datasets (NQ for example, probably because of URLs contained in documents).
Motivation
It would avoid having truncation logic outside the endpoint.
Your contribution
I could try to open a PR but I'm not fluent in rust sadly.