Problems trying to convert & run whisper model

[RyanMetcalfeInt8]

For whisper, there seem to be both conversion issues, and GenAI runtime errors.

onnxruntime:

commit a3c3e2f038216d03a184e76ac32e452612a45e33 (HEAD -> main, origin/main, origin/HEAD)
Author: Yulong Wang <7679871+fs-eire@users.noreply.github.com>
Date:   Mon Jul 7 02:02:43 2025 -0700

onnxruntime-genai:

commit 2f2ad90b7e5c132d2a6aa986031305ac7898ef48 (HEAD -> main, origin/main, origin/HEAD)
Author: Kinfey <93169410+kinfey@users.noreply.github.com>
Date:   Tue Jul 8 03:01:22 2025 +0800

Conversion Issues

I tried following instructions from here: https://github.com/microsoft/onnxruntime/tree/main/onnxruntime/python/tools/transformers/models/whisper#exporting-whisper-for-onnx-runtime-genai

For this command:
python3 -m models.whisper.convert_to_onnx -m openai/whisper-large-v3-turbo --output whisper-turbo --precision fp32 --provider cpu --use_external_data_format --optimize_onnx --no_beam_search_op --output_cross_qk

I received this runtime error:
(Python 3.10, Ubuntu 24.04)

Arguments:Namespace(model_name_or_path='openai/whisper-large-v3-turbo', model_impl='hf', cache_dir='./cache_models', output='whisper-turbo', optimize_onnx=False, use_gpu=False, precision=<Precision.FLOAT32: 'fp32'>, use_int64_inputs=False, provider='cpu', verbose=False, use_external_data_format=True, overwrite=False, separate_encoder_and_decoder_init=False, no_beam_search_op=True, use_decoder_masked_mha=False, use_vocab_mask=False, use_prefix_vocab_mask=False, use_forced_decoder_ids=False, use_logits_processor=False, collect_cross_qk=True, extra_decoding_ids=False, use_temperature=False, no_repeat_ngram_size=0, output_sequence_scores=False, output_scores=False, output_cross_qk=True, cross_qk_onnx_model=None, output_no_speech_probs=False, quantize_embedding_layer=False, quantize_per_channel=False, quantize_reduce_range=False, use_specific_logits_processor=False)
PyTorch Version:2.7.1+cu126
Transformers Version:4.53.1
OnnxRuntime Version:1.22.0
Skip exporting: existing ONNX model whisper-turbo/whisper-large-v3-turbo_decoder.onnx
Exporting ONNX model to whisper-turbo/whisper-large-v3-turbo_encoder.onnx
/home/rdmetca/Whisper/onnxruntime/onnxruntime/python/tools/transformers/models/whisper/my_env_310/lib/python3.10/site-packages/transformers/models/whisper/modeling_whisper.py:676: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if input_features.shape[-1] != expected_seq_length:
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/rdmetca/Whisper/onnxruntime/onnxruntime/python/tools/transformers/models/whisper/convert_to_onnx.py", line 573, in <module>
    main()
  File "/home/rdmetca/Whisper/onnxruntime/onnxruntime/python/tools/transformers/models/whisper/convert_to_onnx.py", line 479, in main
    output_paths = export_onnx_models(
  File "/home/rdmetca/Whisper/onnxruntime/onnxruntime/python/tools/transformers/models/whisper/convert_to_onnx.py", line 381, in export_onnx_models
    WhisperHelper.export_onnx(
  File "/home/rdmetca/Whisper/onnxruntime/onnxruntime/python/tools/transformers/models/whisper/whisper_helper.py", line 728, in export_onnx
    model.export_onnx(
  File "/home/rdmetca/Whisper/onnxruntime/onnxruntime/python/tools/transformers/models/whisper/whisper_encoder_decoder_init.py", line 288, in export_onnx
    torch.onnx.export(
  File "/home/rdmetca/Whisper/onnxruntime/onnxruntime/python/tools/transformers/models/whisper/my_env_310/lib/python3.10/site-packages/torch/onnx/__init__.py", line 396, in export
    export(
  File "/home/rdmetca/Whisper/onnxruntime/onnxruntime/python/tools/transformers/models/whisper/my_env_310/lib/python3.10/site-packages/torch/onnx/utils.py", line 529, in export
    _export(
  File "/home/rdmetca/Whisper/onnxruntime/onnxruntime/python/tools/transformers/models/whisper/my_env_310/lib/python3.10/site-packages/torch/onnx/utils.py", line 1467, in _export
    graph, params_dict, torch_out = _model_to_graph(
  File "/home/rdmetca/Whisper/onnxruntime/onnxruntime/python/tools/transformers/models/whisper/my_env_310/lib/python3.10/site-packages/torch/onnx/utils.py", line 1143, in _model_to_graph
    _set_input_and_output_names(graph, input_names, output_names)
  File "/home/rdmetca/Whisper/onnxruntime/onnxruntime/python/tools/transformers/models/whisper/my_env_310/lib/python3.10/site-packages/torch/onnx/utils.py", line 1596, in _set_input_and_output_names
    set_names(list(graph.outputs()), output_names, "output")
  File "/home/rdmetca/Whisper/onnxruntime/onnxruntime/python/tools/transformers/models/whisper/my_env_310/lib/python3.10/site-packages/torch/onnx/utils.py", line 1573, in set_names
    raise RuntimeError(
RuntimeError: number of output names provided (65) exceeded number of outputs (9)
========> Handling decoder model......
========> Handling encoder model......

I also received the same error on Windows, using python 3.10.

When I replaced openai/whisper-large-v3-turbo with openai/whisper-base, it seemed to work okay on Ubuntu. But on Windows, it seemed to fail during some ninja / build post-conversion step.

Whisper GenAI example Issues

Since I was able to produce a whisper-base model, I continued on to try to run the sample. I seem to get some reshape errors:

(ortgenai_python_env) C:\Users\LocalAdmin\ort_build\BuildArtifacts-20250707_111001\onnxruntime-genai\examples\python>python whisper.py -m C:\Users\LocalAdmin\ort_build\WhisperModels\whisper-base
Loading model...
Audio Paths (comma separated): C:\Users\LocalAdmin\ort_build\BuildArtifacts-20250707_111001\onnxruntime-genai\test\test_models\audios\jfk.flac
Loading audio...
Processing audio...
2025-07-08 07:07:27.7721011 [E:onnxruntime:onnxruntime-genai, sequential_executor.cc:572 onnxruntime::ExecuteKernel] Non-zero status code returned while running Reshape node. Name:'/decoder/Reshape_1' Status Message: C:\Users\LocalAdmin\ort_build\BuildArtifacts-20250707_111001\onnxruntime\onnxruntime\core\providers\cpu\tensor\reshape_helper.h:47 onnxruntime::ReshapeHelper::ReshapeHelper input_shape_size == size was false. The input tensor cannot be reshaped to the requested shape. Input shape:{4}, requested shape:{1,1}

Traceback (most recent call last):
  File "C:\Users\LocalAdmin\ort_build\BuildArtifacts-20250707_111001\onnxruntime-genai\examples\python\whisper.py", line 122, in <module>
    run(args)
  File "C:\Users\LocalAdmin\ort_build\BuildArtifacts-20250707_111001\onnxruntime-genai\examples\python\whisper.py", line 66, in run
    generator.set_inputs(inputs)
RuntimeError: Non-zero status code returned while running Reshape node. Name:'/decoder/Reshape_1' Status Message: C:\Users\LocalAdmin\ort_build\BuildArtifacts-20250707_111001\onnxruntime\onnxruntime\core\providers\cpu\tensor\reshape_helper.h:47 onnxruntime::ReshapeHelper::ReshapeHelper input_shape_size == size was false. The input tensor cannot be reshaped to the requested shape. Input shape:{4}, requested shape:{1,1}

Please advise.

[kunal-vaishnavi]

For the export issues, it seems that Hugging Face's Transformers updated the modeling code. This has changed how the graph looks and may break the graph fusions in ONNX Runtime's transformer optimizer.

Can you use transformers==4.52.3 to export? The runtime issues are stemming from the exported version that you are using and should go away with the older transformers version.

An already-exported version of Whisper tiny is stored in the ONNX Runtime GenAI CIs. It was produced with transformers==4.52.3 and a nightly version of ONNX Runtime. There is an end-to-end test that runs in the CIs to ensure the model can run.

The ninja failure can happen while attempting to create the ONNX model for calculating the jump times that are subsequently used in calculating word-level timestamps. If you remove --output_cross_qk from your command, the third ONNX model will not be created and the failure will go away.

[RyanMetcalfeInt8]

Can you use transformers==4.52.3 to export? The runtime issues are stemming from the exported version that you are using and should go away with the older transformers version.

I'll try it and let you know.

An already-exported version of Whisper tiny is stored in the ONNX Runtime GenAI CIs.

This is something that I would have access to?

If you remove --output_cross_qk from your command, the third ONNX model will not be created and the failure will go away.

If this step is skipped, is there some downside? Does ORT GenAI use this to improve accuracy of word-level timestamps?

[RyanMetcalfeInt8]

Hi @kunal-vaishnavi, I tried with transformers==4.52.3, but I still get the same error. Note that I also cleared the cache folders, etc.

(my_env) C:\Users\rmet0\ort_build\BuildArtifacts-20250707_051440\onnxruntime\onnxruntime\python\tools\transformers>python -m models.whisper.convert_to_onnx -m openai/whisper-large-v3-turbo --output whisper-turbo --use_external_data_format --no_beam_search_op
Arguments:Namespace(model_name_or_path='openai/whisper-large-v3-turbo', model_impl='hf', cache_dir='.\\cache_models', output='whisper-turbo', optimize_onnx=False, use_gpu=False, precision=<Precision.FLOAT32: 'fp32'>, use_int64_inputs=False, provider='cpu', verbose=False, use_external_data_format=True, overwrite=False, separate_encoder_and_decoder_init=False, no_beam_search_op=True, use_decoder_masked_mha=False, use_vocab_mask=False, use_prefix_vocab_mask=False, use_forced_decoder_ids=False, use_logits_processor=False, collect_cross_qk=False, extra_decoding_ids=False, use_temperature=False, no_repeat_ngram_size=0, output_sequence_scores=False, output_scores=False, output_cross_qk=False, cross_qk_onnx_model=None, output_no_speech_probs=False, quantize_embedding_layer=False, quantize_per_channel=False, quantize_reduce_range=False, use_specific_logits_processor=False)
PyTorch Version:2.7.1+cpu
Transformers Version:4.52.3
OnnxRuntime Version:1.22.0
config.json: 1.26kB [00:00, 1.28MB/s]
C:\Users\rmet0\ort_build\BuildArtifacts-20250707_051440\onnxruntime\onnxruntime\python\tools\transformers\models\whisper\my_env\lib\site-packages\huggingface_hub\file_download.py:143: UserWarning: `huggingface_hub` cache-system uses symlinks by default to efficiently store duplicated files but your machine does not support them in C:\Users\rmet0\ort_build\BuildArtifacts-20250707_051440\onnxruntime\onnxruntime\python\tools\transformers\cache_models\models--openai--whisper-large-v3-turbo. Caching files will still work but in a degraded version that might require more space on your disk. This warning can be disabled by setting the `HF_HUB_DISABLE_SYMLINKS_WARNING` environment variable. For more details, see https://huggingface.co/docs/huggingface_hub/how-to-cache#limitations.
To support symlinks on Windows, you either need to activate Developer Mode or to run Python as an administrator. In order to activate developer mode, see this article: https://docs.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development
  warnings.warn(message)
Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
model.safetensors: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.62G/1.62G [00:24<00:00, 66.7MB/s]
generation_config.json: 3.77kB [00:00, ?B/s]
========> Handling decoder model......
Exporting ONNX model to whisper-turbo\whisper-large-v3-turbo_decoder.onnx
Passing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.43.0. You should pass an instance of `EncoderDecoderCache` instead, e.g. `past_key_values=EncoderDecoderCache.from_legacy_cache(past_key_values)`.
C:\Users\rmet0\ort_build\BuildArtifacts-20250707_051440\onnxruntime\onnxruntime\python\tools\transformers\models\whisper\my_env\lib\site-packages\transformers\cache_utils.py:556: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  or not self.key_cache[layer_idx].numel()  # the layer has no cache
C:\Users\rmet0\ort_build\BuildArtifacts-20250707_051440\onnxruntime\onnxruntime\python\tools\transformers\models\whisper\my_env\lib\site-packages\transformers\models\whisper\modeling_whisper.py:1341: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if sequence_length != 1:
C:\Users\rmet0\ort_build\BuildArtifacts-20250707_051440\onnxruntime\onnxruntime\python\tools\transformers\models\whisper\my_env\lib\site-packages\transformers\cache_utils.py:539: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  elif (
C:\Users\rmet0\ort_build\BuildArtifacts-20250707_051440\onnxruntime\onnxruntime\python\tools\transformers\models\whisper\my_env\lib\site-packages\transformers\models\whisper\modeling_whisper.py:330: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attn_output.size() != (bsz, self.num_heads, tgt_len, self.head_dim):
Comparing logits...
Max diff: 6.67572021484375e-06
Comparing present_key_self_0...
Max diff: 9.5367431640625e-07
Comparing present_value_self_0...
Max diff: 9.685754776000977e-08
Comparing present_key_self_1...
Max diff: 1.6093254089355469e-06
Comparing present_value_self_1...
Max diff: 2.384185791015625e-07
Comparing present_key_self_2...
Max diff: 1.6093254089355469e-06
Comparing present_value_self_2...
Max diff: 3.0547380447387695e-07
Comparing present_key_self_3...
Max diff: 1.6689300537109375e-06
Comparing present_value_self_3...
Max diff: 4.76837158203125e-07
Comparing present_key_self_4...
Max diff: 0.0
Comparing present_value_self_4...
Max diff: 0.0
Comparing present_key_self_5...
Max diff: 0.0
Comparing present_value_self_5...
Max diff: 0.0
Comparing present_key_self_6...
Max diff: 0.0
Comparing present_value_self_6...
Max diff: 0.0
Comparing present_key_self_7...
Max diff: 0.0
Comparing present_value_self_7...
Max diff: 0.0
Comparing present_key_self_8...
Max diff: 0.0
Comparing present_value_self_8...
Max diff: 0.0
Comparing present_key_self_9...
Max diff: 0.0
Comparing present_value_self_9...
Max diff: 0.0
Comparing present_key_self_10...
Max diff: 0.0
Comparing present_value_self_10...
Max diff: 0.0
Comparing present_key_self_11...
Max diff: 0.0
Comparing present_value_self_11...
Max diff: 0.0
Comparing present_key_self_12...
Max diff: 0.0
Comparing present_value_self_12...
Max diff: 0.0
Comparing present_key_self_13...
Max diff: 0.0
Comparing present_value_self_13...
Max diff: 0.0
Comparing present_key_self_14...
Max diff: 0.0
Comparing present_value_self_14...
Max diff: 0.0
Comparing present_key_self_15...
Max diff: 0.0
Comparing present_value_self_15...
Max diff: 0.0
Comparing present_key_self_16...
Max diff: 0.0
Comparing present_value_self_16...
Max diff: 0.0
Comparing present_key_self_17...
Max diff: 0.0
Comparing present_value_self_17...
Max diff: 0.0
Comparing present_key_self_18...
Max diff: 0.0
Comparing present_value_self_18...
Max diff: 0.0
Comparing present_key_self_19...
Max diff: 0.0
Comparing present_value_self_19...
Max diff: 0.0
Comparing present_key_self_20...
Max diff: 0.0
Comparing present_value_self_20...
Max diff: 0.0
Comparing present_key_self_21...
Max diff: 0.0
Comparing present_value_self_21...
Max diff: 0.0
Comparing present_key_self_22...
Max diff: 0.0
Comparing present_value_self_22...
Max diff: 0.0
Comparing present_key_self_23...
Max diff: 0.0
Comparing present_value_self_23...
Max diff: 0.0
Comparing present_key_self_24...
Max diff: 0.0
Comparing present_value_self_24...
Max diff: 0.0
Comparing present_key_self_25...
Max diff: 0.0
Comparing present_value_self_25...
Max diff: 0.0
Comparing present_key_self_26...
Max diff: 0.0
Comparing present_value_self_26...
Max diff: 0.0
Comparing present_key_self_27...
Max diff: 0.0
Comparing present_value_self_27...
Max diff: 0.0
Comparing present_key_self_28...
Max diff: 0.0
Comparing present_value_self_28...
Max diff: 0.0
Comparing present_key_self_29...
Max diff: 0.0
Comparing present_value_self_29...
Max diff: 0.0
Comparing present_key_self_30...
Max diff: 0.0
Comparing present_value_self_30...
Max diff: 0.0
Comparing present_key_self_31...
Max diff: 0.0
Comparing present_value_self_31...
Max diff: 0.0
========> Handling encoder model......
Exporting ONNX model to whisper-turbo\whisper-large-v3-turbo_encoder.onnx
C:\Users\rmet0\ort_build\BuildArtifacts-20250707_051440\onnxruntime\onnxruntime\python\tools\transformers\models\whisper\my_env\lib\site-packages\transformers\models\whisper\modeling_whisper.py:881: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if input_features.shape[-1] != expected_seq_length:
Traceback (most recent call last):
  File "C:\Users\rmet0\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\rmet0\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\rmet0\ort_build\BuildArtifacts-20250707_051440\onnxruntime\onnxruntime\python\tools\transformers\models\whisper\convert_to_onnx.py", line 573, in <module>
    main()
  File "C:\Users\rmet0\ort_build\BuildArtifacts-20250707_051440\onnxruntime\onnxruntime\python\tools\transformers\models\whisper\convert_to_onnx.py", line 479, in main
    output_paths = export_onnx_models(
  File "C:\Users\rmet0\ort_build\BuildArtifacts-20250707_051440\onnxruntime\onnxruntime\python\tools\transformers\models\whisper\convert_to_onnx.py", line 381, in export_onnx_models
    WhisperHelper.export_onnx(
  File "C:\Users\rmet0\ort_build\BuildArtifacts-20250707_051440\onnxruntime\onnxruntime\python\tools\transformers\models\whisper\whisper_helper.py", line 728, in export_onnx
    model.export_onnx(
  File "C:\Users\rmet0\ort_build\BuildArtifacts-20250707_051440\onnxruntime\onnxruntime\python\tools\transformers\models\whisper\whisper_encoder_decoder_init.py", line 288, in export_onnx
    torch.onnx.export(
  File "C:\Users\rmet0\ort_build\BuildArtifacts-20250707_051440\onnxruntime\onnxruntime\python\tools\transformers\models\whisper\my_env\lib\site-packages\torch\onnx\__init__.py", line 396, in export
    export(
  File "C:\Users\rmet0\ort_build\BuildArtifacts-20250707_051440\onnxruntime\onnxruntime\python\tools\transformers\models\whisper\my_env\lib\site-packages\torch\onnx\utils.py", line 529, in export
    _export(
  File "C:\Users\rmet0\ort_build\BuildArtifacts-20250707_051440\onnxruntime\onnxruntime\python\tools\transformers\models\whisper\my_env\lib\site-packages\torch\onnx\utils.py", line 1467, in _export
    graph, params_dict, torch_out = _model_to_graph(
  File "C:\Users\rmet0\ort_build\BuildArtifacts-20250707_051440\onnxruntime\onnxruntime\python\tools\transformers\models\whisper\my_env\lib\site-packages\torch\onnx\utils.py", line 1143, in _model_to_graph
    _set_input_and_output_names(graph, input_names, output_names)
  File "C:\Users\rmet0\ort_build\BuildArtifacts-20250707_051440\onnxruntime\onnxruntime\python\tools\transformers\models\whisper\my_env\lib\site-packages\torch\onnx\utils.py", line 1596, in _set_input_and_output_names
    set_names(list(graph.outputs()), output_names, "output")
  File "C:\Users\rmet0\ort_build\BuildArtifacts-20250707_051440\onnxruntime\onnxruntime\python\tools\transformers\models\whisper\my_env\lib\site-packages\torch\onnx\utils.py", line 1573, in set_names
    raise RuntimeError(
RuntimeError: number of output names provided (65) exceeded number of outputs (9)

Here's my pip list (python 3.10):

Package                Version
---------------------- -----------
aiohappyeyeballs       2.6.1
aiohttp                3.12.13
aiosignal              1.4.0
async-timeout          5.0.1
attrs                  25.3.0
audioread              3.0.1
certifi                2025.6.15
cffi                   1.17.1
charset-normalizer     3.4.2
colorama               0.4.6
coloredlogs            15.0.1
datasets               3.6.0
decorator              5.2.1
dill                   0.3.8
ffmpeg-python          0.2.0
filelock               3.18.0
flatbuffers            25.2.10
frozenlist             1.7.0
fsspec                 2025.3.0
future                 1.0.0
huggingface-hub        0.33.2
humanfriendly          10.0
idna                   3.10
Jinja2                 3.1.6
joblib                 1.5.1
lazy_loader            0.4
librosa                0.11.0
llvmlite               0.43.0
MarkupSafe             3.0.2
more-itertools         10.7.0
mpmath                 1.3.0
msgpack                1.1.1
multidict              6.6.3
multiprocess           0.70.16
networkx               3.4.2
ninja                  1.11.1.4
numba                  0.60.0
numpy                  1.23.3
onnx                   1.17.0
onnxruntime            1.22.0
onnxruntime_extensions 0.14.0
openai-whisper         20240927
optimum                1.26.1
packaging              25.0
pandas                 2.3.1
pillow                 11.3.0
pip                    23.0.1
platformdirs           4.3.8
pooch                  1.8.2
propcache              0.3.2
protobuf               3.20.2
psutil                 7.0.0
py-cpuinfo             9.0.0
py3nvml                0.2.7
pyarrow                20.0.0
pycparser              2.22
pyreadline3            3.5.4
python-dateutil        2.9.0.post0
pytz                   2025.2
PyYAML                 6.0.2
regex                  2024.11.6
requests               2.32.4
safetensors            0.5.3
scikit-learn           1.7.0
scipy                  1.13.1
sentencepiece          0.2.0
setuptools             65.5.0
six                    1.17.0
soundfile              0.13.1
soxr                   0.5.0.post1
sympy                  1.14.0
threadpoolctl          3.6.0
tiktoken               0.9.0
tokenizers             0.21.2
torch                  2.7.1
tqdm                   4.67.1
transformers           4.52.3
typing_extensions      4.14.1
tzdata                 2025.2
urllib3                2.5.0
xmltodict              0.14.2
xxhash                 3.5.0
yarl                   1.20.1

[RyanMetcalfeInt8]

Also, I reconverted whisper-base with transformers==4.52.3. When I run the ORT GenAI whisper python sample with this version, it does seem to work better than before:

(ortgenai_python_env) C:\Users\rmet0\ort_build\BuildArtifacts-20250707_051440\onnxruntime-genai\examples\python>python whisper.py -m C:\Users\rmet0\ort_build\BuildArtifacts-20250707_051440\onnxruntime\onnxruntime\python\tools\transformers\whisper-base
Loading model...
Audio Paths (comma separated): C:\Users\rmet0\ort_build\BuildArtifacts-20250707_051440\onnxruntime-genai\test\test_models\audios\jfk.flac
Loading audio...
Processing audio...

Transcription:
    batch 0, beam 0:  And so my fellow Americans, ask not what your country can do for you, ask what you can do for your country.
Transcription:
    batch 0, beam 1:  And so my fellow Americans ask not what your country can do for you, ask what you can do for your country.
Transcription:
    batch 0, beam 2:  And so my fellow Americans ask not what your country can do for you. Ask what you can do for your country.
Transcription:
    batch 0, beam 3:  And so my fellow Americans, ask not what your country can do for you, ask what you can do for your country!

One question I have is, why does it show output for 4 beams here? I exported the model using --no_beam_search_op, and I also have "num_beams": 1, within the search parameters defined in genai_config.json. Is this expected?

Edit: Okay, I see what's going on -- the whisper example defaults --num_beams to 4:

parser.add_argument(
        "-b", "--num_beams", type=int, default=4, help="Number of beams"
    )

Adding -b 1 seems to give expected output now. Anyway, still find it a bit strange that the example defaults this to 4..

[RyanMetcalfeInt8]

One additional issue that I see, is that the whisper python example only seems to support audio snippets of up to 30 seconds.

For example, if I give a .wav that is 31.1 seconds, it crashes. Most of the time, it seems to crash within:

inputs = processor(prompts, audios=audios)

But sometimes it makes it past this point, and crashes within

generator.set_inputs(inputs)

So is it intended that the processor can only support audio of up to 30s? If so, it's up to user to implement the chunking loop?

[kunal-vaishnavi]

This is something that I would have access to?

The CI models are uploaded in an internal storage account so they can't be easily shared. You can re-create them with the Whisper conversion toolchain though.

If this step is skipped, is there some downside? Does ORT GenAI use this to improve accuracy of word-level timestamps?

There is no downside to omitting the --output_cross_qk flag from a performance standpoint when not computing word-level timestamps. As mentioned in the Whisper PR description, support for word-level timestamps is still a work in progress because the end-to-end working example from OpenAI's implementation is still under development.

Hi @kunal-vaishnavi, I tried with transformers==4.52.3, but I still get the same error. Note that I also cleared the cache folders, etc.

The export issue with whisper-large-v3-turbo stems from the encoder and decoder having a different number of layers (encoder uses 32 layers and decoder uses 4 layers). I will make a PR fix in ORT for this.

One question I have is, why does it show output for 4 beams here? I exported the model using --no_beam_search_op, and I also have "num_beams": 1, within the search parameters defined in genai_config.json. Is this expected?

The number of beams is passed as a flag to the end-to-end example because speech-to-text models commonly use beam search to find the best transcription output.

There are two Whisper solutions.

1. The original one which produces one ONNX model containing the WhisperBeamSearch op and does not require ONNX Runtime GenAI to run
2. The redesigned one which produces two ONNX models (neither one having the WhisperBeamSearch op) and can run beam search with ONNX Runtime GenAI

Both solutions use beam search for the best accuracy.

One additional issue that I see, is that the whisper python example only seems to support audio snippets of up to 30 seconds. So is it intended that the processor can only support audio of up to 30s? If so, it's up to user to implement the chunking loop?

Yes, this is because the model can only support audio up to 30s. The chunking loop is part of the end-to-end working example from OpenAI's implementation.

[RyanMetcalfeInt8]

There are two Whisper solutions.

Can you give some more details about how the two solutions that you describe relate back to the onnx conversion recipes, here?

And I guess the 'redesigned' one is the recommended approach? Given the conversion recipes that I used, shared above, is this what I'm using?

Yes, this is because the model can only support audio up to 30s. The chunking loop is part of the end-to-end working example from OpenAI's implementation.

Okay, I understand that each invocation of whisper (encoder + decoder sequence) operates on 30s chunks. But is there a plan to add an example to this repo that illustrates the recommended flow/logic to perform transcription on an audio segment of arbitrary length?

And when you mention OpenAI's implementation, are you saying that there is already an implementation of this that I can reference which uses onnxruntime-genai?

[kunal-vaishnavi]

Can you give some more details about how the two solutions that you describe relate back to the onnx conversion recipes, here?

The original approach was used as the ONNX solution for Whisper. However, it became hard to maintain and customize the decoding loop over time.

ONNX Runtime GenAI was created to handle this work (in fact, Whisper is the motivation for why this library exists). The redesigned approach was built to work with ONNX Runtime GenAI. The new approach also shows how encoder-decoder models can be generally supported in an efficient way (e.g. pre-compute cross-attention KV caches in the encoder vs. re-computing the cross-attention KV caches every time when running the decoder).

And I guess the 'redesigned' one is the recommended approach? Given the conversion recipes that I used, shared above, is this what I'm using?

Yes, this is the recommended approach and it is what you are using when you use --no_beam_search_op. The recipes will eventually be added into Olive as well.

Okay, I understand that each invocation of whisper (encoder + decoder sequence) operates on 30s chunks. But is there a plan to add an example to this repo that illustrates the recommended flow/logic to perform transcription on an audio segment of arbitrary length?

And when you mention OpenAI's implementation, are you saying that there is already an implementation of this that I can reference which uses onnxruntime-genai?

Yes, the PR Iinked above has a link to a fork of the OpenAI implementation containing the recommended flow and I have modified it to run with ONNX Runtime GenAI. The changes in the fork are still a work in progress though.

[RyanMetcalfeInt8]

Hi @kunal-vaishnavi, thanks for the additional details!

Yes, the PR Iinked above has a link to a fork of the OpenAI implementation containing the recommended flow and I have modified it to run with ONNX Runtime GenAI. The changes in the fork are still a work in progress though.

I see, thanks. So given that I have the standalone whisper example working with whisper-base, is it pretty straightforward to use your openai/whisper branch (https://github.com/kunal-vaishnavi/whisper/tree/kvaishnavi/onnxruntime-genai) to run full audio clips? I didn't see much in the README about how to actually enable the use of the onnxruntime_genai bits that you added... I could probably figure it out, but would you mind giving a quick example cmd?

[kunal-vaishnavi]

I see, thanks. So given that I have the standalone whisper example working with whisper-base, is it pretty straightforward to use your openai/whisper branch (https://github.com/kunal-vaishnavi/whisper/tree/kvaishnavi/onnxruntime-genai) to run full audio clips? I didn't see much in the README about how to actually enable the use of the onnxruntime_genai bits that you added... I could probably figure it out, but would you mind giving a quick example cmd?

Yes, it can currently run. Here is an example (this may change since it is still under development).

#!/bin/bash

audio="/path/to/audio/file"
onnx="/path/to/model/folder"
size="name_of_model_size" (choose from the keys listed here: https://github.com/openai/whisper/blob/c0d2f624c09dc18e709e37c2ad90c039a4eb72a2/whisper/__init__.py#L17)

python transcribe.py $audio --onnx_model_dir $onnx --model $size --model_dir ./cache_dir --device cpu --output_dir ./cache_dir --word_timestamps False --best_of 1 --beam_size 5 --batch_size 1

Given the branch's state, I expect it will fail to run and there will be several bugs. This is because it was based on previous commits in the Whisper ORT GenAI PR and not the final version that was merged.