[Web] WebGPU issues tracking

[fs-eire]

This issue is for tracking WebGPU related problems. WebGPU EP is available since ONNX Runtime Web v1.15.0 as experimental feature. We are working on improving stability, operator coverage and performance.

For a list of supported/WIP operators, comments or any operator specific issues: #15952

---

Can not consume

Q: How to build?
A: Building ort-web with webgpu support from source: please refer to this gist

Q: [Web] An error occurred during model execution: "TypeError: Cannot read properties of undefined (reading 'apply')".
A: #15780 <--- this PR fixed it

Q: no available backend found. ERR: ...
A: Need to make sure webgpu is available in the current context. Upgrade to latest Chrome or Edge (v113), and served in a secured location ( https or localhost )

Runtime failures

Q: Non-zero status code returned while running Transpose node. ....
A: #15819 <--- This PR should fix it

Q: crash in the transpose optimizer for various models (#15869: cannot load model https://huggingface.co/runwayml/stable-diffusion-v1-5/tree/onnx/vae_encoder)
A: issue being investigated - see the PR for detailed info

Kernel coverage or running slow

Q: General investigation tips?
A: a few tools that can be used to taking deeper look at it: ( don't do them together, it will generate too many logs )

• env.logLevel = 'verbose'; env.debug = true; - This will let onnxruntime-web to output some logs helpful for analysing the execution. including telling which operators are running on webgpu and which are on CPU (fallback). to improve performance caused by fallback we need to improve the operator coverage. I can help to implement the missing ops.
• env.webgpu.profilingMode = 'default'; - This will output quite a lot of logs into console for each webgpu shaders - by aggregating and analyzing those we can know which shader is slow. Need to launch chrome/edge with flag --disable-dawn-features=disallow_unsafe_apis.
• set sessionOptions.enableProfiling = true when creating inference session. This shows which operator running on GPU, which fallback to CPU.

Q: running slow on image classification model. (logs)
A: jsepCopyGpuToCpu occurred 114 times, which indicating frequent CPU <--> GPU data transfer. Adding implementation of the missing operators may improve performance.

[xenova]

For now, need chrome/edge canary launched with flag --enable-unsafe-webgpu, and served in a secured location ( https or localhost )

WebGPU is now supported in the latest version of the official Chrome build (no longer only Canary, and not locked behind the flag). That said, I do not know about support for other browsers.

[xenova]

Q: Non-zero status code returned while running Transpose node. ....
A: investigating. @xenova could you share the model and corresponding input?

Here are the model files.

Here is some example input (for the encoder):

let input = {
    attention_mask: new Tensor(
        'int64',
        new BigInt64Array([1n, 1n, 1n, 1n, 1n, 1n, 1n, 1n, 1n, 1n, 1n, 1n]),
        [1, 12]
    ),
    input_ids: new Tensor(
        'int64',
        new BigInt64Array([13959n, 1566n, 12n, 2379n, 10n, 8774n, 6n, 149n, 33n, 25n, 58n, 1n]),
        [1, 12]
    )
}

Note: These are the same as I mentioned in the original issue: #15719 (comment)

[nagadomi]

Q: Non-zero status code returned while running Transpose node. ....

I got the similar error.

23-05-04 08:10:11.420400 [W:onnxruntime:, session_state.cc:1171 VerifyEachNodeIsAssignedToAnEp] Rerunning with verbose output on a non-minimal build will show node assignments.
i @ ort-wasm.js:49
23-05-04 08:10:11.427000 [E:onnxruntime:, sequential_executor.cc:514 ExecuteKernel] Non-zero status code returned while running Transpose node. Name:'/Transpose' Status Message: Failed to run JSEP kernel
Non-zero status code returned while running Transpose node. Name:'/Transpose' Status Message: Failed to run JSEP kernel
i @ ort-wasm.js:49
caught (in promise) Error: failed to call OrtRun(). error code = 1.
    at t.run (wasm-core-impl.ts:282:15)
    at async t.OnnxruntimeWebAssemblySessionHandler.run (session-handler.ts:81:11)
    at async d.run (inference-session-impl.js:91:15)
    at async Object.padding (script.js:498:19)
    at async Object.tiled_render (script.js:397:17)
    at async process (script.js:617:9)
    at async HTMLInputElement.<anonymous> (script.js:718:13)

The model(Object.padding) is very simple reflection padding.

PyTorch module
https://github.com/nagadomi/nunif/blob/6b605f8687b0ff5439f4c3776fd794f104cc2e15/nunif/models/onnx_helper_models.py#L12-L38
javascript caller
https://github.com/nagadomi/nunif/blob/6b605f8687b0ff5439f4c3776fd794f104cc2e15/waifu2x/unlimited_waifu2x/public_html/script.js#L483-L494
onnx file
pad.zip

[fs-eire]

The Transpose issue is figured out and a fix is in PR: #15819.

However, a model with a Transpose node requiring uint8 may not be running with a good performance. WGSL does not support uint8 (spec), so it's hard to get to hardware accelerated from int8 quantized model. So far we only support f32. f16 may be taken into consideration, but since Float16Array is not ready, it's hard to get it work E2E.

[DK013]

@fs-eire with the latest build (following your gist: here) with the following code with the same model and inputs as @xenova :

<script src="./dist/ort.webgpu.min.js"></script>
<script>
    document.addEventListener('DOMContentLoaded', async () => {

     // Load model
     let url = 'https://huggingface.co/Xenova/bert-base-cased_web/resolve/main/onnx/model_quantized.onnx'
     let model = await fetch(url);
     let buffer = await model.arrayBuffer();
     let array = new Uint8Array(buffer);

     // Create a new session
     ort.env.wasm.simd = false;
     ort.env.wasm.numThreads = 1;
     let session = await ort.InferenceSession.create(array)
      ...
   })
</script>

results in the following error at line ort.InferenceSession.create :

Uncaught (in promise) Error: no available backend found. ERR: [wasm] RuntimeError: Aborted(TypeError: WebAssembly.instantiate(): Import #0 module="a" error: module is not an object or function), [cpu] Error: previous call to 'initializeWebAssembly()' failed., [xnnpack] Error: previous call to 'initializeWebAssembly()' failed., [webgpu] Error: previous call to 'initializeWebAssembly()' failed.

If I provide only 'webgpu' as executionProviders, the error is as follows:

Uncaught (in promise) Error: no available backend found. ERR: [wasm] RuntimeError: Aborted(TypeError: WebAssembly.instantiate(): Import #0 module="a" error: module is not an object or function)

which is the same as before.

If I use <script src="./dist/ort.js"></script> instead of ort.webgpu.min.js and use webgpu as executionProviders, the error is: Uncaught (in promise) Error: no available backend found.

[fs-eire]

@DK013 The error message looks like the corresponding .wasm file is not served. please check from the devtool/network tab, if any 404 error occurs on *.wasm

[DK013]

@DK013 The error message looks like the corresponding .wasm file is not served. please check from the devtool/network tab, if any 404 error occurs on *.wasm

Doesn't look like it. However I was under the impression it's gonna load the jsep wasm files which exist in dist directory after build (I double checked). Seems like that's not the case.

[fs-eire]

I think I know the reason. so it is because of this line
ort.env.wasm.simd = false;

I don't provide a non-simd version of webgpu wasm file because I assume every environment that supports webgpu should always also support wasm fixed-size SIMD

so remove the line should make it work.

I think I can add a warning message if simd is off and webgpu is request.

[xenova]

Is there a list of supported WebGPU ops, as well as those planned to be implemented?