Allow configuration template to disable some SIMD. #20215

jslap-ubi · 2024-04-05T20:49:16Z

Description

Add compile flags to disable some SIMD optimizations

Motivation and Context

There is a mechanism to detect at runtime if some SIMD optimizations are available, and skip them if it is not the case, But on some platform, some compiler SIMD optimization are not available, even at the compilation step.

jslap-ubi · 2024-04-05T20:51:59Z

@microsoft-github-policy-service agree company="Ubisoft"

yihonglyu · 2024-04-08T18:32:52Z

onnxruntime/core/mlas/lib/convsym.cpp


 #if defined(MLAS_TARGET_AMD64)

+#if !defined(ORT_DISABLE_AVX2)


Where do these macros come from?

I tried to follow the way enabling/disabling certain things was done in the code (for instance: ORT_ENABLE_STREAM). Then the macros should be defined with a '-D CMAKE_CXX_FLAGS=ORT_DISABLE_AVX2'.

Would there be a preferred way of doing this? Should it be done through cmake option() ?

I think it would be preferable to make them CMake options. They should be documented too.

Are you sure? I don't want to overcrowd the cmake options. That would be 5 more options.
If you're positive, I will add the options and documentation.

I think they should be documented. it's fine to add additional CMake options. these are additional build configuration options so it seems like a reasonable place for them.

yufenglee · 2024-04-09T00:14:11Z

The instruction set will be chosen dynamically at runtime. What is the reason to add those macros? Could you please explain more about it?

jslap-ubi · 2024-04-09T13:35:57Z

Some SDKs just don't have the include files needed for the intrinsic. So making it optional would allow more platforms to be able to use the library.

yufenglee · 2024-04-10T22:23:34Z

Some SDKs just don't have the include files needed for the intrinsic. So making it optional would allow more platforms to be able to use the library.

Which SDKs don't have the intrinsic files?

jslap-ubi · 2024-04-12T20:20:11Z

There is one of our gaming platform that links inclusion of the files containing the symbols for the intrinsics (__m512 for instance) with the compiler actually generating code with the instruction set (-mavx512 for instance).
And we would like to have the flexibility to use the compile flag or not.

jslap-ubi · 2024-04-23T13:55:00Z

Have you had the chance to review with my last comments?

jslap-ubi · 2024-05-01T12:43:02Z

Bump. Just to make sure we don't forget about this one.

jslap-ubi · 2024-07-02T20:47:17Z

Any news on this? @yufenglee ?

yufenglee · 2024-07-31T17:59:30Z

/azp run Windows ARM64 QNN CI Pipeline,Windows x64 QNN CI Pipeline,Windows CPU CI Pipeline,Windows GPU CI Pipeline,Windows GPU TensorRT CI Pipeline,ONNX Runtime Web CI Pipeline,Linux CPU CI Pipeline,Linux CPU Minimal Build E2E CI Pipeline,Linux GPU CI Pipeline,Linux GPU TensorRT CI Pipeline

yufenglee · 2024-07-31T17:59:38Z

/azp run Linux OpenVINO CI Pipeline,Linux QNN CI Pipeline,MacOS CI Pipeline,orttraining-amd-gpu-ci-pipeline,orttraining-linux-ci-pipeline,orttraining-linux-gpu-ci-pipeline,orttraining-ortmodule-distributed,onnxruntime-binary-size-checks-ci-pipeline,Big Models,Android CI Pipeline

yufenglee · 2024-07-31T17:59:44Z

/azp run iOS CI Pipeline,ONNX Runtime React Native CI Pipeline

azure-pipelines · 2024-07-31T17:59:57Z

Azure Pipelines successfully started running 2 pipeline(s).

azure-pipelines · 2024-07-31T18:00:06Z

Azure Pipelines successfully started running 9 pipeline(s).

azure-pipelines · 2024-07-31T18:00:20Z

Azure Pipelines successfully started running 10 pipeline(s).

yufenglee

yufenglee · 2024-08-01T18:19:53Z

/azp run Linux Android Emulator QNN CI Pipeline

azure-pipelines · 2024-08-01T18:20:02Z

Azure Pipelines successfully started running 1 pipeline(s).

yufenglee · 2024-08-01T18:20:05Z

/azp run Python format

azure-pipelines · 2024-08-01T18:20:11Z

No pipelines are associated with this pull request.

yufenglee · 2024-08-01T18:21:19Z

@jslap-ubi, you need to sync to latest main branch to trigger the Python format CI.

yufenglee · 2024-08-01T22:35:06Z

/azp run Windows ARM64 QNN CI Pipeline,Windows x64 QNN CI Pipeline,Windows CPU CI Pipeline,Windows GPU CI Pipeline,Windows GPU TensorRT CI Pipeline,ONNX Runtime Web CI Pipeline,Linux CPU CI Pipeline,Linux CPU Minimal Build E2E CI Pipeline,Linux GPU CI Pipeline,Linux GPU TensorRT CI Pipeline

yufenglee · 2024-08-01T22:35:28Z

/azp run Linux OpenVINO CI Pipeline,Linux QNN CI Pipeline,MacOS CI Pipeline,orttraining-amd-gpu-ci-pipeline,orttraining-linux-ci-pipeline,orttraining-linux-gpu-ci-pipeline,orttraining-ortmodule-distributed,onnxruntime-binary-size-checks-ci-pipeline,Big Models,Android CI Pipeline

azure-pipelines · 2024-08-01T22:35:44Z

Azure Pipelines successfully started running 9 pipeline(s).

azure-pipelines · 2024-08-01T22:36:07Z

Azure Pipelines successfully started running 10 pipeline(s).

yufenglee · 2024-08-01T22:38:07Z

/azp run Linux Android Emulator QNN CI Pipeline

azure-pipelines · 2024-08-01T22:38:17Z

Azure Pipelines successfully started running 1 pipeline(s).

jslap-ubi · 2024-08-08T15:37:46Z

Thanks for the approval @yufenglee , what is the next step for this?

jslap-ubi · 2024-08-23T14:53:47Z

Is there anything else I need to do to go forward with the merge? @yufenglee

jslap-ubi · 2024-09-23T17:00:22Z

I just rebase on main since there was now a file conflicting.

@yufenglee : I think we need to renew the approval. Also, What would then be missing for having this MR merged?

jslap-ubi · 2024-11-29T21:57:06Z

The MR has been rebased, it would be ready to merge.

@yufenglee @yihonglyu : Could it be possible to re-approve this?

edgchen1 · 2025-09-12T23:02:19Z

onnxruntime/core/mlas/lib/convsym.cpp


 #if defined(MLAS_TARGET_AMD64)

+#if !defined(ORT_DISABLE_AVX2)


I think it would be preferable to make them CMake options. They should be documented too.

edgchen1 · 2025-09-12T23:08:48Z

onnxruntime/core/mlas/lib/platform.cpp

                }

-#ifndef __APPLE__
+#if !defined(__APPLE__) && !defined(ORT_DISABLE_AMX)


This !defined(ORT_DISABLE_AMX) block also contains this AVX kernel assignment.

https://github.com/jslap-ubi/onnxruntime/blob/7ca7306ae5b65a9606e57ca98d730ae438f568d1/onnxruntime/core/mlas/lib/platform.cpp#L510-L512

Why is this guarded by the ORT_DISABLE_AMX macro?

Good catch, I think it happened when I rebased. I will fix it.

edgchen1 · 2025-09-12T23:22:15Z

There is a mechanism to detect at runtime if some SIMD optimizations are available, and skip them if it is not the case, But on some platform, some compiler SIMD optimization are not available, even at the compilation step.

Do you want to skip compilation of certain optimized kernels? It looks like this change only prevents them from being selected at runtime.

Would you mind elaborating on your use case?

jslap-ubi · 2025-09-15T14:14:59Z

Would you mind elaborating on your use case?

On some gaming platform, compiling the kernels that target those architecture will just fail bacause the intrinsics are not present at all. So we need to skip the compilation of these specific kernel, and to skip where these function pointers are used.

edgchen1 · 2025-09-15T17:27:08Z

Would you mind elaborating on your use case?

On some gaming platform, compiling the kernels that target those architecture will just fail bacause the intrinsics are not present at all. So we need to skip the compilation of these specific kernel, and to skip where these function pointers are used.

I see. Does this change work as-is? I would expect that you'd also need to disable the actual compilation of the kernel source files elsewhere, like in onnxruntime_mlas.cmake.

jslap-ubi · 2025-09-15T17:31:13Z

I see. Does this change work as-is? I would expect that you'd also need to disable the actual compilation of the kernel source files elsewhere, like in onnxruntime_mlas.cmake.

Yes, the actual kernel is located in the file qgemm_kernel_amx.cpp. (we do exclude this file from the build on our side), So I think the exclusion of the files should be integrated in the cmake files, right?

edgchen1 · 2025-09-15T18:16:43Z

So I think the exclusion of the files should be integrated in the cmake files, right?

Yes, that seems reasonable.

jslap-ubi · 2025-09-15T20:59:36Z

I did the fixes in the cmake files, and I rebased.

edgchen1 · 2025-09-15T23:20:23Z

cmake/onnxruntime_mlas.cmake

        set_source_files_properties(${mlas_platform_srcs_sse2} PROPERTIES COMPILE_FLAGS "-msse2")

+        set(mlas_platform_srcs_sse41
+          ${MLAS_SRC_DIR}/qgemm_kernel_sse41.cpp


why are we adding this file here?

FYI, there's a comment saying it's only for Windows:

onnxruntime/onnxruntime/core/mlas/lib/qgemm_kernel_sse41.cpp

Lines 20 to 21 in 1c9a388

// N.B. MSVC does not require turning on SSE 4.1 intrinsics and the current use

// for this code is Windows only, so restrict this kernel to that environment.

and also this ifdef checking for _MSC_VER, which as far as I know implies Windows:

onnxruntime/onnxruntime/core/mlas/lib/platform.cpp

Lines 316 to 329 in 1c9a388

#if defined(_MSC_VER)

//

// Check if the processor supports SSE 4.1 instructions.

//

#ifndef FORCE_GENERIC_ALGORITHMS

if ((Cpuid1[2] & 0x80000) != 0) {

#else // FORCE_GENERIC_ALGORITHMS

if (false) {

#endif // FORCE_GENERIC_ALGORITHMS

this->GemmU8S8Dispatch = &MlasGemmU8S8DispatchSse41;

}

#endif

This flag is necessary for compiling with clang on windows. clang on windows defines _MSC_VER but won't have the necessary sse41 symbols for the gemm op.

ah I see, I didn't realize you all were using Clang on Windows. I'm not sure how much we've tested with that.

one question - would a check with check_cxx_compiler_flag (e.g., check_cxx_compiler_flag("-msse4.1" HAS_INTEL_SSE_4_1)) be sufficient to determine whether the file can be compiled in your particular case? that could be a more automatic way to exclude files requiring specific instruction support which may be unavailable.

this particular change would also enable compilation of qgemm_kernel_sse41.cpp on non-Windows platforms. that may be fine, but it would be good to verify that first. for this PR, it may be simpler to only include qgemm_kernel_sse41.cpp in setup_mlas_source_for_windows().

Checking the compiler flag might work, but I agree it's probably simpler for this PR just to limit the inclusion to the setup for windows -- that's what I've done in the most recent version.

edgchen1 · 2025-09-15T23:22:00Z

cmake/onnxruntime_mlas.cmake

        )

-        if (NOT onnxruntime_ORT_MINIMAL_BUILD)
+        if (NOT onnxruntime_DISABLE_SSE4)


this is not the only place which would need to be modified. see also setup_mlas_source_for_windows():

onnxruntime/cmake/onnxruntime_mlas.cmake

Line 71 in 1c9a388

function(setup_mlas_source_for_windows)

I believe I've updated all locations where these checks need to be done.

edgchen1 · 2025-09-15T23:23:18Z

cmake/CMakeLists.txt

 endif()

+if (onnxruntime_DISABLE_SSE4)
+  add_compile_definitions(DISABLE_SSE4)


can we limit the definition's scope to the onnxruntime_mlas target?

edgchen1 · 2025-09-15T23:30:38Z

cmake/CMakeLists.txt

+endif()
+
+if (onnxruntime_DISABLE_AVX)
+  add_compile_definitions(DISABLE_AVX)


these defines don't match up with the macros in the code. e.g., ORT_DISABLE_AVX. please verify that they work as expected. we don't have CI builds which test these options.

You are right. Sorry. I tested it this morning. Will fix it.

cgaudreau-ubisoft · 2025-09-17T21:27:20Z

Hi! Just a heads up, I will be taking over for @jslap-ubi on the PR. I'm looking into your comments and getting caught up, I hope to address them early next week.

edgchen1 · 2025-09-23T15:57:22Z

cmake/onnxruntime_mlas.cmake

+#  Usually, SIMD instructions in kernel compile fine, and we detect at run-time if they are supported
+#   But on some platforms, even compiling these SIMD specific instructions could fail.
+#   So we provide options to disable compiling these SIMD instructions.
+option(onnxruntime_DISABLE_SSE4 "Disable compiling kernel with SSE4 instructions" OFF)
+option(onnxruntime_DISABLE_AVX "Disable compiling kernel with AVX instructions" OFF)
+option(onnxruntime_DISABLE_AVX2 "Disable compiling kernel with AVX2 instructions" OFF)
+option(onnxruntime_DISABLE_AVX512 "Disable compiling kernel with AVX512 instructions" OFF)
+option(onnxruntime_DISABLE_AMX "Disable compiling kernel with AMX instructions" OFF)


could we keep these option definitions next to the others in cmake/CMakeLists.txt?

edgchen1 · 2025-09-23T15:59:36Z

cmake/onnxruntime_mlas.cmake

+option(onnxruntime_DISABLE_AMX "Disable compiling kernel with AMX instructions" OFF)
+
+if (onnxruntime_DISABLE_SSE4)
+  add_compile_definitions(ORT_DISABLE_SSE4)


please limit the definitions to the onnxruntime_mlas target, e.g., with target_compile_definitions(onnxruntime_mlas PRIVATE ...)

edgchen1 · 2025-09-23T16:06:55Z

cmake/onnxruntime_mlas.cmake


+    if (NOT onnxruntime_DISABLE_SSE4)
+      target_sources(onnxruntime_mlas PRIVATE
+        ${mlas_platform_srcs_sse41}


will mlas_platform_srcs_sse41 have been set to anything here?

You're right, it wouldn't have been set to anything. Fixed.

edgchen1 · 2025-09-23T16:37:36Z

cmake/onnxruntime_mlas.cmake

        set_source_files_properties(${mlas_platform_srcs_sse2} PROPERTIES COMPILE_FLAGS "-msse2")

+        set(mlas_platform_srcs_sse41
+          ${MLAS_SRC_DIR}/qgemm_kernel_sse41.cpp


ah I see, I didn't realize you all were using Clang on Windows. I'm not sure how much we've tested with that.

one question - would a check with check_cxx_compiler_flag (e.g., check_cxx_compiler_flag("-msse4.1" HAS_INTEL_SSE_4_1)) be sufficient to determine whether the file can be compiled in your particular case? that could be a more automatic way to exclude files requiring specific instruction support which may be unavailable.

edgchen1 · 2025-09-23T16:44:50Z

cmake/onnxruntime_mlas.cmake

        set_source_files_properties(${mlas_platform_srcs_sse2} PROPERTIES COMPILE_FLAGS "-msse2")

+        set(mlas_platform_srcs_sse41
+          ${MLAS_SRC_DIR}/qgemm_kernel_sse41.cpp


this particular change would also enable compilation of qgemm_kernel_sse41.cpp on non-Windows platforms. that may be fine, but it would be good to verify that first. for this PR, it may be simpler to only include qgemm_kernel_sse41.cpp in setup_mlas_source_for_windows().

edgchen1 · 2025-09-23T17:11:52Z

onnxruntime/core/mlas/lib/platform.cpp

+#endif // !defined(ORT_DISABLE_AVX2)

 #endif // MLAS_TARGET_AMD64
+#endif  // !defined(ORT_DISABLE_AVX)


currently there is a hierarchy of the code disabled by these macros. e.g., ORT_DISABLE_AVX -> ORT_DISABLE_AVX2 -> ORT_DISABLE_AVX512, where if ORT_DISABLE_AVX is defined, the code controlled by ORT_DISABLE_AVX512 will also be disabled.

this hierarchy is not reflected in the CMake logic that specifies what files are compiled. could it be made consistent?

e.g., flatten the ifdefs here so that they only surround the assignments of kernels that require the corresponding instruction support.

Also, only add the qgemm sse4 file when on windows.

edgchen1

could you elaborate a bit more on your use case(s)? what compilers are you using and what errors do you encounter?

I wonder if we can improve our automatic configuration instead of needing to add options to disable things manually. but it's not yet obvious to me what we're trying to fix here.

edgchen1 · 2025-09-25T17:21:27Z

cmake/onnxruntime_mlas.cmake

+    set(mlas_platform_srcs_sse41
+      ${MLAS_SRC_DIR}/qgemm_kernel_sse41.cpp
+    )
+    set_source_files_properties(${mlas_platform_srcs_sse41} PROPERTIES COMPILE_FLAGS "-msse4.1")


-msse4.1 won't work for MSVC, right? that is the compiler for which setup_mlas_source_for_windows() will be called.

I've removed this, you're right this doesn't make sense for MSVC.

edgchen1 · 2025-09-25T17:52:26Z

onnxruntime/core/mlas/lib/platform.cpp


            if (((Cpuid1[2] & 0x1000) != 0) && ((Cpuid7[1] & 0x20) != 0)) {
-
+#if !defined(ORT_DISABLE_AVX2)


this ifdef block also includes FMA3 kernels. should those also be controlled by the macro to disable AVX2?

You're right, I've re-grouped the FMA3 kernels together and moved them outside the block.

cgaudreau-ubisoft · 2025-10-01T19:47:28Z

#20215 (comment)

encounter

I apologies for the lack of clarity (and delay, I was on holiday). We have a case where we need to compile the project using clang on windows with only SSE2 instructions and not SSE4 (this is to support older machines).

…lock

jslap-ubi requested a review from a team as a code owner April 5, 2024 20:49

yihonglyu reviewed Apr 8, 2024

View reviewed changes

yufenglee previously approved these changes Aug 1, 2024

View reviewed changes

jslap-ubi force-pushed the js/allow-disable-SIMD branch from 2f67b8f to 46f8996 Compare August 1, 2024 19:49

jslap-ubi dismissed yufenglee’s stale review via d0aada7 September 23, 2024 16:58

jslap-ubi force-pushed the js/allow-disable-SIMD branch from 46f8996 to d0aada7 Compare September 23, 2024 16:58

jslap-ubi force-pushed the js/allow-disable-SIMD branch from d0aada7 to 7ca7306 Compare November 29, 2024 21:54

edgchen1 reviewed Sep 12, 2025

View reviewed changes

jslap-ubi force-pushed the js/allow-disable-SIMD branch from 7908f8b to 4568d96 Compare September 15, 2025 20:14

edgchen1 reviewed Sep 15, 2025

View reviewed changes

edgchen1 reviewed Sep 23, 2025

View reviewed changes

jslap-ubi and others added 6 commits September 24, 2025 16:41

Allow configuration template to disable some SIMD.

41b5568

Add cmake options and fix bad ifdef

18f9c69

fix typo

9e115a7

Remove the sse41 flag since it's windows only

d14e4b3

Re-enable sse4 disable flag

7a7dbd0

Flattened the hierarchy of macros for SIMD switches

74adb0e

Also, only add the qgemm sse4 file when on windows.

cgaudreau-ubisoft force-pushed the js/allow-disable-SIMD branch from 07bc7cf to 74adb0e Compare September 24, 2025 21:42

Minor adjustments

d01bcc8

edgchen1 reviewed Sep 25, 2025

View reviewed changes

Removed sse flag for windows and moved FMA3 code out of AVX2 switch b…

a2090b5

…lock


		#if defined(MLAS_TARGET_AMD64)

		#if !defined(ORT_DISABLE_AVX2)

	// N.B. MSVC does not require turning on SSE 4.1 intrinsics and the current use
	// for this code is Windows only, so restrict this kernel to that environment.

	#if defined(_MSC_VER)

	//
	// Check if the processor supports SSE 4.1 instructions.
	//
	#ifndef FORCE_GENERIC_ALGORITHMS
	if ((Cpuid1[2] & 0x80000) != 0) {
	#else // FORCE_GENERIC_ALGORITHMS
	if (false) {
	#endif // FORCE_GENERIC_ALGORITHMS
	this->GemmU8S8Dispatch = &MlasGemmU8S8DispatchSse41;
	}

	#endif


		if (((Cpuid1[2] & 0x1000) != 0) && ((Cpuid7[1] & 0x20) != 0)) {

		#if !defined(ORT_DISABLE_AVX2)

Allow configuration template to disable some SIMD. #20215

Are you sure you want to change the base?

Allow configuration template to disable some SIMD. #20215

Conversation

jslap-ubi commented Apr 5, 2024

Description

Motivation and Context

Uh oh!

jslap-ubi commented Apr 5, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yufenglee commented Apr 9, 2024

Uh oh!

jslap-ubi commented Apr 9, 2024

Uh oh!

yufenglee commented Apr 10, 2024

Uh oh!

jslap-ubi commented Apr 12, 2024

Uh oh!

jslap-ubi commented Apr 23, 2024

Uh oh!

jslap-ubi commented May 1, 2024

Uh oh!

jslap-ubi commented Jul 2, 2024

Uh oh!

yufenglee commented Jul 31, 2024

Uh oh!

yufenglee commented Jul 31, 2024

Uh oh!

yufenglee commented Jul 31, 2024

Uh oh!

azure-pipelines bot commented Jul 31, 2024

Uh oh!

azure-pipelines bot commented Jul 31, 2024

Uh oh!

azure-pipelines bot commented Jul 31, 2024

Uh oh!

yufenglee left a comment

Choose a reason for hiding this comment

Uh oh!

yufenglee commented Aug 1, 2024

Uh oh!

azure-pipelines bot commented Aug 1, 2024

Uh oh!

yufenglee commented Aug 1, 2024

Uh oh!

azure-pipelines bot commented Aug 1, 2024

Uh oh!

yufenglee commented Aug 1, 2024

Uh oh!

yufenglee commented Aug 1, 2024

Uh oh!

yufenglee commented Aug 1, 2024

Uh oh!

azure-pipelines bot commented Aug 1, 2024

Uh oh!

azure-pipelines bot commented Aug 1, 2024

Uh oh!

yufenglee commented Aug 1, 2024

Uh oh!

azure-pipelines bot commented Aug 1, 2024

Uh oh!

jslap-ubi commented Aug 8, 2024

Uh oh!

jslap-ubi commented Aug 23, 2024

Uh oh!

jslap-ubi commented Sep 23, 2024

Uh oh!

jslap-ubi commented Nov 29, 2024

Uh oh!

Choose a reason for hiding this comment

cgaudreau-ubisoft Sep 23, 2025 •

edited

Loading