Linux build fails randomly with "Unable to load shared library 'jitinterface_x64'" error in crossgen

[kg]

Randomly my build on Debian gets stuck in a broken state where all builds fail with this:

  All projects are up-to-date for restore.
./artifacts/bin/Crossgen2Tasks/Debug/net10.0/Microsoft.NET.CrossGen.targets(464,5): error : Unhandled exception. System.DllNotFoundException: Unable to load shared library 'jitinterface_x64' or one of its dependencies. In order to help diagnose loading problems, consider using a tool like strace. If you're using glibc, consider setting the LD_DEBUG environment variable:  [./src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.CoreCLR.sfxproj]
./artifacts/bin/Crossgen2Tasks/Debug/net10.0/Microsoft.NET.CrossGen.targets(464,5): error : ./artifacts/bin/coreclr/linux.x64.Debug/x64/crossgen2/jitinterface_x64.so: cannot open shared object file: No such file or directory [./src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.CoreCLR.sfxproj]
./artifacts/bin/Crossgen2Tasks/Debug/net10.0/Microsoft.NET.CrossGen.targets(464,5): error : ./artifacts/bin/coreclr/linux.x64.Debug/x64/crossgen2/libjitinterface_x64.so: cannot open shared object file: No such file or directory [./src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.CoreCLR.sfxproj]
./artifacts/bin/Crossgen2Tasks/Debug/net10.0/Microsoft.NET.CrossGen.targets(464,5): error : ./artifacts/bin/coreclr/linux.x64.Debug/x64/crossgen2/jitinterface_x64: cannot open shared object file: No such file or directory [./src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.CoreCLR.sfxproj]
./artifacts/bin/Crossgen2Tasks/Debug/net10.0/Microsoft.NET.CrossGen.targets(464,5): error : ./artifacts/bin/coreclr/linux.x64.Debug/x64/crossgen2/libjitinterface_x64: cannot open shared object file: No such file or directory [./src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.CoreCLR.sfxproj]
./artifacts/bin/Crossgen2Tasks/Debug/net10.0/Microsoft.NET.CrossGen.targets(464,5): error :  [./src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.CoreCLR.sfxproj]
./artifacts/bin/Crossgen2Tasks/Debug/net10.0/Microsoft.NET.CrossGen.targets(464,5): error :    at System.Runtime.InteropServices.NativeLibrary.LoadLibraryByName(String libraryName, Assembly assembly, Nullable`1 searchPath, Boolean throwOnError) [./src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.CoreCLR.sfxproj]
./artifacts/bin/Crossgen2Tasks/Debug/net10.0/Microsoft.NET.CrossGen.targets(464,5): error :    at Internal.JitInterface.JitConfigProvider.<>c__DisplayClass5_0.<Initialize>b__0(String libName, Assembly assembly, Nullable`1 searchPath) [./src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.CoreCLR.sfxproj]
./artifacts/bin/Crossgen2Tasks/Debug/net10.0/Microsoft.NET.CrossGen.targets(464,5): error :    at System.Runtime.InteropServices.NativeLibrary.LoadLibraryCallbackStub(String libraryName, Assembly assembly, Boolean hasDllImportSearchPathFlags, UInt32 dllImportSearchPathFlags) [./src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.CoreCLR.sfxproj]
./artifacts/bin/Crossgen2Tasks/Debug/net10.0/Microsoft.NET.CrossGen.targets(464,5): error :    at Internal.JitInterface.CorInfoImpl.Startup(CORINFO_OS os) [./src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.CoreCLR.sfxproj]
./artifacts/bin/Crossgen2Tasks/Debug/net10.0/Microsoft.NET.CrossGen.targets(464,5): error :    at Internal.JitInterface.JitConfigProvider.Initialize(TargetDetails target, IEnumerable`1 jitFlags, IEnumerable`1 parameters, String jitPath) [./src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.CoreCLR.sfxproj]
./artifacts/bin/Crossgen2Tasks/Debug/net10.0/Microsoft.NET.CrossGen.targets(464,5): error :    at ILCompiler.ReadyToRunCodegenCompilationBuilder.ToCompilation() [./src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.CoreCLR.sfxproj]
./artifacts/bin/Crossgen2Tasks/Debug/net10.0/Microsoft.NET.CrossGen.targets(464,5): error :    at ILCompiler.Program.RunSingleCompilation(Dictionary`2 inFilePaths, InstructionSetSupport instructionSetSupport, String compositeRootPath, Dictionary`2 unrootedInputFilePaths, HashSet`1 versionBubbleModulesHash, ReadyToRunCompilerContext typeSystemContext, Logger logger) in ./src/coreclr/tools/aot/crossgen2/Program.cs:line 634 [./src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.CoreCLR.sfxproj]
./artifacts/bin/Crossgen2Tasks/Debug/net10.0/Microsoft.NET.CrossGen.targets(464,5): error :    at ILCompiler.Program.Run() in ./src/coreclr/tools/aot/crossgen2/Program.cs:line 302 [./src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.CoreCLR.sfxproj]
./artifacts/bin/Crossgen2Tasks/Debug/net10.0/Microsoft.NET.CrossGen.targets(464,5): error :    at ILCompiler.Crossgen2RootCommand.<>c__DisplayClass205_0.<.ctor>b__0(ParseResult result) in ./src/coreclr/tools/aot/crossgen2/Crossgen2RootCommand.cs:line 261 [./src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.CoreCLR.sfxproj]
./artifacts/bin/Crossgen2Tasks/Debug/net10.0/Microsoft.NET.CrossGen.targets(464,5): error :    at System.CommandLine.Invocation.InvocationPipeline.Invoke(ParseResult parseResult) [./src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.CoreCLR.sfxproj]
./artifacts/bin/Crossgen2Tasks/Debug/net10.0/Microsoft.NET.CrossGen.targets(464,5): error :    at System.CommandLine.ParseResult.Invoke() [./src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.CoreCLR.sfxproj]
./artifacts/bin/Crossgen2Tasks/Debug/net10.0/Microsoft.NET.CrossGen.targets(464,5): error :    at ILCompiler.Program.Main(String[] args) in ./src/coreclr/tools/aot/crossgen2/Program.cs:line 914 [./src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.CoreCLR.sfxproj]

Blowing away artifacts fixes it temporarily but it tends to come back. This seems like a new issue, I haven't seen it before the last couple weeks (but it's also been a bit since I did lots of runtime builds on linux.)

See https://gist.github.com/kg/eb732af8d9d1338775ca25984f9106ae for LD_DEBUG log output and the location of libjitinterface SO files from my environment.

[kg]

I'd say I hit this multiple times a day now.

[MichalStrehovsky]

I'd say I hit this multiple times a day now.

Do you switch branches by any chance? There were some changes to how crossgen builds (and maybe which binary we run - I haven't paid close attention) in the past week or two and I would not be surprised if they don't play well with incremental build (clean build, switch branch/pull, followed by incremental build).

[kg]

I'd say I hit this multiple times a day now.

Do you switch branches by any chance? There were some changes to how crossgen builds (and maybe which binary we run - I haven't paid close attention) in the past week or two and I would not be surprised if they don't play well with incremental build (clean build, switch branch/pull, followed by incremental build).

This occurs without any branch switches, just by making small changes to the mono runtime and doing a regular build.sh incremental build. Clean build (after nuking artifacts) seems to always work.

So it seems like incremental build even without a branch swap is broken.

[kg]

I diffed find | grep libjitinterface between a good working clean build and a bad build:

good.txt contains 2 additional items:
./artifacts/bin/coreclr/linux.x64.Debug/x64/crossgen2/libjitinterface_x64.so
./artifacts/bin/coreclr/linux.x64.Debug/x64/ilc/libjitinterface_x64.so
bad.txt contains 1 additional items:
./artifacts/obj/coreclr/linux.x64.Debug/tools/aot/jitinterface/CMakeFiles/jitinterface_x64.dir/compiler_depend.internal

I'm not sure why/how those two .so files went missing.

[kg]

It looks like #110471 could have caused this but I'm not sure why. My theory is that JitInterfaceLibraryName could point to a nonexistent file or a slightly incorrect path and then the Include wouldn't find any matching files so the Content wouldn't copy anything to the output directory, if I remember correctly how msbuild works...

[MihaZupan]

I would not be surprised if they don't play well with incremental build (clean build, switch branch/pull, followed by incremental build).

FWIW I'm consistently seeing this with my automated logic that does exactly this (build main, switch to pr, rebuild, test something).

[AaronRobinsonMSFT]

I'm hitting this regularly on macOS too.

@mangod9 or @agocke Can we get someone to look into this?

[AaronRobinsonMSFT]

Error:

Generating native image of System.Private.CoreLib for osx.arm64.Checked. Logging to .../src/runtime/artifacts/bin/coreclr/osx.arm64.Checked/arm64/crossgen2/crossgen2 -o:.../src/runtime/artifacts/bin/coreclr/osx.arm64.Checked/System.Private.CoreLib.dll -r:.../src/runtime/artifacts/bin/coreclr/osx.arm64.Checked/IL/*.dll --targetarch:arm64 --targetos:osx -m:.../src/runtime/artifacts/bin/coreclr/osx.arm64.Checked/StandardOptimizationData.mibc --embed-pgo-data -O --verify-type-and-field-layout --enable-cached-interface-dispatch-support .../src/runtime/artifacts/bin/coreclr/osx.arm64.Checked/IL/System.Private.CoreLib.dll
Unhandled exception. System.DllNotFoundException: Unable to load shared library 'jitinterface_arm64' or one of its dependencies. In order to help diagnose loading problems, consider setting the DYLD_PRINT_LIBRARIES environment variable: dlopen(.../src/runtime/artifacts/bin/coreclr/osx.arm64.Checked/arm64/crossgen2/jitinterface_arm64.dylib, 0x0001): tried: '.../src/runtime/artifacts/bin/coreclr/osx.arm64.Checked/arm64/crossgen2/jitinterface_arm64.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/.../src/runtime/artifacts/bin/coreclr/osx.arm64.Checked/arm64/crossgen2/jitinterface_arm64.dylib' (no such file), '.../src/runtime/artifacts/bin/coreclr/osx.arm64.Checked/arm64/crossgen2/jitinterface_arm64.dylib' (no such file)
dlopen(jitinterface_arm64.dylib, 0x0001): tried: 'jitinterface_arm64.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSjitinterface_arm64.dylib' (no such file), '/usr/lib/jitinterface_arm64.dylib' (no such file, not in dyld cache), 'jitinterface_arm64.dylib' (no such file)
dlopen(.../src/runtime/artifacts/bin/coreclr/osx.arm64.Checked/arm64/crossgen2/libjitinterface_arm64.dylib, 0x0001): tried: '.../src/runtime/artifacts/bin/coreclr/osx.arm64.Checked/arm64/crossgen2/libjitinterface_arm64.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/.../src/runtime/artifacts/bin/coreclr/osx.arm64.Checked/arm64/crossgen2/libjitinterface_arm64.dylib' (no such file), '.../src/runtime/artifacts/bin/coreclr/osx.arm64.Checked/arm64/crossgen2/libjitinterface_arm64.dylib' (no such file)
dlopen(libjitinterface_arm64.dylib, 0x0001): tried: 'libjitinterface_arm64.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibjitinterface_arm64.dylib' (no such file), '/usr/lib/libjitinterface_arm64.dylib' (no such file, not in dyld cache), 'libjitinterface_arm64.dylib' (no such file)
dlopen(.../src/runtime/artifacts/bin/coreclr/osx.arm64.Checked/arm64/crossgen2/jitinterface_arm64, 0x0001): tried: '.../src/runtime/artifacts/bin/coreclr/osx.arm64.Checked/arm64/crossgen2/jitinterface_arm64' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/.../src/runtime/artifacts/bin/coreclr/osx.arm64.Checked/arm64/crossgen2/jitinterface_arm64' (no such file), '.../src/runtime/artifacts/bin/coreclr/osx.arm64.Checked/arm64/crossgen2/jitinterface_arm64' (no such file)
dlopen(jitinterface_arm64, 0x0001): tried: 'jitinterface_arm64' (no such file), '/System/Volumes/Preboot/Cryptexes/OSjitinterface_arm64' (no such file), '/usr/lib/jitinterface_arm64' (no such file, not in dyld cache), 'jitinterface_arm64' (no such file)
dlopen(.../src/runtime/artifacts/bin/coreclr/osx.arm64.Checked/arm64/crossgen2/libjitinterface_arm64, 0x0001): tried: '.../src/runtime/artifacts/bin/coreclr/osx.arm64.Checked/arm64/crossgen2/libjitinterface_arm64' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/.../src/runtime/artifacts/bin/coreclr/osx.arm64.Checked/arm64/crossgen2/libjitinterface_arm64' (no such file), '.../src/runtime/artifacts/bin/coreclr/osx.arm64.Checked/arm64/crossgen2/libjitinterface_arm64' (no such file)
dlopen(libjitinterface_arm64, 0x0001): tried: 'libjitinterface_arm64' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibjitinterface_arm64' (no such file), '/usr/lib/libjitinterface_arm64.dylib' (no such file, not in dyld cache), 'libjitinterface_arm64.dylib' (no such file)
at System.Runtime.InteropServices.NativeLibrary.LoadLibraryByName(String libraryName, Assembly assembly, Nullable`1 searchPath, Boolean throwOnError)
at Internal.JitInterface.JitConfigProvider.<>c__DisplayClass5_0.<Initialize>b__0(String libName, Assembly assembly, Nullable`1 searchPath)
at System.Runtime.InteropServices.NativeLibrary.LoadLibraryCallbackStub(String libraryName, Assembly assembly, Boolean hasDllImportSearchPathFlags, UInt32 dllImportSearchPathFlags)
at Internal.JitInterface.CorInfoImpl.Startup(CORINFO_OS os)
at ILCompiler.ReadyToRunCodegenCompilationBuilder.ToCompilation()
at ILCompiler.Program.RunSingleCompilation(Dictionary`2 inFilePaths, InstructionSetSupport instructionSetSupport, String compositeRootPath, Dictionary`2 unrootedInputFilePaths, HashSet`1 versionBubbleModulesHash, ReadyToRunCompilerContext typeSystemContext, Logger logger) in .../src/runtime/src/coreclr/tools/aot/crossgen2/Program.cs:line 635
at ILCompiler.Program.Run() in .../src/runtime/src/coreclr/tools/aot/crossgen2/Program.cs:line 302
at ILCompiler.Crossgen2RootCommand.<>c__DisplayClass208_0.<.ctor>b__0(ParseResult result) in .../src/runtime/src/coreclr/tools/aot/crossgen2/Crossgen2RootCommand.cs:line 264
at System.CommandLine.Invocation.InvocationPipeline.Invoke(ParseResult parseResult)
at ILCompiler.Program.Main(String[] args) in .../src/runtime/src/coreclr/tools/aot/crossgen2/Program.cs:line 915
.../tmp9308f3fd9f054dc9a80d876922c7efbe.exec.cmd: line 2: 29678 Abort trap: 6
.../src/runtime/artifacts/bin/coreclr/osx.arm64.Checked/arm64/crossgen2/crossgen2 -o:.../src/runtime/artifacts/bin/coreclr/osx.arm64.Checked/System.Private.CoreLib.dll -r:.../src/runtime/artifacts/bin/coreclr/osx.arm64.Checked/IL/*.dll --targetarch:arm64 --targetos:osx -m:.../src/runtime/artifacts/bin/coreclr/osx.arm64.Checked/StandardOptimizationData.mibc --embed-pgo-data -O --verify-type-and-field-layout --enable-cached-interface-dispatch-support .../src/runtime/artifacts/bin/coreclr/osx.arm64.Checked/IL/System.Private.CoreLib.dll

[AaronRobinsonMSFT]

It looks like #110471 could have caused this but I'm not sure why. My theory is that JitInterfaceLibraryName could point to a nonexistent file or a slightly incorrect path and then the Include wouldn't find any matching files so the Content wouldn't copy anything to the output directory, if I remember correctly how msbuild works...

This looks close to or is the issue. What I'm seeing is the version under .../artifacts/bin/coreclr/osx.arm64.Checked/arm64/crossgen2, which appears to be single file, versus the one under .../artifacts/bin/coreclr/osx.arm64.Checked/crossgen2, which is not single file, is used and fails on my macOS. If I copy libjitinterface_arm64.dylib and libclrjit_universal_arm64_arm64.dylib from the "non-single file" directory into the "single file" directory then crossgen2 works successfully and I get a build.

/cc @steveisok and @grendello for potential impact from the linked PR.

[steveisok]

I got this to happen on the second pass of the native runtime build. You have to run the second pass w/ libs in order to get it to error at all.

./build.sh -subset clr+clr.hosts -rc Checked -arch arm64 -hc Release --cmakeargs "-DCLR_CMAKE_APPLE_DSYM=TRUE"
 
./build.sh -subset Libs -configuration Release -arch arm64 -restore -build -rc Checked -hc Release -runtimeFlavor CoreCLR

When you run the native build the first time, the dylibs are all in the single file crossgen folder. On subsequent runs they are cleaned.

[steveisok]

@jkoritzinsky should the libs always publish with crossgen2 single file?