Unmanaged memory leak in clrjit (amd64) in .NET 10 RC1

[rosslovas]

Description

We've been experiencing a significant unmanaged memory leak when testing our application with .NET 10 RC1 (on Linux, amd64). Here's an example graph showing actual memory usage steadily growing while the managed heap remains steady:

I ran the application for a few hours with heaptrack. It identified the leaked allocations were coming from libclrjit.so.

To get more details, I compiled the runtime myself (on branch release/10.0-rc1) with native symbols and repeated the heaptrack experiment, which pointed to this line:

LinearScan::LinearScan(Compiler*)
  at /runtime/src/coreclr/jit/lsra.cpp:878
  in /home/octoross/server/libclrjit.so

Line 878 in coreclr/jit/lsra.cpp

I've included more info from heaptrack including full stacktraces below under "Other information".

Reproduction Steps

I haven't managed to reproduce the problem in a small project at this stage.

Expected behavior

Unmanaged memory consumption does not continuously grow

Actual behavior

Significant amount of unmanaged memory allocated and never reclaimed (several megabytes per minute in our test scenario)

Regression?

We do not experience this memory leak in the latest releases of .NET 8 or 9.

Known Workarounds

No response

Configuration

Linux, amd64 (Debian 12)

Other information

Memory consumption from a ~90 minute run graphed in heaptrack_gui:

Expand for relevant `heaptrack_print --print-leaks true` output from that same run

MEMORY LEAKS
238.22M leaked over 1044826 calls from
LinearScan::LinearScan(Compiler*)
  at /runtime/src/coreclr/jit/lsra.cpp:878
  in /home/octoross/server/libclrjit.so
17.37M leaked over 76174 calls from:
    getLinearScanAllocator(Compiler*)
      at /runtime/src/coreclr/jit/lsra.cpp:829
      in /home/octoross/server/libclrjit.so
    Compiler::compCompile(void**, unsigned int*, JitFlags*)
      at /runtime/src/coreclr/jit/compiler.cpp:4981
      in /home/octoross/server/libclrjit.so
    Compiler::compCompileHelper(CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*)
      at /runtime/src/coreclr/jit/compiler.cpp:7086
      in /home/octoross/server/libclrjit.so
    Compiler::compCompile(CORINFO_MODULE_STRUCT_*, void**, unsigned int*, JitFlags*)::$_7::operator()(Compiler::compCompile(CORINFO_MODULE_STRUCT_*, void**, unsigned int*, JitFlags*)::__JITParam*) const
      at /runtime/src/coreclr/jit/compiler.cpp:6274
      in /home/octoross/server/libclrjit.so
    Compiler::compCompile(CORINFO_MODULE_STRUCT_*, void**, unsigned int*, JitFlags*)
      at /runtime/src/coreclr/jit/compiler.cpp:6293
    jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::$_9::operator()(jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::__JITParam*) const::{lambda(jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::$_9::operator()(jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::__JITParam*) const::__JITParam*)#1}::operator()(jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::$_9::operator()(jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::__JITParam*) const::__JITParam*) const
      at /runtime/src/coreclr/jit/compiler.cpp:7732
      in /home/octoross/server/libclrjit.so
    jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::$_9::operator()(jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::__JITParam*) const
      at /runtime/src/coreclr/jit/compiler.cpp:7756
    jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)
      at /runtime/src/coreclr/jit/compiler.cpp:7758
    CILJit::compileMethod(ICorJitInfo*, CORINFO_METHOD_INFO*, unsigned int, unsigned char**, unsigned int*)
      at /runtime/src/coreclr/jit/ee_il_dll.cpp:302
      in /home/octoross/server/libclrjit.so
    invokeCompileMethod(EECodeGenManager*, CEECodeGenInfo*, unsigned char**, unsigned int*, NativeCodeVersion)
      at /runtime/src/coreclr/vm/jitinterface.cpp:12856
      in /home/octoross/server/libcoreclr.so
    UnsafeJitFunctionWorker(EECodeGenManager*, CEECodeGenInfo*, NativeCodeVersion, unsigned int*)
      at /runtime/src/coreclr/vm/jitinterface.cpp:13079
    UnsafeJitFunction(PrepareCodeConfig*, COR_ILMETHOD_DECODER*, bool*, bool*, unsigned int*)
      at /runtime/src/coreclr/vm/jitinterface.cpp:13410
    MethodDesc::JitCompileCodeLocked(PrepareCodeConfig*, COR_ILMETHOD_DECODER*, ListLockEntryBase<>*, unsigned int*, bool*)
      at /runtime/src/coreclr/vm/prestub.cpp:921
      in /home/octoross/server/libcoreclr.so
    MethodDesc::JitCompileCodeLockedEventWrapper(PrepareCodeConfig*, ListLockEntryBase<>*)
      at /runtime/src/coreclr/vm/prestub.cpp:819
      in /home/octoross/server/libcoreclr.so
    MethodDesc::JitCompileCode(PrepareCodeConfig*)
      at /runtime/src/coreclr/vm/prestub.cpp:694
      in /home/octoross/server/libcoreclr.so
    MethodDesc::PrepareILBasedCode(PrepareCodeConfig*)
      at /runtime/src/coreclr/vm/prestub.cpp:424
      in /home/octoross/server/libcoreclr.so
    TieredCompilationManager::CompileCodeVersion(NativeCodeVersion)
      at /runtime/src/coreclr/vm/tieredcompilation.cpp:956
      in /home/octoross/server/libcoreclr.so
    TieredCompilationManager::OptimizeMethod(NativeCodeVersion)
      at /runtime/src/coreclr/vm/tieredcompilation.cpp:933
      in /home/octoross/server/libcoreclr.so
    TieredCompilationManager::DoBackgroundWork(long*, long, long)
      at /runtime/src/coreclr/vm/tieredcompilation.cpp:819
    TieredCompilationManager::BackgroundWorkerStart()
      at /runtime/src/coreclr/vm/tieredcompilation.cpp:533
      in /home/octoross/server/libcoreclr.so
    TieredCompilationManager::BackgroundWorkerBootstrapper1(void*)
      at /runtime/src/coreclr/vm/tieredcompilation.cpp:483
      in /home/octoross/server/libcoreclr.so
    ManagedThreadBase_DispatchInner(ManagedThreadCallState*)
      at /runtime/src/coreclr/vm/threads.cpp:6830
      in /home/octoross/server/libcoreclr.so
    ManagedThreadBase_DispatchMiddle(ManagedThreadCallState*)
      at /runtime/src/coreclr/vm/threads.cpp:6874
    ManagedThreadBase_DispatchOuter(ManagedThreadCallState*)::$_5::operator()(ManagedThreadBase_DispatchOuter(ManagedThreadCallState*)::TryArgs*) const::{lambda(Param*)#1}::operator()(Param*) const
      at /runtime/src/coreclr/vm/threads.cpp:7032
    ManagedThreadBase_DispatchOuter(ManagedThreadCallState*)::$_5::operator()(ManagedThreadBase_DispatchOuter(ManagedThreadCallState*)::TryArgs*) const
      at /runtime/src/coreclr/vm/threads.cpp:7034
    ManagedThreadBase_DispatchOuter(ManagedThreadCallState*)
      at /runtime/src/coreclr/vm/threads.cpp:7058
    ManagedThreadBase::KickOff(void (*)(void*), void*)
      at /runtime/src/coreclr/vm/threads.cpp:7072
    TieredCompilationManager::BackgroundWorkerBootstrapper0(void*)
      at /runtime/src/coreclr/vm/tieredcompilation.cpp:464
      in /home/octoross/server/libcoreclr.so
    CorUnix::CPalThread::ThreadEntry(void*)
      at /runtime/src/coreclr/pal/src/thread/thread.cpp:1622
      in /home/octoross/server/libcoreclr.so
    0x7fffff3351f4
      in /lib/x86_64-linux-gnu/libc.so.6
    __clone
      in /lib/x86_64-linux-gnu/libc.so.6
13.67M leaked over 59950 calls from:
    getLinearScanAllocator(Compiler*)
      at /runtime/src/coreclr/jit/lsra.cpp:829
      in /home/octoross/server/libclrjit.so
    Compiler::compCompile(void**, unsigned int*, JitFlags*)
      at /runtime/src/coreclr/jit/compiler.cpp:4981
      in /home/octoross/server/libclrjit.so
    Compiler::compCompileHelper(CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*)
      at /runtime/src/coreclr/jit/compiler.cpp:7086
      in /home/octoross/server/libclrjit.so
    Compiler::compCompile(CORINFO_MODULE_STRUCT_*, void**, unsigned int*, JitFlags*)::$_7::operator()(Compiler::compCompile(CORINFO_MODULE_STRUCT_*, void**, unsigned int*, JitFlags*)::__JITParam*) const
      at /runtime/src/coreclr/jit/compiler.cpp:6274
      in /home/octoross/server/libclrjit.so
    Compiler::compCompile(CORINFO_MODULE_STRUCT_*, void**, unsigned int*, JitFlags*)
      at /runtime/src/coreclr/jit/compiler.cpp:6293
    jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::$_9::operator()(jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::__JITParam*) const::{lambda(jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::$_9::operator()(jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::__JITParam*) const::__JITParam*)#1}::operator()(jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::$_9::operator()(jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::__JITParam*) const::__JITParam*) const
      at /runtime/src/coreclr/jit/compiler.cpp:7732
      in /home/octoross/server/libclrjit.so
    jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::$_9::operator()(jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::__JITParam*) const
      at /runtime/src/coreclr/jit/compiler.cpp:7756
    jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)
      at /runtime/src/coreclr/jit/compiler.cpp:7758
    CILJit::compileMethod(ICorJitInfo*, CORINFO_METHOD_INFO*, unsigned int, unsigned char**, unsigned int*)
      at /runtime/src/coreclr/jit/ee_il_dll.cpp:302
      in /home/octoross/server/libclrjit.so
    invokeCompileMethod(EECodeGenManager*, CEECodeGenInfo*, unsigned char**, unsigned int*, NativeCodeVersion)
      at /runtime/src/coreclr/vm/jitinterface.cpp:12856
      in /home/octoross/server/libcoreclr.so
    UnsafeJitFunctionWorker(EECodeGenManager*, CEECodeGenInfo*, NativeCodeVersion, unsigned int*)
      at /runtime/src/coreclr/vm/jitinterface.cpp:13079
    UnsafeJitFunction(PrepareCodeConfig*, COR_ILMETHOD_DECODER*, bool*, bool*, unsigned int*)
      at /runtime/src/coreclr/vm/jitinterface.cpp:13410
    MethodDesc::JitCompileCodeLocked(PrepareCodeConfig*, COR_ILMETHOD_DECODER*, ListLockEntryBase<>*, unsigned int*, bool*)
      at /runtime/src/coreclr/vm/prestub.cpp:921
      in /home/octoross/server/libcoreclr.so
    MethodDesc::JitCompileCodeLockedEventWrapper(PrepareCodeConfig*, ListLockEntryBase<>*)
      at /runtime/src/coreclr/vm/prestub.cpp:819
      in /home/octoross/server/libcoreclr.so
    MethodDesc::JitCompileCode(PrepareCodeConfig*)
      at /runtime/src/coreclr/vm/prestub.cpp:694
      in /home/octoross/server/libcoreclr.so
    MethodDesc::PrepareILBasedCode(PrepareCodeConfig*)
      at /runtime/src/coreclr/vm/prestub.cpp:424
      in /home/octoross/server/libcoreclr.so
    MethodDesc::PrepareCode(PrepareCodeConfig*)
      at /runtime/src/coreclr/vm/prestub.cpp:312
      in /home/octoross/server/libcoreclr.so
    MethodDesc::PrepareInitialCode(CallerGCMode)
      at /runtime/src/coreclr/vm/prestub.cpp:302
    MethodDesc::DoPrestub(MethodTable*, CallerGCMode)
      at /runtime/src/coreclr/vm/prestub.cpp:2278
    ReflectionInvocation_CompileMethod
      at /runtime/src/coreclr/vm/reflectioninvocation.cpp:1209
      in /home/octoross/server/libcoreclr.so
    0x7fff949d37b8
      in ??
    0x7fff934b33bd
      in ??
    0x7fff958aae45
      in ??
    0x7fff958a8d66
      in ??
    0x7fff95886c50
      in ??
    0x7fff945b7f1b
      in ??
    0x7fff945bf750
      in ??
    0x7fff945e28ba
      in ??
    0x7fff945e9d5b
      in ??
    0x7fff945e999f
      in ??
    0x7fff945c930e
      in ??
    0x7fff945c8fe6
      in ??
    0x7fff945c3684
      in ??
    0x7fff945c2dc7
      in ??
    0x7fff945c24bf
      in ??
    0x7fff945caa82
      in ??
    0x7fff945e99b6
      in ??
    0x7fff945c930e
      in ??
    0x7fff945c8fe6
      in ??
    0x7fff945e8e3c
      in ??
    0x7fff945c3612
      in ??
    0x7fff945c2dc7
      in ??
    0x7fff945c24bf
      in ??
    0x7fff9497e306
      in ??
    0x7fff95fd9a41
      in ??
    0x7fff8a904fce
      in ??
    0x7fff945e99b6
      in ??
    0x7fff945c930e
      in ??
    0x7fff945c8fe6
      in ??
    0x7fff945c3684
      in ??
    0x7fff945c2dc7
      in ??
    0x7fff945c24bf
      in ??
    0x7fff958bb3e6
      in ??
    0x7fff945cbdce
      in ??
    0x7fff945cba5a
      in ??
    0x7fff945c8fe6
      in ??
    0x7fff945c3684
      in ??
    0x7fff945c2dc7
      in ??
    0x7fff945c24bf
      in ??
    0x7fff945caa82
      in ??
    0x7fff945e99b6
      in ??
    0x7fff945c930e
      in ??
    0x7fff945c8fe6
      in ??
    0x7fff945c3684
      in ??
    0x7fff945c2dc7
      in ??
    0x7fff945c24bf
      in ??
    0x7fff9497e306
      in ??
11.00M leaked over 48266 calls from:
    getLinearScanAllocator(Compiler*)
      at /runtime/src/coreclr/jit/lsra.cpp:829
      in /home/octoross/server/libclrjit.so
    Compiler::compCompile(void**, unsigned int*, JitFlags*)
      at /runtime/src/coreclr/jit/compiler.cpp:4981
      in /home/octoross/server/libclrjit.so
    Compiler::compCompileHelper(CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*)
      at /runtime/src/coreclr/jit/compiler.cpp:7086
      in /home/octoross/server/libclrjit.so
    Compiler::compCompile(CORINFO_MODULE_STRUCT_*, void**, unsigned int*, JitFlags*)::$_7::operator()(Compiler::compCompile(CORINFO_MODULE_STRUCT_*, void**, unsigned int*, JitFlags*)::__JITParam*) const
      at /runtime/src/coreclr/jit/compiler.cpp:6274
      in /home/octoross/server/libclrjit.so
    Compiler::compCompile(CORINFO_MODULE_STRUCT_*, void**, unsigned int*, JitFlags*)
      at /runtime/src/coreclr/jit/compiler.cpp:6293
    jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::$_9::operator()(jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::__JITParam*) const::{lambda(jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::$_9::operator()(jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::__JITParam*) const::__JITParam*)#1}::operator()(jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::$_9::operator()(jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::__JITParam*) const::__JITParam*) const
      at /runtime/src/coreclr/jit/compiler.cpp:7732
      in /home/octoross/server/libclrjit.so
    jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::$_9::operator()(jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::__JITParam*) const
      at /runtime/src/coreclr/jit/compiler.cpp:7756
    jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)
      at /runtime/src/coreclr/jit/compiler.cpp:7758
    CILJit::compileMethod(ICorJitInfo*, CORINFO_METHOD_INFO*, unsigned int, unsigned char**, unsigned int*)
      at /runtime/src/coreclr/jit/ee_il_dll.cpp:302
      in /home/octoross/server/libclrjit.so
    invokeCompileMethod(EECodeGenManager*, CEECodeGenInfo*, unsigned char**, unsigned int*, NativeCodeVersion)
      at /runtime/src/coreclr/vm/jitinterface.cpp:12856
      in /home/octoross/server/libcoreclr.so
    UnsafeJitFunctionWorker(EECodeGenManager*, CEECodeGenInfo*, NativeCodeVersion, unsigned int*)
      at /runtime/src/coreclr/vm/jitinterface.cpp:13079
    UnsafeJitFunction(PrepareCodeConfig*, COR_ILMETHOD_DECODER*, bool*, bool*, unsigned int*)
      at /runtime/src/coreclr/vm/jitinterface.cpp:13410
    MethodDesc::JitCompileCodeLocked(PrepareCodeConfig*, COR_ILMETHOD_DECODER*, ListLockEntryBase<>*, unsigned int*, bool*)
      at /runtime/src/coreclr/vm/prestub.cpp:921
      in /home/octoross/server/libcoreclr.so
    MethodDesc::JitCompileCodeLockedEventWrapper(PrepareCodeConfig*, ListLockEntryBase<>*)
      at /runtime/src/coreclr/vm/prestub.cpp:819
      in /home/octoross/server/libcoreclr.so
    MethodDesc::JitCompileCode(PrepareCodeConfig*)
      at /runtime/src/coreclr/vm/prestub.cpp:694
      in /home/octoross/server/libcoreclr.so
    MethodDesc::PrepareILBasedCode(PrepareCodeConfig*)
      at /runtime/src/coreclr/vm/prestub.cpp:424
      in /home/octoross/server/libcoreclr.so
    MethodDesc::PrepareCode(PrepareCodeConfig*)
      at /runtime/src/coreclr/vm/prestub.cpp:312
      in /home/octoross/server/libcoreclr.so
    MethodDesc::PrepareInitialCode(CallerGCMode)
      at /runtime/src/coreclr/vm/prestub.cpp:302
    MethodDesc::DoPrestub(MethodTable*, CallerGCMode)
      at /runtime/src/coreclr/vm/prestub.cpp:2278
    ReflectionInvocation_CompileMethod
      at /runtime/src/coreclr/vm/reflectioninvocation.cpp:1209
      in /home/octoross/server/libcoreclr.so
    0x7fff949d37b8
      in ??
    0x7fff934b33bd
      in ??
    0x7fff958aae45
      in ??
    0x7fff958a8d66
      in ??
    0x7fff95886c50
      in ??
    0x7fff945b7f1b
      in ??
    0x7fff945bf750
      in ??
    0x7fff945e28ba
      in ??
    0x7fff945e9d5b
      in ??
    0x7fff945e999f
      in ??
    0x7fff95878b59
      in ??
    0x7fff945c8fe6
      in ??
    0x7fff945e8e3c
      in ??
    0x7fff945c3612
      in ??
    0x7fff945c2dc7
      in ??
    0x7fff945c24bf
      in ??
    0x7fff945caa82
      in ??
    0x7fff945e99b6
      in ??
    0x7fff945c930e
      in ??
    0x7fff95878b59
      in ??
    0x7fff945c8fe6
      in ??
    0x7fff945e8e3c
      in ??
    0x7fff945c3612
      in ??
    0x7fff945c2dc7
      in ??
    0x7fff945c24bf
      in ??
    0x7fff9497e306
      in ??
    0x7fff945c6952
      in ??
    0x7fff945c719c
      in ??
    0x7fff95d1c708
      in ??
    0x7fff95d4fe75
      in ??
    0x7fff95dbe037
      in ??
    0x7fff9499567b
      in ??
    0x7fff95fbd96f
      in ??
    0x7fff9499a31d
      in ??
    0x7fff949811cb
      in ??
    0x7fff9583a761
      in ??
    0x7fff95d1cdbc
      in ??
    0x7fff9499567b
      in ??
    0x7fff95d5d481
      in ??
    0x7fff9499a31d
      in ??
    0x7fff949811cb
      in ??
    0x7fff95d1bd71
      in ??
    0x7fff9499567b
      in ??
    0x7fff95d5d13f
      in ??
    0x7fff9499a31d
      in ??
    0x7fff949811cb
      in ??
    0x7fff9583a761
      in ??
7.92M leaked over 34755 calls from:
    getLinearScanAllocator(Compiler*)
      at /runtime/src/coreclr/jit/lsra.cpp:829
      in /home/octoross/server/libclrjit.so
    Compiler::compCompile(void**, unsigned int*, JitFlags*)
      at /runtime/src/coreclr/jit/compiler.cpp:4981
      in /home/octoross/server/libclrjit.so
    Compiler::compCompileHelper(CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*)
      at /runtime/src/coreclr/jit/compiler.cpp:7086
      in /home/octoross/server/libclrjit.so
    Compiler::compCompile(CORINFO_MODULE_STRUCT_*, void**, unsigned int*, JitFlags*)::$_7::operator()(Compiler::compCompile(CORINFO_MODULE_STRUCT_*, void**, unsigned int*, JitFlags*)::__JITParam*) const
      at /runtime/src/coreclr/jit/compiler.cpp:6274
      in /home/octoross/server/libclrjit.so
    Compiler::compCompile(CORINFO_MODULE_STRUCT_*, void**, unsigned int*, JitFlags*)
      at /runtime/src/coreclr/jit/compiler.cpp:6293
    jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::$_9::operator()(jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::__JITParam*) const::{lambda(jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::$_9::operator()(jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::__JITParam*) const::__JITParam*)#1}::operator()(jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::$_9::operator()(jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::__JITParam*) const::__JITParam*) const
      at /runtime/src/coreclr/jit/compiler.cpp:7732
      in /home/octoross/server/libclrjit.so
    jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::$_9::operator()(jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::__JITParam*) const
      at /runtime/src/coreclr/jit/compiler.cpp:7756
    jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)
      at /runtime/src/coreclr/jit/compiler.cpp:7758
    CILJit::compileMethod(ICorJitInfo*, CORINFO_METHOD_INFO*, unsigned int, unsigned char**, unsigned int*)
      at /runtime/src/coreclr/jit/ee_il_dll.cpp:302
      in /home/octoross/server/libclrjit.so
    invokeCompileMethod(EECodeGenManager*, CEECodeGenInfo*, unsigned char**, unsigned int*, NativeCodeVersion)
      at /runtime/src/coreclr/vm/jitinterface.cpp:12856
      in /home/octoross/server/libcoreclr.so
    UnsafeJitFunctionWorker(EECodeGenManager*, CEECodeGenInfo*, NativeCodeVersion, unsigned int*)
      at /runtime/src/coreclr/vm/jitinterface.cpp:13079
    UnsafeJitFunction(PrepareCodeConfig*, COR_ILMETHOD_DECODER*, bool*, bool*, unsigned int*)
      at /runtime/src/coreclr/vm/jitinterface.cpp:13410
    MethodDesc::JitCompileCodeLocked(PrepareCodeConfig*, COR_ILMETHOD_DECODER*, ListLockEntryBase<>*, unsigned int*, bool*)
      at /runtime/src/coreclr/vm/prestub.cpp:921
      in /home/octoross/server/libcoreclr.so
    MethodDesc::JitCompileCodeLockedEventWrapper(PrepareCodeConfig*, ListLockEntryBase<>*)
      at /runtime/src/coreclr/vm/prestub.cpp:819
      in /home/octoross/server/libcoreclr.so
    MethodDesc::JitCompileCode(PrepareCodeConfig*)
      at /runtime/src/coreclr/vm/prestub.cpp:694
      in /home/octoross/server/libcoreclr.so
    MethodDesc::PrepareILBasedCode(PrepareCodeConfig*)
      at /runtime/src/coreclr/vm/prestub.cpp:424
      in /home/octoross/server/libcoreclr.so
    MethodDesc::PrepareCode(PrepareCodeConfig*)
      at /runtime/src/coreclr/vm/prestub.cpp:312
      in /home/octoross/server/libcoreclr.so
    MethodDesc::PrepareInitialCode(CallerGCMode)
      at /runtime/src/coreclr/vm/prestub.cpp:302
    MethodDesc::DoPrestub(MethodTable*, CallerGCMode)
      at /runtime/src/coreclr/vm/prestub.cpp:2278
    ReflectionInvocation_CompileMethod
      at /runtime/src/coreclr/vm/reflectioninvocation.cpp:1209
      in /home/octoross/server/libcoreclr.so
    0x7fff949d37b8
      in ??
    0x7fff934b33bd
      in ??
    0x7fff958aae45
      in ??
    0x7fff958a8d66
      in ??
    0x7fff95886c50
      in ??
    0x7fff945b7f1b
      in ??
    0x7fff945bf750
      in ??
    0x7fff945e28ba
      in ??
    0x7fff945e9d5b
      in ??
    0x7fff945e999f
      in ??
    0x7fff945c930e
      in ??
    0x7fff945c8fe6
      in ??
    0x7fff945c3684
      in ??
    0x7fff945c2dc7
      in ??
    0x7fff945c24bf
      in ??
    0x7fff945caa82
      in ??
    0x7fff945e99b6
      in ??
    0x7fff945c930e
      in ??
    0x7fff945c8fe6
      in ??
    0x7fff945e8e3c
      in ??
    0x7fff945c3612
      in ??
    0x7fff945c2dc7
      in ??
    0x7fff945c24bf
      in ??
    0x7fff9497e306
      in ??
    0x7fff95db99df
      in ??
    0x7fff9605294c
      in ??
    0x7fff95d60587
      in ??
    0x7fff96051b5b
      in ??
    0x7fff95dba5b5
      in ??
    0x7fff960523ba
      in ??
    0x7fff963cef5f
      in ??
    0x7fff963b9c7c
      in ??
    0x7fff963b94a7
      in ??
    0x7fff964ba9cb
      in ??
    0x7fff9630c8c7
      in ??
    0x7fff9632da6b
      in ??
    0x7fff9630c222
      in ??
    0x7fff964ba5a6
      in ??
    0x7fff95fff4b2
      in ??
    0x7fff963ab19d
      in ??
    0x7fff963aa75a
      in ??
    0x7fff964b9a7e
      in ??
    0x7fff95fff4b2
      in ??
    0x7fff9602e98b
      in ??
    0x7fff9636404e
      in ??
    0x7fff9644ad9b
      in ??
    0x7fff9499567b
      in ??
5.43M leaked over 23832 calls from:
    getLinearScanAllocator(Compiler*)
      at /runtime/src/coreclr/jit/lsra.cpp:829
      in /home/octoross/server/libclrjit.so
    Compiler::compCompile(void**, unsigned int*, JitFlags*)
      at /runtime/src/coreclr/jit/compiler.cpp:4981
      in /home/octoross/server/libclrjit.so
    Compiler::compCompileHelper(CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*)
      at /runtime/src/coreclr/jit/compiler.cpp:7086
      in /home/octoross/server/libclrjit.so
    Compiler::compCompile(CORINFO_MODULE_STRUCT_*, void**, unsigned int*, JitFlags*)::$_7::operator()(Compiler::compCompile(CORINFO_MODULE_STRUCT_*, void**, unsigned int*, JitFlags*)::__JITParam*) const
      at /runtime/src/coreclr/jit/compiler.cpp:6274
      in /home/octoross/server/libclrjit.so
    Compiler::compCompile(CORINFO_MODULE_STRUCT_*, void**, unsigned int*, JitFlags*)
      at /runtime/src/coreclr/jit/compiler.cpp:6293
    jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::$_9::operator()(jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::__JITParam*) const::{lambda(jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::$_9::operator()(jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::__JITParam*) const::__JITParam*)#1}::operator()(jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::$_9::operator()(jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::__JITParam*) const::__JITParam*) const
      at /runtime/src/coreclr/jit/compiler.cpp:7732
      in /home/octoross/server/libclrjit.so
    jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::$_9::operator()(jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)::__JITParam*) const
      at /runtime/src/coreclr/jit/compiler.cpp:7756
    jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*)
      at /runtime/src/coreclr/jit/compiler.cpp:7758
    CILJit::compileMethod(ICorJitInfo*, CORINFO_METHOD_INFO*, unsigned int, unsigned char**, unsigned int*)
      at /runtime/src/coreclr/jit/ee_il_dll.cpp:302
      in /home/octoross/server/libclrjit.so
    invokeCompileMethod(EECodeGenManager*, CEECodeGenInfo*, unsigned char**, unsigned int*, NativeCodeVersion)
      at /runtime/src/coreclr/vm/jitinterface.cpp:12856
      in /home/octoross/server/libcoreclr.so
    UnsafeJitFunctionWorker(EECodeGenManager*, CEECodeGenInfo*, NativeCodeVersion, unsigned int*)
      at /runtime/src/coreclr/vm/jitinterface.cpp:13079
    UnsafeJitFunction(PrepareCodeConfig*, COR_ILMETHOD_DECODER*, bool*, bool*, unsigned int*)
      at /runtime/src/coreclr/vm/jitinterface.cpp:13410
    MethodDesc::JitCompileCodeLocked(PrepareCodeConfig*, COR_ILMETHOD_DECODER*, ListLockEntryBase<>*, unsigned int*, bool*)
      at /runtime/src/coreclr/vm/prestub.cpp:921
      in /home/octoross/server/libcoreclr.so
    MethodDesc::JitCompileCodeLockedEventWrapper(PrepareCodeConfig*, ListLockEntryBase<>*)
      at /runtime/src/coreclr/vm/prestub.cpp:819
      in /home/octoross/server/libcoreclr.so
    MethodDesc::JitCompileCode(PrepareCodeConfig*)
      at /runtime/src/coreclr/vm/prestub.cpp:694
      in /home/octoross/server/libcoreclr.so
    MethodDesc::PrepareILBasedCode(PrepareCodeConfig*)
      at /runtime/src/coreclr/vm/prestub.cpp:424
      in /home/octoross/server/libcoreclr.so
    MethodDesc::PrepareCode(PrepareCodeConfig*)
      at /runtime/src/coreclr/vm/prestub.cpp:312
      in /home/octoross/server/libcoreclr.so
    MethodDesc::PrepareInitialCode(CallerGCMode)
      at /runtime/src/coreclr/vm/prestub.cpp:302
    MethodDesc::DoPrestub(MethodTable*, CallerGCMode)
      at /runtime/src/coreclr/vm/prestub.cpp:2278
    ReflectionInvocation_CompileMethod
      at /runtime/src/coreclr/vm/reflectioninvocation.cpp:1209
      in /home/octoross/server/libcoreclr.so
    0x7fff949d37b8
      in ??
    0x7fff934b33bd
      in ??
    0x7fff958aae45
      in ??
    0x7fff958a8d66
      in ??
    0x7fff95886c50
      in ??
    0x7fff945b7f1b
      in ??
    0x7fff945bf750
      in ??
    0x7fff945e28ba
      in ??
    0x7fff945e9d5b
      in ??
    0x7fff945e999f
      in ??
    0x7fff945c930e
      in ??
    0x7fff945c8fe6
      in ??
    0x7fff945e8e3c
      in ??
    0x7fff945c3612
      in ??
    0x7fff945c2dc7
      in ??
    0x7fff945c24bf
      in ??
    0x7fff945caa82
      in ??
    0x7fff945e99b6
      in ??
    0x7fff945c930e
      in ??
    0x7fff95878b59
      in ??
    0x7fff945c8fe6
      in ??
    0x7fff945e8e3c
      in ??
    0x7fff945c3612
      in ??
    0x7fff945c2dc7
      in ??
    0x7fff945c24bf
      in ??
    0x7fff945caa82
      in ??
    0x7fff945e99b6
      in ??
    0x7fff945c930e
      in ??
    0x7fff945c8fe6
      in ??
    0x7fff945e8e3c
      in ??
    0x7fff945c3612
      in ??
    0x7fff95886e30
      in ??
    0x7fff95886e30
      in ??
    0x7fff95886e30
      in ??
    0x7fff95886e30
      in ??
    0x7fff95886e30
      in ??
    0x7fff95886e30
      in ??
    0x7fff95886e30
      in ??
    0x7fff945c2dc7
      in ??
    0x7fff945c24bf
      in ??
    0x7fff9497e306
      in ??
    0x7fff945c6952
      in ??
    0x7fff945c719c
      in ??
    0x7fff960bf8d1
      in ??
    0x7fff963655de
      in ??
    0x7fff95d1afb0
      in ??
    0x7fff95d1b7e4
      in ??
  and 182.82M from 303918 other places

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[jakobbotsch]

Thanks for the great report and for pointing at the exact line with the problem! This looks like a regression from #113988. We generally should never be heap allocating in the JIT.