Github Actions added to pre-compile artifacts for flang #7

michalpasztamobica · 2020-11-13T16:24:24Z

This PR is required in order to provide prebuilt artifacts for Flang project's CI as planned in flang-compiler/flang#948.

The CI job needs to produce binary, precompiled artifacts to be used in Flang repository, os that the PR checks don't last for 3 hours, but 10 minutes, instead. The existing jobs in this repository do not provide this kind of artifacts. The job will only run after release_100 gets new commits, so that the latest build is always available for Flang job to use.

@kiranchandramohan , @RichBarton-Arm , please provide your review and please set up other reivewers appropriately. Thanks!

gklimowicz

I am not in a position to test this, so I will defer to the other reviewers. LGTM.

bryanpkc

LGTM. Thanks @michalpasztamobica.

michalpasztamobica · 2020-11-23T16:31:53Z

@shivaramaarao, are there any review remarks from your side?

shivaramaarao

Looks good. However I am not able to test this patch.

michalpasztamobica · 2020-11-23T16:46:04Z

Github actions are now enabled for this repository, I'll try to push this branch and see if that triggers them to test this patch for us 😉

michalpasztamobica · 2020-11-23T17:08:50Z

Nope, a push didn't trigger the build, not sure why. @kiranchandramohan are you able to trigger the github actions for this build somehow?

michalpasztamobica · 2020-11-23T17:11:32Z

@shivaramaarao , I hope this script will produce something of this kind once it's merged.

kiranchandramohan · 2020-11-23T17:23:08Z

@michalpasztamobica Not clear to me why it is not running. The CI was always running, the patch I added only limited it to the passing runs. In the case of this patch, it did not previously also run github actions.

kiranchandramohan · 2020-11-23T18:40:18Z

@michalpasztamobica The existing workflows have the following yaml at the beginning. Could you check whether it has something to the way a branch is named?

on:
  push:
    branches:
      - 'release/**'
...

michalpasztamobica · 2020-11-23T19:52:11Z

@kiranchandramohan , right - that's the problem. The job is scheduled to run on push event meaning - "commit to master" and not on pull requests. That's why it isn't running until it gets merged. This is intended - we just want to generate artifacts with this job, it doesn't really test anything.

Why the other jobs are not running is also because of how the yaml scripts are configured. They will only trigger if PR touches the paths:

  pull_request:
    paths:
      - 'clang/**'
      - 'llvm/**'

michalpasztamobica · 2020-11-23T20:03:36Z

@kiranchandramohan , sorry for the confusion, we won't see this job run until we merge and we won't see any jobs running for this PR because it doesn't change clang or llvm.
I tested how it runs within my private fork, so I expect no trouble here.
All reviewers approved, please let me know if anything is looking suspicious to you.

kiranchandramohan

LGTM.

When `Target::GetEntryPointAddress()` calls `exe_module->GetObjectFile()->GetEntryPointAddress()`, and the returned `entry_addr` is valid, it can immediately be returned. However, just before that, an `llvm::Error` value has been setup, but in this case it is not consumed before returning, like is done further below in the function. In https://bugs.freebsd.org/248745 we got a bug report for this, where a very simple test case aborts and dumps core: ``` * thread #1, name = 'testcase', stop reason = breakpoint 1.1 frame #0: 0x00000000002018d4 testcase`main(argc=1, argv=0x00007fffffffea18) at testcase.c:3:5 1 int main(int argc, char *argv[]) 2 { -> 3 return 0; 4 } (lldb) p argc Program aborted due to an unhandled Error: Error value was Success. (Note: Success values must still be checked prior to being destroyed). Thread 1 received signal SIGABRT, Aborted. thr_kill () at thr_kill.S:3 3 thr_kill.S: No such file or directory. (gdb) bt #0 thr_kill () at thr_kill.S:3 #1 0x00000008049a0004 in __raise (s=6) at /usr/src/lib/libc/gen/raise.c:52 #2 0x0000000804916229 in abort () at /usr/src/lib/libc/stdlib/abort.c:67 #3 0x000000000451b5f5 in fatalUncheckedError () at /usr/src/contrib/llvm-project/llvm/lib/Support/Error.cpp:112 #4 0x00000000019cf008 in GetEntryPointAddress () at /usr/src/contrib/llvm-project/llvm/include/llvm/Support/Error.h:267 flang-compiler#5 0x0000000001bccbd8 in ConstructorSetup () at /usr/src/contrib/llvm-project/lldb/source/Target/ThreadPlanCallFunction.cpp:67 flang-compiler#6 0x0000000001bcd2c0 in ThreadPlanCallFunction () at /usr/src/contrib/llvm-project/lldb/source/Target/ThreadPlanCallFunction.cpp:114 flang-compiler#7 0x00000000020076d4 in InferiorCallMmap () at /usr/src/contrib/llvm-project/lldb/source/Plugins/Process/Utility/InferiorCallPOSIX.cpp:97 flang-compiler#8 0x0000000001f4be33 in DoAllocateMemory () at /usr/src/contrib/llvm-project/lldb/source/Plugins/Process/FreeBSD/ProcessFreeBSD.cpp:604 flang-compiler#9 0x0000000001fe51b9 in AllocatePage () at /usr/src/contrib/llvm-project/lldb/source/Target/Memory.cpp:347 flang-compiler#10 0x0000000001fe5385 in AllocateMemory () at /usr/src/contrib/llvm-project/lldb/source/Target/Memory.cpp:383 flang-compiler#11 0x0000000001974da2 in AllocateMemory () at /usr/src/contrib/llvm-project/lldb/source/Target/Process.cpp:2301 flang-compiler#12 CanJIT () at /usr/src/contrib/llvm-project/lldb/source/Target/Process.cpp:2331 flang-compiler#13 0x0000000001a1bf3d in Evaluate () at /usr/src/contrib/llvm-project/lldb/source/Expression/UserExpression.cpp:190 flang-compiler#14 0x00000000019ce7a2 in EvaluateExpression () at /usr/src/contrib/llvm-project/lldb/source/Target/Target.cpp:2372 flang-compiler#15 0x0000000001ad784c in EvaluateExpression () at /usr/src/contrib/llvm-project/lldb/source/Commands/CommandObjectExpression.cpp:414 flang-compiler#16 0x0000000001ad86ae in DoExecute () at /usr/src/contrib/llvm-project/lldb/source/Commands/CommandObjectExpression.cpp:646 flang-compiler#17 0x0000000001a5e3ed in Execute () at /usr/src/contrib/llvm-project/lldb/source/Interpreter/CommandObject.cpp:1003 flang-compiler#18 0x0000000001a6c4a3 in HandleCommand () at /usr/src/contrib/llvm-project/lldb/source/Interpreter/CommandInterpreter.cpp:1762 flang-compiler#19 0x0000000001a6f98c in IOHandlerInputComplete () at /usr/src/contrib/llvm-project/lldb/source/Interpreter/CommandInterpreter.cpp:2760 flang-compiler#20 0x0000000001a90b08 in Run () at /usr/src/contrib/llvm-project/lldb/source/Core/IOHandler.cpp:548 flang-compiler#21 0x00000000019a6c6a in ExecuteIOHandlers () at /usr/src/contrib/llvm-project/lldb/source/Core/Debugger.cpp:903 flang-compiler#22 0x0000000001a70337 in RunCommandInterpreter () at /usr/src/contrib/llvm-project/lldb/source/Interpreter/CommandInterpreter.cpp:2946 flang-compiler#23 0x0000000001d9d812 in RunCommandInterpreter () at /usr/src/contrib/llvm-project/lldb/source/API/SBDebugger.cpp:1169 flang-compiler#24 0x0000000001918be8 in MainLoop () at /usr/src/contrib/llvm-project/lldb/tools/driver/Driver.cpp:675 flang-compiler#25 0x000000000191a114 in main () at /usr/src/contrib/llvm-project/lldb/tools/driver/Driver.cpp:890``` Fix the incorrect error catch by only instantiating an `Error` object if it is necessary. Reviewed By: JDevlieghere Differential Revision: https://reviews.llvm.org/D86355 (cherry picked from commit 1ce07cd)

Andrei Matei reported a llvm11 core dump for his bpf program https://bugs.llvm.org/show_bug.cgi?id=48578 The core dump happens in LiveVariables analysis phase. flang-compiler#4 0x00007fce54356bb0 __restore_rt flang-compiler#5 0x00007fce4d51785e llvm::LiveVariables::HandleVirtRegUse(unsigned int, llvm::MachineBasicBlock*, llvm::MachineInstr&) flang-compiler#6 0x00007fce4d519abe llvm::LiveVariables::runOnInstr(llvm::MachineInstr&, llvm::SmallVectorImpl<unsigned int>&) flang-compiler#7 0x00007fce4d519ec6 llvm::LiveVariables::runOnBlock(llvm::MachineBasicBlock*, unsigned int) flang-compiler#8 0x00007fce4d51a4bf llvm::LiveVariables::runOnMachineFunction(llvm::MachineFunction&) The bug can be reproduced with llvm12 and latest trunk as well. Futher analysis shows that there is a bug in BPF peephole TRUNC elimination optimization, which tries to remove unnecessary TRUNC operations (a <<= 32; a >>= 32). Specifically, the compiler did wrong transformation for the following patterns: %1 = LDW ... %2 = SLL_ri %1, 32 %3 = SRL_ri %2, 32 ... %3 ... %4 = SRA_ri %2, 32 ... %4 ... The current transformation did not check how many uses of %2 and did transformation like %1 = LDW ... ... %1 ... %4 = SRL_ri %2, 32 ... %4 ... and pseudo register %2 is used by not defined and caused LiveVariables analysis core dump. To fix the issue, when traversing back from SRL_ri to SLL_ri, check to ensure SLL_ri has only one use. Otherwise, don't do transformation. Differential Revision: https://reviews.llvm.org/D97792 (cherry picked from commit 51cdb78)

Andrei Matei reported a llvm11 core dump for his bpf program https://bugs.llvm.org/show_bug.cgi?id=48578 The core dump happens in LiveVariables analysis phase. #4 0x00007fce54356bb0 __restore_rt #5 0x00007fce4d51785e llvm::LiveVariables::HandleVirtRegUse(unsigned int, llvm::MachineBasicBlock*, llvm::MachineInstr&) #6 0x00007fce4d519abe llvm::LiveVariables::runOnInstr(llvm::MachineInstr&, llvm::SmallVectorImpl<unsigned int>&) #7 0x00007fce4d519ec6 llvm::LiveVariables::runOnBlock(llvm::MachineBasicBlock*, unsigned int) #8 0x00007fce4d51a4bf llvm::LiveVariables::runOnMachineFunction(llvm::MachineFunction&) The bug can be reproduced with llvm12 and latest trunk as well. Futher analysis shows that there is a bug in BPF peephole TRUNC elimination optimization, which tries to remove unnecessary TRUNC operations (a <<= 32; a >>= 32). Specifically, the compiler did wrong transformation for the following patterns: %1 = LDW ... %2 = SLL_ri %1, 32 %3 = SRL_ri %2, 32 ... %3 ... %4 = SRA_ri %2, 32 ... %4 ... The current transformation did not check how many uses of %2 and did transformation like %1 = LDW ... ... %1 ... %4 = SRL_ri %2, 32 ... %4 ... and pseudo register %2 is used by not defined and caused LiveVariables analysis core dump. To fix the issue, when traversing back from SRL_ri to SLL_ri, check to ensure SLL_ri has only one use. Otherwise, don't do transformation. Differential Revision: https://reviews.llvm.org/D97792 (cherry picked from commit 51cdb78)

@yrnkrn

This patch re-introduces the fix in the commit llvm/llvm-project@66b0cebf7f736 by @yrnkrn > In DwarfEHPrepare, after all passes are run, RewindFunction may be a dangling > > pointer to a dead function. To make sure it's valid, doFinalization nullptrs > RewindFunction just like the constructor and so it will be found on next run. > > llvm-svn: 217737 It seems that the fix was not migrated to `DwarfEHPrepareLegacyPass`. This patch also updates `llvm/test/CodeGen/X86/dwarf-eh-prepare.ll` to include `-run-twice` to exercise the cleanup. Without this patch `llvm-lit -v llvm/test/CodeGen/X86/dwarf-eh-prepare.ll` fails with ``` -- Testing: 1 tests, 1 workers -- FAIL: LLVM :: CodeGen/X86/dwarf-eh-prepare.ll (1 of 1) ******************** TEST 'LLVM :: CodeGen/X86/dwarf-eh-prepare.ll' FAILED ******************** Script: -- : 'RUN: at line 1'; /home/arakaki/build/llvm-project/main/bin/opt -mtriple=x86_64-linux-gnu -dwarfehprepare -simplifycfg-require-and-preserve-domtree=1 -run-twice < /home/arakaki/repos/watch/llvm-project/llvm/test/CodeGen/X86/dwarf-eh-prepare.ll -S | /home/arakaki/build/llvm-project/main/bin/FileCheck /home/arakaki/repos/watch/llvm-project/llvm/test/CodeGen/X86/dwarf-eh-prepare.ll -- Exit Code: 2 Command Output (stderr): -- Referencing function in another module! call void @_Unwind_Resume(i8* %ehptr) #1 ; ModuleID = '<stdin>' void (i8*)* @_Unwind_Resume ; ModuleID = '<stdin>' in function simple_cleanup_catch LLVM ERROR: Broken function found, compilation aborted! PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace. Stack dump: 0. Program arguments: /home/arakaki/build/llvm-project/main/bin/opt -mtriple=x86_64-linux-gnu -dwarfehprepare -simplifycfg-require-and-preserve-domtree=1 -run-twice -S 1. Running pass 'Function Pass Manager' on module '<stdin>'. 2. Running pass 'Module Verifier' on function '@simple_cleanup_catch' #0 0x000056121b570a2c llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) /home/arakaki/repos/watch/llvm-project/llvm/lib/Support/Unix/Signals.inc:569:0 #1 0x000056121b56eb64 llvm::sys::RunSignalHandlers() /home/arakaki/repos/watch/llvm-project/llvm/lib/Support/Signals.cpp:97:0 #2 0x000056121b56f28e SignalHandler(int) /home/arakaki/repos/watch/llvm-project/llvm/lib/Support/Unix/Signals.inc:397:0 #3 0x00007fc7e9b22980 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x12980) #4 0x00007fc7e87d3fb7 raise /build/glibc-S7xCS9/glibc-2.27/signal/../sysdeps/unix/sysv/linux/raise.c:51:0 #5 0x00007fc7e87d5921 abort /build/glibc-S7xCS9/glibc-2.27/stdlib/abort.c:81:0 #6 0x000056121b4e1386 llvm::raw_svector_ostream::raw_svector_ostream(llvm::SmallVectorImpl<char>&) /home/arakaki/repos/watch/llvm-project/llvm/include/llvm/Support/raw_ostream.h:674:0 #7 0x000056121b4e1386 llvm::report_fatal_error(llvm::Twine const&, bool) /home/arakaki/repos/watch/llvm-project/llvm/lib/Support/ErrorHandling.cpp:114:0 #8 0x000056121b4e1528 (/home/arakaki/build/llvm-project/main/bin/opt+0x29e3528) #9 0x000056121adfd03f llvm::raw_ostream::operator<<(llvm::StringRef) /home/arakaki/repos/watch/llvm-project/llvm/include/llvm/Support/raw_ostream.h:218:0 FileCheck error: '<stdin>' is empty. FileCheck command line: /home/arakaki/build/llvm-project/main/bin/FileCheck /home/arakaki/repos/watch/llvm-project/llvm/test/CodeGen/X86/dwarf-eh-prepare.ll -- ******************** ******************** Failed Tests (1): LLVM :: CodeGen/X86/dwarf-eh-prepare.ll Testing Time: 0.22s Failed: 1 ``` Reviewed By: loladiro Differential Revision: https://reviews.llvm.org/D110979 (cherry picked from commit e8806d7)

We experienced some deadlocks when we used multiple threads for logging using `scan-builds` intercept-build tool when we used multiple threads by e.g. logging `make -j16` ``` (gdb) bt #0 0x00007f2bb3aff110 in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 flang-compiler#1 0x00007f2bb3af70a3 in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0 flang-compiler#2 0x00007f2bb3d152e4 in ?? () flang-compiler#3 0x00007ffcc5f0cc80 in ?? () flang-compiler#4 0x00007f2bb3d2bf5b in ?? () from /lib64/ld-linux-x86-64.so.2 flang-compiler#5 0x00007f2bb3b5da27 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 flang-compiler#6 0x00007f2bb3b5dbe0 in exit () from /lib/x86_64-linux-gnu/libc.so.6 flang-compiler#7 0x00007f2bb3d144ee in ?? () flang-compiler#8 0x746e692f706d742f in ?? () flang-compiler#9 0x692d747065637265 in ?? () flang-compiler#10 0x2f653631326b3034 in ?? () flang-compiler#11 0x646d632e35353532 in ?? () flang-compiler#12 0x0000000000000000 in ?? () ``` I think the gcc's exit call caused the injected `libear.so` to be unloaded by the `ld`, which in turn called the `void on_unload() __attribute__((destructor))`. That tried to acquire an already locked mutex which was left locked in the `bear_report_call()` call, that probably encountered some error and returned early when it forgot to unlock the mutex. All of these are speculation since from the backtrace I could not verify if frames 2 and 3 are in fact corresponding to the `libear.so` module. But I think it's a fairly safe bet. So, hereby I'm releasing the held mutex on *all paths*, even if some failure happens. PS: I would use lock_guards, but it's C. Reviewed-by: NoQ Differential Revision: https://reviews.llvm.org/D118439 (cherry picked from commit d919d02)

michalpasztamobica mentioned this pull request Nov 13, 2020

Expand CI to run with GCC-9/10 and LLVM-9/10/11 flang-compiler/flang#948

Merged

gklimowicz self-requested a review November 18, 2020 16:11

gklimowicz approved these changes Nov 18, 2020

View reviewed changes

bryanpkc approved these changes Nov 22, 2020

View reviewed changes

kiranchandramohan requested a review from shivaramaarao November 22, 2020 21:59

shivaramaarao approved these changes Nov 23, 2020

View reviewed changes

Github Actions added to pre-compile artifacts for flang

42665ab

michalpasztamobica force-pushed the actions_precompile_llvm branch from 6a2e9cf to 42665ab Compare November 23, 2020 16:50

kiranchandramohan approved these changes Nov 25, 2020

View reviewed changes

kiranchandramohan merged commit 39d9fe4 into flang-compiler:release_100 Nov 25, 2020

gklimowicz mentioned this pull request Dec 4, 2020

December 2020 newsletter draft fortran-lang/fortran-lang.org#172

Merged

michalpasztamobica deleted the actions_precompile_llvm branch December 30, 2020 13:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Github Actions added to pre-compile artifacts for flang #7

Github Actions added to pre-compile artifacts for flang #7

Uh oh!

michalpasztamobica commented Nov 13, 2020

Uh oh!

gklimowicz left a comment

Uh oh!

bryanpkc left a comment

Uh oh!

michalpasztamobica commented Nov 23, 2020

Uh oh!

shivaramaarao left a comment

Uh oh!

michalpasztamobica commented Nov 23, 2020

Uh oh!

michalpasztamobica commented Nov 23, 2020

Uh oh!

michalpasztamobica commented Nov 23, 2020

Uh oh!

kiranchandramohan commented Nov 23, 2020

Uh oh!

kiranchandramohan commented Nov 23, 2020

Uh oh!

michalpasztamobica commented Nov 23, 2020

Uh oh!

michalpasztamobica commented Nov 23, 2020

Uh oh!

kiranchandramohan left a comment

Uh oh!

Uh oh!

Github Actions added to pre-compile artifacts for flang #7

Github Actions added to pre-compile artifacts for flang #7

Uh oh!

Conversation

michalpasztamobica commented Nov 13, 2020

Uh oh!

gklimowicz left a comment

Choose a reason for hiding this comment

Uh oh!

bryanpkc left a comment

Choose a reason for hiding this comment

Uh oh!

michalpasztamobica commented Nov 23, 2020

Uh oh!

shivaramaarao left a comment

Choose a reason for hiding this comment

Uh oh!

michalpasztamobica commented Nov 23, 2020

Uh oh!

michalpasztamobica commented Nov 23, 2020

Uh oh!

michalpasztamobica commented Nov 23, 2020

Uh oh!

kiranchandramohan commented Nov 23, 2020

Uh oh!

kiranchandramohan commented Nov 23, 2020

Uh oh!

michalpasztamobica commented Nov 23, 2020

Uh oh!

michalpasztamobica commented Nov 23, 2020

Uh oh!

kiranchandramohan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!