[arm64] swiftlang bootstrap stage 0 failing to build on Amazon Linux 2023

[justice-adams-apple]

Description

Swiftlang fails to build bootstrap stage 0 when building for arm64. The error seen in the log is

08:59:57  In file included from /home/build-user/swift/lib/Demangling/Punycode.cpp:13:
08:59:57  In file included from /home/build-user/swift/include/swift/Demangling/Punycode.h:30:
08:59:57  In file included from /home/build-user/swift/stdlib/include/llvm/ADT/StringRef.h:12:
08:59:57  In file included from /home/build-user/swift/stdlib/include/llvm/ADT/Hashing.h:47:
08:59:57  In file included from /home/build-user/swift/stdlib/include/llvm/Support/ErrorHandling.h:17:
08:59:57  /home/build-user/swift/stdlib/include/llvm/Support/Compiler.h:19:10: fatal error: 'new' file not found
08:59:57     19 | #include <new>
08:59:57        |          ^~~~~
08:59:57  1 error generated.

The compiler seemingly has trouble finding the C++ libraries and headers. After some investigation, it seems that the clang++ executable which is built has the wrong search paths.

For example: on X86 where the build works, you can see the following search paths from clang++

$ clang++ -v
clang version 17.0.0 (https://github.com/swiftlang/llvm-project.git 68128bc404300d4614691df049db6ab2ae0dcbc9)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/build-user/build/bootstrap_stage0/llvm-linux-x86_64/bin
Build config: +assertions
Found candidate GCC installation: /usr/lib/gcc/x86_64-amazon-linux/11
Selected GCC installation: /usr/lib/gcc/x86_64-amazon-linux/11
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64

$ clang++ -print-search-dirs
programs: =/home/build-user/build/bootstrap_stage0/llvm-linux-x86_64/bin:/usr/lib/gcc/x86_64-amazon-linux/11/../../../../x86_64-amazon-linux/bin
libraries: =/home/build-user/build/bootstrap_stage0/llvm-linux-x86_64/lib/clang/17:/usr/lib/gcc/x86_64-amazon-linux/11:/usr/lib/gcc/x86_64-amazon-linux/11/../../../../lib64:/lib/../lib64:/usr/lib/../lib64:/lib:/usr/lib

$ clang++ -E -v -x c++ /dev/null
clang version 17.0.0 (https://github.com/swiftlang/llvm-project.git 68128bc404300d4614691df049db6ab2ae0dcbc9)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/build-user/build/bootstrap_stage0/llvm-linux-x86_64/bin
Build config: +assertions
Found candidate GCC installation: /usr/lib/gcc/x86_64-amazon-linux/11
Selected GCC installation: /usr/lib/gcc/x86_64-amazon-linux/11
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64
 (in-process)
 "/home/build-user/build/bootstrap_stage0/llvm-linux-x86_64/bin/clang-17" -cc1 -triple x86_64-unknown-linux-gnu -E -disable-free -clear-ast-before-backend -main-file-name null -mrelocation-model pic -pic-level 2 -pic-is-pie -mframe-pointer=all -fmath-errno -ffp-contract=on -fno-rounding-math -mconstructor-aliases -funwind-tables=2 -target-cpu x86-64 -tune-cpu generic -debugger-tuning=gdb -fdebug-compilation-dir=/home/build-user -v -fcoverage-compilation-dir=/home/build-user -resource-dir /home/build-user/build/bootstrap_stage0/llvm-linux-x86_64/lib/clang/17 -internal-isystem /usr/lib/gcc/x86_64-amazon-linux/11/../../../../include/c++/11 -internal-isystem /usr/lib/gcc/x86_64-amazon-linux/11/../../../../include/c++/11/x86_64-amazon-linux -internal-isystem /usr/lib/gcc/x86_64-amazon-linux/11/../../../../include/c++/11/backward -internal-isystem /home/build-user/build/bootstrap_stage0/llvm-linux-x86_64/lib/clang/17/include -internal-isystem /usr/local/include -internal-isystem /usr/lib/gcc/x86_64-amazon-linux/11/../../../../x86_64-amazon-linux/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -fdeprecated-macro -ferror-limit 19 -fgnuc-version=4.2.1 -fskip-odr-check-in-gmf -fcxx-exceptions -fexceptions -fcolor-diagnostics -faddrsig -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o - -x c++ /dev/null
clang -cc1 version 17.0.0 based upon LLVM 17.0.0 default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/usr/lib/gcc/x86_64-amazon-linux/11/../../../../x86_64-amazon-linux/include"
ignoring nonexistent directory "/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/lib/gcc/x86_64-amazon-linux/11/../../../../include/c++/11
 /usr/lib/gcc/x86_64-amazon-linux/11/../../../../include/c++/11/x86_64-amazon-linux
 /usr/lib/gcc/x86_64-amazon-linux/11/../../../../include/c++/11/backward
 /home/build-user/build/bootstrap_stage0/llvm-linux-x86_64/lib/clang/17/include
 /usr/local/include
 /usr/include
End of search list.
# 1 "/dev/null"
# 1 "<built-in>" 1
# 1 "<built-in>" 3
# 468 "<built-in>" 3
# 1 "<command line>" 1
# 1 "<built-in>" 2
# 1 "/dev/null" 2

What you'll notice is that the search paths look for the following prefix in the various folders x86_64-amazon-linux

Compare this to the stage0 clang++ output from the arm64 build

$ clang++ -v
clang version 17.0.0 (https://github.com/swiftlang/llvm-project.git 9682898a32ef144e2accf08e3c0d4d9f031b22d8)
Target: aarch64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/build-user/build/bootstrap_stage0/llvm-linux-aarch64/bin
Build config: +assertions
Found candidate GCC installation: /usr/lib/gcc/aarch64-unknown-linux-gnu/11
Selected GCC installation: /usr/lib/gcc/aarch64-unknown-linux-gnu/11
Candidate multilib: .;@m64
Selected multilib: .;@m64

$ clang++ -print-search-dirs
programs: =/home/build-user/build/bootstrap_stage0/llvm-linux-aarch64/bin:/usr/lib/gcc/aarch64-unknown-linux-gnu/11/../../../../aarch64-unknown-linux-gnu/bin
libraries: =/home/build-user/build/bootstrap_stage0/llvm-linux-aarch64/lib/clang/17:/usr/lib/gcc/aarch64-unknown-linux-gnu/11:/usr/lib/gcc/aarch64-unknown-linux-gnu/11/../../../../lib64:/lib/../lib64:/usr/lib/../lib64:/lib:/usr/lib

$ clang++ -E -v -x c++ /dev/null
clang version 17.0.0 (https://github.com/swiftlang/llvm-project.git 9682898a32ef144e2accf08e3c0d4d9f031b22d8)
Target: aarch64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/build-user/build/bootstrap_stage0/llvm-linux-aarch64/bin
Build config: +assertions
Found candidate GCC installation: /usr/lib/gcc/aarch64-unknown-linux-gnu/11
Selected GCC installation: /usr/lib/gcc/aarch64-unknown-linux-gnu/11
Candidate multilib: .;@m64
Selected multilib: .;@m64
 (in-process)
 "/home/build-user/build/bootstrap_stage0/llvm-linux-aarch64/bin/clang-17" -cc1 -triple aarch64-unknown-linux-gnu -E -disable-free -clear-ast-before-backend -main-file-name null -mrelocation-model pic -pic-level 2 -pic-is-pie -mframe-pointer=non-leaf -fmath-errno -ffp-contract=on -fno-rounding-math -mconstructor-aliases -funwind-tables=2 -target-cpu generic -target-feature +v8a -target-feature +fp-armv8 -target-feature +neon -target-abi aapcs -debugger-tuning=gdb -fdebug-compilation-dir=/home/build-user -v -fcoverage-compilation-dir=/home/build-user -resource-dir /home/build-user/build/bootstrap_stage0/llvm-linux-aarch64/lib/clang/17 -internal-isystem /usr/lib/gcc/aarch64-unknown-linux-gnu/11/../../../../include/c++/11 -internal-isystem /usr/lib/gcc/aarch64-unknown-linux-gnu/11/../../../../include/c++/11/aarch64-unknown-linux-gnu -internal-isystem /usr/lib/gcc/aarch64-unknown-linux-gnu/11/../../../../include/c++/11/backward -internal-isystem /home/build-user/build/bootstrap_stage0/llvm-linux-aarch64/lib/clang/17/include -internal-isystem /usr/local/include -internal-isystem /usr/lib/gcc/aarch64-unknown-linux-gnu/11/../../../../aarch64-unknown-linux-gnu/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -fdeprecated-macro -ferror-limit 19 -fno-signed-char -fgnuc-version=4.2.1 -fskip-odr-check-in-gmf -fcxx-exceptions -fexceptions -fcolor-diagnostics -target-feature +outline-atomics -target-feature -fmv -faddrsig -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o - -x c++ /dev/null
clang -cc1 version 17.0.0 based upon LLVM 17.0.0 default target aarch64-unknown-linux-gnu
ignoring nonexistent directory "/usr/lib/gcc/aarch64-unknown-linux-gnu/11/../../../../aarch64-unknown-linux-gnu/include"
ignoring nonexistent directory "/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/lib/gcc/aarch64-unknown-linux-gnu/11/../../../../include/c++/11
 /usr/lib/gcc/aarch64-unknown-linux-gnu/11/../../../../include/c++/11/aarch64-unknown-linux-gnu
 /usr/lib/gcc/aarch64-unknown-linux-gnu/11/../../../../include/c++/11/backward
 /home/build-user/build/bootstrap_stage0/llvm-linux-aarch64/lib/clang/17/include
 /usr/local/include
 /usr/include
End of search list.
# 1 "/dev/null"
# 1 "<built-in>" 1
# 1 "<built-in>" 3
# 478 "<built-in>" 3
# 1 "<command line>" 1
# 1 "<built-in>" 2
# 1 "/dev/null" 2

The difference is that on arm64 we are adding the prefix aarch64-unknown-linux-gnu to the search paths which does not exist in the docker images

One approach is to symlink the directories from within the image to the directories being searched for with the gnu suffix

$ ln -sf /usr/include/c++/11/aarch64-amazon-linux /usr/include/c++/11/aarch64-unknown-linux-gnu && \
    mkdir -p /usr/lib/gcc/aarch64-unknown-linux-gnu && \
    ln -sf /usr/lib/gcc/aarch64-amazon-linux/11 /usr/lib/gcc/aarch64-unknown-linux-gnu/11 && \
    ln -sf /usr/lib64 /usr/lib/aarch64-unknown-linux-gnu

But this does not seem right. Instead we should fix the search paths clang searches for to be correct on arm64

Reproduction

1. Clone swift-docker
2. Build the image on an aarch64 machine

cd swift-docker && docker build -t 2023-amazon-aarch64 -f ./swift-ci/main/amazon-linux/2023/Bootstrap/Dockerfile .

3. Run the image and build within the container

docker run -it --rm --name test-swift-bootstrap-amazonlinux-2023-aarch64 -m 32G --cpus 14  -e TOOLCHAIN_VERSION -e OS_VERSION -e ARCHIVE -e SWIFT_SOURCE_ROOT -e GIT_ALLOW_PROTOCOL -w /home/build-user/ -v <swift-project>:/source 2023-amazon-aarch64


$ cp -r /source/* /home/build-user
$ ./swift/utils/build-script --preset bootstrap_stage0 build_subdir=bootstrap_stage0 install_destdir=/home/build-user/stages/stage0

Expected behavior

Expect it to build OK

Environment

Swift toolchain version: swift-DEVELOPMENT-SNAPSHOT-2025-09-10-a

Additional information

No response

[edymtt]

This is because for aarch64 the Clang driver does not infer the vendor specific triple for Amazon Linux to search for the resource directory like it does for x86_64.

Looking at https://github.com/llvm/llvm-project/blob/0384f6c9dbdcce283088b5207a56b1567590f927/clang/lib/Driver/ToolChains/Gnu.cpp#L2327, that seems on purpose upstream, since they want the user to either pass the correct triple to every clang invocation, or to set the correct triple as the default for the compiler.

I think it makes sense to match the same change contained the clang shipped in Amazon Linux 2023 (at least, looking at the sources for clang and clang18) to find the vendor specific resource dir when specifying the generic arm64 linux triple.

[shahmishal]

cc: @sebsto

[sebsto]

I confirm the patch proposed by swiftlang/llvm-project#11447 works on my machines. Build Stage 0 now succeeds.

Just a note: last time I proposed a PR to add this triple there, someone reminded me of the comment on the line just above:

// The lists should shrink over time. Please don't add more elements to *Triples.

If you're OK with this and you think the fix can be contributed upstream, let's go for it as this is a simple one line change that unblocks the compilation.

[sebsto]

And a few hours later:

Total Duration: 8268.40 seconds (2h 17m 48s)

[ec2-user@ip-172-31-3-233 swift-project]$ ls -al ../swift-DEVELOPMENT-SNAPSHOT-2025-09-18-amazonlinux2023.tar.gz 
-rw-r--r--. 1 ec2-user ec2-user 1168364697 Sep 18 22:48 ../swift-DEVELOPMENT-SNAPSHOT-2025-09-18-amazonlinux2023.tar.gz

[edymtt]

If you're OK with this and you think the fix can be contributed upstream

I saw the comment as well -- I would say it makes sense to do this only for the swiftlang fork of llvm-project, and reassess long term whether we want to pass a more specific triple in case this "fallback logic" is removed upstream.