☂️ Use llvm-lit's internal shell by default

[hnrklssn]

Description

As mentioned in https://forums.swift.org/t/rfc-enabling-the-internal-shell-by-default-in-llvm-lit-tests/82148, we ought to move towards running tests using lit's internal shell. This issue tracks the progress on converting tests to be compatible with the internal shell. Some tests are quite long-hanging fruit, others require more work. Regardless, I don't expect any tests to require touching compiler internals, so these should all be good first issues for someone looking at getting involved.

List of tests failing on macOS with the internal shell:

• test/AutoDiff/IRGen/loadable_by_address_cross_module.swift
• test/AutoDiff/compiler_crashers_fixed/issue-56397-cross-module-differentiation-differentiable-function-extract-inlining.swift
• test/AutoDiff/validation-test/always_emit_into_client/multi_module.swift
• test/AutoDiff/validation-test/always_emit_into_client/multi_module_protocol.swift
• test/AutoDiff/validation-test/always_emit_into_client/multi_module_struct.swift
• test/AutoDiff/validation-test/always_emit_into_client/multi_module_struct_no_jvp.swift
• test/AutoDiff/validation-test/always_emit_into_client/multi_module_struct_no_vjp.swift
• test/AutoDiff/validation-test/cross_module_derivative_attr.swift
• test/AutoDiff/validation-test/cross_module_differentiation.swift
• test/ClangImporter/print-module-map-paths.swift
• test/Concurrency/Runtime/actor_assert_precondition_executor.swift
• test/Concurrency/Runtime/actor_assert_precondition_executor_checkIsolated_bincompat_crash_swift_6_mode.swift
• test/Concurrency/Runtime/actor_assert_precondition_executor_checkIsolated_bincompat_nocrash_legacy_mode.swift
• test/Concurrency/Runtime/actor_assert_precondition_executor_checkIsolated_dispatch_dispatchMain_swift_6_mode.swift
• test/Concurrency/Runtime/actor_assert_precondition_executor_checkIsolated_dispatch_swift6_mode.swift
• test/Concurrency/Runtime/actor_assert_precondition_executor_checkIsolated_main_customExecutorOnMain_swift6_mode.swift
• test/Concurrency/Runtime/actor_assert_precondition_executor_checkIsolated_swift6_mode.swift
• test/Concurrency/Runtime/actor_assume_executor.swift
• test/Concurrency/Runtime/actor_init_abi.swift
• test/Concurrency/Runtime/async_task_executor_and_serial_executor_both_executor.swift
• test/Concurrency/Runtime/async_task_executor_and_serial_executor_nonisolated_async_func_legacy.swift
• test/Concurrency/Runtime/async_task_executor_and_serial_executor_nonisolated_async_func_swift6.swift
• test/Concurrency/Runtime/async_task_locals_isolated_deinit.swift
• test/Concurrency/Runtime/class_resilience.swift
• test/Concurrency/Runtime/custom_executors_complex_equality_crash.swift
• test/Concurrency/Runtime/data_race_detection_legacy_warning.swift
• test/Concurrency/Runtime/protocol_resilience.swift
• test/Concurrency/deinit_isolation_import/test.swift
• test/Concurrency/voucher_propagation.swift
• test/Distributed/Runtime/distributed_actor_deinit.swift
• test/Distributed/Runtime/distributed_actor_remoteCall_accessibleFunctions.swift
• test/Distributed/Runtime/distributed_actor_remoteCall_accessibleFunctions_crossModule.swift
• test/Driver/filelists.swift
• test/Driver/options-interpreter.swift
• test/IDE/objc_send_collector_1.swift
• test/IDE/print_clang_bool_bridging.swift
• test/IDE/print_clang_swift_name.swift
• test/IRGen/async-inheritance.swift
• test/IRGen/async/run-call-classinstance-int64-to-void.sil
• test/IRGen/async/run-call-classinstance-void-to-void.sil
• test/IRGen/async/run-call-existential-to-void.sil
• test/IRGen/async/run-call-generic-to-generic.sil
• test/IRGen/async/run-call-genericEquatable-x2-to-bool.sil
• test/IRGen/async/run-call-int64-and-int64-to-void.sil
• test/IRGen/async/run-call-int64-to-void.sil
• test/IRGen/async/run-call-nonresilient-classinstance-void-to-void.swift
• test/IRGen/async/run-call-protocolextension_instance-void-to-int64.sil
• test/IRGen/async/run-call-protocolwitness_instance-void-to-int64.sil
• test/IRGen/async/run-call-resilient-classinstance-void-to-void.sil
• test/IRGen/async/run-call-resilient-protocolinstance-void-to-void.swift
• test/IRGen/async/run-call-struct_five_bools-to-void.sil
• test/IRGen/async/run-call-structinstance-int64-to-void.sil
• test/IRGen/async/run-call-void-throws-to-int-throwing.sil
• test/IRGen/async/run-call-void-throws-to-int-throwing_call-async-nothrow_call-sync-throw.sil
• test/IRGen/async/run-call-void-throws-to-int-throwing_call-async-throw.sil
• test/IRGen/async/run-call-void-throws-to-int-throwing_call-sync-nothrow_call-async-throw.sil
• test/IRGen/async/run-call-void-throws-to-int-throwing_call-sync-throw.sil
• test/IRGen/async/run-call-void-to-existential.sil
• test/IRGen/async/run-call-void-to-int64-and-int64.sil
• test/IRGen/async/run-call-void-to-struct_large.sil
• test/IRGen/async/run-call_generic-protocolwitness_instance-generic-to-int64-and-generic.sil
• test/IRGen/async/run-call_generic-protocolwitness_instance-void-to-int64.sil
• test/IRGen/async/run-convertfunction-int64-to-void.sil
• test/IRGen/async/run-partialapply-capture-class-to-void.sil
• test/IRGen/async/run-partialapply-capture-classinstance-to-void.sil
• test/IRGen/async/run-partialapply-capture-generic_conformer-and-generic-to-void.sil
• test/IRGen/async/run-partialapply-capture-inout-generic-and-in-generic-to-generic.sil
• test/IRGen/async/run-partialapply-capture-int64-int64-throws-to-int64.sil
• test/IRGen/async/run-partialapply-capture-int64-int64-to-int64.sil
• test/IRGen/async/run-partialapply-capture-int64-to-generic.sil
• test/IRGen/async/run-partialapply-capture-struct_classinstance_classinstance-and-int64-to-int64.sil
• test/IRGen/async/run-partialapply-capture-structgeneric_classinstance_to_struct_and_error.sil
• test/IRGen/async/run-partialapply-capture-structgeneric_polymorphic_constrained-to-void.sil
• test/IRGen/async/run-partialapply-capture-type_structgeneric_polymorphic_constrained-to-void.sil
• test/IRGen/async/run-partialapply-capture-type_thin-and-classinstance-to-void.sil
• test/IRGen/async/run-thintothick-int64-to-void.sil
• test/IRGen/hermetic-seal-exec.swift
• test/IRGen/moveonly_split_module_source_deinit_library_evolution.swift
• test/IRGen/partial_apply_run_generic_method1.sil
• test/IRGen/run-coroutine_accessors.swift
• test/IRGen/run_variadic_generics.sil
• test/IRGen/testing-enabled-resilient-super-class-external-module.swift
• test/IRGen/testing-enabled-resilient-super-class.swift
• test/IRGen/thinlto.swift
• test/InterfaceHash/added_method-type.swift
• test/InterfaceHash/added_private_class_private_property.swift
• test/InterfaceHash/added_private_class_property.swift
• test/InterfaceHash/added_private_enum_private_property.swift
• test/InterfaceHash/added_private_enum_property.swift
• test/InterfaceHash/added_private_method.swift
• test/InterfaceHash/added_private_method_value_types.swift
• test/InterfaceHash/added_private_protocol_method.swift
• test/InterfaceHash/added_private_protocol_property.swift
• test/InterfaceHash/added_private_struct_private_property.swift
• test/InterfaceHash/added_private_struct_property.swift
• test/InterfaceHash/edited_method_body.swift
• test/InterfaceHash/edited_property_getter.swift
• test/Interpreter/SDK/check_class_for_archiving_log.swift
• test/Interpreter/SDK/objc_getClass.swift
• test/Interpreter/async_resilience.swift
• test/Interpreter/availability_weak_linking.swift
• test/Interpreter/bitwise_copyable.swift
• test/Interpreter/cdecl_implementation_run.swift
• test/Interpreter/class_resilience.swift
• test/Interpreter/conditional_conformances_modules.swift
• test/Interpreter/coroutine_accessors_default_implementations.swift
• test/Interpreter/dynamicReplacement_property_observer.swift
• test/Interpreter/dynamic_replacement.swift
• test/Interpreter/dynamic_replacement_chaining.swift
• test/Interpreter/dynamic_replacement_protocol_self.swift
• test/Interpreter/dynamic_replacement_weak_linked.swift
• test/Interpreter/dynamic_replacement_without_previous_calls.swift
• test/Interpreter/enum_resilience.swift
• test/Interpreter/global_resilience.swift
• test/Interpreter/has_symbol.swift
• test/Interpreter/import_parameter_pack_library.swift
• test/Interpreter/jit_metadata_strategy.swift
• test/Interpreter/layout_string_witnesses_dynamic.swift
• test/Interpreter/layout_string_witnesses_objc.swift
• test/Interpreter/layout_string_witnesses_static.swift
• test/Interpreter/llvm_link_time_opt.swift
• test/Interpreter/lto_static_lib.swift
• test/Interpreter/moveonly_deinit_separate_library.swift
• test/Interpreter/moveonly_swiftskell.swift
• test/Interpreter/objc_class_resilience.swift
• test/Interpreter/objc_class_resilience_stubs.swift
• test/Interpreter/objc_implementation_objc_client.m
• test/Interpreter/objc_implementation_swift_client.swift
• test/Interpreter/opaque_result_type_runtime_call.swift
• test/Interpreter/package_resilience_super_class.swift
• test/Interpreter/polymorphic_builtins.swift
• test/Interpreter/protocol_resilience.swift
• test/Interpreter/resilient_metadata_cycles.swift
• test/Interpreter/resilient_multipayload_enum.swift
• test/Interpreter/struct_resilience.swift
• test/Interpreter/switch_resilience.swift
• test/Interpreter/typed_throws_abi.swift
• test/Interpreter/unresolvable_dynamic_metadata_cycles.swift
• test/Interpreter/variadic_generic_opaque_type.swift
• test/Interpreter/vtables_multifile.swift
• test/Interpreter/vtables_multifile_testable.swift
• test/Macros/macro_default_argument_serialized.swift
• test/Macros/macro_default_argument_source_location.swift
• test/Macros/macro_default_argument_swiftinterface.swift
• test/ModuleInterface/ModuleCache/module-cache-deployment-target-irrelevant.swift
• test/ModuleInterface/ModuleCache/module-cache-options.swift
• test/ModuleInterface/inherited-defaults-execution.swift
• test/ModuleInterface/inherits-superclass-initializers-client.swift
• test/ModuleInterface/stored-properties-client.swift
• test/PrintAsObjC/unmodified.swift
• test/SILGen/magic_identifier_file.swift
• test/SILGen/objc_error_fast_path.swift
• test/SILOptimizer/di-conditional-destroy-scope.swift
• test/SILOptimizer/package-cmo-disallow-bypass-resilience-on-deserialization-fail.swift
• test/SILOptimizer/package-cmo-import-filter.swift
• test/SPI/run_spi_client.swift
• test/ScanDependencies/direct-cc1-dwarf-options.swift
• test/Serialization/incremental-imports.swift
• test/TBD/coroutine_accessors.swift
• test/TBD/package_symbol_with_default_arg.swift
• test/TBD/rdar82045176.swift
• test/attr/attr_backDeployed_evolution.swift
• test/attr/attr_implements_serial.swift
• test/attr/attr_originally_definedin_backward_compatibility.swift
• test/embedded/dependencies-concurrency-custom-executor.swift
• test/embedded/dependencies-concurrency.swift
• test/embedded/dependencies-concurrency2.swift
• test/embedded/dependencies-no-allocations.swift
• test/embedded/dependencies-print.swift
• test/embedded/dependencies-random.swift
• test/embedded/dependencies.swift
• test/multifile/protocol-conformance-redundant.swift
• test/multifile/resilient-module.swift
• test/stdlib/KVOKeyPaths.swift
• test/stdlib/SwiftObjectNSObject.swift

List of tests with REQUIRES: shell:

• test/AutoDiff/SIL/sil_differentiability_witness.sil
• test/AutoDiff/SIL/Serialization/differentiable_function_type.swift
• test/AutoDiff/SIL/differentiability_witness_function_inst_transpose.sil
• test/AutoDiff/SIL/differentiability_witness_function_inst.sil
• test/AutoDiff/SIL/differentiable_function_inst.sil
• test/TypeRoundTrip/round-trip.swift
• test/remote-run/dry-run.test-sh
• test/remote-run/stderr.test-sh
• test/remote-run/exit-code.test-sh
• test/remote-run/run-only.test-sh
• test/remote-run/env.test-sh
• validation-test/Driver/many-inputs.swift
• test/stdlib/WeakMirror.swift
• test/stdlib/Mirror.swift
• test/stdlib/MirrorWithPacks.swift
• test/Interpreter/convenience_init_peer_delegation.swift
• test/ASTGen/availability.swift
• test/SourceKit/Misc/no-driver-outputs.swift
• test/SourceKit/Misc/match-module-cache-with-compiler.swift
• test/SourceKit/CodeComplete/complete_checkdeps_vfs_open.swift
• test/SourceKit/CodeComplete/complete_checkdeps_vfs.swift
• test/SourceKit/CodeComplete/complete_checkdeps_otherfile.swift
• test/SourceKit/CodeComplete/complete_checkdeps_bridged.swift
• test/SourceKit/CodeComplete/complete_checkdeps_avoid_check.swift
• test/SourceKit/CodeComplete/complete_checkdeps_ownfile.swift
• test/SourceKit/CodeComplete/complete_checkdeps_clangmodule.swift
• test/SourceKit/CodeComplete/complete_checkdeps_swiftmodule.swift
• test/SourceKit/CodeComplete/complete_checkdeps_notifyupdate.swift
• test/Driver/linker.swift
• test/Driver/LegacyDriver/legacy-driver-propagates-response-file-to-new-driver.swift
• test/Driver/subcommands.swift

Additional information

No response

[ramonasuncion]

Hey, I’ll take a look at this and start working on it.

[hnrklssn]

Hey, I’ll take a look at this and start working on it.

Great! I've written in some more detail about it here: https://forums.swift.org/t/rfc-enabling-the-internal-shell-by-default-in-llvm-lit-tests/82148

Have you contributed to Swift before? I'd start with the lowest hanging fruit, like converting VAR=value cmd to env VAR=value cmd and cp {a,x}.swift to cp a.swift x.swift. Slightly less trivial is fixing this snippet in the lit config to not use a subshell (the $(basename \1) part):

config.target_build_swift_dylib = SubstituteCaptures(
        f"{escape_for_substitute_captures(config.target_build_swift)}"
        r" -parse-as-library -emit-library -o '\1'"
        r" -Xlinker -install_name -Xlinker @executable_path/$(basename '\1')"
    )

That one is used in quite a few of these tests, so fixing that would remove a big chunk, but getting rid of the subshell is a bit more involved.

Unfortunately I don't have a list of which test has which problem, but if you need help figuring that out just let me know :)
Don't be afraid to make small PRs! Even if it just fixes a single test case, that's way easier to review than one massive PR that fixes a bunch of unrelated issues at once.

[ramonasuncion]

First PR. I’m still getting up to speed and starting with the low-hanging fruit, just like you recommended.