Using libc++ with the "emulated pthreads" mode

[whitequark]

Now that WebAssembly/wasi-libc#518 has been merged thanks to the excellent work by @ArcaneNibble, we have pthread.h exist dormantly in single-threaded builds, only triggering an assertion if someone was using a function like pthread_create which is definitely missing. However, the original point of that PR was to enable the use of <atomic>, <mutex>, and other such headers in single-threaded WASI targets.

Libc++ can be built in single-threaded or multi-threaded mode, which changes a #define it embeds in a header, and also changes code generated for libc++.{a,so}. This is a problem given the direction chosen for handling pthread.h: right now, if I adjust the wasi-sdk build scripts to enable multi-threaded mode at all times, it becomes impossible to build any C++ code without also passing in -D_WASI_EMULATED_PTHREAD. This is obviously not a feasible way forward.

I can see three potential paths forward here:

1. Add a target variant, let's say wasm32-wasi-emuthreads, that has libc++ compiled in the multi-threaded mode and linked with the emulated pthreads library.
2. Ship a multi-threaded libc++ with the existing wasm32-wasi target, but modify the pthread integration code such that -D_WASI_EMULATED_PTHREAD isn't needed to build code that only needs e.g. <optional> but not <atomic>. This would involve extensive hacking at libc++ because right now e.g. <optional> depends on <atomic> (and many other seemingly non-threading-related headers depend, transitively, on pthreads functions).
3. Give up. Applications like Clang or Slang (a SystemVerilog frontend in C++) will not be buildable with upstream WASI SDK for single-threaded targets, limiting their deployability. I will likely switch to using my fork of the SDK and start to recommend it to anyone porting complex C++ applications.

To me, (2) seems not really a feasible solution given the resource constraints in practice, but I can also see that (1) may be viewed as objectionable. I could put in some effort towards implementing either (I'm halfway through solution 1) but I would like clear guidance from the upstream first.

cc @sbc100 @alexcrichton

[whitequark]

An option that makes (2) perhaps more manageable is to (somehow!) make references to functions like pthread_create be hard errors when invoked directly from user code, but soft errors (diagnosed at the linker stage without -lwasi-emulated-pthread and at runtime with -lwasi-emulated-pthread) when invoked indirectly via libc++.

Right now what happens is that libc++ has a function like this:

inline _LIBCPP_HIDE_FROM_ABI int __libcpp_thread_create(__libcpp_thread_t* __t, void* (*__func)(void*), void* __arg) {
  return pthread_create(__t, nullptr, __func, __arg);
}

and libc has a definition like this (without -D_WASI_EMULATED_PTHREAD):

#define pthread_create(...) ({ _Static_assert(0, "This mode of WASI does not have threads enabled; \
to enable stub functions which always fail, \
compile with -D_WASI_EMULATED_PTHREAD and link with -lwasi-emulated-pthread"); 0;})

and apparently if you put an always-failing static assert into an inline function it becomes basically a #error even if the function is never called. (This isn't what I expected, though I assume this is exactly what the standard says should happen.)

I don't know how to adjust this behavior without gross hacks. I don't suppose libc++ allows us to somehow change which pthread_create symbol it refers to, or interpose arbitrary preprocessor code before/after its internal headers (and frankly even if it allowed that I wouldn't want to use it anyway...)

[whitequark]

I can't think of any way to implement the route proposed above except to introduce a pthread_create function which causes a descriptive linker error (e.g. by linking to extern int A_symbol_whose_name_is_a_descriptive_error_message;) which is kind of terrible UX.

[sunfishcode]

I assume adding a new target variant would also have poor UX, in the case where a user doesn't know they need the target variant.

An extern int with a descriptive name is at least easily searchable, so we could add extra documentation explaining it in more detail, so that seems worth considering.

[alexcrichton]

Personally I continue to be in camp "let's just always have these symbols be defined, and they return errors at runtime for bits that aren't implemented". Effectively remove _WASI_EMULATED_PTHREAD and consider it always enabled. I'm under the impression that it would solve the issue here and work for various users.

I also understand though that this is a controversial point that has its own downsides. Personally, however, I view the downsides as far more palatable than "forced to fork the toolchain" or "forced to create a new target string".

[abrown]

Along those lines, if we still care about the ability to fail at compile time, we could flip the logic here from "emulated mode" to "strict mode:" so, instead of having to opt in to "emulated mode" to get things to compile, one would have to opt in to "strict mode" to fail at compile time if a program uses non-implemented functions like pthread_create. This would "just make things work" for people who just want things to compile (and probably don't mind the runtime error) but for those who only want to allow compilation of valid programs, they could opt-in to "strict mode."

[sbc100]

Along those lines, if we still care about the ability to fail at compile time, we could flip the logic here from "emulated mode" to "strict mode:" so, instead of having to opt in to "emulated mode" to get things to compile, one would have to opt in to "strict mode" to fail at compile time if a program uses non-implemented functions like pthread_create. This would "just make things work" for people who just want things to compile (and probably don't mind the runtime error) but for those who only want to allow compilation of valid programs, they could opt-in to "strict mode."

This is also the model that emscripten has where we have a strict mode (https://emscripten.org/docs/tools_reference/settings_reference.html#strict) that will disable unimplemented/emulated syscalls (https://emscripten.org/docs/tools_reference/settings_reference.html#allow-unimplemented-syscalls), but the default is to allow/emulate them.

[abrown]

@whitequark, how do you feel about an option 4 along the lines of what @alexcrichton suggested above? Option 4 would look like:

• by default, the problematic functions would fail at runtime (pthread_create, pthread_detach, pthread_join, pthread_tryjoin_np, pthread_timedjoin_np); this is what you would end up using, right?
• when a special "strict mode" flag is defined, these functions would fail to compile (much like today, but the user would have to opt in)
• optionally, when the "emulated mode" flag is defined, use the return 0 implementations from the current stub pthreads implementation (alternately, we could discuss removing the emulated mode entirely--does anyone still need this?)
• when threads are enabled (defined(_REENTRANT)), we keep the current implementation.

This avoids extra targets and extensive hacking; how do you feel about this approach?

[whitequark]

Yep, this would solve my use case and I have no philosophical objections to it!

[whitequark]

• optionally, when the "emulated mode" flag is defined, use the return 0 implementations from the current stub pthreads implementation (alternately, we could discuss removing the emulated mode entirely--does anyone still need this?)

So what I'm thinking is that these functions would always exist for linking purposes and would always fail (with -EAGAIN and such), but headers would be set up so that in "strict mode" you don't ever get to that linking step. This would mean that anyone using a foreign function interface from a different language would get a stub instead of an error, but I think this is an advantage if anything--very rarely do you actually want to patch the innards of a language stdlib to add more stubs there.

[whitequark]

Should I implement the design we've discussed here?

[alexcrichton]

That'd be most welcome, please feel free to!