std: Use libc for filesystem ops on WASI targets

[alexcrichton]

This commit is a large change to the implementation of filesystem and other system-related operations on WASI targets. Previously the standard library explicitly used the wasi crate at the 0.11.x version track which means that it used WASIp1 APIs directly. This meant that std was hard-coded to use WASIp1 syscalls and there was no separate implementation for the WASIp{2,3} targets, for example. The high-level goal of this commit is to decouple this interaction and avoid the use of the wasi crate on the WASIp2 target.

Historically when WASIp1 was originally added to Rust the wasi-libc library was in a much different position than it is today. Nowadays Rust already depends on wasi-libc on WASI targets for things like memory allocation and environment variable management. As a libc library it also has all the functions necessary to implement all filesystem operations Rust wants. Recently wasi-libc additionally was updated to use WASIp2 APIs directly on the wasm32-wasip2 target instead of using wasm32-wasip1 APIs. This commit is leveraging this work by enabling Rust to completely sever the dependence on WASIp1 APIs when compiling for wasm32-wasip2. This is also intended to make it easier to migrate to wasm32-wasip3 internally in the future where now only libc need be updated and Rust doesn't need to explicitly change as well.

This commit fairly heavily changes the internals of the implementation of WASI filesystem operations. The basis of implementation is now libc-style APIs as opposed to WASIp1-style-APIs which necessitated a number of changes throughout. Additionally the std::os::wasi::fs module has had a few API-breaking changes as well, but this module is also unstable. While this module has been "stable" since inception in the sense that it hasn't changed, this PR is what the unstable status was sort of trying to buy in terms of future flexibility to make changes. For users of stable std::fs interfaces this is not intended to be a breaking change but there nevertheless may be minor issues here and there.

Concrete changes here are:

• std for wasm32-wasip2 no longer depends on wasi 0.11.x
• The implementation of std::os::wasi::fs, which was previously unstable and still is, is now only available on wasm32-wasip1 and is implemented largely directly in terms of WASIp1 APIs by directly using the wasi 0.11.x crate.
• std::os::wasi::fs::MetadataExt, which was previously unstable and still is, has removed methods that duplicate functionality in std::fs::Metadata as there's no point in having them.
• std::os::wasi::fs::FileTypeExt, which was previously unstable and still is, lost the ability to distinguish between dgram/stream sockets. WASIp1 sockets were never really supported, though, so this isn't much of a loss.
• The internal WasiFd type has been "gutted" to have far fewer methods internally. This still represents a file descriptor owned by wasi-libc, however.
• The std::sys::fs::wasi implementation is overhauled to largely use the libc crate and its APIs directly (which go to wasi-libc). This drastically changes the ReadDir implementation, for example.
• The std::sys::fs::wasi::OpenOptions API now has two ways of opening files. By default libc::open is used but of std::os::wasi::fs::OpenOptionsExt is used then a WASIp1-specific API is used to open a file.
• Types like FileAttr and FileType now use libc representations at-rest instead of WASIp1 internals.
• Various transmutations for iovec-style APIs are consolidated into a WASI-specific module close to the definition of the IoSlice{,Mut} types to make it easier to audit unsafe guarantees.
• The wasip2 OS module gained a custom implementation of the helpers module as the WASIp1 version of this module is no longer applicable.

I'd like to do follow-up PRs after this one to reorganize std::os::wasi* a bit to be more amenable for WASIp{2,3} targets as well, but that'll come in a future PR. For now this is tasked with the high-level goal of removing the dependency on WASIp1 APIs in the standard library by relying on wasi-libc to natively use WASIp2 APIs.

[rustbot]

r? @Mark-Simulacrum

rustbot has assigned @Mark-Simulacrum.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[Mark-Simulacrum]

At a high level this seems OK (cc @joboet as you've been working on a bunch of sys cleanups).

If I'm understanding the new code right, it looks like the future we'd expect is to remove the wasi modules entirely in favor of wasipN being cfg(unix) for the most part? Does that understanding seem right? I see that a bunch of the code "added" here seems like it's copy/paste from cfg(unix) modules counterparts.

For example, maybe library/std/src/sys/io/io_slice/wasi.rs should already be removed and cfgs rerouted to the iovec module, with the extra methods gated on cfgs in that? I think that would also apply to other similar modules at this point. Diffing locally (diff -u <(git show cb4a430b8349d9216f2c58d921a1b3a5aa6a6652:library/std/src/sys/io/io_slice/wasi.rs) library/std/src/sys/io/io_slice/iovec.rs | less) it seems like the only real difference is the extra conversion methods.

[alexcrichton]

That's a good point! You have also made me realize that most of the fs implementation is entirely redundant as well. Almost all of the functions the unix variant uses are implemented by wasi-libc which means we could use wasi-libc directly. The original reason for wasi having its own implementation was to implement everything natively in Rust which requires handling preopens to translate "open this path" to "open this sub-path rooted from this fd", or changing open calls to openat effectively. That's all done internally by wasi-libc anyway, however, so there's not really any need for rust's standard library to duplicate all of that any more.

The main difference between wasi and unix comes down to extensions where on Unix there's all sorts of extra symbols/extensions/etc. That could be handled relatively easily internally though with #[cfg] in the unix implementation (which already has a lot of #[cfg]). Basically I'm rationalizing to myself that I believe it still makes sense for wasi to not be cfg(unix), but using the unix implementations locally within the standard library could make sense.

Lemme test out those ideas and see if I can't delete even more code here.

[alexcrichton]

Ok I think that actually worked out really well -- alexcrichton@6760496 -- so I'm going to switch this PR to taking that approach. Effectively WASIp{1,2,3} will look like "unix" for operations related to the filesystem, threads, timers, etc. There's still platform differences for things like getting arguments and other bits here and there but I think those, in the limit of time, can go through libc as well.

Effectively wasi-libc has matured/changed enough to the point that I think it's more reasonable to go through that for various calls and that should reduce the size of the WASI-specific burden in the standard library. Additionally this reduces the distinction between WASIp1 and WASIp2 even further to by unifying their "pal" module into one since there's virtually no difference now. Overall this feels like a much better state of affairs to me.

This all depends on various libc changes, however, so I'll work on getting those upstream. In the meantime I'll mark this as a draft and I'll ping when this is ready for another round once libc is updated.