Separate stores for evaluation, building and storing the result

[edolstra]

For several use cases it would be useful to distinguish between the stores used to store .drv files, to build derivations, and to store the resulting closure.

For example:

• nix path-info --store https://cache.nixos.org nixpkgs#hello doesn't work because the evaluation of nixpkgs#hello tries to create .drv files in cache.nixos.org.
• For NixOps, we want to do evaluation on the local machine, do builds on some arbitrary machines (could be the local machine, the destination machine, or some other builder) and store the result in the destination machine.
• If /nix/store is a high-latency remote store (as with flox), then copying lots of little .drv files to the remote is slow. So it's better to evaluate locally and only copy .drv closures to the remote as needed and in a single round-trip.

[domenkozar]

Another use case is #3946

[edolstra]

So the fully general solution would be that buildPaths is no longer a Store method, but a static function that takes a stores configuration like this:

struct Stores {
  // The store where the build result should appear.
  ref<Store> destStore;

  // The store that contains the .drv file to be built.
  ref<Store> evalStore;

  // The stores that can build stuff. (This replaces the `build-remote` hook.)
  std::vector<ref<Store>> buildStores;
}

[rickynils]

If something like --build-store is ever implemented, couldn't that entirely replace the concept of remote builders too? It would be great to not have the two closely related but separate concepts of "remote builders" and "remote stores". So instead of having --builders you could maybe just add multiple --build-store arguments?

And while I'm here wishing for stuff, I could just as well throw in something like --fetch-store to replace substituters too. I think it would be awesome to "trickle up" the internal store abstraction like this, since it makes the concepts simpler for the user, and at the same time more flexible which opens up for new interesting usage scenarios in distributing builds, caches and deployments.