New virtual store method(s) to support builder stores

[lisanna-dettwyler]

Is your feature request related to a problem?

To allow the build hook mechanism to be fully deprecated but still provide facilities for implementing custom builders, we want to support a concept of "builder stores" which are lightweight store implementations that just implement custom building logic and no-op/error for other unrelated store methods. Today, build hooks have the advantage of receiving the path to build alongside its inputs, whereas with stores this information is passed in separate calls (addMultipleToStore, buildPathsWithResults) with no relation to each other other than timing / sequencing. For a builder store that wants to multiplex builds across multiple machines, it may not know ahead of time what machine to query for queryValidPaths or copy the paths to from addMultipleToStore before it receives a build request and assigns (or is given) a machine for it.

Proposed solution

Virtual store method(s) that allow you to pass a build request along with its inputs, using a similar signature as the existing build hook mechanism. It could look something like this:

virtual BuildResult buildDerivation(
        const StorePath & drvPath,
        const BasicDerivation & drv,
        StorePathSet & inputs,
        BuildMode buildMode = bmNormal);

This would provide a direct method for the daemon to invoke remote building via the store implementation in place of calling out to the build hook or having to call multiple store methods sequentially. Thus, the store can deal with copying the inputs at the same time as it performing the build, so it is guaranteed to know what machine to copy the inputs to.

inputs should be a StorePathSet because the custom store can just open the local store itself to access the underlying paths, just like the build hook currently does.

A default implementation of the virtual method should be provided to cover built-in stores that just calls existing methods in sequence to copy the inputs and build the derivation(s).

A builder store could be used in the following ways:

• A plugin, allowing usage of --store or specifying it in a machines file / --builder argument.
• A custom nix-daemon which passes the builder store to processConnection,
  • ...to create build services like nixbuild.net.
  • ...to enable usage with ssh-ng://localhost?remoteProgram=/path/to/my-daemon.
  • ...that runs as a service exposing a socket to use with NIX_REMOTE.

This also provides usage options with more flexibility for smarter scheduling, allowing the store to receive the entire closure of a build request rather than piecemeal when used with --store or the daemon model.

This should not require any modifications to the daemon protocol.

Alternative solutions

• Keeping the build hook mechanism (undesirable)

Additional context

• Remove the build hook mechanism #1221
• https://github.com/lisanna-dettwyler/nix-scheduler-hook
• obsidiansystems@build-store

Checklist

• checked latest Nix manual (source)
• checked open feature issues and pull requests for possible duplicates

---

Add 👍 to issues you find important.

[xokdvium]

One important note that Eelco brought up about this is that we need proper cancellation. Like how do you interrupt a build on a remote store? That's the primary reason the build hook exists now (at least that's how I understood it when we discussed this on a call some months ago).

Also building a BasicDerivation is a privileged operation IIUC. One has to first write the drv and let the store read it back to realise it.

[lisanna-dettwyler]

I'm not sure what help the build hook provides for cancellation, the build hook just calls sshStore->buildPathsWithResults and the ssh connection will just get immediately severed regardless of whether it's from the hook or the daemon directly.

Update: I see where the issue is, buildPathsWithResults is a blocking call, so to have it be cancellable it needs to either be invoked from a fork or reworked entirely to be non-blocking. Discussed with John and we are going to pursue reworking it to make it non-blocking.