Stream: cull-os

Topic: Things to not do in the realizer protocol

RGBCube (Nov 01 2024 at 15:46):

Well, this will be a list of weird things in the Nix store protocol for now and I'll decide on what to do to avoid them later

RGBCube (Nov 01 2024 at 16:06):

First off: no copying files into realizers. Only derivations.

You know how when you do

derivation {
  builder = ./builder;
}

Nix first

1. Copies the file to the store
2. Then it actually creates the the derivation files with that path referenced

And how that file itself isn't actually a derivation? But instead is just a file?

RGBCube (Nov 01 2024 at 16:31):

We can avoid this by letting people specify paths inside derivations:

derive {
  paths.builder = executable $ fileContent "binary contents of it here";

  command = "builder"; # relative to pwd by default
}

These "paths" inside derivations will be an attribute of the derivation itself.

Now, when the command is run, the process will see this filesystem layout:

/
  var/lib/real/xxxxx-that-drv/ # this is our pwd
    builder

But of course this is quite inefficient. Every time we change that derivation, we have to write another file with the exact same contents. How do we avoid this?

By making a derivation with only that file, of course:

\thatfileder = derive {
  paths = outputDirectory {
    builder = executable $ fileContent "binary contents here";
  };
};

derive {
  command = pathOf thatfileder + "/builder";
}

This will also mandate that our derivation should not require a command, which is a good thing.

This will be what things like ./foo.sh and ./mydir desugar to too. These two are equivalent:

derive {
  paths.`foo.sh` = ./foo.sh;
  paths.mydir = ./mydir;
}

derive {
  paths.foo = (pathOf $ derive {
    paths = outputDirectory {
      # that executable flag for the file object is automatically determined
      `foo.sh` = executable $ fileContents $ read "foo.sh";
    };
  }) + "/foo.sh";

  paths.mydir = (pathOf $ derive {
    paths = outputDirectory {
      idk = fileContents $ readFile "aaa";
      # and so on, no nested derivations (or maybe there will be?)
    };
  });
}

RGBCube (Nov 01 2024 at 16:54):

Notice how we don't need the concept of "outputs" where we magically get mutable store paths, as we can now specify output directories in the filesystem layout, right inside our derivation? These directories will persist even after the derivation builds. But we have another problem now:

How do we make them content addressed and garbage collect them individually, without affecting the integrity? Luckily there is a smart way out.

We now hash the derivation individually with separate views for every output directory, while replacing self references with a fixed magic hash, and move them to their respective /var/lib/real/xxhashofthisoutput-llvm/ and finally replace the cross and self references with the respective hashes of the output derivations.

RGBCube (Nov 01 2024 at 17:01):

Though, as discussed earlier, this will require even more communication for clients to find out paths. I'll tackle this when I'm actually implementing it.

RGBCube (Nov 01 2024 at 17:10):

One thing this WILL break is binaries not using absolute references when cross referencing other outputs, but I'm sure nothing ever does that so it's fine.

Or maybe we can fix this by symlinking the other output directories (of which are not the content addressed one) into every output derivation.

Last updated: Nov 15 2024 at 13:04 UTC