Stream: services-flake

Topic: CUDA acceleration in ollama

view this post on Zulip Ayrton Mercado (Jun 24 2024 at 11:17):

Guys, adding 

            **acceleration = "cuda";**

has turned into a bit of an inside joke fr me. The resulting error is why I almost gave up on NixOS when I first picked it up a few weeks ago, namely I get the ... has an unfree license (‘CUDA EULA’), refusing to evaluate. whith a prescription for solutions.

I have set the env variable

 $env.NIXPKGS_ALLOW_UNFREE
1

added to my flake

  packages.config.allowUnfree = true;

It is my understanding that the allowUnfree property I'm setting is not scoped correctly, but I'm having a tough time figuring out the correct scope.

Found a thread on stackoverflow that probably has the answer but I can't follow it. (square peg/round hole).

My biggest issue is the dissonance between calling a nix file a configuration and what it actually is (IMO) a program. I have yet to crack the language.

I'm going to start here. With some luck I'll be able to fix this by the end of the week :)

view this post on Zulip Notification Bot (Jun 24 2024 at 11:19):

Shivaraj B H has marked this topic as unresolved.

view this post on Zulip Notification Bot (Jun 24 2024 at 11:19):

A message was moved here from #nixify-llm > ✔ Issues with ollama flake by Shivaraj B H.

view this post on Zulip Shivaraj B H (Jun 24 2024 at 11:23):

@Ayrton Mercado I have to document this, you can use this for now:

# Inside perSystem
{
  imports = [
    "${inputs.nixpkgs}/nixos/modules/misc/nixpkgs.nix"
  ];
  nixpkgs = {
    # Required for CUDA
    config.allowUnfree = true;
  };
}

view this post on Zulip Shivaraj B H (Jun 24 2024 at 11:25):

ollama-flake exports a nixpkgs module that would handle this, but since that’s not the case after upstreaming to services-flake, it wasn’t working out of the box for you.

view this post on Zulip Shivaraj B H (Jun 24 2024 at 11:43):

Hold on, there seems to a bug in services-flake that I just fixed: https://github.com/juspay/services-flake/commit/8145ba10cb02dc0a843bba371fc2d42cea7fd226

view this post on Zulip Shivaraj B H (Jun 24 2024 at 11:54):

This commit demonstrates how to do it in the current services-flake’s example/llm: https://github.com/shivaraj-bh/services-flake/commit/bf3bf65a920dc6e3b4eb749ec3809fdeaf4b9360

view this post on Zulip Shivaraj B H (Jun 24 2024 at 12:00):

Also documented it: https://github.com/juspay/services-flake/commit/fb6de01eb6e7d2ac1c540fd399b9c0f447ce0ab7

view this post on Zulip Ayrton Mercado (Jun 24 2024 at 12:57):

Shivaraj B H said:

Also documented it: https://github.com/juspay/services-flake/commit/fb6de01eb6e7d2ac1c540fd399b9c0f447ce0ab7

Confirmed on my end, the flake compiles

Last updated: Nov 15 2024 at 12:33 UTC