Ayrton Mercado (Jun 21 2024 at 13:33): Ayrton Mercado (Jun 21 2024 at 13:33):@Shivaraj B H @Andreas Thanks for the work you've put into this. I am new to Nix and trying the ollama flake that uses cuda. From here
(I am new to both Linux and Nix, please let me know if I am running afoul of etiquette by posting the question here)
After following the instructions:
nix flake init -t github:shivaraj-bh/ollama-flake
and
nix run
I get:
warning: Git tree '/home/ayrton/.config/nixos' is dirty
trace: warning: cudaPackages.autoAddOpenGLRunpathHook is deprecated, use pkgs.autoAddDriverRunpathHook instead
error: builder for '/nix/store/v478jqmc2cwj3jgd6iajkqf4n5apdagv-ctranslate2-4.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.drv' failed with exit code 1;
last 2 log lines:
> Reading index https://pypi.org/simple/ctranslate2/
> The file ctranslate2-4.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl has not be found in the index https://pypi.org/simple/ctranslate2/
For full logs, run 'nix log /nix/store/v478jqmc2cwj3jgd6iajkqf4n5apdagv-ctranslate2-4.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.drv'.
error: 1 dependencies of derivation '/nix/store/9kipiyyaszpb43v83jimim0mrw4k2prk-python3.10-ctranslate2-4.3.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/40il2v03ljm00cyqs7qzlhbjck4hd4sh-python3-3.10.14-env.drv' failed to build
error: 1 dependencies of derivation '/nix/store/9kdigfnxwvb617z7q6am51cw4ii0gkc9-open-webui.drv' failed to build
error: 1 dependencies of derivation '/nix/store/9aagb1idpyycxcjrmwrd1cvn89p5h2ca-open-webui-wrapper.drv' failed to build
error: 1 dependencies of derivation '/nix/store/q84rx52x11vx6m228djw4vrjlbvczdya-process-compose-default.json.drv' failed to build
error: 1 dependencies of derivation '/nix/store/vznn94a1j7vs8b14w8bmnhrs4wsfk0jx-default.drv' failed to build
the nix log further states:
nix log /nix/store/v478jqmc2cwj3jgd6iajkqf4n5apdagv-ctranslate2-4.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.drv
warning: The interpretation of store paths arguments ending in .drv recently changed. If this command is now failing try again with '/nix/store/v478jqmc2cwj3jgd6iajkqf4n5apdagv-ctranslate2-4.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.drv^*'
Reading index https://pypi.org/simple/ctranslate2/
The file ctranslate2-4.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl has not be found in the index https://pypi.org/simple/ctranslate2/
Notification Bot (Jun 21 2024 at 13:34):A message was moved here from #nixify-llm > Mattermost plugin with local OpenAI compatible backend by Andreas.
Andreas (Jun 21 2024 at 13:34):I moved your message to a new topic @Ayrton Mercado
Ayrton Mercado (Jun 21 2024 at 13:35):Thanks!
Andreas (Jun 21 2024 at 13:44):I think I can reproduce the error on my end. I can't fix it though. But it looks like fetching something from PyPI didn't go the way it should have.
Srid (Jun 21 2024 at 13:49):I believe https://github.com/shivaraj-bh/ollama-flake has been upstreamed? cc @Shivaraj B H
@Ayrton Mercado Try
https://github.com/juspay/services-flake/tree/main/example/llm
https://x.com/nixos_asia/status/1803065244568244578
Andreas (Jun 21 2024 at 13:50):maybe the old one might be archived then? Plus an addition to the readme?
Ayrton Mercado (Jun 21 2024 at 13:53):Srid said:
I believe https://github.com/shivaraj-bh/ollama-flake has been upstreamed? cc Shivaraj B H
Ayrton Mercado Try
https://github.com/juspay/services-flake/tree/main/example/llm
Thanks Srid, trying now
Srid (Jun 21 2024 at 13:57):Just made a note about the 9GB download: https://github.com/juspay/services-flake/commit/0dc5d11cf56f4377c542c56653587225796a3416
Srid (Jun 21 2024 at 13:57):There's #services-flake channel where you can post questions about this example as well.
Ayrton Mercado (Jun 21 2024 at 14:20):Worked flawlessly, thank you for your kindness. Now I need to figure out how to switch ollama to use the GPU.
O/t...
Srid, do you have a suggestion as to how I can help in general? Like I said earlier I am new to nix but feel that I need to contribute.
Srid (Jun 21 2024 at 14:22):Ayrton Mercado said:
Now I need to figure out how to switch ollama to use the GPU.
This should help? https://community.flake.parts/services-flake/ollama#acceleration
Srid (Jun 21 2024 at 14:23):Ayrton Mercado said:
Srid, do you have a suggestion as to how I can help in general? Like I said earlier I am new to nix but feel that I need to contribute.
You are welcome to contribute to any of our repos. @Shivaraj B H maintains services-flake.
Documentation and stuff would also be useful to have -- the docs in services-flake as well as https://github.com/nixos-asia/website
Ayrton Mercado (Jun 21 2024 at 14:30):Will take a look. Thanks again.
Shivaraj B H (Jun 22 2024 at 12:03):Andreas said:
maybe the old one might be archived then? Plus an addition to the readme?
I have archived ollama-flake:
https://github.com/shivaraj-bh/ollama-flake
Last updated: Nov 15 2024 at 12:33 UTC