David Arnold (Feb 20 2024 at 17:37): David Arnold (Feb 20 2024 at 17:37):There was a tool, I forgot the name, which abstracts a wide variety of models behind an openAPI _API_, so that they can be used in its stead from other tooling.
What was that name?
David Arnold (Feb 20 2024 at 17:45):It wasn't https://localai.io/, iirc, but it's one such tool.
Andreas (Feb 20 2024 at 17:49):ollama also has OpenAI support now
David Arnold (Feb 20 2024 at 17:51):Andreas schrieb:
ollama also has OpenAI support now
Maybe it was ollama. But do I misremeber that it could potentially run _any_ model from hugginface?
Andreas (Feb 20 2024 at 17:54):ollama pulls in a lot of models from Huggingface. You can take a loot at their library here: https://ollama.com/library
David Arnold (Feb 20 2024 at 17:56):Andreas schrieb:
ollama pulls in a lot of models from Huggingface. You can take a loot at their library here: https://ollama.com/library
Hm, interesting. Why aren't they consumed from upstream? I just wonder and feel like I miss something.
Andreas (Feb 20 2024 at 18:12):consumed from upstream?
Which would be from where?
David Arnold (Feb 20 2024 at 18:13):I mean: why do they host models from hugginface on their own site, if that makes sense as a question? Or if it doesn't I'm maybe having the wrong premise.
David Arnold (Feb 20 2024 at 18:14):Or is it just a matter of having a "registry" of compatible models?
Andreas (Feb 20 2024 at 18:18):that is a good question to which I have no exact answer right now :smiley: let me know if you find out.
However, there is something on the OpenAI compatible API I just found by accident: https://github.com/ollama/ollama/blob/main/docs/openai.md
Tim DeHerrera (Feb 20 2024 at 18:21):I dunno, I know perplexities api is openai compatible, which is one of the reasons I decided to pull the trigger on a pro subscription
David Arnold (Feb 20 2024 at 18:24):Here's a list: https://kleiber.me/blog/2024/01/07/six-ways-running-llm-locally/
But I cant recognize the thing I had in mind. Lost knowledge. :cry:
Andreas (Feb 20 2024 at 18:26):The problem is that this space moves so fast that it almost becomes irrelevant if you knew something existed three months ago
David Arnold (Feb 20 2024 at 18:31):Yeah, that is true! I guess from a user perspective (in my case: editor support), it's important to look for the emerging standard api, and choose tools wisely so that they aren't a one-way door. Ideally, one's pick will be the one you can grow within that ecosystem.
David Arnold (Feb 20 2024 at 18:37):Yeah! Burn them out!
Andreas (Feb 20 2024 at 20:28):Where is that one from?
David Arnold (Feb 22 2024 at 15:34):Oh, I don't remeber, but it's emblematic for misaligned expectaions in those ecosystems.
Andreas (Feb 23 2024 at 09:18):yes the development of the open source generative A.I. landscape is fast and chaotic right now
David Arnold (Feb 24 2024 at 01:05):https://github.com/janhq/nitro
Wasn't it, but seems promising. Small, based on llama.cpp
Andreas (Feb 24 2024 at 09:08):Even ROCm support seems to be on it's way! :confetti: https://github.com/janhq/nitro/issues/323
I might try it. But it looks more or less like lightweight ollama. However the problem is that in order to run it, I need the 20 GB + something docker image from AMD for ROCm. So the ollama-nitro difference pales in comparison.
David Arnold (Feb 24 2024 at 09:09):This also seems to be taken serious, from a rust perspective, even if shapeshifting quite some, atm: https://github.com/rustformers/llm
David Arnold (Feb 24 2024 at 10:37):Re GGUF, see also this choice of gpt4all:
GPT4All v2.5.0 and newer only supports models in GGUF format (.gguf). Models used with a previous version of GPT4All (.bin extension) will no longer work.
Hints are condensing into knowledge, it appears
Last updated: Nov 15 2024 at 11:45 UTC