I looked through documentation of Weaviate but I cant find how to run Weaviate locally.
nix run nixpkgs#weaviate -- --scheme=http
That uses all the default configs, except I ask it not to use TLS, otherwise you will have to provide the location to the certificates and a bunch of other TLS related stuff
In services-flake, we can assume it is http
by default — as we only target local development and CI
workflows.
I wrote this basic weaviate process:
{
settings.processes.weaviate1.command = pkgs.writeShellApplication {
name = "start-weaviate";
runtimeInputs = [ pkgs.weaviate ];
text = ''
exec weaviate --scheme=http --host 0.0.0.0 --port 8080
'';
};
}
But I don't know why running it doesn't always produce logs
swappy-20240522_175414.png
swappy-20240522_175259.png
I am not very certain, but it could be an upstream process-compose bug. I have noticed that when you switch back to the process you see logs, you can do that if there are more than one processes
Anyways, you could also pass “-t=false” to process-compose to disable TUI mode
That should do that job as well
Right now this is my weaviate.nix
file:
{ pkgs, lib, name, config, ... }:
let
inherit (lib) types;
in
{
options = {
enable = lib.mkEnableOption name;
package = lib.mkPackageOption pkgs "weaviate" { };
outputs.settings = lib.mkOption {
type = types.deferredModule;
internal = true;
readOnly = true;
default = {
processes = {
"${name}" =
let
startScript = pkgs.writeShellApplication {
name = "start-weaviate";
runtimeInputs = [ config.package ];
text = ''
exec weaviate --scheme=http --host 0.0.0.0 --port 8080
'';
};
readyScript = pkgs.writeText "ready.py" ''
import weaviate
client = weaviate.connect_to_local(port=8080)
client.close()
'';
in
{
command = startScript;
readiness_probe = {
exec.command = "${(pkgs.python3.withPackages (p: [ p.weaviate-client ]))}/bin/python ${readyScript}";
initial_delay_seconds = 2;
period_seconds = 10;
timeout_seconds = 4;
success_threshold = 1;
failure_threshold = 5;
};
namespace = name;
# https://github.com/F1bonacc1/process-compose#-auto-restart-if-not-healthy
availability.restart = "on_failure";
};
};
};
};
};
}
and this is weaviate_test.nix
file:
{ pkgs, config, ... }: {
services.weaviate."weaviate1".enable = true;
settings.processes.test =
let
cfg = config.services.weaviate."weaviate1";
testScript = pkgs.writeText "test.py" ''
import weaviate
client = weaviate.connect_to_local(
port=8080,
)
client.close()
'';
in
{
command = pkgs.writeShellApplication {
runtimeInputs = [
cfg.package
(pkgs.python3.withPackages (python-pkgs: [
python-pkgs.weaviate-client
]))
];
text = ''
exec python3 ${testScript}
'';
name = "weaviate-test";
};
depends_on."weaviate1".condition = "process_healthy";
};
}
command just run weaviate
works fine but when i run just test weaviate
I get error:
swappy-20240522_235102.png
can’t reproduce. Tried it on macOS the test is passing.
I added other config options and even doc in this commit. Can you see if there is anything missing
I can reproduce your error on linux
I added other config options and even doc in this commit. Can you see if there is anything missing
Looks neat! Only missing piece will be the test failing on linux. Weaviate is doing something different on macOS, it works there.
It works with just run weaviate
right?
And the only difference i can see is running process-compose with -t=false flag
I added --no-sandbox to
test service:
nix --no-sandbox build ./test#checks.$(nix eval --impure --expr "builtins.currentSystem").{{service}} --override-input services-flake . -L
and it works
I added other config options and even doc in this commit. Can you see if there is anything missing
You are missing dataDir
option for the service
Also, Usage of the weaviate.conf.json file is deprecated and will be removed in the future. Please use environment variables.
I changed the configs to use environment variables commit also added dataDir option.
and found a better way to check for readiness and for test without the python scripts. commit
Great!
I added --no-sandbox to and it works
I don’t why weaviate would require to disable sandbox mode.
I thought it was because of weaviate telemetry but even after disabling it, test won't pass.
I think I figured it out. It has to do with https://github.com/hashicorp/memberlist. Although I am yet to find the exact reason, but setting this ENV makes the test pass:
diff --git a/nix/weaviate_test.nix b/nix/weaviate_test.nix
index a766c58..31d7975 100644
--- a/nix/weaviate_test.nix
+++ b/nix/weaviate_test.nix
@@ -1,5 +1,10 @@
{ pkgs, config, ... }: {
- services.weaviate."weaviate1".enable = true;
+ services.weaviate."weaviate1" = {
+ enable = true;
+ envs = {
+ CLUSTER_ADVERTISE_ADDR = "127.0.0.1";
+ };
+ };
settings.processes.test =
let
This is what lead me to it: https://github.com/weaviate/weaviate/blob/9e924cc88933b3042e27e30ad4abf26ab9ab78f8/usecases/cluster/state.go#L84-L86
I think it has to do with the fact that the default value for AdvertiseAddr
config is an empty string: https://github.com/hashicorp/memberlist/blob/3f82dc10a89f82efe300228752f7077d0d9f87e4/config.go#L308
But what I fail to understand is, how does it work when sandbox is disabled. I think the question can be better framed as: What does empty addr translate to?
When AdvertiseAddr
is empty it goes to this path:
https://github.com/hashicorp/memberlist/blob/3f82dc10a89f82efe300228752f7077d0d9f87e4/net_transport.go#L160
and calls GetPrivateIP
function which I guess in sandbox mode can't find any ip
Should i use types.raw as type of value of environment? other than int, str, list of str and bool, is there anything else? at least for environment values of weaviate
I forgot to update envs to environment in weaviate.md file
https://github.com/juspay/services-flake/blob/main/doc/weaviate.md?plain=1#L32
Last updated: Oct 12 2024 at 21:45 UTC