Stream: nix

Topic: Running python process using process-compose


view this post on Zulip Soumyajit Behera (Jan 02 2024 at 09:46):

I am trying to run some process using process compose.
Using

in
   {
    process-compose =
      {
        settings.processes = commonConfig {
            load-test = {
              command = pkgs.writeShellApplication {
                name = "load-test";
                runtimeInputs = [pkgs.python3];
                text = ''
                  set -x
                  python3 Backend/load-test/setup/auth.py
                '';
              };
            };
          };
        };
  };
}

But this is throwing the below error
error: flake 'git+file:///Users/soumyajit.behera/backend-nammayatri/b-nammayatri' does not provide attribute 'packages.aarch64-darwin.load-test', 'legacyPackages.aarch64-darwin.load-test'

Can someone help me here?

view this post on Zulip Notification Bot (Jan 02 2024 at 09:50):

A message was moved here from #nix > Nix tutorial series by Soumyajit Behera.

view this post on Zulip Srid (Jan 02 2024 at 12:39):

Your config is wrong. It should be process-compose.load-test = not process-compose =

See example here: https://github.com/Platonic-Systems/process-compose-flake/blob/main/example/flake.nix#L20

Also, we have a stream for process-compose: #process-compose-flake

view this post on Zulip Soumyajit Behera (Jan 02 2024 at 13:39):

_:
{
  perSystem = perSystem@{ self', pkgs, lib, ... }: {
    process-compose.load-test-nix =
      {
        settings.processes = {
            echo-web.command = ''
               while true; do
                  echo "heloow"
                done
              '';

            auth = {
              command = pkgs.writeShellApplication {
                name = "auth";
                runtimeInputs = [ pkgs.python3 ];
                text = ''
                    python3 Backend/load-test/setup/auth.py
                '';
              };
            };
          };
      };
  };
}

With this there is no error but the process-compose is not coming up.

view this post on Zulip Srid (Jan 02 2024 at 13:57):

Make sure you are using the latest process-compose. If in nammayatri, use the latest common repo flake input.

view this post on Zulip Soumyajit Behera (Jan 03 2024 at 07:15):

The issue was with the older version of process compose .
Which needs a new argument for -p=0 (port) to come up.
So we can use -p=0 as argument in nix run.

view this post on Zulip Soumyajit Behera (Jan 03 2024 at 10:04):

let
  my-python-packages = ps: with ps; [
    ( buildPythonPackage rec {
      pname = "locust";
      version = "2.16.1";
      format = "pyproject";

      src = fetchFromGitHub {
        owner = "locustio";
        repo = "locust";
        rev = version;
        sha256 = "sha256-MKFzEFpDkD8e+ENZN0GAWYipPK9A4SCZZRHG5/5UaNM=";
      };

      patchPhase = ''
        echo 'version = "${version}"' > locust/_version.py
      '';

      propagatedBuildInputs = with python3Packages; [
        requests
        flask-basicauth
        flask-cors
        msgpack
        gevent
        pyzmq
        roundrobin
        geventhttpclient
        psutil
        typing-extensions
        configargparse
        setuptools
      ];
    })
  ];

I wanted to add locust to my env but this is giving error error: attribute 'buildPythonPackage' missing

I have tried importing { inputs, buildPythonPackage, fetchPypi, fetchFromGitHub, python3Packages, ... }:
But this is not working

view this post on Zulip Shivaraj B H (Jan 03 2024 at 10:12):

buildPythonPackage comes from pkgs.python3Packages.buildPythonPackage

view this post on Zulip Soumyajit Behera (Jan 03 2024 at 12:23):

error: builder for '/nix/store/rs8hwv9ra51064vxbdn05yz08kg1z801-python3.10-locust-2.19.1.drv' failed with exit code 1;
       last 10 log lines:
       >     _version_missing(config)
       >   File "/nix/store/gaafmp68vslf33sawfi97rywids1s1kg-python3.10-setuptools-scm-7.1.0/lib/python3.10/site-packages/setuptools_scm/__init__.py", line 108, in _version_missing
       >     raise LookupError(
       > LookupError: setuptools-scm was unable to detect version for /private/tmp/nix-build-python3.10-locust-2.19.1.drv-0/source.
       >
       > Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work.
       >
       > For example, if you're using pip, instead of https://github.com/user/proj/archive/master.zip use git+https://github.com/user/proj.git#egg=proj
       >
       > ERROR Backend subprocess exited when trying to invoke get_requires_for_build_wheel
       For full logs, run 'nix log /nix/store/rs8hwv9ra51064vxbdn05yz08kg1z801-python3.10-locust-2.19.1.drv'.
error: 1 dependencies of derivation '/nix/store/jgxlnqwwhnyahpvl68niz5qvrffcxpp5-ny-backend-env.drv' failed to build

Adding locust to the env.
Its giving this error.

view this post on Zulip Shivaraj B H (Jan 03 2024 at 13:16):

You can avoid the setuptools-scm problem like this:

{
  inputs.nixpkgs.url = "github:nixos/nixpkgs/nixpkgs-unstable";
  outputs = inputs@{...}: {
    packages.x86_64-linux.default =
    let
      pkgs = inputs.nixpkgs.legacyPackages.x86_64-linux;
    in
    with pkgs;
    with pkgs.python3Packages;
    ( buildPythonPackage rec {
      pname = "locust";
      version = "2.20.1";
      format = "pyproject";

      src = fetchFromGitHub {
        owner = "locustio";
        repo = "locust";
        rev = version;
        sha256 = "sha256-pPtH9rnIDFRUyWPPF7xmPtfpiBFIz9WLYTTV+W0s/es=";
      };

      patchPhase = ''
        echo 'version = "${version}"' > locust/_version.py
      '';
      SETUPTOOLS_SCM_PRETEND_VERSION = version;
      nativeBuildInputs = [ setuptools-scm ];

      propagatedBuildInputs = [
       requests
       flask-basicauth
       flask-cors
       msgpack
       gevent
       pyzmq
       geventhttpclient
       roundrobin
       psutil
       typing-extensions
       configargparse
       setuptools
      ];
    });

  };
}

See here: https://github.com/NixOS/nixpkgs/issues/41136#issuecomment-1101342698

view this post on Zulip Shivaraj B H (Jan 03 2024 at 13:17):

Also after this the package will complaint about roundrobin being missing as it is still an open PR here https://github.com/NixOS/nixpkgs/pull/185087

view this post on Zulip Shivaraj B H (Jan 03 2024 at 13:17):

You will have to create another buildPythonPackage for roundrobin and link it here

view this post on Zulip Shivaraj B H (Jan 03 2024 at 15:55):

@Soumyajit Behera
Here’s a flake that builds locust, just run nix build:

{
  inputs.nixpkgs.url = "github:nixos/nixpkgs/nixpkgs-unstable";
  outputs = inputs@{ ... }:
    let pkgs = inputs.nixpkgs.legacyPackages.x86_64-linux; in {
      packages.x86_64-linux.default =
        with pkgs;
        with pkgs.python3Packages;
        let
          roundrobin = buildPythonPackage rec {
            pname = "roundrobin";
            version = "0.0.4";

            src = fetchFromGitHub {
              owner = "linnik";
              repo = pname;
              rev = version;
              sha256 = "sha256-eedE4PE43sbJE/Ktrc31KjVdfqe2ChKCYUNIl7fir0E=";
            };

            checkPhase = ''
              runHook preCheck
              ${python.interpreter} -m unittest discover
              runHook postCheck
            '';

            pythonImportsCheck = [ pname ];

          };
        in
        (buildPythonPackage rec {
          pname = "locust";
          version = "2.20.1";
          format = "pyproject";

          src = fetchFromGitHub {
            owner = "locustio";
            repo = "locust";
            rev = version;
            sha256 = "sha256-pPtH9rnIDFRUyWPPF7xmPtfpiBFIz9WLYTTV+W0s/es=";
          };

          patchPhase = ''
            echo 'version = "${version}"' > locust/_version.py
          '';
          SETUPTOOLS_SCM_PRETEND_VERSION = version;
          nativeBuildInputs = [ setuptools-scm ];
          pythonImportsCheck = [ pname ];

          propagatedBuildInputs = [
            requests
            flask-basicauth
            flask-cors
            msgpack
            gevent
            pyzmq
            geventhttpclient
            roundrobin
            psutil
            typing-extensions
            configargparse
            setuptools
          ];
        });

    };
}

view this post on Zulip Srid (Jan 03 2024 at 16:04):

(One of these days we probably want to create a ... python-flake)

view this post on Zulip Shivaraj B H (Jan 04 2024 at 07:10):

Agreed


Last updated: Nov 15 2024 at 11:45 UTC