David Arnold (Mar 06 2024 at 19:42): David Arnold (Mar 06 2024 at 19:42):within a systemd service
synapse.http.client: [sentinel] Error sending request to GET https://sistema.frx.localhost/.well-known/openid-configuration: RequestTimedOutError 504: Timeout [...]
From the host:
❯ curl -L https://sistema.frx.localhost/api/method/frappe.integrations.oauth2.openid_configuration
HTTP/2 200
server: nginx
date: Wed, 06 Mar 2024 19:42:27 GMT
content-type: application/json
content-length: 763
vary: Accept-Encoding
set-cookie: sid=Guest; Expires=Sat, 09 Mar 2024 14:42:26 GMT; Secure; HttpOnly; Path=/; SameSite=Lax
set-cookie: system_user=no; Secure; Path=/; SameSite=Lax
set-cookie: full_name=Guest; Secure; Path=/; SameSite=Lax
set-cookie: user_id=Guest; Secure; Path=/; SameSite=Lax
set-cookie: user_image=; Secure; Path=/; SameSite=Lax
x-frame-options: SAMEORIGIN
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
strict-transport-security: max-age=63072000; includeSubDomains; preload
referrer-policy: same-origin, strict-origin-when-cross-origin
{
"issuer": "https://sistema.frx.localhost",
"authorization_endpoint": "https://sistema.frx.localhost/api/method/frappe.integrations.oauth2.authorize",
"token_endpoint": "https://sistema.frx.localhost/api/method/frappe.integrations.oauth2.get_token",
"userinfo_endpoint": "https://sistema.frx.localhost/api/method/frappe.integrations.oauth2.openid_profile",
"revocation_endpoint": "https://sistema.frx.localhost/api/method/frappe.integrations.oauth2.revoke_token",
"introspection_endpoint": "https://sistema.frx.localhost/api/method/frappe.integrations.oauth2.introspect_token",
"response_types_supported": [
"code",
"token",
"code id_token",
"code token id_token",
"id_token",
"id_token token"
],
"subject_types_supported": [
"public"
],
"id_token_signing_alg_values_supported": [
"HS256"
]
}
What's the matter?
David Arnold (Mar 06 2024 at 19:44):❯ cat /etc/nsswitch.conf
───────┬────────────────────────────────────────────────────────────────────────
│ File: /etc/nsswitch.conf
───────┼────────────────────────────────────────────────────────────────────────
1 │ passwd: files systemd
2 │ group: files [success=merge] systemd
3 │ shadow: files
4 │
5 │ hosts: mymachines resolve [!UNAVAIL=return] files myhostname dns
6 │ networks: files
7 │
8 │ ethers: files
9 │ services: files
10 │ protocols: files
11 │ rpc: files
───────┴────────────────────────────────────────────────────────────────────────
^^ it doesn't matter whether I use systemd-resolved or not...
❯ cat /etc/hosts
───────┬────────────────────────────────────────────────────────────────────────
│ File: /etc/hosts
───────┼────────────────────────────────────────────────────────────────────────
1 │ 127.0.0.1 localhost
2 │ ::1 localhost
───────┴────────────────────────────────────────────────────────────────────────
❯ cat /etc/hostname
───────┬────────────────────────────────────────────────────────────────────────
│ File: /etc/hostname
───────┼────────────────────────────────────────────────────────────────────────
1 │ frx
───────┴────────────────────────────────────────────────────────────────────────
is the systemd service configured in a private network namespace or something?
David Arnold (Mar 06 2024 at 19:57):I'm not really sure if I can tell anything apart here:
17 │ ExecStartPre=/nix/store/l9f43mkpwah0md2gi4hhzmsxxdlc56qm-unit-script-ma
│ trix-synapse-pre-start/bin/matrix-synapse-pre-start
18 │ ExecStartPre=+/nix/store/680aj7bs2gbr5q18dgapzwy04vicn1q9-matrix-synaps
│ e-fix-permissions
19 │ Group=matrix-synapse
30 │ ProtectKernelLogs=true
20 │ LockPersonality=true
21 │ NoNewPrivileges=true
22 │ PrivateDevices=true
23 │ PrivateTmp=true
24 │ PrivateUsers=true
25 │ ProcSubset=pid
26 │ ProtectClock=true
27 │ ProtectControlGroups=true
28 │ ProtectHome=true
29 │ ProtectHostname=true
30 │ ProtectKernelLogs=true
31 │ ProtectKernelModules=true
32 │ ProtectKernelTunables=true
33 │ ProtectProc=invisible
34 │ ProtectSystem=strict
35 │ ReadWritePaths=/var/lib/matrix-synapse
36 │ ReadWritePaths=/var/lib/matrix-synapse/media_store
37 │ RemoveIPC=true
38 │ Restart=on-failure
39 │ RestrictAddressFamilies=AF_INET
40 │ RestrictAddressFamilies=AF_INET6
41 │ RestrictAddressFamilies=AF_UNIX
:
43 │ RestrictRealtime=true
44 │ RestrictSUIDSGID=true
45 │ SystemCallArchitectures=native
46 │ SystemCallFilter=@system-service
47 │ SystemCallFilter=~@resources
48 │ SystemCallFilter=~@privileged
49 │ Type=notify
50 │ UMask=0077
51 │ User=matrix-synapse
52 │ WorkingDirectory=/var/lib/matrix-synapse
───────┴────────────────────────────────────────────────────────────────────────
From what I gather, assuming you are trying to run Synapse with OIDC, there might be some issues on Synapse's GitHub related to this:
https://github.com/matrix-org/synapse/issues/9662
https://github.com/matrix-org/synapse/issues/8741
https://github.com/matrix-org/synapse/issues/16004
Judging from a first look, it might have something to do with TLS certificates.
David Arnold (Mar 06 2024 at 20:07):Oh! What a deceiving traceback! :smile:
Tim DeHerrera (Mar 06 2024 at 20:07):PrivateNetwork=true would indicate a private network namespace, so looks like that is not it. TLS would have been my next guess, but that usually doesn't result in a 504 :thinking:
Andreas (Mar 06 2024 at 20:08):It appears Synapse has a use_insecure_ssl_client_just_for_testing_do_not_use: true parameter setting you might try and see if it works...
David Arnold (Mar 06 2024 at 20:08):Yeah, it just doesn't work with my frx.localhost test domain in VM...
I'm setting up static certs, but it seams python doesn't acquire the root properly... I remember this is a pita and one might readily have to patch half of nixpkgs to get it into one of these certify packages that are a transitive dependency to half of the tree.
David Arnold (Mar 06 2024 at 20:39):from twisted.internet.ssl import (
CertificateOptions,
ContextFactory,
TLSVersion,
platformTrust, # this one is twisted :-)
)
platformTrus: [...] any process where the SSL_CERT_FILE environment variable is set to the path of a file containing your desired CA certificates bundle.
Tim DeHerrera (Mar 06 2024 at 20:42):if you are running this service on localhost, perhaps tls isn't necessary anyway?
David Arnold (Mar 06 2024 at 20:44):Unfortunately, it's a requirement that is part of the oidc stack, which otherwise would just error out, somewhat reasonably, actually.
David Arnold (Mar 06 2024 at 21:00):This seemed to have ultimately appeased twisted:
{
systemd.services.matrix-synapse.environment = {
SSL_CERT_FILE = config.environment.etc."ssl/certs/ca-certificates.crt".source;
};
}
Thanks for the hint about certs! :smile:
David Arnold (Mar 06 2024 at 21:03):This is what honors the effect of:
security.pki.certificateFiles = [myca];
David Arnold has marked this topic as resolved.
Last updated: Nov 15 2024 at 12:33 UTC