public inbox for [email protected]help / color / mirror / Atom feed
Patroni 4.1.0 causes issues because of systemd notify 4+ messages / 3 participants [nested] [flat]
* Patroni 4.1.0 causes issues because of systemd notify @ 2025-10-22 16:16 Hauke Bruno Wollentin <[email protected]> 2025-11-06 20:34 ` Re: Patroni 4.1.0 causes issues because of systemd notify Richard Huxton <[email protected]> 2025-11-07 04:06 ` Re: Patroni 4.1.0 causes issues because of systemd notify Aaron Pavely <[email protected]> 0 siblings, 2 replies; 4+ messages in thread From: Hauke Bruno Wollentin @ 2025-10-22 16:16 UTC (permalink / raw) To: [email protected] Hi all, while spinning up new Patroni cluster with version 4.1.0 (which has been added to the Debian repo 2 days ago), my clusters won’t bootstrap properly anymore. At first, logs looks clear but anyways, all systemd operations like start/stop/restart will run into a timeout, causing the service to stop which then will break the cluster at all. It seems like the new systemd notify feature is causing this, since using Patroni 4.0.7 with `Type=simple` in the systemd unit file works pretty fine. I’m on Debian 12 and psql 17. This is the patroni log after bootstrapping. Everything works fine until the 30s timeout of the systemd unit kicks in: Oct 22 17:43:09 hetzner-lab01 patroni[13066]: Success. You can now start the database server using: Oct 22 17:43:09 hetzner-lab01 patroni[13066]: /usr/lib/postgresql/17/bin/pg_ctl -D /var/lib/postgresql/patroni/data -l logfile start Oct 22 17:43:09 hetzner-lab01 patroni[13059]: 2025-10-22 17:43:09,768 INFO: postmaster pid=13083 Oct 22 17:43:09 hetzner-lab01 patroni[13084]: localhost:5432 - no response Oct 22 17:43:09 hetzner-lab01 patroni[13083]: 2025-10-22 17:43:09.780 CEST [13083] LOG: starting PostgreSQL 17.6 (Debian 17.6-2.pgdg12+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14+deb12u1) 12.2.0, 64-bit Oct 22 17:43:09 hetzner-lab01 patroni[13083]: 2025-10-22 17:43:09.780 CEST [13083] LOG: listening on IPv4 address "0.0.0.0", port 5432 Oct 22 17:43:09 hetzner-lab01 patroni[13083]: 2025-10-22 17:43:09.780 CEST [13083] LOG: listening on IPv6 address "::", port 5432 Oct 22 17:43:09 hetzner-lab01 patroni[13083]: 2025-10-22 17:43:09.782 CEST [13083] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432” Oct 22 17:43:09 hetzner-lab01 patroni[13087]: 2025-10-22 17:43:09.789 CEST [13087] LOG: database system was shut down at 2025-10-22 17:43:09 CEST Oct 22 17:43:09 hetzner-lab01 patroni[13083]: 2025-10-22 17:43:09.798 CEST [13083] LOG: database system is ready to accept connections Oct 22 17:43:09 hetzner-lab01 systemd[1]: patroni.service: Got notification message from PID 13083, but reception only permitted for main PID 13059 Oct 22 17:43:10 hetzner-lab01 patroni[13091]: localhost:5432 - accepting connections Oct 22 17:43:10 hetzner-lab01 patroni[13093]: localhost:5432 - accepting connections Oct 22 17:43:10 hetzner-lab01 patroni[13059]: 2025-10-22 17:43:10,814 INFO: establishing a new patroni heartbeat connection to postgres Oct 22 17:43:10 hetzner-lab01 patroni[13059]: 2025-10-22 17:43:10,918 INFO: running post_bootstrap Oct 22 17:43:11 hetzner-lab01 patroni[13059]: 2025-10-22 17:43:11,225 INFO: initialized a new cluster Oct 22 17:43:11 hetzner-lab01 patroni[13059]: 2025-10-22 17:43:11,373 INFO: no action. I am (hetzner-lab01), the leader with the lock Oct 22 17:43:21 hetzner-lab01 patroni[13059]: 2025-10-22 17:43:21,276 INFO: no action. I am (hetzner-lab01), the leader with the lock Oct 22 17:43:31 hetzner-lab01 patroni[13059]: 2025-10-22 17:43:31,368 INFO: no action. I am (hetzner-lab01), the leader with the lock Oct 22 17:43:37 hetzner-lab01 systemd[1]: patroni.service: start operation timed out. Terminating. Oct 22 17:43:37 hetzner-lab01 patroni[13083]: 2025-10-22 17:43:37.987 CEST [13083] LOG: received fast shutdown request Oct 22 17:43:37 hetzner-lab01 patroni[13083]: 2025-10-22 17:43:37.989 CEST [13083] LOG: aborting any active transactions Oct 22 17:43:37 hetzner-lab01 patroni[13095]: 2025-10-22 17:43:37.990 CEST [13095] FATAL: terminating connection due to administrator command Oct 22 17:43:37 hetzner-lab01 systemd[1]: patroni.service: Got notification message from PID 13083, but reception only permitted for main PID 13059 Oct 22 17:43:37 hetzner-lab01 patroni[13083]: 2025-10-22 17:43:37.992 CEST [13083] LOG: background worker "logical replication launcher" (PID 13090) exited with exit code 1 Oct 22 17:43:37 hetzner-lab01 patroni[13085]: 2025-10-22 17:43:37.994 CEST [13085] LOG: shutting down Oct 22 17:43:37 hetzner-lab01 patroni[13085]: 2025-10-22 17:43:37.996 CEST [13085] LOG: checkpoint starting: shutdown immediate Oct 22 17:43:38 hetzner-lab01 patroni[13085]: 2025-10-22 17:43:38.014 CEST [13085] LOG: checkpoint complete: wrote 17 buffers (0.1%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.005 s, sync=0.004 s, total=0.020 s; sync files=13, longest=0.002 s, average=0.001 s; distance=36 kB, estimate=36 kB; lsn=0/152D6F0, redo lsn=0/152D6F0 Oct 22 17:43:38 hetzner-lab01 patroni[13083]: 2025-10-22 17:43:38.020 CEST [13083] LOG: database system is shut down Oct 22 17:43:39 hetzner-lab01 systemd[1]: patroni.service: Failed with result ‘timeout’. Oct 22 17:43:39 hetzner-lab01 systemd[1]: Failed to start patroni.service - Runners to orchestrate a high-availability PostgreSQL. Any recommendations how to fix/workaround this? cheers, Hauke — 🥷 Hauke Bruno Wollentin ^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: Patroni 4.1.0 causes issues because of systemd notify 2025-10-22 16:16 Patroni 4.1.0 causes issues because of systemd notify Hauke Bruno Wollentin <[email protected]> @ 2025-11-06 20:34 ` Richard Huxton <[email protected]> 1 sibling, 0 replies; 4+ messages in thread From: Richard Huxton @ 2025-11-06 20:34 UTC (permalink / raw) To: [email protected] On 2025-10-22 17:16, Hauke Bruno Wollentin wrote: > Hi all, > > while spinning up new Patroni cluster with version 4.1.0 (which has > been added to the Debian repo 2 days ago), my clusters won’t bootstrap > properly anymore. > > At first, logs looks clear but anyways, all systemd operations like > start/stop/restart will run into a timeout, causing the service to stop > which then will break the cluster at all. > > It seems like the new systemd notify feature is causing this, since > using Patroni 4.0.7 with `Type=simple` in the systemd unit file works > pretty fine. I think you may well be right - it seems that patroni is sending the notify from a forked child process and that isn't what systemd is expecting > Oct 22 17:43:09 hetzner-lab01 systemd[1]: patroni.service: Got > notification message from PID 13083, but reception only permitted for > main PID 13059 > Any recommendations how to fix/workaround this? https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#NotifyAccess= I don't have a test patroni installation handy right now, but I think if you set NotifyAccess=all in the service config (which you should be able to do by creating a file with just that line called `/etc/systemd/system/patroni/override.conf`) It defaults to "main" IIRC which is the most restricted, "exec" might work, but "all" (the least restricted) seems the most likely to work first time. Hopefully that will do the trick. If so, maybe try "exec" then and let the list know which works and the package can be updated. -- Richard Huxton ^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: Patroni 4.1.0 causes issues because of systemd notify 2025-10-22 16:16 Patroni 4.1.0 causes issues because of systemd notify Hauke Bruno Wollentin <[email protected]> @ 2025-11-07 04:06 ` Aaron Pavely <[email protected]> 2025-11-07 06:27 ` Re[2]: Patroni 4.1.0 causes issues because of systemd notify Hauke Bruno Wollentin <[email protected]> 1 sibling, 1 reply; 4+ messages in thread From: Aaron Pavely @ 2025-11-07 04:06 UTC (permalink / raw) To: [email protected] On Sun, Nov 2, 2025 at 2:27 PM Hauke Bruno Wollentin <[email protected]> wrote: > Hi all, > > while spinning up new Patroni cluster with version 4.1.0 (which has been > added to the Debian repo 2 days ago), my clusters won’t bootstrap properly > anymore. > > At first, logs looks clear but anyways, all systemd operations like > start/stop/restart will run into a timeout, causing the service to stop > which then will break the cluster at all. > > It seems like the new systemd notify feature is causing this, since using > Patroni 4.0.7 with `Type=simple` in the systemd unit file works pretty fine. > > I’m on Debian 12 and psql 17. > Just a quick question: Has the python3-systemd package been installed as well? There are multiple issues from the upstream Patroni GitHub repository reporting similar problems since the release of 4.1.0, and the corresponding solution is installing that python3-systemd package. It is not (yet) listed as a patroni package dependency, so it may have been missed given the new functionality in 4.1.0. Ref: https://github.com/patroni/patroni/issues/3480 https://github.com/patroni/patroni/issues/3482 -- Aaron Pavely ^ permalink raw reply [nested|flat] 4+ messages in thread
* Re[2]: Patroni 4.1.0 causes issues because of systemd notify 2025-10-22 16:16 Patroni 4.1.0 causes issues because of systemd notify Hauke Bruno Wollentin <[email protected]> 2025-11-07 04:06 ` Re: Patroni 4.1.0 causes issues because of systemd notify Aaron Pavely <[email protected]> @ 2025-11-07 06:27 ` Hauke Bruno Wollentin <[email protected]> 0 siblings, 0 replies; 4+ messages in thread From: Hauke Bruno Wollentin @ 2025-11-07 06:27 UTC (permalink / raw) To: [email protected] --------=_MB003100F6-6C53-4AB7-AEF7-2FFCD3801DBE Content-Type: multipart/mixed; boundary="------=_MB580085C3-364C-4D07-8748-52F7003777BA" --------=_MB580085C3-364C-4D07-8748-52F7003777BA Content-Type: multipart/alternative; boundary="------=_MBC73380EE-0CD4-4B9A-B63F-BDC68EA83E67" --------=_MBC73380EE-0CD4-4B9A-B63F-BDC68EA83E67 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Hi Aaron, I can confirm that installing python3-systemd will resolve this issue,=20 the package doesn't come as a dependency yet. have a nice one, Hauke ------ Original Message ------ ^ permalink raw reply [nested|flat] 4+ messages in thread
end of thread, other threads:[~2025-11-07 06:27 UTC | newest] Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2025-10-22 16:16 Patroni 4.1.0 causes issues because of systemd notify Hauke Bruno Wollentin <[email protected]> 2025-11-06 20:34 ` Richard Huxton <[email protected]> 2025-11-07 04:06 ` Aaron Pavely <[email protected]> 2025-11-07 06:27 ` Re[2]: Patroni 4.1.0 causes issues because of systemd notify Hauke Bruno Wollentin <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox