Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1xSz3G-000000000qj-2ZT5 for pgsql-bugs@arkaria.postgresql.org; Mon, 16 Nov 2026 15:55:27 +0000 Received: from localhost ([127.0.0.1]:56276 helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.98.2) (envelope-from ) id 1xSz3F-000000003Bw-3yWZ for pgsql-bugs@arkaria.postgresql.org; Mon, 16 Nov 2026 15:55:25 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]:60344) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1xSxON-00000001NIQ-2OXh for pgsql-bugs@lists.postgresql.org; Mon, 16 Nov 2026 14:09:07 +0000 Received: from mahout.postgresql.org ([2001:4800:3e1:1::227]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1xSxOL-00000001EO9-47OK for pgsql-bugs@lists.postgresql.org; Mon, 16 Nov 2026 14:09:06 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=postgresql.org; s=20171124; h=Message-ID:Date:Reply-To:Cc:From:To:Subject: Content-Transfer-Encoding:MIME-Version:Content-Type:Sender:Content-ID: Content-Description:In-Reply-To:References; bh=cs7+TLrYe4qq84WAArfSfrq+7RpJ7JzXLXdLzSijEJ8=; b=4Z8Tv5/ar2wFxqWaIAkKd94np7 Y7l66VezI6jkEXF20gACWNgNrViQkU8sjYzAvrSl6/l8q7JV5QXR8qKC250QEvqi5Ual7l6I3jW20 plzNBb9aIJArNzvG+Y+64pxiJ2bGXBZ8YrUv/dkRSB2eVyPY/4Arrdnc9envQ1QBRGZShqa0gJQXC 79rTli3MWnvJT++cjrTBdXBEeUNaCgj4RkB2EPS5ijsZO9Gaf0S1v6mqVAj3lJzTjnlYA28jV3LJz IbJiSCJDh0dGXb3XZWq44wA3zefdZCnlbnNhZK/C8lCDqsIsazQla9ihPVGg+EH6b8WYAuxpStWm1 SGmhx2VA==; Received: from wrigleys.postgresql.org ([2a02:16a8:dc51::60]) by mahout.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1xSxOK-00000003T1x-44dD for pgsql-bugs@lists.postgresql.org; Mon, 16 Nov 2026 14:09:05 +0000 Received: from localhost ([127.0.0.1] helo=wrigleys.postgresql.org) by wrigleys.postgresql.org with esmtp (Exim 4.98.2) (envelope-from ) id 1xSxOJ-00000008fSS-3O9i for pgsql-bugs@lists.postgresql.org; Mon, 16 Nov 2026 14:09:03 +0000 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Subject: BUG #19908: pg_upgrade --check crashes instance if running in a different PID Namespace then the old cluster To: pgsql-bugs@lists.postgresql.org From: PG Bug reporting form Cc: schefner20@gmail.com Reply-To: schefner20@gmail.com, pgsql-bugs@lists.postgresql.org Date: Mon, 16 Nov 2026 14:08:50 +0000 Message-ID: <19908-3db6f3c4f2febdcf@postgresql.org> X-Auto-Response-Suppress: All Auto-Submitted: auto-generated List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk The following bug has been logged on the website: Bug reference: 19908 Logged by: Nick Schefner Email address: schefner20@gmail.com PostgreSQL version: 18.6 Operating system: RHEL Rocky Linux 9.8 Description: =20 We have the following setup: - 2 Postgres Servers | One running as primary the other running as standby - Our Instances are managed though podman quadlets with the following settings: postgresql.container [Unit] Description=3DPodman postgresql.container Documentation=3Dman:podman-generate-systemd(1) Wants=3Dnetwork-online.target After=3Dnetwork-online.target RequiresMountsFor=3D/tmp/podman-run-26/containers StartLimitIntervalSec=3D400 StartLimitBurst=3D3 [Container] Image=3D/:14.24-0-tsdb2.11.0-0-gis3.1.12-0 ContainerName=3Dpostgresql Exec=3D/usr/pgsql-14/bin/postgres -D /postgres// PublishPort=3D0.0.0.0:****:**** Volume=3D/etc/ssl/postgres:/etc/ssl/postgres:ro Volume=3D/var/lib/pgsql/for_podman/:/var/lib/pgsql/from_outside/:ro Volume=3D/postgres/:/postgres/ Volume=3D/etc/repmgr/rdm_tsm:/etc/repmgr/rdm_tsm:ro Environment=3DPGDATA=3D/postgres/ Environment=3DPGPORT=3D**** Environment=3DPGPASSFILE=3D/var/lib/pgsql/from_outside/.pgpass ShmSize=3D20g User=3D26:26 UserNS=3Dkeep-id PodmanArgs=3D--add-host host.containers.internal:host-gateway [Service] Environment=3DREGISTRY_AUTH_FILE=3D/var/lib/pgsql/.config/containers/.auth.= json Restart=3Don-failure RestartSec=3D10 ExecReload=3D/usr/bin/podman exec postgresql /bin/kill -HUP 1 [Install] WantedBy=3Ddefault.target The data directories lie on an NFS share. Now to the bug: I wanted to upgrade the version 14 Instance to the 18.6 version (i sadly just realized that you released 18.7 a few days ago). Before the upgrade I wanted to test 'pg_upgrade --check ...' on the 14 Instance while it was still running, since --check signals to me that nothing should break running it. None the less the command broke my instance. Im going to provide the commands i did now with some errors. 1. I created a one-time podman container for upgrading the instance podman run -it --rm --name pg-upgrade --userns=3Dkeep-id --user r= oot \ -v /postgres:/postgres -v /etc/ssl/postgres:/etc/ssl/postgres:ro \ git.ufz.de:4567/it/databases/postgres/images:14.24-0-tsdb2.11.0-0-gis3.1.12= -0 bash 2. I installed the new versions into the one-time podman container dnf install -y postgresql18-server postgresql18-contrib postgis36_18-3.6.4 repmgr_18 3. I initialized the version 18 Instance into the new directory. (still in the container) su postgres -c "/usr/pgsql-18/bin/initdb -D /postgres/rdm_tsm/18 --data-checksums --encoding=3DUTF-8 --locale=3Dde_DE.utf8 --wal-segsize=3D5= 12" 4. I revoked the old barman functions since they got renamed after Version 14 (on the 14 Instance) REVOKE ALL ON FUNCTION pg_start_backup(text, boolean, boolean) FROM barman; REVOKE ALL ON FUNCTION pg_stop_backup() FROM barman; REVOKE ALL ON FUNCTION pg_stop_backup(boolean, boolean) FROM barman; 5. I ran pg_upgrade --check inside the one-time container cd /postgres/rdm_tsm su postgres -c "/usr/pgsql-18/bin/pg_upgrade -b /usr/pgsql-14/bin -B /usr/pgsql-18/bin -d /postgres/rdm_tsm -D /postgres/rdm_tsm/18 -p 50432 -P 50433 --check" Here it failed with the following error: Performing Consistency Checks on Old Live Server ------------------------------------------------ Checking cluster versions ok could not open file "/postgres/rdm_tsm/postmaster.pid": No such file or directory Failure, exiting Here might be some more errors helping you understand what happen: /postgres/rdm_tsm/18/pg_upgrade_output.d/20261116T073424.501/log/pg_upgrade= _server.log: pg_upgrade run on Mon Nov 16 07:34:24 2026 command: "/usr/pgsql-14/bin/pg_ctl" -w -l ".../pg_upgrade_server.log" -D "/postgres/rdm_tsm" -o "-p 50432 -b -c listen_addresses=3D'' -c unix_socket_permissions=3D0700" start pg_ctl: another server might be running; trying to start server anyway waiting for server to start....2026-11-16 07:34:24 CET [253] @ INFO: timescaledb loaded 2026-11-16 07:34:25 CET [253] @ LOG: redirecting log output to logging collector process . stopped waiting pg_ctl: could not start server live server log: 07:34:50 [1] @ LOG: performing immediate shutdown because data directory lock file is invalid 07:34:50 ... FATAL: the database system is shutting down (pgbouncer, monitoring, barman, repmgr) And this after restart attempts: PANIC: could not find redo location ... (WAL already deleted by "rm %p" in the archive_command) 07:48:45 [3] @ FATAL: requested timeline 10 does not contain minimum recovery point 48AB/7B318D48 on timeline 0 07:48:45 [1] @ LOG: aborting startup due to startup process failure (The "rm %p" is probably wrong and resulted into the recovery failure) Apperently the upgrade command couldn't correctly retrieve the postmaster.pid file or thought it was stale and tried to start the server anyway. As a side effect it completely wiped the postmaster.pid and created a new one breaking the instance. Possible Explanation: - pg_upgrade read the postmaster.pid and checked if the PID exists - it then couldn't find a process with that PID because the container, where pg_upgrade was running on, has its own PID namespace - since it can't find the PID, pg_upgrade startet a second instance Suggestion: I've tried to figure out if this might be fixed in 18.7 but looking through the commits of pg_upgrade.c I couldn't find one that solved the problem. A Solution could be to just not try to start the instance when a postmaster.pid was found e.g: In 'pg_upgrade.c' : static void check_clusters_stopped(void) { check_pghost_envvar(); if (pid_lock_file_exists(old_cluster.pgdata)) { if (!user_opts.check) pg_fatal("A postmaster.pid file exists in the old cluster data directory \"%s\".\n" "Shut down the postmaster servicing the old cluster and try again.\n" "If the file is stale, start and cleanly stop the old server, or remove the file\n" "once you are certain no server (in any container or on any host) uses this directory.", old_cluster.pgdata); user_opts.live_check =3D true; } if (pid_lock_file_exists(new_cluster.pgdata)) pg_fatal("A postmaster.pid file exists in the new cluster data directory \"%s\".\n" "Shut down the postmaster servicing the new cluster and try again.\n" "If the file is stale, remove it once you are certain no server uses this directory.", new_cluster.pgdata); } I rewrote the function, rebuild via meson and ran the tests and all came back as passed. The drawback would be that stale postmaster.pid files cant be ignored. So a different solution could be preferable. Documentation: The Docs of pg_upgrade don't mention this behaviour. It states that pg_upgrade starts a short lived postmaster on both instances but doesnt state the same for --check (where I expect that nothing can possibly break). With this information I could have known that i need to provide the current clusters --pid and --ipc variables when starting the one-time upgrade container. So I would maybe suggest a note in the documentation, describing this issue. Thank you for you work and sorry if I wrote to much Feel free to contact me if you need further configs etc.