pg.ddx.io  pgsql-bugs@postgresql.org mailing list archive  
help / color / mirror / Atom feed
BUG #19908: pg_upgrade --check crashes instance if running in a different PID Namespace then the old cluster
7+ messages / 4 participants
[nested] [flat]

* BUG #19908: pg_upgrade --check crashes instance if running in a different PID Namespace then the old cluster
@ 2026-11-16 14:08  PG Bug reporting form <noreply@postgresql.org>
  0 siblings, 1 reply; 7+ messages in thread

From: PG Bug reporting form @ 2026-11-16 14:08 UTC (permalink / raw)
  To: pgsql-bugs@lists.postgresql.org; +Cc: schefner20@gmail.com

The following bug has been logged on the website:

Bug reference:      19908
Logged by:          Nick Schefner
Email address:      schefner20@gmail.com
PostgreSQL version: 18.6
Operating system:   RHEL Rocky Linux 9.8
Description:        

We have the following setup:

- 2 Postgres Servers | One running as primary the other running as standby
- Our Instances are managed though podman quadlets with the following
settings:

postgresql.container
[Unit]
  Description=Podman postgresql.container
  Documentation=man:podman-generate-systemd(1)
  Wants=network-online.target
  After=network-online.target
  RequiresMountsFor=/tmp/podman-run-26/containers
  StartLimitIntervalSec=400
  StartLimitBurst=3

[Container]
  Image=<host>/<images>:14.24-0-tsdb2.11.0-0-gis3.1.12-0
  ContainerName=postgresql
  Exec=/usr/pgsql-14/bin/postgres -D /postgres/<instance>/
  PublishPort=0.0.0.0:****:****
  Volume=/etc/ssl/postgres:/etc/ssl/postgres:ro
  Volume=/var/lib/pgsql/for_podman/:/var/lib/pgsql/from_outside/:ro
  Volume=/postgres/<instance>:/postgres/<instance>
  Volume=/etc/repmgr/rdm_tsm:/etc/repmgr/rdm_tsm:ro
  Environment=PGDATA=/postgres/<instance>
  Environment=PGPORT=****
  Environment=PGPASSFILE=/var/lib/pgsql/from_outside/.pgpass
  ShmSize=20g
  User=26:26
  UserNS=keep-id
  PodmanArgs=--add-host host.containers.internal:host-gateway
[Service]
Environment=REGISTRY_AUTH_FILE=/var/lib/pgsql/.config/containers/.auth.json
  Restart=on-failure
  RestartSec=10
  ExecReload=/usr/bin/podman exec postgresql /bin/kill -HUP 1
[Install]
  WantedBy=default.target

The data directories lie on an NFS share.

Now to the bug:

I wanted to upgrade the version 14 Instance to the 18.6 version (i sadly
just realized that you released 18.7 a few days ago).
Before the upgrade I wanted to test 'pg_upgrade --check ...' on the 14
Instance while it was still running, since --check signals to me that
nothing should break running it.
None the less the command broke my instance. Im going to provide the
commands i did now with some errors.

1. I created a one-time podman container for upgrading the instance
          podman run -it --rm --name pg-upgrade --userns=keep-id --user root
\
          -v /postgres:/postgres -v /etc/ssl/postgres:/etc/ssl/postgres:ro \
git.ufz.de:4567/it/databases/postgres/images:14.24-0-tsdb2.11.0-0-gis3.1.12-0
bash
2. I installed the new versions into the one-time podman container
          dnf install -y postgresql18-server postgresql18-contrib
postgis36_18-3.6.4 repmgr_18
3. I initialized the version 18 Instance into the new directory. (still in
the container)
          su postgres -c "/usr/pgsql-18/bin/initdb -D /postgres/rdm_tsm/18
--data-checksums --encoding=UTF-8 --locale=de_DE.utf8 --wal-segsize=512"
4. I revoked the old barman functions since they got renamed after Version
14 (on the 14 Instance)
        REVOKE ALL ON FUNCTION pg_start_backup(text, boolean, boolean) FROM
barman;
        REVOKE ALL ON FUNCTION pg_stop_backup() FROM barman;
        REVOKE ALL ON FUNCTION pg_stop_backup(boolean, boolean) FROM barman;
5. I ran pg_upgrade --check inside the one-time container
        cd /postgres/rdm_tsm
        su postgres -c "/usr/pgsql-18/bin/pg_upgrade -b /usr/pgsql-14/bin -B
/usr/pgsql-18/bin -d /postgres/rdm_tsm -D /postgres/rdm_tsm/18 -p 50432 -P
50433 --check"

Here it failed with the following error:

  Performing Consistency Checks on Old Live Server
  ------------------------------------------------
  Checking cluster versions                                     ok

  could not open file "/postgres/rdm_tsm/postmaster.pid": No such file or
directory
  Failure, exiting

Here might be some more errors helping you understand what happen:

/postgres/rdm_tsm/18/pg_upgrade_output.d/20261116T073424.501/log/pg_upgrade_server.log:
  pg_upgrade run on Mon Nov 16 07:34:24 2026
  command: "/usr/pgsql-14/bin/pg_ctl" -w -l ".../pg_upgrade_server.log" -D
"/postgres/rdm_tsm" -o "-p 50432 -b  -c listen_addresses='' -c
unix_socket_permissions=0700" start
  pg_ctl: another server might be running; trying to start server anyway
  waiting for server to start....2026-11-16 07:34:24 CET [253] @ INFO:
timescaledb loaded
  2026-11-16 07:34:25 CET [253] @ LOG:  redirecting log output to logging
collector process
  . stopped waiting
  pg_ctl: could not start server

live server log:
  07:34:50 [1] @ LOG:  performing immediate shutdown because data directory
lock file is invalid
  07:34:50 ... FATAL:  the database system is shutting down   (pgbouncer,
monitoring, barman, repmgr)

And this after restart attempts:
  PANIC: could not find redo location ...   (WAL already deleted by "rm %p"
in the archive_command)
  07:48:45 [3] @ FATAL:  requested timeline 10 does not contain minimum
recovery point 48AB/7B318D48 on timeline 0
  07:48:45 [1] @ LOG:  aborting startup due to startup process failure
(The "rm %p" is probably wrong and resulted into the recovery failure)

Apperently the upgrade command couldn't correctly retrieve the
postmaster.pid file or thought it was stale and tried to start the server
anyway.
As a side effect it completely wiped the postmaster.pid and created a new
one breaking the instance.

Possible Explanation:
 - pg_upgrade read the postmaster.pid and checked if the PID exists
 - it then couldn't find a process with that PID because the container,
where pg_upgrade was running on, has its own PID namespace
 - since it can't find the PID, pg_upgrade startet a second instance

Suggestion:
  I've tried to figure out if this might be fixed in 18.7 but looking
through the commits of pg_upgrade.c I couldn't find one that solved the
problem.
  A Solution could be to just not try to start the instance when a
postmaster.pid was found e.g:
       In 'pg_upgrade.c' :

  static void check_clusters_stopped(void) {
       check_pghost_envvar();
       if (pid_lock_file_exists(old_cluster.pgdata))
       {
              if (!user_opts.check)
                  pg_fatal("A postmaster.pid file exists in the old cluster
data directory \"%s\".\n"
                                "Shut down the postmaster servicing the old
cluster and try again.\n"
                                "If the file is stale, start and cleanly
stop the old server, or remove the file\n"
                                "once you are certain no server (in any
container or on any host) uses this directory.",
                                old_cluster.pgdata);
              user_opts.live_check = true;
       }

        if (pid_lock_file_exists(new_cluster.pgdata))
              pg_fatal("A postmaster.pid file exists in the new cluster data
directory \"%s\".\n"
                            "Shut down the postmaster servicing the new
cluster and try again.\n"
                            "If the file is stale, remove it once you are
certain no server uses this directory.",
                            new_cluster.pgdata);
  }
I rewrote the function, rebuild via meson and ran the tests and all came
back as passed.
The drawback would be that stale postmaster.pid files cant be ignored.
So a different solution could be preferable.

Documentation:
The Docs of pg_upgrade don't mention this behaviour. It states that
pg_upgrade starts a short lived postmaster on both instances but doesnt
state the same for --check (where I expect that nothing can possibly break).
With this information I could have known that i need to provide the current
clusters --pid and --ipc variables when starting the one-time upgrade
container.
So I would maybe suggest a note in the documentation, describing this issue.

Thank you for you work and sorry if I wrote to much
Feel free to contact me if you need further configs etc.








^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* Re: BUG #19908: pg_upgrade --check crashes instance if running in a different PID Namespace then the old cluster
@ 2026-11-16 20:16  Robert Haas <robertmhaas@gmail.com>
  parent: PG Bug reporting form <noreply@postgresql.org>
  0 siblings, 1 reply; 7+ messages in thread

From: Robert Haas @ 2026-11-16 20:16 UTC (permalink / raw)
  To: schefner20@gmail.com; pgsql-bugs@lists.postgresql.org

On Mon, Nov 16, 2026 at 10:55 AM PG Bug reporting form
<noreply@postgresql.org> wrote:
> Before the upgrade I wanted to test 'pg_upgrade --check ...' on the 14
> Instance while it was still running, since --check signals to me that
> nothing should break running it.
> None the less the command broke my instance. Im going to provide the
> commands i did now with some errors.
>
> Possible Explanation:
>  - pg_upgrade read the postmaster.pid and checked if the PID exists
>  - it then couldn't find a process with that PID because the container,
> where pg_upgrade was running on, has its own PID namespace
>  - since it can't find the PID, pg_upgrade startet a second instance

This is unfortunate, but I'm not sure what to do about it. Because the
servers are running in different PID namespaces, and apparently also
different IPC namespaces, the usual interlocks don't work, so when
pg_upgrade tries to start a new postmaster, bad things can happen. The
obvious answer is to never try to start a new server in --check mode,
but then --check wouldn't be able to ignore a stale postmaster.pid
file any more. I'm not sure how important it is for us to be able to
ignore stale postmaster.pid files is here, but my guess is that some
people would complain if we didn't.

-- 
Robert Haas
Databricks






^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* Re: BUG #19908: pg_upgrade --check crashes instance if running in a different PID Namespace then the old cluster
@ 2026-11-16 20:40  Tom Lane <tgl@sss.pgh.pa.us>
  parent: Robert Haas <robertmhaas@gmail.com>
  0 siblings, 1 reply; 7+ messages in thread

From: Tom Lane @ 2026-11-16 20:40 UTC (permalink / raw)
  To: Robert Haas <robertmhaas@gmail.com>; +Cc: schefner20@gmail.com; pgsql-bugs@lists.postgresql.org

Robert Haas <robertmhaas@gmail.com> writes:
> On Mon, Nov 16, 2026 at 10:55 AM PG Bug reporting form
> <noreply@postgresql.org> wrote:
>> Possible Explanation:
>> - pg_upgrade read the postmaster.pid and checked if the PID exists
>> - it then couldn't find a process with that PID because the container,
>> where pg_upgrade was running on, has its own PID namespace
>> - since it can't find the PID, pg_upgrade startet a second instance

> This is unfortunate, but I'm not sure what to do about it. Because the
> servers are running in different PID namespaces, and apparently also
> different IPC namespaces, the usual interlocks don't work, so when
> pg_upgrade tries to start a new postmaster, bad things can happen. The
> obvious answer is to never try to start a new server in --check mode,
> but then --check wouldn't be able to ignore a stale postmaster.pid
> file any more. I'm not sure how important it is for us to be able to
> ignore stale postmaster.pid files is here, but my guess is that some
> people would complain if we didn't.

Yeah, I think skipping this step would be a cure substantially more
dangerous than the disease.  Note the comment in
check_clusters_stopped about ensuring that WAL replay is complete.

Moreover, I think making it safe or easy to run PG in this environment
is a bad idea.  If you've got namespacing or other stuff that can
inhibit the ability of one would-be postmaster to see another one,
you are playing with fire.  You need to not run things that way, or
someday you'll find your data eaten.

			regards, tom lane






^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* Re: BUG #19908: pg_upgrade --check crashes instance if running in a different PID Namespace then the old cluster
@ 2026-11-16 21:36  Robert Haas <robertmhaas@gmail.com>
  parent: Tom Lane <tgl@sss.pgh.pa.us>
  0 siblings, 2 replies; 7+ messages in thread

From: Robert Haas @ 2026-11-16 21:36 UTC (permalink / raw)
  To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: schefner20@gmail.com; pgsql-bugs@lists.postgresql.org

On Mon, Nov 16, 2026 at 3:40 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Moreover, I think making it safe or easy to run PG in this environment
> is a bad idea.  If you've got namespacing or other stuff that can
> inhibit the ability of one would-be postmaster to see another one,
> you are playing with fire.  You need to not run things that way, or
> someday you'll find your data eaten.

Maybe, but such setups are increasingly common. I'm not sure that we
can just tell people "don't do that"; I think everyone already is.

I also feel like it's counterintuitive that pg_upgrade --check can
destroy data, regardless of anything else. One could certainly fail to
read the documentation and fail to realize that risk.

None of this is to say that I have a good idea, because I don't.

-- 
Robert Haas
Databricks






^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* Re: BUG #19908: pg_upgrade --check crashes instance if running in a different PID Namespace then the old cluster
@ 2026-11-16 21:51  Tom Lane <tgl@sss.pgh.pa.us>
  parent: Robert Haas <robertmhaas@gmail.com>
  1 sibling, 0 replies; 7+ messages in thread

From: Tom Lane @ 2026-11-16 21:51 UTC (permalink / raw)
  To: Robert Haas <robertmhaas@gmail.com>; +Cc: schefner20@gmail.com; pgsql-bugs@lists.postgresql.org

Robert Haas <robertmhaas@gmail.com> writes:
> I also feel like it's counterintuitive that pg_upgrade --check can
> destroy data, regardless of anything else. One could certainly fail to
> read the documentation and fail to realize that risk.

Perhaps it'd be reasonable for --check mode to not risk a
start_postmaster() action, but just bail out if it sees a
postmaster.pid file.  However, that doesn't do diddly-squat
for what I consider the real hazard here: the described
environment breaks our safeguards against starting two postmasters
in the same data directory.  That WILL eat your data someday,
and there's nothing we can do to prevent it.

			regards, tom lane






^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* Re: BUG #19908: pg_upgrade --check crashes instance if running in a different PID Namespace then the old cluster
@ 2026-11-19 12:38  Daniel Gustafsson <daniel@yesql.se>
  parent: Robert Haas <robertmhaas@gmail.com>
  1 sibling, 1 reply; 7+ messages in thread

From: Daniel Gustafsson @ 2026-11-19 12:38 UTC (permalink / raw)
  To: Robert Haas <robertmhaas@gmail.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; schefner20@gmail.com; pgsql-bugs@lists.postgresql.org

> On 16 Nov 2026, at 22:36, Robert Haas <robertmhaas@gmail.com> wrote:

> I also feel like it's counterintuitive that pg_upgrade --check can
> destroy data, regardless of anything else. One could certainly fail to
> read the documentation and fail to realize that risk.
> 
> None of this is to say that I have a good idea, because I don't.

I wasn't able to come up with a good idea either, but I propose that we at
least add a note to the documentation.  How about something like the attached
to at least make it visible until we have a solution?

--
Daniel Gustafsson

Attachments:

pg_upgrade_processpace.diff (application/octet-stream, 901B) download diff --git a/doc/src/sgml/ref/pgupgrade.sgml b/doc/src/sgml/ref/pgupgrade.sgml index e4e8c02e6d6..34a2a7ef129 100644 --- a/doc/src/sgml/ref/pgupgrade.sgml +++ b/doc/src/sgml/ref/pgupgrade.sgml @@ -604,6 +604,16 @@ pg_upgrade.exe <command>pg_upgrade</command> requires write permission in the current directory. </para> + <note> + <para> + In order for <application>pg_upgrade</application> to safely be able + to start and stop the clusters they must reside in a shared process + namespace. If either cluster is isolated in a container it must share + process namespace with <application>pg_upgrade</application> to avoid + the risk of data corruption. + </para> + </note> + <para> Obviously, no one should be accessing the clusters during the upgrade. <application>pg_upgrade</application> defaults to running servers
^ permalink raw reply [nested|flat] 7+ messages in thread

* Re: BUG #19908: pg_upgrade --check crashes instance if running in a different PID Namespace then the old cluster
@ 2026-11-19 17:55  Robert Haas <robertmhaas@gmail.com>
  parent: Daniel Gustafsson <daniel@yesql.se>
  0 siblings, 0 replies; 7+ messages in thread

From: Robert Haas @ 2026-11-19 17:55 UTC (permalink / raw)
  To: Daniel Gustafsson <daniel@yesql.se>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; schefner20@gmail.com; pgsql-bugs@lists.postgresql.org

On Thu, Nov 19, 2026 at 7:38 AM Daniel Gustafsson <daniel@yesql.se> wrote:
> I wasn't able to come up with a good idea either, but I propose that we at
> least add a note to the documentation.  How about something like the attached
> to at least make it visible until we have a solution?

I'm generally in favor of adding a note, but I wouldn't put it here or
say this. One thing I think we should do is change the doc string for
--check, which currently says "check clusters only, don't change any
data," but could just as well say "check clusters, but don't perform
the upgrade". But at the point where you added this, I'd suggest
writing something more like this: Note that, even when --check is
used, pg_upgrade will start and stop the old cluster, unless it is
already running. It is, therefore, not a strictly read-only operation,
and must only be performed in a context where the old server is
accessible, and from which starting and stopping it is safe.

I'd rather not make this specifically about containers.

-- 
Robert Haas
Databricks






^ permalink  raw  reply  [nested|flat] 7+ messages in thread


end of thread, other threads:[~2026-11-19 17:55 UTC | newest]

Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-11-16 14:08 BUG #19908: pg_upgrade --check crashes instance if running in a different PID Namespace then the old cluster PG Bug reporting form <noreply@postgresql.org>
2026-11-16 20:16 ` Robert Haas <robertmhaas@gmail.com>
2026-11-16 20:40   ` Tom Lane <tgl@sss.pgh.pa.us>
2026-11-16 21:36     ` Robert Haas <robertmhaas@gmail.com>
2026-11-16 21:51       ` Tom Lane <tgl@sss.pgh.pa.us>
2026-11-19 12:38       ` Daniel Gustafsson <daniel@yesql.se>
2026-11-19 17:55         ` Robert Haas <robertmhaas@gmail.com>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox