pg.ddx.io  pgsql-hackers@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Alexander Lakhin <exclusion@gmail.com>
To: Fujii Masao <masao.fujii@gmail.com>
To: Ayush Tiwari <ayushtiwari.slg01@gmail.com>
Cc: Michael Paquier <michael@paquier.xyz>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>
Subject: Re: 041_checkpoint_at_promote.pl might fail due to race condition on child kill
Date: Sat, 14 Nov 2026 16:00:01 +0200
Message-ID: <1c733765-62af-48b3-b0d7-1e744c8b8522@gmail.com> (raw)
In-Reply-To: <CAHGQGwG3NJ0UZO0-u+GFxqO6x7RPu408OO0v0mjMdmYEPSaC-A@mail.gmail.com>
References: <e7abd282-655d-495e-94c9-e38a58917ebf@gmail.com>
	<CAHGQGwFvuGE5cJJfTFtaYY855=3pmuiczAd13JvcpTVWnUVgFg@mail.gmail.com>
	<c011d010-d227-4f27-b229-e9e0f2b668e0@gmail.com>
	<CAHGQGwFtmZAuG+VruhcHWQcmv4=YcQtXLc9jcN3J8JPF_p4o-w@mail.gmail.com>
	<CAJTYsWXkw4OqxNKVrp8dkhJjG8huFyvCjxEvSL7imSFembaP_g@mail.gmail.com>
	<CAHGQGwG3NJ0UZO0-u+GFxqO6x7RPu408OO0v0mjMdmYEPSaC-A@mail.gmail.com>

Hello Fujii-san and Ayush,

30.10.2026 13:55, Fujii Masao wrote:
> On Fri, Oct 30, 2026 at 3:59 PM Ayush Tiwari
> <ayushtiwari.slg01@gmail.com> wrote:
>> Did multiple runs with it, and the changes lgtm.
> I've pushed the patch. Thanks!

Please look at a new failure of 041_checkpoint_at_promote produced by
tamandua recently [1]:
[03:10:10.869](0.025s) ok 4 - psql query died successfully after SIGKILL
[03:13:15.804](184.935s) # die: timed out waiting for file 
/home/bf/bf-build/tamandua/REL_19_STABLE/pgsql.build/testrun/recovery/041_checkpoint_at_promote/log/041_checkpoint_at_promote_standby1.log 
contents to match: (?^:all server processes terminated; reinitializing) at 
/home/bf/bf-build/tamandua/REL_19_STABLE/pgsql/src/test/recovery/t/041_checkpoint_at_promote.pl line 166.
[03:13:15.806](0.003s) 1..4
timed out waiting for file 
/home/bf/bf-build/tamandua/REL_19_STABLE/pgsql.build/testrun/recovery/041_checkpoint_at_promote/log/041_checkpoint_at_promote_standby1.log 
contents to match: (?^:all server processes terminated; reinitializing) at 
/home/bf/bf-build/tamandua/REL_19_STABLE/pgsql/src/test/recovery/t/041_checkpoint_at_promote.pl line 166.
# No postmaster PID for node "master"

041_checkpoint_at_promote_standby1.log contains:
2026-11-12 03:10:10.826 CET [3578531][client backend][6/2:0] LOG: statement: SELECT pg_backend_pid();
2026-11-12 03:10:10.861 CET [3577203][postmaster][:0] LOG:  client backend (PID 3578531) was terminated by signal 9: Killed
2026-11-12 03:10:10.861 CET [3577203][postmaster][:0] DETAIL: Failed process was running: SELECT pg_backend_pid();
2026-11-12 03:10:10.862 CET [3577203][postmaster][:0] LOG: terminating any other active server processes
2026-11-12 03:10:10.928 CET [3577203][postmaster][:0] LOG:  database system is ready to accept connections

There is no "terminating any other active server processes" message
indeed, the server was not shut down.

I managed to reproduce this locally (with the patch attached and with the
same parallel loop as before) and discovered that this can happen when
PMState goes from PM_HOT_STANDBY to PM_RUN (via PM_WAIT_BACKENDS) instead
of PM_HOT_STANDBY -> PM_WAIT_BACKENDS -> PM_WAIT_DEAD_END ->
PM_NO_CHILDREN -> "all server processes terminated; reinitializing".

This can be explained by two race conditions, with one precondition: the
startup process doesn't exit before pg_ctl kill.
1) The startup process becomes zombie for the postmaster if it exits when
waitpid() doesn't wait.
...
2026-11-14 13:29:02.900 UTC [695404][startup][:0] DEBUG: proc_exit(-1): 0 callbacks to make
...
2026-11-14 13:29:03.088 UTC [695367][postmaster][:0] DEBUG:  sending signal 3/SIGQUIT to startup process with pid 695404
2026-11-14 13:29:03.100 UTC [695367][postmaster][:0] LOG: !!!signal_child| delayed signal 3/SIGQUIT to startup process 
with pid 695404
...
2026-11-14 13:29:03.170 UTC [695367][postmaster][:0] DEBUG:  startup process 695404 is still running
...
2026-11-14 13:29:03.198 UTC [695367][postmaster][:0] DEBUG: updating PMState from PM_WAIT_BACKENDS to PM_RUN


2) SIGQUIT sent to the startup process when it's being executed inside
exit(0), after the point when signal handlers of a process are not called
anymore, so SIGQUIT is ignored:
...
2026-11-14 07:54:44.900 EST [1110043][startup][:0] LOG: !!!proc_exit| delayed exit(0) for startup process
...
2026-11-14 07:54:44.900 EST [1110043][startup][:0] DEBUG: proc_exit(-1): 0 callbacks to make
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] LOG: !!!signal_child| delayed signal 3/SIGQUIT to startup process 
with pid 1110043
...
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: startup process 1110043 is still running
...
2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110043, status: 0
2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 89
2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] DEBUG: updating PMState from PM_WAIT_BACKENDS to PM_RUN

Corresponding logs for these 2 cases and also a normal run's log are
attached for reference.

[1] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=tamandua&dt=2026-11-12%2002%3A04%3A37

Best regards,
Alexander

Attachments:

recovery-041-debugging.patch (text/x-patch, 2.2K) download diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c index f53aa9c404d..d1db634644b 100644 --- a/src/backend/postmaster/postmaster.c +++ b/src/backend/postmaster/postmaster.c @@ -68,6 +68,7 @@ #include <unistd.h> #include <signal.h> #include <time.h> +#include <sys/time.h> #include <sys/wait.h> #include <ctype.h> #include <sys/stat.h> @@ -2253,6 +2254,16 @@ handle_pm_child_exit_signal(SIGNAL_ARGS) SetLatch(MyLatch); } +static pid_t +_waitpid(pid_t pid, int *status, int options) +{ +pid_t ret; +elog(LOG, "!!!waitpid| pid: %d", pid); +ret = waitpid(pid, status, options); +elog(LOG, "!!!waitpid| ret: %d, status: %d", ret, *status); +return ret; +} + /* * Cleanup after a child process dies. */ @@ -2267,7 +2278,7 @@ process_pm_child_exit(void) ereport(DEBUG4, (errmsg_internal("reaping dead processes"))); - while ((pid = waitpid(-1, &exitstatus, WNOHANG)) > 0) + while ((pid = _waitpid(-1, &exitstatus, WNOHANG)) > 0) { PMChild *pmchild; @@ -3519,6 +3530,13 @@ signal_child(PMChild *pmchild, int signal) GetBackendTypeDesc(pmchild->bkend_type), (int) pmchild->pid))); +if (pmchild == StartupPMChild && signal == SIGQUIT) +{ +struct timeval current_time; +do gettimeofday(&current_time, NULL); while ((current_time.tv_usec / 1000) % 100 >= 2); +elog(LOG, "!!!signal_child| delayed signal %d/%s to startup process with pid %d", signal, pm_signame(signal), (int) pmchild->pid); +} + if (kill(pid, signal) < 0) elog(DEBUG3, "kill(%ld,%d) failed: %m", (long) pid, signal); #ifdef HAVE_SETSID diff --git a/src/backend/storage/ipc/ipc.c b/src/backend/storage/ipc/ipc.c index cb944edd8df..e96b3c2a035 100644 --- a/src/backend/storage/ipc/ipc.c +++ b/src/backend/storage/ipc/ipc.c @@ -22,6 +22,7 @@ #include <signal.h> #include <unistd.h> #include <sys/stat.h> +#include <sys/time.h> #include "miscadmin.h" #ifdef PROFILE_PID_DIR @@ -154,6 +155,13 @@ proc_exit(int code) elog(DEBUG3, "exit(%d)", code); +if (AmStartupProcess() && code == 0) +{ +struct timeval current_time; +pg_usleep(250000); +do gettimeofday(&current_time, NULL); while ((current_time.tv_usec / 1000) % 100 >= 2); +elog(LOG, "!!!proc_exit| delayed exit(0) for startup process"); +} exit(code); }
041_checkpoint_at_promote_standby1-2.log (text/x-log, 43.4K) download ... 2026-11-14 07:54:44.636 EST [1110043][startup][37/0:0] LOG: received promote request 2026-11-14 07:54:44.636 EST [1110043][startup][37/0:0] LOG: redo done at 0/04000028 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.58 s 2026-11-14 07:54:44.636 EST [1110043][startup][37/0:0] LOG: last completed transaction was at log time 2026-11-14 07:54:44.226403-05 2026-11-14 07:54:44.636 EST [1110043][startup][37/0:0] DEBUG: resetting unlogged relations: cleanup 0 init 1 2026-11-14 07:54:44.637 EST [1110043][startup][37/0:0] LOG: selected new timeline ID: 2 2026-11-14 07:54:44.637 EST [1110043][startup][37/0:0] DEBUG: updated min recovery point to 0/040000B0 on timeline 1 2026-11-14 07:54:44.648 EST [1110043][startup][37/0:0] DEBUG: could not remove file "pg_wal/000000020000000000000004": No such file or directory 2026-11-14 07:54:44.648 EST [1110043][startup][37/0:0] LOG: archive recovery complete 2026-11-14 07:54:44.648 EST [1110043][startup][37/0:0] DEBUG: MultiXactId wrap limit is 2147483648, limited by database with OID 1 2026-11-14 07:54:44.648 EST [1110043][startup][37/0:0] DEBUG: attempting to remove WAL segments newer than log file 000000020000000000000004 2026-11-14 07:54:44.648 EST [1110043][startup][37/0:0] DEBUG: removing all KnownAssignedXids 2026-11-14 07:54:44.648 EST [1110043][startup][37/0:0] DEBUG: release all standby locks 2026-11-14 07:54:44.648 EST [1110043][startup][37/0:0] DEBUG: shmem_exit(0): 5 before_shmem_exit callbacks to make 2026-11-14 07:54:44.648 EST [1110043][startup][37/0:0] DEBUG: shmem_exit(0): 8 on_shmem_exit callbacks to make 2026-11-14 07:54:44.648 EST [1110043][startup][:0] DEBUG: proc_exit(0): 1 callbacks to make 2026-11-14 07:54:44.648 EST [1110043][startup][:0] DEBUG: exit(0) 2026-11-14 07:54:44.738 EST [1110037][postmaster][:0] DEBUG: assigned pm child slot 2 for client backend 2026-11-14 07:54:44.738 EST [1110037][postmaster][:0] DEBUG: forked new client backend, pid=1110667 socket=8 2026-11-14 07:54:44.738 EST [1110667][client backend][:0] LOG: connection received: host=[local] 2026-11-14 07:54:44.738 EST [1110667][client backend][5/0:0] DEBUG: InitPostgres 2026-11-14 07:54:44.739 EST [1110667][client backend][5/1:0] LOG: connection authenticated: ... 2026-11-14 07:54:44.739 EST [1110667][client backend][5/1:0] LOG: connection authorized: ... 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] LOG: statement: SELECT injection_points_wakeup('create-restart-point'); 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: find_in_path: trying ".../tmp_install/usr/local/pgsql/lib/injection_points" 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: find_in_path: trying ".../tmp_install/usr/local/pgsql/lib/injection_points.so" 2026-11-14 07:54:44.741 EST [1110667][client backend][5/0:0] DEBUG: shmem_exit(0): 5 before_shmem_exit callbacks to make 2026-11-14 07:54:44.741 EST [1110667][client backend][5/0:0] DEBUG: shmem_exit(0): 7 on_shmem_exit callbacks to make 2026-11-14 07:54:44.741 EST [1110667][client backend][:0] DEBUG: proc_exit(0): 3 callbacks to make 2026-11-14 07:54:44.741 EST [1110667][client backend][:0] LOG: disconnection: session time: 0:00:00.002 ... 2026-11-14 07:54:44.741 EST [1110667][client backend][:0] DEBUG: exit(0) 2026-11-14 07:54:44.741 EST [1110667][client backend][:0] DEBUG: shmem_exit(-1): 0 before_shmem_exit callbacks to make 2026-11-14 07:54:44.741 EST [1110667][client backend][:0] DEBUG: shmem_exit(-1): 0 on_shmem_exit callbacks to make 2026-11-14 07:54:44.741 EST [1110667][client backend][:0] DEBUG: proc_exit(-1): 0 callbacks to make 2026-11-14 07:54:44.741 EST [1110037][postmaster][:0] DEBUG: reaping dead processes 2026-11-14 07:54:44.741 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.741 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110667, status: 0 2026-11-14 07:54:44.741 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 2 2026-11-14 07:54:44.741 EST [1110037][postmaster][:0] DEBUG: client backend (PID 1110667) exited with exit code 0 2026-11-14 07:54:44.741 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.741 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 0 2026-11-14 07:54:44.749 EST [1110041][checkpointer][:0] DEBUG: attempting to remove WAL segments older than log file 000000000000000000000002 2026-11-14 07:54:44.749 EST [1110041][checkpointer][:0] DEBUG: recycled write-ahead log file "000000010000000000000002" 2026-11-14 07:54:44.749 EST [1110041][checkpointer][:0] DEBUG: SlruScanDirectory invoking callback on pg_subtrans/0000 2026-11-14 07:54:44.749 EST [1110041][checkpointer][:0] LOG: restartpoint complete: fast wait: wrote 38 buffers (29.7%), wrote 2 SLRU buffers; 0 WAL file(s) added, 0 removed, 1 recycled; write=0.001 s, sync=0.001 s, total=0.528 s; sync files=0, longest=0.000 s, average=0.000 s; distance=16384 kB, estimate=16384 kB; lsn=0/03000088, redo lsn=0/03000028 2026-11-14 07:54:44.749 EST [1110041][checkpointer][:0] LOG: recovery restart point at 0/03000028 2026-11-14 07:54:44.749 EST [1110041][checkpointer][:0] DETAIL: Last completed transaction was at log time 2026-11-14 07:54:44.226403-05. 2026-11-14 07:54:44.749 EST [1110041][checkpointer][:0] LOG: checkpoint starting: force 2026-11-14 07:54:44.843 EST [1110037][postmaster][:0] DEBUG: assigned pm child slot 2 for client backend 2026-11-14 07:54:44.843 EST [1110037][postmaster][:0] DEBUG: forked new client backend, pid=1110740 socket=8 2026-11-14 07:54:44.843 EST [1110740][client backend][:0] LOG: connection received: host=[local] 2026-11-14 07:54:44.843 EST [1110740][client backend][6/0:0] DEBUG: InitPostgres 2026-11-14 07:54:44.844 EST [1110740][client backend][6/1:0] LOG: connection authenticated: ... 2026-11-14 07:54:44.844 EST [1110740][client backend][6/1:0] LOG: connection authorized: ... 2026-11-14 07:54:44.844 EST [1110740][client backend][6/2:0] LOG: statement: SELECT pg_backend_pid(); 2026-11-14 07:54:44.844 EST [1110740][client backend][6/2:0] DEBUG: io 6 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8 2026-11-14 07:54:44.844 EST [1110740][client backend][6/2:0] DEBUG: io 6 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8 2026-11-14 07:54:44.844 EST [1110740][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8 2026-11-14 07:54:44.844 EST [1110740][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8 2026-11-14 07:54:44.844 EST [1110740][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8 2026-11-14 07:54:44.844 EST [1110740][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8 2026-11-14 07:54:44.844 EST [1110740][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.844 EST [1110740][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8 2026-11-14 07:54:44.844 EST [1110740][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.844 EST [1110740][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8 2026-11-14 07:54:44.845 EST [1110037][postmaster][:0] DEBUG: reaping dead processes 2026-11-14 07:54:44.845 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.845 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110740, status: 9 2026-11-14 07:54:44.845 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 2 2026-11-14 07:54:44.845 EST [1110037][postmaster][:0] LOG: client backend (PID 1110740) was terminated by signal 9: Killed 2026-11-14 07:54:44.845 EST [1110037][postmaster][:0] DETAIL: Failed process was running: SELECT pg_backend_pid(); 2026-11-14 07:54:44.845 EST [1110037][postmaster][:0] LOG: terminating any other active server processes 2026-11-14 07:54:44.845 EST [1110037][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to client backend process with pid 1110383 2026-11-14 07:54:44.845 EST [1110037][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to startup process with pid 1110043 2026-11-14 07:54:44.849 EST [1110041][checkpointer][:0] DEBUG: performing replication slot checkpoint 2026-11-14 07:54:44.849 EST [1110041][checkpointer][:0] DEBUG: snapshot of 0+0 running transaction ids (lsn 0/04000148 oldest xid 667 latest complete 666 next xid 667) 2026-11-14 07:54:44.849 EST [1110041][checkpointer][:0] DEBUG: attempting to remove WAL segments older than log file 000000000000000000000003 2026-11-14 07:54:44.849 EST [1110041][checkpointer][:0] DEBUG: recycled write-ahead log file "000000010000000000000003" 2026-11-14 07:54:44.849 EST [1110041][checkpointer][:0] DEBUG: SlruScanDirectory invoking callback on pg_subtrans/0000 2026-11-14 07:54:44.849 EST [1110041][checkpointer][:0] LOG: checkpoint complete: force: wrote 1 buffers (0.8%), wrote 2 SLRU buffers; 0 WAL file(s) added, 0 removed, 1 recycled; write=0.101 s, sync=0.001 s, total=0.101 s; sync files=0, longest=0.000 s, average=0.000 s; distance=16384 kB, estimate=16384 kB; lsn=0/04000148, redo lsn=0/040000E8 2026-11-14 07:54:44.900 EST [1110043][startup][:0] LOG: !!!proc_exit| delayed exit(0) for startup process 2026-11-14 07:54:44.900 EST [1110043][startup][:0] DEBUG: shmem_exit(-1): 0 before_shmem_exit callbacks to make 2026-11-14 07:54:44.900 EST [1110043][startup][:0] DEBUG: shmem_exit(-1): 0 on_shmem_exit callbacks to make 2026-11-14 07:54:44.900 EST [1110043][startup][:0] DEBUG: proc_exit(-1): 0 callbacks to make 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] LOG: !!!signal_child| delayed signal 3/SIGQUIT to startup process with pid 1110043 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to background writer process with pid 1110042 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to checkpointer process with pid 1110041 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to io worker process with pid 1110040 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to io worker process with pid 1110039 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: updating PMState from PM_HOT_STANDBY to PM_WAIT_BACKENDS 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110383, status: 512 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 1 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 512 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: startup process 1110043 is still running 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: background writer process 1110042 is still running 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: checkpointer process 1110041 is still running 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: io worker process 1110040 is still running 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: io worker process 1110039 is still running 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: reaping dead processes 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 43690 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: startup process 1110043 is still running 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: background writer process 1110042 is still running 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: checkpointer process 1110041 is still running 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: io worker process 1110040 is still running 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: io worker process 1110039 is still running 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: reaping dead processes 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110040, status: 512 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 58 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 512 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: startup process 1110043 is still running 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: background writer process 1110042 is still running 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: checkpointer process 1110041 is still running 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: io worker process 1110039 is still running 2026-11-14 07:54:44.902 EST [1110037][postmaster][:0] DEBUG: reaping dead processes 2026-11-14 07:54:44.902 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.902 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110039, status: 512 2026-11-14 07:54:44.902 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 57 2026-11-14 07:54:44.902 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.902 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110041, status: 512 2026-11-14 07:54:44.902 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 56 2026-11-14 07:54:44.902 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.902 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110042, status: 512 2026-11-14 07:54:44.902 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 55 2026-11-14 07:54:44.902 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.902 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 512 2026-11-14 07:54:44.902 EST [1110037][postmaster][:0] DEBUG: startup process 1110043 is still running 2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] DEBUG: reaping dead processes 2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110043, status: 0 2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 89 2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] DEBUG: updating PMState from PM_WAIT_BACKENDS to PM_RUN 2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] LOG: database system is ready to accept connections 2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: -1, status: 0 2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] DEBUG: assigned pm child slot 57 for io worker 2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] DEBUG: assigned pm child slot 58 for io worker 2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] DEBUG: assigned pm child slot 56 for checkpointer 2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] DEBUG: assigned pm child slot 55 for background writer 2026-11-14 07:54:44.904 EST [1110037][postmaster][:0] DEBUG: assigned pm child slot 92 for walwriter 2026-11-14 07:54:44.904 EST [1110037][postmaster][:0] DEBUG: assigned pm child slot 45 for background worker 2026-11-14 07:54:44.904 EST [1110037][postmaster][:0] DEBUG: starting background worker process "logical replication launcher" 2026-11-14 07:54:44.904 EST [1110788][checkpointer][:0] DEBUG: checkpointer updated shared memory configuration values 2026-11-14 07:54:44.904 EST [1110793][logical replication launcher][15/0:0] DEBUG: logical replication launcher started 2026-11-14 07:54:44.904 EST [1110793][logical replication launcher][15/0:0] DEBUG: InitPostgres 2026-11-14 07:55:00.050 EST [1110790][background writer][:0] DEBUG: snapshot of 0+0 running transaction ids (lsn 0/04000208 oldest xid 667 latest complete 666 next xid 667) 2026-11-14 07:55:44.967 EST [1110786][io worker][:0] DEBUG: shmem_exit(0): 4 before_shmem_exit callbacks to make 2026-11-14 07:55:44.967 EST [1110786][io worker][:0] DEBUG: shmem_exit(0): 7 on_shmem_exit callbacks to make 2026-11-14 07:55:44.967 EST [1110786][io worker][:0] DEBUG: proc_exit(0): 1 callbacks to make 2026-11-14 07:55:44.967 EST [1110786][io worker][:0] DEBUG: exit(0) 2026-11-14 07:55:44.967 EST [1110786][io worker][:0] DEBUG: shmem_exit(-1): 0 before_shmem_exit callbacks to make 2026-11-14 07:55:44.967 EST [1110786][io worker][:0] DEBUG: shmem_exit(-1): 0 on_shmem_exit callbacks to make 2026-11-14 07:55:44.967 EST [1110786][io worker][:0] DEBUG: proc_exit(-1): 0 callbacks to make 2026-11-14 07:55:44.968 EST [1110037][postmaster][:0] DEBUG: reaping dead processes 2026-11-14 07:55:44.968 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:55:44.968 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110786, status: 0 2026-11-14 07:55:44.968 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 58 2026-11-14 07:55:44.968 EST [1110037][postmaster][:0] DEBUG: assigned pm child slot 58 for io worker 2026-11-14 07:55:44.969 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:55:44.969 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 0 2026-11-14 07:56:45.031 EST [1111045][io worker][:0] DEBUG: shmem_exit(0): 4 before_shmem_exit callbacks to make 2026-11-14 07:56:45.032 EST [1111045][io worker][:0] DEBUG: shmem_exit(0): 7 on_shmem_exit callbacks to make 2026-11-14 07:56:45.032 EST [1111045][io worker][:0] DEBUG: proc_exit(0): 1 callbacks to make 2026-11-14 07:56:45.032 EST [1111045][io worker][:0] DEBUG: exit(0) 2026-11-14 07:56:45.032 EST [1111045][io worker][:0] DEBUG: shmem_exit(-1): 0 before_shmem_exit callbacks to make 2026-11-14 07:56:45.032 EST [1111045][io worker][:0] DEBUG: shmem_exit(-1): 0 on_shmem_exit callbacks to make 2026-11-14 07:56:45.032 EST [1111045][io worker][:0] DEBUG: proc_exit(-1): 0 callbacks to make 2026-11-14 07:56:45.032 EST [1110037][postmaster][:0] DEBUG: reaping dead processes 2026-11-14 07:56:45.032 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:56:45.033 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1111045, status: 0 2026-11-14 07:56:45.033 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 58 2026-11-14 07:56:45.033 EST [1110037][postmaster][:0] DEBUG: assigned pm child slot 58 for io worker 2026-11-14 07:56:45.033 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:56:45.033 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 0 2026-11-14 07:57:45.098 EST [1111065][io worker][:0] DEBUG: shmem_exit(0): 4 before_shmem_exit callbacks to make 2026-11-14 07:57:45.098 EST [1111065][io worker][:0] DEBUG: shmem_exit(0): 7 on_shmem_exit callbacks to make 2026-11-14 07:57:45.098 EST [1111065][io worker][:0] DEBUG: proc_exit(0): 1 callbacks to make 2026-11-14 07:57:45.098 EST [1111065][io worker][:0] DEBUG: exit(0) 2026-11-14 07:57:45.098 EST [1111065][io worker][:0] DEBUG: shmem_exit(-1): 0 before_shmem_exit callbacks to make 2026-11-14 07:57:45.098 EST [1111065][io worker][:0] DEBUG: shmem_exit(-1): 0 on_shmem_exit callbacks to make 2026-11-14 07:57:45.098 EST [1111065][io worker][:0] DEBUG: proc_exit(-1): 0 callbacks to make 2026-11-14 07:57:45.099 EST [1110037][postmaster][:0] DEBUG: reaping dead processes 2026-11-14 07:57:45.099 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:57:45.099 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1111065, status: 0 2026-11-14 07:57:45.099 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 58 2026-11-14 07:57:45.099 EST [1110037][postmaster][:0] DEBUG: assigned pm child slot 58 for io worker 2026-11-14 07:57:45.099 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:57:45.099 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 0 2026-11-14 07:57:49.231 EST [1110037][postmaster][:0] DEBUG: postmaster received shutdown request signal 2026-11-14 07:57:49.231 EST [1110037][postmaster][:0] LOG: received immediate shutdown request 2026-11-14 07:57:49.231 EST [1110037][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to io worker process with pid 1111082 2026-11-14 07:57:49.231 EST [1110037][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to background worker process with pid 1110793 2026-11-14 07:57:49.231 EST [1110037][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to walwriter process with pid 1110791 2026-11-14 07:57:49.231 EST [1110037][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to background writer process with pid 1110790 2026-11-14 07:57:49.231 EST [1110037][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to checkpointer process with pid 1110788 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to io worker process with pid 1110784 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: updating PMState from PM_RUN to PM_WAIT_BACKENDS 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: io worker process 1111082 is still running 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: background worker process 1110793 is still running 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: walwriter process 1110791 is still running 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: background writer process 1110790 is still running 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: checkpointer process 1110788 is still running 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: io worker process 1110784 is still running 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: reaping dead processes 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110790, status: 512 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 55 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110788, status: 512 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 56 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110791, status: 512 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 92 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110784, status: 512 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 57 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110793, status: 512 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 45 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:57:49.233 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1111082, status: 512 2026-11-14 07:57:49.233 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 58 2026-11-14 07:57:49.233 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:57:49.233 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: -1, status: 512 2026-11-14 07:57:49.233 EST [1110037][postmaster][:0] DEBUG: updating PMState from PM_WAIT_BACKENDS to PM_WAIT_DEAD_END 2026-11-14 07:57:49.233 EST [1110037][postmaster][:0] DEBUG: updating PMState from PM_WAIT_DEAD_END to PM_NO_CHILDREN 2026-11-14 07:57:49.233 EST [1110037][postmaster][:0] DEBUG: shmem_exit(0): 0 before_shmem_exit callbacks to make 2026-11-14 07:57:49.233 EST [1110037][postmaster][:0] DEBUG: shmem_exit(0): 5 on_shmem_exit callbacks to make 2026-11-14 07:57:49.233 EST [1110037][postmaster][:0] DEBUG: cleaning up orphaned dynamic shared memory with ID 1459897042 2026-11-14 07:57:49.233 EST [1110037][postmaster][:0] DEBUG: cleaning up orphaned dynamic shared memory with ID 3069910578 2026-11-14 07:57:49.233 EST [1110037][postmaster][:0] DEBUG: cleaning up orphaned dynamic shared memory with ID 3859456016 2026-11-14 07:57:49.233 EST [1110037][postmaster][:0] DEBUG: cleaning up dynamic shared memory control segment with ID 2869141000 2026-11-14 07:57:49.234 EST [1110037][postmaster][:0] DEBUG: proc_exit(0): 2 callbacks to make 2026-11-14 07:57:49.234 EST [1110037][postmaster][:0] LOG: database system is shut down 2026-11-14 07:57:49.234 EST [1110037][postmaster][:0] DEBUG: exit(0) 2026-11-14 07:57:49.234 EST [1110037][postmaster][:0] DEBUG: shmem_exit(-1): 0 before_shmem_exit callbacks to make 2026-11-14 07:57:49.234 EST [1110037][postmaster][:0] DEBUG: shmem_exit(-1): 0 on_shmem_exit callbacks to make 2026-11-14 07:57:49.234 EST [1110037][postmaster][:0] DEBUG: proc_exit(-1): 0 callbacks to make
041_checkpoint_at_promote_standby1-1.log (text/x-log, 71.0K) download
041_checkpoint_at_promote_standby1-0.log (text/x-log, 44.4K) download ... 2026-11-14 07:54:44.154 EST [1109760][startup][37/0:0] LOG: received promote request 2026-11-14 07:54:44.154 EST [1109760][startup][37/0:0] LOG: redo done at 0/04000028 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.33 s 2026-11-14 07:54:44.154 EST [1109760][startup][37/0:0] LOG: last completed transaction was at log time 2026-11-14 07:54:44.016107-05 2026-11-14 07:54:44.154 EST [1109760][startup][37/0:0] DEBUG: resetting unlogged relations: cleanup 0 init 1 2026-11-14 07:54:44.154 EST [1109760][startup][37/0:0] LOG: selected new timeline ID: 2 2026-11-14 07:54:44.155 EST [1109760][startup][37/0:0] DEBUG: updated min recovery point to 0/040000B0 on timeline 1 2026-11-14 07:54:44.172 EST [1109760][startup][37/0:0] DEBUG: could not remove file "pg_wal/000000020000000000000004": No such file or directory 2026-11-14 07:54:44.172 EST [1109760][startup][37/0:0] LOG: archive recovery complete 2026-11-14 07:54:44.172 EST [1109760][startup][37/0:0] DEBUG: MultiXactId wrap limit is 2147483648, limited by database with OID 1 2026-11-14 07:54:44.172 EST [1109760][startup][37/0:0] DEBUG: attempting to remove WAL segments newer than log file 000000020000000000000004 2026-11-14 07:54:44.172 EST [1109760][startup][37/0:0] DEBUG: removing all KnownAssignedXids 2026-11-14 07:54:44.172 EST [1109760][startup][37/0:0] DEBUG: release all standby locks 2026-11-14 07:54:44.172 EST [1109760][startup][37/0:0] DEBUG: shmem_exit(0): 5 before_shmem_exit callbacks to make 2026-11-14 07:54:44.172 EST [1109760][startup][37/0:0] DEBUG: shmem_exit(0): 8 on_shmem_exit callbacks to make 2026-11-14 07:54:44.172 EST [1109760][startup][:0] DEBUG: proc_exit(0): 1 callbacks to make 2026-11-14 07:54:44.172 EST [1109760][startup][:0] DEBUG: exit(0) 2026-11-14 07:54:44.256 EST [1109746][postmaster][:0] DEBUG: assigned pm child slot 2 for client backend 2026-11-14 07:54:44.256 EST [1109746][postmaster][:0] DEBUG: forked new client backend, pid=1110418 socket=8 2026-11-14 07:54:44.256 EST [1110418][client backend][:0] LOG: connection received: host=[local] 2026-11-14 07:54:44.257 EST [1110418][client backend][5/0:0] DEBUG: InitPostgres 2026-11-14 07:54:44.257 EST [1110418][client backend][5/1:0] LOG: connection authenticated: ... 2026-11-14 07:54:44.257 EST [1110418][client backend][5/1:0] LOG: connection authorized: ... 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] LOG: statement: SELECT injection_points_wakeup('create-restart-point'); 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: find_in_path: trying ".../tmp_install/usr/local/pgsql/lib/injection_points" 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: find_in_path: trying ".../tmp_install/usr/local/pgsql/lib/injection_points.so" 2026-11-14 07:54:44.259 EST [1110418][client backend][5/0:0] DEBUG: shmem_exit(0): 5 before_shmem_exit callbacks to make 2026-11-14 07:54:44.259 EST [1110418][client backend][5/0:0] DEBUG: shmem_exit(0): 7 on_shmem_exit callbacks to make 2026-11-14 07:54:44.259 EST [1110418][client backend][:0] DEBUG: proc_exit(0): 3 callbacks to make 2026-11-14 07:54:44.259 EST [1110418][client backend][:0] LOG: disconnection: session time: 0:00:00.002 ... 2026-11-14 07:54:44.259 EST [1110418][client backend][:0] DEBUG: exit(0) 2026-11-14 07:54:44.259 EST [1110418][client backend][:0] DEBUG: shmem_exit(-1): 0 before_shmem_exit callbacks to make 2026-11-14 07:54:44.259 EST [1110418][client backend][:0] DEBUG: shmem_exit(-1): 0 on_shmem_exit callbacks to make 2026-11-14 07:54:44.259 EST [1110418][client backend][:0] DEBUG: proc_exit(-1): 0 callbacks to make 2026-11-14 07:54:44.260 EST [1109746][postmaster][:0] DEBUG: reaping dead processes 2026-11-14 07:54:44.260 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.260 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 1110418, status: 0 2026-11-14 07:54:44.260 EST [1109746][postmaster][:0] DEBUG: releasing pm child slot 2 2026-11-14 07:54:44.260 EST [1109746][postmaster][:0] DEBUG: client backend (PID 1110418) exited with exit code 0 2026-11-14 07:54:44.260 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.260 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 0 2026-11-14 07:54:44.273 EST [1109758][checkpointer][:0] DEBUG: attempting to remove WAL segments older than log file 000000000000000000000002 2026-11-14 07:54:44.273 EST [1109758][checkpointer][:0] DEBUG: recycled write-ahead log file "000000010000000000000002" 2026-11-14 07:54:44.273 EST [1109758][checkpointer][:0] DEBUG: SlruScanDirectory invoking callback on pg_subtrans/0000 2026-11-14 07:54:44.273 EST [1109758][checkpointer][:0] LOG: restartpoint complete: fast wait: wrote 16 buffers (12.5%), wrote 2 SLRU buffers; 0 WAL file(s) added, 0 removed, 1 recycled; write=0.001 s, sync=0.001 s, total=0.261 s; sync files=0, longest=0.000 s, average=0.000 s; distance=16384 kB, estimate=16384 kB; lsn=0/03000088, redo lsn=0/03000028 2026-11-14 07:54:44.273 EST [1109758][checkpointer][:0] LOG: recovery restart point at 0/03000028 2026-11-14 07:54:44.273 EST [1109758][checkpointer][:0] DETAIL: Last completed transaction was at log time 2026-11-14 07:54:44.016107-05. 2026-11-14 07:54:44.273 EST [1109758][checkpointer][:0] LOG: checkpoint starting: force 2026-11-14 07:54:44.361 EST [1109746][postmaster][:0] DEBUG: assigned pm child slot 2 for client backend 2026-11-14 07:54:44.361 EST [1109746][postmaster][:0] DEBUG: forked new client backend, pid=1110482 socket=8 2026-11-14 07:54:44.361 EST [1110482][client backend][:0] LOG: connection received: host=[local] 2026-11-14 07:54:44.362 EST [1110482][client backend][6/0:0] DEBUG: InitPostgres 2026-11-14 07:54:44.362 EST [1110482][client backend][6/1:0] LOG: connection authenticated: ... 2026-11-14 07:54:44.362 EST [1110482][client backend][6/1:0] LOG: connection authorized: ... 2026-11-14 07:54:44.362 EST [1110482][client backend][6/2:0] LOG: statement: SELECT pg_backend_pid(); 2026-11-14 07:54:44.363 EST [1110482][client backend][6/2:0] DEBUG: io 6 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8 2026-11-14 07:54:44.363 EST [1110482][client backend][6/2:0] DEBUG: io 6 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8 2026-11-14 07:54:44.363 EST [1110482][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8 2026-11-14 07:54:44.363 EST [1110482][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8 2026-11-14 07:54:44.363 EST [1110482][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8 2026-11-14 07:54:44.363 EST [1110482][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8 2026-11-14 07:54:44.363 EST [1110482][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.363 EST [1110482][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8 2026-11-14 07:54:44.363 EST [1110482][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.363 EST [1110482][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8 2026-11-14 07:54:44.364 EST [1109746][postmaster][:0] DEBUG: reaping dead processes 2026-11-14 07:54:44.364 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.364 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 1110482, status: 9 2026-11-14 07:54:44.364 EST [1109746][postmaster][:0] DEBUG: releasing pm child slot 2 2026-11-14 07:54:44.364 EST [1109746][postmaster][:0] LOG: client backend (PID 1110482) was terminated by signal 9: Killed 2026-11-14 07:54:44.364 EST [1109746][postmaster][:0] DETAIL: Failed process was running: SELECT pg_backend_pid(); 2026-11-14 07:54:44.364 EST [1109746][postmaster][:0] LOG: terminating any other active server processes 2026-11-14 07:54:44.364 EST [1109746][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to client backend process with pid 1109914 2026-11-14 07:54:44.364 EST [1109746][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to startup process with pid 1109760 2026-11-14 07:54:44.375 EST [1109758][checkpointer][:0] DEBUG: performing replication slot checkpoint 2026-11-14 07:54:44.375 EST [1109758][checkpointer][:0] DEBUG: snapshot of 0+0 running transaction ids (lsn 0/04000148 oldest xid 667 latest complete 666 next xid 667) 2026-11-14 07:54:44.375 EST [1109758][checkpointer][:0] DEBUG: attempting to remove WAL segments older than log file 000000000000000000000003 2026-11-14 07:54:44.375 EST [1109758][checkpointer][:0] DEBUG: recycled write-ahead log file "000000010000000000000003" 2026-11-14 07:54:44.375 EST [1109758][checkpointer][:0] DEBUG: SlruScanDirectory invoking callback on pg_subtrans/0000 2026-11-14 07:54:44.375 EST [1109758][checkpointer][:0] LOG: checkpoint complete: force: wrote 1 buffers (0.8%), wrote 2 SLRU buffers; 0 WAL file(s) added, 0 removed, 1 recycled; write=0.101 s, sync=0.001 s, total=0.103 s; sync files=0, longest=0.000 s, average=0.000 s; distance=16384 kB, estimate=16384 kB; lsn=0/04000148, redo lsn=0/040000E8 2026-11-14 07:54:44.400 EST [1109746][postmaster][:0] LOG: !!!signal_child| delayed signal 3/SIGQUIT to startup process with pid 1109760 2026-11-14 07:54:44.500 EST [1109760][startup][:0] LOG: !!!proc_exit| delayed exit(0) for startup process 2026-11-14 07:54:44.539 EST [1109746][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to background writer process with pid 1109759 2026-11-14 07:54:44.539 EST [1109746][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to checkpointer process with pid 1109758 2026-11-14 07:54:44.539 EST [1109746][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to io worker process with pid 1109757 2026-11-14 07:54:44.539 EST [1109746][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to io worker process with pid 1109756 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: updating PMState from PM_HOT_STANDBY to PM_WAIT_BACKENDS 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 1109756, status: 512 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: releasing pm child slot 57 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 1109757, status: 512 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: releasing pm child slot 58 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 1109758, status: 512 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: releasing pm child slot 56 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 1109759, status: 512 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: releasing pm child slot 55 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 1109760, status: 512 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: releasing pm child slot 89 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 1109914, status: 512 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: releasing pm child slot 1 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: -1, status: 512 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: updating PMState from PM_WAIT_BACKENDS to PM_WAIT_DEAD_END 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: updating PMState from PM_WAIT_DEAD_END to PM_NO_CHILDREN 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: all server processes terminated; reinitializing 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: shmem_exit(1): 0 before_shmem_exit callbacks to make 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: shmem_exit(1): 5 on_shmem_exit callbacks to make 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: cleaning up orphaned dynamic shared memory with ID 924198510 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: cleaning up orphaned dynamic shared memory with ID 1616401164 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: cleaning up orphaned dynamic shared memory with ID 3295781356 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: cleaning up dynamic shared memory control segment with ID 2272285588 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: invoking IpcMemoryCreate(size=5873664) 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: mmap(6291456) with MAP_HUGETLB failed, huge pages disabled: Cannot allocate memory 2026-11-14 07:54:44.541 EST [1109746][postmaster][:0] DEBUG: dynamic shared memory system will support 229 segments 2026-11-14 07:54:44.541 EST [1109746][postmaster][:0] DEBUG: created dynamic shared memory control segment 4225595752 (9176 bytes) 2026-11-14 07:54:44.541 EST [1109746][postmaster][:0] DEBUG: updating PMState from PM_NO_CHILDREN to PM_STARTUP 2026-11-14 07:54:44.541 EST [1109746][postmaster][:0] DEBUG: assigned pm child slot 58 for io worker 2026-11-14 07:54:44.541 EST [1109746][postmaster][:0] DEBUG: assigned pm child slot 57 for io worker 2026-11-14 07:54:44.541 EST [1109746][postmaster][:0] DEBUG: assigned pm child slot 89 for startup 2026-11-14 07:54:44.541 EST [1109746][postmaster][:0] DEBUG: assigned pm child slot 56 for checkpointer 2026-11-14 07:54:44.541 EST [1109746][postmaster][:0] DEBUG: assigned pm child slot 55 for background writer 2026-11-14 07:54:44.542 EST [1109746][postmaster][:0] DEBUG: reaping dead processes 2026-11-14 07:54:44.542 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.542 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 43690 2026-11-14 07:54:44.542 EST [1110541][startup][:0] LOG: database system was interrupted; last known up at 2026-11-14 07:54:44 EST 2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: removing all temporary WAL segments 2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: checkpoint record is at 0/04000148 2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: redo record is at 0/040000E8; shutdown false 2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: next transaction ID: 667; next OID: 16398 2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: next MultiXactId: 1; next MultiXactOffset: 1 2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: oldest unfrozen transaction ID: 656, in database 1 2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: oldest MultiXactId: 1, in database 1 2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: commit timestamp Xid oldest/newest: 0/0 2026-11-14 07:54:44.542 EST [1110541][startup][:0] LOG: database system was not properly shut down; automatic recovery in progress 2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: transaction ID wrap limit is 2147484303, limited by database with OID 1 2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: MultiXactId wrap limit is 2147483648, limited by database with OID 1 2026-11-14 07:54:44.542 EST [1110542][checkpointer][:0] DEBUG: checkpointer updated shared memory configuration values 2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: starting up replication slots 2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: xmin required by slots: data 0, catalog 0 2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: starting up replication origin progress state 2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: didn't need to unlink permanent stats file "pg_stat/pgstat.stat" - didn't exist 2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: resetting unlogged relations: cleanup 1 init 0 2026-11-14 07:54:44.542 EST [1110541][startup][:0] LOG: redo starts at 0/040000E8 2026-11-14 07:54:44.542 EST [1110541][startup][:0] LOG: invalid record length at 0/040001D0: expected at least 24, got 0 2026-11-14 07:54:44.542 EST [1110541][startup][:0] LOG: redo done at 0/04000148 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s 2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: resetting unlogged relations: cleanup 0 init 1 2026-11-14 07:54:44.542 EST [1109746][postmaster][:0] DEBUG: postmaster received pmsignal signal 2026-11-14 07:54:44.542 EST [1109746][postmaster][:0] DEBUG: updating PMState from PM_STARTUP to PM_RECOVERY 2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: MultiXactId wrap limit is 2147483648, limited by database with OID 1 2026-11-14 07:54:44.543 EST [1110542][checkpointer][:0] LOG: checkpoint starting: end-of-recovery fast wait 2026-11-14 07:54:44.543 EST [1110542][checkpointer][:0] DEBUG: performing replication slot checkpoint 2026-11-14 07:54:44.543 EST [1110542][checkpointer][:0] DEBUG: attempting to remove WAL segments older than log file 000000000000000000000003 2026-11-14 07:54:44.544 EST [1110542][checkpointer][:0] LOG: checkpoint complete: end-of-recovery fast wait: wrote 0 buffers (0.0%), wrote 3 SLRU buffers; 0 WAL file(s) added, 0 removed, 0 recycled; write=0.001 s, sync=0.001 s, total=0.001 s; sync files=0, longest=0.000 s, average=0.000 s; distance=0 kB, estimate=0 kB; lsn=0/040001D0, redo lsn=0/040001D0 2026-11-14 07:54:44.544 EST [1110541][startup][:0] DEBUG: shmem_exit(0): 5 before_shmem_exit callbacks to make 2026-11-14 07:54:44.544 EST [1110541][startup][:0] DEBUG: shmem_exit(0): 7 on_shmem_exit callbacks to make 2026-11-14 07:54:44.544 EST [1110541][startup][:0] DEBUG: proc_exit(0): 1 callbacks to make 2026-11-14 07:54:44.544 EST [1110541][startup][:0] DEBUG: exit(0) 2026-11-14 07:54:44.569 EST [1109746][postmaster][:0] DEBUG: allocating dead-end child 2026-11-14 07:54:44.569 EST [1109746][postmaster][:0] DEBUG: forked new dead-end client backend, pid=1110598 socket=8 2026-11-14 07:54:44.569 EST [1110598][dead-end client backend][:0] LOG: connection received: host=[local] 2026-11-14 07:54:44.569 EST [1110598][dead-end client backend][:0] FATAL: the database system is not yet accepting connections 2026-11-14 07:54:44.569 EST [1110598][dead-end client backend][:0] DETAIL: Consistent recovery state has not been yet reached. 2026-11-14 07:54:44.569 EST [1110598][dead-end client backend][:0] DEBUG: shmem_exit(1): 0 before_shmem_exit callbacks to make 2026-11-14 07:54:44.569 EST [1110598][dead-end client backend][:0] DEBUG: shmem_exit(1): 0 on_shmem_exit callbacks to make 2026-11-14 07:54:44.569 EST [1110598][dead-end client backend][:0] DEBUG: proc_exit(1): 1 callbacks to make 2026-11-14 07:54:44.569 EST [1110598][dead-end client backend][:0] DEBUG: exit(1) 2026-11-14 07:54:44.569 EST [1110598][dead-end client backend][:0] DEBUG: shmem_exit(-1): 0 before_shmem_exit callbacks to make 2026-11-14 07:54:44.569 EST [1110598][dead-end client backend][:0] DEBUG: shmem_exit(-1): 0 on_shmem_exit callbacks to make 2026-11-14 07:54:44.569 EST [1110598][dead-end client backend][:0] DEBUG: proc_exit(-1): 0 callbacks to make 2026-11-14 07:54:44.570 EST [1109746][postmaster][:0] DEBUG: reaping dead processes 2026-11-14 07:54:44.570 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.570 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 1110598, status: 256 2026-11-14 07:54:44.570 EST [1109746][postmaster][:0] DEBUG: releasing dead-end backend 2026-11-14 07:54:44.570 EST [1109746][postmaster][:0] DEBUG: dead-end client backend (PID 1110598) exited with exit code 1 2026-11-14 07:54:44.570 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.570 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 256 2026-11-14 07:54:44.672 EST [1109746][postmaster][:0] DEBUG: allocating dead-end child 2026-11-14 07:54:44.672 EST [1109746][postmaster][:0] DEBUG: forked new dead-end client backend, pid=1110640 socket=8 2026-11-14 07:54:44.673 EST [1110640][dead-end client backend][:0] LOG: connection received: host=[local] 2026-11-14 07:54:44.673 EST [1110640][dead-end client backend][:0] FATAL: the database system is not yet accepting connections 2026-11-14 07:54:44.673 EST [1110640][dead-end client backend][:0] DETAIL: Consistent recovery state has not been yet reached. 2026-11-14 07:54:44.673 EST [1110640][dead-end client backend][:0] DEBUG: shmem_exit(1): 0 before_shmem_exit callbacks to make 2026-11-14 07:54:44.673 EST [1110640][dead-end client backend][:0] DEBUG: shmem_exit(1): 0 on_shmem_exit callbacks to make 2026-11-14 07:54:44.673 EST [1110640][dead-end client backend][:0] DEBUG: proc_exit(1): 1 callbacks to make 2026-11-14 07:54:44.673 EST [1110640][dead-end client backend][:0] DEBUG: exit(1) 2026-11-14 07:54:44.673 EST [1110640][dead-end client backend][:0] DEBUG: shmem_exit(-1): 0 before_shmem_exit callbacks to make 2026-11-14 07:54:44.673 EST [1110640][dead-end client backend][:0] DEBUG: shmem_exit(-1): 0 on_shmem_exit callbacks to make 2026-11-14 07:54:44.673 EST [1110640][dead-end client backend][:0] DEBUG: proc_exit(-1): 0 callbacks to make 2026-11-14 07:54:44.673 EST [1109746][postmaster][:0] DEBUG: reaping dead processes 2026-11-14 07:54:44.673 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.673 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 1110640, status: 256 2026-11-14 07:54:44.673 EST [1109746][postmaster][:0] DEBUG: releasing dead-end backend 2026-11-14 07:54:44.673 EST [1109746][postmaster][:0] DEBUG: dead-end client backend (PID 1110640) exited with exit code 1 2026-11-14 07:54:44.673 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.673 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 256 2026-11-14 07:54:44.775 EST [1109746][postmaster][:0] DEBUG: allocating dead-end child 2026-11-14 07:54:44.775 EST [1109746][postmaster][:0] DEBUG: forked new dead-end client backend, pid=1110689 socket=8 2026-11-14 07:54:44.775 EST [1110689][dead-end client backend][:0] LOG: connection received: host=[local] 2026-11-14 07:54:44.775 EST [1110689][dead-end client backend][:0] FATAL: the database system is not yet accepting connections 2026-11-14 07:54:44.775 EST [1110689][dead-end client backend][:0] DETAIL: Consistent recovery state has not been yet reached. 2026-11-14 07:54:44.775 EST [1110689][dead-end client backend][:0] DEBUG: shmem_exit(1): 0 before_shmem_exit callbacks to make 2026-11-14 07:54:44.775 EST [1110689][dead-end client backend][:0] DEBUG: shmem_exit(1): 0 on_shmem_exit callbacks to make 2026-11-14 07:54:44.775 EST [1110689][dead-end client backend][:0] DEBUG: proc_exit(1): 1 callbacks to make 2026-11-14 07:54:44.775 EST [1110689][dead-end client backend][:0] DEBUG: exit(1) 2026-11-14 07:54:44.775 EST [1110689][dead-end client backend][:0] DEBUG: shmem_exit(-1): 0 before_shmem_exit callbacks to make 2026-11-14 07:54:44.775 EST [1110689][dead-end client backend][:0] DEBUG: shmem_exit(-1): 0 on_shmem_exit callbacks to make 2026-11-14 07:54:44.775 EST [1110689][dead-end client backend][:0] DEBUG: proc_exit(-1): 0 callbacks to make 2026-11-14 07:54:44.776 EST [1109746][postmaster][:0] DEBUG: reaping dead processes 2026-11-14 07:54:44.776 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.776 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 1110689, status: 256 2026-11-14 07:54:44.776 EST [1109746][postmaster][:0] DEBUG: releasing dead-end backend 2026-11-14 07:54:44.776 EST [1109746][postmaster][:0] DEBUG: dead-end client backend (PID 1110689) exited with exit code 1 2026-11-14 07:54:44.776 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.776 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 256 2026-11-14 07:54:44.800 EST [1110541][startup][:0] LOG: !!!proc_exit| delayed exit(0) for startup process 2026-11-14 07:54:44.800 EST [1110541][startup][:0] DEBUG: shmem_exit(-1): 0 before_shmem_exit callbacks to make 2026-11-14 07:54:44.800 EST [1110541][startup][:0] DEBUG: shmem_exit(-1): 0 on_shmem_exit callbacks to make 2026-11-14 07:54:44.800 EST [1110541][startup][:0] DEBUG: proc_exit(-1): 0 callbacks to make 2026-11-14 07:54:44.800 EST [1109746][postmaster][:0] DEBUG: reaping dead processes 2026-11-14 07:54:44.800 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.800 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 1110541, status: 0 2026-11-14 07:54:44.800 EST [1109746][postmaster][:0] DEBUG: releasing pm child slot 89 2026-11-14 07:54:44.800 EST [1109746][postmaster][:0] DEBUG: updating PMState from PM_RECOVERY to PM_RUN 2026-11-14 07:54:44.800 EST [1109746][postmaster][:0] LOG: database system is ready to accept connections 2026-11-14 07:54:44.800 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.800 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 0 2026-11-14 07:54:44.800 EST [1109746][postmaster][:0] DEBUG: assigned pm child slot 92 for walwriter 2026-11-14 07:54:44.801 EST [1109746][postmaster][:0] DEBUG: assigned pm child slot 45 for background worker 2026-11-14 07:54:44.801 EST [1109746][postmaster][:0] DEBUG: starting background worker process "logical replication launcher" 2026-11-14 07:54:44.801 EST [1110704][logical replication launcher][15/0:0] DEBUG: logical replication launcher started 2026-11-14 07:54:44.801 EST [1110704][logical replication launcher][15/0:0] DEBUG: InitPostgres 2026-11-14 07:54:44.884 EST [1109746][postmaster][:0] DEBUG: assigned pm child slot 1 for client backend 2026-11-14 07:54:44.884 EST [1109746][postmaster][:0] DEBUG: forked new client backend, pid=1110770 socket=8 2026-11-14 07:54:44.884 EST [1110770][client backend][:0] LOG: connection received: host=[local] 2026-11-14 07:54:44.884 EST [1110770][client backend][0/0:0] DEBUG: InitPostgres ...

view thread (12+ messages)  latest in thread

Message-ID: <1c733765-62af-48b3-b0d7-1e744c8b8522@gmail.com>
Permalink:  ../1c733765-62af-48b3-b0d7-1e744c8b8522@gmail.com/
Also on:    postgresql.org/message-id/1c733765-62af-48b3-b0d7-1e744c8b8522@gmail.com

 ·  · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-hackers@postgresql.org
  Cc: exclusion@gmail.com, masao.fujii@gmail.com, ayushtiwari.slg01@gmail.com, michael@paquier.xyz
  Subject: Re: 041_checkpoint_at_promote.pl might fail due to race condition on child kill
  In-Reply-To: <1c733765-62af-48b3-b0d7-1e744c8b8522@gmail.com>
  References: <e7abd282-655d-495e-94c9-e38a58917ebf@gmail.com> <CAHGQGwFvuGE5cJJfTFtaYY855=3pmuiczAd13JvcpTVWnUVgFg@mail.gmail.com> <c011d010-d227-4f27-b229-e9e0f2b668e0@gmail.com> <CAHGQGwFtmZAuG+VruhcHWQcmv4=YcQtXLc9jcN3J8JPF_p4o-w@mail.gmail.com> <CAJTYsWXkw4OqxNKVrp8dkhJjG8huFyvCjxEvSL7imSFembaP_g@mail.gmail.com> <CAHGQGwG3NJ0UZO0-u+GFxqO6x7RPu408OO0v0mjMdmYEPSaC-A@mail.gmail.com> <1c733765-62af-48b3-b0d7-1e744c8b8522@gmail.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox