pg.ddx.io  pgsql-hackers@postgresql.org mailing list archive  
help / color / mirror / Atom feed
041_checkpoint_at_promote.pl might fail due to race condition on child kill
12+ messages / 3 participants
[nested] [flat]

Commits from this thread:
  2026-11-18 83cefb7b73 63df88f3ab 504bd2e8da 2bce773e2e c9c98c5f8c e21e8164db  Fix handling of backend crashes during standby promotion
  2026-10-30 eda9d98fac 6621f2b062 7ba657cba1 abab508718  Fix crash-restart race in 041_checkpoint_at_promote.pl
* 041_checkpoint_at_promote.pl might fail due to race condition on child kill
@ 2026-10-07 04:00  Alexander Lakhin <exclusion@gmail.com>
  0 siblings, 1 reply; 12+ messages in thread

From: Alexander Lakhin @ 2026-10-07 04:00 UTC (permalink / raw)
  To: Michael Paquier <michael@paquier.xyz>; pgsql-hackers

Hello hackers,

Please look at a new failure of 041_checkpoint_at_promote.pl produced by
bushmaster, [1]:
[23:56:13.841](0.002s) ok 4 - psql query died successfully after SIGKILL
[23:56:13.902](0.061s) not ok 5 - psql connect success
[23:56:13.902](0.000s)
[23:56:13.902](0.000s) #   Failed test 'psql connect success'
#   at t/041_checkpoint_at_promote.pl line 168.
[23:56:13.902](0.000s) #          got: '2'
#     expected: '0'
[23:56:13.902](0.000s) not ok 6 - psql select 1
[23:56:13.902](0.000s)
[23:56:13.902](0.000s) #   Failed test 'psql select 1'
#   at t/041_checkpoint_at_promote.pl line 169.
[23:56:13.902](0.000s) #          got: ''
#     expected: '1'
[23:56:13.902](0.000s) 1..6

pgsql.build/src/test/recovery/tmp_check/log/041_checkpoint_at_promote_standby1.log
2026-10-03 23:56:13.852 CEST [4159149][postmaster][:0] LOG:  client backend (PID 4159398) was terminated by signal 9: Killed
2026-10-03 23:56:13.852 CEST [4159149][postmaster][:0] DETAIL: Failed process was running: SELECT pg_backend_pid();
2026-10-03 23:56:13.852 CEST [4159149][postmaster][:0] LOG: terminating any other active server processes
2026-10-03 23:56:13.855 CEST [4159149][postmaster][:0] LOG:  all server processes terminated; reinitializing
2026-10-03 23:56:13.899 CEST [4159414][startup][:0] LOG:  database system was interrupted; last known up at 2026-10-03 
23:56:13 CEST
2026-10-03 23:56:13.899 CEST [4159414][startup][:0] LOG:  database system was not properly shut down; automatic recovery 
in progress
2026-10-03 23:56:13.899 CEST [4159414][startup][:0] LOG:  crash recovery starts in timeline 1 and has target timeline 2
2026-10-03 23:56:13.900 CEST [4159414][startup][:0] LOG:  redo starts at 0/3000028
2026-10-03 23:56:13.900 CEST [4159417][not initialized][:0] LOG: connection received: host=[local]
2026-10-03 23:56:13.900 CEST [4159417][client backend][:0] FATAL: the database system is in recovery mode

I've reproduced this failure locally, running multiple tests concurrently inside a 4-core VM, in a loop:
for i in {1..50}; do echo "Iteration $i"; parallel -j40 --linebuffer --tag PROVE_TESTS="t/041*" NO_TEMP_INSTALL=1 
TEMP_CONFIG=~/extra.config make -s check -s -C src/test/recovery_{} PROVE_FLAGS="--timer" ::: `seq 25` || break; done;
(with bushmaster's parameters put in ~/extra.config)

This failed for me on iterations 14, 30, 10:
...
Iteration 10
...
10
10      #   Failed test 'psql connect success'
10      #   at t/041_checkpoint_at_promote.pl line 168.
10      #          got: '2'
10      #     expected: '0'
10
10      #   Failed test 'psql select 1'
10      #   at t/041_checkpoint_at_promote.pl line 169.
10      #          got: ''
10      #     expected: '1'
10      # Looks like you failed 2 tests of 6.
10      [03:51:18] t/041_checkpoint_at_promote.pl ..
10      Dubious, test returned 2 (wstat 512, 0x200)
10      Failed 2/6 subtests
10      [03:51:18]
10
10      Test Summary Report
10      -------------------
10      t/041_checkpoint_at_promote.pl (Wstat: 512 (exited 2) Tests: 6 Failed: 2)
10        Failed tests:  5-6
10        Non-zero exit status: 2

It looks like the same race condition was fixed in 022_crash_temp_files.pl
by e37ad5fa4 two years before 041_checkpoint_at_promote.pl appeared: [2].

[1] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=bushmaster&dt=2026-10-03%2021%3A37%3A08
[2] https://postgr.es/m/1801850.1649047827@sss.pgh.pa.us

Best regards,
Alexander






^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* Re: 041_checkpoint_at_promote.pl might fail due to race condition on child kill
@ 2026-10-07 16:46  Fujii Masao <masao.fujii@gmail.com>
  parent: Alexander Lakhin <exclusion@gmail.com>
  0 siblings, 1 reply; 12+ messages in thread

From: Fujii Masao @ 2026-10-07 16:46 UTC (permalink / raw)
  To: Alexander Lakhin <exclusion@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; pgsql-hackers

On Wed, Oct 7, 2026 at 1:00 PM Alexander Lakhin <exclusion@gmail.com> wrote:
> It looks like the same race condition was fixed in 022_crash_temp_files.pl
> by e37ad5fa4 two years before 041_checkpoint_at_promote.pl appeared: [2].

I tried creating the patch based on that fix. How about the attached patch?

Regards,

-- 
Fujii Masao

Attachments:

v1-0001-Fix-crash-restart-race-in-041_checkpoint_at_promo.patch (application/octet-stream, 2.6K) download From 9f6392513a307226ca75c12bf173bcf552bc6aa6 Mon Sep 17 00:00:00 2001 From: Fujii Masao <fujii@postgresql.org> Date: Wed, 7 Oct 2026 21:31:47 +0900 Subject: [PATCH v1] Fix crash restart race in 041_checkpoint_at_promote.pl --- .../recovery/t/041_checkpoint_at_promote.pl | 44 +++++++++++++++++-- 1 file changed, 41 insertions(+), 3 deletions(-) diff --git a/src/test/recovery/t/041_checkpoint_at_promote.pl b/src/test/recovery/t/041_checkpoint_at_promote.pl index d0783fef9ae..8c9e65502dc 100644 --- a/src/test/recovery/t/041_checkpoint_at_promote.pl +++ b/src/test/recovery/t/041_checkpoint_at_promote.pl @@ -142,10 +142,33 @@ chomp($pid); $killme_stdout = ''; $killme_stderr = ''; +# Keep another backend connected so we can tell when the postmaster has +# noticed the crashed backend and begun restarting the server. +my ($monitor_stdin, $monitor_stdout, $monitor_stderr) = ('', '', ''); +my $monitor = IPC::Run::start( + [ + 'psql', '--no-psqlrc', '--quiet', '--no-align', '--tuples-only', + '--set' => 'ON_ERROR_STOP=1', + '--file' => '-', + '--dbname' => $node_standby->connstr('postgres') + ], + '<' => \$monitor_stdin, + '>' => \$monitor_stdout, + '2>' => \$monitor_stderr, + $psql_timeout); +$monitor_stdin .= q[ +SELECT $$psql-connected$$; +]; +ok( pump_until( + $monitor, $psql_timeout, \$monitor_stdout, qr/psql-connected/m), + 'monitor connected'); +$monitor_stdout = ''; +$monitor_stderr = ''; + my $ret = PostgreSQL::Test::Utils::system_log('pg_ctl', 'kill', 'KILL', $pid); is($ret, 0, 'killed process with KILL'); -# Wait until the server restarts, finish consuming output. +# Finish consuming output from the killed backend. $killme_stdin .= q[ SELECT 1; ]; @@ -158,8 +181,23 @@ ok( pump_until( "psql query died successfully after SIGKILL"); $killme->finish; -# Wait till server finishes restarting. -$node_standby->poll_query_until('postgres', undef, ''); +# Wait until the monitor backend is terminated by the postmaster. The killed +# backend's own connection can fail before the postmaster begins restarting. +$monitor_stdin .= qq[ +SELECT pg_sleep($PostgreSQL::Test::Utils::timeout_default); +]; +ok( pump_until( + $monitor, + $psql_timeout, + \$monitor_stderr, + qr/WARNING: terminating connection because of crash of another server process|server closed the connection unexpectedly|connection to server was lost|could not send data to server/m + ), + "psql monitor died successfully after SIGKILL"); +$monitor->finish; + +# Now wait till server finishes restarting. +is($node_standby->poll_query_until('postgres', undef, ''), + "1", "reconnected after SIGKILL"); # After recovery, the server should be able to start. my $stdout; -- 2.55.0
^ permalink raw reply [nested|flat] 12+ messages in thread

* Re: 041_checkpoint_at_promote.pl might fail due to race condition on child kill
@ 2026-10-08 19:00  Alexander Lakhin <exclusion@gmail.com>
  parent: Fujii Masao <masao.fujii@gmail.com>
  0 siblings, 1 reply; 12+ messages in thread

From: Alexander Lakhin @ 2026-10-08 19:00 UTC (permalink / raw)
  To: Fujii Masao <masao.fujii@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; pgsql-hackers

Hello Fujii-san,

07.10.2026 19:46, Fujii Masao wrote:
> On Wed, Oct 7, 2026 at 1:00 PM Alexander Lakhin<exclusion@gmail.com> wrote:
>> It looks like the same race condition was fixed in 022_crash_temp_files.pl
>> by e37ad5fa4 two years before 041_checkpoint_at_promote.pl appeared: [2].
> I tried creating the patch based on that fix. How about the attached patch?

Your patch works for me test-wise. The same test loop passed 100+
iterations successfully. Thank you for paying attention to this!

Best regards,
Alexander

^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* Re: 041_checkpoint_at_promote.pl might fail due to race condition on child kill
@ 2026-10-13 07:17  Fujii Masao <masao.fujii@gmail.com>
  parent: Alexander Lakhin <exclusion@gmail.com>
  0 siblings, 2 replies; 12+ messages in thread

From: Fujii Masao @ 2026-10-13 07:17 UTC (permalink / raw)
  To: Alexander Lakhin <exclusion@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; pgsql-hackers

On Fri, Oct 9, 2026 at 4:00 AM Alexander Lakhin <exclusion@gmail.com> wrote:
> Your patch works for me test-wise. The same test loop passed 100+
> iterations successfully. Thank you for paying attention to this!

Thanks for the test!

On second thought, we can simplify the test by waiting for the postmaster to
log "all server processes terminated; reinitializing" instead of using
another backend connection to detect the crash. This ensures that the
postmaster has detected the crash and started restarting. The attached
patch takes this approach. Thoughts?

Regards,

-- 
Fujii Masao

Attachments:

v2-0001-Fix-crash-restart-race-in-041_checkpoint_at_promo.patch (application/octet-stream, 3.0K) download From 794d7b6a45d13fc92399cb27f235e48143871eae Mon Sep 17 00:00:00 2001 From: Fujii Masao <fujii@postgresql.org> Date: Wed, 7 Oct 2026 21:31:47 +0900 Subject: [PATCH v2] Fix crash-restart race in 041_checkpoint_at_promote.pl The buildfarm member bushmaster reported failures in the final test query in 041_checkpoint_at_promote.pl with "the database system is in recovery mode". After killing a backend to trigger crash recovery, the test should wait for the postmaster to terminate the remaining server processes and restart the server, then wait for recovery to finish and connections to be accepted before running the final query. Previously, the test assumed that the killed backend's connection closing meant the postmaster had noticed the crash and begun restarting. However, the connection can close before the postmaster notices the crash. The subsequent connection check can then succeed before recovery starts, causing the test to mistake this for recovery having completed. The final query could then run during recovery and fail. Fix this by waiting for the postmaster to log that all server processes have terminated and it is reinitializing before checking that connections are accepted again. This ensures that the postmaster has noticed the crash and begun restarting before the test waits for recovery to finish. Backpatch to v17, where this test was introduced. Reported-by: Alexander Lakhin <exclusion@gmail.com> Author: Fujii Masao <masao.fujii@gmail.com> Discussion: https://postgr.es/m/e7abd282-655d-495e-94c9-e38a58917ebf@gmail.com Backpatch-through: 17 --- src/test/recovery/t/041_checkpoint_at_promote.pl | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/src/test/recovery/t/041_checkpoint_at_promote.pl b/src/test/recovery/t/041_checkpoint_at_promote.pl index d0783fef9ae..82173b9d86e 100644 --- a/src/test/recovery/t/041_checkpoint_at_promote.pl +++ b/src/test/recovery/t/041_checkpoint_at_promote.pl @@ -142,10 +142,13 @@ chomp($pid); $killme_stdout = ''; $killme_stderr = ''; +# Record the log position before triggering crash recovery. +my $crash_logstart = -s $node_standby->logfile; + my $ret = PostgreSQL::Test::Utils::system_log('pg_ctl', 'kill', 'KILL', $pid); is($ret, 0, 'killed process with KILL'); -# Wait until the server restarts, finish consuming output. +# Finish consuming output from the killed backend. $killme_stdin .= q[ SELECT 1; ]; @@ -158,8 +161,15 @@ ok( pump_until( "psql query died successfully after SIGKILL"); $killme->finish; -# Wait till server finishes restarting. -$node_standby->poll_query_until('postgres', undef, ''); +# Wait until the postmaster has noticed the crash and begun restarting. +# The killed backend's connection can fail before this happens. +$node_standby->wait_for_log( + qr/all server processes terminated; reinitializing/, + $crash_logstart); + +# Wait until recovery finishes and connections are accepted. +is($node_standby->poll_query_until('postgres', undef, ''), + 1, 'reconnected after SIGKILL'); # After recovery, the server should be able to start. my $stdout; -- 2.56.0
^ permalink raw reply [nested|flat] 12+ messages in thread

* Re: 041_checkpoint_at_promote.pl might fail due to race condition on child kill
@ 2026-10-29 12:37  Fujii Masao <masao.fujii@gmail.com>
  parent: Fujii Masao <masao.fujii@gmail.com>
  1 sibling, 0 replies; 12+ messages in thread

From: Fujii Masao @ 2026-10-29 12:37 UTC (permalink / raw)
  To: Alexander Lakhin <exclusion@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; pgsql-hackers

On Tue, Oct 13, 2026 at 4:17 PM Fujii Masao <masao.fujii@gmail.com> wrote:
> On second thought, we can simplify the test by waiting for the postmaster to
> log "all server processes terminated; reinitializing" instead of using
> another backend connection to detect the crash. This ensures that the
> postmaster has detected the crash and started restarting. The attached
> patch takes this approach. Thoughts?

Barring any objections, I will commit this patch.

Regards,

-- 
Fujii Masao






^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* Re: 041_checkpoint_at_promote.pl might fail due to race condition on child kill
@ 2026-10-30 06:58  Ayush Tiwari <ayushtiwari.slg01@gmail.com>
  parent: Fujii Masao <masao.fujii@gmail.com>
  1 sibling, 1 reply; 12+ messages in thread

From: Ayush Tiwari @ 2026-10-30 06:58 UTC (permalink / raw)
  To: Fujii Masao <masao.fujii@gmail.com>; +Cc: Alexander Lakhin <exclusion@gmail.com>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers

Hi,

On Tue, 13 Oct 2026 at 12:47, Fujii Masao <masao.fujii@gmail.com> wrote:
>
> On Fri, Oct 9, 2026 at 4:00 AM Alexander Lakhin <exclusion@gmail.com> wrote:
> > Your patch works for me test-wise. The same test loop passed 100+
> > iterations successfully. Thank you for paying attention to this!
>
> Thanks for the test!
>
> On second thought, we can simplify the test by waiting for the postmaster to
> log "all server processes terminated; reinitializing" instead of using
> another backend connection to detect the crash. This ensures that the
> postmaster has detected the crash and started restarting. The attached
> patch takes this approach. Thoughts?

Thanks for the patch.

Did multiple runs with it, and the changes lgtm.

Regards,
Ayush






^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* Re: 041_checkpoint_at_promote.pl might fail due to race condition on child kill
@ 2026-10-30 11:55  Fujii Masao <masao.fujii@gmail.com>
  parent: Ayush Tiwari <ayushtiwari.slg01@gmail.com>
  0 siblings, 1 reply; 12+ messages in thread

From: Fujii Masao @ 2026-10-30 11:55 UTC (permalink / raw)
  To: Ayush Tiwari <ayushtiwari.slg01@gmail.com>; +Cc: Alexander Lakhin <exclusion@gmail.com>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers

On Fri, Oct 30, 2026 at 3:59 PM Ayush Tiwari
<ayushtiwari.slg01@gmail.com> wrote:
> Did multiple runs with it, and the changes lgtm.

I've pushed the patch. Thanks!

Regards,

-- 
Fujii Masao






^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* Re: 041_checkpoint_at_promote.pl might fail due to race condition on child kill
@ 2026-11-14 14:00  Alexander Lakhin <exclusion@gmail.com>
  parent: Fujii Masao <masao.fujii@gmail.com>
  0 siblings, 1 reply; 12+ messages in thread

From: Alexander Lakhin @ 2026-11-14 14:00 UTC (permalink / raw)
  To: Fujii Masao <masao.fujii@gmail.com>; Ayush Tiwari <ayushtiwari.slg01@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; pgsql-hackers

Hello Fujii-san and Ayush,

30.10.2026 13:55, Fujii Masao wrote:
> On Fri, Oct 30, 2026 at 3:59 PM Ayush Tiwari
> <ayushtiwari.slg01@gmail.com> wrote:
>> Did multiple runs with it, and the changes lgtm.
> I've pushed the patch. Thanks!

Please look at a new failure of 041_checkpoint_at_promote produced by
tamandua recently [1]:
[03:10:10.869](0.025s) ok 4 - psql query died successfully after SIGKILL
[03:13:15.804](184.935s) # die: timed out waiting for file 
/home/bf/bf-build/tamandua/REL_19_STABLE/pgsql.build/testrun/recovery/041_checkpoint_at_promote/log/041_checkpoint_at_promote_standby1.log 
contents to match: (?^:all server processes terminated; reinitializing) at 
/home/bf/bf-build/tamandua/REL_19_STABLE/pgsql/src/test/recovery/t/041_checkpoint_at_promote.pl line 166.
[03:13:15.806](0.003s) 1..4
timed out waiting for file 
/home/bf/bf-build/tamandua/REL_19_STABLE/pgsql.build/testrun/recovery/041_checkpoint_at_promote/log/041_checkpoint_at_promote_standby1.log 
contents to match: (?^:all server processes terminated; reinitializing) at 
/home/bf/bf-build/tamandua/REL_19_STABLE/pgsql/src/test/recovery/t/041_checkpoint_at_promote.pl line 166.
# No postmaster PID for node "master"

041_checkpoint_at_promote_standby1.log contains:
2026-11-12 03:10:10.826 CET [3578531][client backend][6/2:0] LOG: statement: SELECT pg_backend_pid();
2026-11-12 03:10:10.861 CET [3577203][postmaster][:0] LOG:  client backend (PID 3578531) was terminated by signal 9: Killed
2026-11-12 03:10:10.861 CET [3577203][postmaster][:0] DETAIL: Failed process was running: SELECT pg_backend_pid();
2026-11-12 03:10:10.862 CET [3577203][postmaster][:0] LOG: terminating any other active server processes
2026-11-12 03:10:10.928 CET [3577203][postmaster][:0] LOG:  database system is ready to accept connections

There is no "terminating any other active server processes" message
indeed, the server was not shut down.

I managed to reproduce this locally (with the patch attached and with the
same parallel loop as before) and discovered that this can happen when
PMState goes from PM_HOT_STANDBY to PM_RUN (via PM_WAIT_BACKENDS) instead
of PM_HOT_STANDBY -> PM_WAIT_BACKENDS -> PM_WAIT_DEAD_END ->
PM_NO_CHILDREN -> "all server processes terminated; reinitializing".

This can be explained by two race conditions, with one precondition: the
startup process doesn't exit before pg_ctl kill.
1) The startup process becomes zombie for the postmaster if it exits when
waitpid() doesn't wait.
...
2026-11-14 13:29:02.900 UTC [695404][startup][:0] DEBUG: proc_exit(-1): 0 callbacks to make
...
2026-11-14 13:29:03.088 UTC [695367][postmaster][:0] DEBUG:  sending signal 3/SIGQUIT to startup process with pid 695404
2026-11-14 13:29:03.100 UTC [695367][postmaster][:0] LOG: !!!signal_child| delayed signal 3/SIGQUIT to startup process 
with pid 695404
...
2026-11-14 13:29:03.170 UTC [695367][postmaster][:0] DEBUG:  startup process 695404 is still running
...
2026-11-14 13:29:03.198 UTC [695367][postmaster][:0] DEBUG: updating PMState from PM_WAIT_BACKENDS to PM_RUN


2) SIGQUIT sent to the startup process when it's being executed inside
exit(0), after the point when signal handlers of a process are not called
anymore, so SIGQUIT is ignored:
...
2026-11-14 07:54:44.900 EST [1110043][startup][:0] LOG: !!!proc_exit| delayed exit(0) for startup process
...
2026-11-14 07:54:44.900 EST [1110043][startup][:0] DEBUG: proc_exit(-1): 0 callbacks to make
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] LOG: !!!signal_child| delayed signal 3/SIGQUIT to startup process 
with pid 1110043
...
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: startup process 1110043 is still running
...
2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110043, status: 0
2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 89
2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] DEBUG: updating PMState from PM_WAIT_BACKENDS to PM_RUN

Corresponding logs for these 2 cases and also a normal run's log are
attached for reference.

[1] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=tamandua&dt=2026-11-12%2002%3A04%3A37

Best regards,
Alexander

Attachments:

recovery-041-debugging.patch (text/x-patch, 2.2K) download diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c index f53aa9c404d..d1db634644b 100644 --- a/src/backend/postmaster/postmaster.c +++ b/src/backend/postmaster/postmaster.c @@ -68,6 +68,7 @@ #include <unistd.h> #include <signal.h> #include <time.h> +#include <sys/time.h> #include <sys/wait.h> #include <ctype.h> #include <sys/stat.h> @@ -2253,6 +2254,16 @@ handle_pm_child_exit_signal(SIGNAL_ARGS) SetLatch(MyLatch); } +static pid_t +_waitpid(pid_t pid, int *status, int options) +{ +pid_t ret; +elog(LOG, "!!!waitpid| pid: %d", pid); +ret = waitpid(pid, status, options); +elog(LOG, "!!!waitpid| ret: %d, status: %d", ret, *status); +return ret; +} + /* * Cleanup after a child process dies. */ @@ -2267,7 +2278,7 @@ process_pm_child_exit(void) ereport(DEBUG4, (errmsg_internal("reaping dead processes"))); - while ((pid = waitpid(-1, &exitstatus, WNOHANG)) > 0) + while ((pid = _waitpid(-1, &exitstatus, WNOHANG)) > 0) { PMChild *pmchild; @@ -3519,6 +3530,13 @@ signal_child(PMChild *pmchild, int signal) GetBackendTypeDesc(pmchild->bkend_type), (int) pmchild->pid))); +if (pmchild == StartupPMChild && signal == SIGQUIT) +{ +struct timeval current_time; +do gettimeofday(&current_time, NULL); while ((current_time.tv_usec / 1000) % 100 >= 2); +elog(LOG, "!!!signal_child| delayed signal %d/%s to startup process with pid %d", signal, pm_signame(signal), (int) pmchild->pid); +} + if (kill(pid, signal) < 0) elog(DEBUG3, "kill(%ld,%d) failed: %m", (long) pid, signal); #ifdef HAVE_SETSID diff --git a/src/backend/storage/ipc/ipc.c b/src/backend/storage/ipc/ipc.c index cb944edd8df..e96b3c2a035 100644 --- a/src/backend/storage/ipc/ipc.c +++ b/src/backend/storage/ipc/ipc.c @@ -22,6 +22,7 @@ #include <signal.h> #include <unistd.h> #include <sys/stat.h> +#include <sys/time.h> #include "miscadmin.h" #ifdef PROFILE_PID_DIR @@ -154,6 +155,13 @@ proc_exit(int code) elog(DEBUG3, "exit(%d)", code); +if (AmStartupProcess() && code == 0) +{ +struct timeval current_time; +pg_usleep(250000); +do gettimeofday(&current_time, NULL); while ((current_time.tv_usec / 1000) % 100 >= 2); +elog(LOG, "!!!proc_exit| delayed exit(0) for startup process"); +} exit(code); }
041_checkpoint_at_promote_standby1-2.log (text/x-log, 43.4K) download ... 2026-11-14 07:54:44.636 EST [1110043][startup][37/0:0] LOG: received promote request 2026-11-14 07:54:44.636 EST [1110043][startup][37/0:0] LOG: redo done at 0/04000028 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.58 s 2026-11-14 07:54:44.636 EST [1110043][startup][37/0:0] LOG: last completed transaction was at log time 2026-11-14 07:54:44.226403-05 2026-11-14 07:54:44.636 EST [1110043][startup][37/0:0] DEBUG: resetting unlogged relations: cleanup 0 init 1 2026-11-14 07:54:44.637 EST [1110043][startup][37/0:0] LOG: selected new timeline ID: 2 2026-11-14 07:54:44.637 EST [1110043][startup][37/0:0] DEBUG: updated min recovery point to 0/040000B0 on timeline 1 2026-11-14 07:54:44.648 EST [1110043][startup][37/0:0] DEBUG: could not remove file "pg_wal/000000020000000000000004": No such file or directory 2026-11-14 07:54:44.648 EST [1110043][startup][37/0:0] LOG: archive recovery complete 2026-11-14 07:54:44.648 EST [1110043][startup][37/0:0] DEBUG: MultiXactId wrap limit is 2147483648, limited by database with OID 1 2026-11-14 07:54:44.648 EST [1110043][startup][37/0:0] DEBUG: attempting to remove WAL segments newer than log file 000000020000000000000004 2026-11-14 07:54:44.648 EST [1110043][startup][37/0:0] DEBUG: removing all KnownAssignedXids 2026-11-14 07:54:44.648 EST [1110043][startup][37/0:0] DEBUG: release all standby locks 2026-11-14 07:54:44.648 EST [1110043][startup][37/0:0] DEBUG: shmem_exit(0): 5 before_shmem_exit callbacks to make 2026-11-14 07:54:44.648 EST [1110043][startup][37/0:0] DEBUG: shmem_exit(0): 8 on_shmem_exit callbacks to make 2026-11-14 07:54:44.648 EST [1110043][startup][:0] DEBUG: proc_exit(0): 1 callbacks to make 2026-11-14 07:54:44.648 EST [1110043][startup][:0] DEBUG: exit(0) 2026-11-14 07:54:44.738 EST [1110037][postmaster][:0] DEBUG: assigned pm child slot 2 for client backend 2026-11-14 07:54:44.738 EST [1110037][postmaster][:0] DEBUG: forked new client backend, pid=1110667 socket=8 2026-11-14 07:54:44.738 EST [1110667][client backend][:0] LOG: connection received: host=[local] 2026-11-14 07:54:44.738 EST [1110667][client backend][5/0:0] DEBUG: InitPostgres 2026-11-14 07:54:44.739 EST [1110667][client backend][5/1:0] LOG: connection authenticated: ... 2026-11-14 07:54:44.739 EST [1110667][client backend][5/1:0] LOG: connection authorized: ... 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] LOG: statement: SELECT injection_points_wakeup('create-restart-point'); 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: find_in_path: trying ".../tmp_install/usr/local/pgsql/lib/injection_points" 2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: find_in_path: trying ".../tmp_install/usr/local/pgsql/lib/injection_points.so" 2026-11-14 07:54:44.741 EST [1110667][client backend][5/0:0] DEBUG: shmem_exit(0): 5 before_shmem_exit callbacks to make 2026-11-14 07:54:44.741 EST [1110667][client backend][5/0:0] DEBUG: shmem_exit(0): 7 on_shmem_exit callbacks to make 2026-11-14 07:54:44.741 EST [1110667][client backend][:0] DEBUG: proc_exit(0): 3 callbacks to make 2026-11-14 07:54:44.741 EST [1110667][client backend][:0] LOG: disconnection: session time: 0:00:00.002 ... 2026-11-14 07:54:44.741 EST [1110667][client backend][:0] DEBUG: exit(0) 2026-11-14 07:54:44.741 EST [1110667][client backend][:0] DEBUG: shmem_exit(-1): 0 before_shmem_exit callbacks to make 2026-11-14 07:54:44.741 EST [1110667][client backend][:0] DEBUG: shmem_exit(-1): 0 on_shmem_exit callbacks to make 2026-11-14 07:54:44.741 EST [1110667][client backend][:0] DEBUG: proc_exit(-1): 0 callbacks to make 2026-11-14 07:54:44.741 EST [1110037][postmaster][:0] DEBUG: reaping dead processes 2026-11-14 07:54:44.741 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.741 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110667, status: 0 2026-11-14 07:54:44.741 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 2 2026-11-14 07:54:44.741 EST [1110037][postmaster][:0] DEBUG: client backend (PID 1110667) exited with exit code 0 2026-11-14 07:54:44.741 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.741 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 0 2026-11-14 07:54:44.749 EST [1110041][checkpointer][:0] DEBUG: attempting to remove WAL segments older than log file 000000000000000000000002 2026-11-14 07:54:44.749 EST [1110041][checkpointer][:0] DEBUG: recycled write-ahead log file "000000010000000000000002" 2026-11-14 07:54:44.749 EST [1110041][checkpointer][:0] DEBUG: SlruScanDirectory invoking callback on pg_subtrans/0000 2026-11-14 07:54:44.749 EST [1110041][checkpointer][:0] LOG: restartpoint complete: fast wait: wrote 38 buffers (29.7%), wrote 2 SLRU buffers; 0 WAL file(s) added, 0 removed, 1 recycled; write=0.001 s, sync=0.001 s, total=0.528 s; sync files=0, longest=0.000 s, average=0.000 s; distance=16384 kB, estimate=16384 kB; lsn=0/03000088, redo lsn=0/03000028 2026-11-14 07:54:44.749 EST [1110041][checkpointer][:0] LOG: recovery restart point at 0/03000028 2026-11-14 07:54:44.749 EST [1110041][checkpointer][:0] DETAIL: Last completed transaction was at log time 2026-11-14 07:54:44.226403-05. 2026-11-14 07:54:44.749 EST [1110041][checkpointer][:0] LOG: checkpoint starting: force 2026-11-14 07:54:44.843 EST [1110037][postmaster][:0] DEBUG: assigned pm child slot 2 for client backend 2026-11-14 07:54:44.843 EST [1110037][postmaster][:0] DEBUG: forked new client backend, pid=1110740 socket=8 2026-11-14 07:54:44.843 EST [1110740][client backend][:0] LOG: connection received: host=[local] 2026-11-14 07:54:44.843 EST [1110740][client backend][6/0:0] DEBUG: InitPostgres 2026-11-14 07:54:44.844 EST [1110740][client backend][6/1:0] LOG: connection authenticated: ... 2026-11-14 07:54:44.844 EST [1110740][client backend][6/1:0] LOG: connection authorized: ... 2026-11-14 07:54:44.844 EST [1110740][client backend][6/2:0] LOG: statement: SELECT pg_backend_pid(); 2026-11-14 07:54:44.844 EST [1110740][client backend][6/2:0] DEBUG: io 6 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8 2026-11-14 07:54:44.844 EST [1110740][client backend][6/2:0] DEBUG: io 6 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8 2026-11-14 07:54:44.844 EST [1110740][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8 2026-11-14 07:54:44.844 EST [1110740][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8 2026-11-14 07:54:44.844 EST [1110740][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8 2026-11-14 07:54:44.844 EST [1110740][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8 2026-11-14 07:54:44.844 EST [1110740][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.844 EST [1110740][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8 2026-11-14 07:54:44.844 EST [1110740][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.844 EST [1110740][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8 2026-11-14 07:54:44.845 EST [1110037][postmaster][:0] DEBUG: reaping dead processes 2026-11-14 07:54:44.845 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.845 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110740, status: 9 2026-11-14 07:54:44.845 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 2 2026-11-14 07:54:44.845 EST [1110037][postmaster][:0] LOG: client backend (PID 1110740) was terminated by signal 9: Killed 2026-11-14 07:54:44.845 EST [1110037][postmaster][:0] DETAIL: Failed process was running: SELECT pg_backend_pid(); 2026-11-14 07:54:44.845 EST [1110037][postmaster][:0] LOG: terminating any other active server processes 2026-11-14 07:54:44.845 EST [1110037][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to client backend process with pid 1110383 2026-11-14 07:54:44.845 EST [1110037][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to startup process with pid 1110043 2026-11-14 07:54:44.849 EST [1110041][checkpointer][:0] DEBUG: performing replication slot checkpoint 2026-11-14 07:54:44.849 EST [1110041][checkpointer][:0] DEBUG: snapshot of 0+0 running transaction ids (lsn 0/04000148 oldest xid 667 latest complete 666 next xid 667) 2026-11-14 07:54:44.849 EST [1110041][checkpointer][:0] DEBUG: attempting to remove WAL segments older than log file 000000000000000000000003 2026-11-14 07:54:44.849 EST [1110041][checkpointer][:0] DEBUG: recycled write-ahead log file "000000010000000000000003" 2026-11-14 07:54:44.849 EST [1110041][checkpointer][:0] DEBUG: SlruScanDirectory invoking callback on pg_subtrans/0000 2026-11-14 07:54:44.849 EST [1110041][checkpointer][:0] LOG: checkpoint complete: force: wrote 1 buffers (0.8%), wrote 2 SLRU buffers; 0 WAL file(s) added, 0 removed, 1 recycled; write=0.101 s, sync=0.001 s, total=0.101 s; sync files=0, longest=0.000 s, average=0.000 s; distance=16384 kB, estimate=16384 kB; lsn=0/04000148, redo lsn=0/040000E8 2026-11-14 07:54:44.900 EST [1110043][startup][:0] LOG: !!!proc_exit| delayed exit(0) for startup process 2026-11-14 07:54:44.900 EST [1110043][startup][:0] DEBUG: shmem_exit(-1): 0 before_shmem_exit callbacks to make 2026-11-14 07:54:44.900 EST [1110043][startup][:0] DEBUG: shmem_exit(-1): 0 on_shmem_exit callbacks to make 2026-11-14 07:54:44.900 EST [1110043][startup][:0] DEBUG: proc_exit(-1): 0 callbacks to make 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] LOG: !!!signal_child| delayed signal 3/SIGQUIT to startup process with pid 1110043 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to background writer process with pid 1110042 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to checkpointer process with pid 1110041 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to io worker process with pid 1110040 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to io worker process with pid 1110039 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: updating PMState from PM_HOT_STANDBY to PM_WAIT_BACKENDS 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110383, status: 512 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 1 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 512 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: startup process 1110043 is still running 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: background writer process 1110042 is still running 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: checkpointer process 1110041 is still running 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: io worker process 1110040 is still running 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: io worker process 1110039 is still running 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: reaping dead processes 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 43690 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: startup process 1110043 is still running 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: background writer process 1110042 is still running 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: checkpointer process 1110041 is still running 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: io worker process 1110040 is still running 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: io worker process 1110039 is still running 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: reaping dead processes 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110040, status: 512 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 58 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 512 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: startup process 1110043 is still running 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: background writer process 1110042 is still running 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: checkpointer process 1110041 is still running 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: io worker process 1110039 is still running 2026-11-14 07:54:44.902 EST [1110037][postmaster][:0] DEBUG: reaping dead processes 2026-11-14 07:54:44.902 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.902 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110039, status: 512 2026-11-14 07:54:44.902 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 57 2026-11-14 07:54:44.902 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.902 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110041, status: 512 2026-11-14 07:54:44.902 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 56 2026-11-14 07:54:44.902 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.902 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110042, status: 512 2026-11-14 07:54:44.902 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 55 2026-11-14 07:54:44.902 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.902 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 512 2026-11-14 07:54:44.902 EST [1110037][postmaster][:0] DEBUG: startup process 1110043 is still running 2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] DEBUG: reaping dead processes 2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110043, status: 0 2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 89 2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] DEBUG: updating PMState from PM_WAIT_BACKENDS to PM_RUN 2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] LOG: database system is ready to accept connections 2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: -1, status: 0 2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] DEBUG: assigned pm child slot 57 for io worker 2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] DEBUG: assigned pm child slot 58 for io worker 2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] DEBUG: assigned pm child slot 56 for checkpointer 2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] DEBUG: assigned pm child slot 55 for background writer 2026-11-14 07:54:44.904 EST [1110037][postmaster][:0] DEBUG: assigned pm child slot 92 for walwriter 2026-11-14 07:54:44.904 EST [1110037][postmaster][:0] DEBUG: assigned pm child slot 45 for background worker 2026-11-14 07:54:44.904 EST [1110037][postmaster][:0] DEBUG: starting background worker process "logical replication launcher" 2026-11-14 07:54:44.904 EST [1110788][checkpointer][:0] DEBUG: checkpointer updated shared memory configuration values 2026-11-14 07:54:44.904 EST [1110793][logical replication launcher][15/0:0] DEBUG: logical replication launcher started 2026-11-14 07:54:44.904 EST [1110793][logical replication launcher][15/0:0] DEBUG: InitPostgres 2026-11-14 07:55:00.050 EST [1110790][background writer][:0] DEBUG: snapshot of 0+0 running transaction ids (lsn 0/04000208 oldest xid 667 latest complete 666 next xid 667) 2026-11-14 07:55:44.967 EST [1110786][io worker][:0] DEBUG: shmem_exit(0): 4 before_shmem_exit callbacks to make 2026-11-14 07:55:44.967 EST [1110786][io worker][:0] DEBUG: shmem_exit(0): 7 on_shmem_exit callbacks to make 2026-11-14 07:55:44.967 EST [1110786][io worker][:0] DEBUG: proc_exit(0): 1 callbacks to make 2026-11-14 07:55:44.967 EST [1110786][io worker][:0] DEBUG: exit(0) 2026-11-14 07:55:44.967 EST [1110786][io worker][:0] DEBUG: shmem_exit(-1): 0 before_shmem_exit callbacks to make 2026-11-14 07:55:44.967 EST [1110786][io worker][:0] DEBUG: shmem_exit(-1): 0 on_shmem_exit callbacks to make 2026-11-14 07:55:44.967 EST [1110786][io worker][:0] DEBUG: proc_exit(-1): 0 callbacks to make 2026-11-14 07:55:44.968 EST [1110037][postmaster][:0] DEBUG: reaping dead processes 2026-11-14 07:55:44.968 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:55:44.968 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110786, status: 0 2026-11-14 07:55:44.968 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 58 2026-11-14 07:55:44.968 EST [1110037][postmaster][:0] DEBUG: assigned pm child slot 58 for io worker 2026-11-14 07:55:44.969 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:55:44.969 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 0 2026-11-14 07:56:45.031 EST [1111045][io worker][:0] DEBUG: shmem_exit(0): 4 before_shmem_exit callbacks to make 2026-11-14 07:56:45.032 EST [1111045][io worker][:0] DEBUG: shmem_exit(0): 7 on_shmem_exit callbacks to make 2026-11-14 07:56:45.032 EST [1111045][io worker][:0] DEBUG: proc_exit(0): 1 callbacks to make 2026-11-14 07:56:45.032 EST [1111045][io worker][:0] DEBUG: exit(0) 2026-11-14 07:56:45.032 EST [1111045][io worker][:0] DEBUG: shmem_exit(-1): 0 before_shmem_exit callbacks to make 2026-11-14 07:56:45.032 EST [1111045][io worker][:0] DEBUG: shmem_exit(-1): 0 on_shmem_exit callbacks to make 2026-11-14 07:56:45.032 EST [1111045][io worker][:0] DEBUG: proc_exit(-1): 0 callbacks to make 2026-11-14 07:56:45.032 EST [1110037][postmaster][:0] DEBUG: reaping dead processes 2026-11-14 07:56:45.032 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:56:45.033 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1111045, status: 0 2026-11-14 07:56:45.033 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 58 2026-11-14 07:56:45.033 EST [1110037][postmaster][:0] DEBUG: assigned pm child slot 58 for io worker 2026-11-14 07:56:45.033 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:56:45.033 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 0 2026-11-14 07:57:45.098 EST [1111065][io worker][:0] DEBUG: shmem_exit(0): 4 before_shmem_exit callbacks to make 2026-11-14 07:57:45.098 EST [1111065][io worker][:0] DEBUG: shmem_exit(0): 7 on_shmem_exit callbacks to make 2026-11-14 07:57:45.098 EST [1111065][io worker][:0] DEBUG: proc_exit(0): 1 callbacks to make 2026-11-14 07:57:45.098 EST [1111065][io worker][:0] DEBUG: exit(0) 2026-11-14 07:57:45.098 EST [1111065][io worker][:0] DEBUG: shmem_exit(-1): 0 before_shmem_exit callbacks to make 2026-11-14 07:57:45.098 EST [1111065][io worker][:0] DEBUG: shmem_exit(-1): 0 on_shmem_exit callbacks to make 2026-11-14 07:57:45.098 EST [1111065][io worker][:0] DEBUG: proc_exit(-1): 0 callbacks to make 2026-11-14 07:57:45.099 EST [1110037][postmaster][:0] DEBUG: reaping dead processes 2026-11-14 07:57:45.099 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:57:45.099 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1111065, status: 0 2026-11-14 07:57:45.099 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 58 2026-11-14 07:57:45.099 EST [1110037][postmaster][:0] DEBUG: assigned pm child slot 58 for io worker 2026-11-14 07:57:45.099 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:57:45.099 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 0 2026-11-14 07:57:49.231 EST [1110037][postmaster][:0] DEBUG: postmaster received shutdown request signal 2026-11-14 07:57:49.231 EST [1110037][postmaster][:0] LOG: received immediate shutdown request 2026-11-14 07:57:49.231 EST [1110037][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to io worker process with pid 1111082 2026-11-14 07:57:49.231 EST [1110037][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to background worker process with pid 1110793 2026-11-14 07:57:49.231 EST [1110037][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to walwriter process with pid 1110791 2026-11-14 07:57:49.231 EST [1110037][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to background writer process with pid 1110790 2026-11-14 07:57:49.231 EST [1110037][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to checkpointer process with pid 1110788 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to io worker process with pid 1110784 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: updating PMState from PM_RUN to PM_WAIT_BACKENDS 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: io worker process 1111082 is still running 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: background worker process 1110793 is still running 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: walwriter process 1110791 is still running 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: background writer process 1110790 is still running 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: checkpointer process 1110788 is still running 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: io worker process 1110784 is still running 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: reaping dead processes 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110790, status: 512 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 55 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110788, status: 512 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 56 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110791, status: 512 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 92 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110784, status: 512 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 57 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110793, status: 512 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 45 2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:57:49.233 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1111082, status: 512 2026-11-14 07:57:49.233 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 58 2026-11-14 07:57:49.233 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:57:49.233 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: -1, status: 512 2026-11-14 07:57:49.233 EST [1110037][postmaster][:0] DEBUG: updating PMState from PM_WAIT_BACKENDS to PM_WAIT_DEAD_END 2026-11-14 07:57:49.233 EST [1110037][postmaster][:0] DEBUG: updating PMState from PM_WAIT_DEAD_END to PM_NO_CHILDREN 2026-11-14 07:57:49.233 EST [1110037][postmaster][:0] DEBUG: shmem_exit(0): 0 before_shmem_exit callbacks to make 2026-11-14 07:57:49.233 EST [1110037][postmaster][:0] DEBUG: shmem_exit(0): 5 on_shmem_exit callbacks to make 2026-11-14 07:57:49.233 EST [1110037][postmaster][:0] DEBUG: cleaning up orphaned dynamic shared memory with ID 1459897042 2026-11-14 07:57:49.233 EST [1110037][postmaster][:0] DEBUG: cleaning up orphaned dynamic shared memory with ID 3069910578 2026-11-14 07:57:49.233 EST [1110037][postmaster][:0] DEBUG: cleaning up orphaned dynamic shared memory with ID 3859456016 2026-11-14 07:57:49.233 EST [1110037][postmaster][:0] DEBUG: cleaning up dynamic shared memory control segment with ID 2869141000 2026-11-14 07:57:49.234 EST [1110037][postmaster][:0] DEBUG: proc_exit(0): 2 callbacks to make 2026-11-14 07:57:49.234 EST [1110037][postmaster][:0] LOG: database system is shut down 2026-11-14 07:57:49.234 EST [1110037][postmaster][:0] DEBUG: exit(0) 2026-11-14 07:57:49.234 EST [1110037][postmaster][:0] DEBUG: shmem_exit(-1): 0 before_shmem_exit callbacks to make 2026-11-14 07:57:49.234 EST [1110037][postmaster][:0] DEBUG: shmem_exit(-1): 0 on_shmem_exit callbacks to make 2026-11-14 07:57:49.234 EST [1110037][postmaster][:0] DEBUG: proc_exit(-1): 0 callbacks to make
041_checkpoint_at_promote_standby1-1.log (text/x-log, 71.0K) download
041_checkpoint_at_promote_standby1-0.log (text/x-log, 44.4K) download ... 2026-11-14 07:54:44.154 EST [1109760][startup][37/0:0] LOG: received promote request 2026-11-14 07:54:44.154 EST [1109760][startup][37/0:0] LOG: redo done at 0/04000028 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.33 s 2026-11-14 07:54:44.154 EST [1109760][startup][37/0:0] LOG: last completed transaction was at log time 2026-11-14 07:54:44.016107-05 2026-11-14 07:54:44.154 EST [1109760][startup][37/0:0] DEBUG: resetting unlogged relations: cleanup 0 init 1 2026-11-14 07:54:44.154 EST [1109760][startup][37/0:0] LOG: selected new timeline ID: 2 2026-11-14 07:54:44.155 EST [1109760][startup][37/0:0] DEBUG: updated min recovery point to 0/040000B0 on timeline 1 2026-11-14 07:54:44.172 EST [1109760][startup][37/0:0] DEBUG: could not remove file "pg_wal/000000020000000000000004": No such file or directory 2026-11-14 07:54:44.172 EST [1109760][startup][37/0:0] LOG: archive recovery complete 2026-11-14 07:54:44.172 EST [1109760][startup][37/0:0] DEBUG: MultiXactId wrap limit is 2147483648, limited by database with OID 1 2026-11-14 07:54:44.172 EST [1109760][startup][37/0:0] DEBUG: attempting to remove WAL segments newer than log file 000000020000000000000004 2026-11-14 07:54:44.172 EST [1109760][startup][37/0:0] DEBUG: removing all KnownAssignedXids 2026-11-14 07:54:44.172 EST [1109760][startup][37/0:0] DEBUG: release all standby locks 2026-11-14 07:54:44.172 EST [1109760][startup][37/0:0] DEBUG: shmem_exit(0): 5 before_shmem_exit callbacks to make 2026-11-14 07:54:44.172 EST [1109760][startup][37/0:0] DEBUG: shmem_exit(0): 8 on_shmem_exit callbacks to make 2026-11-14 07:54:44.172 EST [1109760][startup][:0] DEBUG: proc_exit(0): 1 callbacks to make 2026-11-14 07:54:44.172 EST [1109760][startup][:0] DEBUG: exit(0) 2026-11-14 07:54:44.256 EST [1109746][postmaster][:0] DEBUG: assigned pm child slot 2 for client backend 2026-11-14 07:54:44.256 EST [1109746][postmaster][:0] DEBUG: forked new client backend, pid=1110418 socket=8 2026-11-14 07:54:44.256 EST [1110418][client backend][:0] LOG: connection received: host=[local] 2026-11-14 07:54:44.257 EST [1110418][client backend][5/0:0] DEBUG: InitPostgres 2026-11-14 07:54:44.257 EST [1110418][client backend][5/1:0] LOG: connection authenticated: ... 2026-11-14 07:54:44.257 EST [1110418][client backend][5/1:0] LOG: connection authorized: ... 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] LOG: statement: SELECT injection_points_wakeup('create-restart-point'); 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: find_in_path: trying ".../tmp_install/usr/local/pgsql/lib/injection_points" 2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: find_in_path: trying ".../tmp_install/usr/local/pgsql/lib/injection_points.so" 2026-11-14 07:54:44.259 EST [1110418][client backend][5/0:0] DEBUG: shmem_exit(0): 5 before_shmem_exit callbacks to make 2026-11-14 07:54:44.259 EST [1110418][client backend][5/0:0] DEBUG: shmem_exit(0): 7 on_shmem_exit callbacks to make 2026-11-14 07:54:44.259 EST [1110418][client backend][:0] DEBUG: proc_exit(0): 3 callbacks to make 2026-11-14 07:54:44.259 EST [1110418][client backend][:0] LOG: disconnection: session time: 0:00:00.002 ... 2026-11-14 07:54:44.259 EST [1110418][client backend][:0] DEBUG: exit(0) 2026-11-14 07:54:44.259 EST [1110418][client backend][:0] DEBUG: shmem_exit(-1): 0 before_shmem_exit callbacks to make 2026-11-14 07:54:44.259 EST [1110418][client backend][:0] DEBUG: shmem_exit(-1): 0 on_shmem_exit callbacks to make 2026-11-14 07:54:44.259 EST [1110418][client backend][:0] DEBUG: proc_exit(-1): 0 callbacks to make 2026-11-14 07:54:44.260 EST [1109746][postmaster][:0] DEBUG: reaping dead processes 2026-11-14 07:54:44.260 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.260 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 1110418, status: 0 2026-11-14 07:54:44.260 EST [1109746][postmaster][:0] DEBUG: releasing pm child slot 2 2026-11-14 07:54:44.260 EST [1109746][postmaster][:0] DEBUG: client backend (PID 1110418) exited with exit code 0 2026-11-14 07:54:44.260 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.260 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 0 2026-11-14 07:54:44.273 EST [1109758][checkpointer][:0] DEBUG: attempting to remove WAL segments older than log file 000000000000000000000002 2026-11-14 07:54:44.273 EST [1109758][checkpointer][:0] DEBUG: recycled write-ahead log file "000000010000000000000002" 2026-11-14 07:54:44.273 EST [1109758][checkpointer][:0] DEBUG: SlruScanDirectory invoking callback on pg_subtrans/0000 2026-11-14 07:54:44.273 EST [1109758][checkpointer][:0] LOG: restartpoint complete: fast wait: wrote 16 buffers (12.5%), wrote 2 SLRU buffers; 0 WAL file(s) added, 0 removed, 1 recycled; write=0.001 s, sync=0.001 s, total=0.261 s; sync files=0, longest=0.000 s, average=0.000 s; distance=16384 kB, estimate=16384 kB; lsn=0/03000088, redo lsn=0/03000028 2026-11-14 07:54:44.273 EST [1109758][checkpointer][:0] LOG: recovery restart point at 0/03000028 2026-11-14 07:54:44.273 EST [1109758][checkpointer][:0] DETAIL: Last completed transaction was at log time 2026-11-14 07:54:44.016107-05. 2026-11-14 07:54:44.273 EST [1109758][checkpointer][:0] LOG: checkpoint starting: force 2026-11-14 07:54:44.361 EST [1109746][postmaster][:0] DEBUG: assigned pm child slot 2 for client backend 2026-11-14 07:54:44.361 EST [1109746][postmaster][:0] DEBUG: forked new client backend, pid=1110482 socket=8 2026-11-14 07:54:44.361 EST [1110482][client backend][:0] LOG: connection received: host=[local] 2026-11-14 07:54:44.362 EST [1110482][client backend][6/0:0] DEBUG: InitPostgres 2026-11-14 07:54:44.362 EST [1110482][client backend][6/1:0] LOG: connection authenticated: ... 2026-11-14 07:54:44.362 EST [1110482][client backend][6/1:0] LOG: connection authorized: ... 2026-11-14 07:54:44.362 EST [1110482][client backend][6/2:0] LOG: statement: SELECT pg_backend_pid(); 2026-11-14 07:54:44.363 EST [1110482][client backend][6/2:0] DEBUG: io 6 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8 2026-11-14 07:54:44.363 EST [1110482][client backend][6/2:0] DEBUG: io 6 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8 2026-11-14 07:54:44.363 EST [1110482][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8 2026-11-14 07:54:44.363 EST [1110482][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8 2026-11-14 07:54:44.363 EST [1110482][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8 2026-11-14 07:54:44.363 EST [1110482][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8 2026-11-14 07:54:44.363 EST [1110482][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.363 EST [1110482][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8 2026-11-14 07:54:44.363 EST [1110482][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8 2026-11-14 07:54:44.363 EST [1110482][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8 2026-11-14 07:54:44.364 EST [1109746][postmaster][:0] DEBUG: reaping dead processes 2026-11-14 07:54:44.364 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.364 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 1110482, status: 9 2026-11-14 07:54:44.364 EST [1109746][postmaster][:0] DEBUG: releasing pm child slot 2 2026-11-14 07:54:44.364 EST [1109746][postmaster][:0] LOG: client backend (PID 1110482) was terminated by signal 9: Killed 2026-11-14 07:54:44.364 EST [1109746][postmaster][:0] DETAIL: Failed process was running: SELECT pg_backend_pid(); 2026-11-14 07:54:44.364 EST [1109746][postmaster][:0] LOG: terminating any other active server processes 2026-11-14 07:54:44.364 EST [1109746][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to client backend process with pid 1109914 2026-11-14 07:54:44.364 EST [1109746][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to startup process with pid 1109760 2026-11-14 07:54:44.375 EST [1109758][checkpointer][:0] DEBUG: performing replication slot checkpoint 2026-11-14 07:54:44.375 EST [1109758][checkpointer][:0] DEBUG: snapshot of 0+0 running transaction ids (lsn 0/04000148 oldest xid 667 latest complete 666 next xid 667) 2026-11-14 07:54:44.375 EST [1109758][checkpointer][:0] DEBUG: attempting to remove WAL segments older than log file 000000000000000000000003 2026-11-14 07:54:44.375 EST [1109758][checkpointer][:0] DEBUG: recycled write-ahead log file "000000010000000000000003" 2026-11-14 07:54:44.375 EST [1109758][checkpointer][:0] DEBUG: SlruScanDirectory invoking callback on pg_subtrans/0000 2026-11-14 07:54:44.375 EST [1109758][checkpointer][:0] LOG: checkpoint complete: force: wrote 1 buffers (0.8%), wrote 2 SLRU buffers; 0 WAL file(s) added, 0 removed, 1 recycled; write=0.101 s, sync=0.001 s, total=0.103 s; sync files=0, longest=0.000 s, average=0.000 s; distance=16384 kB, estimate=16384 kB; lsn=0/04000148, redo lsn=0/040000E8 2026-11-14 07:54:44.400 EST [1109746][postmaster][:0] LOG: !!!signal_child| delayed signal 3/SIGQUIT to startup process with pid 1109760 2026-11-14 07:54:44.500 EST [1109760][startup][:0] LOG: !!!proc_exit| delayed exit(0) for startup process 2026-11-14 07:54:44.539 EST [1109746][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to background writer process with pid 1109759 2026-11-14 07:54:44.539 EST [1109746][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to checkpointer process with pid 1109758 2026-11-14 07:54:44.539 EST [1109746][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to io worker process with pid 1109757 2026-11-14 07:54:44.539 EST [1109746][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to io worker process with pid 1109756 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: updating PMState from PM_HOT_STANDBY to PM_WAIT_BACKENDS 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 1109756, status: 512 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: releasing pm child slot 57 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 1109757, status: 512 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: releasing pm child slot 58 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 1109758, status: 512 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: releasing pm child slot 56 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 1109759, status: 512 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: releasing pm child slot 55 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 1109760, status: 512 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: releasing pm child slot 89 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 1109914, status: 512 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: releasing pm child slot 1 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: -1, status: 512 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: updating PMState from PM_WAIT_BACKENDS to PM_WAIT_DEAD_END 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: updating PMState from PM_WAIT_DEAD_END to PM_NO_CHILDREN 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: all server processes terminated; reinitializing 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: shmem_exit(1): 0 before_shmem_exit callbacks to make 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: shmem_exit(1): 5 on_shmem_exit callbacks to make 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: cleaning up orphaned dynamic shared memory with ID 924198510 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: cleaning up orphaned dynamic shared memory with ID 1616401164 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: cleaning up orphaned dynamic shared memory with ID 3295781356 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: cleaning up dynamic shared memory control segment with ID 2272285588 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: invoking IpcMemoryCreate(size=5873664) 2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: mmap(6291456) with MAP_HUGETLB failed, huge pages disabled: Cannot allocate memory 2026-11-14 07:54:44.541 EST [1109746][postmaster][:0] DEBUG: dynamic shared memory system will support 229 segments 2026-11-14 07:54:44.541 EST [1109746][postmaster][:0] DEBUG: created dynamic shared memory control segment 4225595752 (9176 bytes) 2026-11-14 07:54:44.541 EST [1109746][postmaster][:0] DEBUG: updating PMState from PM_NO_CHILDREN to PM_STARTUP 2026-11-14 07:54:44.541 EST [1109746][postmaster][:0] DEBUG: assigned pm child slot 58 for io worker 2026-11-14 07:54:44.541 EST [1109746][postmaster][:0] DEBUG: assigned pm child slot 57 for io worker 2026-11-14 07:54:44.541 EST [1109746][postmaster][:0] DEBUG: assigned pm child slot 89 for startup 2026-11-14 07:54:44.541 EST [1109746][postmaster][:0] DEBUG: assigned pm child slot 56 for checkpointer 2026-11-14 07:54:44.541 EST [1109746][postmaster][:0] DEBUG: assigned pm child slot 55 for background writer 2026-11-14 07:54:44.542 EST [1109746][postmaster][:0] DEBUG: reaping dead processes 2026-11-14 07:54:44.542 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.542 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 43690 2026-11-14 07:54:44.542 EST [1110541][startup][:0] LOG: database system was interrupted; last known up at 2026-11-14 07:54:44 EST 2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: removing all temporary WAL segments 2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: checkpoint record is at 0/04000148 2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: redo record is at 0/040000E8; shutdown false 2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: next transaction ID: 667; next OID: 16398 2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: next MultiXactId: 1; next MultiXactOffset: 1 2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: oldest unfrozen transaction ID: 656, in database 1 2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: oldest MultiXactId: 1, in database 1 2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: commit timestamp Xid oldest/newest: 0/0 2026-11-14 07:54:44.542 EST [1110541][startup][:0] LOG: database system was not properly shut down; automatic recovery in progress 2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: transaction ID wrap limit is 2147484303, limited by database with OID 1 2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: MultiXactId wrap limit is 2147483648, limited by database with OID 1 2026-11-14 07:54:44.542 EST [1110542][checkpointer][:0] DEBUG: checkpointer updated shared memory configuration values 2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: starting up replication slots 2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: xmin required by slots: data 0, catalog 0 2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: starting up replication origin progress state 2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: didn't need to unlink permanent stats file "pg_stat/pgstat.stat" - didn't exist 2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: resetting unlogged relations: cleanup 1 init 0 2026-11-14 07:54:44.542 EST [1110541][startup][:0] LOG: redo starts at 0/040000E8 2026-11-14 07:54:44.542 EST [1110541][startup][:0] LOG: invalid record length at 0/040001D0: expected at least 24, got 0 2026-11-14 07:54:44.542 EST [1110541][startup][:0] LOG: redo done at 0/04000148 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s 2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: resetting unlogged relations: cleanup 0 init 1 2026-11-14 07:54:44.542 EST [1109746][postmaster][:0] DEBUG: postmaster received pmsignal signal 2026-11-14 07:54:44.542 EST [1109746][postmaster][:0] DEBUG: updating PMState from PM_STARTUP to PM_RECOVERY 2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: MultiXactId wrap limit is 2147483648, limited by database with OID 1 2026-11-14 07:54:44.543 EST [1110542][checkpointer][:0] LOG: checkpoint starting: end-of-recovery fast wait 2026-11-14 07:54:44.543 EST [1110542][checkpointer][:0] DEBUG: performing replication slot checkpoint 2026-11-14 07:54:44.543 EST [1110542][checkpointer][:0] DEBUG: attempting to remove WAL segments older than log file 000000000000000000000003 2026-11-14 07:54:44.544 EST [1110542][checkpointer][:0] LOG: checkpoint complete: end-of-recovery fast wait: wrote 0 buffers (0.0%), wrote 3 SLRU buffers; 0 WAL file(s) added, 0 removed, 0 recycled; write=0.001 s, sync=0.001 s, total=0.001 s; sync files=0, longest=0.000 s, average=0.000 s; distance=0 kB, estimate=0 kB; lsn=0/040001D0, redo lsn=0/040001D0 2026-11-14 07:54:44.544 EST [1110541][startup][:0] DEBUG: shmem_exit(0): 5 before_shmem_exit callbacks to make 2026-11-14 07:54:44.544 EST [1110541][startup][:0] DEBUG: shmem_exit(0): 7 on_shmem_exit callbacks to make 2026-11-14 07:54:44.544 EST [1110541][startup][:0] DEBUG: proc_exit(0): 1 callbacks to make 2026-11-14 07:54:44.544 EST [1110541][startup][:0] DEBUG: exit(0) 2026-11-14 07:54:44.569 EST [1109746][postmaster][:0] DEBUG: allocating dead-end child 2026-11-14 07:54:44.569 EST [1109746][postmaster][:0] DEBUG: forked new dead-end client backend, pid=1110598 socket=8 2026-11-14 07:54:44.569 EST [1110598][dead-end client backend][:0] LOG: connection received: host=[local] 2026-11-14 07:54:44.569 EST [1110598][dead-end client backend][:0] FATAL: the database system is not yet accepting connections 2026-11-14 07:54:44.569 EST [1110598][dead-end client backend][:0] DETAIL: Consistent recovery state has not been yet reached. 2026-11-14 07:54:44.569 EST [1110598][dead-end client backend][:0] DEBUG: shmem_exit(1): 0 before_shmem_exit callbacks to make 2026-11-14 07:54:44.569 EST [1110598][dead-end client backend][:0] DEBUG: shmem_exit(1): 0 on_shmem_exit callbacks to make 2026-11-14 07:54:44.569 EST [1110598][dead-end client backend][:0] DEBUG: proc_exit(1): 1 callbacks to make 2026-11-14 07:54:44.569 EST [1110598][dead-end client backend][:0] DEBUG: exit(1) 2026-11-14 07:54:44.569 EST [1110598][dead-end client backend][:0] DEBUG: shmem_exit(-1): 0 before_shmem_exit callbacks to make 2026-11-14 07:54:44.569 EST [1110598][dead-end client backend][:0] DEBUG: shmem_exit(-1): 0 on_shmem_exit callbacks to make 2026-11-14 07:54:44.569 EST [1110598][dead-end client backend][:0] DEBUG: proc_exit(-1): 0 callbacks to make 2026-11-14 07:54:44.570 EST [1109746][postmaster][:0] DEBUG: reaping dead processes 2026-11-14 07:54:44.570 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.570 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 1110598, status: 256 2026-11-14 07:54:44.570 EST [1109746][postmaster][:0] DEBUG: releasing dead-end backend 2026-11-14 07:54:44.570 EST [1109746][postmaster][:0] DEBUG: dead-end client backend (PID 1110598) exited with exit code 1 2026-11-14 07:54:44.570 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.570 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 256 2026-11-14 07:54:44.672 EST [1109746][postmaster][:0] DEBUG: allocating dead-end child 2026-11-14 07:54:44.672 EST [1109746][postmaster][:0] DEBUG: forked new dead-end client backend, pid=1110640 socket=8 2026-11-14 07:54:44.673 EST [1110640][dead-end client backend][:0] LOG: connection received: host=[local] 2026-11-14 07:54:44.673 EST [1110640][dead-end client backend][:0] FATAL: the database system is not yet accepting connections 2026-11-14 07:54:44.673 EST [1110640][dead-end client backend][:0] DETAIL: Consistent recovery state has not been yet reached. 2026-11-14 07:54:44.673 EST [1110640][dead-end client backend][:0] DEBUG: shmem_exit(1): 0 before_shmem_exit callbacks to make 2026-11-14 07:54:44.673 EST [1110640][dead-end client backend][:0] DEBUG: shmem_exit(1): 0 on_shmem_exit callbacks to make 2026-11-14 07:54:44.673 EST [1110640][dead-end client backend][:0] DEBUG: proc_exit(1): 1 callbacks to make 2026-11-14 07:54:44.673 EST [1110640][dead-end client backend][:0] DEBUG: exit(1) 2026-11-14 07:54:44.673 EST [1110640][dead-end client backend][:0] DEBUG: shmem_exit(-1): 0 before_shmem_exit callbacks to make 2026-11-14 07:54:44.673 EST [1110640][dead-end client backend][:0] DEBUG: shmem_exit(-1): 0 on_shmem_exit callbacks to make 2026-11-14 07:54:44.673 EST [1110640][dead-end client backend][:0] DEBUG: proc_exit(-1): 0 callbacks to make 2026-11-14 07:54:44.673 EST [1109746][postmaster][:0] DEBUG: reaping dead processes 2026-11-14 07:54:44.673 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.673 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 1110640, status: 256 2026-11-14 07:54:44.673 EST [1109746][postmaster][:0] DEBUG: releasing dead-end backend 2026-11-14 07:54:44.673 EST [1109746][postmaster][:0] DEBUG: dead-end client backend (PID 1110640) exited with exit code 1 2026-11-14 07:54:44.673 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.673 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 256 2026-11-14 07:54:44.775 EST [1109746][postmaster][:0] DEBUG: allocating dead-end child 2026-11-14 07:54:44.775 EST [1109746][postmaster][:0] DEBUG: forked new dead-end client backend, pid=1110689 socket=8 2026-11-14 07:54:44.775 EST [1110689][dead-end client backend][:0] LOG: connection received: host=[local] 2026-11-14 07:54:44.775 EST [1110689][dead-end client backend][:0] FATAL: the database system is not yet accepting connections 2026-11-14 07:54:44.775 EST [1110689][dead-end client backend][:0] DETAIL: Consistent recovery state has not been yet reached. 2026-11-14 07:54:44.775 EST [1110689][dead-end client backend][:0] DEBUG: shmem_exit(1): 0 before_shmem_exit callbacks to make 2026-11-14 07:54:44.775 EST [1110689][dead-end client backend][:0] DEBUG: shmem_exit(1): 0 on_shmem_exit callbacks to make 2026-11-14 07:54:44.775 EST [1110689][dead-end client backend][:0] DEBUG: proc_exit(1): 1 callbacks to make 2026-11-14 07:54:44.775 EST [1110689][dead-end client backend][:0] DEBUG: exit(1) 2026-11-14 07:54:44.775 EST [1110689][dead-end client backend][:0] DEBUG: shmem_exit(-1): 0 before_shmem_exit callbacks to make 2026-11-14 07:54:44.775 EST [1110689][dead-end client backend][:0] DEBUG: shmem_exit(-1): 0 on_shmem_exit callbacks to make 2026-11-14 07:54:44.775 EST [1110689][dead-end client backend][:0] DEBUG: proc_exit(-1): 0 callbacks to make 2026-11-14 07:54:44.776 EST [1109746][postmaster][:0] DEBUG: reaping dead processes 2026-11-14 07:54:44.776 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.776 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 1110689, status: 256 2026-11-14 07:54:44.776 EST [1109746][postmaster][:0] DEBUG: releasing dead-end backend 2026-11-14 07:54:44.776 EST [1109746][postmaster][:0] DEBUG: dead-end client backend (PID 1110689) exited with exit code 1 2026-11-14 07:54:44.776 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.776 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 256 2026-11-14 07:54:44.800 EST [1110541][startup][:0] LOG: !!!proc_exit| delayed exit(0) for startup process 2026-11-14 07:54:44.800 EST [1110541][startup][:0] DEBUG: shmem_exit(-1): 0 before_shmem_exit callbacks to make 2026-11-14 07:54:44.800 EST [1110541][startup][:0] DEBUG: shmem_exit(-1): 0 on_shmem_exit callbacks to make 2026-11-14 07:54:44.800 EST [1110541][startup][:0] DEBUG: proc_exit(-1): 0 callbacks to make 2026-11-14 07:54:44.800 EST [1109746][postmaster][:0] DEBUG: reaping dead processes 2026-11-14 07:54:44.800 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.800 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 1110541, status: 0 2026-11-14 07:54:44.800 EST [1109746][postmaster][:0] DEBUG: releasing pm child slot 89 2026-11-14 07:54:44.800 EST [1109746][postmaster][:0] DEBUG: updating PMState from PM_RECOVERY to PM_RUN 2026-11-14 07:54:44.800 EST [1109746][postmaster][:0] LOG: database system is ready to accept connections 2026-11-14 07:54:44.800 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1 2026-11-14 07:54:44.800 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 0 2026-11-14 07:54:44.800 EST [1109746][postmaster][:0] DEBUG: assigned pm child slot 92 for walwriter 2026-11-14 07:54:44.801 EST [1109746][postmaster][:0] DEBUG: assigned pm child slot 45 for background worker 2026-11-14 07:54:44.801 EST [1109746][postmaster][:0] DEBUG: starting background worker process "logical replication launcher" 2026-11-14 07:54:44.801 EST [1110704][logical replication launcher][15/0:0] DEBUG: logical replication launcher started 2026-11-14 07:54:44.801 EST [1110704][logical replication launcher][15/0:0] DEBUG: InitPostgres 2026-11-14 07:54:44.884 EST [1109746][postmaster][:0] DEBUG: assigned pm child slot 1 for client backend 2026-11-14 07:54:44.884 EST [1109746][postmaster][:0] DEBUG: forked new client backend, pid=1110770 socket=8 2026-11-14 07:54:44.884 EST [1110770][client backend][:0] LOG: connection received: host=[local] 2026-11-14 07:54:44.884 EST [1110770][client backend][0/0:0] DEBUG: InitPostgres ...
^ permalink raw reply [nested|flat] 12+ messages in thread

* Re: 041_checkpoint_at_promote.pl might fail due to race condition on child kill
@ 2026-11-16 05:26  Ayush Tiwari <ayushtiwari.slg01@gmail.com>
  parent: Alexander Lakhin <exclusion@gmail.com>
  0 siblings, 1 reply; 12+ messages in thread

From: Ayush Tiwari @ 2026-11-16 05:26 UTC (permalink / raw)
  To: Alexander Lakhin <exclusion@gmail.com>; +Cc: Fujii Masao <masao.fujii@gmail.com>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers

Hi,

On Sat, 14 Nov 2026 at 19:30, Alexander Lakhin <exclusion@gmail.com> wrote:
>
> Hello Fujii-san and Ayush,
>
> 30.10.2026 13:55, Fujii Masao wrote:
> > On Fri, Oct 30, 2026 at 3:59 PM Ayush Tiwari
> > <ayushtiwari.slg01@gmail.com> wrote:
> >> Did multiple runs with it, and the changes lgtm.
> > I've pushed the patch. Thanks!
>
> Please look at a new failure of 041_checkpoint_at_promote produced by
> tamandua recently [1]:
> [03:10:10.869](0.025s) ok 4 - psql query died successfully after SIGKILL
> [03:13:15.804](184.935s) # die: timed out waiting for file
>
/home/bf/bf-build/tamandua/REL_19_STABLE/pgsql.build/testrun/recovery/041_checkpoint_at_promote/log/041_checkpoint_at_promote_standby1.log
> contents to match: (?^:all server processes terminated; reinitializing) at
> /home/bf/bf-build/tamandua/REL_19_STABLE/pgsql/src/test/recovery/t/
041_checkpoint_at_promote.pl line 166.
> [03:13:15.806](0.003s) 1..4
> timed out waiting for file
>
/home/bf/bf-build/tamandua/REL_19_STABLE/pgsql.build/testrun/recovery/041_checkpoint_at_promote/log/041_checkpoint_at_promote_standby1.log
> contents to match: (?^:all server processes terminated; reinitializing) at
> /home/bf/bf-build/tamandua/REL_19_STABLE/pgsql/src/test/recovery/t/
041_checkpoint_at_promote.pl line 166.
> # No postmaster PID for node "master"
>
> 041_checkpoint_at_promote_standby1.log contains:
> 2026-11-12 03:10:10.826 CET [3578531][client backend][6/2:0] LOG:
statement: SELECT pg_backend_pid();
> 2026-11-12 03:10:10.861 CET [3577203][postmaster][:0] LOG:  client
backend (PID 3578531) was terminated by signal 9: Killed
> 2026-11-12 03:10:10.861 CET [3577203][postmaster][:0] DETAIL: Failed
process was running: SELECT pg_backend_pid();
> 2026-11-12 03:10:10.862 CET [3577203][postmaster][:0] LOG: terminating
any other active server processes
> 2026-11-12 03:10:10.928 CET [3577203][postmaster][:0] LOG:  database
system is ready to accept connections
>
> There is no "terminating any other active server processes" message
> indeed, the server was not shut down.
>
> I managed to reproduce this locally (with the patch attached and with the
> same parallel loop as before) and discovered that this can happen when
> PMState goes from PM_HOT_STANDBY to PM_RUN (via PM_WAIT_BACKENDS) instead
> of PM_HOT_STANDBY -> PM_WAIT_BACKENDS -> PM_WAIT_DEAD_END ->
> PM_NO_CHILDREN -> "all server processes terminated; reinitializing".
>
> This can be explained by two race conditions, with one precondition: the
> startup process doesn't exit before pg_ctl kill.
> 1) The startup process becomes zombie for the postmaster if it exits when
> waitpid() doesn't wait.
> ...
> 2026-11-14 13:29:02.900 UTC [695404][startup][:0] DEBUG: proc_exit(-1): 0
callbacks to make
> ...
> 2026-11-14 13:29:03.088 UTC [695367][postmaster][:0] DEBUG:  sending
signal 3/SIGQUIT to startup process with pid 695404
> 2026-11-14 13:29:03.100 UTC [695367][postmaster][:0] LOG:
!!!signal_child| delayed signal 3/SIGQUIT to startup process
> with pid 695404
> ...
> 2026-11-14 13:29:03.170 UTC [695367][postmaster][:0] DEBUG:  startup
process 695404 is still running
> ...
> 2026-11-14 13:29:03.198 UTC [695367][postmaster][:0] DEBUG: updating
PMState from PM_WAIT_BACKENDS to PM_RUN
>
>
> 2) SIGQUIT sent to the startup process when it's being executed inside
> exit(0), after the point when signal handlers of a process are not called
> anymore, so SIGQUIT is ignored:
> ...
> 2026-11-14 07:54:44.900 EST [1110043][startup][:0] LOG: !!!proc_exit|
delayed exit(0) for startup process
> ...
> 2026-11-14 07:54:44.900 EST [1110043][startup][:0] DEBUG: proc_exit(-1):
0 callbacks to make
> 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] LOG:
!!!signal_child| delayed signal 3/SIGQUIT to startup process
> with pid 1110043
> ...
> 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: startup
process 1110043 is still running
> ...
> 2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] LOG: !!!waitpid|
ret: 1110043, status: 0
> 2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] DEBUG: releasing pm
child slot 89
> 2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] DEBUG: updating
PMState from PM_WAIT_BACKENDS to PM_RUN
>
> Corresponding logs for these 2 cases and also a normal run's log are
> attached for reference.

Thanks, I reproduced this with your debugging patch. Looks like a
postmaster bug rather than a problem with the test. Let me try to
write here what I understood.

Normally, startup exits with status 0 after promotion, and the postmaster
takes that as the signal to enter PM_RUN. A backend crash needs the
opposite, stop the other children and rebuild shared memory before
letting the server run again.

Here those two things happen at almost the same time:

1) Startup finishes promotion and is exiting, but the postmaster hasn't
   collected its exit status yet.

2) The test kills a backend. The postmaster notices that death first,
   sets FatalError and sends SIGQUIT to the remaining children. It recrds
   that it told startup to quit by setting STARTUP_SIGNALED.

3) SIGQUIT is too late for startup, it has already exited, or is too far
   into exit(0) to handle it? Its exit status is basically still 0.

4) The postmaster then collects that 0 and takes the normal "startup
   succeeded" path. It clears FatalError and enters PM_RUN, skipping the
   shared-memory reset needed for the backend crash.

That's why the test never sees "all server processes terminated;
reinitializing".

I think we should check STARTUP_SIGNALED even when the exit status is 0.
Something like [2]. With that diff, I tried your patch and reran the tests
multiple
times and it all passed.

Regards,
Ayush

> [1]
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=tamandua&dt=2026-11-12%2002%3A04%3A37
[2]

diff --git a/src/backend/postmaster/postmaster.c
b/src/backend/postmaster/postmaster.c
index f53aa9c404d..a2e52b61902 100644
inline patch 1: src/backend/postmaster/postmaster.c (+6/-5 lines) download inline-1.patch--- a/src/backend/postmaster/postmaster.c +++ b/src/backend/postmaster/postmaster.c @@ -2317,12 +2317,13 @@ process_pm_child_exit(void) * during PM_STARTUP due to some dead-end child crashing: in that * situation, if the startup process dies on the SIGQUIT, we need * to transition to PM_WAIT_BACKENDS state which will allow - * PostmasterStateMachine to restart the startup process. (On the - * other hand, the startup process might complete normally, if we - * were too late with the SIGQUIT. In that case we'll fall - * through and commence normal operations.) + * PostmasterStateMachine to restart the startup process. + * + * SIGQUIT may arrive too late to change the startup process's exit + * status. If we signaled it, continue the crash restart even if it + * exits normally. */ - if (!EXIT_STATUS_0(exitstatus)) + if (!EXIT_STATUS_0(exitstatus) || StartupStatus == STARTUP_SIGNALED) { if (StartupStatus == STARTUP_SIGNALED) {
^ permalink raw reply [nested|flat] 12+ messages in thread

* Re: 041_checkpoint_at_promote.pl might fail due to race condition on child kill
@ 2026-11-16 06:19  Fujii Masao <masao.fujii@gmail.com>
  parent: Ayush Tiwari <ayushtiwari.slg01@gmail.com>
  0 siblings, 1 reply; 12+ messages in thread

From: Fujii Masao @ 2026-11-16 06:19 UTC (permalink / raw)
  To: Ayush Tiwari <ayushtiwari.slg01@gmail.com>; +Cc: Alexander Lakhin <exclusion@gmail.com>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers

On Mon, Nov 16, 2026 at 2:26 PM Ayush Tiwari
<ayushtiwari.slg01@gmail.com> wrote:
> I think we should check STARTUP_SIGNALED even when the exit status is 0.
> Something like [2].

LGTM, thanks!

This should be backpatched to all supported versions?

Regards,

-- 
Fujii Masao






^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* Re: 041_checkpoint_at_promote.pl might fail due to race condition on child kill
@ 2026-11-16 06:50  Ayush Tiwari <ayushtiwari.slg01@gmail.com>
  parent: Fujii Masao <masao.fujii@gmail.com>
  0 siblings, 1 reply; 12+ messages in thread

From: Ayush Tiwari @ 2026-11-16 06:50 UTC (permalink / raw)
  To: Fujii Masao <masao.fujii@gmail.com>; +Cc: Alexander Lakhin <exclusion@gmail.com>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers

Hi,

On Mon, 16 Nov 2026 at 11:49, Fujii Masao <masao.fujii@gmail.com> wrote:
>
> On Mon, Nov 16, 2026 at 2:26 PM Ayush Tiwari
> <ayushtiwari.slg01@gmail.com> wrote:
> > I think we should check STARTUP_SIGNALED even when the exit status is 0.
> > Something like [2].
>
> LGTM, thanks!

Thanks for the review, Fujii-san!

> This should be backpatched to all supported versions?

Yes, back till 15.

Regards,
Ayush






^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* Re: 041_checkpoint_at_promote.pl might fail due to race condition on child kill
@ 2026-11-18 04:44  Fujii Masao <masao.fujii@gmail.com>
  parent: Ayush Tiwari <ayushtiwari.slg01@gmail.com>
  0 siblings, 0 replies; 12+ messages in thread

From: Fujii Masao @ 2026-11-18 04:44 UTC (permalink / raw)
  To: Ayush Tiwari <ayushtiwari.slg01@gmail.com>; +Cc: Alexander Lakhin <exclusion@gmail.com>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers

On Mon, Nov 16, 2026 at 3:50 PM Ayush Tiwari
<ayushtiwari.slg01@gmail.com> wrote:
> Yes, back till 15.

I've pushed the patch. Thanks!

Regards,

-- 
Fujii Masao






^ permalink  raw  reply  [nested|flat] 12+ messages in thread


end of thread, other threads:[~2026-11-18 04:44 UTC | newest]

Thread overview: 12+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-10-07 04:00 041_checkpoint_at_promote.pl might fail due to race condition on child kill Alexander Lakhin <exclusion@gmail.com>
2026-10-07 16:46 ` Fujii Masao <masao.fujii@gmail.com>
2026-10-08 19:00   ` Alexander Lakhin <exclusion@gmail.com>
2026-10-13 07:17     ` Fujii Masao <masao.fujii@gmail.com>
2026-10-29 12:37       ` Fujii Masao <masao.fujii@gmail.com>
2026-10-30 06:58       ` Ayush Tiwari <ayushtiwari.slg01@gmail.com>
2026-10-30 11:55         ` Fujii Masao <masao.fujii@gmail.com>
2026-11-14 14:00           ` Alexander Lakhin <exclusion@gmail.com>
2026-11-16 05:26             ` Ayush Tiwari <ayushtiwari.slg01@gmail.com>
2026-11-16 06:19               ` Fujii Masao <masao.fujii@gmail.com>
2026-11-16 06:50                 ` Ayush Tiwari <ayushtiwari.slg01@gmail.com>
2026-11-18 04:44                   ` Fujii Masao <masao.fujii@gmail.com>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox