pg.ddx.io pgsql-hackers@postgresql.org mailing list archive
help / color / mirror / Atom feed041_checkpoint_at_promote.pl might fail due to race condition on child kill
12+ messages / 3 participants
[nested] [flat]
Commits from this thread:
2026-11-18 83cefb7b73 63df88f3ab 504bd2e8da 2bce773e2e c9c98c5f8c e21e8164db Fix handling of backend crashes during standby promotion
2026-10-30 eda9d98fac 6621f2b062 7ba657cba1 abab508718 Fix crash-restart race in 041_checkpoint_at_promote.pl
* 041_checkpoint_at_promote.pl might fail due to race condition on child kill
@ 2026-10-07 04:00 Alexander Lakhin <exclusion@gmail.com>
0 siblings, 1 reply; 12+ messages in thread
From: Alexander Lakhin @ 2026-10-07 04:00 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; pgsql-hackers
Hello hackers,
Please look at a new failure of 041_checkpoint_at_promote.pl produced by
bushmaster, [1]:
[23:56:13.841](0.002s) ok 4 - psql query died successfully after SIGKILL
[23:56:13.902](0.061s) not ok 5 - psql connect success
[23:56:13.902](0.000s)
[23:56:13.902](0.000s) # Failed test 'psql connect success'
# at t/041_checkpoint_at_promote.pl line 168.
[23:56:13.902](0.000s) # got: '2'
# expected: '0'
[23:56:13.902](0.000s) not ok 6 - psql select 1
[23:56:13.902](0.000s)
[23:56:13.902](0.000s) # Failed test 'psql select 1'
# at t/041_checkpoint_at_promote.pl line 169.
[23:56:13.902](0.000s) # got: ''
# expected: '1'
[23:56:13.902](0.000s) 1..6
pgsql.build/src/test/recovery/tmp_check/log/041_checkpoint_at_promote_standby1.log
2026-10-03 23:56:13.852 CEST [4159149][postmaster][:0] LOG: client backend (PID 4159398) was terminated by signal 9: Killed
2026-10-03 23:56:13.852 CEST [4159149][postmaster][:0] DETAIL: Failed process was running: SELECT pg_backend_pid();
2026-10-03 23:56:13.852 CEST [4159149][postmaster][:0] LOG: terminating any other active server processes
2026-10-03 23:56:13.855 CEST [4159149][postmaster][:0] LOG: all server processes terminated; reinitializing
2026-10-03 23:56:13.899 CEST [4159414][startup][:0] LOG: database system was interrupted; last known up at 2026-10-03
23:56:13 CEST
2026-10-03 23:56:13.899 CEST [4159414][startup][:0] LOG: database system was not properly shut down; automatic recovery
in progress
2026-10-03 23:56:13.899 CEST [4159414][startup][:0] LOG: crash recovery starts in timeline 1 and has target timeline 2
2026-10-03 23:56:13.900 CEST [4159414][startup][:0] LOG: redo starts at 0/3000028
2026-10-03 23:56:13.900 CEST [4159417][not initialized][:0] LOG: connection received: host=[local]
2026-10-03 23:56:13.900 CEST [4159417][client backend][:0] FATAL: the database system is in recovery mode
I've reproduced this failure locally, running multiple tests concurrently inside a 4-core VM, in a loop:
for i in {1..50}; do echo "Iteration $i"; parallel -j40 --linebuffer --tag PROVE_TESTS="t/041*" NO_TEMP_INSTALL=1
TEMP_CONFIG=~/extra.config make -s check -s -C src/test/recovery_{} PROVE_FLAGS="--timer" ::: `seq 25` || break; done;
(with bushmaster's parameters put in ~/extra.config)
This failed for me on iterations 14, 30, 10:
...
Iteration 10
...
10
10 # Failed test 'psql connect success'
10 # at t/041_checkpoint_at_promote.pl line 168.
10 # got: '2'
10 # expected: '0'
10
10 # Failed test 'psql select 1'
10 # at t/041_checkpoint_at_promote.pl line 169.
10 # got: ''
10 # expected: '1'
10 # Looks like you failed 2 tests of 6.
10 [03:51:18] t/041_checkpoint_at_promote.pl ..
10 Dubious, test returned 2 (wstat 512, 0x200)
10 Failed 2/6 subtests
10 [03:51:18]
10
10 Test Summary Report
10 -------------------
10 t/041_checkpoint_at_promote.pl (Wstat: 512 (exited 2) Tests: 6 Failed: 2)
10 Failed tests: 5-6
10 Non-zero exit status: 2
It looks like the same race condition was fixed in 022_crash_temp_files.pl
by e37ad5fa4 two years before 041_checkpoint_at_promote.pl appeared: [2].
[1] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=bushmaster&dt=2026-10-03%2021%3A37%3A08
[2] https://postgr.es/m/1801850.1649047827@sss.pgh.pa.us
Best regards,
Alexander
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: 041_checkpoint_at_promote.pl might fail due to race condition on child kill
@ 2026-10-07 16:46 Fujii Masao <masao.fujii@gmail.com>
parent: Alexander Lakhin <exclusion@gmail.com>
0 siblings, 1 reply; 12+ messages in thread
From: Fujii Masao @ 2026-10-07 16:46 UTC (permalink / raw)
To: Alexander Lakhin <exclusion@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; pgsql-hackers
On Wed, Oct 7, 2026 at 1:00 PM Alexander Lakhin <exclusion@gmail.com> wrote:
> It looks like the same race condition was fixed in 022_crash_temp_files.pl
> by e37ad5fa4 two years before 041_checkpoint_at_promote.pl appeared: [2].
I tried creating the patch based on that fix. How about the attached patch?
Regards,
--
Fujii Masao
Attachments:
v1-0001-Fix-crash-restart-race-in-041_checkpoint_at_promo.patch (application/octet-stream, 2.6K) download
From 9f6392513a307226ca75c12bf173bcf552bc6aa6 Mon Sep 17 00:00:00 2001
From: Fujii Masao <fujii@postgresql.org>
Date: Wed, 7 Oct 2026 21:31:47 +0900
Subject: [PATCH v1] Fix crash restart race in 041_checkpoint_at_promote.pl
---
.../recovery/t/041_checkpoint_at_promote.pl | 44 +++++++++++++++++--
1 file changed, 41 insertions(+), 3 deletions(-)
diff --git a/src/test/recovery/t/041_checkpoint_at_promote.pl b/src/test/recovery/t/041_checkpoint_at_promote.pl
index d0783fef9ae..8c9e65502dc 100644
--- a/src/test/recovery/t/041_checkpoint_at_promote.pl
+++ b/src/test/recovery/t/041_checkpoint_at_promote.pl
@@ -142,10 +142,33 @@ chomp($pid);
$killme_stdout = '';
$killme_stderr = '';
+# Keep another backend connected so we can tell when the postmaster has
+# noticed the crashed backend and begun restarting the server.
+my ($monitor_stdin, $monitor_stdout, $monitor_stderr) = ('', '', '');
+my $monitor = IPC::Run::start(
+ [
+ 'psql', '--no-psqlrc', '--quiet', '--no-align', '--tuples-only',
+ '--set' => 'ON_ERROR_STOP=1',
+ '--file' => '-',
+ '--dbname' => $node_standby->connstr('postgres')
+ ],
+ '<' => \$monitor_stdin,
+ '>' => \$monitor_stdout,
+ '2>' => \$monitor_stderr,
+ $psql_timeout);
+$monitor_stdin .= q[
+SELECT $$psql-connected$$;
+];
+ok( pump_until(
+ $monitor, $psql_timeout, \$monitor_stdout, qr/psql-connected/m),
+ 'monitor connected');
+$monitor_stdout = '';
+$monitor_stderr = '';
+
my $ret = PostgreSQL::Test::Utils::system_log('pg_ctl', 'kill', 'KILL', $pid);
is($ret, 0, 'killed process with KILL');
-# Wait until the server restarts, finish consuming output.
+# Finish consuming output from the killed backend.
$killme_stdin .= q[
SELECT 1;
];
@@ -158,8 +181,23 @@ ok( pump_until(
"psql query died successfully after SIGKILL");
$killme->finish;
-# Wait till server finishes restarting.
-$node_standby->poll_query_until('postgres', undef, '');
+# Wait until the monitor backend is terminated by the postmaster. The killed
+# backend's own connection can fail before the postmaster begins restarting.
+$monitor_stdin .= qq[
+SELECT pg_sleep($PostgreSQL::Test::Utils::timeout_default);
+];
+ok( pump_until(
+ $monitor,
+ $psql_timeout,
+ \$monitor_stderr,
+ qr/WARNING: terminating connection because of crash of another server process|server closed the connection unexpectedly|connection to server was lost|could not send data to server/m
+ ),
+ "psql monitor died successfully after SIGKILL");
+$monitor->finish;
+
+# Now wait till server finishes restarting.
+is($node_standby->poll_query_until('postgres', undef, ''),
+ "1", "reconnected after SIGKILL");
# After recovery, the server should be able to start.
my $stdout;
--
2.55.0
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: 041_checkpoint_at_promote.pl might fail due to race condition on child kill
@ 2026-10-08 19:00 Alexander Lakhin <exclusion@gmail.com>
parent: Fujii Masao <masao.fujii@gmail.com>
0 siblings, 1 reply; 12+ messages in thread
From: Alexander Lakhin @ 2026-10-08 19:00 UTC (permalink / raw)
To: Fujii Masao <masao.fujii@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; pgsql-hackers
Hello Fujii-san,
07.10.2026 19:46, Fujii Masao wrote:
> On Wed, Oct 7, 2026 at 1:00 PM Alexander Lakhin<exclusion@gmail.com> wrote:
>> It looks like the same race condition was fixed in 022_crash_temp_files.pl
>> by e37ad5fa4 two years before 041_checkpoint_at_promote.pl appeared: [2].
> I tried creating the patch based on that fix. How about the attached patch?
Your patch works for me test-wise. The same test loop passed 100+
iterations successfully. Thank you for paying attention to this!
Best regards,
Alexander
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: 041_checkpoint_at_promote.pl might fail due to race condition on child kill
@ 2026-10-13 07:17 Fujii Masao <masao.fujii@gmail.com>
parent: Alexander Lakhin <exclusion@gmail.com>
0 siblings, 2 replies; 12+ messages in thread
From: Fujii Masao @ 2026-10-13 07:17 UTC (permalink / raw)
To: Alexander Lakhin <exclusion@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; pgsql-hackers
On Fri, Oct 9, 2026 at 4:00 AM Alexander Lakhin <exclusion@gmail.com> wrote:
> Your patch works for me test-wise. The same test loop passed 100+
> iterations successfully. Thank you for paying attention to this!
Thanks for the test!
On second thought, we can simplify the test by waiting for the postmaster to
log "all server processes terminated; reinitializing" instead of using
another backend connection to detect the crash. This ensures that the
postmaster has detected the crash and started restarting. The attached
patch takes this approach. Thoughts?
Regards,
--
Fujii Masao
Attachments:
v2-0001-Fix-crash-restart-race-in-041_checkpoint_at_promo.patch (application/octet-stream, 3.0K) download
From 794d7b6a45d13fc92399cb27f235e48143871eae Mon Sep 17 00:00:00 2001
From: Fujii Masao <fujii@postgresql.org>
Date: Wed, 7 Oct 2026 21:31:47 +0900
Subject: [PATCH v2] Fix crash-restart race in 041_checkpoint_at_promote.pl
The buildfarm member bushmaster reported failures in the final test query
in 041_checkpoint_at_promote.pl with "the database system is in recovery
mode".
After killing a backend to trigger crash recovery, the test should wait
for the postmaster to terminate the remaining server processes and restart
the server, then wait for recovery to finish and connections to be accepted
before running the final query.
Previously, the test assumed that the killed backend's connection closing
meant the postmaster had noticed the crash and begun restarting. However,
the connection can close before the postmaster notices the crash. The
subsequent connection check can then succeed before recovery starts,
causing the test to mistake this for recovery having completed. The final
query could then run during recovery and fail.
Fix this by waiting for the postmaster to log that all server processes
have terminated and it is reinitializing before checking that connections
are accepted again. This ensures that the postmaster has noticed the crash
and begun restarting before the test waits for recovery to finish.
Backpatch to v17, where this test was introduced.
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Author: Fujii Masao <masao.fujii@gmail.com>
Discussion: https://postgr.es/m/e7abd282-655d-495e-94c9-e38a58917ebf@gmail.com
Backpatch-through: 17
---
src/test/recovery/t/041_checkpoint_at_promote.pl | 16 +++++++++++++---
1 file changed, 13 insertions(+), 3 deletions(-)
diff --git a/src/test/recovery/t/041_checkpoint_at_promote.pl b/src/test/recovery/t/041_checkpoint_at_promote.pl
index d0783fef9ae..82173b9d86e 100644
--- a/src/test/recovery/t/041_checkpoint_at_promote.pl
+++ b/src/test/recovery/t/041_checkpoint_at_promote.pl
@@ -142,10 +142,13 @@ chomp($pid);
$killme_stdout = '';
$killme_stderr = '';
+# Record the log position before triggering crash recovery.
+my $crash_logstart = -s $node_standby->logfile;
+
my $ret = PostgreSQL::Test::Utils::system_log('pg_ctl', 'kill', 'KILL', $pid);
is($ret, 0, 'killed process with KILL');
-# Wait until the server restarts, finish consuming output.
+# Finish consuming output from the killed backend.
$killme_stdin .= q[
SELECT 1;
];
@@ -158,8 +161,15 @@ ok( pump_until(
"psql query died successfully after SIGKILL");
$killme->finish;
-# Wait till server finishes restarting.
-$node_standby->poll_query_until('postgres', undef, '');
+# Wait until the postmaster has noticed the crash and begun restarting.
+# The killed backend's connection can fail before this happens.
+$node_standby->wait_for_log(
+ qr/all server processes terminated; reinitializing/,
+ $crash_logstart);
+
+# Wait until recovery finishes and connections are accepted.
+is($node_standby->poll_query_until('postgres', undef, ''),
+ 1, 'reconnected after SIGKILL');
# After recovery, the server should be able to start.
my $stdout;
--
2.56.0
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: 041_checkpoint_at_promote.pl might fail due to race condition on child kill
@ 2026-10-29 12:37 Fujii Masao <masao.fujii@gmail.com>
parent: Fujii Masao <masao.fujii@gmail.com>
1 sibling, 0 replies; 12+ messages in thread
From: Fujii Masao @ 2026-10-29 12:37 UTC (permalink / raw)
To: Alexander Lakhin <exclusion@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; pgsql-hackers
On Tue, Oct 13, 2026 at 4:17 PM Fujii Masao <masao.fujii@gmail.com> wrote:
> On second thought, we can simplify the test by waiting for the postmaster to
> log "all server processes terminated; reinitializing" instead of using
> another backend connection to detect the crash. This ensures that the
> postmaster has detected the crash and started restarting. The attached
> patch takes this approach. Thoughts?
Barring any objections, I will commit this patch.
Regards,
--
Fujii Masao
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: 041_checkpoint_at_promote.pl might fail due to race condition on child kill
@ 2026-10-30 06:58 Ayush Tiwari <ayushtiwari.slg01@gmail.com>
parent: Fujii Masao <masao.fujii@gmail.com>
1 sibling, 1 reply; 12+ messages in thread
From: Ayush Tiwari @ 2026-10-30 06:58 UTC (permalink / raw)
To: Fujii Masao <masao.fujii@gmail.com>; +Cc: Alexander Lakhin <exclusion@gmail.com>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers
Hi,
On Tue, 13 Oct 2026 at 12:47, Fujii Masao <masao.fujii@gmail.com> wrote:
>
> On Fri, Oct 9, 2026 at 4:00 AM Alexander Lakhin <exclusion@gmail.com> wrote:
> > Your patch works for me test-wise. The same test loop passed 100+
> > iterations successfully. Thank you for paying attention to this!
>
> Thanks for the test!
>
> On second thought, we can simplify the test by waiting for the postmaster to
> log "all server processes terminated; reinitializing" instead of using
> another backend connection to detect the crash. This ensures that the
> postmaster has detected the crash and started restarting. The attached
> patch takes this approach. Thoughts?
Thanks for the patch.
Did multiple runs with it, and the changes lgtm.
Regards,
Ayush
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: 041_checkpoint_at_promote.pl might fail due to race condition on child kill
@ 2026-10-30 11:55 Fujii Masao <masao.fujii@gmail.com>
parent: Ayush Tiwari <ayushtiwari.slg01@gmail.com>
0 siblings, 1 reply; 12+ messages in thread
From: Fujii Masao @ 2026-10-30 11:55 UTC (permalink / raw)
To: Ayush Tiwari <ayushtiwari.slg01@gmail.com>; +Cc: Alexander Lakhin <exclusion@gmail.com>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers
On Fri, Oct 30, 2026 at 3:59 PM Ayush Tiwari
<ayushtiwari.slg01@gmail.com> wrote:
> Did multiple runs with it, and the changes lgtm.
I've pushed the patch. Thanks!
Regards,
--
Fujii Masao
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: 041_checkpoint_at_promote.pl might fail due to race condition on child kill
@ 2026-11-14 14:00 Alexander Lakhin <exclusion@gmail.com>
parent: Fujii Masao <masao.fujii@gmail.com>
0 siblings, 1 reply; 12+ messages in thread
From: Alexander Lakhin @ 2026-11-14 14:00 UTC (permalink / raw)
To: Fujii Masao <masao.fujii@gmail.com>; Ayush Tiwari <ayushtiwari.slg01@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; pgsql-hackers
Hello Fujii-san and Ayush,
30.10.2026 13:55, Fujii Masao wrote:
> On Fri, Oct 30, 2026 at 3:59 PM Ayush Tiwari
> <ayushtiwari.slg01@gmail.com> wrote:
>> Did multiple runs with it, and the changes lgtm.
> I've pushed the patch. Thanks!
Please look at a new failure of 041_checkpoint_at_promote produced by
tamandua recently [1]:
[03:10:10.869](0.025s) ok 4 - psql query died successfully after SIGKILL
[03:13:15.804](184.935s) # die: timed out waiting for file
/home/bf/bf-build/tamandua/REL_19_STABLE/pgsql.build/testrun/recovery/041_checkpoint_at_promote/log/041_checkpoint_at_promote_standby1.log
contents to match: (?^:all server processes terminated; reinitializing) at
/home/bf/bf-build/tamandua/REL_19_STABLE/pgsql/src/test/recovery/t/041_checkpoint_at_promote.pl line 166.
[03:13:15.806](0.003s) 1..4
timed out waiting for file
/home/bf/bf-build/tamandua/REL_19_STABLE/pgsql.build/testrun/recovery/041_checkpoint_at_promote/log/041_checkpoint_at_promote_standby1.log
contents to match: (?^:all server processes terminated; reinitializing) at
/home/bf/bf-build/tamandua/REL_19_STABLE/pgsql/src/test/recovery/t/041_checkpoint_at_promote.pl line 166.
# No postmaster PID for node "master"
041_checkpoint_at_promote_standby1.log contains:
2026-11-12 03:10:10.826 CET [3578531][client backend][6/2:0] LOG: statement: SELECT pg_backend_pid();
2026-11-12 03:10:10.861 CET [3577203][postmaster][:0] LOG: client backend (PID 3578531) was terminated by signal 9: Killed
2026-11-12 03:10:10.861 CET [3577203][postmaster][:0] DETAIL: Failed process was running: SELECT pg_backend_pid();
2026-11-12 03:10:10.862 CET [3577203][postmaster][:0] LOG: terminating any other active server processes
2026-11-12 03:10:10.928 CET [3577203][postmaster][:0] LOG: database system is ready to accept connections
There is no "terminating any other active server processes" message
indeed, the server was not shut down.
I managed to reproduce this locally (with the patch attached and with the
same parallel loop as before) and discovered that this can happen when
PMState goes from PM_HOT_STANDBY to PM_RUN (via PM_WAIT_BACKENDS) instead
of PM_HOT_STANDBY -> PM_WAIT_BACKENDS -> PM_WAIT_DEAD_END ->
PM_NO_CHILDREN -> "all server processes terminated; reinitializing".
This can be explained by two race conditions, with one precondition: the
startup process doesn't exit before pg_ctl kill.
1) The startup process becomes zombie for the postmaster if it exits when
waitpid() doesn't wait.
...
2026-11-14 13:29:02.900 UTC [695404][startup][:0] DEBUG: proc_exit(-1): 0 callbacks to make
...
2026-11-14 13:29:03.088 UTC [695367][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to startup process with pid 695404
2026-11-14 13:29:03.100 UTC [695367][postmaster][:0] LOG: !!!signal_child| delayed signal 3/SIGQUIT to startup process
with pid 695404
...
2026-11-14 13:29:03.170 UTC [695367][postmaster][:0] DEBUG: startup process 695404 is still running
...
2026-11-14 13:29:03.198 UTC [695367][postmaster][:0] DEBUG: updating PMState from PM_WAIT_BACKENDS to PM_RUN
2) SIGQUIT sent to the startup process when it's being executed inside
exit(0), after the point when signal handlers of a process are not called
anymore, so SIGQUIT is ignored:
...
2026-11-14 07:54:44.900 EST [1110043][startup][:0] LOG: !!!proc_exit| delayed exit(0) for startup process
...
2026-11-14 07:54:44.900 EST [1110043][startup][:0] DEBUG: proc_exit(-1): 0 callbacks to make
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] LOG: !!!signal_child| delayed signal 3/SIGQUIT to startup process
with pid 1110043
...
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: startup process 1110043 is still running
...
2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110043, status: 0
2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 89
2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] DEBUG: updating PMState from PM_WAIT_BACKENDS to PM_RUN
Corresponding logs for these 2 cases and also a normal run's log are
attached for reference.
[1] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=tamandua&dt=2026-11-12%2002%3A04%3A37
Best regards,
Alexander
Attachments:
recovery-041-debugging.patch (text/x-patch, 2.2K) download
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index f53aa9c404d..d1db634644b 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -68,6 +68,7 @@
#include <unistd.h>
#include <signal.h>
#include <time.h>
+#include <sys/time.h>
#include <sys/wait.h>
#include <ctype.h>
#include <sys/stat.h>
@@ -2253,6 +2254,16 @@ handle_pm_child_exit_signal(SIGNAL_ARGS)
SetLatch(MyLatch);
}
+static pid_t
+_waitpid(pid_t pid, int *status, int options)
+{
+pid_t ret;
+elog(LOG, "!!!waitpid| pid: %d", pid);
+ret = waitpid(pid, status, options);
+elog(LOG, "!!!waitpid| ret: %d, status: %d", ret, *status);
+return ret;
+}
+
/*
* Cleanup after a child process dies.
*/
@@ -2267,7 +2278,7 @@ process_pm_child_exit(void)
ereport(DEBUG4,
(errmsg_internal("reaping dead processes")));
- while ((pid = waitpid(-1, &exitstatus, WNOHANG)) > 0)
+ while ((pid = _waitpid(-1, &exitstatus, WNOHANG)) > 0)
{
PMChild *pmchild;
@@ -3519,6 +3530,13 @@ signal_child(PMChild *pmchild, int signal)
GetBackendTypeDesc(pmchild->bkend_type),
(int) pmchild->pid)));
+if (pmchild == StartupPMChild && signal == SIGQUIT)
+{
+struct timeval current_time;
+do gettimeofday(¤t_time, NULL); while ((current_time.tv_usec / 1000) % 100 >= 2);
+elog(LOG, "!!!signal_child| delayed signal %d/%s to startup process with pid %d", signal, pm_signame(signal), (int) pmchild->pid);
+}
+
if (kill(pid, signal) < 0)
elog(DEBUG3, "kill(%ld,%d) failed: %m", (long) pid, signal);
#ifdef HAVE_SETSID
diff --git a/src/backend/storage/ipc/ipc.c b/src/backend/storage/ipc/ipc.c
index cb944edd8df..e96b3c2a035 100644
--- a/src/backend/storage/ipc/ipc.c
+++ b/src/backend/storage/ipc/ipc.c
@@ -22,6 +22,7 @@
#include <signal.h>
#include <unistd.h>
#include <sys/stat.h>
+#include <sys/time.h>
#include "miscadmin.h"
#ifdef PROFILE_PID_DIR
@@ -154,6 +155,13 @@ proc_exit(int code)
elog(DEBUG3, "exit(%d)", code);
+if (AmStartupProcess() && code == 0)
+{
+struct timeval current_time;
+pg_usleep(250000);
+do gettimeofday(¤t_time, NULL); while ((current_time.tv_usec / 1000) % 100 >= 2);
+elog(LOG, "!!!proc_exit| delayed exit(0) for startup process");
+}
exit(code);
}
041_checkpoint_at_promote_standby1-2.log (text/x-log, 43.4K) download
...
2026-11-14 07:54:44.636 EST [1110043][startup][37/0:0] LOG: received promote request
2026-11-14 07:54:44.636 EST [1110043][startup][37/0:0] LOG: redo done at 0/04000028 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.58 s
2026-11-14 07:54:44.636 EST [1110043][startup][37/0:0] LOG: last completed transaction was at log time 2026-11-14 07:54:44.226403-05
2026-11-14 07:54:44.636 EST [1110043][startup][37/0:0] DEBUG: resetting unlogged relations: cleanup 0 init 1
2026-11-14 07:54:44.637 EST [1110043][startup][37/0:0] LOG: selected new timeline ID: 2
2026-11-14 07:54:44.637 EST [1110043][startup][37/0:0] DEBUG: updated min recovery point to 0/040000B0 on timeline 1
2026-11-14 07:54:44.648 EST [1110043][startup][37/0:0] DEBUG: could not remove file "pg_wal/000000020000000000000004": No such file or directory
2026-11-14 07:54:44.648 EST [1110043][startup][37/0:0] LOG: archive recovery complete
2026-11-14 07:54:44.648 EST [1110043][startup][37/0:0] DEBUG: MultiXactId wrap limit is 2147483648, limited by database with OID 1
2026-11-14 07:54:44.648 EST [1110043][startup][37/0:0] DEBUG: attempting to remove WAL segments newer than log file 000000020000000000000004
2026-11-14 07:54:44.648 EST [1110043][startup][37/0:0] DEBUG: removing all KnownAssignedXids
2026-11-14 07:54:44.648 EST [1110043][startup][37/0:0] DEBUG: release all standby locks
2026-11-14 07:54:44.648 EST [1110043][startup][37/0:0] DEBUG: shmem_exit(0): 5 before_shmem_exit callbacks to make
2026-11-14 07:54:44.648 EST [1110043][startup][37/0:0] DEBUG: shmem_exit(0): 8 on_shmem_exit callbacks to make
2026-11-14 07:54:44.648 EST [1110043][startup][:0] DEBUG: proc_exit(0): 1 callbacks to make
2026-11-14 07:54:44.648 EST [1110043][startup][:0] DEBUG: exit(0)
2026-11-14 07:54:44.738 EST [1110037][postmaster][:0] DEBUG: assigned pm child slot 2 for client backend
2026-11-14 07:54:44.738 EST [1110037][postmaster][:0] DEBUG: forked new client backend, pid=1110667 socket=8
2026-11-14 07:54:44.738 EST [1110667][client backend][:0] LOG: connection received: host=[local]
2026-11-14 07:54:44.738 EST [1110667][client backend][5/0:0] DEBUG: InitPostgres
2026-11-14 07:54:44.739 EST [1110667][client backend][5/1:0] LOG: connection authenticated: ...
2026-11-14 07:54:44.739 EST [1110667][client backend][5/1:0] LOG: connection authorized: ...
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] LOG: statement: SELECT injection_points_wakeup('create-restart-point');
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8)
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0)
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192)
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1)
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: find_in_path: trying ".../tmp_install/usr/local/pgsql/lib/injection_points"
2026-11-14 07:54:44.740 EST [1110667][client backend][5/2:0] DEBUG: find_in_path: trying ".../tmp_install/usr/local/pgsql/lib/injection_points.so"
2026-11-14 07:54:44.741 EST [1110667][client backend][5/0:0] DEBUG: shmem_exit(0): 5 before_shmem_exit callbacks to make
2026-11-14 07:54:44.741 EST [1110667][client backend][5/0:0] DEBUG: shmem_exit(0): 7 on_shmem_exit callbacks to make
2026-11-14 07:54:44.741 EST [1110667][client backend][:0] DEBUG: proc_exit(0): 3 callbacks to make
2026-11-14 07:54:44.741 EST [1110667][client backend][:0] LOG: disconnection: session time: 0:00:00.002 ...
2026-11-14 07:54:44.741 EST [1110667][client backend][:0] DEBUG: exit(0)
2026-11-14 07:54:44.741 EST [1110667][client backend][:0] DEBUG: shmem_exit(-1): 0 before_shmem_exit callbacks to make
2026-11-14 07:54:44.741 EST [1110667][client backend][:0] DEBUG: shmem_exit(-1): 0 on_shmem_exit callbacks to make
2026-11-14 07:54:44.741 EST [1110667][client backend][:0] DEBUG: proc_exit(-1): 0 callbacks to make
2026-11-14 07:54:44.741 EST [1110037][postmaster][:0] DEBUG: reaping dead processes
2026-11-14 07:54:44.741 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:54:44.741 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110667, status: 0
2026-11-14 07:54:44.741 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 2
2026-11-14 07:54:44.741 EST [1110037][postmaster][:0] DEBUG: client backend (PID 1110667) exited with exit code 0
2026-11-14 07:54:44.741 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:54:44.741 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 0
2026-11-14 07:54:44.749 EST [1110041][checkpointer][:0] DEBUG: attempting to remove WAL segments older than log file 000000000000000000000002
2026-11-14 07:54:44.749 EST [1110041][checkpointer][:0] DEBUG: recycled write-ahead log file "000000010000000000000002"
2026-11-14 07:54:44.749 EST [1110041][checkpointer][:0] DEBUG: SlruScanDirectory invoking callback on pg_subtrans/0000
2026-11-14 07:54:44.749 EST [1110041][checkpointer][:0] LOG: restartpoint complete: fast wait: wrote 38 buffers (29.7%), wrote 2 SLRU buffers; 0 WAL file(s) added, 0 removed, 1 recycled; write=0.001 s, sync=0.001 s, total=0.528 s; sync files=0, longest=0.000 s, average=0.000 s; distance=16384 kB, estimate=16384 kB; lsn=0/03000088, redo lsn=0/03000028
2026-11-14 07:54:44.749 EST [1110041][checkpointer][:0] LOG: recovery restart point at 0/03000028
2026-11-14 07:54:44.749 EST [1110041][checkpointer][:0] DETAIL: Last completed transaction was at log time 2026-11-14 07:54:44.226403-05.
2026-11-14 07:54:44.749 EST [1110041][checkpointer][:0] LOG: checkpoint starting: force
2026-11-14 07:54:44.843 EST [1110037][postmaster][:0] DEBUG: assigned pm child slot 2 for client backend
2026-11-14 07:54:44.843 EST [1110037][postmaster][:0] DEBUG: forked new client backend, pid=1110740 socket=8
2026-11-14 07:54:44.843 EST [1110740][client backend][:0] LOG: connection received: host=[local]
2026-11-14 07:54:44.843 EST [1110740][client backend][6/0:0] DEBUG: InitPostgres
2026-11-14 07:54:44.844 EST [1110740][client backend][6/1:0] LOG: connection authenticated: ...
2026-11-14 07:54:44.844 EST [1110740][client backend][6/1:0] LOG: connection authorized: ...
2026-11-14 07:54:44.844 EST [1110740][client backend][6/2:0] LOG: statement: SELECT pg_backend_pid();
2026-11-14 07:54:44.844 EST [1110740][client backend][6/2:0] DEBUG: io 6 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8
2026-11-14 07:54:44.844 EST [1110740][client backend][6/2:0] DEBUG: io 6 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8
2026-11-14 07:54:44.844 EST [1110740][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8
2026-11-14 07:54:44.844 EST [1110740][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8
2026-11-14 07:54:44.844 EST [1110740][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8
2026-11-14 07:54:44.844 EST [1110740][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8
2026-11-14 07:54:44.844 EST [1110740][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8
2026-11-14 07:54:44.844 EST [1110740][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8
2026-11-14 07:54:44.844 EST [1110740][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8
2026-11-14 07:54:44.844 EST [1110740][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8
2026-11-14 07:54:44.845 EST [1110037][postmaster][:0] DEBUG: reaping dead processes
2026-11-14 07:54:44.845 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:54:44.845 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110740, status: 9
2026-11-14 07:54:44.845 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 2
2026-11-14 07:54:44.845 EST [1110037][postmaster][:0] LOG: client backend (PID 1110740) was terminated by signal 9: Killed
2026-11-14 07:54:44.845 EST [1110037][postmaster][:0] DETAIL: Failed process was running: SELECT pg_backend_pid();
2026-11-14 07:54:44.845 EST [1110037][postmaster][:0] LOG: terminating any other active server processes
2026-11-14 07:54:44.845 EST [1110037][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to client backend process with pid 1110383
2026-11-14 07:54:44.845 EST [1110037][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to startup process with pid 1110043
2026-11-14 07:54:44.849 EST [1110041][checkpointer][:0] DEBUG: performing replication slot checkpoint
2026-11-14 07:54:44.849 EST [1110041][checkpointer][:0] DEBUG: snapshot of 0+0 running transaction ids (lsn 0/04000148 oldest xid 667 latest complete 666 next xid 667)
2026-11-14 07:54:44.849 EST [1110041][checkpointer][:0] DEBUG: attempting to remove WAL segments older than log file 000000000000000000000003
2026-11-14 07:54:44.849 EST [1110041][checkpointer][:0] DEBUG: recycled write-ahead log file "000000010000000000000003"
2026-11-14 07:54:44.849 EST [1110041][checkpointer][:0] DEBUG: SlruScanDirectory invoking callback on pg_subtrans/0000
2026-11-14 07:54:44.849 EST [1110041][checkpointer][:0] LOG: checkpoint complete: force: wrote 1 buffers (0.8%), wrote 2 SLRU buffers; 0 WAL file(s) added, 0 removed, 1 recycled; write=0.101 s, sync=0.001 s, total=0.101 s; sync files=0, longest=0.000 s, average=0.000 s; distance=16384 kB, estimate=16384 kB; lsn=0/04000148, redo lsn=0/040000E8
2026-11-14 07:54:44.900 EST [1110043][startup][:0] LOG: !!!proc_exit| delayed exit(0) for startup process
2026-11-14 07:54:44.900 EST [1110043][startup][:0] DEBUG: shmem_exit(-1): 0 before_shmem_exit callbacks to make
2026-11-14 07:54:44.900 EST [1110043][startup][:0] DEBUG: shmem_exit(-1): 0 on_shmem_exit callbacks to make
2026-11-14 07:54:44.900 EST [1110043][startup][:0] DEBUG: proc_exit(-1): 0 callbacks to make
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] LOG: !!!signal_child| delayed signal 3/SIGQUIT to startup process with pid 1110043
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to background writer process with pid 1110042
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to checkpointer process with pid 1110041
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to io worker process with pid 1110040
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to io worker process with pid 1110039
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: updating PMState from PM_HOT_STANDBY to PM_WAIT_BACKENDS
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110383, status: 512
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 1
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 512
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: startup process 1110043 is still running
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: background writer process 1110042 is still running
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: checkpointer process 1110041 is still running
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: io worker process 1110040 is still running
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: io worker process 1110039 is still running
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: reaping dead processes
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 43690
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: startup process 1110043 is still running
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: background writer process 1110042 is still running
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: checkpointer process 1110041 is still running
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: io worker process 1110040 is still running
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: io worker process 1110039 is still running
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: reaping dead processes
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110040, status: 512
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 58
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 512
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: startup process 1110043 is still running
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: background writer process 1110042 is still running
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: checkpointer process 1110041 is still running
2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: io worker process 1110039 is still running
2026-11-14 07:54:44.902 EST [1110037][postmaster][:0] DEBUG: reaping dead processes
2026-11-14 07:54:44.902 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:54:44.902 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110039, status: 512
2026-11-14 07:54:44.902 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 57
2026-11-14 07:54:44.902 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:54:44.902 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110041, status: 512
2026-11-14 07:54:44.902 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 56
2026-11-14 07:54:44.902 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:54:44.902 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110042, status: 512
2026-11-14 07:54:44.902 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 55
2026-11-14 07:54:44.902 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:54:44.902 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 512
2026-11-14 07:54:44.902 EST [1110037][postmaster][:0] DEBUG: startup process 1110043 is still running
2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] DEBUG: reaping dead processes
2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110043, status: 0
2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 89
2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] DEBUG: updating PMState from PM_WAIT_BACKENDS to PM_RUN
2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] LOG: database system is ready to accept connections
2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: -1, status: 0
2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] DEBUG: assigned pm child slot 57 for io worker
2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] DEBUG: assigned pm child slot 58 for io worker
2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] DEBUG: assigned pm child slot 56 for checkpointer
2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] DEBUG: assigned pm child slot 55 for background writer
2026-11-14 07:54:44.904 EST [1110037][postmaster][:0] DEBUG: assigned pm child slot 92 for walwriter
2026-11-14 07:54:44.904 EST [1110037][postmaster][:0] DEBUG: assigned pm child slot 45 for background worker
2026-11-14 07:54:44.904 EST [1110037][postmaster][:0] DEBUG: starting background worker process "logical replication launcher"
2026-11-14 07:54:44.904 EST [1110788][checkpointer][:0] DEBUG: checkpointer updated shared memory configuration values
2026-11-14 07:54:44.904 EST [1110793][logical replication launcher][15/0:0] DEBUG: logical replication launcher started
2026-11-14 07:54:44.904 EST [1110793][logical replication launcher][15/0:0] DEBUG: InitPostgres
2026-11-14 07:55:00.050 EST [1110790][background writer][:0] DEBUG: snapshot of 0+0 running transaction ids (lsn 0/04000208 oldest xid 667 latest complete 666 next xid 667)
2026-11-14 07:55:44.967 EST [1110786][io worker][:0] DEBUG: shmem_exit(0): 4 before_shmem_exit callbacks to make
2026-11-14 07:55:44.967 EST [1110786][io worker][:0] DEBUG: shmem_exit(0): 7 on_shmem_exit callbacks to make
2026-11-14 07:55:44.967 EST [1110786][io worker][:0] DEBUG: proc_exit(0): 1 callbacks to make
2026-11-14 07:55:44.967 EST [1110786][io worker][:0] DEBUG: exit(0)
2026-11-14 07:55:44.967 EST [1110786][io worker][:0] DEBUG: shmem_exit(-1): 0 before_shmem_exit callbacks to make
2026-11-14 07:55:44.967 EST [1110786][io worker][:0] DEBUG: shmem_exit(-1): 0 on_shmem_exit callbacks to make
2026-11-14 07:55:44.967 EST [1110786][io worker][:0] DEBUG: proc_exit(-1): 0 callbacks to make
2026-11-14 07:55:44.968 EST [1110037][postmaster][:0] DEBUG: reaping dead processes
2026-11-14 07:55:44.968 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:55:44.968 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110786, status: 0
2026-11-14 07:55:44.968 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 58
2026-11-14 07:55:44.968 EST [1110037][postmaster][:0] DEBUG: assigned pm child slot 58 for io worker
2026-11-14 07:55:44.969 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:55:44.969 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 0
2026-11-14 07:56:45.031 EST [1111045][io worker][:0] DEBUG: shmem_exit(0): 4 before_shmem_exit callbacks to make
2026-11-14 07:56:45.032 EST [1111045][io worker][:0] DEBUG: shmem_exit(0): 7 on_shmem_exit callbacks to make
2026-11-14 07:56:45.032 EST [1111045][io worker][:0] DEBUG: proc_exit(0): 1 callbacks to make
2026-11-14 07:56:45.032 EST [1111045][io worker][:0] DEBUG: exit(0)
2026-11-14 07:56:45.032 EST [1111045][io worker][:0] DEBUG: shmem_exit(-1): 0 before_shmem_exit callbacks to make
2026-11-14 07:56:45.032 EST [1111045][io worker][:0] DEBUG: shmem_exit(-1): 0 on_shmem_exit callbacks to make
2026-11-14 07:56:45.032 EST [1111045][io worker][:0] DEBUG: proc_exit(-1): 0 callbacks to make
2026-11-14 07:56:45.032 EST [1110037][postmaster][:0] DEBUG: reaping dead processes
2026-11-14 07:56:45.032 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:56:45.033 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1111045, status: 0
2026-11-14 07:56:45.033 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 58
2026-11-14 07:56:45.033 EST [1110037][postmaster][:0] DEBUG: assigned pm child slot 58 for io worker
2026-11-14 07:56:45.033 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:56:45.033 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 0
2026-11-14 07:57:45.098 EST [1111065][io worker][:0] DEBUG: shmem_exit(0): 4 before_shmem_exit callbacks to make
2026-11-14 07:57:45.098 EST [1111065][io worker][:0] DEBUG: shmem_exit(0): 7 on_shmem_exit callbacks to make
2026-11-14 07:57:45.098 EST [1111065][io worker][:0] DEBUG: proc_exit(0): 1 callbacks to make
2026-11-14 07:57:45.098 EST [1111065][io worker][:0] DEBUG: exit(0)
2026-11-14 07:57:45.098 EST [1111065][io worker][:0] DEBUG: shmem_exit(-1): 0 before_shmem_exit callbacks to make
2026-11-14 07:57:45.098 EST [1111065][io worker][:0] DEBUG: shmem_exit(-1): 0 on_shmem_exit callbacks to make
2026-11-14 07:57:45.098 EST [1111065][io worker][:0] DEBUG: proc_exit(-1): 0 callbacks to make
2026-11-14 07:57:45.099 EST [1110037][postmaster][:0] DEBUG: reaping dead processes
2026-11-14 07:57:45.099 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:57:45.099 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1111065, status: 0
2026-11-14 07:57:45.099 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 58
2026-11-14 07:57:45.099 EST [1110037][postmaster][:0] DEBUG: assigned pm child slot 58 for io worker
2026-11-14 07:57:45.099 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:57:45.099 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 0
2026-11-14 07:57:49.231 EST [1110037][postmaster][:0] DEBUG: postmaster received shutdown request signal
2026-11-14 07:57:49.231 EST [1110037][postmaster][:0] LOG: received immediate shutdown request
2026-11-14 07:57:49.231 EST [1110037][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to io worker process with pid 1111082
2026-11-14 07:57:49.231 EST [1110037][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to background worker process with pid 1110793
2026-11-14 07:57:49.231 EST [1110037][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to walwriter process with pid 1110791
2026-11-14 07:57:49.231 EST [1110037][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to background writer process with pid 1110790
2026-11-14 07:57:49.231 EST [1110037][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to checkpointer process with pid 1110788
2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to io worker process with pid 1110784
2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: updating PMState from PM_RUN to PM_WAIT_BACKENDS
2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: io worker process 1111082 is still running
2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: background worker process 1110793 is still running
2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: walwriter process 1110791 is still running
2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: background writer process 1110790 is still running
2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: checkpointer process 1110788 is still running
2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: io worker process 1110784 is still running
2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: reaping dead processes
2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110790, status: 512
2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 55
2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110788, status: 512
2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 56
2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110791, status: 512
2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 92
2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110784, status: 512
2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 57
2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1110793, status: 512
2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 45
2026-11-14 07:57:49.232 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:57:49.233 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: 1111082, status: 512
2026-11-14 07:57:49.233 EST [1110037][postmaster][:0] DEBUG: releasing pm child slot 58
2026-11-14 07:57:49.233 EST [1110037][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:57:49.233 EST [1110037][postmaster][:0] LOG: !!!waitpid| ret: -1, status: 512
2026-11-14 07:57:49.233 EST [1110037][postmaster][:0] DEBUG: updating PMState from PM_WAIT_BACKENDS to PM_WAIT_DEAD_END
2026-11-14 07:57:49.233 EST [1110037][postmaster][:0] DEBUG: updating PMState from PM_WAIT_DEAD_END to PM_NO_CHILDREN
2026-11-14 07:57:49.233 EST [1110037][postmaster][:0] DEBUG: shmem_exit(0): 0 before_shmem_exit callbacks to make
2026-11-14 07:57:49.233 EST [1110037][postmaster][:0] DEBUG: shmem_exit(0): 5 on_shmem_exit callbacks to make
2026-11-14 07:57:49.233 EST [1110037][postmaster][:0] DEBUG: cleaning up orphaned dynamic shared memory with ID 1459897042
2026-11-14 07:57:49.233 EST [1110037][postmaster][:0] DEBUG: cleaning up orphaned dynamic shared memory with ID 3069910578
2026-11-14 07:57:49.233 EST [1110037][postmaster][:0] DEBUG: cleaning up orphaned dynamic shared memory with ID 3859456016
2026-11-14 07:57:49.233 EST [1110037][postmaster][:0] DEBUG: cleaning up dynamic shared memory control segment with ID 2869141000
2026-11-14 07:57:49.234 EST [1110037][postmaster][:0] DEBUG: proc_exit(0): 2 callbacks to make
2026-11-14 07:57:49.234 EST [1110037][postmaster][:0] LOG: database system is shut down
2026-11-14 07:57:49.234 EST [1110037][postmaster][:0] DEBUG: exit(0)
2026-11-14 07:57:49.234 EST [1110037][postmaster][:0] DEBUG: shmem_exit(-1): 0 before_shmem_exit callbacks to make
2026-11-14 07:57:49.234 EST [1110037][postmaster][:0] DEBUG: shmem_exit(-1): 0 on_shmem_exit callbacks to make
2026-11-14 07:57:49.234 EST [1110037][postmaster][:0] DEBUG: proc_exit(-1): 0 callbacks to make
041_checkpoint_at_promote_standby1-1.log (text/x-log, 71.0K) download
041_checkpoint_at_promote_standby1-0.log (text/x-log, 44.4K) download
...
2026-11-14 07:54:44.154 EST [1109760][startup][37/0:0] LOG: received promote request
2026-11-14 07:54:44.154 EST [1109760][startup][37/0:0] LOG: redo done at 0/04000028 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.33 s
2026-11-14 07:54:44.154 EST [1109760][startup][37/0:0] LOG: last completed transaction was at log time 2026-11-14 07:54:44.016107-05
2026-11-14 07:54:44.154 EST [1109760][startup][37/0:0] DEBUG: resetting unlogged relations: cleanup 0 init 1
2026-11-14 07:54:44.154 EST [1109760][startup][37/0:0] LOG: selected new timeline ID: 2
2026-11-14 07:54:44.155 EST [1109760][startup][37/0:0] DEBUG: updated min recovery point to 0/040000B0 on timeline 1
2026-11-14 07:54:44.172 EST [1109760][startup][37/0:0] DEBUG: could not remove file "pg_wal/000000020000000000000004": No such file or directory
2026-11-14 07:54:44.172 EST [1109760][startup][37/0:0] LOG: archive recovery complete
2026-11-14 07:54:44.172 EST [1109760][startup][37/0:0] DEBUG: MultiXactId wrap limit is 2147483648, limited by database with OID 1
2026-11-14 07:54:44.172 EST [1109760][startup][37/0:0] DEBUG: attempting to remove WAL segments newer than log file 000000020000000000000004
2026-11-14 07:54:44.172 EST [1109760][startup][37/0:0] DEBUG: removing all KnownAssignedXids
2026-11-14 07:54:44.172 EST [1109760][startup][37/0:0] DEBUG: release all standby locks
2026-11-14 07:54:44.172 EST [1109760][startup][37/0:0] DEBUG: shmem_exit(0): 5 before_shmem_exit callbacks to make
2026-11-14 07:54:44.172 EST [1109760][startup][37/0:0] DEBUG: shmem_exit(0): 8 on_shmem_exit callbacks to make
2026-11-14 07:54:44.172 EST [1109760][startup][:0] DEBUG: proc_exit(0): 1 callbacks to make
2026-11-14 07:54:44.172 EST [1109760][startup][:0] DEBUG: exit(0)
2026-11-14 07:54:44.256 EST [1109746][postmaster][:0] DEBUG: assigned pm child slot 2 for client backend
2026-11-14 07:54:44.256 EST [1109746][postmaster][:0] DEBUG: forked new client backend, pid=1110418 socket=8
2026-11-14 07:54:44.256 EST [1110418][client backend][:0] LOG: connection received: host=[local]
2026-11-14 07:54:44.257 EST [1110418][client backend][5/0:0] DEBUG: InitPostgres
2026-11-14 07:54:44.257 EST [1110418][client backend][5/1:0] LOG: connection authenticated: ...
2026-11-14 07:54:44.257 EST [1110418][client backend][5/1:0] LOG: connection authorized: ...
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] LOG: statement: SELECT injection_points_wakeup('create-restart-point');
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8)
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0)
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192)
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1)
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: io 5 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: find_in_path: trying ".../tmp_install/usr/local/pgsql/lib/injection_points"
2026-11-14 07:54:44.258 EST [1110418][client backend][5/2:0] DEBUG: find_in_path: trying ".../tmp_install/usr/local/pgsql/lib/injection_points.so"
2026-11-14 07:54:44.259 EST [1110418][client backend][5/0:0] DEBUG: shmem_exit(0): 5 before_shmem_exit callbacks to make
2026-11-14 07:54:44.259 EST [1110418][client backend][5/0:0] DEBUG: shmem_exit(0): 7 on_shmem_exit callbacks to make
2026-11-14 07:54:44.259 EST [1110418][client backend][:0] DEBUG: proc_exit(0): 3 callbacks to make
2026-11-14 07:54:44.259 EST [1110418][client backend][:0] LOG: disconnection: session time: 0:00:00.002 ...
2026-11-14 07:54:44.259 EST [1110418][client backend][:0] DEBUG: exit(0)
2026-11-14 07:54:44.259 EST [1110418][client backend][:0] DEBUG: shmem_exit(-1): 0 before_shmem_exit callbacks to make
2026-11-14 07:54:44.259 EST [1110418][client backend][:0] DEBUG: shmem_exit(-1): 0 on_shmem_exit callbacks to make
2026-11-14 07:54:44.259 EST [1110418][client backend][:0] DEBUG: proc_exit(-1): 0 callbacks to make
2026-11-14 07:54:44.260 EST [1109746][postmaster][:0] DEBUG: reaping dead processes
2026-11-14 07:54:44.260 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:54:44.260 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 1110418, status: 0
2026-11-14 07:54:44.260 EST [1109746][postmaster][:0] DEBUG: releasing pm child slot 2
2026-11-14 07:54:44.260 EST [1109746][postmaster][:0] DEBUG: client backend (PID 1110418) exited with exit code 0
2026-11-14 07:54:44.260 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:54:44.260 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 0
2026-11-14 07:54:44.273 EST [1109758][checkpointer][:0] DEBUG: attempting to remove WAL segments older than log file 000000000000000000000002
2026-11-14 07:54:44.273 EST [1109758][checkpointer][:0] DEBUG: recycled write-ahead log file "000000010000000000000002"
2026-11-14 07:54:44.273 EST [1109758][checkpointer][:0] DEBUG: SlruScanDirectory invoking callback on pg_subtrans/0000
2026-11-14 07:54:44.273 EST [1109758][checkpointer][:0] LOG: restartpoint complete: fast wait: wrote 16 buffers (12.5%), wrote 2 SLRU buffers; 0 WAL file(s) added, 0 removed, 1 recycled; write=0.001 s, sync=0.001 s, total=0.261 s; sync files=0, longest=0.000 s, average=0.000 s; distance=16384 kB, estimate=16384 kB; lsn=0/03000088, redo lsn=0/03000028
2026-11-14 07:54:44.273 EST [1109758][checkpointer][:0] LOG: recovery restart point at 0/03000028
2026-11-14 07:54:44.273 EST [1109758][checkpointer][:0] DETAIL: Last completed transaction was at log time 2026-11-14 07:54:44.016107-05.
2026-11-14 07:54:44.273 EST [1109758][checkpointer][:0] LOG: checkpoint starting: force
2026-11-14 07:54:44.361 EST [1109746][postmaster][:0] DEBUG: assigned pm child slot 2 for client backend
2026-11-14 07:54:44.361 EST [1109746][postmaster][:0] DEBUG: forked new client backend, pid=1110482 socket=8
2026-11-14 07:54:44.361 EST [1110482][client backend][:0] LOG: connection received: host=[local]
2026-11-14 07:54:44.362 EST [1110482][client backend][6/0:0] DEBUG: InitPostgres
2026-11-14 07:54:44.362 EST [1110482][client backend][6/1:0] LOG: connection authenticated: ...
2026-11-14 07:54:44.362 EST [1110482][client backend][6/1:0] LOG: connection authorized: ...
2026-11-14 07:54:44.362 EST [1110482][client backend][6/2:0] LOG: statement: SELECT pg_backend_pid();
2026-11-14 07:54:44.363 EST [1110482][client backend][6/2:0] DEBUG: io 6 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb at character 8
2026-11-14 07:54:44.363 EST [1110482][client backend][6/2:0] DEBUG: io 6 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb at character 8
2026-11-14 07:54:44.363 EST [1110482][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(8) at character 8
2026-11-14 07:54:44.363 EST [1110482][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state STAGED : staged (synchronous: 1, in_batch: 0) at character 8
2026-11-14 07:54:44.363 EST [1110482][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state COMPLETED_IO : calling cb #2, id 1/aio_md_readv_cb->complete_shared(0) with distilled result: (status OK, id 0, error_data 0, result 8192) at character 8
2026-11-14 07:54:44.363 EST [1110482][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state COMPLETED_IO : calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_shared(8) with distilled result: (status OK, id 0, error_data 0, result 1) at character 8
2026-11-14 07:54:44.363 EST [1110482][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 at character 8
2026-11-14 07:54:44.363 EST [1110482][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state COMPLETED_SHARED: calling cb #1, id 2/aio_shared_buffer_readv_cb->complete_local(8) with distilled result: status OK, id 0, error_data 0, result 1 at character 8
2026-11-14 07:54:44.363 EST [1110482][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 at character 8
2026-11-14 07:54:44.363 EST [1110482][client backend][6/2:0] DEBUG: io 6 |op readv|target smgr|state COMPLETED_LOCAL : reclaiming: distilled_result: (status OK, id 0, error_data 0), raw_result: 8192 at character 8
2026-11-14 07:54:44.364 EST [1109746][postmaster][:0] DEBUG: reaping dead processes
2026-11-14 07:54:44.364 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:54:44.364 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 1110482, status: 9
2026-11-14 07:54:44.364 EST [1109746][postmaster][:0] DEBUG: releasing pm child slot 2
2026-11-14 07:54:44.364 EST [1109746][postmaster][:0] LOG: client backend (PID 1110482) was terminated by signal 9: Killed
2026-11-14 07:54:44.364 EST [1109746][postmaster][:0] DETAIL: Failed process was running: SELECT pg_backend_pid();
2026-11-14 07:54:44.364 EST [1109746][postmaster][:0] LOG: terminating any other active server processes
2026-11-14 07:54:44.364 EST [1109746][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to client backend process with pid 1109914
2026-11-14 07:54:44.364 EST [1109746][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to startup process with pid 1109760
2026-11-14 07:54:44.375 EST [1109758][checkpointer][:0] DEBUG: performing replication slot checkpoint
2026-11-14 07:54:44.375 EST [1109758][checkpointer][:0] DEBUG: snapshot of 0+0 running transaction ids (lsn 0/04000148 oldest xid 667 latest complete 666 next xid 667)
2026-11-14 07:54:44.375 EST [1109758][checkpointer][:0] DEBUG: attempting to remove WAL segments older than log file 000000000000000000000003
2026-11-14 07:54:44.375 EST [1109758][checkpointer][:0] DEBUG: recycled write-ahead log file "000000010000000000000003"
2026-11-14 07:54:44.375 EST [1109758][checkpointer][:0] DEBUG: SlruScanDirectory invoking callback on pg_subtrans/0000
2026-11-14 07:54:44.375 EST [1109758][checkpointer][:0] LOG: checkpoint complete: force: wrote 1 buffers (0.8%), wrote 2 SLRU buffers; 0 WAL file(s) added, 0 removed, 1 recycled; write=0.101 s, sync=0.001 s, total=0.103 s; sync files=0, longest=0.000 s, average=0.000 s; distance=16384 kB, estimate=16384 kB; lsn=0/04000148, redo lsn=0/040000E8
2026-11-14 07:54:44.400 EST [1109746][postmaster][:0] LOG: !!!signal_child| delayed signal 3/SIGQUIT to startup process with pid 1109760
2026-11-14 07:54:44.500 EST [1109760][startup][:0] LOG: !!!proc_exit| delayed exit(0) for startup process
2026-11-14 07:54:44.539 EST [1109746][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to background writer process with pid 1109759
2026-11-14 07:54:44.539 EST [1109746][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to checkpointer process with pid 1109758
2026-11-14 07:54:44.539 EST [1109746][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to io worker process with pid 1109757
2026-11-14 07:54:44.539 EST [1109746][postmaster][:0] DEBUG: sending signal 3/SIGQUIT to io worker process with pid 1109756
2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: updating PMState from PM_HOT_STANDBY to PM_WAIT_BACKENDS
2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 1109756, status: 512
2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: releasing pm child slot 57
2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 1109757, status: 512
2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: releasing pm child slot 58
2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 1109758, status: 512
2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: releasing pm child slot 56
2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 1109759, status: 512
2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: releasing pm child slot 55
2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 1109760, status: 512
2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: releasing pm child slot 89
2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 1109914, status: 512
2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: releasing pm child slot 1
2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: -1, status: 512
2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: updating PMState from PM_WAIT_BACKENDS to PM_WAIT_DEAD_END
2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: updating PMState from PM_WAIT_DEAD_END to PM_NO_CHILDREN
2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] LOG: all server processes terminated; reinitializing
2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: shmem_exit(1): 0 before_shmem_exit callbacks to make
2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: shmem_exit(1): 5 on_shmem_exit callbacks to make
2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: cleaning up orphaned dynamic shared memory with ID 924198510
2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: cleaning up orphaned dynamic shared memory with ID 1616401164
2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: cleaning up orphaned dynamic shared memory with ID 3295781356
2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: cleaning up dynamic shared memory control segment with ID 2272285588
2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: invoking IpcMemoryCreate(size=5873664)
2026-11-14 07:54:44.540 EST [1109746][postmaster][:0] DEBUG: mmap(6291456) with MAP_HUGETLB failed, huge pages disabled: Cannot allocate memory
2026-11-14 07:54:44.541 EST [1109746][postmaster][:0] DEBUG: dynamic shared memory system will support 229 segments
2026-11-14 07:54:44.541 EST [1109746][postmaster][:0] DEBUG: created dynamic shared memory control segment 4225595752 (9176 bytes)
2026-11-14 07:54:44.541 EST [1109746][postmaster][:0] DEBUG: updating PMState from PM_NO_CHILDREN to PM_STARTUP
2026-11-14 07:54:44.541 EST [1109746][postmaster][:0] DEBUG: assigned pm child slot 58 for io worker
2026-11-14 07:54:44.541 EST [1109746][postmaster][:0] DEBUG: assigned pm child slot 57 for io worker
2026-11-14 07:54:44.541 EST [1109746][postmaster][:0] DEBUG: assigned pm child slot 89 for startup
2026-11-14 07:54:44.541 EST [1109746][postmaster][:0] DEBUG: assigned pm child slot 56 for checkpointer
2026-11-14 07:54:44.541 EST [1109746][postmaster][:0] DEBUG: assigned pm child slot 55 for background writer
2026-11-14 07:54:44.542 EST [1109746][postmaster][:0] DEBUG: reaping dead processes
2026-11-14 07:54:44.542 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:54:44.542 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 43690
2026-11-14 07:54:44.542 EST [1110541][startup][:0] LOG: database system was interrupted; last known up at 2026-11-14 07:54:44 EST
2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: removing all temporary WAL segments
2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: checkpoint record is at 0/04000148
2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: redo record is at 0/040000E8; shutdown false
2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: next transaction ID: 667; next OID: 16398
2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: next MultiXactId: 1; next MultiXactOffset: 1
2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: oldest unfrozen transaction ID: 656, in database 1
2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: oldest MultiXactId: 1, in database 1
2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: commit timestamp Xid oldest/newest: 0/0
2026-11-14 07:54:44.542 EST [1110541][startup][:0] LOG: database system was not properly shut down; automatic recovery in progress
2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: transaction ID wrap limit is 2147484303, limited by database with OID 1
2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: MultiXactId wrap limit is 2147483648, limited by database with OID 1
2026-11-14 07:54:44.542 EST [1110542][checkpointer][:0] DEBUG: checkpointer updated shared memory configuration values
2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: starting up replication slots
2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: xmin required by slots: data 0, catalog 0
2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: starting up replication origin progress state
2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: didn't need to unlink permanent stats file "pg_stat/pgstat.stat" - didn't exist
2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: resetting unlogged relations: cleanup 1 init 0
2026-11-14 07:54:44.542 EST [1110541][startup][:0] LOG: redo starts at 0/040000E8
2026-11-14 07:54:44.542 EST [1110541][startup][:0] LOG: invalid record length at 0/040001D0: expected at least 24, got 0
2026-11-14 07:54:44.542 EST [1110541][startup][:0] LOG: redo done at 0/04000148 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s
2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: resetting unlogged relations: cleanup 0 init 1
2026-11-14 07:54:44.542 EST [1109746][postmaster][:0] DEBUG: postmaster received pmsignal signal
2026-11-14 07:54:44.542 EST [1109746][postmaster][:0] DEBUG: updating PMState from PM_STARTUP to PM_RECOVERY
2026-11-14 07:54:44.542 EST [1110541][startup][:0] DEBUG: MultiXactId wrap limit is 2147483648, limited by database with OID 1
2026-11-14 07:54:44.543 EST [1110542][checkpointer][:0] LOG: checkpoint starting: end-of-recovery fast wait
2026-11-14 07:54:44.543 EST [1110542][checkpointer][:0] DEBUG: performing replication slot checkpoint
2026-11-14 07:54:44.543 EST [1110542][checkpointer][:0] DEBUG: attempting to remove WAL segments older than log file 000000000000000000000003
2026-11-14 07:54:44.544 EST [1110542][checkpointer][:0] LOG: checkpoint complete: end-of-recovery fast wait: wrote 0 buffers (0.0%), wrote 3 SLRU buffers; 0 WAL file(s) added, 0 removed, 0 recycled; write=0.001 s, sync=0.001 s, total=0.001 s; sync files=0, longest=0.000 s, average=0.000 s; distance=0 kB, estimate=0 kB; lsn=0/040001D0, redo lsn=0/040001D0
2026-11-14 07:54:44.544 EST [1110541][startup][:0] DEBUG: shmem_exit(0): 5 before_shmem_exit callbacks to make
2026-11-14 07:54:44.544 EST [1110541][startup][:0] DEBUG: shmem_exit(0): 7 on_shmem_exit callbacks to make
2026-11-14 07:54:44.544 EST [1110541][startup][:0] DEBUG: proc_exit(0): 1 callbacks to make
2026-11-14 07:54:44.544 EST [1110541][startup][:0] DEBUG: exit(0)
2026-11-14 07:54:44.569 EST [1109746][postmaster][:0] DEBUG: allocating dead-end child
2026-11-14 07:54:44.569 EST [1109746][postmaster][:0] DEBUG: forked new dead-end client backend, pid=1110598 socket=8
2026-11-14 07:54:44.569 EST [1110598][dead-end client backend][:0] LOG: connection received: host=[local]
2026-11-14 07:54:44.569 EST [1110598][dead-end client backend][:0] FATAL: the database system is not yet accepting connections
2026-11-14 07:54:44.569 EST [1110598][dead-end client backend][:0] DETAIL: Consistent recovery state has not been yet reached.
2026-11-14 07:54:44.569 EST [1110598][dead-end client backend][:0] DEBUG: shmem_exit(1): 0 before_shmem_exit callbacks to make
2026-11-14 07:54:44.569 EST [1110598][dead-end client backend][:0] DEBUG: shmem_exit(1): 0 on_shmem_exit callbacks to make
2026-11-14 07:54:44.569 EST [1110598][dead-end client backend][:0] DEBUG: proc_exit(1): 1 callbacks to make
2026-11-14 07:54:44.569 EST [1110598][dead-end client backend][:0] DEBUG: exit(1)
2026-11-14 07:54:44.569 EST [1110598][dead-end client backend][:0] DEBUG: shmem_exit(-1): 0 before_shmem_exit callbacks to make
2026-11-14 07:54:44.569 EST [1110598][dead-end client backend][:0] DEBUG: shmem_exit(-1): 0 on_shmem_exit callbacks to make
2026-11-14 07:54:44.569 EST [1110598][dead-end client backend][:0] DEBUG: proc_exit(-1): 0 callbacks to make
2026-11-14 07:54:44.570 EST [1109746][postmaster][:0] DEBUG: reaping dead processes
2026-11-14 07:54:44.570 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:54:44.570 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 1110598, status: 256
2026-11-14 07:54:44.570 EST [1109746][postmaster][:0] DEBUG: releasing dead-end backend
2026-11-14 07:54:44.570 EST [1109746][postmaster][:0] DEBUG: dead-end client backend (PID 1110598) exited with exit code 1
2026-11-14 07:54:44.570 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:54:44.570 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 256
2026-11-14 07:54:44.672 EST [1109746][postmaster][:0] DEBUG: allocating dead-end child
2026-11-14 07:54:44.672 EST [1109746][postmaster][:0] DEBUG: forked new dead-end client backend, pid=1110640 socket=8
2026-11-14 07:54:44.673 EST [1110640][dead-end client backend][:0] LOG: connection received: host=[local]
2026-11-14 07:54:44.673 EST [1110640][dead-end client backend][:0] FATAL: the database system is not yet accepting connections
2026-11-14 07:54:44.673 EST [1110640][dead-end client backend][:0] DETAIL: Consistent recovery state has not been yet reached.
2026-11-14 07:54:44.673 EST [1110640][dead-end client backend][:0] DEBUG: shmem_exit(1): 0 before_shmem_exit callbacks to make
2026-11-14 07:54:44.673 EST [1110640][dead-end client backend][:0] DEBUG: shmem_exit(1): 0 on_shmem_exit callbacks to make
2026-11-14 07:54:44.673 EST [1110640][dead-end client backend][:0] DEBUG: proc_exit(1): 1 callbacks to make
2026-11-14 07:54:44.673 EST [1110640][dead-end client backend][:0] DEBUG: exit(1)
2026-11-14 07:54:44.673 EST [1110640][dead-end client backend][:0] DEBUG: shmem_exit(-1): 0 before_shmem_exit callbacks to make
2026-11-14 07:54:44.673 EST [1110640][dead-end client backend][:0] DEBUG: shmem_exit(-1): 0 on_shmem_exit callbacks to make
2026-11-14 07:54:44.673 EST [1110640][dead-end client backend][:0] DEBUG: proc_exit(-1): 0 callbacks to make
2026-11-14 07:54:44.673 EST [1109746][postmaster][:0] DEBUG: reaping dead processes
2026-11-14 07:54:44.673 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:54:44.673 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 1110640, status: 256
2026-11-14 07:54:44.673 EST [1109746][postmaster][:0] DEBUG: releasing dead-end backend
2026-11-14 07:54:44.673 EST [1109746][postmaster][:0] DEBUG: dead-end client backend (PID 1110640) exited with exit code 1
2026-11-14 07:54:44.673 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:54:44.673 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 256
2026-11-14 07:54:44.775 EST [1109746][postmaster][:0] DEBUG: allocating dead-end child
2026-11-14 07:54:44.775 EST [1109746][postmaster][:0] DEBUG: forked new dead-end client backend, pid=1110689 socket=8
2026-11-14 07:54:44.775 EST [1110689][dead-end client backend][:0] LOG: connection received: host=[local]
2026-11-14 07:54:44.775 EST [1110689][dead-end client backend][:0] FATAL: the database system is not yet accepting connections
2026-11-14 07:54:44.775 EST [1110689][dead-end client backend][:0] DETAIL: Consistent recovery state has not been yet reached.
2026-11-14 07:54:44.775 EST [1110689][dead-end client backend][:0] DEBUG: shmem_exit(1): 0 before_shmem_exit callbacks to make
2026-11-14 07:54:44.775 EST [1110689][dead-end client backend][:0] DEBUG: shmem_exit(1): 0 on_shmem_exit callbacks to make
2026-11-14 07:54:44.775 EST [1110689][dead-end client backend][:0] DEBUG: proc_exit(1): 1 callbacks to make
2026-11-14 07:54:44.775 EST [1110689][dead-end client backend][:0] DEBUG: exit(1)
2026-11-14 07:54:44.775 EST [1110689][dead-end client backend][:0] DEBUG: shmem_exit(-1): 0 before_shmem_exit callbacks to make
2026-11-14 07:54:44.775 EST [1110689][dead-end client backend][:0] DEBUG: shmem_exit(-1): 0 on_shmem_exit callbacks to make
2026-11-14 07:54:44.775 EST [1110689][dead-end client backend][:0] DEBUG: proc_exit(-1): 0 callbacks to make
2026-11-14 07:54:44.776 EST [1109746][postmaster][:0] DEBUG: reaping dead processes
2026-11-14 07:54:44.776 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:54:44.776 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 1110689, status: 256
2026-11-14 07:54:44.776 EST [1109746][postmaster][:0] DEBUG: releasing dead-end backend
2026-11-14 07:54:44.776 EST [1109746][postmaster][:0] DEBUG: dead-end client backend (PID 1110689) exited with exit code 1
2026-11-14 07:54:44.776 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:54:44.776 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 256
2026-11-14 07:54:44.800 EST [1110541][startup][:0] LOG: !!!proc_exit| delayed exit(0) for startup process
2026-11-14 07:54:44.800 EST [1110541][startup][:0] DEBUG: shmem_exit(-1): 0 before_shmem_exit callbacks to make
2026-11-14 07:54:44.800 EST [1110541][startup][:0] DEBUG: shmem_exit(-1): 0 on_shmem_exit callbacks to make
2026-11-14 07:54:44.800 EST [1110541][startup][:0] DEBUG: proc_exit(-1): 0 callbacks to make
2026-11-14 07:54:44.800 EST [1109746][postmaster][:0] DEBUG: reaping dead processes
2026-11-14 07:54:44.800 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:54:44.800 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 1110541, status: 0
2026-11-14 07:54:44.800 EST [1109746][postmaster][:0] DEBUG: releasing pm child slot 89
2026-11-14 07:54:44.800 EST [1109746][postmaster][:0] DEBUG: updating PMState from PM_RECOVERY to PM_RUN
2026-11-14 07:54:44.800 EST [1109746][postmaster][:0] LOG: database system is ready to accept connections
2026-11-14 07:54:44.800 EST [1109746][postmaster][:0] LOG: !!!waitpid| pid: -1
2026-11-14 07:54:44.800 EST [1109746][postmaster][:0] LOG: !!!waitpid| ret: 0, status: 0
2026-11-14 07:54:44.800 EST [1109746][postmaster][:0] DEBUG: assigned pm child slot 92 for walwriter
2026-11-14 07:54:44.801 EST [1109746][postmaster][:0] DEBUG: assigned pm child slot 45 for background worker
2026-11-14 07:54:44.801 EST [1109746][postmaster][:0] DEBUG: starting background worker process "logical replication launcher"
2026-11-14 07:54:44.801 EST [1110704][logical replication launcher][15/0:0] DEBUG: logical replication launcher started
2026-11-14 07:54:44.801 EST [1110704][logical replication launcher][15/0:0] DEBUG: InitPostgres
2026-11-14 07:54:44.884 EST [1109746][postmaster][:0] DEBUG: assigned pm child slot 1 for client backend
2026-11-14 07:54:44.884 EST [1109746][postmaster][:0] DEBUG: forked new client backend, pid=1110770 socket=8
2026-11-14 07:54:44.884 EST [1110770][client backend][:0] LOG: connection received: host=[local]
2026-11-14 07:54:44.884 EST [1110770][client backend][0/0:0] DEBUG: InitPostgres
...
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: 041_checkpoint_at_promote.pl might fail due to race condition on child kill
@ 2026-11-16 05:26 Ayush Tiwari <ayushtiwari.slg01@gmail.com>
parent: Alexander Lakhin <exclusion@gmail.com>
0 siblings, 1 reply; 12+ messages in thread
From: Ayush Tiwari @ 2026-11-16 05:26 UTC (permalink / raw)
To: Alexander Lakhin <exclusion@gmail.com>; +Cc: Fujii Masao <masao.fujii@gmail.com>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers
Hi,
On Sat, 14 Nov 2026 at 19:30, Alexander Lakhin <exclusion@gmail.com> wrote:
>
> Hello Fujii-san and Ayush,
>
> 30.10.2026 13:55, Fujii Masao wrote:
> > On Fri, Oct 30, 2026 at 3:59 PM Ayush Tiwari
> > <ayushtiwari.slg01@gmail.com> wrote:
> >> Did multiple runs with it, and the changes lgtm.
> > I've pushed the patch. Thanks!
>
> Please look at a new failure of 041_checkpoint_at_promote produced by
> tamandua recently [1]:
> [03:10:10.869](0.025s) ok 4 - psql query died successfully after SIGKILL
> [03:13:15.804](184.935s) # die: timed out waiting for file
>
/home/bf/bf-build/tamandua/REL_19_STABLE/pgsql.build/testrun/recovery/041_checkpoint_at_promote/log/041_checkpoint_at_promote_standby1.log
> contents to match: (?^:all server processes terminated; reinitializing) at
> /home/bf/bf-build/tamandua/REL_19_STABLE/pgsql/src/test/recovery/t/
041_checkpoint_at_promote.pl line 166.
> [03:13:15.806](0.003s) 1..4
> timed out waiting for file
>
/home/bf/bf-build/tamandua/REL_19_STABLE/pgsql.build/testrun/recovery/041_checkpoint_at_promote/log/041_checkpoint_at_promote_standby1.log
> contents to match: (?^:all server processes terminated; reinitializing) at
> /home/bf/bf-build/tamandua/REL_19_STABLE/pgsql/src/test/recovery/t/
041_checkpoint_at_promote.pl line 166.
> # No postmaster PID for node "master"
>
> 041_checkpoint_at_promote_standby1.log contains:
> 2026-11-12 03:10:10.826 CET [3578531][client backend][6/2:0] LOG:
statement: SELECT pg_backend_pid();
> 2026-11-12 03:10:10.861 CET [3577203][postmaster][:0] LOG: client
backend (PID 3578531) was terminated by signal 9: Killed
> 2026-11-12 03:10:10.861 CET [3577203][postmaster][:0] DETAIL: Failed
process was running: SELECT pg_backend_pid();
> 2026-11-12 03:10:10.862 CET [3577203][postmaster][:0] LOG: terminating
any other active server processes
> 2026-11-12 03:10:10.928 CET [3577203][postmaster][:0] LOG: database
system is ready to accept connections
>
> There is no "terminating any other active server processes" message
> indeed, the server was not shut down.
>
> I managed to reproduce this locally (with the patch attached and with the
> same parallel loop as before) and discovered that this can happen when
> PMState goes from PM_HOT_STANDBY to PM_RUN (via PM_WAIT_BACKENDS) instead
> of PM_HOT_STANDBY -> PM_WAIT_BACKENDS -> PM_WAIT_DEAD_END ->
> PM_NO_CHILDREN -> "all server processes terminated; reinitializing".
>
> This can be explained by two race conditions, with one precondition: the
> startup process doesn't exit before pg_ctl kill.
> 1) The startup process becomes zombie for the postmaster if it exits when
> waitpid() doesn't wait.
> ...
> 2026-11-14 13:29:02.900 UTC [695404][startup][:0] DEBUG: proc_exit(-1): 0
callbacks to make
> ...
> 2026-11-14 13:29:03.088 UTC [695367][postmaster][:0] DEBUG: sending
signal 3/SIGQUIT to startup process with pid 695404
> 2026-11-14 13:29:03.100 UTC [695367][postmaster][:0] LOG:
!!!signal_child| delayed signal 3/SIGQUIT to startup process
> with pid 695404
> ...
> 2026-11-14 13:29:03.170 UTC [695367][postmaster][:0] DEBUG: startup
process 695404 is still running
> ...
> 2026-11-14 13:29:03.198 UTC [695367][postmaster][:0] DEBUG: updating
PMState from PM_WAIT_BACKENDS to PM_RUN
>
>
> 2) SIGQUIT sent to the startup process when it's being executed inside
> exit(0), after the point when signal handlers of a process are not called
> anymore, so SIGQUIT is ignored:
> ...
> 2026-11-14 07:54:44.900 EST [1110043][startup][:0] LOG: !!!proc_exit|
delayed exit(0) for startup process
> ...
> 2026-11-14 07:54:44.900 EST [1110043][startup][:0] DEBUG: proc_exit(-1):
0 callbacks to make
> 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] LOG:
!!!signal_child| delayed signal 3/SIGQUIT to startup process
> with pid 1110043
> ...
> 2026-11-14 07:54:44.900 EST [1110037][postmaster][:0] DEBUG: startup
process 1110043 is still running
> ...
> 2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] LOG: !!!waitpid|
ret: 1110043, status: 0
> 2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] DEBUG: releasing pm
child slot 89
> 2026-11-14 07:54:44.903 EST [1110037][postmaster][:0] DEBUG: updating
PMState from PM_WAIT_BACKENDS to PM_RUN
>
> Corresponding logs for these 2 cases and also a normal run's log are
> attached for reference.
Thanks, I reproduced this with your debugging patch. Looks like a
postmaster bug rather than a problem with the test. Let me try to
write here what I understood.
Normally, startup exits with status 0 after promotion, and the postmaster
takes that as the signal to enter PM_RUN. A backend crash needs the
opposite, stop the other children and rebuild shared memory before
letting the server run again.
Here those two things happen at almost the same time:
1) Startup finishes promotion and is exiting, but the postmaster hasn't
collected its exit status yet.
2) The test kills a backend. The postmaster notices that death first,
sets FatalError and sends SIGQUIT to the remaining children. It recrds
that it told startup to quit by setting STARTUP_SIGNALED.
3) SIGQUIT is too late for startup, it has already exited, or is too far
into exit(0) to handle it? Its exit status is basically still 0.
4) The postmaster then collects that 0 and takes the normal "startup
succeeded" path. It clears FatalError and enters PM_RUN, skipping the
shared-memory reset needed for the backend crash.
That's why the test never sees "all server processes terminated;
reinitializing".
I think we should check STARTUP_SIGNALED even when the exit status is 0.
Something like [2]. With that diff, I tried your patch and reran the tests
multiple
times and it all passed.
Regards,
Ayush
> [1]
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=tamandua&dt=2026-11-12%2002%3A04%3A37
[2]
diff --git a/src/backend/postmaster/postmaster.c
b/src/backend/postmaster/postmaster.c
index f53aa9c404d..a2e52b61902 100644
inline patch 1: src/backend/postmaster/postmaster.c (+6/-5 lines) download inline-1.patch
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -2317,12 +2317,13 @@ process_pm_child_exit(void)
* during PM_STARTUP due to some dead-end child crashing: in that
* situation, if the startup process dies on the SIGQUIT, we need
* to transition to PM_WAIT_BACKENDS state which will allow
- * PostmasterStateMachine to restart the startup process. (On the
- * other hand, the startup process might complete normally, if we
- * were too late with the SIGQUIT. In that case we'll fall
- * through and commence normal operations.)
+ * PostmasterStateMachine to restart the startup process.
+ *
+ * SIGQUIT may arrive too late to change the startup process's exit
+ * status. If we signaled it, continue the crash restart even if it
+ * exits normally.
*/
- if (!EXIT_STATUS_0(exitstatus))
+ if (!EXIT_STATUS_0(exitstatus) || StartupStatus == STARTUP_SIGNALED)
{
if (StartupStatus == STARTUP_SIGNALED)
{
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: 041_checkpoint_at_promote.pl might fail due to race condition on child kill
@ 2026-11-16 06:19 Fujii Masao <masao.fujii@gmail.com>
parent: Ayush Tiwari <ayushtiwari.slg01@gmail.com>
0 siblings, 1 reply; 12+ messages in thread
From: Fujii Masao @ 2026-11-16 06:19 UTC (permalink / raw)
To: Ayush Tiwari <ayushtiwari.slg01@gmail.com>; +Cc: Alexander Lakhin <exclusion@gmail.com>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers
On Mon, Nov 16, 2026 at 2:26 PM Ayush Tiwari
<ayushtiwari.slg01@gmail.com> wrote:
> I think we should check STARTUP_SIGNALED even when the exit status is 0.
> Something like [2].
LGTM, thanks!
This should be backpatched to all supported versions?
Regards,
--
Fujii Masao
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: 041_checkpoint_at_promote.pl might fail due to race condition on child kill
@ 2026-11-16 06:50 Ayush Tiwari <ayushtiwari.slg01@gmail.com>
parent: Fujii Masao <masao.fujii@gmail.com>
0 siblings, 1 reply; 12+ messages in thread
From: Ayush Tiwari @ 2026-11-16 06:50 UTC (permalink / raw)
To: Fujii Masao <masao.fujii@gmail.com>; +Cc: Alexander Lakhin <exclusion@gmail.com>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers
Hi,
On Mon, 16 Nov 2026 at 11:49, Fujii Masao <masao.fujii@gmail.com> wrote:
>
> On Mon, Nov 16, 2026 at 2:26 PM Ayush Tiwari
> <ayushtiwari.slg01@gmail.com> wrote:
> > I think we should check STARTUP_SIGNALED even when the exit status is 0.
> > Something like [2].
>
> LGTM, thanks!
Thanks for the review, Fujii-san!
> This should be backpatched to all supported versions?
Yes, back till 15.
Regards,
Ayush
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: 041_checkpoint_at_promote.pl might fail due to race condition on child kill
@ 2026-11-18 04:44 Fujii Masao <masao.fujii@gmail.com>
parent: Ayush Tiwari <ayushtiwari.slg01@gmail.com>
0 siblings, 0 replies; 12+ messages in thread
From: Fujii Masao @ 2026-11-18 04:44 UTC (permalink / raw)
To: Ayush Tiwari <ayushtiwari.slg01@gmail.com>; +Cc: Alexander Lakhin <exclusion@gmail.com>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers
On Mon, Nov 16, 2026 at 3:50 PM Ayush Tiwari
<ayushtiwari.slg01@gmail.com> wrote:
> Yes, back till 15.
I've pushed the patch. Thanks!
Regards,
--
Fujii Masao
^ permalink raw reply [nested|flat] 12+ messages in thread
end of thread, other threads:[~2026-11-18 04:44 UTC | newest]
Thread overview: 12+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-10-07 04:00 041_checkpoint_at_promote.pl might fail due to race condition on child kill Alexander Lakhin <exclusion@gmail.com>
2026-10-07 16:46 ` Fujii Masao <masao.fujii@gmail.com>
2026-10-08 19:00 ` Alexander Lakhin <exclusion@gmail.com>
2026-10-13 07:17 ` Fujii Masao <masao.fujii@gmail.com>
2026-10-29 12:37 ` Fujii Masao <masao.fujii@gmail.com>
2026-10-30 06:58 ` Ayush Tiwari <ayushtiwari.slg01@gmail.com>
2026-10-30 11:55 ` Fujii Masao <masao.fujii@gmail.com>
2026-11-14 14:00 ` Alexander Lakhin <exclusion@gmail.com>
2026-11-16 05:26 ` Ayush Tiwari <ayushtiwari.slg01@gmail.com>
2026-11-16 06:19 ` Fujii Masao <masao.fujii@gmail.com>
2026-11-16 06:50 ` Ayush Tiwari <ayushtiwari.slg01@gmail.com>
2026-11-18 04:44 ` Fujii Masao <masao.fujii@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox