($INBOX_DIR/description missing)
help / color / mirror / Atom feedRe: [HACKERS] WAL logging problem in 9.4.3?
45+ messages / 5 participants
[nested] [flat]
* Re: [HACKERS] WAL logging problem in 9.4.3?
@ 2019-06-28 23:46 Amit Kapila <[email protected]>
0 siblings, 0 replies; 45+ messages in thread
From: Amit Kapila @ 2019-06-28 23:46 UTC (permalink / raw)
To: Kyotaro HORIGUCHI <[email protected]>; +Cc: pgsql-hackers; Dmitry Dolgov <[email protected]>; Andrew Dunstan <[email protected]>; hlinnaka <[email protected]>; Robert Haas <[email protected]>; Michael Paquier <[email protected]>; Noah Misch <[email protected]>
On Tue, May 28, 2019 at 4:33 AM Noah Misch <[email protected]> wrote:
>
> On Mon, May 27, 2019 at 02:08:26PM +0900, Kyotaro HORIGUCHI wrote:
> > At Fri, 24 May 2019 19:33:32 -0700, Noah Misch <[email protected]> wrote in <[email protected]>
> > > On Mon, May 20, 2019 at 03:54:30PM +0900, Kyotaro HORIGUCHI wrote:
> > > > Following this direction, the attached PoC works *at least for*
> > > > the wal_optimization TAP tests, but doing pending flush not in
> > > > smgr but in relcache.
> > >
> > > This task, syncing files created in the current transaction, is not the kind
> > > of task normally assigned to a cache. We already have a module, storage.c,
> > > that maintains state about files created in the current transaction. Why did
> > > you use relcache instead of storage.c?
> >
> > The reason was at-commit sync needs buffer flush beforehand. But
> > FlushRelationBufferWithoutRelCache() in v11 can do
> > that. storage.c is reasonable as the place.
>
> Okay. I do want this to work in 9.5 and later, but I'm not aware of a reason
> relcache.c would be a better code location in older branches. Unless you
> think of a reason to prefer relcache.c, please use storage.c.
>
> > > On Tue, May 21, 2019 at 09:29:48PM +0900, Kyotaro HORIGUCHI wrote:
> > > > This is a tidier version of the patch.
> > >
> > > > - Move the substantial work to table/index AMs.
> > > >
> > > > Each AM can decide whether to support WAL skip or not.
> > > > Currently heap and nbtree support it.
> > >
> > > Why would an AM find it important to disable WAL skip?
> >
> > The reason is currently it's AM's responsibility to decide
> > whether to skip WAL or not.
>
> I see. Skipping the sync would be a mere optimization; no AM would require it
> for correctness. An AM might want RelationNeedsWAL() to keep returning true
> despite the sync happening, perhaps because it persists data somewhere other
> than the forks of pg_class.relfilenode. Since the index and table APIs
> already assume one relfilenode captures all persistent data, I'm not seeing a
> use case for an AM overriding this behavior. Let's take away the AM's
> responsibility for this decision, making the system simpler. A future patch
> could let AM code decide, if someone find a real-world use case for
> AM-specific logic around when to skip WAL.
>
It seems there is some feedback for this patch and the CF is going to
start in 2 days. Are you planning to work on this patch for next CF,
if not then it is better to bump this? It is not a good idea to see
the patch in "waiting on author" in the beginning of CF unless the
author is actively working on the patch and is going to produce a
version in next few days.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: Race conditions in 019_replslot_limit.pl
@ 2022-02-17 02:14 Andres Freund <[email protected]>
2022-02-17 03:11 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
0 siblings, 1 reply; 45+ messages in thread
From: Andres Freund @ 2022-02-17 02:14 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; pgsql-hackers
Hi,
On 2022-02-16 20:22:23 -0500, Tom Lane wrote:
> There's no disconnection log entry for either, which I suppose means
> that somebody didn't bother logging disconnection for walsenders ...
The thing is, we actually *do* log disconnection for walsenders:
2022-02-16 17:53:55.780 PST [2459806][walsender][:0] LOG: disconnection: session time: 0:00:01.321 user=andres database= host=[local]
The only way I see for the disconnection not to be logged is an immediate
shutdown.
It'd be one thing if the missing walsender disconnect wasn't logged because we
shut down just after. But unless I misunderstand something, between the
basebackup and the failing test, we actually start standby_3:
# Running: pg_basebackup -D /home/bf/build/buildfarm-serinus/HEAD/pgsql.build/src/test/recovery/tmp_check/t_019_replslot_limit_primary3_data/backup/my_backup -h /tmp/hyrtnyd2RU -p 53774 --checkpoint fast --no-sync
# Backup finished
...
# Running: pg_ctl -w -D /home/bf/build/buildfarm-serinus/HEAD/pgsql.build/src/test/recovery/tmp_check/t_019_replslot_limit_standby_3_data/pgdata -l /home/bf/build/buildfarm-serinus/HEAD/pgsql.build/src/test/recovery/tmp_check/log/019_replslot_limit_standby_3.log -o --cluster-name=standby_3 start
and also wait till replication has caught up:
Waiting for replication conn standby_3's replay_lsn to pass 0/700000 on primary3
done
and only then we have
not ok 17 - have walsender pid 3682154
# 3682136
# Failed test 'have walsender pid 3682154
# 3682136'
# at t/019_replslot_limit.pl line 335.
# '3682154
# 3682136'
# doesn't match '(?^:^[0-9]+$)'
Starting a node in recovery and having it connect to the primary seems like a
mighty long time for a process to exit, unless it's stuck behind something.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: Race conditions in 019_replslot_limit.pl
2022-02-17 02:14 Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
@ 2022-02-17 03:11 ` Tom Lane <[email protected]>
2022-02-17 03:15 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-17 03:46 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
0 siblings, 2 replies; 45+ messages in thread
From: Tom Lane @ 2022-02-17 03:11 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; pgsql-hackers
Andres Freund <[email protected]> writes:
> On 2022-02-16 20:22:23 -0500, Tom Lane wrote:
>> There's no disconnection log entry for either, which I suppose means
>> that somebody didn't bother logging disconnection for walsenders ...
> The thing is, we actually *do* log disconnection for walsenders:
Ah, my mistake, now I do see a disconnection entry for the other walsender
launched by the basebackup.
> Starting a node in recovery and having it connect to the primary seems like a
> mighty long time for a process to exit, unless it's stuck behind something.
Fair point. Also, 019_replslot_limit.pl hasn't been changed in any
material way in months, but *something's* changed recently, because
this just started. I scraped the buildfarm for instances of
"Failed test 'have walsender pid" going back 6 months, and what I find is
sysname | branch | snapshot | stage | l
--------------+--------+---------------------+---------------+---------------------------------------------
desmoxytes | HEAD | 2022-02-15 04:42:05 | recoveryCheck | # Failed test 'have walsender pid 1685516
idiacanthus | HEAD | 2022-02-15 07:24:05 | recoveryCheck | # Failed test 'have walsender pid 2758549
serinus | HEAD | 2022-02-15 11:00:08 | recoveryCheck | # Failed test 'have walsender pid 3682154
desmoxytes | HEAD | 2022-02-15 11:04:05 | recoveryCheck | # Failed test 'have walsender pid 3775359
flaviventris | HEAD | 2022-02-15 18:03:48 | recoveryCheck | # Failed test 'have walsender pid 1517077
idiacanthus | HEAD | 2022-02-15 22:48:05 | recoveryCheck | # Failed test 'have walsender pid 2494972
desmoxytes | HEAD | 2022-02-15 23:48:04 | recoveryCheck | # Failed test 'have walsender pid 3055399
desmoxytes | HEAD | 2022-02-16 10:48:05 | recoveryCheck | # Failed test 'have walsender pid 1593461
komodoensis | HEAD | 2022-02-16 21:16:04 | recoveryCheck | # Failed test 'have walsender pid 3726703
serinus | HEAD | 2022-02-17 01:18:17 | recoveryCheck | # Failed test 'have walsender pid 208363
So (a) it broke around 48 hours ago, which is already a useful
bit of info, and (b) your animals seem far more susceptible than
anyone else's. Why do you suppose that is?
regards, tom lane
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: Race conditions in 019_replslot_limit.pl
2022-02-17 02:14 Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 03:11 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
@ 2022-02-17 03:15 ` Tom Lane <[email protected]>
1 sibling, 0 replies; 45+ messages in thread
From: Tom Lane @ 2022-02-17 03:15 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; pgsql-hackers
I wrote:
> So (a) it broke around 48 hours ago, which is already a useful
> bit of info, and (b) your animals seem far more susceptible than
> anyone else's. Why do you suppose that is?
Eyeballing the commit log for potential causes in that time frame,
I can't help noticing
2 days ago Andres Freund Move replication slot release to before_shmem_exit().
regards, tom lane
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: Race conditions in 019_replslot_limit.pl
2022-02-17 02:14 Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 03:11 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
@ 2022-02-17 03:46 ` Andres Freund <[email protected]>
2022-02-17 03:58 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-17 04:03 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
1 sibling, 2 replies; 45+ messages in thread
From: Andres Freund @ 2022-02-17 03:46 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; pgsql-hackers
Hi,
On 2022-02-16 22:11:30 -0500, Tom Lane wrote:
> So (a) it broke around 48 hours ago, which is already a useful
> bit of info
Indeed. 2f6501fa3c54bbe4568e3bcccd9a60d26a46b5ee seems like the obvious commit
to blame.
We document before_shmem_exit hooks as
/*
* Call before_shmem_exit callbacks.
*
* These should be things that need most of the system to still be up and
* working, such as cleanup of temp relations, which requires catalog
* access; or things that need to be completed because later cleanup steps
* depend on them, such as releasing lwlocks.
*/
and several before_shmem_exit callbacks use lwlocks.
But right now I'm not seeing what prevents us from throwing a FATAL error
while holding an lwlock?
> , and (b) your animals seem far more susceptible than
> anyone else's. Why do you suppose that is?
Flaviventris, serinus use the newest snapshot of gcc available in debian, one
with -O0, the other with O3.
desmoxytes, idiacanthus, komodoensis all have JIT forced for every query.
They all run on the same host - looking at stats it doesn't look crazily
overcommitted or such. But it might have more scheduler variance than most?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: Race conditions in 019_replslot_limit.pl
2022-02-17 02:14 Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 03:11 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-17 03:46 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
@ 2022-02-17 03:58 ` Tom Lane <[email protected]>
2022-02-17 04:32 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
1 sibling, 1 reply; 45+ messages in thread
From: Tom Lane @ 2022-02-17 03:58 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; pgsql-hackers
Andres Freund <[email protected]> writes:
> But right now I'm not seeing what prevents us from throwing a FATAL error
> while holding an lwlock?
If we're inside a transaction, then ShutdownPostgres ->
AbortOutOfAnyTransaction would release LWLocks. I wonder
though if an exiting walsender would always take that path.
Maybe it'd be good if we released all LWLocks even if we
don't think we're in a transaction?
More generally, it wasn't obvious to me why you thought that
BaseInit was a good spot to place the on_shmem_exit call.
It obviously can't be *before* that because of the pgstats
dependency, but maybe it should be later?
regards, tom lane
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: Race conditions in 019_replslot_limit.pl
2022-02-17 02:14 Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 03:11 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-17 03:46 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 03:58 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
@ 2022-02-17 04:32 ` Andres Freund <[email protected]>
0 siblings, 0 replies; 45+ messages in thread
From: Andres Freund @ 2022-02-17 04:32 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; pgsql-hackers
Hi,
On 2022-02-16 22:58:19 -0500, Tom Lane wrote:
> I wonder though if an exiting walsender would always take that path.
You're right - the CFI() in PostgresMain(), several replication commands, are
not at all guaranteed to be in a transaction when throwing a FATAL.
I don't quite see where we could be throwing an error with a relevant lwlock
held yet though.
> Maybe it'd be good if we released all LWLocks even if we don't think we're
> in a transaction?
Yea, probably a good idea. I think there's a few hooks could that could
conceivably be in trouble otherwise.
> More generally, it wasn't obvious to me why you thought that
> BaseInit was a good spot to place the on_shmem_exit call.
> It obviously can't be *before* that because of the pgstats
> dependency, but maybe it should be later?
Yea, I agree, it probably could be done later. It's not super obvious when
though. If e.g. somebody wrote a non-database connected bgworker to
synchronize slot state with another node (for failover, there's even patches
for it), it'd be just after BaseInit(). Additionally, it was previously in
ProcKill(), so moving it something registered early-ish seemed sensible.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: Race conditions in 019_replslot_limit.pl
2022-02-17 02:14 Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 03:11 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-17 03:46 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
@ 2022-02-17 04:03 ` Andres Freund <[email protected]>
2022-02-18 00:34 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
1 sibling, 1 reply; 45+ messages in thread
From: Andres Freund @ 2022-02-17 04:03 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; pgsql-hackers
Hi,
On 2022-02-16 19:46:26 -0800, Andres Freund wrote:
> But right now I'm not seeing what prevents us from throwing a FATAL error
> while holding an lwlock?
Nothing. But we register ShutdownPostgres() / ShutdownAuxiliaryProcess() to
take care of that.
I checked and walsenders do have ShutdownPostgres() and I don't see anything
else that's missing it either.
I'll run the test in a loop, perhaps I can reproduce...
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: Race conditions in 019_replslot_limit.pl
2022-02-17 02:14 Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 03:11 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-17 03:46 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 04:03 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
@ 2022-02-18 00:34 ` Andres Freund <[email protected]>
2022-02-18 03:41 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 04:11 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
0 siblings, 2 replies; 45+ messages in thread
From: Andres Freund @ 2022-02-18 00:34 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; pgsql-hackers
Hi,
On 2022-02-16 20:03:00 -0800, Andres Freund wrote:
> I'll run the test in a loop, perhaps I can reproduce...
I've done this for a couple hundred iterations, under load, subsequently with
additional assertions, without being able to reproduce.
There've not been any new reports in the last 18 hours, but that's probably
just due to lower commit activity triggering fewer runs.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: Race conditions in 019_replslot_limit.pl
2022-02-17 02:14 Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 03:11 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-17 03:46 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 04:03 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 00:34 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
@ 2022-02-18 03:41 ` Andres Freund <[email protected]>
2022-02-18 05:55 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
1 sibling, 1 reply; 45+ messages in thread
From: Andres Freund @ 2022-02-18 03:41 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; pgsql-hackers
Hi,
On 2022-02-17 16:34:34 -0800, Andres Freund wrote:
> I've done this for a couple hundred iterations, under load, subsequently with
> additional assertions, without being able to reproduce.
Playing around with this further I did get into one interesting state:
I started psql with replication=1, created a slot
SELECT pg_drop_replication_slot('t');SELECT pg_create_physical_replication_slot('t', true);
which was at 0/200002C8 at this point
START_REPLICATION SLOT "t" 0/200002C8 TIMELINE 1
then did
SELECT pg_switch_wal();checkpoint;
in another session, which tried to invalidate that slot because of
max_slot_wal_keep_size=1MB
At this point the psql "walsender" is *not* terminated, because the fatal
message is waiting behind all the WAL data sent. Which also means that the
slot isn't yet released.
At this point checkpointer is stuck, because it is waiting for the connection
to end.
Quitting the "psql walsender" or terminating it again resolves the issue.
Not sure how this could be related, but it seems interesting.
"psql walsender":
(gdb) bt
#0 0x00007faf4a570ec6 in epoll_wait (epfd=4, events=0x7faf4c201458, maxevents=1, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1 0x00007faf4b8ced5c in WaitEventSetWaitBlock (set=0x7faf4c2013e0, cur_timeout=-1, occurred_events=0x7ffe47df2320, nevents=1)
at /home/andres/src/postgresql/src/backend/storage/ipc/latch.c:1465
#2 0x00007faf4b8cebe3 in WaitEventSetWait (set=0x7faf4c2013e0, timeout=-1, occurred_events=0x7ffe47df2320, nevents=1, wait_event_info=100663297)
at /home/andres/src/postgresql/src/backend/storage/ipc/latch.c:1411
#3 0x00007faf4b70f48b in secure_write (port=0x7faf4c22da50, ptr=0x7faf4c2f1210, len=21470) at /home/andres/src/postgresql/src/backend/libpq/be-secure.c:298
#4 0x00007faf4b71aecb in internal_flush () at /home/andres/src/postgresql/src/backend/libpq/pqcomm.c:1380
#5 0x00007faf4b71ada1 in internal_putbytes (s=0x7ffe47df23dc "E\177", len=1) at /home/andres/src/postgresql/src/backend/libpq/pqcomm.c:1326
#6 0x00007faf4b71b0cf in socket_putmessage (msgtype=69 'E', s=0x7faf4c201700 "SFATAL", len=112)
at /home/andres/src/postgresql/src/backend/libpq/pqcomm.c:1507
#7 0x00007faf4b71c151 in pq_endmessage (buf=0x7ffe47df2460) at /home/andres/src/postgresql/src/backend/libpq/pqformat.c:301
#8 0x00007faf4babbb5e in send_message_to_frontend (edata=0x7faf4be2f1c0 <errordata>) at /home/andres/src/postgresql/src/backend/utils/error/elog.c:3253
#9 0x00007faf4bab8aa0 in EmitErrorReport () at /home/andres/src/postgresql/src/backend/utils/error/elog.c:1541
#10 0x00007faf4bab5ec0 in errfinish (filename=0x7faf4bc9770d "postgres.c", lineno=3192, funcname=0x7faf4bc99170 <__func__.8> "ProcessInterrupts")
at /home/andres/src/postgresql/src/backend/utils/error/elog.c:592
#11 0x00007faf4b907e73 in ProcessInterrupts () at /home/andres/src/postgresql/src/backend/tcop/postgres.c:3192
#12 0x00007faf4b8920af in WalSndLoop (send_data=0x7faf4b8927f2 <XLogSendPhysical>) at /home/andres/src/postgresql/src/backend/replication/walsender.c:2404
#13 0x00007faf4b88f82e in StartReplication (cmd=0x7faf4c293fc0) at /home/andres/src/postgresql/src/backend/replication/walsender.c:834
#14 0x00007faf4b89136f in exec_replication_command (cmd_string=0x7faf4c2073c0 "START_REPLICATION SLOT \"t\" 0/1B000000 TIMELINE 1\n;")
at /home/andres/src/postgresql/src/backend/replication/walsender.c:1771
#15 0x00007faf4b909842 in PostgresMain (dbname=0x7faf4c22fce8 "", username=0x7faf4c22fcc8 "andres")
at /home/andres/src/postgresql/src/backend/tcop/postgres.c:4494
#16 0x00007faf4b830e7e in BackendRun (port=0x7faf4c22da50) at /home/andres/src/postgresql/src/backend/postmaster/postmaster.c:4593
#17 0x00007faf4b83075f in BackendStartup (port=0x7faf4c22da50) at /home/andres/src/postgresql/src/backend/postmaster/postmaster.c:4321
#18 0x00007faf4b82c55b in ServerLoop () at /home/andres/src/postgresql/src/backend/postmaster/postmaster.c:1801
#19 0x00007faf4b82bd71 in PostmasterMain (argc=49, argv=0x7faf4c1ff1e0) at /home/andres/src/postgresql/src/backend/postmaster/postmaster.c:1473
checkpointer:
(gdb) bt
#0 0x00007f0d4ee1fec6 in epoll_wait (epfd=10, events=0x7f0d51489b78, maxevents=1, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1 0x00007f0d5017dd5c in WaitEventSetWaitBlock (set=0x7f0d51489b18, cur_timeout=-1, occurred_events=0x7fffe7c5f410, nevents=1)
at /home/andres/src/postgresql/src/backend/storage/ipc/latch.c:1465
#2 0x00007f0d5017dbe3 in WaitEventSetWait (set=0x7f0d51489b18, timeout=-1, occurred_events=0x7fffe7c5f410, nevents=1, wait_event_info=134217772)
at /home/andres/src/postgresql/src/backend/storage/ipc/latch.c:1411
#3 0x00007f0d5017cec4 in WaitLatch (latch=0x7f0d4c27db04, wakeEvents=33, timeout=-1, wait_event_info=134217772)
at /home/andres/src/postgresql/src/backend/storage/ipc/latch.c:472
#4 0x00007f0d5018fe34 in ConditionVariableTimedSleep (cv=0x7f0d4c342268, timeout=-1, wait_event_info=134217772)
at /home/andres/src/postgresql/src/backend/storage/lmgr/condition_variable.c:163
#5 0x00007f0d5018fd6c in ConditionVariableSleep (cv=0x7f0d4c342268, wait_event_info=134217772)
at /home/andres/src/postgresql/src/backend/storage/lmgr/condition_variable.c:100
#6 0x00007f0d50132525 in InvalidatePossiblyObsoleteSlot (s=0x7f0d4c342188, oldestLSN=553648128, invalidated=0x7fffe7c5f577)
at /home/andres/src/postgresql/src/backend/replication/slot.c:1290
#7 0x00007f0d501326af in InvalidateObsoleteReplicationSlots (oldestSegno=33) at /home/andres/src/postgresql/src/backend/replication/slot.c:1359
#8 0x00007f0d4fd84072 in CreateCheckPoint (flags=108) at /home/andres/src/postgresql/src/backend/access/transam/xlog.c:6620
#9 0x00007f0d500cd278 in CheckpointerMain () at /home/andres/src/postgresql/src/backend/postmaster/checkpointer.c:445
#10 0x00007f0d500cad7a in AuxiliaryProcessMain (auxtype=CheckpointerProcess) at /home/andres/src/postgresql/src/backend/postmaster/auxprocess.c:153
#11 0x00007f0d500e034c in StartChildProcess (type=CheckpointerProcess) at /home/andres/src/postgresql/src/backend/postmaster/postmaster.c:5531
#12 0x00007f0d500dacf1 in PostmasterMain (argc=49, argv=0x7f0d514891e0) at /home/andres/src/postgresql/src/backend/postmaster/postmaster.c:1458
#13 0x00007f0d4ffcfc19 in main (argc=49, argv=0x7f0d514891e0) at /home/andres/src/postgresql/src/backend/main/main.c:202
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: Race conditions in 019_replslot_limit.pl
2022-02-17 02:14 Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 03:11 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-17 03:46 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 04:03 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 00:34 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 03:41 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
@ 2022-02-18 05:55 ` Andres Freund <[email protected]>
2022-02-18 22:42 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
0 siblings, 1 reply; 45+ messages in thread
From: Andres Freund @ 2022-02-18 05:55 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; pgsql-hackers
Hi,
On 2022-02-17 19:41:40 -0800, Andres Freund wrote:
> At this point the psql "walsender" is *not* terminated, because the fatal
> message is waiting behind all the WAL data sent. Which also means that the
> slot isn't yet released.
Isn't it pretty bonkers that we allow error processing to get stuck behind
network traffic, *before* we have have released resources (locks etc)?
I wonder if we should try to send, but do it in a nonblocking way.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: Race conditions in 019_replslot_limit.pl
2022-02-17 02:14 Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 03:11 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-17 03:46 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 04:03 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 00:34 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 03:41 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 05:55 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
@ 2022-02-18 22:42 ` Andres Freund <[email protected]>
2022-02-18 23:14 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 23:15 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
0 siblings, 2 replies; 45+ messages in thread
From: Andres Freund @ 2022-02-18 22:42 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; pgsql-hackers
Hi,
On 2022-02-17 21:55:21 -0800, Andres Freund wrote:
> Isn't it pretty bonkers that we allow error processing to get stuck behind
> network traffic, *before* we have have released resources (locks etc)?
This is particularly likely to be a problem for walsenders, because they often
have a large output buffer filled, because walsender uses
pq_putmessage_noblock() to send WAL data. Which obviously can be large.
In the stacktrace upthread you can see:
#3 0x00007faf4b70f48b in secure_write (port=0x7faf4c22da50, ptr=0x7faf4c2f1210, len=21470) at /home/andres/src/postgresql/src/backend/libpq/be-secure.c:29
which certainly is more than in most other cases of error messages being
sent. And it obviously might not be the first to have gone out.
> I wonder if we should try to send, but do it in a nonblocking way.
I think we should probably do so at least during FATAL error processing. But
also consider doing so for ERROR, because not releasing resources after
getting cancelled / terminated is pretty nasty imo.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: Race conditions in 019_replslot_limit.pl
2022-02-17 02:14 Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 03:11 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-17 03:46 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 04:03 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 00:34 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 03:41 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 05:55 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 22:42 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
@ 2022-02-18 23:14 ` Andres Freund <[email protected]>
2022-02-23 02:06 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
1 sibling, 1 reply; 45+ messages in thread
From: Andres Freund @ 2022-02-18 23:14 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; pgsql-hackers
Hi,
On 2022-02-18 14:42:48 -0800, Andres Freund wrote:
> On 2022-02-17 21:55:21 -0800, Andres Freund wrote:
> > Isn't it pretty bonkers that we allow error processing to get stuck behind
> > network traffic, *before* we have have released resources (locks etc)?
>
> This is particularly likely to be a problem for walsenders, because they often
> have a large output buffer filled, because walsender uses
> pq_putmessage_noblock() to send WAL data. Which obviously can be large.
>
> In the stacktrace upthread you can see:
> #3 0x00007faf4b70f48b in secure_write (port=0x7faf4c22da50, ptr=0x7faf4c2f1210, len=21470) at /home/andres/src/postgresql/src/backend/libpq/be-secure.c:29
>
> which certainly is more than in most other cases of error messages being
> sent. And it obviously might not be the first to have gone out.
>
>
> > I wonder if we should try to send, but do it in a nonblocking way.
>
> I think we should probably do so at least during FATAL error processing. But
> also consider doing so for ERROR, because not releasing resources after
> getting cancelled / terminated is pretty nasty imo.
Is it possible that what we're seeing is a deadlock, with both walsender and
the pg_basebackup child trying to send data, but neither receiving?
But that'd require that somehow the basebackup child process didn't exit with
its parent. And I don't really see how that'd happen.
I'm running out of ideas for how to try to reproduce this. I think we might
need some additional debugging information to get more information from the
buildfarm.
I'm thinking of adding log_min_messages=DEBUG2 to primary3, passing --verbose
to pg_basebackup in $node_primary3->backup(...).
It might also be worth adding DEBUG2 messages to ReplicationSlotShmemExit(),
ReplicationSlotCleanup(), InvalidateObsoleteReplicationSlots().
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: Race conditions in 019_replslot_limit.pl
2022-02-17 02:14 Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 03:11 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-17 03:46 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 04:03 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 00:34 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 03:41 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 05:55 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 22:42 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 23:14 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
@ 2022-02-23 02:06 ` Andres Freund <[email protected]>
2022-02-25 19:29 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
0 siblings, 1 reply; 45+ messages in thread
From: Andres Freund @ 2022-02-23 02:06 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; pgsql-hackers
Hi,
I think I did find a bug related to the test, but afaics not the cause of the
test failures we're seeing. See
https://www.postgresql.org/message-id/20220223014855.4lsddr464i7mymk2%40alap3.anarazel.de
I don't think it's related to the problem of this thread, because the logs of
primary3 don't have a single mention of
ereport(LOG,
(errmsg("terminating process %d to release replication slot \"%s\"",
active_pid, NameStr(slotname))));
On 2022-02-18 15:14:15 -0800, Andres Freund wrote:
> I'm running out of ideas for how to try to reproduce this. I think we might
> need some additional debugging information to get more information from the
> buildfarm.
> I'm thinking of adding log_min_messages=DEBUG2 to primary3, passing --verbose
> to pg_basebackup in $node_primary3->backup(...).
>
> It might also be worth adding DEBUG2 messages to ReplicationSlotShmemExit(),
> ReplicationSlotCleanup(), InvalidateObsoleteReplicationSlots().
Planning to commit something like the attached.
Greetings,
Andres Freund
Attachments:
[text/x-diff] v1-0001-Add-temporary-debug-info-to-help-debug-019_replsl.patch (4.5K, ../../[email protected]/2-v1-0001-Add-temporary-debug-info-to-help-debug-019_replsl.patch)
download | inline diff:
From afdeff10526e29e3fc63b18c08100458780489d9 Mon Sep 17 00:00:00 2001
From: Andres Freund <[email protected]>
Date: Tue, 22 Feb 2022 18:02:34 -0800
Subject: [PATCH v1] Add temporary debug info to help debug
019_replslot_limit.pl failures.
I have not been able to reproduce the occasional failures of
019_replslot_limit.pl we are seeing in the buildfarm and not for lack of
trying. The additional logging and increased log level will hopefully help.
Will be reverted once the cause is identified.
Discussion: https://postgr.es/m/[email protected]
---
src/backend/replication/slot.c | 21 +++++++++++++++++++++
src/bin/pg_basebackup/pg_basebackup.c | 10 +++++++++-
src/test/recovery/t/019_replslot_limit.pl | 5 ++++-
3 files changed, 34 insertions(+), 2 deletions(-)
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 5da5fa825a2..3d39fddaaef 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -177,6 +177,10 @@ ReplicationSlotInitialize(void)
static void
ReplicationSlotShmemExit(int code, Datum arg)
{
+ /* temp debugging aid to analyze 019_replslot_limit failures */
+ elog(DEBUG3, "replication slot exit hook, %s active slot",
+ MyReplicationSlot != NULL ? "with" : "without");
+
/* Make sure active replication slots are released */
if (MyReplicationSlot != NULL)
ReplicationSlotRelease();
@@ -554,6 +558,9 @@ ReplicationSlotCleanup(void)
Assert(MyReplicationSlot == NULL);
restart:
+ /* temp debugging aid to analyze 019_replslot_limit failures */
+ elog(DEBUG3, "temporary replication slot cleanup: begin");
+
LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
for (i = 0; i < max_replication_slots; i++)
{
@@ -579,6 +586,8 @@ restart:
}
LWLockRelease(ReplicationSlotControlLock);
+
+ elog(DEBUG3, "temporary replication slot cleanup: done");
}
/*
@@ -1284,6 +1293,12 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlot *s, XLogRecPtr oldestLSN,
(void) kill(active_pid, SIGTERM);
last_signaled_pid = active_pid;
}
+ else
+ {
+ /* temp debugging aid to analyze 019_replslot_limit failures */
+ elog(DEBUG3, "not signalling process %d during invalidation of slot \"%s\"",
+ active_pid, NameStr(slotname));
+ }
/* Wait until the slot is released. */
ConditionVariableSleep(&s->active_cv,
@@ -1347,6 +1362,10 @@ InvalidateObsoleteReplicationSlots(XLogSegNo oldestSegno)
XLogSegNoOffsetToRecPtr(oldestSegno, 0, wal_segment_size, oldestLSN);
restart:
+ /* temp debugging aid to analyze 019_replslot_limit failures */
+ elog(DEBUG3, "begin invalidating obsolete replication slots older than %X/%X",
+ LSN_FORMAT_ARGS(oldestLSN));
+
LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
for (int i = 0; i < max_replication_slots; i++)
{
@@ -1372,6 +1391,8 @@ restart:
ReplicationSlotsComputeRequiredLSN();
}
+ elog(DEBUG3, "done invalidating obsolete replication slots");
+
return invalidated;
}
diff --git a/src/bin/pg_basebackup/pg_basebackup.c b/src/bin/pg_basebackup/pg_basebackup.c
index 08b07d5a06e..8c77c533e64 100644
--- a/src/bin/pg_basebackup/pg_basebackup.c
+++ b/src/bin/pg_basebackup/pg_basebackup.c
@@ -700,8 +700,16 @@ StartLogStreamer(char *startpos, uint32 timeline, char *sysidentifier)
bgchild = fork();
if (bgchild == 0)
{
+ int ret;
+
/* in child process */
- exit(LogStreamerMain(param));
+ ret = LogStreamerMain(param);
+
+ /* temp debugging aid to analyze 019_replslot_limit failures */
+ if (verbose)
+ pg_log_info("log streamer with pid %d exiting", getpid());
+
+ exit(ret);
}
else if (bgchild < 0)
{
diff --git a/src/test/recovery/t/019_replslot_limit.pl b/src/test/recovery/t/019_replslot_limit.pl
index 4257bd4d35a..0c9da9bf272 100644
--- a/src/test/recovery/t/019_replslot_limit.pl
+++ b/src/test/recovery/t/019_replslot_limit.pl
@@ -316,13 +316,16 @@ $node_primary3->append_conf(
max_wal_size = 2MB
log_checkpoints = yes
max_slot_wal_keep_size = 1MB
+
+ # temp debugging aid to analyze 019_replslot_limit failures
+ log_min_messages=debug3
));
$node_primary3->start;
$node_primary3->safe_psql('postgres',
"SELECT pg_create_physical_replication_slot('rep3')");
# Take backup
$backup_name = 'my_backup';
-$node_primary3->backup($backup_name);
+$node_primary3->backup($backup_name, backup_options => ['--verbose']);
# Create standby
my $node_standby3 = PostgreSQL::Test::Cluster->new('standby_3');
$node_standby3->init_from_backup($node_primary3, $backup_name,
--
2.34.0
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: Race conditions in 019_replslot_limit.pl
2022-02-17 02:14 Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 03:11 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-17 03:46 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 04:03 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 00:34 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 03:41 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 05:55 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 22:42 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 23:14 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-23 02:06 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
@ 2022-02-25 19:29 ` Andres Freund <[email protected]>
2022-02-25 20:07 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
0 siblings, 1 reply; 45+ messages in thread
From: Andres Freund @ 2022-02-25 19:29 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; pgsql-hackers
Hi,
On 2022-02-22 18:06:24 -0800, Andres Freund wrote:
> On 2022-02-18 15:14:15 -0800, Andres Freund wrote:
> > I'm running out of ideas for how to try to reproduce this. I think we might
> > need some additional debugging information to get more information from the
> > buildfarm.
>
> > I'm thinking of adding log_min_messages=DEBUG2 to primary3, passing --verbose
> > to pg_basebackup in $node_primary3->backup(...).
> >
> > It might also be worth adding DEBUG2 messages to ReplicationSlotShmemExit(),
> > ReplicationSlotCleanup(), InvalidateObsoleteReplicationSlots().
>
> Planning to commit something like the attached.
This did provide us a bit more detail.
Seems to suggest something is holding a problematic lock in a way that I do not understand yet:
https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=crake&dt=2022-02-23%2013%3A47%3A20...
2022-02-23 09:09:52.299 EST [2022-02-23 09:09:52 EST 1997084:6] 019_replslot_limit.pl LOG: received replication command: CREATE_REPLICATION_SLOT "pg_basebackup_1997084" TEMPORARY PHYSICAL ( RESERVE_WAL)
...
2022-02-23 09:09:52.518 EST [2022-02-23 09:09:52 EST 1997084:14] 019_replslot_limit.pl DEBUG: shmem_exit(0): 4 before_shmem_exit callbacks to make
2022-02-23 09:09:52.518 EST [2022-02-23 09:09:52 EST 1997084:15] 019_replslot_limit.pl DEBUG: replication slot exit hook, without active slot
2022-02-23 09:09:52.518 EST [2022-02-23 09:09:52 EST 1997084:16] 019_replslot_limit.pl DEBUG: temporary replication slot cleanup: begin
last message from 1997084 until the immediate shutdown. Just checking for
temporary replication slots that need to be dropped requires
ReplicationSlotControlLock. Actually dropping also requires
ReplicationSlotAllocationLock
1997084 does have to a temporary replication slot to clean up.
2022-02-23 09:09:52.519 EST [2022-02-23 09:09:52 EST 1997083:35] 019_replslot_limit.pl DEBUG: shmem_exit(0): 4 before_shmem_exit callbacks to make
2022-02-23 09:09:52.519 EST [2022-02-23 09:09:52 EST 1997083:36] 019_replslot_limit.pl DEBUG: replication slot exit hook, without active slot
2022-02-23 09:09:52.519 EST [2022-02-23 09:09:52 EST 1997083:37] 019_replslot_limit.pl DEBUG: temporary replication slot cleanup: begin
2022-02-23 09:09:52.519 EST [2022-02-23 09:09:52 EST 1997083:38] 019_replslot_limit.pl DEBUG: temporary replication slot cleanup: done
1997083 succeeds in doing the cleanup. It does not have a temporary
replication slot. Making it look like somehow ReplicationSlotAllocationLock
hasn't been released.
2022-02-23 09:09:52.519 EST [2022-02-23 09:09:52 EST 1997083:39] 019_replslot_limit.pl DEBUG: shmem_exit(0): 7 on_shmem_exit callbacks to make
...
2022-02-23 09:09:53.076 EST [2022-02-23 09:09:52 EST 1997072:87] LOG: received immediate shutdown request
...
2022-02-23 09:09:53.095 EST [2022-02-23 09:09:52 EST 1997072:90] DEBUG: server process (PID 1997084) exited with exit code 2
It's *possible*, but not likely, that somehow 1997084 just doesn't get
scheduled for a prolonged amount of time.
We could be more certain if we shut down the cluster in fast rather than
immediate mode. So I'm thinking of doing something like
# We've seen occasionales cases where multiple walsender pids are active. An
# immediate shutdown may hide evidence of a locking bug. So if multiple
# walsenders are observed, shut down in fast mode, and collect some more
# information.
if (not like($senderpid, qr/^[0-9]+$/, "have walsender pid $senderpid"))
{
my ($stdout, $stderr);
$node_primary3->psql('postgres',
"\\a\\t\nSELECT * FROM pg_stat_activity",
stdout => \$stdout, stderr => \$stderr);
diag $stdout, $stderr;
$node_primary3->stop('fast');
$node_standby3->stop('fast');
die "could not determine walsender pid, can't continue";
}
Does that make sense? Better ideas?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: Race conditions in 019_replslot_limit.pl
2022-02-17 02:14 Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 03:11 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-17 03:46 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 04:03 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 00:34 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 03:41 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 05:55 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 22:42 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 23:14 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-23 02:06 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-25 19:29 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
@ 2022-02-25 20:07 ` Tom Lane <[email protected]>
2022-02-25 20:15 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
0 siblings, 1 reply; 45+ messages in thread
From: Tom Lane @ 2022-02-25 20:07 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; pgsql-hackers
Andres Freund <[email protected]> writes:
> Seems to suggest something is holding a problematic lock in a way that I do not understand yet:
> https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=crake&dt=2022-02-23%2013%3A47%3A20...
> 2022-02-23 09:09:52.299 EST [2022-02-23 09:09:52 EST 1997084:6] 019_replslot_limit.pl LOG: received replication command: CREATE_REPLICATION_SLOT "pg_basebackup_1997084" TEMPORARY PHYSICAL ( RESERVE_WAL)
> ...
> 2022-02-23 09:09:52.518 EST [2022-02-23 09:09:52 EST 1997084:14] 019_replslot_limit.pl DEBUG: shmem_exit(0): 4 before_shmem_exit callbacks to make
> 2022-02-23 09:09:52.518 EST [2022-02-23 09:09:52 EST 1997084:15] 019_replslot_limit.pl DEBUG: replication slot exit hook, without active slot
> 2022-02-23 09:09:52.518 EST [2022-02-23 09:09:52 EST 1997084:16] 019_replslot_limit.pl DEBUG: temporary replication slot cleanup: begin
> last message from 1997084 until the immediate shutdown.
Hmm. Maybe put a couple more debug messages into ReplicationSlotCleanup
and/or ReplicationSlotDropPtr? It doesn't seem very clear where in that
sequence it's hanging up.
> We could be more certain if we shut down the cluster in fast rather than
> immediate mode. So I'm thinking of doing something like
Does that risk an indefinite hangup of the buildfarm run?
regards, tom lane
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: Race conditions in 019_replslot_limit.pl
2022-02-17 02:14 Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 03:11 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-17 03:46 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 04:03 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 00:34 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 03:41 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 05:55 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 22:42 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 23:14 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-23 02:06 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-25 19:29 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-25 20:07 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
@ 2022-02-25 20:15 ` Andres Freund <[email protected]>
2022-02-26 07:23 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
0 siblings, 1 reply; 45+ messages in thread
From: Andres Freund @ 2022-02-25 20:15 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; pgsql-hackers
Hi,
On 2022-02-25 15:07:01 -0500, Tom Lane wrote:
> Andres Freund <[email protected]> writes:
> > Seems to suggest something is holding a problematic lock in a way that I do not understand yet:
>
> > https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=crake&dt=2022-02-23%2013%3A47%3A20...
> > 2022-02-23 09:09:52.299 EST [2022-02-23 09:09:52 EST 1997084:6] 019_replslot_limit.pl LOG: received replication command: CREATE_REPLICATION_SLOT "pg_basebackup_1997084" TEMPORARY PHYSICAL ( RESERVE_WAL)
> > ...
> > 2022-02-23 09:09:52.518 EST [2022-02-23 09:09:52 EST 1997084:14] 019_replslot_limit.pl DEBUG: shmem_exit(0): 4 before_shmem_exit callbacks to make
> > 2022-02-23 09:09:52.518 EST [2022-02-23 09:09:52 EST 1997084:15] 019_replslot_limit.pl DEBUG: replication slot exit hook, without active slot
> > 2022-02-23 09:09:52.518 EST [2022-02-23 09:09:52 EST 1997084:16] 019_replslot_limit.pl DEBUG: temporary replication slot cleanup: begin
>
> > last message from 1997084 until the immediate shutdown.
>
> Hmm. Maybe put a couple more debug messages into ReplicationSlotCleanup
> and/or ReplicationSlotDropPtr? It doesn't seem very clear where in that
> sequence it's hanging up.
Yea, was thinking that as well.
I'm also wondering whether it's worth adding an assert, or at least a WARNING,
about no lwlocks held to the tail end of ShutdownPostgres? I don't want to
add an LWLockReleaseAll() yet, before I understand what's actually happening.
> > We could be more certain if we shut down the cluster in fast rather than
> > immediate mode. So I'm thinking of doing something like
>
> Does that risk an indefinite hangup of the buildfarm run?
I think not. The pg_ctl stop -m fast should time out after PGCTLTIMEOUT,
$self->_update_pid(-1); should notice it's not dead. The END{} block should
then shut it down in immediate mode.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: Race conditions in 019_replslot_limit.pl
2022-02-17 02:14 Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 03:11 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-17 03:46 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 04:03 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 00:34 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 03:41 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 05:55 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 22:42 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 23:14 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-23 02:06 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-25 19:29 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-25 20:07 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-25 20:15 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
@ 2022-02-26 07:23 ` Andres Freund <[email protected]>
2022-03-24 19:05 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
0 siblings, 1 reply; 45+ messages in thread
From: Andres Freund @ 2022-02-26 07:23 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; pgsql-hackers
Hi,
On 2022-02-25 12:15:58 -0800, Andres Freund wrote:
> On 2022-02-25 15:07:01 -0500, Tom Lane wrote:
> > Andres Freund <[email protected]> writes:
> > Hmm. Maybe put a couple more debug messages into ReplicationSlotCleanup
> > and/or ReplicationSlotDropPtr? It doesn't seem very clear where in that
> > sequence it's hanging up.
>
> Yea, was thinking that as well.
>
>
> I'm also wondering whether it's worth adding an assert, or at least a WARNING,
> about no lwlocks held to the tail end of ShutdownPostgres? I don't want to
> add an LWLockReleaseAll() yet, before I understand what's actually happening.
Did those things. Sure looks like there's some interaction with
checkpointer. There's a similar sequence in the two failures since the
additional debugging.
2022-02-26 06:35:52.453 CET [6219bc37.8717f:20] DEBUG: replication slot drop: pg_basebackup_553343: removed on-disk
2022-02-26 06:35:52.453 CET [6219bc37.87168:17] DEBUG: snapshot of 0+0 running transaction ids (lsn 0/700060 oldest xid 720 latest complete 719 next xid 720)
2022-02-26 06:35:52.453 CET [6219bc37.87168:18] DEBUG: begin invalidating obsolete replication slots older than 0/700000
2022-02-26 06:35:52.453 CET [6219bc37.87168:19] DEBUG: done invalidating obsolete replication slots
2022-02-26 06:35:52.453 CET [6219bc37.87168:20] DEBUG: attempting to remove WAL segments older than log file 000000000000000000000006
2022-02-26 06:35:52.453 CET [6219bc37.87168:21] DEBUG: removing write-ahead log file "000000010000000000000006"
2022-02-26 06:35:52.453 CET [6219bc37.87168:22] DEBUG: SlruScanDirectory invoking callback on pg_subtrans/0000
2022-02-26 06:35:52.453 CET [6219bc37.87168:23] LOG: checkpoint complete: wrote 0 buffers (0.0%); 0 WAL file(s) added, 1 removed, 0 recycled; write=0.001 s, sync=0.001 s, total=0.351 s; sync files=0, longest=0.000 s, average=0.000 s; distance=1024 kB, estimate=1024 kB
[...]
2022-02-26 06:35:52.970 CET [6219bc37.87139:71] LOG: received fast shutdown request
2022-02-26 06:35:52.970 CET [6219bc37.87139:72] LOG: aborting any active transactions
[...]
2022-02-26 06:35:52.971 CET [6219bc37.87168:24] LOG: shutting down
2022-02-26 06:35:52.976 CET [6219bc37.8717f:21] DEBUG: replication slot drop: pg_basebackup_553343: done
2022-02-26 06:35:52.976 CET [6219bc37.8717f:22] DEBUG: temporary replication slot cleanup: begin
2022-02-26 06:35:52.976 CET [6219bc37.8717f:23] DEBUG: temporary replication slot cleanup: 0 in use, active_pid: 553385
2022-02-26 06:35:52.976 CET [6219bc37.8717f:24] DEBUG: temporary replication slot cleanup: done
2022-02-26 06:35:52.977 CET [6219bc37.8717f:25] DEBUG: shmem_exit(0): 7 on_shmem_exit callbacks to make
2022-02-26 06:35:52.977 CET [6219bc37.8717f:26] DEBUG: proc_exit(0): 3 callbacks to make
2022-02-26 06:35:52.977 CET [6219bc37.8717f:27] LOG: disconnection: session time: 0:00:00.996 user=bf database= host=[local]
2022-02-26 06:35:52.977 CET [6219bc37.8717f:28] DEBUG: exit(0)
2022-02-26 06:35:52.977 CET [6219bc37.8717f:29] DEBUG: shmem_exit(-1): 0 before_shmem_exit callbacks to make
2022-02-26 06:35:52.977 CET [6219bc37.8717f:30] DEBUG: shmem_exit(-1): 0 on_shmem_exit callbacks to make
2022-02-26 06:35:52.977 CET [6219bc37.8717f:31] DEBUG: proc_exit(-1): 0 callbacks to make
So the backend (8717f/553343) is in the middle of ReplicationSlotDrop(), after
removing on-disk data. And then for 500ms nothing happens until checkpointer
wakes up again. As soon as it does, the slot drop continues.
Just before calling ReplicationSlotDrop() we were able to acquire
ReplicationSlotControlLock in share mode. Just after the log message after
which there's the delay, ReplicationSlotControlLock is locked in exclusive
mode.
Too tired now..
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: Race conditions in 019_replslot_limit.pl
2022-02-17 02:14 Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 03:11 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-17 03:46 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 04:03 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 00:34 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 03:41 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 05:55 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 22:42 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 23:14 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-23 02:06 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-25 19:29 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-25 20:07 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-25 20:15 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-26 07:23 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
@ 2022-03-24 19:05 ` Tom Lane <[email protected]>
2022-03-25 00:02 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
0 siblings, 1 reply; 45+ messages in thread
From: Tom Lane @ 2022-03-24 19:05 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; pgsql-hackers
I just noticed something very interesting: in a couple of recent
buildfarm runs with this failure, the pg_stat_activity printout
no longer shows the extra walsender:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=serinus&dt=2022-03-24%2017%3A50%3A10
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=xenodermus&dt=2022-03-23%2011%3A00%3A05
This is just two of the 33 such failures in the past ten days,
so maybe it's not surprising that we didn't see it already.
(I got bored before looking back further than that.)
What this suggests to me is that maybe the extra walsender is
indeed not blocked on anything, but is just taking its time
about exiting. In these two runs, as well as in all the
non-failing runs, it had enough time to do so.
I suggest that we add a couple-of-seconds sleep in front of
the query that collects walsender PIDs, and maybe a couple more
seconds before the pg_stat_activity probe in the failure path,
and see if the behavior changes at all. That should be enough
to confirm or disprove this idea pretty quickly. If it is
right, a permanent fix could be to wait for the basebackup's
walsender to disappear from node_primary3's pg_stat_activity
before we start the one for node_standby3.
regards, tom lane
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: Race conditions in 019_replslot_limit.pl
2022-02-17 02:14 Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 03:11 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-17 03:46 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 04:03 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 00:34 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 03:41 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 05:55 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 22:42 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 23:14 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-23 02:06 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-25 19:29 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-25 20:07 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-25 20:15 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-26 07:23 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-03-24 19:05 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
@ 2022-03-25 00:02 ` Andres Freund <[email protected]>
2022-03-25 00:08 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-03-27 19:28 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
0 siblings, 2 replies; 45+ messages in thread
From: Andres Freund @ 2022-03-25 00:02 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; pgsql-hackers
Hi,
On 2022-03-24 15:05:40 -0400, Tom Lane wrote:
> I just noticed something very interesting: in a couple of recent
> buildfarm runs with this failure, the pg_stat_activity printout
> no longer shows the extra walsender:
Oh. That is interesting. Thanks for catching that.
> What this suggests to me is that maybe the extra walsender is
> indeed not blocked on anything, but is just taking its time
> about exiting. In these two runs, as well as in all the
> non-failing runs, it had enough time to do so.
Still odd that it started with the bugfix I committed. And the locking pattern
in [1] still seems suspicious. But the obove does seem to suggest those might
just have been red herrings.
I had previously tried to make process exit slow by inserting delays in
various points and only succeeded in tests failing due too many
connections. For some reason I didn't increase max_connections...
I now increased the test's max_connections and managed to get a kind of
similar failure by sticking
elog(LOG, "received disconnect");
pg_usleep(220000);
into the 'X' case in PostgresMain().
However, I always see three pids or the expected one pid, not two like the BF
cases. But it's not too surprising that such an artificial, uniform, slowdown
would have different symptoms than reality.
> I suggest that we add a couple-of-seconds sleep in front of
> the query that collects walsender PIDs, and maybe a couple more
> seconds before the pg_stat_activity probe in the failure path,
> and see if the behavior changes at all.
Makes sense. I was previously thinking that it'd make sense to sleep for a bit
before the ->stop('fast'); calls, so that we can see whether it's just the
shutdown unblocking things (which [1] seems to suggest).
How about the attached variation, which retries (for 15s, with 100ms sleeps)
if there are multiple walsenders, printing the debugging info each time? It'll
still fail the test if later iterations find just one walsender, which seems
the right behaviour for now.
> That should be enough to confirm or disprove this idea pretty quickly. If
> it is right, a permanent fix could be to wait for the basebackup's walsender
> to disappear from node_primary3's pg_stat_activity before we start the one
> for node_standby3.
For some tests a "confimed disconnect" mode would be useful. Basically the
client waiting until it receives EOF after sending 'X'.
Greetings,
Andres Freund
[1] https://postgr.es/m/20220226072325.wtvkwvvga2wc3nkn%40alap3.anarazel.de
Attachments:
[text/x-diff] 019_replslot_limit_debug.diff (2.2K, ../../[email protected]/2-019_replslot_limit_debug.diff)
download | inline diff:
diff --git i/src/test/recovery/t/019_replslot_limit.pl w/src/test/recovery/t/019_replslot_limit.pl
index f62b7b32f66..77bb401bc5f 100644
--- i/src/test/recovery/t/019_replslot_limit.pl
+++ w/src/test/recovery/t/019_replslot_limit.pl
@@ -333,23 +333,41 @@ $node_standby3->init_from_backup($node_primary3, $backup_name,
$node_standby3->append_conf('postgresql.conf', "primary_slot_name = 'rep3'");
$node_standby3->start;
$node_primary3->wait_for_catchup($node_standby3);
-my $senderpid = $node_primary3->safe_psql('postgres',
- "SELECT pid FROM pg_stat_activity WHERE backend_type = 'walsender'");
-# We've seen occasional cases where multiple walsender pids are active. An
-# immediate shutdown may hide evidence of a locking bug. So if multiple
-# walsenders are observed, shut down in fast mode, and collect some more
-# information.
-if (not like($senderpid, qr/^[0-9]+$/, "have walsender pid $senderpid"))
+my $senderpid;
+
+# We've seen occasional cases where multiple walsender pids are active. It
+# could be that we're just observing process shutdown being slow. To collect
+# more information, retry a couple times, print a bit of debugging information
+# each iteration. For now report a test failure even if later iterations
+# succeed.
+my $i = 0;
+while (1)
{
my ($stdout, $stderr);
+
+ $senderpid = $node_primary3->safe_psql('postgres',
+ "SELECT pid FROM pg_stat_activity WHERE backend_type = 'walsender'");
+
+ last if like($senderpid, qr/^[0-9]+$/, "have walsender pid $senderpid");
+
+ # show information about all active connections
$node_primary3->psql('postgres',
"\\a\\t\nSELECT * FROM pg_stat_activity",
stdout => \$stdout, stderr => \$stderr);
diag $stdout, $stderr;
- $node_primary3->stop('fast');
- $node_standby3->stop('fast');
- die "could not determine walsender pid, can't continue";
+
+ # unlikely that the problem would resolve after 15s, so give up at point
+ if ($i++ == 150)
+ {
+ # An immediate shutdown may hide evidence of a locking bug. If
+ # retrying didn't resolve the issue, shut down in fast mode.
+ $node_primary3->stop('fast');
+ $node_standby3->stop('fast');
+ die "could not determine walsender pid, can't continue";
+ }
+
+ usleep(100_000);
}
my $receiverpid = $node_standby3->safe_psql('postgres',
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: Race conditions in 019_replslot_limit.pl
2022-02-17 02:14 Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 03:11 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-17 03:46 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 04:03 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 00:34 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 03:41 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 05:55 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 22:42 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 23:14 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-23 02:06 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-25 19:29 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-25 20:07 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-25 20:15 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-26 07:23 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-03-24 19:05 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-03-25 00:02 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
@ 2022-03-25 00:08 ` Tom Lane <[email protected]>
1 sibling, 0 replies; 45+ messages in thread
From: Tom Lane @ 2022-03-25 00:08 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; pgsql-hackers
Andres Freund <[email protected]> writes:
> How about the attached variation, which retries (for 15s, with 100ms sleeps)
> if there are multiple walsenders, printing the debugging info each time? It'll
> still fail the test if later iterations find just one walsender, which seems
> the right behaviour for now.
OK by me.
regards, tom lane
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: Race conditions in 019_replslot_limit.pl
2022-02-17 02:14 Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 03:11 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-17 03:46 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 04:03 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 00:34 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 03:41 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 05:55 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 22:42 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 23:14 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-23 02:06 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-25 19:29 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-25 20:07 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-25 20:15 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-26 07:23 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-03-24 19:05 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-03-25 00:02 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
@ 2022-03-27 19:28 ` Tom Lane <[email protected]>
2022-03-27 21:32 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
1 sibling, 1 reply; 45+ messages in thread
From: Tom Lane @ 2022-03-27 19:28 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; pgsql-hackers
Andres Freund <[email protected]> writes:
> How about the attached variation, which retries (for 15s, with 100ms sleeps)
> if there are multiple walsenders, printing the debugging info each time? It'll
> still fail the test if later iterations find just one walsender, which seems
> the right behaviour for now.
We have now four instances of failures with this version of the test,
and in every case the second iteration succeeded. Is that enough
evidence yet?
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=crake&dt=2022-03-27%2017%3A04%3A18
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=flaviventris&dt=2022-03-25%2009%3A00%3A0...
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=desmoxytes&dt=2022-03-25%2008%3A02%3A05
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=flaviventris&dt=2022-03-25%2003%3A00%3A1...
I'd like to silence this noise so that we can start tracking
lower-probability failure modes, like say these:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=olingo&dt=2022-03-26%2002%3A59%3A03
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=skink&dt=2022-03-26%2015%3A53%3A51
regards, tom lane
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: Race conditions in 019_replslot_limit.pl
2022-02-17 02:14 Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 03:11 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-17 03:46 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 04:03 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 00:34 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 03:41 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 05:55 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 22:42 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 23:14 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-23 02:06 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-25 19:29 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-25 20:07 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-25 20:15 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-26 07:23 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-03-24 19:05 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-03-25 00:02 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-03-27 19:28 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
@ 2022-03-27 21:32 ` Andres Freund <[email protected]>
2022-03-27 21:36 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
0 siblings, 1 reply; 45+ messages in thread
From: Andres Freund @ 2022-03-27 21:32 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; pgsql-hackers
Hi,
On 2022-03-27 15:28:05 -0400, Tom Lane wrote:
> Andres Freund <[email protected]> writes:
> > How about the attached variation, which retries (for 15s, with 100ms sleeps)
> > if there are multiple walsenders, printing the debugging info each time? It'll
> > still fail the test if later iterations find just one walsender, which seems
> > the right behaviour for now.
>
> We have now four instances of failures with this version of the test,
> and in every case the second iteration succeeded. Is that enough
> evidence yet?
I still feel like there's something off here. But that's probably not enough
to keep causing failures. I'm inclined to leave the debugging in for a bit
longer, but not fail the test anymore?
The way the temporary slot removal hangs for a while seems just off:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=crake&dt=2022-03-27%2017%3A04%3A18
2022-03-27 13:30:56.993 EDT [2022-03-27 13:30:56 EDT 750695:20] 019_replslot_limit.pl DEBUG: replication slot drop: pg_basebackup_750695: removed on-disk
...
2022-03-27 13:30:57.456 EDT [2022-03-27 13:30:57 EDT 750759:3] [unknown] LOG: connection authorized: user=andrew database=postgres application_name=019_replslot_limit.pl
2022-03-27 13:30:57.466 EDT [2022-03-27 13:30:57 EDT 750759:4] 019_replslot_limit.pl LOG: statement: SELECT * FROM pg_stat_activity
.
2022-03-27 13:30:57.474 EDT [2022-03-27 13:30:56 EDT 750679:87] DEBUG: server process (PID 750759) exited with exit code 0
2022-03-27 13:30:57.507 EDT [2022-03-27 13:30:56 EDT 750695:21] 019_replslot_limit.pl DEBUG: replication slot drop: pg_basebackup_750695: done
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=flaviventris&dt=2022-03-25%2009%3A00%3A0...
2022-03-25 10:13:30.364 CET [4022819][walsender][4/0:0][019_replslot_limit.pl] DEBUG: replication slot drop: pg_basebackup_4022819: begin
2022-03-25 10:13:30.364 CET [4022819][walsender][4/0:0][019_replslot_limit.pl] DEBUG: replication slot drop: pg_basebackup_4022819: removed on-disk
...
2022-03-25 10:13:31.038 CET [4022841][client backend][5/7:0][[unknown]] LOG: connection authorized: user=bf database=postgres application_name=019_replslot_limit.pl
2022-03-25 10:13:31.039 CET [4022841][client backend][5/8:0][019_replslot_limit.pl] LOG: statement: SELECT * FROM pg_stat_activity
...
2022-03-25 10:13:31.045 CET [4022807][postmaster][:0][] DEBUG: server process (PID 4022841) exited with exit code 0
2022-03-25 10:13:31.056 CET [4022819][walsender][4/0:0][019_replslot_limit.pl] DEBUG: replication slot drop: pg_basebackup_4022819: done
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=desmoxytes&dt=2022-03-25%2008%3A02%3A05
2022-03-25 09:15:20.558 CET [3730425][walsender][4/0:0][019_replslot_limit.pl] DEBUG: replication slot drop: pg_basebackup_3730425: removed on-disk
...
2022-03-25 09:15:20.803 CET [3730461][client backend][5/7:0][[unknown]] LOG: connection authorized: user=bf database=postgres application_name=019_replslot_limit.pl
2022-03-25 09:15:20.804 CET [3730461][client backend][5/8:0][019_replslot_limit.pl] LOG: statement: SELECT * FROM pg_stat_activity
...
2022-03-25 09:15:20.834 CET [3730381][postmaster][:0][] DEBUG: server process (PID 3730461) exited with exit code 0
2022-03-25 09:15:20.861 CET [3730425][walsender][4/0:0][019_replslot_limit.pl] DEBUG: replication slot drop: pg_basebackup_3730425: done
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=flaviventris&dt=2022-03-25%2003%3A00%3A1...
2022-03-25 04:14:03.025 CET [2674398][walsender][4/0:0][019_replslot_limit.pl] DEBUG: replication slot drop: pg_basebackup_2674398: removed on-disk
...
2022-03-25 04:14:03.264 CET [2674463][client backend][5/7:0][[unknown]] LOG: connection authorized: user=bf database=postgres application_name=019_replslot_limit.pl
2022-03-25 04:14:03.265 CET [2674463][client backend][5/8:0][019_replslot_limit.pl] LOG: statement: SELECT * FROM pg_stat_activity
...
2022-03-25 04:14:03.270 CET [2674384][postmaster][:0][] DEBUG: server process (PID 2674463) exited with exit code 0
...
2022-03-25 04:14:03.324 CET [2674398][walsender][4/0:0][019_replslot_limit.pl] DEBUG: replication slot drop: pg_basebackup_2674398: done
We are able to start / finish several new connections between the two debug
elog()sin ReplicationSlotDropPtr()?
I wonder if there's something odd going on with ConditionVariableBroadcast().
Might be useful to add another debug message before/after
ConditionVariableBroadcast() and rmtree(). If the delay is due to rmtree()
being slow under concurrent tests I'd feel less concerned (although that
machine has dual NVMe drives...).
I really wish I could reproduce the failure. I went through a few hundred
cycles of that test in a separate checkout on that machine.
> I'd like to silence this noise so that we can start tracking
> lower-probability failure modes, like say these:
>
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=olingo&dt=2022-03-26%2002%3A59%3A03
That one was a missing compiler flag. I see that failure reproducibly locally
when using asan with clang, unless I use -fsanitize-address-use-after-return=never. gcc has a different default
for the option, which is why I hadn't configured it.
If I understand correctly, the problem is that
-fsanitize-address-use-after-return uses an alternative stack. Sometimes our
stack depths functions get called with the "proper" stack, and sometimes with
a "shadow" stack. Which breaks our stack depth checking.
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=skink&dt=2022-03-26%2015%3A53%3A51
This one I have no idea about yet. I assume it's just a race in a new test...
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: Race conditions in 019_replslot_limit.pl
2022-02-17 02:14 Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 03:11 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-17 03:46 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 04:03 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 00:34 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 03:41 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 05:55 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 22:42 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 23:14 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-23 02:06 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-25 19:29 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-25 20:07 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-25 20:15 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-26 07:23 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-03-24 19:05 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-03-25 00:02 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-03-27 19:28 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-03-27 21:32 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
@ 2022-03-27 21:36 ` Tom Lane <[email protected]>
2022-03-28 05:37 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
0 siblings, 1 reply; 45+ messages in thread
From: Tom Lane @ 2022-03-27 21:36 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; pgsql-hackers
Andres Freund <[email protected]> writes:
> I still feel like there's something off here. But that's probably not enough
> to keep causing failures. I'm inclined to leave the debugging in for a bit
> longer, but not fail the test anymore?
WFM.
> The way the temporary slot removal hangs for a while seems just off:
Perhaps, but right now we're causing noise in the buildfarm and
learning nothing.
regards, tom lane
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: Race conditions in 019_replslot_limit.pl
2022-02-17 02:14 Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 03:11 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-17 03:46 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 04:03 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 00:34 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 03:41 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 05:55 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 22:42 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 23:14 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-23 02:06 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-25 19:29 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-25 20:07 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-25 20:15 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-26 07:23 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-03-24 19:05 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-03-25 00:02 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-03-27 19:28 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-03-27 21:32 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-03-27 21:36 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
@ 2022-03-28 05:37 ` Andres Freund <[email protected]>
2022-05-30 19:01 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
0 siblings, 1 reply; 45+ messages in thread
From: Andres Freund @ 2022-03-28 05:37 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; pgsql-hackers
Hi,
On 2022-03-27 17:36:14 -0400, Tom Lane wrote:
> Andres Freund <[email protected]> writes:
> > I still feel like there's something off here. But that's probably not enough
> > to keep causing failures. I'm inclined to leave the debugging in for a bit
> > longer, but not fail the test anymore?
>
> WFM.
I've done so now.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: Race conditions in 019_replslot_limit.pl
2022-02-17 02:14 Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 03:11 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-17 03:46 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 04:03 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 00:34 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 03:41 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 05:55 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 22:42 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 23:14 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-23 02:06 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-25 19:29 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-25 20:07 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-25 20:15 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-26 07:23 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-03-24 19:05 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-03-25 00:02 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-03-27 19:28 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-03-27 21:32 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-03-27 21:36 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-03-28 05:37 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
@ 2022-05-30 19:01 ` Andres Freund <[email protected]>
2022-05-31 01:31 ` Re: Race conditions in 019_replslot_limit.pl Kyotaro Horiguchi <[email protected]>
0 siblings, 1 reply; 45+ messages in thread
From: Andres Freund @ 2022-05-30 19:01 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; pgsql-hackers
Hi,
On 2022-03-27 22:37:34 -0700, Andres Freund wrote:
> On 2022-03-27 17:36:14 -0400, Tom Lane wrote:
> > Andres Freund <[email protected]> writes:
> > > I still feel like there's something off here. But that's probably not enough
> > > to keep causing failures. I'm inclined to leave the debugging in for a bit
> > > longer, but not fail the test anymore?
> >
> > WFM.
>
> I've done so now.
I did look over the test results a couple times since then and once more
today. There were a few cases with pretty significant numbers of iterations:
The highest is
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=dragonet&dt=2022-04-07%2022%3A14%3A03
showing:
# multiple walsenders active in iteration 19
It's somewhat interesting that the worst case was just around the feature
freeze, where the load on my buildfarm animal boxes was higher than normal.
I comparison to earlier approaches, with the current in-tree approach, we
don't do anything when hitting the "problem", other than wait. Which does give
us additional information - afaics there's nothing at all indicating that some
other backend existed allowing the replication slot drop to finish.
It just looks like for reasons I still do not understand, removing a directory
and 2 files or so takes multiple seconds (at least ~36 new connections, 18
pg_usleep(100_100)), while there are no other indications of problems.
I also still don't have a theory why this suddenly started to happen.
Unless somebody has another idea, I'm planning to remove all the debugging
code added, but keep the retry based approach in 019_replslot_limit.pl, so we
don't again get all the spurious failures.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: Race conditions in 019_replslot_limit.pl
2022-02-17 02:14 Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 03:11 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-17 03:46 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 04:03 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 00:34 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 03:41 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 05:55 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 22:42 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 23:14 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-23 02:06 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-25 19:29 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-25 20:07 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-25 20:15 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-26 07:23 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-03-24 19:05 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-03-25 00:02 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-03-27 19:28 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-03-27 21:32 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-03-27 21:36 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-03-28 05:37 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-05-30 19:01 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
@ 2022-05-31 01:31 ` Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 45+ messages in thread
From: Kyotaro Horiguchi @ 2022-05-31 01:31 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; pgsql-hackers
At Mon, 30 May 2022 12:01:55 -0700, Andres Freund <[email protected]> wrote in
> Hi,
>
> On 2022-03-27 22:37:34 -0700, Andres Freund wrote:
> > On 2022-03-27 17:36:14 -0400, Tom Lane wrote:
> > > Andres Freund <[email protected]> writes:
> > > > I still feel like there's something off here. But that's probably not enough
> > > > to keep causing failures. I'm inclined to leave the debugging in for a bit
> > > > longer, but not fail the test anymore?
> > >
> > > WFM.
> >
> > I've done so now.
>
> I did look over the test results a couple times since then and once more
> today. There were a few cases with pretty significant numbers of iterations:
>
> The highest is
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=dragonet&dt=2022-04-07%2022%3A14%3A03
> showing:
> # multiple walsenders active in iteration 19
>
> It's somewhat interesting that the worst case was just around the feature
> freeze, where the load on my buildfarm animal boxes was higher than normal.
If disk is too busy, CheckPointReplicationSlots may take very long.
> I comparison to earlier approaches, with the current in-tree approach, we
> don't do anything when hitting the "problem", other than wait. Which does give
> us additional information - afaics there's nothing at all indicating that some
> other backend existed allowing the replication slot drop to finish.
preventing? Only checkpointer and a client backend that ran "SELECT * FROM
pg_stat_activity" are the only processes that are running during the
blocking state.
> It just looks like for reasons I still do not understand, removing a
directory
> and 2 files or so takes multiple seconds (at least ~36 new connections, 18
> pg_usleep(100_100)), while there are no other indications of problems.
That fact suports that CheckPointReplicationSlots took long time.
> I also still don't have a theory why this suddenly started to happen.
Maybe we need to see the load of disks at that time OS-wide. Couldn't
compiler or other non-postgres tools put significant load to disks?
> Unless somebody has another idea, I'm planning to remove all the debugging
> code added, but keep the retry based approach in 019_replslot_limit.pl, so we
> don't again get all the spurious failures.
+1.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: Race conditions in 019_replslot_limit.pl
2022-02-17 02:14 Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 03:11 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-17 03:46 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 04:03 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 00:34 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 03:41 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 05:55 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 22:42 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
@ 2022-02-18 23:15 ` Tom Lane <[email protected]>
2022-02-18 23:40 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
1 sibling, 1 reply; 45+ messages in thread
From: Tom Lane @ 2022-02-18 23:15 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; pgsql-hackers
Andres Freund <[email protected]> writes:
> On 2022-02-17 21:55:21 -0800, Andres Freund wrote:
>> Isn't it pretty bonkers that we allow error processing to get stuck behind
>> network traffic, *before* we have have released resources (locks etc)?
It's more or less intentional, per elog.c:
/*
* This flush is normally not necessary, since postgres.c will flush out
* waiting data when control returns to the main loop. But it seems best
* to leave it here, so that the client has some clue what happened if the
* backend dies before getting back to the main loop ... error/notice
* messages should not be a performance-critical path anyway, so an extra
* flush won't hurt much ...
*/
pq_flush();
Perhaps it'd be sensible to do this only in debugging (ie Assert)
builds?
regards, tom lane
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: Race conditions in 019_replslot_limit.pl
2022-02-17 02:14 Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 03:11 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-17 03:46 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 04:03 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 00:34 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 03:41 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 05:55 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 22:42 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 23:15 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
@ 2022-02-18 23:40 ` Andres Freund <[email protected]>
2022-02-18 23:49 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
0 siblings, 1 reply; 45+ messages in thread
From: Andres Freund @ 2022-02-18 23:40 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; pgsql-hackers
Hi,
On 2022-02-18 18:15:21 -0500, Tom Lane wrote:
> Andres Freund <[email protected]> writes:
> > On 2022-02-17 21:55:21 -0800, Andres Freund wrote:
> >> Isn't it pretty bonkers that we allow error processing to get stuck behind
> >> network traffic, *before* we have have released resources (locks etc)?
>
> It's more or less intentional, per elog.c:
>
> /*
> * This flush is normally not necessary, since postgres.c will flush out
> * waiting data when control returns to the main loop. But it seems best
> * to leave it here, so that the client has some clue what happened if the
> * backend dies before getting back to the main loop ... error/notice
> * messages should not be a performance-critical path anyway, so an extra
> * flush won't hurt much ...
> */
> pq_flush();
>
> Perhaps it'd be sensible to do this only in debugging (ie Assert)
> builds?
That seems not great, because it pretty clearly can lead to hangs, which is
problematic in tests too. What about using pq_flush_if_writable()? In nearly
all situations that'd still push the failure to the client.
We'd also need to add pq_endmessage_noblock(), because the pq_endmessage()
obviously tries to send (as in the backtrace upthread) if the output buffer is
large enough, which it often will be in walsender.
I guess we could try to flush in a blocking manner sometime later in the
shutdown sequence, after we've released resources? But I'm doubtful it's a
good idea, we don't really want to block waiting to exit when e.g. the network
connection is dead without the TCP stack knowing.
Hm. There already is code trying to short-circuit sending errors to the client
if a backend gets terminated. Introduced in
commit 2ddb9149d14de9a2e7ac9ec6accf3ad442702b24
Author: Tom Lane <[email protected]>
Date: 2018-10-19 21:39:21 -0400
Server-side fix for delayed NOTIFY and SIGTERM processing.
and predecessors.
If ProcessClientWriteInterrupt() sees ProcDiePending, we'll stop trying to
send stuff to the client if writes block. However, this doesn't work here,
because we've already unset ProcDiePending:
#7 0x00007faf4b71c151 in pq_endmessage (buf=0x7ffe47df2460) at /home/andres/src/postgresql/src/backend/libpq/pqformat.c:301
#8 0x00007faf4babbb5e in send_message_to_frontend (edata=0x7faf4be2f1c0 <errordata>) at /home/andres/src/postgresql/src/backend/utils/error/elog.c:3253
#9 0x00007faf4bab8aa0 in EmitErrorReport () at /home/andres/src/postgresql/src/backend/utils/error/elog.c:1541
#10 0x00007faf4bab5ec0 in errfinish (filename=0x7faf4bc9770d "postgres.c", lineno=3192, funcname=0x7faf4bc99170 <__func__.8> "ProcessInterrupts")
at /home/andres/src/postgresql/src/backend/utils/error/elog.c:592
#11 0x00007faf4b907e73 in ProcessInterrupts () at /home/andres/src/postgresql/src/backend/tcop/postgres.c:3192
#12 0x00007faf4b8920af in WalSndLoop (send_data=0x7faf4b8927f2 <XLogSendPhysical>) at /home/andres/src/postgresql/src/backend/replication/walsender.c:2404
#13 0x00007faf4b88f82e in StartReplication (cmd=0x7faf4c293fc0) at /home/andres/src/postgresql/src/backend/replication/walsender.c:834
Before ProcessInterrupts() FATALs due to a SIGTERM, it resets ProcDiePending.
This seems not optimal.
We can't just leave ProcDiePending set, otherwise we'll probably end up
throwing more errors during the shutdown sequence. But it seems we need
something similar to proc_exit_inprogress, except set earlier? And then take
that into account in ProcessClientWriteInterrupt()?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: Race conditions in 019_replslot_limit.pl
2022-02-17 02:14 Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 03:11 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-17 03:46 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 04:03 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 00:34 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 03:41 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 05:55 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 22:42 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 23:15 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-18 23:40 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
@ 2022-02-18 23:49 ` Tom Lane <[email protected]>
2022-02-19 00:08 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
0 siblings, 1 reply; 45+ messages in thread
From: Tom Lane @ 2022-02-18 23:49 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; pgsql-hackers
Andres Freund <[email protected]> writes:
> On 2022-02-18 18:15:21 -0500, Tom Lane wrote:
>> Perhaps it'd be sensible to do this only in debugging (ie Assert)
>> builds?
> That seems not great, because it pretty clearly can lead to hangs, which is
> problematic in tests too. What about using pq_flush_if_writable()? In nearly
> all situations that'd still push the failure to the client.
That'd be okay by me.
> We'd also need to add pq_endmessage_noblock(), because the pq_endmessage()
> obviously tries to send (as in the backtrace upthread) if the output buffer is
> large enough, which it often will be in walsender.
I don't see that as "obvious". If we're there, we do not have an
error situation.
> I guess we could try to flush in a blocking manner sometime later in the
> shutdown sequence, after we've released resources? But I'm doubtful it's a
> good idea, we don't really want to block waiting to exit when e.g. the network
> connection is dead without the TCP stack knowing.
I think you are trying to move in the direction of possibly exiting
without ever sending at all, which does NOT seem like an improvement.
regards, tom lane
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: Race conditions in 019_replslot_limit.pl
2022-02-17 02:14 Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 03:11 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-17 03:46 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 04:03 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 00:34 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 03:41 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 05:55 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 22:42 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 23:15 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-18 23:40 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 23:49 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
@ 2022-02-19 00:08 ` Andres Freund <[email protected]>
0 siblings, 0 replies; 45+ messages in thread
From: Andres Freund @ 2022-02-19 00:08 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; pgsql-hackers
Hi,
On 2022-02-18 18:49:14 -0500, Tom Lane wrote:
> Andres Freund <[email protected]> writes:
> > We'd also need to add pq_endmessage_noblock(), because the pq_endmessage()
> > obviously tries to send (as in the backtrace upthread) if the output buffer is
> > large enough, which it often will be in walsender.
>
> I don't see that as "obvious". If we're there, we do not have an
> error situation.
The problem is that due walsender using pq_putmessage_noblock(), the output
buffer is often going to be too full for a plain pq_endmessage() to not send
out data. Because walsender will have an output buffer > PQ_SEND_BUFFER_SIZE
a lot of the time, errors will commonly be thrown with the output buffer
full.
That leads the pq_endmessage() in send_message_to_frontend() to block sending
the error message and the preceding WAL data. Leading to backtraces like:
> #0 0x00007faf4a570ec6 in epoll_wait (epfd=4, events=0x7faf4c201458, maxevents=1, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
> #1 0x00007faf4b8ced5c in WaitEventSetWaitBlock (set=0x7faf4c2013e0, cur_timeout=-1, occurred_events=0x7ffe47df2320, nevents=1)
> at /home/andres/src/postgresql/src/backend/storage/ipc/latch.c:1465
> #2 0x00007faf4b8cebe3 in WaitEventSetWait (set=0x7faf4c2013e0, timeout=-1, occurred_events=0x7ffe47df2320, nevents=1, wait_event_info=100663297)
> at /home/andres/src/postgresql/src/backend/storage/ipc/latch.c:1411
> #3 0x00007faf4b70f48b in secure_write (port=0x7faf4c22da50, ptr=0x7faf4c2f1210, len=21470) at /home/andres/src/postgresql/src/backend/libpq/be-secure.c:298
> #4 0x00007faf4b71aecb in internal_flush () at /home/andres/src/postgresql/src/backend/libpq/pqcomm.c:1380
> #5 0x00007faf4b71ada1 in internal_putbytes (s=0x7ffe47df23dc "E\177", len=1) at /home/andres/src/postgresql/src/backend/libpq/pqcomm.c:1326
> #6 0x00007faf4b71b0cf in socket_putmessage (msgtype=69 'E', s=0x7faf4c201700 "SFATAL", len=112)
> at /home/andres/src/postgresql/src/backend/libpq/pqcomm.c:1507
> #7 0x00007faf4b71c151 in pq_endmessage (buf=0x7ffe47df2460) at /home/andres/src/postgresql/src/backend/libpq/pqformat.c:301
> #8 0x00007faf4babbb5e in send_message_to_frontend (edata=0x7faf4be2f1c0 <errordata>) at /home/andres/src/postgresql/src/backend/utils/error/elog.c:3253
> #9 0x00007faf4bab8aa0 in EmitErrorReport () at /home/andres/src/postgresql/src/backend/utils/error/elog.c:1541
> #10 0x00007faf4bab5ec0 in errfinish (filename=0x7faf4bc9770d "postgres.c", lineno=3192, funcname=0x7faf4bc99170 <__func__.8> "ProcessInterrupts")
> at /home/andres/src/postgresql/src/backend/utils/error/elog.c:592
> #11 0x00007faf4b907e73 in ProcessInterrupts () at /home/andres/src/postgresql/src/backend/tcop/postgres.c:3192
> #12 0x00007faf4b8920af in WalSndLoop (send_data=0x7faf4b8927f2 <XLogSendPhysical>) at /home/andres/src/postgresql/src/backend/replication/walsender.c:2404
> > I guess we could try to flush in a blocking manner sometime later in the
> > shutdown sequence, after we've released resources? But I'm doubtful it's a
> > good idea, we don't really want to block waiting to exit when e.g. the network
> > connection is dead without the TCP stack knowing.
>
> I think you are trying to move in the direction of possibly exiting
> without ever sending at all, which does NOT seem like an improvement.
I was just talking about doing another blocking pq_flush(), in addition to the
pq_flush_if_writable() earlier. That'd be trying harder to send out data, not
less hard...
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 45+ messages in thread
* Re: Race conditions in 019_replslot_limit.pl
2022-02-17 02:14 Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 03:11 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-17 03:46 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 04:03 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 00:34 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
@ 2022-02-18 04:11 ` Tom Lane <[email protected]>
1 sibling, 0 replies; 45+ messages in thread
From: Tom Lane @ 2022-02-18 04:11 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; pgsql-hackers
Andres Freund <[email protected]> writes:
> There've not been any new reports in the last 18 hours, but that's probably
> just due to lower commit activity triggering fewer runs.
myna just showed the same symptom:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=myna&dt=2022-02-18%2003%3A00%3A17
# Failed test 'have walsender pid 13681
# 13670'
# at t/019_replslot_limit.pl line 335.
# '13681
# 13670'
# doesn't match '(?^:^[0-9]+$)'
# Tests were run but no plan was declared and done_testing() was not seen.
# Looks like your test exited with 29 just after 18.
[03:13:09] t/019_replslot_limit.pl ..............
I think that's the first one on something outside your menagerie.
regards, tom lane
^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH v14 5/7] rename archive-modules.sgml to archive-and-restore-modules.sgml
@ 2023-02-15 22:45 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 45+ messages in thread
From: Nathan Bossart @ 2023-02-15 22:45 UTC (permalink / raw)
---
.../{archive-modules.sgml => archive-and-restore-modules.sgml} | 0
doc/src/sgml/filelist.sgml | 2 +-
doc/src/sgml/postgres.sgml | 2 +-
3 files changed, 2 insertions(+), 2 deletions(-)
rename doc/src/sgml/{archive-modules.sgml => archive-and-restore-modules.sgml} (100%)
diff --git a/doc/src/sgml/archive-modules.sgml b/doc/src/sgml/archive-and-restore-modules.sgml
similarity index 100%
rename from doc/src/sgml/archive-modules.sgml
rename to doc/src/sgml/archive-and-restore-modules.sgml
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index 0d6be9a2fa..66a4de4762 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -100,7 +100,7 @@
<!ENTITY custom-scan SYSTEM "custom-scan.sgml">
<!ENTITY logicaldecoding SYSTEM "logicaldecoding.sgml">
<!ENTITY replication-origins SYSTEM "replication-origins.sgml">
-<!ENTITY archive-modules SYSTEM "archive-modules.sgml">
+<!ENTITY archive-and-restore-modules SYSTEM "archive-and-restore-modules.sgml">
<!ENTITY protocol SYSTEM "protocol.sgml">
<!ENTITY sources SYSTEM "sources.sgml">
<!ENTITY storage SYSTEM "storage.sgml">
diff --git a/doc/src/sgml/postgres.sgml b/doc/src/sgml/postgres.sgml
index 2e271862fc..f1ff6bb451 100644
--- a/doc/src/sgml/postgres.sgml
+++ b/doc/src/sgml/postgres.sgml
@@ -233,7 +233,7 @@ break is not needed in a wider output rendering.
&bgworker;
&logicaldecoding;
&replication-origins;
- &archive-modules;
+ &archive-and-restore-modules;
</part>
--
2.25.1
--IS0zKkzwUGydFO0o
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v14-0006-restructure-archive-modules-docs-in-preparation-.patch"
^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH v15 5/7] rename archive-modules.sgml to archive-and-restore-modules.sgml
@ 2023-02-15 22:45 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 45+ messages in thread
From: Nathan Bossart @ 2023-02-15 22:45 UTC (permalink / raw)
---
.../{archive-modules.sgml => archive-and-restore-modules.sgml} | 0
doc/src/sgml/filelist.sgml | 2 +-
doc/src/sgml/postgres.sgml | 2 +-
3 files changed, 2 insertions(+), 2 deletions(-)
rename doc/src/sgml/{archive-modules.sgml => archive-and-restore-modules.sgml} (100%)
diff --git a/doc/src/sgml/archive-modules.sgml b/doc/src/sgml/archive-and-restore-modules.sgml
similarity index 100%
rename from doc/src/sgml/archive-modules.sgml
rename to doc/src/sgml/archive-and-restore-modules.sgml
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index 0d6be9a2fa..66a4de4762 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -100,7 +100,7 @@
<!ENTITY custom-scan SYSTEM "custom-scan.sgml">
<!ENTITY logicaldecoding SYSTEM "logicaldecoding.sgml">
<!ENTITY replication-origins SYSTEM "replication-origins.sgml">
-<!ENTITY archive-modules SYSTEM "archive-modules.sgml">
+<!ENTITY archive-and-restore-modules SYSTEM "archive-and-restore-modules.sgml">
<!ENTITY protocol SYSTEM "protocol.sgml">
<!ENTITY sources SYSTEM "sources.sgml">
<!ENTITY storage SYSTEM "storage.sgml">
diff --git a/doc/src/sgml/postgres.sgml b/doc/src/sgml/postgres.sgml
index 2e271862fc..f1ff6bb451 100644
--- a/doc/src/sgml/postgres.sgml
+++ b/doc/src/sgml/postgres.sgml
@@ -233,7 +233,7 @@ break is not needed in a wider output rendering.
&bgworker;
&logicaldecoding;
&replication-origins;
- &archive-modules;
+ &archive-and-restore-modules;
</part>
--
2.25.1
--k+w/mQv8wyuph6w0
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v15-0006-restructure-archive-modules-docs-in-preparation-.patch"
^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH v12 4/6] rename archive-modules.sgml to archive-and-restore-modules.sgml
@ 2023-02-15 22:45 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 45+ messages in thread
From: Nathan Bossart @ 2023-02-15 22:45 UTC (permalink / raw)
---
.../{archive-modules.sgml => archive-and-restore-modules.sgml} | 0
doc/src/sgml/filelist.sgml | 2 +-
doc/src/sgml/postgres.sgml | 2 +-
3 files changed, 2 insertions(+), 2 deletions(-)
rename doc/src/sgml/{archive-modules.sgml => archive-and-restore-modules.sgml} (100%)
diff --git a/doc/src/sgml/archive-modules.sgml b/doc/src/sgml/archive-and-restore-modules.sgml
similarity index 100%
rename from doc/src/sgml/archive-modules.sgml
rename to doc/src/sgml/archive-and-restore-modules.sgml
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index 0d6be9a2fa..66a4de4762 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -100,7 +100,7 @@
<!ENTITY custom-scan SYSTEM "custom-scan.sgml">
<!ENTITY logicaldecoding SYSTEM "logicaldecoding.sgml">
<!ENTITY replication-origins SYSTEM "replication-origins.sgml">
-<!ENTITY archive-modules SYSTEM "archive-modules.sgml">
+<!ENTITY archive-and-restore-modules SYSTEM "archive-and-restore-modules.sgml">
<!ENTITY protocol SYSTEM "protocol.sgml">
<!ENTITY sources SYSTEM "sources.sgml">
<!ENTITY storage SYSTEM "storage.sgml">
diff --git a/doc/src/sgml/postgres.sgml b/doc/src/sgml/postgres.sgml
index 2e271862fc..f1ff6bb451 100644
--- a/doc/src/sgml/postgres.sgml
+++ b/doc/src/sgml/postgres.sgml
@@ -233,7 +233,7 @@ break is not needed in a wider output rendering.
&bgworker;
&logicaldecoding;
&replication-origins;
- &archive-modules;
+ &archive-and-restore-modules;
</part>
--
2.25.1
--YiEDa0DAkWCtVeE4
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v12-0005-restructure-archive-modules-docs-in-preparation-.patch"
^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH v13 5/7] rename archive-modules.sgml to archive-and-restore-modules.sgml
@ 2023-02-15 22:45 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 45+ messages in thread
From: Nathan Bossart @ 2023-02-15 22:45 UTC (permalink / raw)
---
.../{archive-modules.sgml => archive-and-restore-modules.sgml} | 0
doc/src/sgml/filelist.sgml | 2 +-
doc/src/sgml/postgres.sgml | 2 +-
3 files changed, 2 insertions(+), 2 deletions(-)
rename doc/src/sgml/{archive-modules.sgml => archive-and-restore-modules.sgml} (100%)
diff --git a/doc/src/sgml/archive-modules.sgml b/doc/src/sgml/archive-and-restore-modules.sgml
similarity index 100%
rename from doc/src/sgml/archive-modules.sgml
rename to doc/src/sgml/archive-and-restore-modules.sgml
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index 0d6be9a2fa..66a4de4762 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -100,7 +100,7 @@
<!ENTITY custom-scan SYSTEM "custom-scan.sgml">
<!ENTITY logicaldecoding SYSTEM "logicaldecoding.sgml">
<!ENTITY replication-origins SYSTEM "replication-origins.sgml">
-<!ENTITY archive-modules SYSTEM "archive-modules.sgml">
+<!ENTITY archive-and-restore-modules SYSTEM "archive-and-restore-modules.sgml">
<!ENTITY protocol SYSTEM "protocol.sgml">
<!ENTITY sources SYSTEM "sources.sgml">
<!ENTITY storage SYSTEM "storage.sgml">
diff --git a/doc/src/sgml/postgres.sgml b/doc/src/sgml/postgres.sgml
index 2e271862fc..f1ff6bb451 100644
--- a/doc/src/sgml/postgres.sgml
+++ b/doc/src/sgml/postgres.sgml
@@ -233,7 +233,7 @@ break is not needed in a wider output rendering.
&bgworker;
&logicaldecoding;
&replication-origins;
- &archive-modules;
+ &archive-and-restore-modules;
</part>
--
2.25.1
--5vNYLRcllDrimb99
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v13-0006-restructure-archive-modules-docs-in-preparation-.patch"
^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH v18 3/5] rename archive-modules.sgml to archive-and-restore-modules.sgml
@ 2023-02-15 22:45 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 45+ messages in thread
From: Nathan Bossart @ 2023-02-15 22:45 UTC (permalink / raw)
---
.../{archive-modules.sgml => archive-and-restore-modules.sgml} | 0
doc/src/sgml/filelist.sgml | 2 +-
doc/src/sgml/postgres.sgml | 2 +-
3 files changed, 2 insertions(+), 2 deletions(-)
rename doc/src/sgml/{archive-modules.sgml => archive-and-restore-modules.sgml} (100%)
diff --git a/doc/src/sgml/archive-modules.sgml b/doc/src/sgml/archive-and-restore-modules.sgml
similarity index 100%
rename from doc/src/sgml/archive-modules.sgml
rename to doc/src/sgml/archive-and-restore-modules.sgml
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index e0dca81cb2..31a57943f7 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -101,7 +101,7 @@
<!ENTITY custom-scan SYSTEM "custom-scan.sgml">
<!ENTITY logicaldecoding SYSTEM "logicaldecoding.sgml">
<!ENTITY replication-origins SYSTEM "replication-origins.sgml">
-<!ENTITY archive-modules SYSTEM "archive-modules.sgml">
+<!ENTITY archive-and-restore-modules SYSTEM "archive-and-restore-modules.sgml">
<!ENTITY protocol SYSTEM "protocol.sgml">
<!ENTITY sources SYSTEM "sources.sgml">
<!ENTITY storage SYSTEM "storage.sgml">
diff --git a/doc/src/sgml/postgres.sgml b/doc/src/sgml/postgres.sgml
index 2c107199d3..23522ccaf3 100644
--- a/doc/src/sgml/postgres.sgml
+++ b/doc/src/sgml/postgres.sgml
@@ -228,7 +228,7 @@ break is not needed in a wider output rendering.
&bgworker;
&logicaldecoding;
&replication-origins;
- &archive-modules;
+ &archive-and-restore-modules;
</part>
--
2.25.1
--IJpNTDwzlM2Ie8A6
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v18-0004-restructure-archive-modules-docs-in-preparation-.patch"
^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH v18 3/5] rename archive-modules.sgml to archive-and-restore-modules.sgml
@ 2023-02-15 22:45 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 45+ messages in thread
From: Nathan Bossart @ 2023-02-15 22:45 UTC (permalink / raw)
---
.../{archive-modules.sgml => archive-and-restore-modules.sgml} | 0
doc/src/sgml/filelist.sgml | 2 +-
doc/src/sgml/postgres.sgml | 2 +-
3 files changed, 2 insertions(+), 2 deletions(-)
rename doc/src/sgml/{archive-modules.sgml => archive-and-restore-modules.sgml} (100%)
diff --git a/doc/src/sgml/archive-modules.sgml b/doc/src/sgml/archive-and-restore-modules.sgml
similarity index 100%
rename from doc/src/sgml/archive-modules.sgml
rename to doc/src/sgml/archive-and-restore-modules.sgml
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index e0dca81cb2..31a57943f7 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -101,7 +101,7 @@
<!ENTITY custom-scan SYSTEM "custom-scan.sgml">
<!ENTITY logicaldecoding SYSTEM "logicaldecoding.sgml">
<!ENTITY replication-origins SYSTEM "replication-origins.sgml">
-<!ENTITY archive-modules SYSTEM "archive-modules.sgml">
+<!ENTITY archive-and-restore-modules SYSTEM "archive-and-restore-modules.sgml">
<!ENTITY protocol SYSTEM "protocol.sgml">
<!ENTITY sources SYSTEM "sources.sgml">
<!ENTITY storage SYSTEM "storage.sgml">
diff --git a/doc/src/sgml/postgres.sgml b/doc/src/sgml/postgres.sgml
index 2c107199d3..23522ccaf3 100644
--- a/doc/src/sgml/postgres.sgml
+++ b/doc/src/sgml/postgres.sgml
@@ -228,7 +228,7 @@ break is not needed in a wider output rendering.
&bgworker;
&logicaldecoding;
&replication-origins;
- &archive-modules;
+ &archive-and-restore-modules;
</part>
--
2.25.1
--IJpNTDwzlM2Ie8A6
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v18-0004-restructure-archive-modules-docs-in-preparation-.patch"
^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH v19 3/5] rename archive-modules.sgml to archive-and-restore-modules.sgml
@ 2023-02-15 22:45 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 45+ messages in thread
From: Nathan Bossart @ 2023-02-15 22:45 UTC (permalink / raw)
---
.../{archive-modules.sgml => archive-and-restore-modules.sgml} | 0
doc/src/sgml/filelist.sgml | 2 +-
doc/src/sgml/postgres.sgml | 2 +-
3 files changed, 2 insertions(+), 2 deletions(-)
rename doc/src/sgml/{archive-modules.sgml => archive-and-restore-modules.sgml} (100%)
diff --git a/doc/src/sgml/archive-modules.sgml b/doc/src/sgml/archive-and-restore-modules.sgml
similarity index 100%
rename from doc/src/sgml/archive-modules.sgml
rename to doc/src/sgml/archive-and-restore-modules.sgml
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index e0dca81cb2..31a57943f7 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -101,7 +101,7 @@
<!ENTITY custom-scan SYSTEM "custom-scan.sgml">
<!ENTITY logicaldecoding SYSTEM "logicaldecoding.sgml">
<!ENTITY replication-origins SYSTEM "replication-origins.sgml">
-<!ENTITY archive-modules SYSTEM "archive-modules.sgml">
+<!ENTITY archive-and-restore-modules SYSTEM "archive-and-restore-modules.sgml">
<!ENTITY protocol SYSTEM "protocol.sgml">
<!ENTITY sources SYSTEM "sources.sgml">
<!ENTITY storage SYSTEM "storage.sgml">
diff --git a/doc/src/sgml/postgres.sgml b/doc/src/sgml/postgres.sgml
index 2c107199d3..23522ccaf3 100644
--- a/doc/src/sgml/postgres.sgml
+++ b/doc/src/sgml/postgres.sgml
@@ -228,7 +228,7 @@ break is not needed in a wider output rendering.
&bgworker;
&logicaldecoding;
&replication-origins;
- &archive-modules;
+ &archive-and-restore-modules;
</part>
--
2.25.1
--oyUTqETQ0mS9luUI
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v19-0004-restructure-archive-modules-docs-in-preparation-.patch"
^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH v20 3/5] rename archive-modules.sgml to archive-and-restore-modules.sgml
@ 2023-02-15 22:45 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 45+ messages in thread
From: Nathan Bossart @ 2023-02-15 22:45 UTC (permalink / raw)
---
.../{archive-modules.sgml => archive-and-restore-modules.sgml} | 0
doc/src/sgml/filelist.sgml | 2 +-
doc/src/sgml/postgres.sgml | 2 +-
3 files changed, 2 insertions(+), 2 deletions(-)
rename doc/src/sgml/{archive-modules.sgml => archive-and-restore-modules.sgml} (100%)
diff --git a/doc/src/sgml/archive-modules.sgml b/doc/src/sgml/archive-and-restore-modules.sgml
similarity index 100%
rename from doc/src/sgml/archive-modules.sgml
rename to doc/src/sgml/archive-and-restore-modules.sgml
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index e0dca81cb2..31a57943f7 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -101,7 +101,7 @@
<!ENTITY custom-scan SYSTEM "custom-scan.sgml">
<!ENTITY logicaldecoding SYSTEM "logicaldecoding.sgml">
<!ENTITY replication-origins SYSTEM "replication-origins.sgml">
-<!ENTITY archive-modules SYSTEM "archive-modules.sgml">
+<!ENTITY archive-and-restore-modules SYSTEM "archive-and-restore-modules.sgml">
<!ENTITY protocol SYSTEM "protocol.sgml">
<!ENTITY sources SYSTEM "sources.sgml">
<!ENTITY storage SYSTEM "storage.sgml">
diff --git a/doc/src/sgml/postgres.sgml b/doc/src/sgml/postgres.sgml
index 2c107199d3..23522ccaf3 100644
--- a/doc/src/sgml/postgres.sgml
+++ b/doc/src/sgml/postgres.sgml
@@ -228,7 +228,7 @@ break is not needed in a wider output rendering.
&bgworker;
&logicaldecoding;
&replication-origins;
- &archive-modules;
+ &archive-and-restore-modules;
</part>
--
2.25.1
--3V7upXqbjpZ4EhLz
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v20-0004-restructure-archive-modules-docs-in-preparation-.patch"
^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH v19 3/5] rename archive-modules.sgml to archive-and-restore-modules.sgml
@ 2023-02-15 22:45 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 45+ messages in thread
From: Nathan Bossart @ 2023-02-15 22:45 UTC (permalink / raw)
---
.../{archive-modules.sgml => archive-and-restore-modules.sgml} | 0
doc/src/sgml/filelist.sgml | 2 +-
doc/src/sgml/postgres.sgml | 2 +-
3 files changed, 2 insertions(+), 2 deletions(-)
rename doc/src/sgml/{archive-modules.sgml => archive-and-restore-modules.sgml} (100%)
diff --git a/doc/src/sgml/archive-modules.sgml b/doc/src/sgml/archive-and-restore-modules.sgml
similarity index 100%
rename from doc/src/sgml/archive-modules.sgml
rename to doc/src/sgml/archive-and-restore-modules.sgml
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index e0dca81cb2..31a57943f7 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -101,7 +101,7 @@
<!ENTITY custom-scan SYSTEM "custom-scan.sgml">
<!ENTITY logicaldecoding SYSTEM "logicaldecoding.sgml">
<!ENTITY replication-origins SYSTEM "replication-origins.sgml">
-<!ENTITY archive-modules SYSTEM "archive-modules.sgml">
+<!ENTITY archive-and-restore-modules SYSTEM "archive-and-restore-modules.sgml">
<!ENTITY protocol SYSTEM "protocol.sgml">
<!ENTITY sources SYSTEM "sources.sgml">
<!ENTITY storage SYSTEM "storage.sgml">
diff --git a/doc/src/sgml/postgres.sgml b/doc/src/sgml/postgres.sgml
index 2c107199d3..23522ccaf3 100644
--- a/doc/src/sgml/postgres.sgml
+++ b/doc/src/sgml/postgres.sgml
@@ -228,7 +228,7 @@ break is not needed in a wider output rendering.
&bgworker;
&logicaldecoding;
&replication-origins;
- &archive-modules;
+ &archive-and-restore-modules;
</part>
--
2.25.1
--oyUTqETQ0mS9luUI
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v19-0004-restructure-archive-modules-docs-in-preparation-.patch"
^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH v19 3/5] rename archive-modules.sgml to archive-and-restore-modules.sgml
@ 2023-02-15 22:45 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 45+ messages in thread
From: Nathan Bossart @ 2023-02-15 22:45 UTC (permalink / raw)
---
.../{archive-modules.sgml => archive-and-restore-modules.sgml} | 0
doc/src/sgml/filelist.sgml | 2 +-
doc/src/sgml/postgres.sgml | 2 +-
3 files changed, 2 insertions(+), 2 deletions(-)
rename doc/src/sgml/{archive-modules.sgml => archive-and-restore-modules.sgml} (100%)
diff --git a/doc/src/sgml/archive-modules.sgml b/doc/src/sgml/archive-and-restore-modules.sgml
similarity index 100%
rename from doc/src/sgml/archive-modules.sgml
rename to doc/src/sgml/archive-and-restore-modules.sgml
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index e0dca81cb2..31a57943f7 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -101,7 +101,7 @@
<!ENTITY custom-scan SYSTEM "custom-scan.sgml">
<!ENTITY logicaldecoding SYSTEM "logicaldecoding.sgml">
<!ENTITY replication-origins SYSTEM "replication-origins.sgml">
-<!ENTITY archive-modules SYSTEM "archive-modules.sgml">
+<!ENTITY archive-and-restore-modules SYSTEM "archive-and-restore-modules.sgml">
<!ENTITY protocol SYSTEM "protocol.sgml">
<!ENTITY sources SYSTEM "sources.sgml">
<!ENTITY storage SYSTEM "storage.sgml">
diff --git a/doc/src/sgml/postgres.sgml b/doc/src/sgml/postgres.sgml
index 2c107199d3..23522ccaf3 100644
--- a/doc/src/sgml/postgres.sgml
+++ b/doc/src/sgml/postgres.sgml
@@ -228,7 +228,7 @@ break is not needed in a wider output rendering.
&bgworker;
&logicaldecoding;
&replication-origins;
- &archive-modules;
+ &archive-and-restore-modules;
</part>
--
2.25.1
--oyUTqETQ0mS9luUI
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v19-0004-restructure-archive-modules-docs-in-preparation-.patch"
^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH v18 3/5] rename archive-modules.sgml to archive-and-restore-modules.sgml
@ 2023-02-15 22:45 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 45+ messages in thread
From: Nathan Bossart @ 2023-02-15 22:45 UTC (permalink / raw)
---
.../{archive-modules.sgml => archive-and-restore-modules.sgml} | 0
doc/src/sgml/filelist.sgml | 2 +-
doc/src/sgml/postgres.sgml | 2 +-
3 files changed, 2 insertions(+), 2 deletions(-)
rename doc/src/sgml/{archive-modules.sgml => archive-and-restore-modules.sgml} (100%)
diff --git a/doc/src/sgml/archive-modules.sgml b/doc/src/sgml/archive-and-restore-modules.sgml
similarity index 100%
rename from doc/src/sgml/archive-modules.sgml
rename to doc/src/sgml/archive-and-restore-modules.sgml
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index e0dca81cb2..31a57943f7 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -101,7 +101,7 @@
<!ENTITY custom-scan SYSTEM "custom-scan.sgml">
<!ENTITY logicaldecoding SYSTEM "logicaldecoding.sgml">
<!ENTITY replication-origins SYSTEM "replication-origins.sgml">
-<!ENTITY archive-modules SYSTEM "archive-modules.sgml">
+<!ENTITY archive-and-restore-modules SYSTEM "archive-and-restore-modules.sgml">
<!ENTITY protocol SYSTEM "protocol.sgml">
<!ENTITY sources SYSTEM "sources.sgml">
<!ENTITY storage SYSTEM "storage.sgml">
diff --git a/doc/src/sgml/postgres.sgml b/doc/src/sgml/postgres.sgml
index 2c107199d3..23522ccaf3 100644
--- a/doc/src/sgml/postgres.sgml
+++ b/doc/src/sgml/postgres.sgml
@@ -228,7 +228,7 @@ break is not needed in a wider output rendering.
&bgworker;
&logicaldecoding;
&replication-origins;
- &archive-modules;
+ &archive-and-restore-modules;
</part>
--
2.25.1
--IJpNTDwzlM2Ie8A6
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v18-0004-restructure-archive-modules-docs-in-preparation-.patch"
^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH v20 3/5] rename archive-modules.sgml to archive-and-restore-modules.sgml
@ 2023-02-15 22:45 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 45+ messages in thread
From: Nathan Bossart @ 2023-02-15 22:45 UTC (permalink / raw)
---
.../{archive-modules.sgml => archive-and-restore-modules.sgml} | 0
doc/src/sgml/filelist.sgml | 2 +-
doc/src/sgml/postgres.sgml | 2 +-
3 files changed, 2 insertions(+), 2 deletions(-)
rename doc/src/sgml/{archive-modules.sgml => archive-and-restore-modules.sgml} (100%)
diff --git a/doc/src/sgml/archive-modules.sgml b/doc/src/sgml/archive-and-restore-modules.sgml
similarity index 100%
rename from doc/src/sgml/archive-modules.sgml
rename to doc/src/sgml/archive-and-restore-modules.sgml
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index e0dca81cb2..31a57943f7 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -101,7 +101,7 @@
<!ENTITY custom-scan SYSTEM "custom-scan.sgml">
<!ENTITY logicaldecoding SYSTEM "logicaldecoding.sgml">
<!ENTITY replication-origins SYSTEM "replication-origins.sgml">
-<!ENTITY archive-modules SYSTEM "archive-modules.sgml">
+<!ENTITY archive-and-restore-modules SYSTEM "archive-and-restore-modules.sgml">
<!ENTITY protocol SYSTEM "protocol.sgml">
<!ENTITY sources SYSTEM "sources.sgml">
<!ENTITY storage SYSTEM "storage.sgml">
diff --git a/doc/src/sgml/postgres.sgml b/doc/src/sgml/postgres.sgml
index 2c107199d3..23522ccaf3 100644
--- a/doc/src/sgml/postgres.sgml
+++ b/doc/src/sgml/postgres.sgml
@@ -228,7 +228,7 @@ break is not needed in a wider output rendering.
&bgworker;
&logicaldecoding;
&replication-origins;
- &archive-modules;
+ &archive-and-restore-modules;
</part>
--
2.25.1
--3V7upXqbjpZ4EhLz
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v20-0004-restructure-archive-modules-docs-in-preparation-.patch"
^ permalink raw reply [nested|flat] 45+ messages in thread
* [PATCH v16 3/5] rename archive-modules.sgml to archive-and-restore-modules.sgml
@ 2023-02-15 22:45 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 45+ messages in thread
From: Nathan Bossart @ 2023-02-15 22:45 UTC (permalink / raw)
---
.../{archive-modules.sgml => archive-and-restore-modules.sgml} | 0
doc/src/sgml/filelist.sgml | 2 +-
doc/src/sgml/postgres.sgml | 2 +-
3 files changed, 2 insertions(+), 2 deletions(-)
rename doc/src/sgml/{archive-modules.sgml => archive-and-restore-modules.sgml} (100%)
diff --git a/doc/src/sgml/archive-modules.sgml b/doc/src/sgml/archive-and-restore-modules.sgml
similarity index 100%
rename from doc/src/sgml/archive-modules.sgml
rename to doc/src/sgml/archive-and-restore-modules.sgml
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index 4c63a7e768..b175152e6b 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -101,7 +101,7 @@
<!ENTITY custom-scan SYSTEM "custom-scan.sgml">
<!ENTITY logicaldecoding SYSTEM "logicaldecoding.sgml">
<!ENTITY replication-origins SYSTEM "replication-origins.sgml">
-<!ENTITY archive-modules SYSTEM "archive-modules.sgml">
+<!ENTITY archive-and-restore-modules SYSTEM "archive-and-restore-modules.sgml">
<!ENTITY protocol SYSTEM "protocol.sgml">
<!ENTITY sources SYSTEM "sources.sgml">
<!ENTITY storage SYSTEM "storage.sgml">
diff --git a/doc/src/sgml/postgres.sgml b/doc/src/sgml/postgres.sgml
index f629524be0..b12a9f8dd5 100644
--- a/doc/src/sgml/postgres.sgml
+++ b/doc/src/sgml/postgres.sgml
@@ -233,7 +233,7 @@ break is not needed in a wider output rendering.
&bgworker;
&logicaldecoding;
&replication-origins;
- &archive-modules;
+ &archive-and-restore-modules;
</part>
--
2.25.1
--SLDf9lqlvOQaIe6s
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v16-0004-restructure-archive-modules-docs-in-preparation-.patch"
^ permalink raw reply [nested|flat] 45+ messages in thread
end of thread, other threads:[~2023-02-15 22:45 UTC | newest]
Thread overview: 45+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2019-06-28 23:46 Re: [HACKERS] WAL logging problem in 9.4.3? Amit Kapila <[email protected]>
2022-02-17 02:14 Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 03:11 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-17 03:15 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-17 03:46 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 03:58 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-17 04:32 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-17 04:03 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 00:34 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 03:41 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 05:55 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 22:42 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 23:14 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-23 02:06 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-25 19:29 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-25 20:07 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-25 20:15 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-26 07:23 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-03-24 19:05 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-03-25 00:02 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-03-25 00:08 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-03-27 19:28 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-03-27 21:32 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-03-27 21:36 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-03-28 05:37 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-05-30 19:01 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-05-31 01:31 ` Re: Race conditions in 019_replslot_limit.pl Kyotaro Horiguchi <[email protected]>
2022-02-18 23:15 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-18 23:40 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 23:49 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2022-02-19 00:08 ` Re: Race conditions in 019_replslot_limit.pl Andres Freund <[email protected]>
2022-02-18 04:11 ` Re: Race conditions in 019_replslot_limit.pl Tom Lane <[email protected]>
2023-02-15 22:45 [PATCH v14 5/7] rename archive-modules.sgml to archive-and-restore-modules.sgml Nathan Bossart <[email protected]>
2023-02-15 22:45 [PATCH v15 5/7] rename archive-modules.sgml to archive-and-restore-modules.sgml Nathan Bossart <[email protected]>
2023-02-15 22:45 [PATCH v12 4/6] rename archive-modules.sgml to archive-and-restore-modules.sgml Nathan Bossart <[email protected]>
2023-02-15 22:45 [PATCH v13 5/7] rename archive-modules.sgml to archive-and-restore-modules.sgml Nathan Bossart <[email protected]>
2023-02-15 22:45 [PATCH v18 3/5] rename archive-modules.sgml to archive-and-restore-modules.sgml Nathan Bossart <[email protected]>
2023-02-15 22:45 [PATCH v18 3/5] rename archive-modules.sgml to archive-and-restore-modules.sgml Nathan Bossart <[email protected]>
2023-02-15 22:45 [PATCH v19 3/5] rename archive-modules.sgml to archive-and-restore-modules.sgml Nathan Bossart <[email protected]>
2023-02-15 22:45 [PATCH v20 3/5] rename archive-modules.sgml to archive-and-restore-modules.sgml Nathan Bossart <[email protected]>
2023-02-15 22:45 [PATCH v19 3/5] rename archive-modules.sgml to archive-and-restore-modules.sgml Nathan Bossart <[email protected]>
2023-02-15 22:45 [PATCH v19 3/5] rename archive-modules.sgml to archive-and-restore-modules.sgml Nathan Bossart <[email protected]>
2023-02-15 22:45 [PATCH v18 3/5] rename archive-modules.sgml to archive-and-restore-modules.sgml Nathan Bossart <[email protected]>
2023-02-15 22:45 [PATCH v20 3/5] rename archive-modules.sgml to archive-and-restore-modules.sgml Nathan Bossart <[email protected]>
2023-02-15 22:45 [PATCH v16 3/5] rename archive-modules.sgml to archive-and-restore-modules.sgml Nathan Bossart <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox