pg.ddx.io  pgsql-hackers@postgresql.org mailing list archive  
help / color / mirror / Atom feed
crash with synchronized_standby_slots
5+ messages / 3 participants
[nested] [flat]

Commits from this thread:
  2024-12-03 3c5f9f12c8 9abdc1841e  Fix synchronized_standby_slots GUC check hook
* crash with synchronized_standby_slots
@ 2024-11-28 12:16  Alvaro Herrera <alvherre@alvh.no-ip.org>
  0 siblings, 2 replies; 5+ messages in thread

From: Alvaro Herrera @ 2024-11-28 12:16 UTC (permalink / raw)
  To: Pg Hackers <pgsql-hackers@lists.postgresql.org>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Zhijie Hou (Fujitsu) <houzj.fnst@fujitsu.com>; Gabriele Bartolini <gabriele.bartolini@enterprisedb.com>

Hello,

Gabriele just reported a crash when changing synchronized_standby_slots
under SIGHUP and logging collector working.  The problem apparently is
that validate_sync_standby_slots is run for the GUC check routine, and
it requires acquiring an LWLock; but syslogger doesn't have a PGPROC so
that doesn't work.

Apparently we already have a hack in place against the same problem in
postmaster (and elsewhere?), by testing that ReplicationSlotCtl is not
null.  But that hack seems to be incomplete, as the crash here attests.

To reproduce, simply start with no synchronized_standby_slots setting
and logging_collector=on, then change the value in postgresql.conf and
reload.

One option to fix this might be as attached.  The point here is that
processes without a PGPROC don't need or care to have a correct setting,
so we just skip verifying it on those.  AFAICS this is more general than
the test for ReplicationSlotCtl, so I just removed that one.

Here's the backtrace Gabriele showed me.

[New LWP 29]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `postgres: freddie: logger                  '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  pg_atomic_read_u32_impl (ptr=0x7f0cf21a1f84) at ./build/../src/include/port/atomics/generic.h:48
Download failed: Invalid argument.  Continuing without source file ./build/src/backend/storage/lmgr/./build/../src/include/port/atomics/generic.h.
48	./build/../src/include/port/atomics/generic.h: No such file or directory.
#0  pg_atomic_read_u32_impl (ptr=0x7f0cf21a1f84) at ./build/../src/include/port/atomics/generic.h:48
No locals.
#1  pg_atomic_read_u32 (ptr=0x7f0cf21a1f84) at ./build/../src/include/port/atomics.h:237
No locals.
#2  LWLockAttemptLock (lock=lock@entry=0x7f0cf21a1f80, mode=mode@entry=LW_SHARED) at ./build/../src/backend/storage/lmgr/lwlock.c:796
        old_state = <optimized out>
#3  0x0000557cbe97ba60 in LWLockAcquire (lock=0x7f0cf21a1f80, mode=mode@entry=LW_SHARED) at ./build/../src/backend/storage/lmgr/lwlock.c:1235
        mustwait = <optimized out>
        proc = 0x0
        result = true
        extraWaits = 0
        __func__ = "LWLockAcquire"
#4  0x0000557cbe935919 in validate_sync_standby_slots (elemlist=0x7ffda5aa60d0, rawname=0x557cfd704c98 "_cnpg_freddie_2") at ./build/../src/backend/replication/slot.c:2443
        ok = true
        ok = <optimized out>
        name = <optimized out>
        name__outerloop = <optimized out>
        name__state = {l = <optimized out>, i = <optimized out>}
        slot = <optimized out>
#5  check_synchronized_standby_slots (newval=<optimized out>, extra=0x7ffda5aa6300, source=<optimized out>) at ./build/../src/backend/replication/slot.c:2494
        rawname = 0x557cfd704c98 "_cnpg_freddie_2"
        ptr = <optimized out>
        elemlist = 0x557cfd7041c8
        size = <optimized out>
        ok = <optimized out>
        config = <optimized out>
#6  0x0000557cbeae95de in call_string_check_hook (conf=conf@entry=0x557cbee2f8b0 <ConfigureNamesString+14000>, newval=newval@entry=0x7ffda5aa62f8, extra=extra@entry=0x7ffda5aa6300, source=source@entry=PGC_S_FILE, elevel=elevel@entry=12) at ./build/../src/backend/utils/misc/guc.c:6925
        _save_exception_stack = 0x0
        _save_context_stack = 0x0
        _local_sigjmp_buf = {{__jmpbuf = {93994266851504, -6990900416883569069, 140727382860536, 12, 93994266851504, 93995316414744, -6990900416982135213, -3748856269321122221}, __mask_was_saved = 0, __saved_mask = {__val = {140727382860536, 140727382860272, 93995316305072, 139693771467072, 93994263427300, 4662694561863172096, 93994263387537, 140727382860536, 93994263505961, 93994266851504, 140727382860536, 140727382860336, 93994263421816, 140727382860536, 12, 93994266851504}}}}
        _do_rethrow = false
        result = true
        __func__ = "call_string_check_hook"
#7  0x0000557cbeaec1ba in parse_and_validate_value (record=0x557cbee2f8b0 <ConfigureNamesString+14000>, name=0x557cfd765840 "synchronized_standby_slots", value=<optimized out>, source=PGC_S_FILE, elevel=12, newval=0x7ffda5aa62f8, newextra=0x7ffda5aa6300) at ./build/../src/backend/utils/misc/guc.c:3260
        conf = 0x557cbee2f8b0 <ConfigureNamesString+14000>
        __func__ = "parse_and_validate_value"
#8  0x0000557cbeaecd6f in set_config_with_handle (name=0x557cfd765840 "synchronized_standby_slots", handle=<optimized out>, value=0x557cfd765868 "_cnpg_freddie_2", context=PGC_SIGHUP, source=PGC_S_FILE, srole=10, action=GUC_ACTION_SET, changeVal=<optimized out>, elevel=12, is_reload=false) at ./build/../src/backend/utils/misc/guc.c:4013
        conf = 0x557cbee2f8b0 <ConfigureNamesString+14000>
        orig_context = PGC_SIGHUP
        orig_source = PGC_S_FILE
        orig_srole = 10
        record = 0x557cbee2f8b0 <ConfigureNamesString+14000>
        newval_union = {boolval = 24, intval = -42858216, realval = 4.6439856710502817e-310, stringval = 0x557cfd720918 "_cnpg_freddie_2", enumval = -42858216}
        newextra = 0x0
        prohibitValueChange = false
        makeDefault = <optimized out>
        __func__ = "set_config_with_handle"
#9  0x0000557cbeaef400 in set_config_option (is_reload=false, elevel=0, changeVal=true, action=GUC_ACTION_SET, source=PGC_S_FILE, context=PGC_SIGHUP, value=<optimized out>, name=<optimized out>) at ./build/../src/backend/utils/misc/guc.c:3363
        srole = 10
        srole = <optimized out>
#10 ProcessConfigFileInternal (context=context@entry=PGC_SIGHUP, applySettings=applySettings@entry=true, elevel=elevel@entry=13) at ./build/../src/backend/utils/misc/guc.c:557
        pre_value = 0x0
        scres = <optimized out>
        error = false
        applying = true
        ConfFileWithError = 0x557cfd720100 "/var/lib/postgresql/data/pgdata/postgresql.conf"
        item = 0x557cfd765880
        head = 0x557cfd704248
        tail = 0x557cfd7664c0
        status = {hashp = 0x557cfd7219c8, curBucket = 512, curEntry = 0x0}
        hentry = <optimized out>
        bail_out = <optimized out>
        __func__ = "ProcessConfigFileInternal"
#11 0x0000557cbeaf2e81 in ProcessConfigFile (context=context@entry=PGC_SIGHUP) at ./build/../src/backend/utils/misc/guc-file.l:152
        elevel = 13
        config_cxt = 0x557cfd7040d0
        caller_cxt = 0x557cfd6fd800
#12 0x0000557cbe9001cc in SysLoggerMain (startup_data=<optimized out>, startup_data_len=<optimized out>) at ./build/../src/backend/postmaster/syslogger.c:369
        time_based_rotation = false
        size_rotation_for = 0
        cur_timeout = <optimized out>
        event = {pos = 0, events = 1, fd = -1, user_data = 0x0}
        rc = <optimized out>
        logbuffer = "\000\000\300\000\034\000\000\000!2024-11-28 10:19:42.415 UTC,,,28,,6748439f.1c,8,,2024-11-28 10:19:11 UTC,,0,LOG,00000,\"parameter \"\"synchronous_standby_names\"\" changed to \"\"ANY 1 (\"\"freddie-2\"\")\"\"\",,,,,,,,,\"\",\"postmaster\",,0"...
        bytes_in_logbuffer = 0
        currentLogDir = 0x557cfd6fe6d8 "/controller/log"
        currentLogFilename = 0x557cfd6fe6f0 "postgres"
        currentLogRotationAge = 0
        now = 1732789151
        wes = 0x557cfd6fe708
        __func__ = "SysLoggerMain"
#13 0x0000557cbe8f908a in postmaster_child_launch (child_type=child_type@entry=B_LOGGER, startup_data=startup_data@entry=0x0, startup_data_len=startup_data_len@entry=0, client_sock=client_sock@entry=0x0) at ./build/../src/backend/postmaster/launch_backend.c:277
        pid = <optimized out>
#14 0x0000557cbe8ff9c2 in SysLogger_Start () at ./build/../src/backend/postmaster/syslogger.c:706
        sysloggerPid = <optimized out>
        filename = 0x557cfd75a160 "og output will go to log destination \"csvlog\".\n"
        __func__ = "SysLogger_Start"
#15 0x0000557cbe8fe42e in PostmasterMain (argc=argc@entry=3, argv=argv@entry=0x557cfd6fd7a0) at ./build/../src/backend/postmaster/postmaster.c:1054
        opt = <optimized out>
        status = <optimized out>
        userDoption = <optimized out>
        listen_addr_saved = false
        output_config_variable = 0x0
        __func__ = "PostmasterMain"
#16 0x0000557cbe6258ec in main (argc=3, argv=0x557cfd6fd7a0) at ./build/../src/backend/main/main.c:197
        do_check_root = <optimized out>

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/
"Once again, thank you and all of the developers for your hard work on
PostgreSQL.  This is by far the most pleasant management experience of
any database I've worked on."                             (Dan Harris)
http://archives.postgresql.org/pgsql-performance/2006-04/msg00247.php

Attachments:

0001-Fix-synchronized_standby_slots-GUC-check-hook.patch (text/x-diff, 1.6K) download From 2d38870be86200deefb9979b9dd0a164d5986499 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=81lvaro=20Herrera?= <alvherre@alvh.no-ip.org> Date: Thu, 28 Nov 2024 13:11:20 +0100 Subject: [PATCH] Fix synchronized_standby_slots GUC check hook It requires an LWLock, so it cannot run in processes without PGPROC. Skip it there; it makes no difference. Reported-by: Gabriele Bartolini <gabriele.bartolini@enterprisedb.com> --- src/backend/replication/slot.c | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c index 77bae536915..b6e43c1a89e 100644 --- a/src/backend/replication/slot.c +++ b/src/backend/replication/slot.c @@ -2428,18 +2428,16 @@ validate_sync_standby_slots(char *rawname, List **elemlist) { GUC_check_errdetail("List syntax is invalid."); } - else if (!ReplicationSlotCtl) + else if (MyProc) { /* - * We cannot validate the replication slot if the replication slots' - * data has not been initialized. This is ok as we will anyway - * validate the specified slot when waiting for them to catch up. See - * StandbySlotsHaveCaughtup() for details. + * Check that the specified slots exist and are logical slots. Note + * that because we need an LWLock, we cannot do this on processes + * without a PGPROC (hello syslogger!) so we skip it there, but it + * doesn't have an effect for those anyway. + * + * See StandbySlotsHaveCaughtup() for details. */ - } - else - { - /* Check that the specified slots exist and are logical slots */ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED); foreach_ptr(char, name, *elemlist) -- 2.39.5
^ permalink raw reply [nested|flat] 5+ messages in thread

* Re: crash with synchronized_standby_slots
@ 2024-11-29 06:01  Amit Kapila <amit.kapila16@gmail.com>
  parent: Alvaro Herrera <alvherre@alvh.no-ip.org>
  1 sibling, 1 reply; 5+ messages in thread

From: Amit Kapila @ 2024-11-29 06:01 UTC (permalink / raw)
  To: Alvaro Herrera <alvherre@alvh.no-ip.org>; +Cc: Pg Hackers <pgsql-hackers@lists.postgresql.org>; Zhijie Hou (Fujitsu) <houzj.fnst@fujitsu.com>; Gabriele Bartolini <gabriele.bartolini@enterprisedb.com>

On Thu, Nov 28, 2024 at 5:54 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
>
> Gabriele just reported a crash when changing synchronized_standby_slots
> under SIGHUP and logging collector working.  The problem apparently is
> that validate_sync_standby_slots is run for the GUC check routine, and
> it requires acquiring an LWLock; but syslogger doesn't have a PGPROC so
> that doesn't work.
>
> Apparently we already have a hack in place against the same problem in
> postmaster (and elsewhere?), by testing that ReplicationSlotCtl is not
> null.  But that hack seems to be incomplete, as the crash here attests.
>

I tried it on my Windows machine and noticed that ReplicationSlotCtl
is NULL for syslogger, so the problem doesn't occur. The reason is
that we don't attach to shared memory in syslogger, so ideally
ReplicationSlotCtl should be NULL. Because we inherit everything
through the fork for Linux systems and then later for processes that
don't want to attach to shared memory, we call PGSharedMemoryDetach()
from postmaster_child_launch(). The PGSharedMemoryDetach won't
reinitialize the memory pointed to by ReplicationSlotCtl, so, it would
be an invalid memory.

> To reproduce, simply start with no synchronized_standby_slots setting
> and logging_collector=on, then change the value in postgresql.conf and
> reload.
>
> One option to fix this might be as attached.  The point here is that
> processes without a PGPROC don't need or care to have a correct setting,
> so we just skip verifying it on those.  AFAICS this is more general than
> the test for ReplicationSlotCtl, so I just removed that one.
>

Yeah, I can't think of a better fix for this.

-- 
With Regards,
Amit Kapila.





^ permalink  raw  reply  [nested|flat] 5+ messages in thread

* RE: crash with synchronized_standby_slots
@ 2024-11-29 08:19  Zhijie Hou (Fujitsu) <houzj.fnst@fujitsu.com>
  parent: Alvaro Herrera <alvherre@alvh.no-ip.org>
  1 sibling, 0 replies; 5+ messages in thread

From: Zhijie Hou (Fujitsu) @ 2024-11-29 08:19 UTC (permalink / raw)
  To: Alvaro Herrera <alvherre@alvh.no-ip.org>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Gabriele Bartolini <gabriele.bartolini@enterprisedb.com>; Pg Hackers <pgsql-hackers@lists.postgresql.org>

On Thursday, November 28, 2024 8:16 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
> 
> Gabriele just reported a crash when changing synchronized_standby_slots
> under SIGHUP and logging collector working.  The problem apparently is
> that validate_sync_standby_slots is run for the GUC check routine, and
> it requires acquiring an LWLock; but syslogger doesn't have a PGPROC so
> that doesn't work.
> 
> Apparently we already have a hack in place against the same problem in
> postmaster (and elsewhere?), by testing that ReplicationSlotCtl is not
> null.  But that hack seems to be incomplete, as the crash here attests.
> 
> To reproduce, simply start with no synchronized_standby_slots setting
> and logging_collector=on, then change the value in postgresql.conf and
> reload.
> 
> One option to fix this might be as attached.  The point here is that
> processes without a PGPROC don't need or care to have a correct setting,
> so we just skip verifying it on those.  AFAICS this is more general than
> the test for ReplicationSlotCtl, so I just removed that one.

Thanks for the fix! It looks good to me.

I can also reproduce this bug and confirmed that the bug is fixed after
applying the patch. In addition to the regression tests, I also manually tested
the behavior of the postmaster, walsender, and user backend after reloading the
configuration, and they all work as expected.

Best Regards,
Hou zj


^ permalink  raw  reply  [nested|flat] 5+ messages in thread

* Re: crash with synchronized_standby_slots
@ 2024-12-03 17:04  Alvaro Herrera <alvherre@alvh.no-ip.org>
  parent: Amit Kapila <amit.kapila16@gmail.com>
  0 siblings, 1 reply; 5+ messages in thread

From: Alvaro Herrera @ 2024-12-03 17:04 UTC (permalink / raw)
  To: Amit Kapila <amit.kapila16@gmail.com>; Zhijie Hou (Fujitsu) <houzj.fnst@fujitsu.com>; +Cc: Pg Hackers <pgsql-hackers@lists.postgresql.org>; Gabriele Bartolini <gabriele.bartolini@enterprisedb.com>

On 2024-Nov-29, Amit Kapila wrote:

> I tried it on my Windows machine and noticed that ReplicationSlotCtl
> is NULL for syslogger, so the problem doesn't occur. The reason is
> that we don't attach to shared memory in syslogger, so ideally
> ReplicationSlotCtl should be NULL. Because we inherit everything
> through the fork for Linux systems and then later for processes that
> don't want to attach to shared memory, we call PGSharedMemoryDetach()
> from postmaster_child_launch(). The PGSharedMemoryDetach won't
> reinitialize the memory pointed to by ReplicationSlotCtl, so, it would
> be an invalid memory.

Heh, interesting.  I'm not sure if we should try to do something about
invalid pointers being left around after shmem initialization.  Also, is
this the first GUC check_hook that needs to take an LWLock?

Anyway, I have pushed this.

BTW it occurs to me that there might well be some sort of thundering
herd problem if every process needs to run the check_hook when a SIGHUP
is broadcast, and they'll all be waiting on that particular lwlock and
run the same validation locally again and again.  I bet if you have a
few thousand backends (hi Jakub! [1]) it's problematic.  Maybe we need a
different way to validate the GUC, but I don't know what that might be;
but doing the validation once and storing the result in shmem might be
better.

On 2024-Nov-29, Zhijie Hou (Fujitsu) wrote:

> I can also reproduce this bug and confirmed that the bug is fixed
> after applying the patch. In addition to the regression tests, I also
> manually tested the behavior of the postmaster, walsender, and user
> backend after reloading the configuration, and they all work as
> expected.

Many thanks for testing!

[1] https://postgr.es/m/CAKZiRmwrBjCbCJ433wV5zjvwt_OuY7BsVX12MBKiBu+eNZDm6g@mail.gmail.com

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/
"I apologize for the confusion in my previous responses.
 There appears to be an error."                   (ChatGPT)





^ permalink  raw  reply  [nested|flat] 5+ messages in thread

* Re: crash with synchronized_standby_slots
@ 2024-12-04 05:32  Amit Kapila <amit.kapila16@gmail.com>
  parent: Alvaro Herrera <alvherre@alvh.no-ip.org>
  0 siblings, 0 replies; 5+ messages in thread

From: Amit Kapila @ 2024-12-04 05:32 UTC (permalink / raw)
  To: Alvaro Herrera <alvherre@alvh.no-ip.org>; +Cc: Zhijie Hou (Fujitsu) <houzj.fnst@fujitsu.com>; Pg Hackers <pgsql-hackers@lists.postgresql.org>; Gabriele Bartolini <gabriele.bartolini@enterprisedb.com>

On Tue, Dec 3, 2024 at 10:34 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
>
> On 2024-Nov-29, Amit Kapila wrote:
>
> BTW it occurs to me that there might well be some sort of thundering
> herd problem if every process needs to run the check_hook when a SIGHUP
> is broadcast, and they'll all be waiting on that particular lwlock and
> run the same validation locally again and again.  I bet if you have a
> few thousand backends (hi Jakub! [1]) it's problematic.
>

The lock is taken in shared mode, so, ideally, it shouldn't create a
problem but if it ever creates a problem, we can even skip that check
during validation. The validation will anyway happen later during
replication in StandbySlotsHaveCaughtup(). This check is mostly to
detect the error in GUC early.

>  Maybe we need a
> different way to validate the GUC, but I don't know what that might be;
> but doing the validation once and storing the result in shmem might be
> better.
>

What if that particular GUC changes again? We may need some additional
invalidation mechanism.

-- 
With Regards,
Amit Kapila.





^ permalink  raw  reply  [nested|flat] 5+ messages in thread


end of thread, other threads:[~2024-12-04 05:32 UTC | newest]

Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2024-11-28 12:16 crash with synchronized_standby_slots Alvaro Herrera <alvherre@alvh.no-ip.org>
2024-11-29 06:01 ` Amit Kapila <amit.kapila16@gmail.com>
2024-12-03 17:04   ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2024-12-04 05:32     ` Amit Kapila <amit.kapila16@gmail.com>
2024-11-29 08:19 ` Zhijie Hou (Fujitsu) <houzj.fnst@fujitsu.com>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox