Subject: Re: v15b1: FailedAssertion("segment_map->header->magic == (DSA_SEGMENT_HEADER_MAGIC ^ area->control->handle ^ index)", File: "dsa.c", ..)

From: Thomas Munro <[email-masked]>
Date: Wed, 12 Apr 2023 11:18:36 +1200

On Wed, Apr 12, 2023 at 7:46 AM Justin Pryzby <[email-masked]> wrote:
> Unfortunately:
> (gdb) p area->control->handle
> $3 = 0
> (gdb) p segment_map->header->magic
> value has been optimized out
> (gdb) p index
> $4 = <value optimized out>

Hmm, well index I can find from parameters:

> #2  0x0000000000991470 in ExceptionalCondition (conditionName=<value optimized out>, errorType=<value optimized out>, fileName=<value optimized out>, lineNumber=1770) at assert.c:69
> #3  0x00000000009b9f97 in get_segment_by_index (area=0x22818c0, index=<value optimized out>) at dsa.c:1769
> #4  0x00000000009ba192 in dsa_get_address (area=0x22818c0, dp=1099511703168) at dsa.c:953

We have dp=1099511703168 == 0x10000012680, so index == 1 and the rest
is the offset into that segment.  It's not the initial segment in the
main shared memory area created by the postmaster with
dsa_create_in_place() (that'd be index 0), it's in an extra segment
that was created with shm_open().  We managed to open and mmap() that
segment, but it contains unexpected garbage.

Can you print *area->control?  And then can you see that the DSM
handle is in index 1 in "segment_handles" in there?  Then can you see
if your system has a file with that number in its name under
/dev/shm/, and can you tell me what "od -c /dev/shm/..." shows as the
first few lines of stuff at the top, so we can see what that
unexpected garbage looks like?

Side rant:  I don't think there's any particular indication that it's
the issue here, but while it's on my mind:  I really wish we didn't
use random numbers for DSM handles.  I understand where it came from:
the need to manage SysV shmem keyspace (a DSM mode that almost nobody
uses, but whose limitations apply to all modes).  We've debugged
issues relating to handle collisions before, causing unrelated DSM
segments to be confused, back when the random seed was not different
in each backend making collisions likely.  For every other mode, we
could instead use something like (slot, generation) to keep collisions
as far apart as possible (generation wraparound), and avoid collisions
between unrelated clusters by using the pgdata path as a shm_open()
prefix.  Another idea is to add a new DSM mode that would use memfd
and similar things and pass fds between backends, so that the segments
are entirely anonymous and don't need to be cleaned up after a crash
(I thought about that while studying the reasons why PostgreSQL can't
run on Capsicum (a capabilities research project) or Android (a
telephone), both of which banned SysV *and* POSIX shm because
system-global namespaces are bad).

---

From: Justin Pryzby <[email-masked]>
Date: Tue, 11 Apr 2023 18:37:24 -0500

On Wed, Apr 12, 2023 at 11:18:36AM +1200, Thomas Munro wrote:
> Can you print *area->control?

(gdb) p *area->control
$1 = {segment_header = {magic = 216163848, usable_pages = 62, size = 1048576, prev = 1, next = 18446744073709551615, bin = 4, freed = false}, handle = 0, segment_handles = {0, 3696856876, 433426374, 1403332952, 2754923922, 
    0 <repeats 1019 times>}, segment_bins = {18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 3, 4, 18446744073709551615, 18446744073709551615, 18446744073709551615, 
    18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615}, pools = {{lock = {tranche = 72, state = {value = 536870912}, 
        waiters = {head = 2147483647, tail = 2147483647}}, spans = {0, 2199025295360, 8192, 0}}, {lock = {tranche = 72, state = {value = 536870912}, waiters = {head = 2147483647, tail = 2147483647}}, spans = {0, 
        2199025296088, 0, 0}}, {lock = {tranche = 72, state = {value = 536870912}, waiters = {head = 2147483647, tail = 2147483647}}, spans = {0, 0, 0, 0}}, {lock = {tranche = 72, state = {value = 536870912}, waiters = {
          head = 2147483647, tail = 2147483647}}, spans = {0, 0, 0, 0}}, {lock = {tranche = 72, state = {value = 536870912}, waiters = {head = 2147483647, tail = 2147483647}}, spans = {0, 0, 0, 0}}, {lock = {tranche = 72, 
        state = {value = 536870912}, waiters = {head = 2147483647, tail = 2147483647}}, spans = {0, 0, 0, 0}}, {lock = {tranche = 72, state = {value = 536870912}, waiters = {head = 2147483647, tail = 2147483647}}, spans = {0, 
        0, 0, 0}}, {lock = {tranche = 72, state = {value = 536870912}, waiters = {head = 2147483647, tail = 2147483647}}, spans = {0, 2199025296648, 2199025298608, 0}}, {lock = {tranche = 72, state = {value = 536870912}, 
        waiters = {head = 2147483647, tail = 2147483647}}, spans = {0, 0, 0, 0}}, {lock = {tranche = 72, state = {value = 536870912}, waiters = {head = 2147483647, tail = 2147483647}}, spans = {0, 0, 0, 0}}, {lock = {
        tranche = 72, state = {value = 536870912}, waiters = {head = 2147483647, tail = 2147483647}}, spans = {0, 0, 0, 0}}, {lock = {tranche = 72, state = {value = 536870912}, waiters = {head = 2147483647, 
          tail = 2147483647}}, spans = {0, 0, 0, 0}}, {lock = {tranche = 72, state = {value = 536870912}, waiters = {head = 2147483647, tail = 2147483647}}, spans = {0, 0, 0, 0}}, {lock = {tranche = 72, state = {
          value = 536870912}, waiters = {head = 2147483647, tail = 2147483647}}, spans = {0, 0, 0, 0}}, {lock = {tranche = 72, state = {value = 536870912}, waiters = {head = 2147483647, tail = 2147483647}}, spans = {0, 0, 0, 
        0}}, {lock = {tranche = 72, state = {value = 536870912}, waiters = {head = 2147483647, tail = 2147483647}}, spans = {0, 2199025298496, 2199025298664, 2199025297936}}, {lock = {tranche = 72, state = {
          value = 536870912}, waiters = {head = 2147483647, tail = 2147483647}}, spans = {0, 0, 0, 0}}, {lock = {tranche = 72, state = {value = 536870912}, waiters = {head = 2147483647, tail = 2147483647}}, spans = {0, 0, 0, 
        0}}, {lock = {tranche = 72, state = {value = 536870912}, waiters = {head = 2147483647, tail = 2147483647}}, spans = {0, 8416, 0, 0}}, {lock = {tranche = 72, state = {value = 536870912}, waiters = {head = 2147483647, 
          tail = 2147483647}}, spans = {0, 0, 0, 0}}, {lock = {tranche = 72, state = {value = 536870912}, waiters = {head = 2147483647, tail = 2147483647}}, spans = {0, 0, 0, 0}}, {lock = {tranche = 72, state = {
          value = 536870912}, waiters = {head = 2147483647, tail = 2147483647}}, spans = {0, 0, 0, 0}}, {lock = {tranche = 72, state = {value = 536870912}, waiters = {head = 2147483647, tail = 2147483647}}, spans = {0, 0, 0, 
        0}}, {lock = {tranche = 72, state = {value = 536870912}, waiters = {head = 2147483647, tail = 2147483647}}, spans = {0, 0, 0, 0}}, {lock = {tranche = 72, state = {value = 536870912}, waiters = {head = 2147483647, 
          tail = 2147483647}}, spans = {0, 0, 0, 0}}, {lock = {tranche = 72, state = {value = 536870912}, waiters = {head = 2147483647, tail = 2147483647}}, spans = {0, 8304, 0, 0}}, {lock = {tranche = 72, state = {
          value = 536870912}, waiters = {head = 2147483647, tail = 2147483647}}, spans = {0, 0, 0, 0}}, {lock = {tranche = 72, state = {value = 536870912}, waiters = {head = 2147483647, tail = 2147483647}}, spans = {0, 0, 0, 
        0}}, {lock = {tranche = 72, state = {value = 536870912}, waiters = {head = 2147483647, tail = 2147483647}}, spans = {0, 0, 0, 0}}, {lock = {tranche = 72, state = {value = 536870912}, waiters = {head = 2147483647, 
          tail = 2147483647}}, spans = {0, 8528, 0, 0}}, {lock = {tranche = 72, state = {value = 536870912}, waiters = {head = 2147483647, tail = 2147483647}}, spans = {0, 0, 0, 0}}, {lock = {tranche = 72, state = {
          value = 536870912}, waiters = {head = 2147483647, tail = 2147483647}}, spans = {0, 8248, 0, 0}}, {lock = {tranche = 72, state = {value = 536870912}, waiters = {head = 2147483647, tail = 2147483647}}, spans = {0, 0, 
        0, 0}}, {lock = {tranche = 72, state = {value = 536870912}, waiters = {head = 2147483647, tail = 2147483647}}, spans = {0, 8584, 0, 0}}, {lock = {tranche = 72, state = {value = 536870912}, waiters = {
          head = 2147483647, tail = 2147483647}}, spans = {0, 0, 0, 0}}, {lock = {tranche = 72, state = {value = 536870912}, waiters = {head = 2147483647, tail = 2147483647}}, spans = {0, 0, 0, 0}}, {lock = {tranche = 72, 
        state = {value = 536870912}, waiters = {head = 2147483647, tail = 2147483647}}, spans = {0, 0, 0, 0}}, {lock = {tranche = 72, state = {value = 536870912}, waiters = {head = 2147483647, tail = 2147483647}}, spans = {0, 
        8640, 0, 0}}}, total_segment_size = 9699328, max_total_segment_size = 18446744073709551615, high_segment_index = 4, refcnt = 8455469, pinned = true, freed_segment_counter = 0, lwlock_tranche_id = 72, lock = {
    tranche = 72, state = {value = 536870912}, waiters = {head = 2147483647, tail = 2147483647}}}

> And then can you see that the DSM handle is in index 1 in "segment_handles"
> in there?

(gdb) p area->control->segment_handles 
$2 = {0, 3696856876, 433426374, 1403332952, 2754923922, 0 <repeats 1019 times>}

> Then can you see if your system has a file with that number in its name under
> /dev/shm/,

$ ls /dev/shm/ |grep 3696856876 || echo not found
not found

(In case it matters: the vm has been up for 1558 days).

If it's helpful, I could provide the corefile, unstripped binaries, and
libc.so, which would be enough to use gdb on your side with "set
solib-search-path".

-- 
Justin

---

From: Thomas Munro <[email-masked]>
Date: Wed, 12 Apr 2023 11:49:51 +1200

> $ ls /dev/shm/ |grep 3696856876 || echo not found

Oh, of course it would have restarted after it crashed and unlinked
that...  So the remaining traces of that memory *might* be in the core
file, depending (IIRC) on the core filter settings (you definitely get
shared anonymous memory like our main shm region by default, but IIRC
there's something extra needed if you want the shm_open'd DSM segments
to be dumped too...)

> (In case it matters: the vm has been up for 1558 days).

I will refrain from invoking cosmic radiation at this point :-)

> libc.so, which would be enough to use gdb on your side with "set

Sounds good, thanks, please send them over off-list and I'll see if I
can figure anything out ...