pg.ddx.io pgsql-bugs@postgresql.org mailing list archive help / color / mirror / Atom feed From: Marco Boeringa <marco@boeringa.demon.nl>
To: Andres Freund <andres@anarazel.de>
Cc: pgsql-bugs@lists.postgresql.org, Thom Brown <thom@linux.com>
Subject: Re: Potential "AIO / io workers" inter-worker locking issue in PG18?
Date: Wed, 8 Oct 2025 00:13:34 +0200
Message-ID: <c31b47d9-c5e4-4c2d-b1dc-22b07e86019e@boeringa.demon.nl> (raw )
In-Reply-To: <cc003455-1773-40b0-a488-281459942542@boeringa.demon.nl >
References: <cc003455-1773-40b0-a488-281459942542@boeringa.demon.nl >
Hi Andres,
This looks much better, doesn't it? I hope this helps. Let me know if
you need anything else.
Marco
*** sudo perf top: ***
Samples: 2M of event 'cpu-clock:ppp', 4000 Hz, Event count (approx.):
593240647834 lost: 0/0 drop: 0/0
Overhead Shared Object Symbol
24,66% [kernel] [k] pv_native_safe_halt
9,35% postgres [.] LWLockAttemptLock
7,69% postgres [.] heap_hot_search_buffer
6,68% postgres [.]
tts_heap_getsomeattrs.lto_priv.0
4,93% postgres [.] LWLockReleaseInternal
4,26% postgres [.] ExecInterpExpr
4,12% postgres [.]
hash_search_with_hash_value
3,42% postgres [.] PinBuffer
2,54% postgres [.]
heapam_index_fetch_tuple.lto_priv.0
2,46% postgres [.] ReleaseAndReadBuffer
1,72% postgres [.] LWLockAcquire
1,72% postgres [.]
HeapTupleSatisfiesVisibility
1,48% postgres [.] index_fetch_heap
1,44% postgres [.] index_getnext_tid
1,40% postgres [.] heap_page_prune_opt
1,28% postgres [.] _bt_readpage
1,28% postgres [.] ExecScan
1,23% postgres [.] LWLockRelease
1,10% postgres [.] IndexNext
1,10% postgres [.] XidInMVCCSnapshot
1,07% postgres [.] _bt_checkkeys
0,98% postgres [.] _bt_next
0,90% postgres [.]
UnpinBufferNoOwner.lto_priv.0
0,82% postgres [.]
GetPrivateRefCountEntry.lto_priv.0
0,81% postgres [.] StartReadBuffer
0,80% postgres [.] btgettuple
0,69% postgres [.]
_bt_check_compare.lto_priv.0
0,67% libxorgxrdp.so [.] crc_process_data
0,56% postgres [.]
ExecStoreBufferHeapTuple
0,55% postgres [.] MemoryContextReset
0,50% postgres [.] ExecEvalSysVar
0,50% postgres [.] hash_bytes
0,48% postgres [.]
tts_virtual_clear.lto_priv.0
0,48% postgres [.] ExecNestLoop
0,43% postgres [.] ResourceOwnerForget
0,41% postgres [.] GlobalVisTestFor
0,37% postgres [.] HeapTupleIsSurelyDead
0,33% postgres [.] slot_getsomeattrs_int
0,33% postgres [.] PredicateLockTID
0,29% postgres [.] ReadBufferExtended
0,27% postgres [.] _bt_saveitem
0,23% postgres [.] _bt_setuppostingitems
0,22% libc.so.6 [.] __memcmp_sse2
0,21% postgres [.] ExecIndexScan
0,20% postgres [.] ReleaseBuffer
0,19% postgres [.] ResourceOwnerEnlarge
*** sudo perf -p <PID of one stuck postgres backend> -g -d 10 ***
*** sudo perf report -g ***
Samples: 40K of event 'task-clock:ppp', Event count (approx.): 10008250000
Children Self Command Shared Object Symbol
+ 100,00% 0,00% postgres postgres [.] _start
+ 100,00% 0,00% postgres libc.so.6 [.]
__libc_start_main@@GLIBC_2.34
+ 100,00% 0,00% postgres libc.so.6 [.]
__libc_start_call_main
+ 100,00% 0,00% postgres postgres [.] main
+ 100,00% 0,00% postgres postgres [.] PostmasterMain
+ 100,00% 0,00% postgres postgres [.] ServerLoop.isra.0
+ 100,00% 0,00% postgres postgres [.]
postmaster_child_launch
+ 100,00% 0,00% postgres postgres [.] 0x00005f3570fb9dbf
+ 100,00% 0,00% postgres postgres [.] PostgresMain
+ 100,00% 0,00% postgres postgres [.] exec_simple_query
+ 100,00% 0,63% postgres postgres [.] ExecNestLoop
+ 100,00% 0,00% postgres postgres [.] PortalRun
+ 100,00% 0,00% postgres postgres [.] PortalRunMulti
+ 100,00% 0,00% postgres postgres [.] ProcessQuery
+ 100,00% 0,00% postgres postgres [.] standard_ExecutorRun
+ 100,00% 0,00% postgres postgres [.] ExecModifyTable
+ 94,63% 1,47% postgres postgres [.] ExecScan
+ 78,76% 1,49% postgres postgres [.] IndexNext
+ 66,89% 1,96% postgres postgres [.] index_fetch_heap
+ 64,35% 3,61% postgres postgres [.]
heapam_index_fetch_tuple.lto_priv.0
+ 21,92% 0,40% postgres postgres [.] ReadBufferExtended
+ 20,36% 1,19% postgres postgres [.] StartReadBuffer
+ 16,23% 5,79% postgres postgres [.] ExecInterpExpr
+ 15,00% 10,70% postgres postgres [.]
heap_hot_search_buffer
+ 12,81% 12,81% postgres postgres [.] LWLockAttemptLock
+ 10,31% 2,36% postgres postgres [.] index_getnext_tid
+ 9,69% 0,48% postgres postgres [.] slot_getsomeattrs_int
+ 9,27% 9,27% postgres postgres [.]
tts_heap_getsomeattrs.lto_priv.0
+ 8,45% 1,85% postgres postgres [.] LWLockRelease
+ 7,91% 1,08% postgres postgres [.] btgettuple
+ 6,89% 1,12% postgres postgres [.] _bt_next
+ 6,77% 6,77% postgres postgres [.] LWLockReleaseInternal
+ 5,89% 5,89% postgres postgres [.]
hash_search_with_hash_value
+ 5,78% 0,10% postgres postgres [.] _bt_readnextpage
+ 4,48% 4,14% postgres postgres [.] PinBuffer
+ 4,41% 1,90% postgres postgres [.] _bt_readpage
+ 4,09% 3,00% postgres postgres [.] ReleaseAndReadBuffer
+ 2,99% 2,38% postgres postgres [.]
HeapTupleSatisfiesVisibility
+ 2,58% 0,82% postgres postgres [.]
ExecStoreBufferHeapTuple
+ 2,43% 2,43% postgres postgres [.] LWLockAcquire
+ 2,28% 1,47% postgres postgres [.] _bt_checkkeys
+ 1,74% 1,74% postgres postgres [.] heap_page_prune_opt
+ 1,67% 1,10% postgres postgres [.]
UnpinBufferNoOwner.lto_priv.0
+ 1,47% 1,47% postgres postgres [.] XidInMVCCSnapshot
+ 1,18% 1,18% postgres postgres [.]
GetPrivateRefCountEntry.lto_priv.0
+ 1,09% 1,09% postgres postgres [.] MemoryContextReset
Op 7-10-2025 om 23:38 schreef Marco Boeringa:
> Hi Andres,
>
> Searching a bit more, and reading the PostgreSQL wiki pages about
> debugging, I have now found the 'find-dbgsym-packages' command, part
> of the 'debian-goodies'
>
> Installing and running this against a postgres process ID, returned
> the following debug symbol packages. Does this list seem about right
> for you?
>
> Marco
>
> lib32stdc++6-14-dbg
>
> libkrb5-dbg
>
> libstdc++6-14-dbg
>
> libx32stdc++6-14-dbg
>
> postgresql-18-dbgsym
>
view thread (56+ messages) latest in thread
Message-ID: <c31b47d9-c5e4-4c2d-b1dc-22b07e86019e@boeringa.demon.nl>
Permalink: ../c31b47d9-c5e4-4c2d-b1dc-22b07e86019e@boeringa.demon.nl/
Also on: postgresql.org/message-id/c31b47d9-c5e4-4c2d-b1dc-22b07e86019e@boeringa.demon.nl
copy link · copy postgr.es
reply Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-bugs@postgresql.org
Cc: marco@boeringa.demon.nl, andres@anarazel.de, thom@linux.com
Subject: Re: Potential "AIO / io workers" inter-worker locking issue in PG18?
In-Reply-To: <c31b47d9-c5e4-4c2d-b1dc-22b07e86019e@boeringa.demon.nl>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox