public inbox for [email protected]
help / color / mirror / Atom feedRe: Streaming replica hangs periodically for ~ 1 second - how to diagnose/debug
9+ messages / 7 participants
[nested] [flat]
* Re: Streaming replica hangs periodically for ~ 1 second - how to diagnose/debug
@ 2025-08-19 18:39 Adrian Klaver <[email protected]>
0 siblings, 2 replies; 9+ messages in thread
From: Adrian Klaver @ 2025-08-19 18:39 UTC (permalink / raw)
To: [email protected]; PostgreSQL General <[email protected]>
On 8/19/25 10:21, hubert depesz lubaczewski wrote:
> Hi,
> we have following situation: Pg 14.17 (yes, I know, but it can't be
> upgraded now/soon), on Ubuntu focal, in AWS cloud on EC2 server using
> arm64 architecture.
>
> All works, is fine.
>
> Every now and then (usually every 3-5 minutes, but not through the whole
> day), we see situations where every query suddently takes ~ 1 second.
>
Given the subject line, what you are reporting is happening on the
replica, correct?
If so where is the replica relative to the primary, in terms of network
distance?
Also what are the 'hardware' specifications on the replica instance?
--
Adrian Klaver
[email protected]
^ permalink raw reply [nested|flat] 9+ messages in thread
* RE: Streaming replica hangs periodically for ~ 1 second - how to diagnose/debug
@ 2025-08-19 20:25 Scot Kreienkamp <[email protected]>
parent: Adrian Klaver <[email protected]>
1 sibling, 2 replies; 9+ messages in thread
From: Scot Kreienkamp @ 2025-08-19 20:25 UTC (permalink / raw)
To: Adrian Klaver <[email protected]>; [email protected] <[email protected]>; PostgreSQL General <[email protected]>
Synchronous commit? Or asynchronous?
Scot Kreienkamp | Applications Infrastructure Architect | La-Z-Boy Corporate
(734) 384-6403 | 1-734-915-1444 | [email protected]
One La-Z-Boy Drive | Monroe, Michigan 48162 | la-z-boy.com<http://www.la-z-boy.com/;
facebook.com/lazboy<http://facebook.com/lazboy; | instagram.com/lazboy<https://instagram.com/lazboy; | youtube.com/lazboy<http://youtube.com/lazboy;
[cid:lazboy_2024_inc_navy_pms2189_rgb_rev2025_e40a94f2-e344-4a4a-a02c-fc31996e127c.png]
This message is intended only for the individual or entity to which it is addressed. It may contain privileged, confidential information which is exempt from disclosure under applicable laws. If you are not the intended recipient, you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information. If you have received this communication in error, please notify us immediately by e-mail or by telephone at the above number. Thank you.
Attachments:
[application/ms-tnef] winmail.dat (136.2K, 2-winmail.dat)
download
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Streaming replica hangs periodically for ~ 1 second - how to diagnose/debug
@ 2025-08-20 09:41 Alban Hertroys <[email protected]>
parent: Scot Kreienkamp <[email protected]>
1 sibling, 3 replies; 9+ messages in thread
From: Alban Hertroys @ 2025-08-20 09:41 UTC (permalink / raw)
To: Scot Kreienkamp <[email protected]>; +Cc: Adrian Klaver <[email protected]>; [email protected] <[email protected]>; PostgreSQL General <[email protected]>
> On 19 Aug 2025, at 22:25, Scot Kreienkamp <[email protected]> wrote:
>
> 匀礀渀挀栀爀漀渀漀甀猀 挀漀洀洀椀琀㼀 伀爀 愀猀礀渀挀栀爀漀渀漀甀猀㼀ഀഀഀ匀挀漀琀 䬀爀攀椀攀渀欀愀洀瀀 簀 䄀瀀瀀氀椀挀愀琀椀漀渀猀 䤀渀昀爀愀猀琀爀甀挀琀甀爀攀 䄀爀挀栀椀琀攀挀琀 簀 䰀愀ⴀ娀ⴀ䈀漀礀 䌀漀爀瀀漀爀愀琀攀ഀ⠀㜀㌀㐀⤀ ㌀㠀㐀ⴀ㘀㐀 ㌀ 簀 ⴀ㜀㌀㐀ⴀ㤀㔀ⴀ㐀㐀㐀 簀 匀挀漀琀⸀䬀
This looks like a bug in Outlook (at work) that I ran into over 3 years ago after enabling beta-support for UTF-8 encoding… The fun part was that the messages looked perfectly fine on my end, but were gibberish on the receiving ends only. Is that bug still present perhaps?
Or is it just my end that turns Scott's (and only Scott's) messages into gibberish Chinese?
Alban Hertroys
--
There is always an exception to always.
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Streaming replica hangs periodically for ~ 1 second - how to diagnose/debug
@ 2025-08-20 10:53 Ron Johnson <[email protected]>
parent: Alban Hertroys <[email protected]>
2 siblings, 0 replies; 9+ messages in thread
From: Ron Johnson @ 2025-08-20 10:53 UTC (permalink / raw)
To: PostgreSQL General <[email protected]>
Scot's email wasn't gibberish to me (reading from chrome). Spaces between
every character, and 99.9689% signature block (I did the math!), but all
English (and a giant png file).
On Wed, Aug 20, 2025 at 5:41 AM Alban Hertroys <[email protected]> wrote:
>
>
> > On 19 Aug 2025, at 22:25, Scot Kreienkamp <[email protected]>
> wrote:
> >
> > 匀礀渀挀栀爀漀渀漀甀猀 挀漀洀洀椀琀㼀 伀爀 愀猀礀渀挀栀爀漀渀漀甀猀㼀ഀഀഀ匀挀漀琀 䬀爀攀椀攀渀欀愀洀瀀 簀 䄀瀀瀀氀椀挀愀琀椀漀渀猀
> 䤀渀昀爀愀猀琀爀甀挀琀甀爀攀 䄀爀挀栀椀琀攀挀琀 簀 䰀愀ⴀ娀ⴀ䈀漀礀 䌀漀爀瀀漀爀愀琀攀ഀ⠀㜀㌀㐀⤀ ㌀㠀㐀ⴀ㘀㐀 ㌀ 簀
> ⴀ㜀㌀㐀ⴀ㤀㔀ⴀ㐀㐀㐀 簀 匀挀漀琀⸀䬀
>
> This looks like a bug in Outlook (at work) that I ran into over 3 years
> ago after enabling beta-support for UTF-8 encoding… The fun part was that
> the messages looked perfectly fine on my end, but were gibberish on the
> receiving ends only. Is that bug still present perhaps?
>
> Or is it just my end that turns Scott's (and only Scott's) messages into
> gibberish Chinese?
>
> Alban Hertroys
> --
> There is always an exception to always.
>
>
>
>
>
>
>
--
Death to <Redacted>, and butter sauce.
Don't boil me, I'm still alive.
<Redacted> lobster!
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Streaming replica hangs periodically for ~ 1 second - how to diagnose/debug
@ 2025-08-20 11:32 hubert depesz lubaczewski <[email protected]>
parent: Adrian Klaver <[email protected]>
1 sibling, 0 replies; 9+ messages in thread
From: hubert depesz lubaczewski @ 2025-08-20 11:32 UTC (permalink / raw)
To: Adrian Klaver <[email protected]>; +Cc: PostgreSQL General <[email protected]>
On Tue, Aug 19, 2025 at 11:39:03AM -0700, Adrian Klaver wrote:
> > Every now and then (usually every 3-5 minutes, but not through the whole
> > day), we see situations where every query suddently takes ~ 1 second.
> Given the subject line, what you are reporting is happening on the replica,
> correct?
Yes.
> If so where is the replica relative to the primary, in terms of network
> distance?
=$ ping -c 10 primary
reports:
10 packets transmitted, 10 received, 0% packet loss, time 9181ms
rtt min/avg/max/mdev = 0.942/0.956/0.991/0.012 ms
> Also what are the 'hardware' specifications on the replica instance?
c8g.48xlarge ec2 instance. It is arm64, 192 cores, with 384 gb of ram.
As for storage, this is relatitvely slow, because this db is rather
small:
gp3 500gb volume, with 6000 iops. At no point is IO in any way close to
limits, the whole db fits easily in ram.
Best regards,
depesz
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Streaming replica hangs periodically for ~ 1 second - how to diagnose/debug
@ 2025-08-20 11:35 hubert depesz lubaczewski <[email protected]>
parent: Scot Kreienkamp <[email protected]>
1 sibling, 0 replies; 9+ messages in thread
From: hubert depesz lubaczewski @ 2025-08-20 11:35 UTC (permalink / raw)
To: Scot Kreienkamp <[email protected]>; +Cc: Adrian Klaver <[email protected]>; PostgreSQL General <[email protected]>
On Tue, Aug 19, 2025 at 08:25:26PM +0000, Scot Kreienkamp wrote:
> Synchronous commit? Or asynchronous?
Isn't this potential problem on primary, and not replica? Anyway,
synchronous commit is set to "on", and "synchronous_standby_names" is
empty.
Best regards,
depesz
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Streaming replica hangs periodically for ~ 1 second - how to diagnose/debug
@ 2025-08-20 11:42 Pawel Veselov <[email protected]>
parent: Alban Hertroys <[email protected]>
2 siblings, 0 replies; 9+ messages in thread
From: Pawel Veselov @ 2025-08-20 11:42 UTC (permalink / raw)
To: Alban Hertroys <[email protected]>; +Cc: Scot Kreienkamp <[email protected]>; Adrian Klaver <[email protected]>; [email protected] <[email protected]>; PostgreSQL General <[email protected]>
> > 匀礀渀挀栀爀漀渀漀甀猀 挀漀洀洀椀琀㼀 伀爀 愀猀礀渀挀栀爀漀渀漀甀猀㼀ഀഀഀ匀挀漀琀 䬀爀攀椀攀渀欀愀洀瀀 簀 䄀瀀瀀氀椀挀愀琀椀漀渀猀 䤀渀昀爀愀猀琀爀甀挀琀甀爀攀 䄀爀挀栀椀琀攀挀琀 簀 䰀愀ⴀ娀ⴀ䈀漀礀 䌀漀爀瀀漀爀愀琀攀ഀ⠀㜀㌀㐀⤀ ㌀㠀㐀ⴀ㘀㐀 ㌀ 簀 ⴀ㜀㌀㐀ⴀ㤀㔀ⴀ㐀㐀㐀 簀 匀挀漀琀⸀䬀
> This looks like a bug in Outlook (at work) that I ran into over 3 years ago after enabling beta-support for UTF-8 encoding… The fun part was that the messages looked perfectly fine on my end, but were gibberish on the receiving ends only. Is that bug still present perhaps?
Interesting.
Your email client probably incorrectly interprets UTF-16 encoding.
That's what the content charset is in Scott's email.
The first text character in Scott's email is "S" (U0053), but when
sending(?) it translates to 匀 (U5300).
Gmail web also is struggling with it, judging by the spaces I see
between the characters in the Scott's original. Those spaces are
replaced '0x00' chars from the first byte of each UTF-16 char.
>
> Or is it just my end that turns Scott's (and only Scott's) messages into gibberish Chinese?
>
> Alban Hertroys
> --
> There is always an exception to always.
>
>
>
>
>
>
--
With best of best regards
Pawel S. Veselov
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Streaming replica hangs periodically for ~ 1 second - how to diagnose/debug
@ 2025-08-20 12:03 Tom Lane <[email protected]>
parent: Alban Hertroys <[email protected]>
2 siblings, 1 reply; 9+ messages in thread
From: Tom Lane @ 2025-08-20 12:03 UTC (permalink / raw)
To: Alban Hertroys <[email protected]>; +Cc: Scot Kreienkamp <[email protected]>; Adrian Klaver <[email protected]>; [email protected] <[email protected]>; PostgreSQL General <[email protected]>
Alban Hertroys <[email protected]> writes:
> Or is it just my end that turns Scott's (and only Scott's) messages into gibberish Chinese?
What it looks like from here is that the message body is in UTF16:
every other byte is zero. That gives my email client indigestion,
too.
regards, tom lane
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Streaming replica hangs periodically for ~ 1 second - how to diagnose/debug
@ 2025-08-20 20:03 Alban Hertroys <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 0 replies; 9+ messages in thread
From: Alban Hertroys @ 2025-08-20 20:03 UTC (permalink / raw)
To: Scot Kreienkamp <[email protected]>; +Cc: Adrian Klaver <[email protected]>; [email protected] <[email protected]>; PostgreSQL General <[email protected]>; Tom Lane <[email protected]>
> On 20 Aug 2025, at 14:03, Tom Lane <[email protected]> wrote:
>
> Alban Hertroys <[email protected]> writes:
>> Or is it just my end that turns Scott's (and only Scott's) messages into gibberish Chinese?
>
> What it looks like from here is that the message body is in UTF16:
> every other byte is zero. That gives my email client indigestion,
> too.
Actually, I think it’s misreporting the message content as UTF-16, while it is in fact encoded as UTF-8. So the error appears to be on the sender side.
Apparently, Apple Mail.app is rather strict about following the reported encoding, while Gmail for example does some attempt to guess. That would explain the less terrible results with Gmail.
I can base64 decode the message just fine on the command line, which runs in a terminal emulator (Apple’s Terminal.app) that does handle UTF-8, but not UTF-16 by default.
Alban Hertroys
--
Illustratief voor het verschil tussen steek- en snijwapens:
Met mensen die een punthoofd hebben,
kun je goed de draak steken.
Met mensen die scherp zijn,
lukt dat juist slecht.
^ permalink raw reply [nested|flat] 9+ messages in thread
end of thread, other threads:[~2025-08-20 20:03 UTC | newest]
Thread overview: 9+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2025-08-19 18:39 Re: Streaming replica hangs periodically for ~ 1 second - how to diagnose/debug Adrian Klaver <[email protected]>
2025-08-19 20:25 ` Scot Kreienkamp <[email protected]>
2025-08-20 09:41 ` Alban Hertroys <[email protected]>
2025-08-20 10:53 ` Ron Johnson <[email protected]>
2025-08-20 11:42 ` Pawel Veselov <[email protected]>
2025-08-20 12:03 ` Tom Lane <[email protected]>
2025-08-20 20:03 ` Alban Hertroys <[email protected]>
2025-08-20 11:35 ` hubert depesz lubaczewski <[email protected]>
2025-08-20 11:32 ` hubert depesz lubaczewski <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox