Date: Fri, 22 Aug 2025 17:30:22 +0200
From: hubert depesz lubaczewski <depesz@depesz.com>
Sender: depesz@depesz.com
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Adrian Klaver <adrian.klaver@aklaver.com>,
	PostgreSQL General <pgsql-general@lists.postgresql.org>,
	Chris Wilson <chris+google@qwirx.com>
Subject: Re: Streaming replica hangs periodically for ~ 1 second - how to
 diagnose/debug
Message-ID: <aKiNDmLNsNe0OEio@depesz.com>
Reply-To: depesz@depesz.com
References: <aKbvygjgBWG1WExx@depesz.com>
 <25334887-f1c3-40a1-94b0-753c7d67ae2b@aklaver.com>
 <aKc3pidKUSzq-1-8@depesz.com>
 <f32fb472-c5f1-4cd6-a4c3-bc7c160e735b@aklaver.com>
 <aKdOoeD7RTB-isuP@depesz.com>
 <2a3e4a8d-e8c2-46d6-ad7d-9e631ce6725e@aklaver.com>
 <aKhkfOFc4UaEWkmr@depesz.com>
 <fc1e4f47-b479-40ae-914b-6acf323388ad@aklaver.com>
 <aKiGtrwmEMBMXBFr@depesz.com>
 <1882312.1755876082@sss.pgh.pa.us>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <1882312.1755876082@sss.pgh.pa.us>
Archived-At: <https://www.postgresql.org/message-id/aKiNDmLNsNe0OEio%40depesz.com>
Precedence: bulk

On Fri, Aug 22, 2025 at 11:21:22AM -0400, Tom Lane wrote:
> hubert depesz lubaczewski <depesz@depesz.com> writes:
> > I got repeatable case today. Is is breaking on its own everyy
> > ~ 5 minutes.
> 
> Interesting.  That futex call is presumably caused by interaction
> with some other process within the standby server, and the only
> plausible candidate really is the startup process (which is replaying
> WAL received from the primary).  There are cases where WAL replay
> will take locks that can block queries on the standby.  Can you
> correlate the delays on the standby server with any DDL events
> occurring on the primary?

Nope. Plus there is certain repetition of these cases, so even if I'd
miss *some* create table/alter, it just isn't going to be happening
every 4-5 minutes.

For example, looking at logs for the last ~2h, and just checking
situation when there are more than 20 messages in the same milisecond,
I can see:

    108 14:02:03.149
     25 14:04:01.619
    110 14:05:36.924
     77 14:05:36.925
    108 14:09:28.155
     38 14:13:52.481
     63 14:13:52.482
     73 14:13:52.484
    146 14:18:19.338
     39 14:18:19.339
     24 14:20:01.694
     82 14:23:07.352
     55 14:23:07.353
     37 14:23:07.353
     45 14:27:44.125
    132 14:27:44.126
    109 14:31:41.593
     70 14:31:41.594
     24 14:32:01.205
     21 14:34:01.477
     79 14:35:36.761
    104 14:35:36.762
     22 14:39:49.541
    151 14:39:49.542
     22 14:39:49.543
    112 14:44:15.607
     73 14:44:15.608
     28 14:48:01.256
     50 14:48:25.588
    131 14:48:25.589
    139 14:52:44.391
     74 14:57:02.369
    117 14:57:02.370
     20 15:00:02.008
    137 15:00:43.982
     34 15:00:43.983
     20 15:01:01.110
     22 15:04:21.037
    153 15:04:21.038
     20 15:08:01.136
     31 15:08:55.798
    126 15:08:55.799
     76 15:13:46.654
     83 15:13:46.655
     20 15:17:01.700
    107 15:18:42.112
     72 15:18:42.113
    124 15:23:48.689
     32 15:23:48.690
     25 15:23:48.690
     28 15:24:01.397

So, while there are outliers, I'd say that most of the problems happens every
3-5 minutes.

depesz