Feedback-ID: i76984098:Fastmail
Message-ID: <25334887-f1c3-40a1-94b0-753c7d67ae2b@aklaver.com>
Date: Thu, 21 Aug 2025 08:04:25 -0700
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: Streaming replica hangs periodically for ~ 1 second - how to
 diagnose/debug
To: depesz@depesz.com
Cc: PostgreSQL General <pgsql-general@lists.postgresql.org>
References: <aKSyq74zIuLS-2B5@depesz.com>
 <d1d40328-ae3a-4089-9ee0-beade92d7c45@aklaver.com>
 <aKWyUpvXqXjJLtVx@depesz.com>
 <05969854-0d19-4726-ae1b-586659dd443b@aklaver.com>
 <aKXy64uaOasow3eK@depesz.com>
 <cbee6b61-1747-45de-9da1-39ce7022d036@aklaver.com>
 <aKbvygjgBWG1WExx@depesz.com>
Content-Language: en-US
From: Adrian Klaver <adrian.klaver@aklaver.com>
In-Reply-To: <aKbvygjgBWG1WExx@depesz.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Archived-At: <https://www.postgresql.org/message-id/25334887-f1c3-40a1-94b0-753c7d67ae2b%40aklaver.com>
Precedence: bulk

On 8/21/25 03:07, hubert depesz lubaczewski wrote:
> On Wed, Aug 20, 2025 at 10:45:13AM -0700, Adrian Klaver wrote:
>> On 8/20/25 09:08, hubert depesz lubaczewski wrote:
>>> On Wed, Aug 20, 2025 at 08:14:47AM -0700, Adrian Klaver wrote:
>>>> Hmm.
>>>>
>>>>   From initial post:
>>>>
>>>> "For ~ 1 second there are no logs going to log (we usually have at 5-20
>>>> messages logged per second), no connection, nothing. And then we get
>>>> bunch (30+) messages with the same milisecond time."
>>>> Are the 30+ messages all coming in on one connection or multiple
>>>> connections?
>>> Multiple connections.
>>>> Also to be clear these are statements that are being run on the replica
>>>> locally, correct?
>>> What do you mean locally?
>> I should have been clearer. Are the queries being run against the replica or
>> the primary?
> 
> All to replica. Primary has its own work, of course, but the problem
> we're experiencing is on replicas.

If I am following there is more then one primary --> replica pair and 
the problem exists across all the pairs.

>> How many applications servers are hitting the database?
> 
> To be honest, I'm not sure. I have visibility into dbs, and bouncers,
> not really into Apps. I know that these are automatically dynamically
> scaled, so number of app server is very varying.
> 
> I'd say anything from 40 to 200 app servers hit first layer of bouncers,
> which we usually have 6-9 (2-3 per az).
> 
> These go to 2nd layer of bouncers, on the db server itself.

By bouncer I assume you mean something like pgBouncer, a connection pooler.

Is it possible to determine what bouncer the queries in question are 
coming from?

> 
> Best regards,
> 
> depesz
> 


-- 
Adrian Klaver
adrian.klaver@aklaver.com