MIME-Version: 1.0
References: <aKSyq74zIuLS-2B5@depesz.com> <d1d40328-ae3a-4089-9ee0-beade92d7c45@aklaver.com>
 <aKWyUpvXqXjJLtVx@depesz.com> <05969854-0d19-4726-ae1b-586659dd443b@aklaver.com>
 <aKXy64uaOasow3eK@depesz.com> <cbee6b61-1747-45de-9da1-39ce7022d036@aklaver.com>
 <aKbvygjgBWG1WExx@depesz.com> <25334887-f1c3-40a1-94b0-753c7d67ae2b@aklaver.com>
 <aKc3pidKUSzq-1-8@depesz.com> <f32fb472-c5f1-4cd6-a4c3-bc7c160e735b@aklaver.com>
 <aKdOoeD7RTB-isuP@depesz.com> <2a3e4a8d-e8c2-46d6-ad7d-9e631ce6725e@aklaver.com>
In-Reply-To: <2a3e4a8d-e8c2-46d6-ad7d-9e631ce6725e@aklaver.com>
From: Chris Wilson <chris+google@qwirx.com>
Date: Thu, 21 Aug 2025 19:38:34 +0100
Message-ID: <CAOg7f83dNTc4xmngUqTOPiVLtMsTqjdz9Sr2szfpiYtL+Sf9Tg@mail.gmail.com>
Subject: Re: Streaming replica hangs periodically for ~ 1 second - how to diagnose/debug
To: Adrian Klaver <adrian.klaver@aklaver.com>
Cc: depesz@depesz.com, PostgreSQL General <pgsql-general@lists.postgresql.org>
Content-Type: multipart/alternative; boundary="000000000000bd8df4063ce467bd"
Archived-At: <https://www.postgresql.org/message-id/CAOg7f83dNTc4xmngUqTOPiVLtMsTqjdz9Sr2szfpiYtL%2BSf9Tg%40mail.gmail.com>
Precedence: bulk

--000000000000bd8df4063ce467bd
Content-Type: text/plain; charset="UTF-8"

If all your queries are coming through pgBouncer, and only those hang (the
server itself responds if you connect directly to it), then it might be
this pgBouncer issue:

https://github.com/pgbouncer/pgbouncer/issues/1054

Although that issue is now "closed", because the invisible "debug" log
message was upgraded to a warning (and I don't think that change is in any
released version), the underlying problem still exists: pgbouncer hangs
completely (stops forwarding packets) for a while if the PAM
authentication queue becomes full.

If you have a relatively slow PAM service (such as pam_ldap) then you can
trigger it by opening ~100 connections to pgBouncer simultaneously (without
waiting for previous ones to authenticate), something like this:

for i in `seq 1 100`; do psql -h pgbouncer -p 6432 -U user db_name -c
"SELECT 1" & done


Thanks, Chris.

On Thu, 21 Aug 2025 at 19:17, Adrian Klaver <adrian.klaver@aklaver.com>
wrote:

> On 8/21/25 09:51, hubert depesz lubaczewski wrote:
> > On Thu, Aug 21, 2025 at 08:59:03AM -0700, Adrian Klaver wrote:
> >> Getting to the bottom of the bag of ideas:
> >> Have you looked at the OS system log for the time period involved?
> >
> > Yes. Mostly dmesg. Nothing interesting logged around the time.
> >
> >> You mentioned this seemed to involve PREPARE and DISCARD ALL.
> >> Is this the same set of statements or is it all over the place?
> >
> > No. From what I can tell it's random sample.
> >
> >> Also it would be helpful to know what bouncer you are actually using and
> >> what mode you are running in?
> >
> > pgBouncer, version 1.23.1. As for more... mostly transaction pooling.
> > Applications go using transaction pooling, but people (dbas, ops) have
> > session pooling.
>
> Have you looked at?:
>
> https://www.pgbouncer.org/changelog.html#pgbouncer-124x
>
> To see if anything stands out.
>
> Then there is:
>
> https://www.pgbouncer.org/config.html#max_prepared_statements
>
> The below may also be worth looking at:
>
> https://github.com/pgbouncer/pgbouncer/pull/1144
>
> I can't help thinking that there is a caching issue at stake, though
> that is just a guess.
>
>
> >
> > Best regards,
> >
> > depesz
> >
>
>
> --
> Adrian Klaver
> adrian.klaver@aklaver.com
>
>
>

--000000000000bd8df4063ce467bd
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">If all your queries are coming through pgBouncer, and only=
 those hang (the server itself responds if you connect directly to it), the=
n it might be this pgBouncer issue:<div><br></div><div><a href=3D"https://g=
ithub.com/pgbouncer/pgbouncer/issues/1054">https://github.com/pgbouncer/pgb=
ouncer/issues/1054</a><br></div><div><br></div><div>Although=C2=A0that issu=
e is now &quot;closed&quot;, because the invisible &quot;debug&quot; log me=
ssage was upgraded to a warning (and I don&#39;t think that change is in an=
y released version), the underlying problem still exists: pgbouncer hangs c=
ompletely (stops forwarding packets) for a while if the PAM authentication=
=C2=A0queue becomes full.</div><div><br></div><div>If you have a relatively=
 slow PAM=C2=A0service (such as pam_ldap) then you can trigger it by openin=
g ~100 connections to pgBouncer simultaneously (without waiting for previou=
s ones to authenticate), something like this:</div><div><br></div><blockquo=
te style=3D"margin:0 0 0 40px;border:none;padding:0px"><div><font face=3D"m=
onospace">for i in `seq 1 100`; do psql -h pgbouncer -p 6432 -U user db_nam=
e -c &quot;SELECT 1&quot; &amp; done</font></div></blockquote><div><br></di=
v><div>Thanks, Chris.</div></div><br><div class=3D"gmail_quote gmail_quote_=
container"><div dir=3D"ltr" class=3D"gmail_attr">On Thu, 21 Aug 2025 at 19:=
17, Adrian Klaver &lt;<a href=3D"mailto:adrian.klaver@aklaver.com">adrian.k=
laver@aklaver.com</a>&gt; wrote:<br></div><blockquote class=3D"gmail_quote"=
 style=3D"margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:=
solid;border-left-color:rgb(204,204,204);padding-left:1ex">On 8/21/25 09:51=
, hubert depesz lubaczewski wrote:<br>
&gt; On Thu, Aug 21, 2025 at 08:59:03AM -0700, Adrian Klaver wrote:<br>
&gt;&gt; Getting to the bottom of the bag of ideas:<br>
&gt;&gt; Have you looked at the OS system log for the time period involved?=
<br>
&gt; <br>
&gt; Yes. Mostly dmesg. Nothing interesting logged around the time.<br>
&gt; <br>
&gt;&gt; You mentioned this seemed to involve PREPARE and DISCARD ALL.<br>
&gt;&gt; Is this the same set of statements or is it all over the place?<br=
>
&gt; <br>
&gt; No. From what I can tell it&#39;s random sample.<br>
&gt; <br>
&gt;&gt; Also it would be helpful to know what bouncer you are actually usi=
ng and<br>
&gt;&gt; what mode you are running in?<br>
&gt; <br>
&gt; pgBouncer, version 1.23.1. As for more... mostly transaction pooling.<=
br>
&gt; Applications go using transaction pooling, but people (dbas, ops) have=
<br>
&gt; session pooling.<br>
<br>
Have you looked at?:<br>
<br>
<a href=3D"https://www.pgbouncer.org/changelog.html#pgbouncer-124x" rel=3D"=
noreferrer" target=3D"_blank">https://www.pgbouncer.org/changelog.html#pgbo=
uncer-124x</a><br>
<br>
To see if anything stands out.<br>
<br>
Then there is:<br>
<br>
<a href=3D"https://www.pgbouncer.org/config.html#max_prepared_statements" r=
el=3D"noreferrer" target=3D"_blank">https://www.pgbouncer.org/config.html#m=
ax_prepared_statements</a><br>
<br>
The below may also be worth looking at:<br>
<br>
<a href=3D"https://github.com/pgbouncer/pgbouncer/pull/1144" rel=3D"norefer=
rer" target=3D"_blank">https://github.com/pgbouncer/pgbouncer/pull/1144</a>=
<br>
<br>
I can&#39;t help thinking that there is a caching issue at stake, though <b=
r>
that is just a guess.<br>
<br>
<br>
&gt; <br>
&gt; Best regards,<br>
&gt; <br>
&gt; depesz<br>
&gt; <br>
<br>
<br>
-- <br>
Adrian Klaver<br>
<a href=3D"mailto:adrian.klaver@aklaver.com" target=3D"_blank">adrian.klave=
r@aklaver.com</a><br>
<br>
<br>
</blockquote></div>

--000000000000bd8df4063ce467bd--