MIME-Version: 1.0
References: 
 <CAB+Nuk9NAH8MxDK9ng9G+Vvs8XAaaV7Jc=F7FWrbF8U8WWaDEw@mail.gmail.com>
 <CAApHDvqCEWF-3Eg=Lmrv3SGFuFX0KzHf8XfZT22Q1XL=8KN0UA@mail.gmail.com>
In-Reply-To: 
 <CAApHDvqCEWF-3Eg=Lmrv3SGFuFX0KzHf8XfZT22Q1XL=8KN0UA@mail.gmail.com>
From: bruno vieira da silva <brunogiovs@gmail.com>
Date: Thu, 16 Jan 2025 09:56:31 -0500
Message-ID: 
 <CAB+Nuk8bW4xcjGEjANy9manpxOpH=0pRFOQRufwzEUvkG_euDQ@mail.gmail.com>
Subject: Re: Query planning read a large amount of buffers for partitioned
 tables
To: David Rowley <dgrowleyml@gmail.com>
Cc: pgsql-performance@lists.postgresql.org
Content-Type: multipart/alternative; boundary="00000000000082ef80062bd40034"
Archived-At: 
 <https://www.postgresql.org/message-id/CAB%2BNuk8bW4xcjGEjANy9manpxOpH%3D0pRFOQRufwzEUvkG_euDQ%40mail.gmail.com>
Precedence: bulk

--00000000000082ef80062bd40034
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hello, Thanks David.

this pg test deployment. anyways I did a vacuum full on the db. and the
number of buffers read increased a bit.


On Wed, Jan 15, 2025 at 3:01=E2=80=AFPM David Rowley <dgrowleyml@gmail.com>=
 wrote:

> On Thu, 16 Jan 2025 at 07:29, bruno vieira da silva
> <brunogiovs@gmail.com> wrote:
> > On pg 17 now we have better visibility on the I/O required during query
> planning.
> > so, as part of an ongoing design work for table partitioning I was
> analyzing the performance implications of having more or less partitions.
> > In one of my tests of a table with 200 partitions using explain showed =
a
> large amount of buffers read during planning. around 12k buffers.
>
> That's a suspiciously high number of buffers.
>
> > I observed that query planning seems to have a caching mechanism as
> subsequent similar queries require only a fraction of buffers read during
> query planning.
> > However, this "caching" seems to be per session as if I end the client
> session and I reconnect the same query execution will require again to re=
ad
> 12k buffer for query planning.
> >
> > Does pg have any mechanism to mitigate this issue ( new sessions need t=
o
> read a large amount of buffers for query planning) ? or should I mitigate
> this issue by the use of connection pooling.
> > How is this caching done? Is there a way to have viability on its usage=
?
> Where is it stored?
>
> The caching is for relation meta-data and for various catalogue data.
> This is stored in local session hash tables. The caching is done
> lazily the first time something is looked up after the session starts.
> If you're doing very little work before ending the session, then
> you'll pay this overhead much more often than you would if you were to
> do more work in each session. A connection pooler would help you do
> that, otherwise it would need to be a redesign of how you're
> connecting to Postgres from your application.
>
> There's no easy way from EXPLAIN to see which tables or catalogue
> tables the IO is occurring on, however, you might want to try looking
> at pg_statio_all_tables directly before and after the query that's
> causing the 12k buffer accesses and then look at what's changed.
>
> I suspect if you're accessing 12k buffers to run EXPLAIN that you have
> some auto-vacuum starvation issues. Is auto-vacuum enabled and
> running? If you look at pg_stat_activity, do you see autovacuum
> running? It's possible that it's running but not configured to run
> quickly enough to keep up with demand.  Alternatively, it may be
> keeping up now, but at some point in the past, it might not have been
> and you have some bloat either in an index or in a catalogue table as
> a result.
>
> David
>


--=20
Bruno Vieira da Silva

--00000000000082ef80062bd40034
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div dir=3D"ltr">Hello, Thanks David.<div><br></div><div>t=
his pg test deployment. anyways I did a vacuum=C2=A0full on the db. and the=
 number of buffers read increased a bit.=C2=A0</div><div><br></div></div></=
div><br><div class=3D"gmail_quote gmail_quote_container"><div dir=3D"ltr" c=
lass=3D"gmail_attr">On Wed, Jan 15, 2025 at 3:01=E2=80=AFPM David Rowley &l=
t;<a href=3D"mailto:dgrowleyml@gmail.com">dgrowleyml@gmail.com</a>&gt; wrot=
e:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0=
.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Thu, 16 Ja=
n 2025 at 07:29, bruno vieira da silva<br>
&lt;<a href=3D"mailto:brunogiovs@gmail.com" target=3D"_blank">brunogiovs@gm=
ail.com</a>&gt; wrote:<br>
&gt; On pg 17 now we have better visibility on the I/O required during quer=
y planning.<br>
&gt; so, as part of an ongoing design work for table partitioning I was ana=
lyzing the performance implications of having more or less partitions.<br>
&gt; In one of my tests of a table with 200 partitions using explain showed=
 a large amount of buffers read during planning. around 12k buffers.<br>
<br>
That&#39;s a suspiciously high number of buffers.<br>
<br>
&gt; I observed that query planning seems to have a caching mechanism as su=
bsequent similar queries require only a fraction of buffers read during que=
ry planning.<br>
&gt; However, this &quot;caching&quot; seems to be per session as if I end =
the client session and I reconnect the same query execution will require ag=
ain to read 12k buffer for query planning.<br>
&gt;<br>
&gt; Does pg have any mechanism to mitigate this issue ( new sessions need =
to read a large amount of buffers for query planning) ? or should I mitigat=
e this issue by the use of connection pooling.<br>
&gt; How is this caching done? Is there a way to have viability on its usag=
e? Where is it stored?<br>
<br>
The caching is for relation meta-data and for various catalogue data.<br>
This is stored in local session hash tables. The caching is done<br>
lazily the first time something is looked up after the session starts.<br>
If you&#39;re doing very little work before ending the session, then<br>
you&#39;ll pay this overhead much more often than you would if you were to<=
br>
do more work in each session. A connection pooler would help you do<br>
that, otherwise it would need to be a redesign of how you&#39;re<br>
connecting to Postgres from your application.<br>
<br>
There&#39;s no easy way from EXPLAIN to see which tables or catalogue<br>
tables the IO is occurring on, however, you might want to try looking<br>
at pg_statio_all_tables directly before and after the query that&#39;s<br>
causing the 12k buffer accesses and then look at what&#39;s changed.<br>
<br>
I suspect if you&#39;re accessing 12k buffers to run EXPLAIN that you have<=
br>
some auto-vacuum starvation issues. Is auto-vacuum enabled and<br>
running? If you look at pg_stat_activity, do you see autovacuum<br>
running? It&#39;s possible that it&#39;s running but not configured to run<=
br>
quickly enough to keep up with demand.=C2=A0 Alternatively, it may be<br>
keeping up now, but at some point in the past, it might not have been<br>
and you have some bloat either in an index or in a catalogue table as<br>
a result.<br>
<br>
David<br>
</blockquote></div><div><br clear=3D"all"></div><div><br></div><span class=
=3D"gmail_signature_prefix">-- </span><br><div dir=3D"ltr" class=3D"gmail_s=
ignature"><div dir=3D"ltr">Bruno Vieira da Silva<br></div></div>

--00000000000082ef80062bd40034--