MIME-Version: 1.0
References: 
 <CAK-MWwS_2UwF7XPy-8XpcVqUjEveDjDQRbXxPyzvJM+ooWeh9A@mail.gmail.com>
 <b7da2330-1054-4bb8-939a-cb05d377f70f@gmail.com>
 <CAApHDvrduvaF6nN9ZQSfOdE0Na9SwPOkwYKn=Rb2a=Q9B72MAQ@mail.gmail.com>
 <CAK-MWwTL=_NjJc9v241VQdavOC4deWgPjoqDxLbvm=wUFKq=ug@mail.gmail.com>
In-Reply-To: 
 <CAK-MWwTL=_NjJc9v241VQdavOC4deWgPjoqDxLbvm=wUFKq=ug@mail.gmail.com>
From: David Rowley <dgrowleyml@gmail.com>
Date: Tue, 13 May 2025 07:20:21 -0400
Message-ID: 
 <CAApHDvoWhzzD9Z1ttvtR-T3+uAULb4JDZLP9o8OrNcb2-gBkaA@mail.gmail.com>
Subject: Re: inefficient/wrong plan cache mode selection for queries with
 partitioned tables (postgresql 17)
To: Maxim Boguk <maxim.boguk@gmail.com>
Cc: Andrei Lepikhov <lepihov@gmail.com>,
	Pgsql Performance <pgsql-performance@lists.postgresql.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: 
 <https://www.postgresql.org/message-id/CAApHDvoWhzzD9Z1ttvtR-T3%2BuAULb4JDZLP9o8OrNcb2-gBkaA%40mail.gmail.com>
Precedence: bulk

On Tue, 13 May 2025 at 03:19, Maxim Boguk <maxim.boguk@gmail.com> wrote:
> On Mon, May 12, 2025 at 6:01=E2=80=AFPM David Rowley <dgrowleyml@gmail.co=
m> wrote:
>> This is just an artifact of the fact that runtime pruning is not factore=
d into the costs. Note the cost of the generic plan. The plan_cache_mode GU=
C is about the only way to overrule the choice to use the custom plan.
>
> Situation quite the opposite - I need to force a generic plan because it =
has the same execution time as a custom plan but performs 20-50x faster (be=
cause in custom plan case - 95-98% time spent in planning not in execution)=
.

You misunderstood. The choice the planner (or choose_custom_plan) made
to use the custom plan can be overridden with SET plan_cache_mode =3D
force_generic_plan;, which seems to be what performs better for you,
per your example EXPLAIN ANALYZE outputs.

> And the problem is that the cost of a custom plan ignores the cost of pla=
nning itself (which is like 2x orders of magnitude worse than the cost of r=
eal time partition pruning of a generic plan). I started thinking of someth=
ing like cost_planner GUC to help with similar issues (where planning cost =
calculated as cost_planned*(some heuristic function with amount involved in=
 query tables).
>
> In my case the high cost of planning itself should force the database to =
use generic plan.

Certainly the cost estimate for planning there is quite crude. I doubt
you'll find anyone arguing that it's not. It is however designed to be
low-overhead.  The estimated planning cost isn't the issue here. It's
(as I mentioned) related to no cost consideration being given to
run-time pruning.  We could certainly adjust things so that is
accounted for, and we (I think Robert and I) have talked about it in
the past.  The problem is that doing that is a wild stab in the dark,
especially so for your range partitioned case where the amount of
actual partitions pruned during executor startup could range from 0 to
all of them.  Unfortunately when we tag those costs onto the plan,
we've no idea what the parameter values are going to be when the plan
is executed. I think Robert suggested multiplying the Append cost by
DEFAULT_INEQ_SEL for this bounded range type pruning. Whether that
will help you or not depends on how many partitions you have and how
evenly populated they are.

In order words, it's a tricky problem with no one-size-fits-all solution.

David