agora inbox for [email protected]
help / color / mirror / Atom feedRe: Should we warn against using too many partitions?
86+ messages / 12 participants
[nested] [flat]
* Re: Should we warn against using too many partitions?
@ 2019-06-06 04:43 David Rowley <[email protected]>
0 siblings, 3 replies; 86+ messages in thread
From: David Rowley @ 2019-06-06 04:43 UTC (permalink / raw)
To: Amit Langote <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Robert Haas <[email protected]>; PostgreSQL Hackers <[email protected]>
On Fri, 24 May 2019 at 22:00, David Rowley <[email protected]> wrote:
> I've attached the pg10 and pg11 patches with that updated... and also
> the master one (unchanged) with the hopes that the CF bot picks that
> one.
I got talking to Andres about this at PGCon after a use case of 250k
partitions was brought to our attention. I was thinking about the best
way to handle this on the long flight home and after studying the
current docs I really feel that they fairly well describe what we've
done so far implementing table partitioning, but they offer next to
nothing on best practices on how to make the most of the feature.
I've done some work on this today and what I've ended up with is an
entirely new section to the partitioning docs about best practices
which provides a bit of detail on how you might go about choosing the
partition key. It gives an example of why LIST partitioning on a set
of values that may grow significantly over time might be a bad idea.
It talks about memory growth with more partitions and mentions that
rel cache might become a problem even if queries are touching a small
number of partitions per query, but a large number per session.
The attached patch is aimed at master. PG11 will need the planner
memory and performance part tweaked and for PG10 I'll do that plus
remove the mention of PRIMARY KEY and UNIQUE constraints on the
partitioned table.
Does anyone see anything wrong with doing this? I don't think there
should be an issue adding a section to the docs right at the end as
it's not causing any resequencing.
Or does anyone have any better ideas or better examples to give? or
any comments?
If it looks okay I can post version for PG11 and PG10 for review, but
I'd like to get this in fairly soon.
--
David Rowley http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Attachments:
[application/octet-stream] add_docs_about_best_partition_practices.patch (4.6K, ../../CAKJS1f-0CizMJLXRFo2nHSr04tm4H3gYX_35qAmE4MnO29kmEQ@mail.gmail.com/2-add_docs_about_best_partition_practices.patch)
download | inline diff:
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index cce1618fc1..ab26630199 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -3450,8 +3450,9 @@ VALUES ('Albany', NULL, NULL, 'NY');
</listitem>
</itemizedlist>
- These deficiencies will probably be fixed in some future release,
- but in the meantime considerable care is needed in deciding whether
+ Some functionality not implemented for inheritance hierarchies is
+ implemented for declarative partitioning.
+ Considerable care is needed in deciding whether partitioning with legacy
inheritance is useful for your application.
</para>
@@ -4674,6 +4675,76 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01';
</itemizedlist>
</para>
</sect2>
+
+ <sect2 id="ddl-partitioning-declarative-best-practices">
+ <title>Declarative Partitioning Best Practices</title>
+
+ <para>
+ The choice of how to partition a table should be considered carefully as
+ the performance of query planning and execution can be negatively affected
+ by poorly made design decisions.
+ </para>
+
+ <para>
+ One of the most critical design decisions will be the column or columns
+ which you partition your data by. Often the best choice will be to
+ partition by the column or set of columns which most commonly appear in
+ <literal>WHERE</literal> clauses of queries being executed on the
+ partitioned table. <literal>WHERE</literal> clause items that match and
+ are compatible with the partition key can be used to prune away unneeded
+ partitions. However, you may be forced into making other decisions by
+ requirements for the <literal>PRIMARY KEY</literal> or a
+ <literal>UNIQUE</literal> constraint.
+ </para>
+
+ <para>
+ Choosing the number of partitions to divide the table into is also a
+ critical decision to make. Not having enough partitions may mean that
+ indexes remain too large and that data locality remains poor which could
+ result in poor cache hit ratios. However, dividing the table into too
+ many partitions can also cause issues. Too many partitions can mean
+ slower query planning times and higher memory consumption during both
+ query planning and execution. It's also important to consider what
+ changes may occur in the future when choosing how to partition your table.
+ For example, if you choose to have one partition per customer and you
+ currently have a small number of large customers, what will the
+ implications be if in several years you obtain a large number of small
+ customers. In this case, it may be better to choose to partition by
+ <literal>HASH</literal> and choose a reasonable amount of partitions
+ rather than trying to partition by <literal>LIST</literal> and hoping that
+ the number of customers does not increase significantly over time.
+ </para>
+
+ <para>
+ It is also important to consider the overhead of partitioning during
+ query planning and execution. The query planner is generally able to
+ handle partition hierarchies up a few thousand partitions fairly well,
+ providing that the vast majority of them can be pruned during query
+ planning. Planning times become much slower and memory consumption
+ becomes higher when more partitions remain after partition pruning.
+ This is particularly true for the <command>UPDATE</command> and
+ <command>DELETE</command> commands. Also, even if most queries are
+ able to prune a high number of partitions during query planning, it still
+ may be undesirable to have a large number of partitions as each partition
+ also will obtain a relation cache entry in each session which uses the
+ partition, and if each session touches a large number of partitions over a
+ period of time then the memory consumption for these cache entries may
+ become significant.
+ </para>
+
+ <para>
+ With data warehouse type workloads it can make sense to use a larger
+ number of partitions than with an OLTP type workload. Generally, in data
+ warehouses, query planning time is less of a concern as the majority of
+ processing time is generally spent during query execution. With either of
+ these two types of workload, it is important to make the right decisions
+ early as re-partitioning large quantities of data can be painstakingly
+ slow. When performance is critical, performing workload simulations to
+ assist in making the correct decisions can be beneficial. Never assume
+ that more partitions are better than fewer partitions and vice-versa.
+ </para>
+ </sect2>
+
</sect1>
<sect1 id="ddl-foreign-data">
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Should we warn against using too many partitions?
@ 2019-06-06 05:29 Justin Pryzby <[email protected]>
parent: David Rowley <[email protected]>
2 siblings, 1 reply; 86+ messages in thread
From: Justin Pryzby @ 2019-06-06 05:29 UTC (permalink / raw)
To: David Rowley <[email protected]>; +Cc: Amit Langote <[email protected]>; Robert Haas <[email protected]>; PostgreSQL Hackers <[email protected]>
I suggest just minor variations on language.
On Thu, Jun 06, 2019 at 04:43:48PM +1200, David Rowley wrote:
>diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
>index cce1618fc1..ab26630199 100644
>--- a/doc/src/sgml/ddl.sgml
>+++ b/doc/src/sgml/ddl.sgml
>@@ -4674,6 +4675,76 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01';
> </itemizedlist>
> </para>
> </sect2>
>+
>+ <sect2 id="ddl-partitioning-declarative-best-practices">
>+ <title>Declarative Partitioning Best Practices</title>
>+
>+ <para>
>+ The choice of how to partition a table should be considered carefully as
Either say "How to partition consider should be .." or "The choice should MADE carefully" ?
>+ <para>
>+ One of the most critical design decisions will be the column or columns
>+ which you partition your data by. Often the best choice will be to
by which ?
>+ <para>
>+ Choosing the number of partitions to divide the table into is also a
the TARGET number of partitions BY WHICH to divide the table ?
>+ critical decision to make. Not having enough partitions may mean that
>+ indexes remain too large and that data locality remains poor which could
>+ result in poor cache hit ratios. However, dividing the table into too
>+ many partitions can also cause issues. Too many partitions can mean
>+ slower query planning times and higher memory consumption during both
>+ query planning and execution. It's also important to consider what
>+ changes may occur in the future when choosing how to partition your table.
>+ For example, if you choose to have one partition per customer and you
>+ currently have a small number of large customers, what will the
have ONLY ?
>+ implications be if in several years you obtain a large number of small
>+ customers. In this case, it may be better to choose to partition by
>+ <literal>HASH</literal> and choose a reasonable amount of partitions
reasonable NUMBER ?
>+ <para>
>+ It is also important to consider the overhead of partitioning during
>+ query planning and execution. The query planner is generally able to
>+ handle partition hierarchies up a few thousand partitions fairly well,
>+ providing that the vast majority of them can be pruned during query
provided ?
I would say: "provided that typical queries prune all but a small number of
partitions during planning time".
>+ <command>DELETE</command> commands. Also, even if most queries are
>+ able to prune a high number of partitions during query planning, it still
LARGE number?
>+ may be undesirable to have a large number of partitions as each partition
may still ?
>+ also will obtain a relation cache entry in each session which uses the
will require ? Or occupy ?
>+ <para>
>+ With data warehouse type workloads it can make sense to use a larger
>+ number of partitions than with an OLTP type workload. Generally, in data
>+ warehouses, query planning time is less of a concern as the majority of
>+ processing time is generally spent during query execution. With either of
remove the 2nd "generally"
>+ these two types of workload, it is important to make the right decisions
>+ early as re-partitioning large quantities of data can be painstakingly
early COMMA ?
PAINFULLY slow
>+ When performance is critical, performing workload simulations to
>+ assist in making the correct decisions can be beneficial.
I would say:
Simulations of the intended workload are beneficial for optimizing partitioning
strategy.
Thanks,
Justin
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Should we warn against using too many partitions?
@ 2019-06-06 05:46 Amit Langote <[email protected]>
parent: David Rowley <[email protected]>
2 siblings, 0 replies; 86+ messages in thread
From: Amit Langote @ 2019-06-06 05:46 UTC (permalink / raw)
To: David Rowley <[email protected]>; +Cc: Amit Langote <[email protected]>; Justin Pryzby <[email protected]>; Robert Haas <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi,
On Thu, Jun 6, 2019 at 1:44 PM David Rowley
<[email protected]> wrote:
>
> On Fri, 24 May 2019 at 22:00, David Rowley <[email protected]> wrote:
> > I've attached the pg10 and pg11 patches with that updated... and also
> > the master one (unchanged) with the hopes that the CF bot picks that
> > one.
>
> I got talking to Andres about this at PGCon after a use case of 250k
> partitions was brought to our attention. I was thinking about the best
> way to handle this on the long flight home and after studying the
> current docs I really feel that they fairly well describe what we've
> done so far implementing table partitioning, but they offer next to
> nothing on best practices on how to make the most of the feature.
Agreed that some "best practices" text is overdue, so thanks for taking that up.
> I've done some work on this today and what I've ended up with is an
> entirely new section to the partitioning docs about best practices
> which provides a bit of detail on how you might go about choosing the
> partition key. It gives an example of why LIST partitioning on a set
> of values that may grow significantly over time might be a bad idea.
Design advice like this is good.
> It talks about memory growth with more partitions and mentions that
> rel cache might become a problem even if queries are touching a small
> number of partitions per query, but a large number per session.
I wasn't sure at first if stuff like this should be mentioned in the
user-facing documentation, but your wording seems fine in general.
Thanks,
Amit
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Should we warn against using too many partitions?
@ 2019-06-06 15:12 Alvaro Herrera <[email protected]>
parent: David Rowley <[email protected]>
2 siblings, 2 replies; 86+ messages in thread
From: Alvaro Herrera @ 2019-06-06 15:12 UTC (permalink / raw)
To: David Rowley <[email protected]>; +Cc: Amit Langote <[email protected]>; Justin Pryzby <[email protected]>; Robert Haas <[email protected]>; PostgreSQL Hackers <[email protected]>
On 2019-Jun-06, David Rowley wrote:
> The attached patch is aimed at master. PG11 will need the planner
> memory and performance part tweaked and for PG10 I'll do that plus
> remove the mention of PRIMARY KEY and UNIQUE constraints on the
> partitioned table.
I think in PG10 something should be mentioned about PK and UNIQUE, so
that people doing their partitioning on that release can think ahead.
We don't want them to have to redesign and redo the whole setup when
upgrading to a newer release. If we had written the pg10 material back
when pg10 was fresh, it wouldn't make sense, but now that we know the
future, I don't see why we wouldn't do it. Maybe something like "The
current version does not support <this>, but future Postgres versions
do; consult their manuals for some limitations that may affect the
choice of partitioning strategy".
In the PG10 version you'll need to elide the mention of HASH
partitioning strategy.
Generally speaking, your material looks good to me. Also generally I +1
Justin's suggestions. The part that mentions a "relation cache entry"
seems too low-level as-is, though ... maybe just say it uses some memory
per partition without being too specific.
I think it'd be worthwhile to mention sub-partitioning.
I wonder if the PG10 manual should just suggest to skip to PG11 if
they're setting up partitioning for the first time.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Should we warn against using too many partitions?
@ 2019-06-06 18:46 David Rowley <[email protected]>
parent: Justin Pryzby <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: David Rowley @ 2019-06-06 18:46 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: Amit Langote <[email protected]>; Robert Haas <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, 6 Jun 2019 at 17:29, Justin Pryzby <[email protected]> wrote:
> >+
> >+ <sect2 id="ddl-partitioning-declarative-best-practices">
> >+ <title>Declarative Partitioning Best Practices</title>
> >+
> >+ <para>
> >+ The choice of how to partition a table should be considered carefully as
>
> Either say "How to partition consider should be .." or "The choice should MADE carefully" ?
I've changed "considered" to "made". I'm unable to make sense of the
first suggestion there :(
> >+ <para>
> >+ One of the most critical design decisions will be the column or columns
> >+ which you partition your data by. Often the best choice will be to
>
> by which ?
okay. I've moved the "by" from after "data" to before "which"
> >+ <para>
> >+ Choosing the number of partitions to divide the table into is also a
>
> the TARGET number of partitions BY WHICH to divide the table ?
Changed.
> >+ critical decision to make. Not having enough partitions may mean that
> >+ indexes remain too large and that data locality remains poor which could
> >+ result in poor cache hit ratios. However, dividing the table into too
> >+ many partitions can also cause issues. Too many partitions can mean
> >+ slower query planning times and higher memory consumption during both
> >+ query planning and execution. It's also important to consider what
> >+ changes may occur in the future when choosing how to partition your table.
> >+ For example, if you choose to have one partition per customer and you
> >+ currently have a small number of large customers, what will the
>
> have ONLY ?
I assume you mean after the "have" before "one partition per
customer"? I don't quite understand that since in the scenario we're
partitioning by customer, so it's not possible to have more than one
partition per customer, only the reverse is possible. It seems to me
injecting "only" there would just confuse things.
> >+ implications be if in several years you obtain a large number of small
> >+ customers. In this case, it may be better to choose to partition by
> >+ <literal>HASH</literal> and choose a reasonable amount of partitions
>
> reasonable NUMBER ?
changed.
> >+ <para>
> >+ It is also important to consider the overhead of partitioning during
> >+ query planning and execution. The query planner is generally able to
> >+ handle partition hierarchies up a few thousand partitions fairly well,
> >+ providing that the vast majority of them can be pruned during query
>
> provided ?
>
> I would say: "provided that typical queries prune all but a small number of
> partitions during planning time".
changed, only I used "during query planning" rather than "during planning time".
> >+ <command>DELETE</command> commands. Also, even if most queries are
> >+ able to prune a high number of partitions during query planning, it still
>
> LARGE number?
changed
> >+ may be undesirable to have a large number of partitions as each partition
>
> may still ?
>
> >+ also will obtain a relation cache entry in each session which uses the
>
> will require ? Or occupy ?
"require" seems better. Although, this may need to be reworded a bit
further per what Alvaro mentions.
> >+ <para>
> >+ With data warehouse type workloads it can make sense to use a larger
> >+ number of partitions than with an OLTP type workload. Generally, in data
> >+ warehouses, query planning time is less of a concern as the majority of
> >+ processing time is generally spent during query execution. With either of
>
> remove the 2nd "generally"
Oops. I should have caught that.
> >+ these two types of workload, it is important to make the right decisions
> >+ early as re-partitioning large quantities of data can be painstakingly
>
> early COMMA ?
removed
> PAINFULLY slow
yeah
> >+ When performance is critical, performing workload simulations to
> >+ assist in making the correct decisions can be beneficial.
>
> I would say:
> Simulations of the intended workload are beneficial for optimizing partitioning
> strategy.
I took that but added "often" before "beneficial"
I'll write the patches for PG10 and PG11 and send them all a bit later.
Thanks for the review.
--
David Rowley http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Should we warn against using too many partitions?
@ 2019-06-06 18:54 Justin Pryzby <[email protected]>
parent: David Rowley <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Justin Pryzby @ 2019-06-06 18:54 UTC (permalink / raw)
To: David Rowley <[email protected]>; +Cc: Amit Langote <[email protected]>; Robert Haas <[email protected]>; PostgreSQL Hackers <[email protected]>
On Fri, Jun 07, 2019 at 06:46:59AM +1200, David Rowley wrote:
> On Thu, 6 Jun 2019 at 17:29, Justin Pryzby <[email protected]> wrote:
> > >+
> > >+ <sect2 id="ddl-partitioning-declarative-best-practices">
> > >+ <title>Declarative Partitioning Best Practices</title>
> > >+
> > >+ <para>
> > >+ The choice of how to partition a table should be considered carefully as
> >
> > Either say "How to partition consider should be .." or "The choice should MADE carefully" ?
>
> I've changed "considered" to "made". I'm unable to make sense of the
> first suggestion there :(
The first option was intended to be:
|How to partition a table should be considered carefully.
(The idea being that the "choice" doesn't need to be considered carefully but
the thing itself).
> > >+ critical decision to make. Not having enough partitions may mean that
> > >+ indexes remain too large and that data locality remains poor which could
> > >+ result in poor cache hit ratios. However, dividing the table into too
> > >+ many partitions can also cause issues. Too many partitions can mean
> > >+ slower query planning times and higher memory consumption during both
> > >+ query planning and execution. It's also important to consider what
> > >+ changes may occur in the future when choosing how to partition your table.
> > >+ For example, if you choose to have one partition per customer and you
> > >+ currently have a small number of large customers, what will the
> >
> > have ONLY ?
>
> I assume you mean after the "have" before "one partition per
> customer"?
No, I meant "currently have ONLY".
> I don't quite understand that since in the scenario we're
> partitioning by customer, so it's not possible to have more than one
> partition per customer, only the reverse is possible. It seems to me
> injecting "only" there would just confuse things.
Thanks,
Justin
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Should we warn against using too many partitions?
@ 2019-06-06 18:59 David Rowley <[email protected]>
parent: Alvaro Herrera <[email protected]>
1 sibling, 0 replies; 86+ messages in thread
From: David Rowley @ 2019-06-06 18:59 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Amit Langote <[email protected]>; Justin Pryzby <[email protected]>; Robert Haas <[email protected]>; PostgreSQL Hackers <[email protected]>
On Fri, 7 Jun 2019 at 03:12, Alvaro Herrera <[email protected]> wrote:
> I think in PG10 something should be mentioned about PK and UNIQUE, so
> that people doing their partitioning on that release can think ahead.
That seems reasonable, but I feel caution would be required as we
don't want to provide any details about what a future version will
support, such information might not age very well. We could say that
future versions of PostgreSQL support PRIMARY KEY and UNIQUE
constraints, but we'll be unable to detail out that these must be a
super-set of the partition columns as if we get global indexes one day
that will no longer be a restriction. I'll have a think about it and
post a PG10 patch later.
> We don't want them to have to redesign and redo the whole setup when
> upgrading to a newer release. If we had written the pg10 material back
> when pg10 was fresh, it wouldn't make sense, but now that we know the
> future, I don't see why we wouldn't do it. Maybe something like "The
> current version does not support <this>, but future Postgres versions
> do; consult their manuals for some limitations that may affect the
> choice of partitioning strategy".
> In the PG10 version you'll need to elide the mention of HASH
> partitioning strategy.
Good point. I might need to rethink that example completely as I'm not
sure if swapping HASH for RANGE is such a great fix.
> Generally speaking, your material looks good to me. Also generally I +1
> Justin's suggestions. The part that mentions a "relation cache entry"
> seems too low-level as-is, though ... maybe just say it uses some memory
> per partition without being too specific.
Yeah, I wondered about that. I did grep the docs for "relation cache"
and saw two other mentions, that's why I ended up going with it, but I
do agree that it may be a problem since there's nothing in the docs
that explain what that actually means.
> I think it'd be worthwhile to mention sub-partitioning.
I'll try to come up with something for that.
> I wonder if the PG10 manual should just suggest to skip to PG11 if
> they're setting up partitioning for the first time.
I don't think so. I mean, if they just happened to have just installed
PG10 that might be okay, but they may already be heavily invested in
that version already. Suggesting an upgrade may not be a well-received
recommendation for some. Maybe a suggestion that significant
improvements have been made in later versions might be enough, but I'm
a bit on the fence about that.
Thanks for having a look. I'll post PG10 and 11 patches later.
--
David Rowley http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Should we warn against using too many partitions?
@ 2019-06-06 19:36 David Rowley <[email protected]>
parent: Justin Pryzby <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: David Rowley @ 2019-06-06 19:36 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: Amit Langote <[email protected]>; Robert Haas <[email protected]>; PostgreSQL Hackers <[email protected]>
On Fri, 7 Jun 2019 at 06:54, Justin Pryzby <[email protected]> wrote:
> > > >+ critical decision to make. Not having enough partitions may mean that
> > > >+ indexes remain too large and that data locality remains poor which could
> > > >+ result in poor cache hit ratios. However, dividing the table into too
> > > >+ many partitions can also cause issues. Too many partitions can mean
> > > >+ slower query planning times and higher memory consumption during both
> > > >+ query planning and execution. It's also important to consider what
> > > >+ changes may occur in the future when choosing how to partition your table.
> > > >+ For example, if you choose to have one partition per customer and you
> > > >+ currently have a small number of large customers, what will the
> > >
> > > have ONLY ?
> >
> > I assume you mean after the "have" before "one partition per
> > customer"?
>
> No, I meant "currently have ONLY".
I see, thanks for explaining. I've left that one out as I think adding
"only" would imply that having a small number of large customers is
less significant that a large number of small customers. I don't
really see why either of those has significance over the other, so I
think "only" is out of place there.
--
David Rowley http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Should we warn against using too many partitions?
@ 2019-06-07 05:34 David Rowley <[email protected]>
parent: Alvaro Herrera <[email protected]>
1 sibling, 2 replies; 86+ messages in thread
From: David Rowley @ 2019-06-07 05:34 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Amit Langote <[email protected]>; Justin Pryzby <[email protected]>; Robert Haas <[email protected]>; PostgreSQL Hackers <[email protected]>
On Fri, 7 Jun 2019 at 03:12, Alvaro Herrera <[email protected]> wrote:
> I think it'd be worthwhile to mention sub-partitioning.
In the attached I did briefly mention about sub-partitioning, however,
I didn't feel I had any very wise words to write about it other than
it can be useful to split up larger partitions.
I rather cheaply did the PG10 ones and just removed the mention about
PRIMARY KEYS and UNIQUE constraints. I also mention that PG11 is able
to handle "a few hundred partitions fairly well", and for PG10 I just
wrote that it's able to handle "a few hundred partitions" without the
"fairly well" part. master gets "a few thousand partitions fairly
well".
I also swapped out HASH for RANGE in the PG10 version which is not
quite perfect since its likely a customer ID would be a serial and
would fill the partitions one-by-one rather than more evenly as HASH
partitioning would.
Anyway comments welcome. If I had a few more minutes to spare I'd
have wrapped OLTP in <acronym> tags, but out of time for now.
--
David Rowley http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Attachments:
[application/octet-stream] part_doc_master.patch (5.3K, ../../CAKJS1f9Cs6ibf2cY5gbd3EPLu+SEQLNR0w8S7fh31N+Dj6me7w@mail.gmail.com/2-part_doc_master.patch)
download | inline diff:
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index cce1618fc1..be2ca3be48 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -3450,8 +3450,9 @@ VALUES ('Albany', NULL, NULL, 'NY');
</listitem>
</itemizedlist>
- These deficiencies will probably be fixed in some future release,
- but in the meantime considerable care is needed in deciding whether
+ Some functionality not implemented for inheritance hierarchies is
+ implemented for declarative partitioning.
+ Considerable care is needed in deciding whether partitioning with legacy
inheritance is useful for your application.
</para>
@@ -4674,6 +4675,88 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01';
</itemizedlist>
</para>
</sect2>
+
+ <sect2 id="ddl-partitioning-declarative-best-practices">
+ <title>Declarative Partitioning Best Practices</title>
+
+ <para>
+ The choice of how to partition a table should be made carefully as the
+ performance of query planning and execution can be negatively affected by
+ poorly made design decisions.
+ </para>
+
+ <para>
+ One of the most critical design decisions will be the column or columns
+ by which you partition your data. Often the best choice will be to
+ partition by the column or set of columns which most commonly appear in
+ <literal>WHERE</literal> clauses of queries being executed on the
+ partitioned table. <literal>WHERE</literal> clause items that match and
+ are compatible with the partition key can be used to prune away unneeded
+ partitions. However, you may be forced into making other decisions by
+ requirements for the <literal>PRIMARY KEY</literal> or a
+ <literal>UNIQUE</literal> constraint. Removal of unwanted data is also
+ a factor to consider when planning your partitioning strategy as an entire
+ partition can be removed fairly quickly. However, if data that you want
+ to keep exists in that partition then that means having to resort to using
+ <command>DELETE</command> instead of removing the partition.
+ </para>
+
+ <para>
+ Choosing the target number of partitions by which the table should be
+ divided into is also a critical decision to make. Not having enough
+ partitions may mean that indexes remain too large and that data locality
+ remains poor which could result in poor cache hit ratios. However,
+ dividing the table into too many partitions can also cause issues.
+ Too many partitions can mean slower query planning times and higher memory
+ consumption during both query planning and execution. It's also important
+ to consider what changes may occur in the future when choosing how to
+ partition your table. For example, if you choose to have one partition
+ per customer and you currently have a small number of large customers,
+ what will the implications be if in several years you obtain a large
+ number of small customers. In this case, it may be better to choose to
+ partition by <literal>HASH</literal> and choose a reasonable number of
+ partitions rather than trying to partition by <literal>LIST</literal> and
+ hoping that the number of customers does not increase significantly over
+ time.
+ </para>
+
+ <para>
+ Sub-partitioning can be useful to further divide partitions that are
+ expected to become larger than other partitions, although excessive
+ sub-partitioning can easily lead to large numbers of partitions and can
+ cause the problems mentioned in the preceding paragraph.
+ </para>
+
+ <para>
+ It is also important to consider the overhead of partitioning during
+ query planning and execution. The query planner is generally able to
+ handle partition hierarchies up a few thousand partitions fairly well,
+ provided that typical queries prune all but a small number of partitions
+ during query planning. Planning times become slower and memory
+ consumption becomes higher when more partitions remain after the planner
+ performs partition pruning. This is particularly true for the
+ <command>UPDATE</command> and <command>DELETE</command> commands. Also,
+ even if most queries are able to prune a large number of partitions during
+ query planning, it still may be undesirable to have a large number of
+ partitions as each partition requires metadata about the partition to be
+ stored in each session that touches it. If each session touches a large
+ number of partitions over a period of time then the memory consumption for
+ this may become significant.
+ </para>
+
+ <para>
+ With data warehouse type workloads it can make sense to use a larger
+ number of partitions than with an OLTP type workload. Generally, in data
+ warehouses, query planning time is less of a concern as the majority of
+ processing time is spent during query execution. With either of these two
+ types of workload it is important to make the right decisions early as
+ re-partitioning large quantities of data can be painfully slow.
+ Simulations of the intended workload are often beneficial for optimizing
+ the partitioning strategy. Never assume that more partitions are better
+ than fewer partitions and vice-versa.
+ </para>
+ </sect2>
+
</sect1>
<sect1 id="ddl-foreign-data">
[application/octet-stream] part_doc_pg11.patch (5.0K, ../../CAKJS1f9Cs6ibf2cY5gbd3EPLu+SEQLNR0w8S7fh31N+Dj6me7w@mail.gmail.com/3-part_doc_pg11.patch)
download | inline diff:
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index 39382e99c7..50031ac1e5 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -2833,8 +2833,9 @@ VALUES ('Albany', NULL, NULL, 'NY');
</listitem>
</itemizedlist>
- These deficiencies will probably be fixed in some future release,
- but in the meantime considerable care is needed in deciding whether
+ Some functionality not implemented for inheritance hierarchies is
+ implemented for declarative partitioning.
+ Considerable care is needed in deciding whether partitioning with legacy
inheritance is useful for your application.
</para>
@@ -4057,6 +4058,85 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01';
</itemizedlist>
</para>
</sect2>
+
+ <sect2 id="ddl-partitioning-declarative-best-practices">
+ <title>Declarative Partitioning Best Practices</title>
+
+ <para>
+ The choice of how to partition a table should be made carefully as the
+ performance of query planning and execution can be negatively affected by
+ poorly made design decisions.
+ </para>
+
+ <para>
+ One of the most critical design decisions will be the column or columns
+ by which you partition your data. Often the best choice will be to
+ partition by the column or set of columns which most commonly appear in
+ <literal>WHERE</literal> clauses of queries being executed on the
+ partitioned table. <literal>WHERE</literal> clause items that match and
+ are compatible with the partition key can be used to prune away unneeded
+ partitions. However, you may be forced into making other decisions by
+ requirements for the <literal>PRIMARY KEY</literal> or a
+ <literal>UNIQUE</literal> constraint. Removal of unwanted data is also
+ a factor to consider when planning your partitioning strategy as an entire
+ partition can be removed fairly quickly. However, if data that you want
+ to keep exists in that partition then that means having to resort to using
+ <command>DELETE</command> instead of removing the partition.
+ </para>
+
+ <para>
+ Choosing the target number of partitions by which the table should be
+ divided into is also a critical decision to make. Not having enough
+ partitions may mean that indexes remain too large and that data locality
+ remains poor which could result in poor cache hit ratios. However,
+ dividing the table into too many partitions can also cause issues.
+ Too many partitions can mean slower query planning times and higher memory
+ consumption during both query planning and execution. It's also important
+ to consider what changes may occur in the future when choosing how to
+ partition your table. For example, if you choose to have one partition
+ per customer and you currently have a small number of large customers,
+ what will the implications be if in several years you obtain a large
+ number of small customers. In this case, it may be better to choose to
+ partition by <literal>HASH</literal> and choose a reasonable number of
+ partitions rather than trying to partition by <literal>LIST</literal> and
+ hoping that the number of customers does not increase significantly over
+ time.
+ </para>
+
+ <para>
+ Sub-partitioning can be useful to further divide partitions that are
+ expected to become larger than other partitions, although excessive
+ sub-partitioning can easily lead to large numbers of partitions and can
+ cause the problems mentioned in the preceding paragraph.
+ </para>
+
+ <para>
+ It is also important to consider the overhead of partitioning during
+ query planning and execution. The query planner is generally able to
+ handle partition hierarchies up a few hundred partitions fairly well.
+ With larger numbers of partitions planning times become slower and memory
+ consumption becomes higher. This is particularly true for the
+ <command>UPDATE</command> and <command>DELETE</command> commands. It also
+ may be undesirable to have a large number of partitions as each partition
+ requires metadata about the partition to be stored in each session that
+ touches it. If each session touches a large number of partitions over a
+ period of time then the memory consumption for this may become
+ significant.
+ </para>
+
+ <para>
+ With data warehouse type workloads it can make sense to use a larger
+ number of partitions than with an OLTP type workload. Generally, in data
+ warehouses, query planning time is less of a concern as the majority of
+ processing time is spent during query execution. With either of these two
+ types of workload it is important to make the right decisions early as
+ re-partitioning large quantities of data can be painfully slow.
+ Simulations of the intended workload are often beneficial for optimizing
+ the partitioning strategy. Never assume that more partitions are better
+ than fewer partitions and vice-versa.
+ </para>
+ </sect2>
+
</sect1>
<sect1 id="ddl-foreign-data">
[application/octet-stream] part_doc_pg10.patch (4.9K, ../../CAKJS1f9Cs6ibf2cY5gbd3EPLu+SEQLNR0w8S7fh31N+Dj6me7w@mail.gmail.com/4-part_doc_pg10.patch)
download | inline diff:
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index 13a1001761..45f369806a 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -2833,8 +2833,9 @@ VALUES ('Albany', NULL, NULL, 'NY');
</listitem>
</itemizedlist>
- These deficiencies will probably be fixed in some future release,
- but in the meantime considerable care is needed in deciding whether
+ Some functionality not implemented for inheritance hierarchies is
+ implemented for declarative partitioning.
+ Considerable care is needed in deciding whether partitioning with legacy
inheritance is useful for your application.
</para>
@@ -3927,6 +3928,83 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01';
</itemizedlist>
</para>
</sect2>
+
+ <sect2 id="ddl-partitioning-declarative-best-practices">
+ <title>Declarative Partitioning Best Practices</title>
+
+ <para>
+ The choice of how to partition a table should be made carefully as the
+ performance of query planning and execution can be negatively affected by
+ poorly made design decisions.
+ </para>
+
+ <para>
+ One of the most critical design decisions will be the column or columns
+ by which you partition your data. Often the best choice will be to
+ partition by the column or set of columns which most commonly appear in
+ <literal>WHERE</literal> clauses of queries being executed on the
+ partitioned table. <literal>WHERE</literal> clause items that match and
+ are compatible with the partition key can be used to prune away unneeded
+ partitions. Removal of unwanted data is also a factor to consider when
+ planning your partitioning strategy as an entire partition can be removed
+ fairly quickly. However, if data that you want to keep exists in that
+ partition then that means having to resort to using
+ <command>DELETE</command> instead of removing the partition.
+ </para>
+
+ <para>
+ Choosing the target number of partitions by which the table should be
+ divided into is also a critical decision to make. Not having enough
+ partitions may mean that indexes remain too large and that data locality
+ remains poor which could result in poor cache hit ratios. However,
+ dividing the table into too many partitions can also cause issues.
+ Too many partitions can mean slower query planning times and higher memory
+ consumption during both query planning and execution. It's also important
+ to consider what changes may occur in the future when choosing how to
+ partition your table. For example, if you choose to have one partition
+ per customer and you currently have a small number of large customers,
+ what will the implications be if in several years you obtain a large
+ number of small customers. In this case, it may be better to choose to
+ partition by <literal>RANGE</literal> and choose a reasonable number of
+ partitions rather than trying to partition by <literal>LIST</literal> and
+ hoping that the number of customers does not increase significantly over
+ time.
+ </para>
+
+ <para>
+ Sub-partitioning can be useful to further divide partitions that are
+ expected to become larger than other partitions, although excessive
+ sub-partitioning can easily lead to large numbers of partitions and can
+ cause the problems mentioned in the preceding paragraph.
+ </para>
+
+ <para>
+ It is also important to consider the overhead of partitioning during
+ query planning and execution. The query planner is generally able to
+ handle partition hierarchies up a few hundred partitions. With larger
+ numbers of partitions planning times become slower and memory consumption
+ becomes higher. This is particularly true for the
+ <command>UPDATE</command> and <command>DELETE</command> commands. It also
+ may be undesirable to have a large number of partitions as each partition
+ requires metadata about the partition to be stored in each session that
+ touches it. If each session touches a large number of partitions over a
+ period of time then the memory consumption for this may become
+ significant.
+ </para>
+
+ <para>
+ With data warehouse type workloads it can make sense to use a larger
+ number of partitions than with an OLTP type workload. Generally, in data
+ warehouses, query planning time is less of a concern as the majority of
+ processing time is spent during query execution. With either of these two
+ types of workload it is important to make the right decisions early as
+ re-partitioning large quantities of data can be painfully slow.
+ Simulations of the intended workload are often beneficial for optimizing
+ the partitioning strategy. Never assume that more partitions are better
+ than fewer partitions and vice-versa.
+ </para>
+ </sect2>
+
</sect1>
<sect1 id="ddl-foreign-data">
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Should we warn against using too many partitions?
@ 2019-06-07 06:59 Amit Langote <[email protected]>
parent: David Rowley <[email protected]>
1 sibling, 1 reply; 86+ messages in thread
From: Amit Langote @ 2019-06-07 06:59 UTC (permalink / raw)
To: David Rowley <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Amit Langote <[email protected]>; Justin Pryzby <[email protected]>; Robert Haas <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi,
Thanks for the updated patches.
On Fri, Jun 7, 2019 at 2:34 PM David Rowley
<[email protected]> wrote:
> Anyway comments welcome. If I had a few more minutes to spare I'd
> have wrapped OLTP in <acronym> tags, but out of time for now.
Some rewording suggestions.
1.
+ ... Removal of unwanted data is also a factor to consider when
+ planning your partitioning strategy as an entire partition can be removed
+ fairly quickly. However, if data that you want to keep exists in that
+ partition then that means having to resort to using
+ <command>DELETE</command> instead of removing the partition.
Not sure if the 2nd sentence is necessary or perhaps should be
rewritten in a way that helps to design to benefit from this.
Maybe:
... Removal of unwanted data is also a factor to consider when
planning your partitioning strategy as an entire partition can be
removed fairly quickly, especially if the partition keys are chosen
such that all data that can be deleted together are grouped into
separate partitions.
2.
+ ... For example, if you choose to have one partition
+ per customer and you currently have a small number of large customers,
+ what will the implications be if in several years you obtain a large
+ number of small customers.
The sentence could be rewritten a bit. Maybe as:
... For example, choosing a design with one partition per customer,
because you currently have a small number of large customers, will not
scale well several years down the line when you might have a large
number of small customers.
Btw, doesn't it suffice here to say "large number of customers"
instead of "large number of small customers"?
3.
+ ... In this case, it may be better to choose to
+ partition by <literal>RANGE</literal> and choose a reasonable number of
+ partitions
Maybe:
... and choose reasonable number of partitions, each containing the
data of a fixed number of customers.
4.
+ ... It also
+ may be undesirable to have a large number of partitions as each partition
+ requires metadata about the partition to be stored in each session that
+ touches it. If each session touches a large number of partitions over a
+ period of time then the memory consumption for this may become
+ significant.
It might be a good idea to reorder the sentences here to put the
problem first and the cause later. Maybe like this:
Another reason to be concerned about having a large number of
partitions is that the server's memory consumption may grow
significantly over a period of time, especially if many sessions touch
large numbers of partitions. That's because each partition requires
its own metadata that must be loaded into the local memory of each
session that touches it.
5.
+ With data warehouse type workloads it can make sense to use a larger
+ number of partitions than with an OLTP type workload.
Is there a comma missing between "With data warehouse type workloads"
and the rest of the sentence?
Thanks,
Amit
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Should we warn against using too many partitions?
@ 2019-06-08 06:38 Justin Pryzby <[email protected]>
parent: David Rowley <[email protected]>
1 sibling, 1 reply; 86+ messages in thread
From: Justin Pryzby @ 2019-06-08 06:38 UTC (permalink / raw)
To: David Rowley <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Amit Langote <[email protected]>; Robert Haas <[email protected]>; PostgreSQL Hackers <[email protected]>
I made another pass, hopefully it's useful and not too much of a pain.
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index cce1618fc1..be2ca3be48 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -4674,6 +4675,88 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01';
</itemizedlist>
</para>
</sect2>
+
+ <sect2 id="ddl-partitioning-declarative-best-practices">
+ <title>Declarative Partitioning Best Practices</title>
+
+ <para>
+ The choice of how to partition a table should be made carefully as the
+ performance of query planning and execution can be negatively affected by
+ poorly made design decisions.
Maybe just "poor design"
+ partitioned table. <literal>WHERE</literal> clause items that match and
+ are compatible with the partition key can be used to prune away unneeded
remove "away" ?
+ requirements for the <literal>PRIMARY KEY</literal> or a
+ <literal>UNIQUE</literal> constraint. Removal of unwanted data is also
+ a factor to consider when planning your partitioning strategy as an entire
+ partition can be removed fairly quickly. However, if data that you want
Can we just say "dropped" ? On my first (re)reading, I briefly thought this
was now referring to "pruning" as "removal".
+ to keep exists in that partition then that means having to resort to using
+ <command>DELETE</command> instead of removing the partition.
+ </para>
+
+ <para>
+ Choosing the target number of partitions by which the table should be
+ divided into is also a critical decision to make. Not having enough
Should be: ".. target number .. into which .. should be divided .."
+ partitions may mean that indexes remain too large and that data locality
+ remains poor which could result in poor cache hit ratios. However,
Change the 2nd remains to "is" and the second poor to "low" ?
+ dividing the table into too many partitions can also cause issues.
+ Too many partitions can mean slower query planning times and higher memory
s/slower/longer/
+ consumption during both query planning and execution. It's also important
+ to consider what changes may occur in the future when choosing how to
+ partition your table. For example, if you choose to have one partition
Remove "when choosing ..."? Or say:
|When choosing how to partition your table, it's also important to consider
|what changes may occur in the future.
+ per customer and you currently have a small number of large customers,
+ what will the implications be if in several years you obtain a large
+ number of small customers. In this case, it may be better to choose to
+ partition by <literal>HASH</literal> and choose a reasonable number of
+ partitions rather than trying to partition by <literal>LIST</literal> and
+ hoping that the number of customers does not increase significantly over
+ time.
+ </para>
It's an unusual thing for which to hope :)
+ <para>
+ Sub-partitioning can be useful to further divide partitions that are
+ expected to become larger than other partitions, although excessive
+ sub-partitioning can easily lead to large numbers of partitions and can
+ cause the problems mentioned in the preceding paragraph.
+ </para>
cause the SAME problems ?
+ It is also important to consider the overhead of partitioning during
+ query planning and execution. The query planner is generally able to
+ handle partition hierarchies up a few thousand partitions fairly well,
+ provided that typical queries prune all but a small number of partitions
+ during query planning. Planning times become slower and memory
s/slower/longer/
Hm, maybe say "typical queries ALLOW PRUNNING .."
+ consumption becomes higher when more partitions remain after the planner
+ performs partition pruning. This is particularly true for the
Just say: "remain after planning" ?
+ <command>UPDATE</command> and <command>DELETE</command> commands. Also,
+ even if most queries are able to prune a large number of partitions during
+ query planning, it still may be undesirable to have a large number of
may still ?
+ partitions as each partition requires metadata about the partition to be
+ stored in each session that touches it. If each session touches a large
stored for ?
+ number of partitions over a period of time then the memory consumption for
+ this may become significant.
+ </para>
Remove "over a period of time" ?
Add a comma?
Maybe say:
|If each session touches a large number of partitions, then the memory
|overhead may become significant.
+ <para>
+ With data warehouse type workloads it can make sense to use a larger
+ number of partitions than with an OLTP type workload. Generally, in data
+ warehouses, query planning time is less of a concern as the majority of
VAST majority? Or "essentially all"? Or " .. query planning time is
insignificant compared to the time spent during query execution.
+ processing time is spent during query execution. With either of these two
+ types of workload it is important to make the right decisions early as
early COMMA
Justin
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Should we warn against using too many partitions?
@ 2019-06-08 20:29 David Rowley <[email protected]>
parent: Amit Langote <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: David Rowley @ 2019-06-08 20:29 UTC (permalink / raw)
To: Amit Langote <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Amit Langote <[email protected]>; Justin Pryzby <[email protected]>; Robert Haas <[email protected]>; PostgreSQL Hackers <[email protected]>
Thanks for these suggestions.
On Fri, 7 Jun 2019 at 19:00, Amit Langote <[email protected]> wrote:
> Some rewording suggestions.
>
> 1.
>
> + ... Removal of unwanted data is also a factor to consider when
> + planning your partitioning strategy as an entire partition can be removed
> + fairly quickly. However, if data that you want to keep exists in that
> + partition then that means having to resort to using
> + <command>DELETE</command> instead of removing the partition.
>
> Not sure if the 2nd sentence is necessary or perhaps should be
> rewritten in a way that helps to design to benefit from this.
>
> Maybe:
>
> ... Removal of unwanted data is also a factor to consider when
> planning your partitioning strategy as an entire partition can be
> removed fairly quickly, especially if the partition keys are chosen
> such that all data that can be deleted together are grouped into
> separate partitions.
It seems like a good idea to change this to have this mention the
benefits rather than the drawbacks. I've reworded it, but not using
your exact words as it seems the "especially" means that a partition
can be removed faster with properly chosen partition keys, which is
not the case.
I also split this out into its own paragraph since it's talking about
something quite different from the previous paragraph.
> 2.
>
> + ... For example, if you choose to have one partition
> + per customer and you currently have a small number of large customers,
> + what will the implications be if in several years you obtain a large
> + number of small customers.
>
> The sentence could be rewritten a bit. Maybe as:
>
> ... For example, choosing a design with one partition per customer,
> because you currently have a small number of large customers, will not
> scale well several years down the line when you might have a large
> number of small customers.
>
> Btw, doesn't it suffice here to say "large number of customers"
> instead of "large number of small customers"?
I'm not really trying to imply to plan for business growth here, I'm
trying to angle it as "what if your business changes". I've reworded
this slightly and it now says "what will the implications be if in
several years you instead find yourself with a large number of small
customers."
> 3.
>
> + ... In this case, it may be better to choose to
> + partition by <literal>RANGE</literal> and choose a reasonable number of
> + partitions
>
> Maybe:
>
> ... and choose reasonable number of partitions, each containing the
> data of a fixed number of customers.
Yeah, that seems better. I'll change that for the PG10 version only.
> 4.
>
> + ... It also
> + may be undesirable to have a large number of partitions as each partition
> + requires metadata about the partition to be stored in each session that
> + touches it. If each session touches a large number of partitions over a
> + period of time then the memory consumption for this may become
> + significant.
>
> It might be a good idea to reorder the sentences here to put the
> problem first and the cause later. Maybe like this:
>
> Another reason to be concerned about having a large number of
> partitions is that the server's memory consumption may grow
> significantly over a period of time, especially if many sessions touch
> large numbers of partitions. That's because each partition requires
> its own metadata that must be loaded into the local memory of each
> session that touches it.
That seems better. I've taken that text.
> 5.
>
> + With data warehouse type workloads it can make sense to use a larger
> + number of partitions than with an OLTP type workload.
>
> Is there a comma missing between "With data warehouse type workloads"
> and the rest of the sentence?
I've added one.
Patches will follow once I've addressed Justin's review.
--
David Rowley http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Should we warn against using too many partitions?
@ 2019-06-09 01:15 David Rowley <[email protected]>
parent: Justin Pryzby <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: David Rowley @ 2019-06-09 01:15 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Amit Langote <[email protected]>; Robert Haas <[email protected]>; PostgreSQL Hackers <[email protected]>
Thanks for having another look.
On Sat, 8 Jun 2019 at 18:39, Justin Pryzby <[email protected]> wrote:
> + <para>
> + The choice of how to partition a table should be made carefully as the
> + performance of query planning and execution can be negatively affected by
> + poorly made design decisions.
>
> Maybe just "poor design"
changed
> + partitioned table. <literal>WHERE</literal> clause items that match and
> + are compatible with the partition key can be used to prune away unneeded
>
> remove "away" ?
removed
> + requirements for the <literal>PRIMARY KEY</literal> or a
> + <literal>UNIQUE</literal> constraint. Removal of unwanted data is also
> + a factor to consider when planning your partitioning strategy as an entire
> + partition can be removed fairly quickly. However, if data that you want
>
> Can we just say "dropped" ? On my first (re)reading, I briefly thought this
> was now referring to "pruning" as "removal".
I used removed because that could be done via DROP TABLE or by DETACH
PARTITION. If I change it to "dropped" then it sounds like we might
only mean DROP TABLE. I've reworded to use "detached" instead.
> + to keep exists in that partition then that means having to resort to using
> + <command>DELETE</command> instead of removing the partition.
> + </para>
> +
> + <para>
> + Choosing the target number of partitions by which the table should be
> + divided into is also a critical decision to make. Not having enough
>
> Should be: ".. target number .. into which .. should be divided .."
I've changed "by" to "into". I think that's what you mean, otherwise,
you've lost me.
> + partitions may mean that indexes remain too large and that data locality
> + remains poor which could result in poor cache hit ratios. However,
>
> Change the 2nd remains to "is" and the second poor to "low" ?
An internet search on "low cache hit ratio" turns up about twice as
many results as "poor cache hit ratio", but both seem fine to me.
However, since the search seems to show more for the former, I change
it to that.
> + dividing the table into too many partitions can also cause issues.
> + Too many partitions can mean slower query planning times and higher memory
>
> s/slower/longer/
changed
> + consumption during both query planning and execution. It's also important
> + to consider what changes may occur in the future when choosing how to
> + partition your table. For example, if you choose to have one partition
>
> Remove "when choosing ..."? Or say:
I don't see how that would make sense.
> |When choosing how to partition your table, it's also important to consider
> |what changes may occur in the future.
Changed to that.
> + per customer and you currently have a small number of large customers,
> + what will the implications be if in several years you obtain a large
> + number of small customers. In this case, it may be better to choose to
> + partition by <literal>HASH</literal> and choose a reasonable number of
> + partitions rather than trying to partition by <literal>LIST</literal> and
> + hoping that the number of customers does not increase significantly over
> + time.
> + </para>
>
> It's an unusual thing for which to hope :)
I have reworded this slightly which may help with that.
> + <para>
> + Sub-partitioning can be useful to further divide partitions that are
> + expected to become larger than other partitions, although excessive
> + sub-partitioning can easily lead to large numbers of partitions and can
> + cause the problems mentioned in the preceding paragraph.
> + </para>
>
> cause the SAME problems ?
Added
> + It is also important to consider the overhead of partitioning during
> + query planning and execution. The query planner is generally able to
> + handle partition hierarchies up a few thousand partitions fairly well,
> + provided that typical queries prune all but a small number of partitions
> + during query planning. Planning times become slower and memory
>
> s/slower/longer/
Changed
> Hm, maybe say "typical queries ALLOW PRUNNING .."
>
> + consumption becomes higher when more partitions remain after the planner
> + performs partition pruning. This is particularly true for the
>
> Just say: "remain after planning" ?
I've changed this around, but not really how you've asked.
> + <command>UPDATE</command> and <command>DELETE</command> commands. Also,
> + even if most queries are able to prune a large number of partitions during
> + query planning, it still may be undesirable to have a large number of
>
> may still ?
This has been rewritten per Amit's review.
> + <para>
> + With data warehouse type workloads it can make sense to use a larger
> + number of partitions than with an OLTP type workload. Generally, in data
> + warehouses, query planning time is less of a concern as the majority of
>
> VAST majority? Or "essentially all"? Or " .. query planning time is
> insignificant compared to the time spent during query execution.
I don't see any benefit in raising the significance of that.
> + processing time is spent during query execution. With either of these two
> + types of workload it is important to make the right decisions early as
>
> early COMMA
I'm not really sure what you mean here as I don't see any comma in
that text. I guess you want me to add one? But I'm confused as you
seemed to ask me to remove a comma there in your previous review.
You wrote:
>>+ these two types of workload, it is important to make the right decisions
>>+ early as re-partitioning large quantities of data can be painstakingly
> early COMMA ?
Can you be more precise to the exact problem that you see with the
text? In the meantime, I've put the comma back where it was in the
original patch.
I've attached the updated patches.
--
David Rowley http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Attachments:
[application/octet-stream] part_doc_pg10_v2.patch (5.0K, ../../CAKJS1f9AD5HwvpwAci5Ha+jHfnjTPetiTrG5-=pUV5jw4ht=wA@mail.gmail.com/2-part_doc_pg10_v2.patch)
download | inline diff:
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index 13a1001761..10447d8d89 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -2833,8 +2833,9 @@ VALUES ('Albany', NULL, NULL, 'NY');
</listitem>
</itemizedlist>
- These deficiencies will probably be fixed in some future release,
- but in the meantime considerable care is needed in deciding whether
+ Some functionality not implemented for inheritance hierarchies is
+ implemented for declarative partitioning.
+ Considerable care is needed in deciding whether partitioning with legacy
inheritance is useful for your application.
</para>
@@ -3927,6 +3928,84 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01';
</itemizedlist>
</para>
</sect2>
+
+ <sect2 id="ddl-partitioning-declarative-best-practices">
+ <title>Declarative Partitioning Best Practices</title>
+
+ <para>
+ The choice of how to partition a table should be made carefully as the
+ performance of query planning and execution can be negatively affected by
+ poor design.
+ </para>
+
+ <para>
+ One of the most critical design decisions will be the column or columns
+ by which you partition your data. Often the best choice will be to
+ partition by the column or set of columns which most commonly appear in
+ <literal>WHERE</literal> clauses of queries being executed on the
+ partitioned table. <literal>WHERE</literal> clause items that match and
+ are compatible with the partition key can be used to prune unneeded
+ partitions. Removal of unwanted data is also a factor to consider when
+ planning your partitioning strategy. An entire partition can be detached
+ fairly quickly, so it may be beneficial to design the partition strategy
+ in such a way that all data to be removed at once is located in a single
+ partition.
+ </para>
+
+ <para>
+ Choosing the target number of partitions into which the table should be
+ divided into is also a critical decision to make. Not having enough
+ partitions may mean that indexes remain too large and that data locality
+ remains poor which could result in low cache hit ratios. However,
+ dividing the table into too many partitions can also cause issues.
+ Too many partitions can mean longer query planning times and higher memory
+ consumption during both query planning and execution. When choosing how
+ to partition your table, it's also important to consider what changes may
+ occur in the future. For example, if you choose to have one partition
+ per customer and you currently have a small number of large customers,
+ what will the implications be if in several years you instead find
+ yourself with a large number of small customers. In this case, it may be
+ better to choose to partition by <literal>RANGE</literal> and choose a
+ reasonable number of partitions, each containing a fixed number of
+ customers, rather than trying to partition by <literal>LIST</literal>
+ and hoping that the number of customers does not increase beyond what it
+ is practical to partition the data by.
+ </para>
+
+ <para>
+ Sub-partitioning can be useful to further divide partitions that are
+ expected to become larger than other partitions, although excessive
+ sub-partitioning can easily lead to large numbers of partitions and can
+ cause the same problems mentioned in the preceding paragraph.
+ </para>
+
+ <para>
+ It is also important to consider the overhead of partitioning during
+ query planning and execution. The query planner is generally able to
+ handle partition hierarchies up a few hundred partition. Planning times
+ become longer and memory consumption becomes higher as more partitions are
+ added. This is particularly true for the <command>UPDATE</command> and
+ <command>DELETE</command> commands. Another reason to be concerned about
+ having a large number of partitions is that the server's memory
+ consumption may grow significantly over a period of time, especially if
+ many sessions touch large numbers of partitions. That's because each
+ partition requires its own metadata that must be loaded into the local
+ memory of each session that touches it.
+ </para>
+
+ <para>
+ With data warehouse type workloads, it can make sense to use a larger
+ number of partitions than with an <acronym>OLTP</acronym> type workload.
+ Generally, in data warehouses, query planning time is less of a concern as
+ the majority of processing time is spent during query execution. With
+ either of these two types of workload, it is important to make the right
+ decisions early as re-partitioning large quantities of data can be
+ painfully slow. Simulations of the intended workload are often beneficial
+ for optimizing the partitioning strategy. Never assume that more
+ partitions are better than fewer partitions and vice-versa.
+ </para>
+ </sect2>
+
</sect1>
<sect1 id="ddl-foreign-data">
[application/octet-stream] part_doc_pg11_v2.patch (5.2K, ../../CAKJS1f9AD5HwvpwAci5Ha+jHfnjTPetiTrG5-=pUV5jw4ht=wA@mail.gmail.com/3-part_doc_pg11_v2.patch)
download | inline diff:
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index 39382e99c7..233ff86e1a 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -2833,8 +2833,9 @@ VALUES ('Albany', NULL, NULL, 'NY');
</listitem>
</itemizedlist>
- These deficiencies will probably be fixed in some future release,
- but in the meantime considerable care is needed in deciding whether
+ Some functionality not implemented for inheritance hierarchies is
+ implemented for declarative partitioning.
+ Considerable care is needed in deciding whether partitioning with legacy
inheritance is useful for your application.
</para>
@@ -4057,6 +4058,87 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01';
</itemizedlist>
</para>
</sect2>
+
+ <sect2 id="ddl-partitioning-declarative-best-practices">
+ <title>Declarative Partitioning Best Practices</title>
+
+ <para>
+ The choice of how to partition a table should be made carefully as the
+ performance of query planning and execution can be negatively affected by
+ poor design.
+ </para>
+
+ <para>
+ One of the most critical design decisions will be the column or columns
+ by which you partition your data. Often the best choice will be to
+ partition by the column or set of columns which most commonly appear in
+ <literal>WHERE</literal> clauses of queries being executed on the
+ partitioned table. <literal>WHERE</literal> clause items that match and
+ are compatible with the partition key can be used to prune unneeded
+ partitions. However, you may be forced into making other decisions by
+ requirements for the <literal>PRIMARY KEY</literal> or a
+ <literal>UNIQUE</literal> constraint. Removal of unwanted data is also a
+ factor to consider when planning your partitioning strategy. An entire
+ partition can be detached fairly quickly, so it may be beneficial to
+ design the partition strategy in such a way that all data to be removed
+ at once is located in a single partition.
+ </para>
+
+ <para>
+ Choosing the target number of partitions into which the table should be
+ divided into is also a critical decision to make. Not having enough
+ partitions may mean that indexes remain too large and that data locality
+ remains poor which could result in low cache hit ratios. However,
+ dividing the table into too many partitions can also cause issues.
+ Too many partitions can mean longer query planning times and higher memory
+ consumption during both query planning and execution. When choosing how
+ to partition your table, it's also important to consider what changes may
+ occur in the future. For example, if you choose to have one partition
+ per customer and you currently have a small number of large customers,
+ what will the implications be if in several years you instead find
+ yourself with a large number of small customers. In this case, it may be
+ better to choose to partition by <literal>HASH</literal> and choose a
+ reasonable number of partitions rather than trying to partition by
+ <literal>LIST</literal> and hoping that the number of customers does
+ not increase beyond what it is practical to partition the data by.
+ </para>
+
+ <para>
+ Sub-partitioning can be useful to further divide partitions that are
+ expected to become larger than other partitions, although excessive
+ sub-partitioning can easily lead to large numbers of partitions and can
+ cause the same problems mentioned in the preceding paragraph.
+ </para>
+
+ <para>
+ It is also important to consider the overhead of partitioning during
+ query planning and execution. The query planner is generally able to
+ handle partition hierarchies up a few hundred partitions fairly well,
+ provided that typical queries allow the query planner to prune all but a
+ small number of partitions. Planning times become longer and memory
+ consumption becomes higher as more partitions are added. This is
+ particularly true for the <command>UPDATE</command> and
+ <command>DELETE</command> commands. Another reason to be concerned about
+ having a large number of partitions is that the server's memory
+ consumption may grow significantly over a period of time, especially if
+ many sessions touch large numbers of partitions. That's because each
+ partition requires its own metadata that must be loaded into the local
+ memory of each session that touches it.
+ </para>
+
+ <para>
+ With data warehouse type workloads, it can make sense to use a larger
+ number of partitions than with an <acronym>OLTP</acronym> type workload.
+ Generally, in data warehouses, query planning time is less of a concern as
+ the majority of processing time is spent during query execution. With
+ either of these two types of workload, it is important to make the right
+ decisions early as re-partitioning large quantities of data can be
+ painfully slow. Simulations of the intended workload are often beneficial
+ for optimizing the partitioning strategy. Never assume that more
+ partitions are better than fewer partitions and vice-versa.
+ </para>
+ </sect2>
+
</sect1>
<sect1 id="ddl-foreign-data">
[application/octet-stream] part_doc_master_v2.patch (5.3K, ../../CAKJS1f9AD5HwvpwAci5Ha+jHfnjTPetiTrG5-=pUV5jw4ht=wA@mail.gmail.com/4-part_doc_master_v2.patch)
download | inline diff:
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index cce1618fc1..6ce0bbf0a5 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -3450,8 +3450,9 @@ VALUES ('Albany', NULL, NULL, 'NY');
</listitem>
</itemizedlist>
- These deficiencies will probably be fixed in some future release,
- but in the meantime considerable care is needed in deciding whether
+ Some functionality not implemented for inheritance hierarchies is
+ implemented for declarative partitioning.
+ Considerable care is needed in deciding whether partitioning with legacy
inheritance is useful for your application.
</para>
@@ -4674,6 +4675,87 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01';
</itemizedlist>
</para>
</sect2>
+
+ <sect2 id="ddl-partitioning-declarative-best-practices">
+ <title>Declarative Partitioning Best Practices</title>
+
+ <para>
+ The choice of how to partition a table should be made carefully as the
+ performance of query planning and execution can be negatively affected by
+ poor design.
+ </para>
+
+ <para>
+ One of the most critical design decisions will be the column or columns
+ by which you partition your data. Often the best choice will be to
+ partition by the column or set of columns which most commonly appear in
+ <literal>WHERE</literal> clauses of queries being executed on the
+ partitioned table. <literal>WHERE</literal> clause items that match and
+ are compatible with the partition key can be used to prune unneeded
+ partitions. However, you may be forced into making other decisions by
+ requirements for the <literal>PRIMARY KEY</literal> or a
+ <literal>UNIQUE</literal> constraint. Removal of unwanted data is also a
+ factor to consider when planning your partitioning strategy. An entire
+ partition can be detached fairly quickly, so it may be beneficial to
+ design the partition strategy in such a way that all data to be removed
+ at once is located in a single partition.
+ </para>
+
+ <para>
+ Choosing the target number of partitions into which the table should be
+ divided into is also a critical decision to make. Not having enough
+ partitions may mean that indexes remain too large and that data locality
+ remains poor which could result in low cache hit ratios. However,
+ dividing the table into too many partitions can also cause issues.
+ Too many partitions can mean longer query planning times and higher memory
+ consumption during both query planning and execution. When choosing how
+ to partition your table, it's also important to consider what changes may
+ occur in the future. For example, if you choose to have one partition
+ per customer and you currently have a small number of large customers,
+ what will the implications be if in several years you instead find
+ yourself with a large number of small customers. In this case, it may be
+ better to choose to partition by <literal>HASH</literal> and choose a
+ reasonable number of partitions rather than trying to partition by
+ <literal>LIST</literal> and hoping that the number of customers does
+ not increase beyond what it is practical to partition the data by.
+ </para>
+
+ <para>
+ Sub-partitioning can be useful to further divide partitions that are
+ expected to become larger than other partitions, although excessive
+ sub-partitioning can easily lead to large numbers of partitions and can
+ cause the same problems mentioned in the preceding paragraph.
+ </para>
+
+ <para>
+ It is also important to consider the overhead of partitioning during
+ query planning and execution. The query planner is generally able to
+ handle partition hierarchies up a few thousand partitions fairly well,
+ provided that typical queries allow the query planner to prune all but a
+ small number of partitions. Planning times become longer and memory
+ consumption becomes higher when more partitions remain after the planner
+ performs partition pruning. This is particularly true for the
+ <command>UPDATE</command> and <command>DELETE</command> commands. Another
+ reason to be concerned about having a large number of partitions is that
+ the server's memory consumption may grow significantly over a period of
+ time, especially if many sessions touch large numbers of partitions.
+ That's because each partition requires its own metadata that must be
+ loaded into the local memory of each session that touches it.
+ </para>
+
+ <para>
+ With data warehouse type workloads, it can make sense to use a larger
+ number of partitions than with an <acronym>OLTP</acronym> type workload.
+ Generally, in data warehouses, query planning time is less of a concern as
+ the majority of processing time is spent during query execution. With
+ either of these two types of workload, it is important to make the right
+ decisions early as re-partitioning large quantities of data can be
+ painfully slow. Simulations of the intended workload are often beneficial
+ for optimizing the partitioning strategy. Never assume that more
+ partitions are better than fewer partitions and vice-versa.
+ </para>
+ </sect2>
+
</sect1>
<sect1 id="ddl-foreign-data">
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Should we warn against using too many partitions?
@ 2019-06-09 04:21 Justin Pryzby <[email protected]>
parent: David Rowley <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Justin Pryzby @ 2019-06-09 04:21 UTC (permalink / raw)
To: David Rowley <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Amit Langote <[email protected]>; Robert Haas <[email protected]>; PostgreSQL Hackers <[email protected]>
On Sun, Jun 09, 2019 at 01:15:09PM +1200, David Rowley wrote:
> Thanks for having another look.
>
> On Sat, 8 Jun 2019 at 18:39, Justin Pryzby <[email protected]> wrote:
> > + to keep exists in that partition then that means having to resort to using
> > + <command>DELETE</command> instead of removing the partition.
> > + </para>
> > +
> > + <para>
> > + Choosing the target number of partitions by which the table should be
> > + divided into is also a critical decision to make. Not having enough
> >
> > Should be: ".. target number .. into which .. should be divided .."
>
> I've changed "by" to "into". I think that's what you mean, otherwise,
> you've lost me.
I meant it should say "into which it should be divided" and not "by which it
should be divided INTO", which has too many prepositions. This is still an
issue:
+ Choosing the target number of partitions into which the table should be
+ divided into is also a critical decision to make. Not having enough
> > + partitions may mean that indexes remain too large and that data locality
> > + remains poor which could result in poor cache hit ratios. However,
> >
> > Change the 2nd remains to "is" and the second poor to "low" ?
> > + consumption during both query planning and execution. It's also important
> > + to consider what changes may occur in the future when choosing how to
> > + partition your table. For example, if you choose to have one partition
> >
> > Remove "when choosing ..."? Or say:
>
> I don't see how that would make sense.
I suggested it because otherwise it can read as: "in the future when choosing ...".
> > + per customer and you currently have a small number of large customers,
> > + what will the implications be if in several years you obtain a large
> > + number of small customers. In this case, it may be better to choose to
> > + partition by <literal>HASH</literal> and choose a reasonable number of
> > + partitions rather than trying to partition by <literal>LIST</literal> and
> > + hoping that the number of customers does not increase significantly over
> > + time.
> > + </para>
> >
> > It's an unusual thing for which to hope :)
>
> I have reworded this slightly which may help with that.
I didn't mean there was any issue with this, just that it's amusing to find
oneself in the unfortunate position of hoping that one's company doesn't end up
with many customers.
> > + processing time is spent during query execution. With either of these two
> > + types of workload it is important to make the right decisions early as
> >
> > early COMMA
>
> I'm not really sure what you mean here as I don't see any comma in
> that text. I guess you want me to add one? But I'm confused as you
> seemed to ask me to remove a comma there in your previous review.
I meant to add one then and now, like:
| these two types of workload, it is important to make the right decisions
| early, as re-partitioning large quantities of data can be ...
Thanks,
Justin
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Should we warn against using too many partitions?
@ 2019-06-09 05:07 David Rowley <[email protected]>
parent: Justin Pryzby <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: David Rowley @ 2019-06-09 05:07 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Amit Langote <[email protected]>; Robert Haas <[email protected]>; PostgreSQL Hackers <[email protected]>
On Sun, 9 Jun 2019 at 16:21, Justin Pryzby <[email protected]> wrote:
> I meant it should say "into which it should be divided" and not "by which it
> should be divided INTO", which has too many prepositions. This is still an
> issue:
It now reads "divided by" instead of "divided into".
> | these two types of workload, it is important to make the right decisions
> | early, as re-partitioning large quantities of data can be ...
I've added a comma after "early".
--
David Rowley http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Attachments:
[application/octet-stream] part_doc_master_v3.patch (5.3K, ../../CAKJS1f-zof2ZAnDLWND+ReuXE4qqczs79_GGOu9zfJPHBftbZA@mail.gmail.com/2-part_doc_master_v3.patch)
download | inline diff:
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index cce1618fc1..d1dd0718ef 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -3450,8 +3450,9 @@ VALUES ('Albany', NULL, NULL, 'NY');
</listitem>
</itemizedlist>
- These deficiencies will probably be fixed in some future release,
- but in the meantime considerable care is needed in deciding whether
+ Some functionality not implemented for inheritance hierarchies is
+ implemented for declarative partitioning.
+ Considerable care is needed in deciding whether partitioning with legacy
inheritance is useful for your application.
</para>
@@ -4674,6 +4675,87 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01';
</itemizedlist>
</para>
</sect2>
+
+ <sect2 id="ddl-partitioning-declarative-best-practices">
+ <title>Declarative Partitioning Best Practices</title>
+
+ <para>
+ The choice of how to partition a table should be made carefully as the
+ performance of query planning and execution can be negatively affected by
+ poor design.
+ </para>
+
+ <para>
+ One of the most critical design decisions will be the column or columns
+ by which you partition your data. Often the best choice will be to
+ partition by the column or set of columns which most commonly appear in
+ <literal>WHERE</literal> clauses of queries being executed on the
+ partitioned table. <literal>WHERE</literal> clause items that match and
+ are compatible with the partition key can be used to prune unneeded
+ partitions. However, you may be forced into making other decisions by
+ requirements for the <literal>PRIMARY KEY</literal> or a
+ <literal>UNIQUE</literal> constraint. Removal of unwanted data is also a
+ factor to consider when planning your partitioning strategy. An entire
+ partition can be detached fairly quickly, so it may be beneficial to
+ design the partition strategy in such a way that all data to be removed
+ at once is located in a single partition.
+ </para>
+
+ <para>
+ Choosing the target number of partitions into which the table should be
+ divided by is also a critical decision to make. Not having enough
+ partitions may mean that indexes remain too large and that data locality
+ remains poor which could result in low cache hit ratios. However,
+ dividing the table into too many partitions can also cause issues.
+ Too many partitions can mean longer query planning times and higher memory
+ consumption during both query planning and execution. When choosing how
+ to partition your table, it's also important to consider what changes may
+ occur in the future. For example, if you choose to have one partition
+ per customer and you currently have a small number of large customers,
+ what will the implications be if in several years you instead find
+ yourself with a large number of small customers. In this case, it may be
+ better to choose to partition by <literal>HASH</literal> and choose a
+ reasonable number of partitions rather than trying to partition by
+ <literal>LIST</literal> and hoping that the number of customers does
+ not increase beyond what it is practical to partition the data by.
+ </para>
+
+ <para>
+ Sub-partitioning can be useful to further divide partitions that are
+ expected to become larger than other partitions, although excessive
+ sub-partitioning can easily lead to large numbers of partitions and can
+ cause the same problems mentioned in the preceding paragraph.
+ </para>
+
+ <para>
+ It is also important to consider the overhead of partitioning during
+ query planning and execution. The query planner is generally able to
+ handle partition hierarchies up a few thousand partitions fairly well,
+ provided that typical queries allow the query planner to prune all but a
+ small number of partitions. Planning times become longer and memory
+ consumption becomes higher when more partitions remain after the planner
+ performs partition pruning. This is particularly true for the
+ <command>UPDATE</command> and <command>DELETE</command> commands. Another
+ reason to be concerned about having a large number of partitions is that
+ the server's memory consumption may grow significantly over a period of
+ time, especially if many sessions touch large numbers of partitions.
+ That's because each partition requires its own metadata that must be
+ loaded into the local memory of each session that touches it.
+ </para>
+
+ <para>
+ With data warehouse type workloads, it can make sense to use a larger
+ number of partitions than with an <acronym>OLTP</acronym> type workload.
+ Generally, in data warehouses, query planning time is less of a concern as
+ the majority of processing time is spent during query execution. With
+ either of these two types of workload, it is important to make the right
+ decisions early, as re-partitioning large quantities of data can be
+ painfully slow. Simulations of the intended workload are often beneficial
+ for optimizing the partitioning strategy. Never assume that more
+ partitions are better than fewer partitions and vice-versa.
+ </para>
+ </sect2>
+
</sect1>
<sect1 id="ddl-foreign-data">
[application/octet-stream] part_doc_pg11_v3.patch (5.2K, ../../CAKJS1f-zof2ZAnDLWND+ReuXE4qqczs79_GGOu9zfJPHBftbZA@mail.gmail.com/3-part_doc_pg11_v3.patch)
download | inline diff:
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index 39382e99c7..233ff86e1a 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -2833,8 +2833,9 @@ VALUES ('Albany', NULL, NULL, 'NY');
</listitem>
</itemizedlist>
- These deficiencies will probably be fixed in some future release,
- but in the meantime considerable care is needed in deciding whether
+ Some functionality not implemented for inheritance hierarchies is
+ implemented for declarative partitioning.
+ Considerable care is needed in deciding whether partitioning with legacy
inheritance is useful for your application.
</para>
@@ -4057,6 +4058,87 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01';
</itemizedlist>
</para>
</sect2>
+
+ <sect2 id="ddl-partitioning-declarative-best-practices">
+ <title>Declarative Partitioning Best Practices</title>
+
+ <para>
+ The choice of how to partition a table should be made carefully as the
+ performance of query planning and execution can be negatively affected by
+ poor design.
+ </para>
+
+ <para>
+ One of the most critical design decisions will be the column or columns
+ by which you partition your data. Often the best choice will be to
+ partition by the column or set of columns which most commonly appear in
+ <literal>WHERE</literal> clauses of queries being executed on the
+ partitioned table. <literal>WHERE</literal> clause items that match and
+ are compatible with the partition key can be used to prune unneeded
+ partitions. However, you may be forced into making other decisions by
+ requirements for the <literal>PRIMARY KEY</literal> or a
+ <literal>UNIQUE</literal> constraint. Removal of unwanted data is also a
+ factor to consider when planning your partitioning strategy. An entire
+ partition can be detached fairly quickly, so it may be beneficial to
+ design the partition strategy in such a way that all data to be removed
+ at once is located in a single partition.
+ </para>
+
+ <para>
+ Choosing the target number of partitions into which the table should be
+ divided by is also a critical decision to make. Not having enough
+ partitions may mean that indexes remain too large and that data locality
+ remains poor which could result in low cache hit ratios. However,
+ dividing the table into too many partitions can also cause issues.
+ Too many partitions can mean longer query planning times and higher memory
+ consumption during both query planning and execution. When choosing how
+ to partition your table, it's also important to consider what changes may
+ occur in the future. For example, if you choose to have one partition
+ per customer and you currently have a small number of large customers,
+ what will the implications be if in several years you instead find
+ yourself with a large number of small customers. In this case, it may be
+ better to choose to partition by <literal>HASH</literal> and choose a
+ reasonable number of partitions rather than trying to partition by
+ <literal>LIST</literal> and hoping that the number of customers does
+ not increase beyond what it is practical to partition the data by.
+ </para>
+
+ <para>
+ Sub-partitioning can be useful to further divide partitions that are
+ expected to become larger than other partitions, although excessive
+ sub-partitioning can easily lead to large numbers of partitions and can
+ cause the same problems mentioned in the preceding paragraph.
+ </para>
+
+ <para>
+ It is also important to consider the overhead of partitioning during
+ query planning and execution. The query planner is generally able to
+ handle partition hierarchies up a few hundred partitions fairly well,
+ provided that typical queries allow the query planner to prune all but a
+ small number of partitions. Planning times become longer and memory
+ consumption becomes higher as more partitions are added. This is
+ particularly true for the <command>UPDATE</command> and
+ <command>DELETE</command> commands. Another reason to be concerned about
+ having a large number of partitions is that the server's memory
+ consumption may grow significantly over a period of time, especially if
+ many sessions touch large numbers of partitions. That's because each
+ partition requires its own metadata that must be loaded into the local
+ memory of each session that touches it.
+ </para>
+
+ <para>
+ With data warehouse type workloads, it can make sense to use a larger
+ number of partitions than with an <acronym>OLTP</acronym> type workload.
+ Generally, in data warehouses, query planning time is less of a concern as
+ the majority of processing time is spent during query execution. With
+ either of these two types of workload, it is important to make the right
+ decisions early, as re-partitioning large quantities of data can be
+ painfully slow. Simulations of the intended workload are often beneficial
+ for optimizing the partitioning strategy. Never assume that more
+ partitions are better than fewer partitions and vice-versa.
+ </para>
+ </sect2>
+
</sect1>
<sect1 id="ddl-foreign-data">
[application/octet-stream] part_doc_pg10_v3.patch (5.0K, ../../CAKJS1f-zof2ZAnDLWND+ReuXE4qqczs79_GGOu9zfJPHBftbZA@mail.gmail.com/4-part_doc_pg10_v3.patch)
download | inline diff:
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index 13a1001761..10447d8d89 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -2833,8 +2833,9 @@ VALUES ('Albany', NULL, NULL, 'NY');
</listitem>
</itemizedlist>
- These deficiencies will probably be fixed in some future release,
- but in the meantime considerable care is needed in deciding whether
+ Some functionality not implemented for inheritance hierarchies is
+ implemented for declarative partitioning.
+ Considerable care is needed in deciding whether partitioning with legacy
inheritance is useful for your application.
</para>
@@ -3927,6 +3928,84 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01';
</itemizedlist>
</para>
</sect2>
+
+ <sect2 id="ddl-partitioning-declarative-best-practices">
+ <title>Declarative Partitioning Best Practices</title>
+
+ <para>
+ The choice of how to partition a table should be made carefully as the
+ performance of query planning and execution can be negatively affected by
+ poor design.
+ </para>
+
+ <para>
+ One of the most critical design decisions will be the column or columns
+ by which you partition your data. Often the best choice will be to
+ partition by the column or set of columns which most commonly appear in
+ <literal>WHERE</literal> clauses of queries being executed on the
+ partitioned table. <literal>WHERE</literal> clause items that match and
+ are compatible with the partition key can be used to prune unneeded
+ partitions. Removal of unwanted data is also a factor to consider when
+ planning your partitioning strategy. An entire partition can be detached
+ fairly quickly, so it may be beneficial to design the partition strategy
+ in such a way that all data to be removed at once is located in a single
+ partition.
+ </para>
+
+ <para>
+ Choosing the target number of partitions into which the table should be
+ divided by is also a critical decision to make. Not having enough
+ partitions may mean that indexes remain too large and that data locality
+ remains poor which could result in low cache hit ratios. However,
+ dividing the table into too many partitions can also cause issues.
+ Too many partitions can mean longer query planning times and higher memory
+ consumption during both query planning and execution. When choosing how
+ to partition your table, it's also important to consider what changes may
+ occur in the future. For example, if you choose to have one partition
+ per customer and you currently have a small number of large customers,
+ what will the implications be if in several years you instead find
+ yourself with a large number of small customers. In this case, it may be
+ better to choose to partition by <literal>RANGE</literal> and choose a
+ reasonable number of partitions, each containing a fixed number of
+ customers, rather than trying to partition by <literal>LIST</literal>
+ and hoping that the number of customers does not increase beyond what it
+ is practical to partition the data by.
+ </para>
+
+ <para>
+ Sub-partitioning can be useful to further divide partitions that are
+ expected to become larger than other partitions, although excessive
+ sub-partitioning can easily lead to large numbers of partitions and can
+ cause the same problems mentioned in the preceding paragraph.
+ </para>
+
+ <para>
+ It is also important to consider the overhead of partitioning during
+ query planning and execution. The query planner is generally able to
+ handle partition hierarchies up a few hundred partition. Planning times
+ become longer and memory consumption becomes higher as more partitions are
+ added. This is particularly true for the <command>UPDATE</command> and
+ <command>DELETE</command> commands. Another reason to be concerned about
+ having a large number of partitions is that the server's memory
+ consumption may grow significantly over a period of time, especially if
+ many sessions touch large numbers of partitions. That's because each
+ partition requires its own metadata that must be loaded into the local
+ memory of each session that touches it.
+ </para>
+
+ <para>
+ With data warehouse type workloads, it can make sense to use a larger
+ number of partitions than with an <acronym>OLTP</acronym> type workload.
+ Generally, in data warehouses, query planning time is less of a concern as
+ the majority of processing time is spent during query execution. With
+ either of these two types of workload, it is important to make the right
+ decisions early, as re-partitioning large quantities of data can be
+ painfully slow. Simulations of the intended workload are often beneficial
+ for optimizing the partitioning strategy. Never assume that more
+ partitions are better than fewer partitions and vice-versa.
+ </para>
+ </sect2>
+
</sect1>
<sect1 id="ddl-foreign-data">
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Should we warn against using too many partitions?
@ 2019-06-09 05:11 Justin Pryzby <[email protected]>
parent: David Rowley <[email protected]>
0 siblings, 2 replies; 86+ messages in thread
From: Justin Pryzby @ 2019-06-09 05:11 UTC (permalink / raw)
To: David Rowley <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Amit Langote <[email protected]>; Robert Haas <[email protected]>; PostgreSQL Hackers <[email protected]>
On Sun, Jun 09, 2019 at 05:07:39PM +1200, David Rowley wrote:
> On Sun, 9 Jun 2019 at 16:21, Justin Pryzby <[email protected]> wrote:
> > I meant it should say "into which it should be divided" and not "by which it
> > should be divided INTO", which has too many prepositions. This is still an
> > issue:
>
> It now reads "divided by" instead of "divided into".
Sorry, but I think this is still an issue:
> Choosing the target number of partitions into which the table should be
> divided by is also a critical decision to make. Not having enough
I think it should say:
| Choosing the target number of partitions into which the table should be
| divided is also a critical decision to make. Not having enough
Justin
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Should we warn against using too many partitions?
@ 2019-06-09 05:44 David Rowley <[email protected]>
parent: Justin Pryzby <[email protected]>
1 sibling, 0 replies; 86+ messages in thread
From: David Rowley @ 2019-06-09 05:44 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Amit Langote <[email protected]>; Robert Haas <[email protected]>; PostgreSQL Hackers <[email protected]>
On Sun, 9 Jun 2019 at 17:11, Justin Pryzby <[email protected]> wrote:
> Sorry, but I think this is still an issue:
>
> > Choosing the target number of partitions into which the table should be
> > divided by is also a critical decision to make. Not having enough
>
> I think it should say:
>
> | Choosing the target number of partitions into which the table should be
> | divided is also a critical decision to make. Not having enough
Alright. I guess I misunderstood you. Updated patches are attached.
--
David Rowley http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Attachments:
[application/octet-stream] part_doc_master_v4.patch (5.3K, ../../CAKJS1f97SGMrfvA67yZDEOyQB0Q2iocLYEcS+=i1P-oaKbDWVw@mail.gmail.com/2-part_doc_master_v4.patch)
download | inline diff:
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index cce1618fc1..d1dd0718ef 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -3450,8 +3450,9 @@ VALUES ('Albany', NULL, NULL, 'NY');
</listitem>
</itemizedlist>
- These deficiencies will probably be fixed in some future release,
- but in the meantime considerable care is needed in deciding whether
+ Some functionality not implemented for inheritance hierarchies is
+ implemented for declarative partitioning.
+ Considerable care is needed in deciding whether partitioning with legacy
inheritance is useful for your application.
</para>
@@ -4674,6 +4675,87 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01';
</itemizedlist>
</para>
</sect2>
+
+ <sect2 id="ddl-partitioning-declarative-best-practices">
+ <title>Declarative Partitioning Best Practices</title>
+
+ <para>
+ The choice of how to partition a table should be made carefully as the
+ performance of query planning and execution can be negatively affected by
+ poor design.
+ </para>
+
+ <para>
+ One of the most critical design decisions will be the column or columns
+ by which you partition your data. Often the best choice will be to
+ partition by the column or set of columns which most commonly appear in
+ <literal>WHERE</literal> clauses of queries being executed on the
+ partitioned table. <literal>WHERE</literal> clause items that match and
+ are compatible with the partition key can be used to prune unneeded
+ partitions. However, you may be forced into making other decisions by
+ requirements for the <literal>PRIMARY KEY</literal> or a
+ <literal>UNIQUE</literal> constraint. Removal of unwanted data is also a
+ factor to consider when planning your partitioning strategy. An entire
+ partition can be detached fairly quickly, so it may be beneficial to
+ design the partition strategy in such a way that all data to be removed
+ at once is located in a single partition.
+ </para>
+
+ <para>
+ Choosing the target number of partitions into which the table should be
+ divided is also a critical decision to make. Not having enough
+ partitions may mean that indexes remain too large and that data locality
+ remains poor which could result in low cache hit ratios. However,
+ dividing the table into too many partitions can also cause issues.
+ Too many partitions can mean longer query planning times and higher memory
+ consumption during both query planning and execution. When choosing how
+ to partition your table, it's also important to consider what changes may
+ occur in the future. For example, if you choose to have one partition
+ per customer and you currently have a small number of large customers,
+ what will the implications be if in several years you instead find
+ yourself with a large number of small customers. In this case, it may be
+ better to choose to partition by <literal>HASH</literal> and choose a
+ reasonable number of partitions rather than trying to partition by
+ <literal>LIST</literal> and hoping that the number of customers does
+ not increase beyond what it is practical to partition the data by.
+ </para>
+
+ <para>
+ Sub-partitioning can be useful to further divide partitions that are
+ expected to become larger than other partitions, although excessive
+ sub-partitioning can easily lead to large numbers of partitions and can
+ cause the same problems mentioned in the preceding paragraph.
+ </para>
+
+ <para>
+ It is also important to consider the overhead of partitioning during
+ query planning and execution. The query planner is generally able to
+ handle partition hierarchies up a few thousand partitions fairly well,
+ provided that typical queries allow the query planner to prune all but a
+ small number of partitions. Planning times become longer and memory
+ consumption becomes higher when more partitions remain after the planner
+ performs partition pruning. This is particularly true for the
+ <command>UPDATE</command> and <command>DELETE</command> commands. Another
+ reason to be concerned about having a large number of partitions is that
+ the server's memory consumption may grow significantly over a period of
+ time, especially if many sessions touch large numbers of partitions.
+ That's because each partition requires its own metadata that must be
+ loaded into the local memory of each session that touches it.
+ </para>
+
+ <para>
+ With data warehouse type workloads, it can make sense to use a larger
+ number of partitions than with an <acronym>OLTP</acronym> type workload.
+ Generally, in data warehouses, query planning time is less of a concern as
+ the majority of processing time is spent during query execution. With
+ either of these two types of workload, it is important to make the right
+ decisions early, as re-partitioning large quantities of data can be
+ painfully slow. Simulations of the intended workload are often beneficial
+ for optimizing the partitioning strategy. Never assume that more
+ partitions are better than fewer partitions and vice-versa.
+ </para>
+ </sect2>
+
</sect1>
<sect1 id="ddl-foreign-data">
[application/octet-stream] part_doc_pg11_v4.patch (5.2K, ../../CAKJS1f97SGMrfvA67yZDEOyQB0Q2iocLYEcS+=i1P-oaKbDWVw@mail.gmail.com/3-part_doc_pg11_v4.patch)
download | inline diff:
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index 39382e99c7..233ff86e1a 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -2833,8 +2833,9 @@ VALUES ('Albany', NULL, NULL, 'NY');
</listitem>
</itemizedlist>
- These deficiencies will probably be fixed in some future release,
- but in the meantime considerable care is needed in deciding whether
+ Some functionality not implemented for inheritance hierarchies is
+ implemented for declarative partitioning.
+ Considerable care is needed in deciding whether partitioning with legacy
inheritance is useful for your application.
</para>
@@ -4057,6 +4058,87 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01';
</itemizedlist>
</para>
</sect2>
+
+ <sect2 id="ddl-partitioning-declarative-best-practices">
+ <title>Declarative Partitioning Best Practices</title>
+
+ <para>
+ The choice of how to partition a table should be made carefully as the
+ performance of query planning and execution can be negatively affected by
+ poor design.
+ </para>
+
+ <para>
+ One of the most critical design decisions will be the column or columns
+ by which you partition your data. Often the best choice will be to
+ partition by the column or set of columns which most commonly appear in
+ <literal>WHERE</literal> clauses of queries being executed on the
+ partitioned table. <literal>WHERE</literal> clause items that match and
+ are compatible with the partition key can be used to prune unneeded
+ partitions. However, you may be forced into making other decisions by
+ requirements for the <literal>PRIMARY KEY</literal> or a
+ <literal>UNIQUE</literal> constraint. Removal of unwanted data is also a
+ factor to consider when planning your partitioning strategy. An entire
+ partition can be detached fairly quickly, so it may be beneficial to
+ design the partition strategy in such a way that all data to be removed
+ at once is located in a single partition.
+ </para>
+
+ <para>
+ Choosing the target number of partitions into which the table should be
+ divided is also a critical decision to make. Not having enough
+ partitions may mean that indexes remain too large and that data locality
+ remains poor which could result in low cache hit ratios. However,
+ dividing the table into too many partitions can also cause issues.
+ Too many partitions can mean longer query planning times and higher memory
+ consumption during both query planning and execution. When choosing how
+ to partition your table, it's also important to consider what changes may
+ occur in the future. For example, if you choose to have one partition
+ per customer and you currently have a small number of large customers,
+ what will the implications be if in several years you instead find
+ yourself with a large number of small customers. In this case, it may be
+ better to choose to partition by <literal>HASH</literal> and choose a
+ reasonable number of partitions rather than trying to partition by
+ <literal>LIST</literal> and hoping that the number of customers does
+ not increase beyond what it is practical to partition the data by.
+ </para>
+
+ <para>
+ Sub-partitioning can be useful to further divide partitions that are
+ expected to become larger than other partitions, although excessive
+ sub-partitioning can easily lead to large numbers of partitions and can
+ cause the same problems mentioned in the preceding paragraph.
+ </para>
+
+ <para>
+ It is also important to consider the overhead of partitioning during
+ query planning and execution. The query planner is generally able to
+ handle partition hierarchies up a few hundred partitions fairly well,
+ provided that typical queries allow the query planner to prune all but a
+ small number of partitions. Planning times become longer and memory
+ consumption becomes higher as more partitions are added. This is
+ particularly true for the <command>UPDATE</command> and
+ <command>DELETE</command> commands. Another reason to be concerned about
+ having a large number of partitions is that the server's memory
+ consumption may grow significantly over a period of time, especially if
+ many sessions touch large numbers of partitions. That's because each
+ partition requires its own metadata that must be loaded into the local
+ memory of each session that touches it.
+ </para>
+
+ <para>
+ With data warehouse type workloads, it can make sense to use a larger
+ number of partitions than with an <acronym>OLTP</acronym> type workload.
+ Generally, in data warehouses, query planning time is less of a concern as
+ the majority of processing time is spent during query execution. With
+ either of these two types of workload, it is important to make the right
+ decisions early, as re-partitioning large quantities of data can be
+ painfully slow. Simulations of the intended workload are often beneficial
+ for optimizing the partitioning strategy. Never assume that more
+ partitions are better than fewer partitions and vice-versa.
+ </para>
+ </sect2>
+
</sect1>
<sect1 id="ddl-foreign-data">
[application/octet-stream] part_doc_pg10_v4.patch (5.0K, ../../CAKJS1f97SGMrfvA67yZDEOyQB0Q2iocLYEcS+=i1P-oaKbDWVw@mail.gmail.com/4-part_doc_pg10_v4.patch)
download | inline diff:
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index 13a1001761..10447d8d89 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -2833,8 +2833,9 @@ VALUES ('Albany', NULL, NULL, 'NY');
</listitem>
</itemizedlist>
- These deficiencies will probably be fixed in some future release,
- but in the meantime considerable care is needed in deciding whether
+ Some functionality not implemented for inheritance hierarchies is
+ implemented for declarative partitioning.
+ Considerable care is needed in deciding whether partitioning with legacy
inheritance is useful for your application.
</para>
@@ -3927,6 +3928,84 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01';
</itemizedlist>
</para>
</sect2>
+
+ <sect2 id="ddl-partitioning-declarative-best-practices">
+ <title>Declarative Partitioning Best Practices</title>
+
+ <para>
+ The choice of how to partition a table should be made carefully as the
+ performance of query planning and execution can be negatively affected by
+ poor design.
+ </para>
+
+ <para>
+ One of the most critical design decisions will be the column or columns
+ by which you partition your data. Often the best choice will be to
+ partition by the column or set of columns which most commonly appear in
+ <literal>WHERE</literal> clauses of queries being executed on the
+ partitioned table. <literal>WHERE</literal> clause items that match and
+ are compatible with the partition key can be used to prune unneeded
+ partitions. Removal of unwanted data is also a factor to consider when
+ planning your partitioning strategy. An entire partition can be detached
+ fairly quickly, so it may be beneficial to design the partition strategy
+ in such a way that all data to be removed at once is located in a single
+ partition.
+ </para>
+
+ <para>
+ Choosing the target number of partitions into which the table should be
+ divided is also a critical decision to make. Not having enough
+ partitions may mean that indexes remain too large and that data locality
+ remains poor which could result in low cache hit ratios. However,
+ dividing the table into too many partitions can also cause issues.
+ Too many partitions can mean longer query planning times and higher memory
+ consumption during both query planning and execution. When choosing how
+ to partition your table, it's also important to consider what changes may
+ occur in the future. For example, if you choose to have one partition
+ per customer and you currently have a small number of large customers,
+ what will the implications be if in several years you instead find
+ yourself with a large number of small customers. In this case, it may be
+ better to choose to partition by <literal>RANGE</literal> and choose a
+ reasonable number of partitions, each containing a fixed number of
+ customers, rather than trying to partition by <literal>LIST</literal>
+ and hoping that the number of customers does not increase beyond what it
+ is practical to partition the data by.
+ </para>
+
+ <para>
+ Sub-partitioning can be useful to further divide partitions that are
+ expected to become larger than other partitions, although excessive
+ sub-partitioning can easily lead to large numbers of partitions and can
+ cause the same problems mentioned in the preceding paragraph.
+ </para>
+
+ <para>
+ It is also important to consider the overhead of partitioning during
+ query planning and execution. The query planner is generally able to
+ handle partition hierarchies up a few hundred partition. Planning times
+ become longer and memory consumption becomes higher as more partitions are
+ added. This is particularly true for the <command>UPDATE</command> and
+ <command>DELETE</command> commands. Another reason to be concerned about
+ having a large number of partitions is that the server's memory
+ consumption may grow significantly over a period of time, especially if
+ many sessions touch large numbers of partitions. That's because each
+ partition requires its own metadata that must be loaded into the local
+ memory of each session that touches it.
+ </para>
+
+ <para>
+ With data warehouse type workloads, it can make sense to use a larger
+ number of partitions than with an <acronym>OLTP</acronym> type workload.
+ Generally, in data warehouses, query planning time is less of a concern as
+ the majority of processing time is spent during query execution. With
+ either of these two types of workload, it is important to make the right
+ decisions early, as re-partitioning large quantities of data can be
+ painfully slow. Simulations of the intended workload are often beneficial
+ for optimizing the partitioning strategy. Never assume that more
+ partitions are better than fewer partitions and vice-versa.
+ </para>
+ </sect2>
+
</sect1>
<sect1 id="ddl-foreign-data">
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Should we warn against using too many partitions?
@ 2019-06-10 08:11 Amit Langote <[email protected]>
parent: David Rowley <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Amit Langote @ 2019-06-10 08:11 UTC (permalink / raw)
To: David Rowley <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Amit Langote <[email protected]>; Justin Pryzby <[email protected]>; Robert Haas <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi,
Thanks for the updated patches.
On Sun, Jun 9, 2019 at 5:29 AM David Rowley
<[email protected]> wrote:
> On Fri, 7 Jun 2019 at 19:00, Amit Langote <[email protected]> wrote:
> > Maybe:
> >
> > ... Removal of unwanted data is also a factor to consider when
> > planning your partitioning strategy as an entire partition can be
> > removed fairly quickly, especially if the partition keys are chosen
> > such that all data that can be deleted together are grouped into
> > separate partitions.
>
> It seems like a good idea to change this to have this mention the
> benefits rather than the drawbacks. I've reworded it, but not using
> your exact words as it seems the "especially" means that a partition
> can be removed faster with properly chosen partition keys, which is
> not the case.
>
> I also split this out into its own paragraph since it's talking about
> something quite different from the previous paragraph.
Did you miss to split? In v4 patches, I still see this point
mentioned in the same paragraph that it was in before:
+ <para>
+ One of the most critical design decisions will be the column or columns
+ by which you partition your data. Often the best choice will be to
+ partition by the column or set of columns which most commonly appear in
+ <literal>WHERE</literal> clauses of queries being executed on the
+ partitioned table. <literal>WHERE</literal> clause items that match and
+ are compatible with the partition key can be used to prune unneeded
+ partitions. Removal of unwanted data is also a factor to consider when
+ planning your partitioning strategy. An entire partition can be detached
+ fairly quickly, so it may be beneficial to design the partition strategy
+ in such a way that all data to be removed at once is located in a single
+ partition.
+ </para>
> > 2.
> >
> > + ... For example, if you choose to have one partition
> > + per customer and you currently have a small number of large customers,
> > + what will the implications be if in several years you obtain a large
> > + number of small customers.
> >
> > The sentence could be rewritten a bit. Maybe as:
> >
> > ... For example, choosing a design with one partition per customer,
> > because you currently have a small number of large customers, will not
> > scale well several years down the line when you might have a large
> > number of small customers.
> >
> > Btw, doesn't it suffice here to say "large number of customers"
> > instead of "large number of small customers"?
>
> I'm not really trying to imply to plan for business growth here, I'm
> trying to angle it as "what if your business changes".
Hmm, okay. I thought you were intending this as an example of how a
particular partitioning design may not *scale with time*.
> I've reworded
> this slightly and it now says "what will the implications be if in
> several years you instead find yourself with a large number of small
> customers."
I suggest "consider the implications" in place of "what will the
implications be...". Also a user may choose a particular design (one
partition per customer) *because* of their business situation (small
number of large customers), so I suggest linking the two clauses with
"because". With these two changes, the whole sentence will read more
connected, imho:
For example, if you choose to have one partition per customer because
you currently have a small number of large customers, consider the
implications if in several years you instead find yourself with a
large number of small customers.
Thanks,
Amit
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Should we warn against using too many partitions?
@ 2019-06-10 09:10 David Rowley <[email protected]>
parent: Amit Langote <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: David Rowley @ 2019-06-10 09:10 UTC (permalink / raw)
To: Amit Langote <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Amit Langote <[email protected]>; Justin Pryzby <[email protected]>; Robert Haas <[email protected]>; PostgreSQL Hackers <[email protected]>
On Mon, 10 Jun 2019 at 20:11, Amit Langote <[email protected]> wrote:
>
> On Sun, Jun 9, 2019 at 5:29 AM David Rowley
> > I also split this out into its own paragraph since it's talking about
> > something quite different from the previous paragraph.
>
> Did you miss to split? In v4 patches, I still see this point
> mentioned in the same paragraph that it was in before:
Not quite. I just changed my mind again after reading it through.
Since both paragraphs were talking about the number of partitions I
decided they should be the same paragraph after all.
> > I've reworded
> > this slightly and it now says "what will the implications be if in
> > several years you instead find yourself with a large number of small
> > customers."
>
> I suggest "consider the implications" in place of "what will the
> implications be...". Also a user may choose a particular design (one
> partition per customer) *because* of their business situation (small
> number of large customers), so I suggest linking the two clauses with
> "because". With these two changes, the whole sentence will read more
> connected, imho:
The disconnect there is on purpose. I don't really want to suggest
they chose to partition by customer because they have a small number
of large customers. The choice to partition by customer could well
have come from "customer_id = ..." always being present in WHERE
clauses and they may be fooled into thinking it's a good idea to
partition by that because of that fact. I'm hoping the text there
points out that it might not always be a good choice.
I have slightly reworded it to be a bit closer to your suggestion, but
I maintained the disconnect.
--
David Rowley http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Attachments:
[application/octet-stream] part_doc_master_v5.patch (5.3K, ../../CAKJS1f-j19_NfbJrqn2XvnZAYECKSdPtisUGzZ+0aBdprh7KNA@mail.gmail.com/2-part_doc_master_v5.patch)
download | inline diff:
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index cce1618fc1..b6a811f4dd 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -3450,8 +3450,9 @@ VALUES ('Albany', NULL, NULL, 'NY');
</listitem>
</itemizedlist>
- These deficiencies will probably be fixed in some future release,
- but in the meantime considerable care is needed in deciding whether
+ Some functionality not implemented for inheritance hierarchies is
+ implemented for declarative partitioning.
+ Considerable care is needed in deciding whether partitioning with legacy
inheritance is useful for your application.
</para>
@@ -4674,6 +4675,87 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01';
</itemizedlist>
</para>
</sect2>
+
+ <sect2 id="ddl-partitioning-declarative-best-practices">
+ <title>Declarative Partitioning Best Practices</title>
+
+ <para>
+ The choice of how to partition a table should be made carefully as the
+ performance of query planning and execution can be negatively affected by
+ poor design.
+ </para>
+
+ <para>
+ One of the most critical design decisions will be the column or columns
+ by which you partition your data. Often the best choice will be to
+ partition by the column or set of columns which most commonly appear in
+ <literal>WHERE</literal> clauses of queries being executed on the
+ partitioned table. <literal>WHERE</literal> clause items that match and
+ are compatible with the partition key can be used to prune unneeded
+ partitions. However, you may be forced into making other decisions by
+ requirements for the <literal>PRIMARY KEY</literal> or a
+ <literal>UNIQUE</literal> constraint. Removal of unwanted data is also a
+ factor to consider when planning your partitioning strategy. An entire
+ partition can be detached fairly quickly, so it may be beneficial to
+ design the partition strategy in such a way that all data to be removed
+ at once is located in a single partition.
+ </para>
+
+ <para>
+ Choosing the target number of partitions into which the table should be
+ divided by is also a critical decision to make. Not having enough
+ partitions may mean that indexes remain too large and that data locality
+ remains poor which could result in low cache hit ratios. However,
+ dividing the table into too many partitions can also cause issues.
+ Too many partitions can mean longer query planning times and higher memory
+ consumption during both query planning and execution. When choosing how
+ to partition your table, it's also important to consider what changes may
+ occur in the future. For example, if you choose to have one partition
+ per customer and you currently have a small number of large customers,
+ consider the implications if in several years you instead find yourself
+ with a large number of small customers. In this case, it may be better
+ to choose to partition by <literal>HASH</literal> and choose a reasonable
+ number of partitions rather than trying to partition by
+ <literal>LIST</literal> and hoping that the number of customers does
+ not increase beyond what it is practical to partition the data by.
+ </para>
+
+ <para>
+ Sub-partitioning can be useful to further divide partitions that are
+ expected to become larger than other partitions, although excessive
+ sub-partitioning can easily lead to large numbers of partitions and can
+ cause the same problems mentioned in the preceding paragraph.
+ </para>
+
+ <para>
+ It is also important to consider the overhead of partitioning during
+ query planning and execution. The query planner is generally able to
+ handle partition hierarchies up a few thousand partitions fairly well,
+ provided that typical queries allow the query planner to prune all but a
+ small number of partitions. Planning times become longer and memory
+ consumption becomes higher when more partitions remain after the planner
+ performs partition pruning. This is particularly true for the
+ <command>UPDATE</command> and <command>DELETE</command> commands. Another
+ reason to be concerned about having a large number of partitions is that
+ the server's memory consumption may grow significantly over a period of
+ time, especially if many sessions touch large numbers of partitions.
+ That's because each partition requires its own metadata that must be
+ loaded into the local memory of each session that touches it.
+ </para>
+
+ <para>
+ With data warehouse type workloads, it can make sense to use a larger
+ number of partitions than with an <acronym>OLTP</acronym> type workload.
+ Generally, in data warehouses, query planning time is less of a concern as
+ the majority of processing time is spent during query execution. With
+ either of these two types of workload, it is important to make the right
+ decisions early, as re-partitioning large quantities of data can be
+ painfully slow. Simulations of the intended workload are often beneficial
+ for optimizing the partitioning strategy. Never assume that more
+ partitions are better than fewer partitions and vice-versa.
+ </para>
+ </sect2>
+
</sect1>
<sect1 id="ddl-foreign-data">
[application/octet-stream] part_doc_pg10_v5.patch (5.0K, ../../CAKJS1f-j19_NfbJrqn2XvnZAYECKSdPtisUGzZ+0aBdprh7KNA@mail.gmail.com/3-part_doc_pg10_v5.patch)
download | inline diff:
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index 13a1001761..3e8aa57942 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -2833,8 +2833,9 @@ VALUES ('Albany', NULL, NULL, 'NY');
</listitem>
</itemizedlist>
- These deficiencies will probably be fixed in some future release,
- but in the meantime considerable care is needed in deciding whether
+ Some functionality not implemented for inheritance hierarchies is
+ implemented for declarative partitioning.
+ Considerable care is needed in deciding whether partitioning with legacy
inheritance is useful for your application.
</para>
@@ -3927,6 +3928,84 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01';
</itemizedlist>
</para>
</sect2>
+
+ <sect2 id="ddl-partitioning-declarative-best-practices">
+ <title>Declarative Partitioning Best Practices</title>
+
+ <para>
+ The choice of how to partition a table should be made carefully as the
+ performance of query planning and execution can be negatively affected by
+ poor design.
+ </para>
+
+ <para>
+ One of the most critical design decisions will be the column or columns
+ by which you partition your data. Often the best choice will be to
+ partition by the column or set of columns which most commonly appear in
+ <literal>WHERE</literal> clauses of queries being executed on the
+ partitioned table. <literal>WHERE</literal> clause items that match and
+ are compatible with the partition key can be used to prune unneeded
+ partitions. Removal of unwanted data is also a factor to consider when
+ planning your partitioning strategy. An entire partition can be detached
+ fairly quickly, so it may be beneficial to design the partition strategy
+ in such a way that all data to be removed at once is located in a single
+ partition.
+ </para>
+
+ <para>
+ Choosing the target number of partitions into which the table should be
+ divided is also a critical decision to make. Not having enough
+ partitions may mean that indexes remain too large and that data locality
+ remains poor which could result in low cache hit ratios. However,
+ dividing the table into too many partitions can also cause issues.
+ Too many partitions can mean longer query planning times and higher memory
+ consumption during both query planning and execution. When choosing how
+ to partition your table, it's also important to consider what changes may
+ occur in the future. For example, if you choose to have one partition
+ per customer and you currently have a small number of large customers,
+ consider the implications if in several years you instead find yourself
+ with a large number of small customers. In this case, it may be better
+ to choose to partition by <literal>RANGE</literal> and choose a reasonable
+ number of partitions, each containing a fixed number of customers, rather
+ than trying to partition by <literal>LIST</literal> and hoping that the
+ number of customers does not increase beyond what it is practical to
+ partition the data by.
+ </para>
+
+ <para>
+ Sub-partitioning can be useful to further divide partitions that are
+ expected to become larger than other partitions, although excessive
+ sub-partitioning can easily lead to large numbers of partitions and can
+ cause the same problems mentioned in the preceding paragraph.
+ </para>
+
+ <para>
+ It is also important to consider the overhead of partitioning during
+ query planning and execution. The query planner is generally able to
+ handle partition hierarchies up a few hundred partition. Planning times
+ become longer and memory consumption becomes higher as more partitions are
+ added. This is particularly true for the <command>UPDATE</command> and
+ <command>DELETE</command> commands. Another reason to be concerned about
+ having a large number of partitions is that the server's memory
+ consumption may grow significantly over a period of time, especially if
+ many sessions touch large numbers of partitions. That's because each
+ partition requires its own metadata that must be loaded into the local
+ memory of each session that touches it.
+ </para>
+
+ <para>
+ With data warehouse type workloads, it can make sense to use a larger
+ number of partitions than with an <acronym>OLTP</acronym> type workload.
+ Generally, in data warehouses, query planning time is less of a concern as
+ the majority of processing time is spent during query execution. With
+ either of these two types of workload, it is important to make the right
+ decisions early, as re-partitioning large quantities of data can be
+ painfully slow. Simulations of the intended workload are often beneficial
+ for optimizing the partitioning strategy. Never assume that more
+ partitions are better than fewer partitions and vice-versa.
+ </para>
+ </sect2>
+
</sect1>
<sect1 id="ddl-foreign-data">
[application/octet-stream] part_doc_pg11_v5.patch (5.2K, ../../CAKJS1f-j19_NfbJrqn2XvnZAYECKSdPtisUGzZ+0aBdprh7KNA@mail.gmail.com/4-part_doc_pg11_v5.patch)
download | inline diff:
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index 39382e99c7..aaefbfa857 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -2833,8 +2833,9 @@ VALUES ('Albany', NULL, NULL, 'NY');
</listitem>
</itemizedlist>
- These deficiencies will probably be fixed in some future release,
- but in the meantime considerable care is needed in deciding whether
+ Some functionality not implemented for inheritance hierarchies is
+ implemented for declarative partitioning.
+ Considerable care is needed in deciding whether partitioning with legacy
inheritance is useful for your application.
</para>
@@ -4057,6 +4058,87 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01';
</itemizedlist>
</para>
</sect2>
+
+ <sect2 id="ddl-partitioning-declarative-best-practices">
+ <title>Declarative Partitioning Best Practices</title>
+
+ <para>
+ The choice of how to partition a table should be made carefully as the
+ performance of query planning and execution can be negatively affected by
+ poor design.
+ </para>
+
+ <para>
+ One of the most critical design decisions will be the column or columns
+ by which you partition your data. Often the best choice will be to
+ partition by the column or set of columns which most commonly appear in
+ <literal>WHERE</literal> clauses of queries being executed on the
+ partitioned table. <literal>WHERE</literal> clause items that match and
+ are compatible with the partition key can be used to prune unneeded
+ partitions. However, you may be forced into making other decisions by
+ requirements for the <literal>PRIMARY KEY</literal> or a
+ <literal>UNIQUE</literal> constraint. Removal of unwanted data is also a
+ factor to consider when planning your partitioning strategy. An entire
+ partition can be detached fairly quickly, so it may be beneficial to
+ design the partition strategy in such a way that all data to be removed
+ at once is located in a single partition.
+ </para>
+
+ <para>
+ Choosing the target number of partitions into which the table should be
+ divided is also a critical decision to make. Not having enough
+ partitions may mean that indexes remain too large and that data locality
+ remains poor which could result in low cache hit ratios. However,
+ dividing the table into too many partitions can also cause issues.
+ Too many partitions can mean longer query planning times and higher memory
+ consumption during both query planning and execution. When choosing how
+ to partition your table, it's also important to consider what changes may
+ occur in the future. For example, if you choose to have one partition
+ per customer and you currently have a small number of large customers,
+ consider the implications if in several years you instead find yourself
+ with a large number of small customers. In this case, it may be better
+ to choose to partition by <literal>HASH</literal> and choose a reasonable
+ number of partitions rather than trying to partition by
+ <literal>LIST</literal> and hoping that the number of customers does
+ not increase beyond what it is practical to partition the data by.
+ </para>
+
+ <para>
+ Sub-partitioning can be useful to further divide partitions that are
+ expected to become larger than other partitions, although excessive
+ sub-partitioning can easily lead to large numbers of partitions and can
+ cause the same problems mentioned in the preceding paragraph.
+ </para>
+
+ <para>
+ It is also important to consider the overhead of partitioning during
+ query planning and execution. The query planner is generally able to
+ handle partition hierarchies up a few hundred partitions fairly well,
+ provided that typical queries allow the query planner to prune all but a
+ small number of partitions. Planning times become longer and memory
+ consumption becomes higher as more partitions are added. This is
+ particularly true for the <command>UPDATE</command> and
+ <command>DELETE</command> commands. Another reason to be concerned about
+ having a large number of partitions is that the server's memory
+ consumption may grow significantly over a period of time, especially if
+ many sessions touch large numbers of partitions. That's because each
+ partition requires its own metadata that must be loaded into the local
+ memory of each session that touches it.
+ </para>
+
+ <para>
+ With data warehouse type workloads, it can make sense to use a larger
+ number of partitions than with an <acronym>OLTP</acronym> type workload.
+ Generally, in data warehouses, query planning time is less of a concern as
+ the majority of processing time is spent during query execution. With
+ either of these two types of workload, it is important to make the right
+ decisions early, as re-partitioning large quantities of data can be
+ painfully slow. Simulations of the intended workload are often beneficial
+ for optimizing the partitioning strategy. Never assume that more
+ partitions are better than fewer partitions and vice-versa.
+ </para>
+ </sect2>
+
</sect1>
<sect1 id="ddl-foreign-data">
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Should we warn against using too many partitions?
@ 2019-06-10 13:44 Alvaro Herrera <[email protected]>
parent: David Rowley <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Alvaro Herrera @ 2019-06-10 13:44 UTC (permalink / raw)
To: David Rowley <[email protected]>; +Cc: Amit Langote <[email protected]>; Amit Langote <[email protected]>; Justin Pryzby <[email protected]>; Robert Haas <[email protected]>; PostgreSQL Hackers <[email protected]>
part_doc_pg10_v5.patch :
+ query planning and execution. The query planner is generally able to
+ handle partition hierarchies up a few hundred partition. Planning times
"up TO a few hundred partition*S*" ?
part_doc_master_v5.patch:
+ Choosing the target number of partitions into which the table should be
+ divided by is also a critical decision to make.
"into which ... should be divided by" seems like a copy-editing
mistake. Did you mean to remove either the "into which" or the "by"?
I think "the target number of partitions THAT the table should be
divided into" is simple and sensible; I'm not sure I trust the version
with "into which" instead of "that", and the role of "by" is not clear
to me ("divide by" implies a divisor, but here we're talking about the
resulting chunks and not the divisor).
In this phrase (all versions):
+ That's because each partition requires its own metadata that must be
+ loaded into the local memory of each session that touches it.
I would replace "requires its own metadata that must be loaded" with
"requires its metadata to be loaded".
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Should we warn against using too many partitions?
@ 2019-06-10 21:45 David Rowley <[email protected]>
parent: Alvaro Herrera <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: David Rowley @ 2019-06-10 21:45 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Amit Langote <[email protected]>; Amit Langote <[email protected]>; Justin Pryzby <[email protected]>; Robert Haas <[email protected]>; PostgreSQL Hackers <[email protected]>
Thanks for looking at this.
On Tue, 11 Jun 2019 at 01:44, Alvaro Herrera <[email protected]> wrote:
>
> part_doc_pg10_v5.patch :
> + query planning and execution. The query planner is generally able to
> + handle partition hierarchies up a few hundred partition. Planning times
>
> "up TO a few hundred partition*S*" ?
Oops. My backspace key must have removed too many chars when I removed
"quite well" out of the PG10 version.
> part_doc_master_v5.patch:
> + Choosing the target number of partitions into which the table should be
> + divided by is also a critical decision to make.
>
> "into which ... should be divided by" seems like a copy-editing
> mistake.
Yes it is. It only existed in the master version. I'm not sure how it
snuck by in there.
> Did you mean to remove either the "into which" or the "by"?
I meant to remove "by", per advice from Justin.
> I think "the target number of partitions THAT the table should be
> divided into" is simple and sensible; I'm not sure I trust the version
> with "into which" instead of "that", and the role of "by" is not clear
> to me ("divide by" implies a divisor, but here we're talking about the
> resulting chunks and not the divisor).
This is tricky. Justin liked it that way and since it took me a few
rounds to get it the way he wanted, I'm quite tempted by the
status-quo.
> In this phrase (all versions):
> + That's because each partition requires its own metadata that must be
> + loaded into the local memory of each session that touches it.
>
> I would replace "requires its own metadata that must be loaded" with
> "requires its metadata to be loaded".
That seems like a good improvement. Changed to that.
v6 versions are attached.
--
David Rowley http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Attachments:
[application/octet-stream] part_doc_pg10_v6.patch (5.0K, ../../CAKJS1f9ZzH_VC_7kNHCYK_RoD6KjCPZiy1rPXiEPQnmVC75xVQ@mail.gmail.com/2-part_doc_pg10_v6.patch)
download | inline diff:
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index 13a1001761..3e8aa57942 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -2833,8 +2833,9 @@ VALUES ('Albany', NULL, NULL, 'NY');
</listitem>
</itemizedlist>
- These deficiencies will probably be fixed in some future release,
- but in the meantime considerable care is needed in deciding whether
+ Some functionality not implemented for inheritance hierarchies is
+ implemented for declarative partitioning.
+ Considerable care is needed in deciding whether partitioning with legacy
inheritance is useful for your application.
</para>
@@ -3927,6 +3928,84 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01';
</itemizedlist>
</para>
</sect2>
+
+ <sect2 id="ddl-partitioning-declarative-best-practices">
+ <title>Declarative Partitioning Best Practices</title>
+
+ <para>
+ The choice of how to partition a table should be made carefully as the
+ performance of query planning and execution can be negatively affected by
+ poor design.
+ </para>
+
+ <para>
+ One of the most critical design decisions will be the column or columns
+ by which you partition your data. Often the best choice will be to
+ partition by the column or set of columns which most commonly appear in
+ <literal>WHERE</literal> clauses of queries being executed on the
+ partitioned table. <literal>WHERE</literal> clause items that match and
+ are compatible with the partition key can be used to prune unneeded
+ partitions. Removal of unwanted data is also a factor to consider when
+ planning your partitioning strategy. An entire partition can be detached
+ fairly quickly, so it may be beneficial to design the partition strategy
+ in such a way that all data to be removed at once is located in a single
+ partition.
+ </para>
+
+ <para>
+ Choosing the target number of partitions into which the table should be
+ divided is also a critical decision to make. Not having enough
+ partitions may mean that indexes remain too large and that data locality
+ remains poor which could result in low cache hit ratios. However,
+ dividing the table into too many partitions can also cause issues.
+ Too many partitions can mean longer query planning times and higher memory
+ consumption during both query planning and execution. When choosing how
+ to partition your table, it's also important to consider what changes may
+ occur in the future. For example, if you choose to have one partition
+ per customer and you currently have a small number of large customers,
+ consider the implications if in several years you instead find yourself
+ with a large number of small customers. In this case, it may be better
+ to choose to partition by <literal>RANGE</literal> and choose a reasonable
+ number of partitions, each containing a fixed number of customers, rather
+ than trying to partition by <literal>LIST</literal> and hoping that the
+ number of customers does not increase beyond what it is practical to
+ partition the data by.
+ </para>
+
+ <para>
+ Sub-partitioning can be useful to further divide partitions that are
+ expected to become larger than other partitions, although excessive
+ sub-partitioning can easily lead to large numbers of partitions and can
+ cause the same problems mentioned in the preceding paragraph.
+ </para>
+
+ <para>
+ It is also important to consider the overhead of partitioning during
+ query planning and execution. The query planner is generally able to
+ handle partition hierarchies up a few hundred partitions. Planning times
+ become longer and memory consumption becomes higher as more partitions are
+ added. This is particularly true for the <command>UPDATE</command> and
+ <command>DELETE</command> commands. Another reason to be concerned about
+ having a large number of partitions is that the server's memory
+ consumption may grow significantly over a period of time, especially if
+ many sessions touch large numbers of partitions. That's because each
+ partition requires its metadata to be loaded into the local memory of
+ each session that touches it.
+ </para>
+
+ <para>
+ With data warehouse type workloads, it can make sense to use a larger
+ number of partitions than with an <acronym>OLTP</acronym> type workload.
+ Generally, in data warehouses, query planning time is less of a concern as
+ the majority of processing time is spent during query execution. With
+ either of these two types of workload, it is important to make the right
+ decisions early, as re-partitioning large quantities of data can be
+ painfully slow. Simulations of the intended workload are often beneficial
+ for optimizing the partitioning strategy. Never assume that more
+ partitions are better than fewer partitions and vice-versa.
+ </para>
+ </sect2>
+
</sect1>
<sect1 id="ddl-foreign-data">
[application/octet-stream] part_doc_pg11_v6.patch (5.2K, ../../CAKJS1f9ZzH_VC_7kNHCYK_RoD6KjCPZiy1rPXiEPQnmVC75xVQ@mail.gmail.com/3-part_doc_pg11_v6.patch)
download | inline diff:
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index 39382e99c7..aaefbfa857 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -2833,8 +2833,9 @@ VALUES ('Albany', NULL, NULL, 'NY');
</listitem>
</itemizedlist>
- These deficiencies will probably be fixed in some future release,
- but in the meantime considerable care is needed in deciding whether
+ Some functionality not implemented for inheritance hierarchies is
+ implemented for declarative partitioning.
+ Considerable care is needed in deciding whether partitioning with legacy
inheritance is useful for your application.
</para>
@@ -4057,6 +4058,87 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01';
</itemizedlist>
</para>
</sect2>
+
+ <sect2 id="ddl-partitioning-declarative-best-practices">
+ <title>Declarative Partitioning Best Practices</title>
+
+ <para>
+ The choice of how to partition a table should be made carefully as the
+ performance of query planning and execution can be negatively affected by
+ poor design.
+ </para>
+
+ <para>
+ One of the most critical design decisions will be the column or columns
+ by which you partition your data. Often the best choice will be to
+ partition by the column or set of columns which most commonly appear in
+ <literal>WHERE</literal> clauses of queries being executed on the
+ partitioned table. <literal>WHERE</literal> clause items that match and
+ are compatible with the partition key can be used to prune unneeded
+ partitions. However, you may be forced into making other decisions by
+ requirements for the <literal>PRIMARY KEY</literal> or a
+ <literal>UNIQUE</literal> constraint. Removal of unwanted data is also a
+ factor to consider when planning your partitioning strategy. An entire
+ partition can be detached fairly quickly, so it may be beneficial to
+ design the partition strategy in such a way that all data to be removed
+ at once is located in a single partition.
+ </para>
+
+ <para>
+ Choosing the target number of partitions into which the table should be
+ divided is also a critical decision to make. Not having enough
+ partitions may mean that indexes remain too large and that data locality
+ remains poor which could result in low cache hit ratios. However,
+ dividing the table into too many partitions can also cause issues.
+ Too many partitions can mean longer query planning times and higher memory
+ consumption during both query planning and execution. When choosing how
+ to partition your table, it's also important to consider what changes may
+ occur in the future. For example, if you choose to have one partition
+ per customer and you currently have a small number of large customers,
+ consider the implications if in several years you instead find yourself
+ with a large number of small customers. In this case, it may be better
+ to choose to partition by <literal>HASH</literal> and choose a reasonable
+ number of partitions rather than trying to partition by
+ <literal>LIST</literal> and hoping that the number of customers does
+ not increase beyond what it is practical to partition the data by.
+ </para>
+
+ <para>
+ Sub-partitioning can be useful to further divide partitions that are
+ expected to become larger than other partitions, although excessive
+ sub-partitioning can easily lead to large numbers of partitions and can
+ cause the same problems mentioned in the preceding paragraph.
+ </para>
+
+ <para>
+ It is also important to consider the overhead of partitioning during
+ query planning and execution. The query planner is generally able to
+ handle partition hierarchies up a few hundred partitions fairly well,
+ provided that typical queries allow the query planner to prune all but a
+ small number of partitions. Planning times become longer and memory
+ consumption becomes higher as more partitions are added. This is
+ particularly true for the <command>UPDATE</command> and
+ <command>DELETE</command> commands. Another reason to be concerned about
+ having a large number of partitions is that the server's memory
+ consumption may grow significantly over a period of time, especially if
+ many sessions touch large numbers of partitions. That's because each
+ partition requires its metadata to be loaded into the local memory of
+ each session that touches it.
+ </para>
+
+ <para>
+ With data warehouse type workloads, it can make sense to use a larger
+ number of partitions than with an <acronym>OLTP</acronym> type workload.
+ Generally, in data warehouses, query planning time is less of a concern as
+ the majority of processing time is spent during query execution. With
+ either of these two types of workload, it is important to make the right
+ decisions early, as re-partitioning large quantities of data can be
+ painfully slow. Simulations of the intended workload are often beneficial
+ for optimizing the partitioning strategy. Never assume that more
+ partitions are better than fewer partitions and vice-versa.
+ </para>
+ </sect2>
+
</sect1>
<sect1 id="ddl-foreign-data">
[application/octet-stream] part_doc_master_v6.patch (5.2K, ../../CAKJS1f9ZzH_VC_7kNHCYK_RoD6KjCPZiy1rPXiEPQnmVC75xVQ@mail.gmail.com/4-part_doc_master_v6.patch)
download | inline diff:
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index cce1618fc1..4399609b82 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -3450,8 +3450,9 @@ VALUES ('Albany', NULL, NULL, 'NY');
</listitem>
</itemizedlist>
- These deficiencies will probably be fixed in some future release,
- but in the meantime considerable care is needed in deciding whether
+ Some functionality not implemented for inheritance hierarchies is
+ implemented for declarative partitioning.
+ Considerable care is needed in deciding whether partitioning with legacy
inheritance is useful for your application.
</para>
@@ -4674,6 +4675,87 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01';
</itemizedlist>
</para>
</sect2>
+
+ <sect2 id="ddl-partitioning-declarative-best-practices">
+ <title>Declarative Partitioning Best Practices</title>
+
+ <para>
+ The choice of how to partition a table should be made carefully as the
+ performance of query planning and execution can be negatively affected by
+ poor design.
+ </para>
+
+ <para>
+ One of the most critical design decisions will be the column or columns
+ by which you partition your data. Often the best choice will be to
+ partition by the column or set of columns which most commonly appear in
+ <literal>WHERE</literal> clauses of queries being executed on the
+ partitioned table. <literal>WHERE</literal> clause items that match and
+ are compatible with the partition key can be used to prune unneeded
+ partitions. However, you may be forced into making other decisions by
+ requirements for the <literal>PRIMARY KEY</literal> or a
+ <literal>UNIQUE</literal> constraint. Removal of unwanted data is also a
+ factor to consider when planning your partitioning strategy. An entire
+ partition can be detached fairly quickly, so it may be beneficial to
+ design the partition strategy in such a way that all data to be removed
+ at once is located in a single partition.
+ </para>
+
+ <para>
+ Choosing the target number of partitions into which the table should be
+ divided is also a critical decision to make. Not having enough
+ partitions may mean that indexes remain too large and that data locality
+ remains poor which could result in low cache hit ratios. However,
+ dividing the table into too many partitions can also cause issues.
+ Too many partitions can mean longer query planning times and higher memory
+ consumption during both query planning and execution. When choosing how
+ to partition your table, it's also important to consider what changes may
+ occur in the future. For example, if you choose to have one partition
+ per customer and you currently have a small number of large customers,
+ consider the implications if in several years you instead find yourself
+ with a large number of small customers. In this case, it may be better
+ to choose to partition by <literal>HASH</literal> and choose a reasonable
+ number of partitions rather than trying to partition by
+ <literal>LIST</literal> and hoping that the number of customers does
+ not increase beyond what it is practical to partition the data by.
+ </para>
+
+ <para>
+ Sub-partitioning can be useful to further divide partitions that are
+ expected to become larger than other partitions, although excessive
+ sub-partitioning can easily lead to large numbers of partitions and can
+ cause the same problems mentioned in the preceding paragraph.
+ </para>
+
+ <para>
+ It is also important to consider the overhead of partitioning during
+ query planning and execution. The query planner is generally able to
+ handle partition hierarchies up a few thousand partitions fairly well,
+ provided that typical queries allow the query planner to prune all but a
+ small number of partitions. Planning times become longer and memory
+ consumption becomes higher when more partitions remain after the planner
+ performs partition pruning. This is particularly true for the
+ <command>UPDATE</command> and <command>DELETE</command> commands. Another
+ reason to be concerned about having a large number of partitions is that
+ the server's memory consumption may grow significantly over a period of
+ time, especially if many sessions touch large numbers of partitions.
+ That's because each partition requires its metadata to be loaded into the
+ local memory of each session that touches it.
+ </para>
+
+ <para>
+ With data warehouse type workloads, it can make sense to use a larger
+ number of partitions than with an <acronym>OLTP</acronym> type workload.
+ Generally, in data warehouses, query planning time is less of a concern as
+ the majority of processing time is spent during query execution. With
+ either of these two types of workload, it is important to make the right
+ decisions early, as re-partitioning large quantities of data can be
+ painfully slow. Simulations of the intended workload are often beneficial
+ for optimizing the partitioning strategy. Never assume that more
+ partitions are better than fewer partitions and vice-versa.
+ </para>
+ </sect2>
+
</sect1>
<sect1 id="ddl-foreign-data">
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Should we warn against using too many partitions?
@ 2019-06-10 22:11 Alvaro Herrera <[email protected]>
parent: Justin Pryzby <[email protected]>
1 sibling, 1 reply; 86+ messages in thread
From: Alvaro Herrera @ 2019-06-10 22:11 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: David Rowley <[email protected]>; Amit Langote <[email protected]>; Robert Haas <[email protected]>; PostgreSQL Hackers <[email protected]>
On 2019-Jun-09, Justin Pryzby wrote:
> I think it should say:
>
> | Choosing the target number of partitions into which the table should be
> | divided is also a critical decision to make. Not having enough
I opined elsewhere in the thread that this phrase can be made into more
straightforward English:
Choosing the target number of partitions THAT the table should be
divided INTO is also a critical decision to make. Not having enough
What do you think of that formulation?
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Should we warn against using too many partitions?
@ 2019-06-10 23:15 Justin Pryzby <[email protected]>
parent: Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Justin Pryzby @ 2019-06-10 23:15 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: David Rowley <[email protected]>; Amit Langote <[email protected]>; Robert Haas <[email protected]>; PostgreSQL Hackers <[email protected]>
On Mon, Jun 10, 2019 at 06:11:35PM -0400, Alvaro Herrera wrote:
> On 2019-Jun-09, Justin Pryzby wrote:
>
> > I think it should say:
> >
> > | Choosing the target number of partitions into which the table should be
> > | divided is also a critical decision to make. Not having enough
>
> I opined elsewhere in the thread that this phrase can be made into more
> straightforward English:
>
> Choosing the target number of partitions THAT the table should be
> divided INTO is also a critical decision to make. Not having enough
>
> What do you think of that formulation?
It originally said:
| Choosing the number of partitions to divide the table into is also a
So this mostly changes it back.
One could also say:
| Another critical decision is [the choice of?] the number of partitions
| into which the table['s content?] should be divided...
I'm okay with it if David is okay making the change :)
Thanks,
Justin
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Should we warn against using too many partitions?
@ 2019-06-11 01:30 David Rowley <[email protected]>
parent: Justin Pryzby <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: David Rowley @ 2019-06-11 01:30 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Amit Langote <[email protected]>; Robert Haas <[email protected]>; PostgreSQL Hackers <[email protected]>
On Tue, 11 Jun 2019 at 11:15, Justin Pryzby <[email protected]> wrote:
>
> On Mon, Jun 10, 2019 at 06:11:35PM -0400, Alvaro Herrera wrote:
> > On 2019-Jun-09, Justin Pryzby wrote:
> >
> > > I think it should say:
> > >
> > > | Choosing the target number of partitions into which the table should be
> > > | divided is also a critical decision to make. Not having enough
> >
> > I opined elsewhere in the thread that this phrase can be made into more
> > straightforward English:
> >
> > Choosing the target number of partitions THAT the table should be
> > divided INTO is also a critical decision to make. Not having enough
> >
> > What do you think of that formulation?
>
> It originally said:
> | Choosing the number of partitions to divide the table into is also a
>
> So this mostly changes it back.
>
> One could also say:
> | Another critical decision is [the choice of?] the number of partitions
> | into which the table['s content?] should be divided...
>
> I'm okay with it if David is okay making the change :)
Changes attached.
--
David Rowley http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Attachments:
[application/octet-stream] part_doc_pg11_v7.patch (5.2K, ../../CAKJS1f9sjpSAP0L2QawBTtPYeLDMBpFkCjWvXaSr7Qzn3LAQWA@mail.gmail.com/2-part_doc_pg11_v7.patch)
download | inline diff:
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index 39382e99c7..b628cac2d3 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -2833,8 +2833,9 @@ VALUES ('Albany', NULL, NULL, 'NY');
</listitem>
</itemizedlist>
- These deficiencies will probably be fixed in some future release,
- but in the meantime considerable care is needed in deciding whether
+ Some functionality not implemented for inheritance hierarchies is
+ implemented for declarative partitioning.
+ Considerable care is needed in deciding whether partitioning with legacy
inheritance is useful for your application.
</para>
@@ -4057,6 +4058,87 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01';
</itemizedlist>
</para>
</sect2>
+
+ <sect2 id="ddl-partitioning-declarative-best-practices">
+ <title>Declarative Partitioning Best Practices</title>
+
+ <para>
+ The choice of how to partition a table should be made carefully as the
+ performance of query planning and execution can be negatively affected by
+ poor design.
+ </para>
+
+ <para>
+ One of the most critical design decisions will be the column or columns
+ by which you partition your data. Often the best choice will be to
+ partition by the column or set of columns which most commonly appear in
+ <literal>WHERE</literal> clauses of queries being executed on the
+ partitioned table. <literal>WHERE</literal> clause items that match and
+ are compatible with the partition key can be used to prune unneeded
+ partitions. However, you may be forced into making other decisions by
+ requirements for the <literal>PRIMARY KEY</literal> or a
+ <literal>UNIQUE</literal> constraint. Removal of unwanted data is also a
+ factor to consider when planning your partitioning strategy. An entire
+ partition can be detached fairly quickly, so it may be beneficial to
+ design the partition strategy in such a way that all data to be removed
+ at once is located in a single partition.
+ </para>
+
+ <para>
+ Choosing the target number of partitions that the table should be divided
+ into is also a critical decision to make. Not having enough partitions
+ may mean that indexes remain too large and that data locality remains poor
+ which could result in low cache hit ratios. However, dividing the table
+ into too many partitions can also cause issues. Too many partitions can
+ mean longer query planning times and higher memory consumption during both
+ query planning and execution. When choosing how to partition your table,
+ it's also important to consider what changes may occur in the future. For
+ example, if you choose to have one partition per customer and you
+ currently have a small number of large customers, consider the
+ implications if in several years you instead find yourself with a large
+ number of small customers. In this case, it may be better to choose to
+ partition by <literal>HASH</literal> and choose a reasonable number of
+ partitions rather than trying to partition by <literal>LIST</literal> and
+ hoping that the number of customers does not increase beyond what it is
+ practical to partition the data by.
+ </para>
+
+ <para>
+ Sub-partitioning can be useful to further divide partitions that are
+ expected to become larger than other partitions, although excessive
+ sub-partitioning can easily lead to large numbers of partitions and can
+ cause the same problems mentioned in the preceding paragraph.
+ </para>
+
+ <para>
+ It is also important to consider the overhead of partitioning during
+ query planning and execution. The query planner is generally able to
+ handle partition hierarchies up a few hundred partitions fairly well,
+ provided that typical queries allow the query planner to prune all but a
+ small number of partitions. Planning times become longer and memory
+ consumption becomes higher as more partitions are added. This is
+ particularly true for the <command>UPDATE</command> and
+ <command>DELETE</command> commands. Another reason to be concerned about
+ having a large number of partitions is that the server's memory
+ consumption may grow significantly over a period of time, especially if
+ many sessions touch large numbers of partitions. That's because each
+ partition requires its metadata to be loaded into the local memory of
+ each session that touches it.
+ </para>
+
+ <para>
+ With data warehouse type workloads, it can make sense to use a larger
+ number of partitions than with an <acronym>OLTP</acronym> type workload.
+ Generally, in data warehouses, query planning time is less of a concern as
+ the majority of processing time is spent during query execution. With
+ either of these two types of workload, it is important to make the right
+ decisions early, as re-partitioning large quantities of data can be
+ painfully slow. Simulations of the intended workload are often beneficial
+ for optimizing the partitioning strategy. Never assume that more
+ partitions are better than fewer partitions and vice-versa.
+ </para>
+ </sect2>
+
</sect1>
<sect1 id="ddl-foreign-data">
[application/octet-stream] part_doc_master_v7.patch (5.2K, ../../CAKJS1f9sjpSAP0L2QawBTtPYeLDMBpFkCjWvXaSr7Qzn3LAQWA@mail.gmail.com/3-part_doc_master_v7.patch)
download | inline diff:
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index cce1618fc1..15505f337c 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -3450,8 +3450,9 @@ VALUES ('Albany', NULL, NULL, 'NY');
</listitem>
</itemizedlist>
- These deficiencies will probably be fixed in some future release,
- but in the meantime considerable care is needed in deciding whether
+ Some functionality not implemented for inheritance hierarchies is
+ implemented for declarative partitioning.
+ Considerable care is needed in deciding whether partitioning with legacy
inheritance is useful for your application.
</para>
@@ -4674,6 +4675,87 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01';
</itemizedlist>
</para>
</sect2>
+
+ <sect2 id="ddl-partitioning-declarative-best-practices">
+ <title>Declarative Partitioning Best Practices</title>
+
+ <para>
+ The choice of how to partition a table should be made carefully as the
+ performance of query planning and execution can be negatively affected by
+ poor design.
+ </para>
+
+ <para>
+ One of the most critical design decisions will be the column or columns
+ by which you partition your data. Often the best choice will be to
+ partition by the column or set of columns which most commonly appear in
+ <literal>WHERE</literal> clauses of queries being executed on the
+ partitioned table. <literal>WHERE</literal> clause items that match and
+ are compatible with the partition key can be used to prune unneeded
+ partitions. However, you may be forced into making other decisions by
+ requirements for the <literal>PRIMARY KEY</literal> or a
+ <literal>UNIQUE</literal> constraint. Removal of unwanted data is also a
+ factor to consider when planning your partitioning strategy. An entire
+ partition can be detached fairly quickly, so it may be beneficial to
+ design the partition strategy in such a way that all data to be removed
+ at once is located in a single partition.
+ </para>
+
+ <para>
+ Choosing the target number of partitions that the table should be divided
+ into is also a critical decision to make. Not having enough partitions
+ may mean that indexes remain too large and that data locality remains poor
+ which could result in low cache hit ratios. However, dividing the table
+ into too many partitions can also cause issues. Too many partitions can
+ mean longer query planning times and higher memory consumption during both
+ query planning and execution. When choosing how to partition your table,
+ it's also important to consider what changes may occur in the future. For
+ example, if you choose to have one partition per customer and you
+ currently have a small number of large customers, consider the
+ implications if in several years you instead find yourself with a large
+ number of small customers. In this case, it may be better to choose to
+ partition by <literal>HASH</literal> and choose a reasonable number of
+ partitions rather than trying to partition by <literal>LIST</literal> and
+ hoping that the number of customers does not increase beyond what it is
+ practical to partition the data by.
+ </para>
+
+ <para>
+ Sub-partitioning can be useful to further divide partitions that are
+ expected to become larger than other partitions, although excessive
+ sub-partitioning can easily lead to large numbers of partitions and can
+ cause the same problems mentioned in the preceding paragraph.
+ </para>
+
+ <para>
+ It is also important to consider the overhead of partitioning during
+ query planning and execution. The query planner is generally able to
+ handle partition hierarchies up a few thousand partitions fairly well,
+ provided that typical queries allow the query planner to prune all but a
+ small number of partitions. Planning times become longer and memory
+ consumption becomes higher when more partitions remain after the planner
+ performs partition pruning. This is particularly true for the
+ <command>UPDATE</command> and <command>DELETE</command> commands. Another
+ reason to be concerned about having a large number of partitions is that
+ the server's memory consumption may grow significantly over a period of
+ time, especially if many sessions touch large numbers of partitions.
+ That's because each partition requires its metadata to be loaded into the
+ local memory of each session that touches it.
+ </para>
+
+ <para>
+ With data warehouse type workloads, it can make sense to use a larger
+ number of partitions than with an <acronym>OLTP</acronym> type workload.
+ Generally, in data warehouses, query planning time is less of a concern as
+ the majority of processing time is spent during query execution. With
+ either of these two types of workload, it is important to make the right
+ decisions early, as re-partitioning large quantities of data can be
+ painfully slow. Simulations of the intended workload are often beneficial
+ for optimizing the partitioning strategy. Never assume that more
+ partitions are better than fewer partitions and vice-versa.
+ </para>
+ </sect2>
+
</sect1>
<sect1 id="ddl-foreign-data">
[application/octet-stream] part_doc_pg10_v7.patch (5.0K, ../../CAKJS1f9sjpSAP0L2QawBTtPYeLDMBpFkCjWvXaSr7Qzn3LAQWA@mail.gmail.com/4-part_doc_pg10_v7.patch)
download | inline diff:
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index 13a1001761..dfcf3c4ab0 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -2833,8 +2833,9 @@ VALUES ('Albany', NULL, NULL, 'NY');
</listitem>
</itemizedlist>
- These deficiencies will probably be fixed in some future release,
- but in the meantime considerable care is needed in deciding whether
+ Some functionality not implemented for inheritance hierarchies is
+ implemented for declarative partitioning.
+ Considerable care is needed in deciding whether partitioning with legacy
inheritance is useful for your application.
</para>
@@ -3927,6 +3928,84 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01';
</itemizedlist>
</para>
</sect2>
+
+ <sect2 id="ddl-partitioning-declarative-best-practices">
+ <title>Declarative Partitioning Best Practices</title>
+
+ <para>
+ The choice of how to partition a table should be made carefully as the
+ performance of query planning and execution can be negatively affected by
+ poor design.
+ </para>
+
+ <para>
+ One of the most critical design decisions will be the column or columns
+ by which you partition your data. Often the best choice will be to
+ partition by the column or set of columns which most commonly appear in
+ <literal>WHERE</literal> clauses of queries being executed on the
+ partitioned table. <literal>WHERE</literal> clause items that match and
+ are compatible with the partition key can be used to prune unneeded
+ partitions. Removal of unwanted data is also a factor to consider when
+ planning your partitioning strategy. An entire partition can be detached
+ fairly quickly, so it may be beneficial to design the partition strategy
+ in such a way that all data to be removed at once is located in a single
+ partition.
+ </para>
+
+ <para>
+ Choosing the target number of partitions that the table should be divided
+ into is also a critical decision to make. Not having enough partitions
+ may mean that indexes remain too large and that data locality remains poor
+ which could result in low cache hit ratios. However, dividing the table
+ into too many partitions can also cause issues. Too many partitions can
+ mean longer query planning times and higher memory consumption during both
+ query planning and execution. When choosing how to partition your table,
+ it's also important to consider what changes may occur in the future. For
+ example, if you choose to have one partition per customer and you
+ currently have a small number of large customers, consider the
+ implications if in several years you instead find yourself with a large
+ number of small customers. In this case, it may be better to choose to
+ partition by <literal>RANGE</literal> and choose a reasonable number of
+ partitions, each containing a fixed number of customers, rather than
+ trying to partition by <literal>LIST</literal> and hoping that the number
+ of customers does not increase beyond what it is practical to partition
+ the data by.
+ </para>
+
+ <para>
+ Sub-partitioning can be useful to further divide partitions that are
+ expected to become larger than other partitions, although excessive
+ sub-partitioning can easily lead to large numbers of partitions and can
+ cause the same problems mentioned in the preceding paragraph.
+ </para>
+
+ <para>
+ It is also important to consider the overhead of partitioning during
+ query planning and execution. The query planner is generally able to
+ handle partition hierarchies up a few hundred partitions. Planning times
+ become longer and memory consumption becomes higher as more partitions are
+ added. This is particularly true for the <command>UPDATE</command> and
+ <command>DELETE</command> commands. Another reason to be concerned about
+ having a large number of partitions is that the server's memory
+ consumption may grow significantly over a period of time, especially if
+ many sessions touch large numbers of partitions. That's because each
+ partition requires its metadata to be loaded into the local memory of
+ each session that touches it.
+ </para>
+
+ <para>
+ With data warehouse type workloads, it can make sense to use a larger
+ number of partitions than with an <acronym>OLTP</acronym> type workload.
+ Generally, in data warehouses, query planning time is less of a concern as
+ the majority of processing time is spent during query execution. With
+ either of these two types of workload, it is important to make the right
+ decisions early, as re-partitioning large quantities of data can be
+ painfully slow. Simulations of the intended workload are often beneficial
+ for optimizing the partitioning strategy. Never assume that more
+ partitions are better than fewer partitions and vice-versa.
+ </para>
+ </sect2>
+
</sect1>
<sect1 id="ddl-foreign-data">
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Should we warn against using too many partitions?
@ 2019-06-11 02:43 Alvaro Herrera <[email protected]>
parent: David Rowley <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Alvaro Herrera @ 2019-06-11 02:43 UTC (permalink / raw)
To: David Rowley <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Amit Langote <[email protected]>; Robert Haas <[email protected]>; PostgreSQL Hackers <[email protected]>
On 2019-Jun-11, David Rowley wrote:
> Changes attached.
Unreserved +1 to these patches.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Should we warn against using too many partitions?
@ 2019-06-11 02:52 Amit Langote <[email protected]>
parent: Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Amit Langote @ 2019-06-11 02:52 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: David Rowley <[email protected]>; Justin Pryzby <[email protected]>; Amit Langote <[email protected]>; Robert Haas <[email protected]>; PostgreSQL Hackers <[email protected]>
On Tue, Jun 11, 2019 at 11:43 AM Alvaro Herrera
<[email protected]> wrote:
>
> On 2019-Jun-11, David Rowley wrote:
>
> > Changes attached.
>
> Unreserved +1 to these patches.
The latest version looks good to me too.
Thanks,
Amit
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Should we warn against using too many partitions?
@ 2019-06-11 20:12 David Rowley <[email protected]>
parent: Amit Langote <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: David Rowley @ 2019-06-11 20:12 UTC (permalink / raw)
To: Amit Langote <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Justin Pryzby <[email protected]>; Amit Langote <[email protected]>; Robert Haas <[email protected]>; PostgreSQL Hackers <[email protected]>
On Tue, 11 Jun 2019 at 14:53, Amit Langote <[email protected]> wrote:
> The latest version looks good to me too.
Pushed. Thank you all for the reviews.
--
David Rowley http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Should we warn against using too many partitions?
@ 2019-06-12 05:48 Amit Langote <[email protected]>
parent: David Rowley <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Amit Langote @ 2019-06-12 05:48 UTC (permalink / raw)
To: David Rowley <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Justin Pryzby <[email protected]>; Amit Langote <[email protected]>; Robert Haas <[email protected]>; PostgreSQL Hackers <[email protected]>
On Wed, Jun 12, 2019 at 5:12 AM David Rowley
<[email protected]> wrote:
>
> On Tue, 11 Jun 2019 at 14:53, Amit Langote <[email protected]> wrote:
> > The latest version looks good to me too.
>
> Pushed. Thank you all for the reviews.
Thanks.
I noticed a typo:
"...able to handle partition hierarchies up a few thousand partitions"
s/up/up to/g
I'm inclined to add one more word though, as:
"...able to handle partition hierarchies with up to a few thousand partitions"
or
"...able to handle partition hierarchies containing up to a few
thousand partitions"
Thanks,
Amit
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Should we warn against using too many partitions?
@ 2019-06-12 22:36 David Rowley <[email protected]>
parent: Amit Langote <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: David Rowley @ 2019-06-12 22:36 UTC (permalink / raw)
To: Amit Langote <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Justin Pryzby <[email protected]>; Amit Langote <[email protected]>; Robert Haas <[email protected]>; PostgreSQL Hackers <[email protected]>
On Wed, 12 Jun 2019 at 17:49, Amit Langote <[email protected]> wrote:
> I noticed a typo:
>
> "...able to handle partition hierarchies up a few thousand partitions"
>
> s/up/up to/g
>
> I'm inclined to add one more word though, as:
>
> "...able to handle partition hierarchies with up to a few thousand partitions"
>
> or
>
> "...able to handle partition hierarchies containing up to a few
> thousand partitions"
Thanks for noticing that. I've pushed a fix.
--
David Rowley http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH 11/18] duplicate words
@ 2021-01-31 00:10 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Justin Pryzby @ 2021-01-31 00:10 UTC (permalink / raw)
commit 9c4f5192f69ed16c99e0d079f0b5faebd7bad212
Allow pg_rewind to use a standby server as the source system.
commit 4a252996d5fda7662b2afdf329a5c95be0fe3b01
Add tests for tuplesort.c.
commit 0a2bc5d61e713e3fe72438f020eea5fcc90b0f0b
Move per-agg and per-trans duplicate finding to the planner.
commit 623a9ba79bbdd11c5eccb30b8bd5c446130e521c
snapshot scalability: cache snapshots using a xact completion counter.
commit 2c03216d831160bedd72d45f712601b6f7d03f1c
Revamp the WAL record format.
---
src/backend/access/transam/xlogutils.c | 3 +--
src/backend/optimizer/prep/prepagg.c | 2 +-
src/backend/storage/ipc/procarray.c | 2 +-
src/bin/pg_rewind/libpq_source.c | 2 +-
src/test/regress/expected/tuplesort.out | 2 +-
src/test/regress/sql/tuplesort.sql | 2 +-
6 files changed, 6 insertions(+), 7 deletions(-)
diff --git a/src/backend/access/transam/xlogutils.c b/src/backend/access/transam/xlogutils.c
index e723253297..25d6df1659 100644
--- a/src/backend/access/transam/xlogutils.c
+++ b/src/backend/access/transam/xlogutils.c
@@ -433,8 +433,7 @@ XLogReadBufferForRedoExtended(XLogReaderState *record,
* NB: A redo function should normally not call this directly. To get a page
* to modify, use XLogReadBufferForRedoExtended instead. It is important that
* all pages modified by a WAL record are registered in the WAL records, or
- * they will be invisible to tools that that need to know which pages are
- * modified.
+ * they will be invisible to tools that need to know which pages are modified.
*/
Buffer
XLogReadBufferExtended(RelFileNode rnode, ForkNumber forknum,
diff --git a/src/backend/optimizer/prep/prepagg.c b/src/backend/optimizer/prep/prepagg.c
index 929a8ea13b..89046f9afb 100644
--- a/src/backend/optimizer/prep/prepagg.c
+++ b/src/backend/optimizer/prep/prepagg.c
@@ -71,7 +71,7 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
*
* Information about the aggregates and transition functions are collected
* in the root->agginfos and root->aggtransinfos lists. The 'aggtranstype',
- * 'aggno', and 'aggtransno' fields in are filled in in each Aggref.
+ * 'aggno', and 'aggtransno' fields of each Aggref are filled in.
*
* NOTE: This modifies the Aggrefs in the input expression in-place!
*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index cf12eda504..b9fbdcb88f 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -2049,7 +2049,7 @@ GetSnapshotDataReuse(Snapshot snapshot)
* holding ProcArrayLock) exclusively). Thus the xactCompletionCount check
* ensures we would detect if the snapshot would have changed.
*
- * As the snapshot contents are the same as it was before, it is is safe
+ * As the snapshot contents are the same as it was before, it is safe
* to re-enter the snapshot's xmin into the PGPROC array. None of the rows
* visible under the snapshot could already have been removed (that'd
* require the set of running transactions to change) and it fulfills the
diff --git a/src/bin/pg_rewind/libpq_source.c b/src/bin/pg_rewind/libpq_source.c
index 86d2adcaee..ac794cf4eb 100644
--- a/src/bin/pg_rewind/libpq_source.c
+++ b/src/bin/pg_rewind/libpq_source.c
@@ -539,7 +539,7 @@ process_queued_fetch_requests(libpq_source *src)
chunkoff, rq->path, (int64) rq->offset);
/*
- * We should not receive receive more data than we requested, or
+ * We should not receive more data than we requested, or
* pg_read_binary_file() messed up. We could receive less,
* though, if the file was truncated in the source after we
* checked its size. That's OK, there should be a WAL record of
diff --git a/src/test/regress/expected/tuplesort.out b/src/test/regress/expected/tuplesort.out
index 3fc1998bf2..418f296a3f 100644
--- a/src/test/regress/expected/tuplesort.out
+++ b/src/test/regress/expected/tuplesort.out
@@ -1,7 +1,7 @@
-- only use parallelism when explicitly intending to do so
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
diff --git a/src/test/regress/sql/tuplesort.sql b/src/test/regress/sql/tuplesort.sql
index 7d7e02f02a..846484d561 100644
--- a/src/test/regress/sql/tuplesort.sql
+++ b/src/test/regress/sql/tuplesort.sql
@@ -2,7 +2,7 @@
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
--
2.17.0
--lc9FT7cWel8HagAv
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0012-doc-review-piecemeal-construction-of-partitioned-ind.patch"
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH 11/18] duplicate words
@ 2021-01-31 00:10 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Justin Pryzby @ 2021-01-31 00:10 UTC (permalink / raw)
commit 9c4f5192f69ed16c99e0d079f0b5faebd7bad212
Allow pg_rewind to use a standby server as the source system.
commit 4a252996d5fda7662b2afdf329a5c95be0fe3b01
Add tests for tuplesort.c.
commit 0a2bc5d61e713e3fe72438f020eea5fcc90b0f0b
Move per-agg and per-trans duplicate finding to the planner.
commit 623a9ba79bbdd11c5eccb30b8bd5c446130e521c
snapshot scalability: cache snapshots using a xact completion counter.
commit 2c03216d831160bedd72d45f712601b6f7d03f1c
Revamp the WAL record format.
---
src/backend/access/transam/xlogutils.c | 3 +--
src/backend/optimizer/prep/prepagg.c | 2 +-
src/backend/storage/ipc/procarray.c | 2 +-
src/bin/pg_rewind/libpq_source.c | 2 +-
src/test/regress/expected/tuplesort.out | 2 +-
src/test/regress/sql/tuplesort.sql | 2 +-
6 files changed, 6 insertions(+), 7 deletions(-)
diff --git a/src/backend/access/transam/xlogutils.c b/src/backend/access/transam/xlogutils.c
index e723253297..25d6df1659 100644
--- a/src/backend/access/transam/xlogutils.c
+++ b/src/backend/access/transam/xlogutils.c
@@ -433,8 +433,7 @@ XLogReadBufferForRedoExtended(XLogReaderState *record,
* NB: A redo function should normally not call this directly. To get a page
* to modify, use XLogReadBufferForRedoExtended instead. It is important that
* all pages modified by a WAL record are registered in the WAL records, or
- * they will be invisible to tools that that need to know which pages are
- * modified.
+ * they will be invisible to tools that need to know which pages are modified.
*/
Buffer
XLogReadBufferExtended(RelFileNode rnode, ForkNumber forknum,
diff --git a/src/backend/optimizer/prep/prepagg.c b/src/backend/optimizer/prep/prepagg.c
index 929a8ea13b..89046f9afb 100644
--- a/src/backend/optimizer/prep/prepagg.c
+++ b/src/backend/optimizer/prep/prepagg.c
@@ -71,7 +71,7 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
*
* Information about the aggregates and transition functions are collected
* in the root->agginfos and root->aggtransinfos lists. The 'aggtranstype',
- * 'aggno', and 'aggtransno' fields in are filled in in each Aggref.
+ * 'aggno', and 'aggtransno' fields of each Aggref are filled in.
*
* NOTE: This modifies the Aggrefs in the input expression in-place!
*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index cf12eda504..b9fbdcb88f 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -2049,7 +2049,7 @@ GetSnapshotDataReuse(Snapshot snapshot)
* holding ProcArrayLock) exclusively). Thus the xactCompletionCount check
* ensures we would detect if the snapshot would have changed.
*
- * As the snapshot contents are the same as it was before, it is is safe
+ * As the snapshot contents are the same as it was before, it is safe
* to re-enter the snapshot's xmin into the PGPROC array. None of the rows
* visible under the snapshot could already have been removed (that'd
* require the set of running transactions to change) and it fulfills the
diff --git a/src/bin/pg_rewind/libpq_source.c b/src/bin/pg_rewind/libpq_source.c
index 86d2adcaee..ac794cf4eb 100644
--- a/src/bin/pg_rewind/libpq_source.c
+++ b/src/bin/pg_rewind/libpq_source.c
@@ -539,7 +539,7 @@ process_queued_fetch_requests(libpq_source *src)
chunkoff, rq->path, (int64) rq->offset);
/*
- * We should not receive receive more data than we requested, or
+ * We should not receive more data than we requested, or
* pg_read_binary_file() messed up. We could receive less,
* though, if the file was truncated in the source after we
* checked its size. That's OK, there should be a WAL record of
diff --git a/src/test/regress/expected/tuplesort.out b/src/test/regress/expected/tuplesort.out
index 3fc1998bf2..418f296a3f 100644
--- a/src/test/regress/expected/tuplesort.out
+++ b/src/test/regress/expected/tuplesort.out
@@ -1,7 +1,7 @@
-- only use parallelism when explicitly intending to do so
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
diff --git a/src/test/regress/sql/tuplesort.sql b/src/test/regress/sql/tuplesort.sql
index 7d7e02f02a..846484d561 100644
--- a/src/test/regress/sql/tuplesort.sql
+++ b/src/test/regress/sql/tuplesort.sql
@@ -2,7 +2,7 @@
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
--
2.17.0
--lc9FT7cWel8HagAv
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0012-doc-review-piecemeal-construction-of-partitioned-ind.patch"
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH 11/18] duplicate words
@ 2021-01-31 00:10 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Justin Pryzby @ 2021-01-31 00:10 UTC (permalink / raw)
commit 9c4f5192f69ed16c99e0d079f0b5faebd7bad212
Allow pg_rewind to use a standby server as the source system.
commit 4a252996d5fda7662b2afdf329a5c95be0fe3b01
Add tests for tuplesort.c.
commit 0a2bc5d61e713e3fe72438f020eea5fcc90b0f0b
Move per-agg and per-trans duplicate finding to the planner.
commit 623a9ba79bbdd11c5eccb30b8bd5c446130e521c
snapshot scalability: cache snapshots using a xact completion counter.
commit 2c03216d831160bedd72d45f712601b6f7d03f1c
Revamp the WAL record format.
---
src/backend/access/transam/xlogutils.c | 3 +--
src/backend/optimizer/prep/prepagg.c | 2 +-
src/backend/storage/ipc/procarray.c | 2 +-
src/bin/pg_rewind/libpq_source.c | 2 +-
src/test/regress/expected/tuplesort.out | 2 +-
src/test/regress/sql/tuplesort.sql | 2 +-
6 files changed, 6 insertions(+), 7 deletions(-)
diff --git a/src/backend/access/transam/xlogutils.c b/src/backend/access/transam/xlogutils.c
index e723253297..25d6df1659 100644
--- a/src/backend/access/transam/xlogutils.c
+++ b/src/backend/access/transam/xlogutils.c
@@ -433,8 +433,7 @@ XLogReadBufferForRedoExtended(XLogReaderState *record,
* NB: A redo function should normally not call this directly. To get a page
* to modify, use XLogReadBufferForRedoExtended instead. It is important that
* all pages modified by a WAL record are registered in the WAL records, or
- * they will be invisible to tools that that need to know which pages are
- * modified.
+ * they will be invisible to tools that need to know which pages are modified.
*/
Buffer
XLogReadBufferExtended(RelFileNode rnode, ForkNumber forknum,
diff --git a/src/backend/optimizer/prep/prepagg.c b/src/backend/optimizer/prep/prepagg.c
index 929a8ea13b..89046f9afb 100644
--- a/src/backend/optimizer/prep/prepagg.c
+++ b/src/backend/optimizer/prep/prepagg.c
@@ -71,7 +71,7 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
*
* Information about the aggregates and transition functions are collected
* in the root->agginfos and root->aggtransinfos lists. The 'aggtranstype',
- * 'aggno', and 'aggtransno' fields in are filled in in each Aggref.
+ * 'aggno', and 'aggtransno' fields of each Aggref are filled in.
*
* NOTE: This modifies the Aggrefs in the input expression in-place!
*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index cf12eda504..b9fbdcb88f 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -2049,7 +2049,7 @@ GetSnapshotDataReuse(Snapshot snapshot)
* holding ProcArrayLock) exclusively). Thus the xactCompletionCount check
* ensures we would detect if the snapshot would have changed.
*
- * As the snapshot contents are the same as it was before, it is is safe
+ * As the snapshot contents are the same as it was before, it is safe
* to re-enter the snapshot's xmin into the PGPROC array. None of the rows
* visible under the snapshot could already have been removed (that'd
* require the set of running transactions to change) and it fulfills the
diff --git a/src/bin/pg_rewind/libpq_source.c b/src/bin/pg_rewind/libpq_source.c
index 86d2adcaee..ac794cf4eb 100644
--- a/src/bin/pg_rewind/libpq_source.c
+++ b/src/bin/pg_rewind/libpq_source.c
@@ -539,7 +539,7 @@ process_queued_fetch_requests(libpq_source *src)
chunkoff, rq->path, (int64) rq->offset);
/*
- * We should not receive receive more data than we requested, or
+ * We should not receive more data than we requested, or
* pg_read_binary_file() messed up. We could receive less,
* though, if the file was truncated in the source after we
* checked its size. That's OK, there should be a WAL record of
diff --git a/src/test/regress/expected/tuplesort.out b/src/test/regress/expected/tuplesort.out
index 3fc1998bf2..418f296a3f 100644
--- a/src/test/regress/expected/tuplesort.out
+++ b/src/test/regress/expected/tuplesort.out
@@ -1,7 +1,7 @@
-- only use parallelism when explicitly intending to do so
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
diff --git a/src/test/regress/sql/tuplesort.sql b/src/test/regress/sql/tuplesort.sql
index 7d7e02f02a..846484d561 100644
--- a/src/test/regress/sql/tuplesort.sql
+++ b/src/test/regress/sql/tuplesort.sql
@@ -2,7 +2,7 @@
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
--
2.17.0
--lc9FT7cWel8HagAv
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0012-doc-review-piecemeal-construction-of-partitioned-ind.patch"
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH 11/18] duplicate words
@ 2021-01-31 00:10 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Justin Pryzby @ 2021-01-31 00:10 UTC (permalink / raw)
commit 9c4f5192f69ed16c99e0d079f0b5faebd7bad212
Allow pg_rewind to use a standby server as the source system.
commit 4a252996d5fda7662b2afdf329a5c95be0fe3b01
Add tests for tuplesort.c.
commit 0a2bc5d61e713e3fe72438f020eea5fcc90b0f0b
Move per-agg and per-trans duplicate finding to the planner.
commit 623a9ba79bbdd11c5eccb30b8bd5c446130e521c
snapshot scalability: cache snapshots using a xact completion counter.
commit 2c03216d831160bedd72d45f712601b6f7d03f1c
Revamp the WAL record format.
---
src/backend/access/transam/xlogutils.c | 3 +--
src/backend/optimizer/prep/prepagg.c | 2 +-
src/backend/storage/ipc/procarray.c | 2 +-
src/bin/pg_rewind/libpq_source.c | 2 +-
src/test/regress/expected/tuplesort.out | 2 +-
src/test/regress/sql/tuplesort.sql | 2 +-
6 files changed, 6 insertions(+), 7 deletions(-)
diff --git a/src/backend/access/transam/xlogutils.c b/src/backend/access/transam/xlogutils.c
index e723253297..25d6df1659 100644
--- a/src/backend/access/transam/xlogutils.c
+++ b/src/backend/access/transam/xlogutils.c
@@ -433,8 +433,7 @@ XLogReadBufferForRedoExtended(XLogReaderState *record,
* NB: A redo function should normally not call this directly. To get a page
* to modify, use XLogReadBufferForRedoExtended instead. It is important that
* all pages modified by a WAL record are registered in the WAL records, or
- * they will be invisible to tools that that need to know which pages are
- * modified.
+ * they will be invisible to tools that need to know which pages are modified.
*/
Buffer
XLogReadBufferExtended(RelFileNode rnode, ForkNumber forknum,
diff --git a/src/backend/optimizer/prep/prepagg.c b/src/backend/optimizer/prep/prepagg.c
index 929a8ea13b..89046f9afb 100644
--- a/src/backend/optimizer/prep/prepagg.c
+++ b/src/backend/optimizer/prep/prepagg.c
@@ -71,7 +71,7 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
*
* Information about the aggregates and transition functions are collected
* in the root->agginfos and root->aggtransinfos lists. The 'aggtranstype',
- * 'aggno', and 'aggtransno' fields in are filled in in each Aggref.
+ * 'aggno', and 'aggtransno' fields of each Aggref are filled in.
*
* NOTE: This modifies the Aggrefs in the input expression in-place!
*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index cf12eda504..b9fbdcb88f 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -2049,7 +2049,7 @@ GetSnapshotDataReuse(Snapshot snapshot)
* holding ProcArrayLock) exclusively). Thus the xactCompletionCount check
* ensures we would detect if the snapshot would have changed.
*
- * As the snapshot contents are the same as it was before, it is is safe
+ * As the snapshot contents are the same as it was before, it is safe
* to re-enter the snapshot's xmin into the PGPROC array. None of the rows
* visible under the snapshot could already have been removed (that'd
* require the set of running transactions to change) and it fulfills the
diff --git a/src/bin/pg_rewind/libpq_source.c b/src/bin/pg_rewind/libpq_source.c
index 86d2adcaee..ac794cf4eb 100644
--- a/src/bin/pg_rewind/libpq_source.c
+++ b/src/bin/pg_rewind/libpq_source.c
@@ -539,7 +539,7 @@ process_queued_fetch_requests(libpq_source *src)
chunkoff, rq->path, (int64) rq->offset);
/*
- * We should not receive receive more data than we requested, or
+ * We should not receive more data than we requested, or
* pg_read_binary_file() messed up. We could receive less,
* though, if the file was truncated in the source after we
* checked its size. That's OK, there should be a WAL record of
diff --git a/src/test/regress/expected/tuplesort.out b/src/test/regress/expected/tuplesort.out
index 3fc1998bf2..418f296a3f 100644
--- a/src/test/regress/expected/tuplesort.out
+++ b/src/test/regress/expected/tuplesort.out
@@ -1,7 +1,7 @@
-- only use parallelism when explicitly intending to do so
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
diff --git a/src/test/regress/sql/tuplesort.sql b/src/test/regress/sql/tuplesort.sql
index 7d7e02f02a..846484d561 100644
--- a/src/test/regress/sql/tuplesort.sql
+++ b/src/test/regress/sql/tuplesort.sql
@@ -2,7 +2,7 @@
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
--
2.17.0
--lc9FT7cWel8HagAv
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0012-doc-review-piecemeal-construction-of-partitioned-ind.patch"
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH 11/18] duplicate words
@ 2021-01-31 00:10 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Justin Pryzby @ 2021-01-31 00:10 UTC (permalink / raw)
commit 9c4f5192f69ed16c99e0d079f0b5faebd7bad212
Allow pg_rewind to use a standby server as the source system.
commit 4a252996d5fda7662b2afdf329a5c95be0fe3b01
Add tests for tuplesort.c.
commit 0a2bc5d61e713e3fe72438f020eea5fcc90b0f0b
Move per-agg and per-trans duplicate finding to the planner.
commit 623a9ba79bbdd11c5eccb30b8bd5c446130e521c
snapshot scalability: cache snapshots using a xact completion counter.
commit 2c03216d831160bedd72d45f712601b6f7d03f1c
Revamp the WAL record format.
---
src/backend/access/transam/xlogutils.c | 3 +--
src/backend/optimizer/prep/prepagg.c | 2 +-
src/backend/storage/ipc/procarray.c | 2 +-
src/bin/pg_rewind/libpq_source.c | 2 +-
src/test/regress/expected/tuplesort.out | 2 +-
src/test/regress/sql/tuplesort.sql | 2 +-
6 files changed, 6 insertions(+), 7 deletions(-)
diff --git a/src/backend/access/transam/xlogutils.c b/src/backend/access/transam/xlogutils.c
index e723253297..25d6df1659 100644
--- a/src/backend/access/transam/xlogutils.c
+++ b/src/backend/access/transam/xlogutils.c
@@ -433,8 +433,7 @@ XLogReadBufferForRedoExtended(XLogReaderState *record,
* NB: A redo function should normally not call this directly. To get a page
* to modify, use XLogReadBufferForRedoExtended instead. It is important that
* all pages modified by a WAL record are registered in the WAL records, or
- * they will be invisible to tools that that need to know which pages are
- * modified.
+ * they will be invisible to tools that need to know which pages are modified.
*/
Buffer
XLogReadBufferExtended(RelFileNode rnode, ForkNumber forknum,
diff --git a/src/backend/optimizer/prep/prepagg.c b/src/backend/optimizer/prep/prepagg.c
index 929a8ea13b..89046f9afb 100644
--- a/src/backend/optimizer/prep/prepagg.c
+++ b/src/backend/optimizer/prep/prepagg.c
@@ -71,7 +71,7 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
*
* Information about the aggregates and transition functions are collected
* in the root->agginfos and root->aggtransinfos lists. The 'aggtranstype',
- * 'aggno', and 'aggtransno' fields in are filled in in each Aggref.
+ * 'aggno', and 'aggtransno' fields of each Aggref are filled in.
*
* NOTE: This modifies the Aggrefs in the input expression in-place!
*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index cf12eda504..b9fbdcb88f 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -2049,7 +2049,7 @@ GetSnapshotDataReuse(Snapshot snapshot)
* holding ProcArrayLock) exclusively). Thus the xactCompletionCount check
* ensures we would detect if the snapshot would have changed.
*
- * As the snapshot contents are the same as it was before, it is is safe
+ * As the snapshot contents are the same as it was before, it is safe
* to re-enter the snapshot's xmin into the PGPROC array. None of the rows
* visible under the snapshot could already have been removed (that'd
* require the set of running transactions to change) and it fulfills the
diff --git a/src/bin/pg_rewind/libpq_source.c b/src/bin/pg_rewind/libpq_source.c
index 86d2adcaee..ac794cf4eb 100644
--- a/src/bin/pg_rewind/libpq_source.c
+++ b/src/bin/pg_rewind/libpq_source.c
@@ -539,7 +539,7 @@ process_queued_fetch_requests(libpq_source *src)
chunkoff, rq->path, (int64) rq->offset);
/*
- * We should not receive receive more data than we requested, or
+ * We should not receive more data than we requested, or
* pg_read_binary_file() messed up. We could receive less,
* though, if the file was truncated in the source after we
* checked its size. That's OK, there should be a WAL record of
diff --git a/src/test/regress/expected/tuplesort.out b/src/test/regress/expected/tuplesort.out
index 3fc1998bf2..418f296a3f 100644
--- a/src/test/regress/expected/tuplesort.out
+++ b/src/test/regress/expected/tuplesort.out
@@ -1,7 +1,7 @@
-- only use parallelism when explicitly intending to do so
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
diff --git a/src/test/regress/sql/tuplesort.sql b/src/test/regress/sql/tuplesort.sql
index 7d7e02f02a..846484d561 100644
--- a/src/test/regress/sql/tuplesort.sql
+++ b/src/test/regress/sql/tuplesort.sql
@@ -2,7 +2,7 @@
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
--
2.17.0
--lc9FT7cWel8HagAv
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0012-doc-review-piecemeal-construction-of-partitioned-ind.patch"
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH 11/18] duplicate words
@ 2021-01-31 00:10 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Justin Pryzby @ 2021-01-31 00:10 UTC (permalink / raw)
commit 9c4f5192f69ed16c99e0d079f0b5faebd7bad212
Allow pg_rewind to use a standby server as the source system.
commit 4a252996d5fda7662b2afdf329a5c95be0fe3b01
Add tests for tuplesort.c.
commit 0a2bc5d61e713e3fe72438f020eea5fcc90b0f0b
Move per-agg and per-trans duplicate finding to the planner.
commit 623a9ba79bbdd11c5eccb30b8bd5c446130e521c
snapshot scalability: cache snapshots using a xact completion counter.
commit 2c03216d831160bedd72d45f712601b6f7d03f1c
Revamp the WAL record format.
---
src/backend/access/transam/xlogutils.c | 3 +--
src/backend/optimizer/prep/prepagg.c | 2 +-
src/backend/storage/ipc/procarray.c | 2 +-
src/bin/pg_rewind/libpq_source.c | 2 +-
src/test/regress/expected/tuplesort.out | 2 +-
src/test/regress/sql/tuplesort.sql | 2 +-
6 files changed, 6 insertions(+), 7 deletions(-)
diff --git a/src/backend/access/transam/xlogutils.c b/src/backend/access/transam/xlogutils.c
index e723253297..25d6df1659 100644
--- a/src/backend/access/transam/xlogutils.c
+++ b/src/backend/access/transam/xlogutils.c
@@ -433,8 +433,7 @@ XLogReadBufferForRedoExtended(XLogReaderState *record,
* NB: A redo function should normally not call this directly. To get a page
* to modify, use XLogReadBufferForRedoExtended instead. It is important that
* all pages modified by a WAL record are registered in the WAL records, or
- * they will be invisible to tools that that need to know which pages are
- * modified.
+ * they will be invisible to tools that need to know which pages are modified.
*/
Buffer
XLogReadBufferExtended(RelFileNode rnode, ForkNumber forknum,
diff --git a/src/backend/optimizer/prep/prepagg.c b/src/backend/optimizer/prep/prepagg.c
index 929a8ea13b..89046f9afb 100644
--- a/src/backend/optimizer/prep/prepagg.c
+++ b/src/backend/optimizer/prep/prepagg.c
@@ -71,7 +71,7 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
*
* Information about the aggregates and transition functions are collected
* in the root->agginfos and root->aggtransinfos lists. The 'aggtranstype',
- * 'aggno', and 'aggtransno' fields in are filled in in each Aggref.
+ * 'aggno', and 'aggtransno' fields of each Aggref are filled in.
*
* NOTE: This modifies the Aggrefs in the input expression in-place!
*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index cf12eda504..b9fbdcb88f 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -2049,7 +2049,7 @@ GetSnapshotDataReuse(Snapshot snapshot)
* holding ProcArrayLock) exclusively). Thus the xactCompletionCount check
* ensures we would detect if the snapshot would have changed.
*
- * As the snapshot contents are the same as it was before, it is is safe
+ * As the snapshot contents are the same as it was before, it is safe
* to re-enter the snapshot's xmin into the PGPROC array. None of the rows
* visible under the snapshot could already have been removed (that'd
* require the set of running transactions to change) and it fulfills the
diff --git a/src/bin/pg_rewind/libpq_source.c b/src/bin/pg_rewind/libpq_source.c
index 86d2adcaee..ac794cf4eb 100644
--- a/src/bin/pg_rewind/libpq_source.c
+++ b/src/bin/pg_rewind/libpq_source.c
@@ -539,7 +539,7 @@ process_queued_fetch_requests(libpq_source *src)
chunkoff, rq->path, (int64) rq->offset);
/*
- * We should not receive receive more data than we requested, or
+ * We should not receive more data than we requested, or
* pg_read_binary_file() messed up. We could receive less,
* though, if the file was truncated in the source after we
* checked its size. That's OK, there should be a WAL record of
diff --git a/src/test/regress/expected/tuplesort.out b/src/test/regress/expected/tuplesort.out
index 3fc1998bf2..418f296a3f 100644
--- a/src/test/regress/expected/tuplesort.out
+++ b/src/test/regress/expected/tuplesort.out
@@ -1,7 +1,7 @@
-- only use parallelism when explicitly intending to do so
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
diff --git a/src/test/regress/sql/tuplesort.sql b/src/test/regress/sql/tuplesort.sql
index 7d7e02f02a..846484d561 100644
--- a/src/test/regress/sql/tuplesort.sql
+++ b/src/test/regress/sql/tuplesort.sql
@@ -2,7 +2,7 @@
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
--
2.17.0
--lc9FT7cWel8HagAv
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0012-doc-review-piecemeal-construction-of-partitioned-ind.patch"
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH 11/18] duplicate words
@ 2021-01-31 00:10 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Justin Pryzby @ 2021-01-31 00:10 UTC (permalink / raw)
commit 9c4f5192f69ed16c99e0d079f0b5faebd7bad212
Allow pg_rewind to use a standby server as the source system.
commit 4a252996d5fda7662b2afdf329a5c95be0fe3b01
Add tests for tuplesort.c.
commit 0a2bc5d61e713e3fe72438f020eea5fcc90b0f0b
Move per-agg and per-trans duplicate finding to the planner.
commit 623a9ba79bbdd11c5eccb30b8bd5c446130e521c
snapshot scalability: cache snapshots using a xact completion counter.
commit 2c03216d831160bedd72d45f712601b6f7d03f1c
Revamp the WAL record format.
---
src/backend/access/transam/xlogutils.c | 3 +--
src/backend/optimizer/prep/prepagg.c | 2 +-
src/backend/storage/ipc/procarray.c | 2 +-
src/bin/pg_rewind/libpq_source.c | 2 +-
src/test/regress/expected/tuplesort.out | 2 +-
src/test/regress/sql/tuplesort.sql | 2 +-
6 files changed, 6 insertions(+), 7 deletions(-)
diff --git a/src/backend/access/transam/xlogutils.c b/src/backend/access/transam/xlogutils.c
index e723253297..25d6df1659 100644
--- a/src/backend/access/transam/xlogutils.c
+++ b/src/backend/access/transam/xlogutils.c
@@ -433,8 +433,7 @@ XLogReadBufferForRedoExtended(XLogReaderState *record,
* NB: A redo function should normally not call this directly. To get a page
* to modify, use XLogReadBufferForRedoExtended instead. It is important that
* all pages modified by a WAL record are registered in the WAL records, or
- * they will be invisible to tools that that need to know which pages are
- * modified.
+ * they will be invisible to tools that need to know which pages are modified.
*/
Buffer
XLogReadBufferExtended(RelFileNode rnode, ForkNumber forknum,
diff --git a/src/backend/optimizer/prep/prepagg.c b/src/backend/optimizer/prep/prepagg.c
index 929a8ea13b..89046f9afb 100644
--- a/src/backend/optimizer/prep/prepagg.c
+++ b/src/backend/optimizer/prep/prepagg.c
@@ -71,7 +71,7 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
*
* Information about the aggregates and transition functions are collected
* in the root->agginfos and root->aggtransinfos lists. The 'aggtranstype',
- * 'aggno', and 'aggtransno' fields in are filled in in each Aggref.
+ * 'aggno', and 'aggtransno' fields of each Aggref are filled in.
*
* NOTE: This modifies the Aggrefs in the input expression in-place!
*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index cf12eda504..b9fbdcb88f 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -2049,7 +2049,7 @@ GetSnapshotDataReuse(Snapshot snapshot)
* holding ProcArrayLock) exclusively). Thus the xactCompletionCount check
* ensures we would detect if the snapshot would have changed.
*
- * As the snapshot contents are the same as it was before, it is is safe
+ * As the snapshot contents are the same as it was before, it is safe
* to re-enter the snapshot's xmin into the PGPROC array. None of the rows
* visible under the snapshot could already have been removed (that'd
* require the set of running transactions to change) and it fulfills the
diff --git a/src/bin/pg_rewind/libpq_source.c b/src/bin/pg_rewind/libpq_source.c
index 86d2adcaee..ac794cf4eb 100644
--- a/src/bin/pg_rewind/libpq_source.c
+++ b/src/bin/pg_rewind/libpq_source.c
@@ -539,7 +539,7 @@ process_queued_fetch_requests(libpq_source *src)
chunkoff, rq->path, (int64) rq->offset);
/*
- * We should not receive receive more data than we requested, or
+ * We should not receive more data than we requested, or
* pg_read_binary_file() messed up. We could receive less,
* though, if the file was truncated in the source after we
* checked its size. That's OK, there should be a WAL record of
diff --git a/src/test/regress/expected/tuplesort.out b/src/test/regress/expected/tuplesort.out
index 3fc1998bf2..418f296a3f 100644
--- a/src/test/regress/expected/tuplesort.out
+++ b/src/test/regress/expected/tuplesort.out
@@ -1,7 +1,7 @@
-- only use parallelism when explicitly intending to do so
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
diff --git a/src/test/regress/sql/tuplesort.sql b/src/test/regress/sql/tuplesort.sql
index 7d7e02f02a..846484d561 100644
--- a/src/test/regress/sql/tuplesort.sql
+++ b/src/test/regress/sql/tuplesort.sql
@@ -2,7 +2,7 @@
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
--
2.17.0
--lc9FT7cWel8HagAv
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0012-doc-review-piecemeal-construction-of-partitioned-ind.patch"
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH 11/18] duplicate words
@ 2021-01-31 00:10 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Justin Pryzby @ 2021-01-31 00:10 UTC (permalink / raw)
commit 9c4f5192f69ed16c99e0d079f0b5faebd7bad212
Allow pg_rewind to use a standby server as the source system.
commit 4a252996d5fda7662b2afdf329a5c95be0fe3b01
Add tests for tuplesort.c.
commit 0a2bc5d61e713e3fe72438f020eea5fcc90b0f0b
Move per-agg and per-trans duplicate finding to the planner.
commit 623a9ba79bbdd11c5eccb30b8bd5c446130e521c
snapshot scalability: cache snapshots using a xact completion counter.
commit 2c03216d831160bedd72d45f712601b6f7d03f1c
Revamp the WAL record format.
---
src/backend/access/transam/xlogutils.c | 3 +--
src/backend/optimizer/prep/prepagg.c | 2 +-
src/backend/storage/ipc/procarray.c | 2 +-
src/bin/pg_rewind/libpq_source.c | 2 +-
src/test/regress/expected/tuplesort.out | 2 +-
src/test/regress/sql/tuplesort.sql | 2 +-
6 files changed, 6 insertions(+), 7 deletions(-)
diff --git a/src/backend/access/transam/xlogutils.c b/src/backend/access/transam/xlogutils.c
index e723253297..25d6df1659 100644
--- a/src/backend/access/transam/xlogutils.c
+++ b/src/backend/access/transam/xlogutils.c
@@ -433,8 +433,7 @@ XLogReadBufferForRedoExtended(XLogReaderState *record,
* NB: A redo function should normally not call this directly. To get a page
* to modify, use XLogReadBufferForRedoExtended instead. It is important that
* all pages modified by a WAL record are registered in the WAL records, or
- * they will be invisible to tools that that need to know which pages are
- * modified.
+ * they will be invisible to tools that need to know which pages are modified.
*/
Buffer
XLogReadBufferExtended(RelFileNode rnode, ForkNumber forknum,
diff --git a/src/backend/optimizer/prep/prepagg.c b/src/backend/optimizer/prep/prepagg.c
index 929a8ea13b..89046f9afb 100644
--- a/src/backend/optimizer/prep/prepagg.c
+++ b/src/backend/optimizer/prep/prepagg.c
@@ -71,7 +71,7 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
*
* Information about the aggregates and transition functions are collected
* in the root->agginfos and root->aggtransinfos lists. The 'aggtranstype',
- * 'aggno', and 'aggtransno' fields in are filled in in each Aggref.
+ * 'aggno', and 'aggtransno' fields of each Aggref are filled in.
*
* NOTE: This modifies the Aggrefs in the input expression in-place!
*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index cf12eda504..b9fbdcb88f 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -2049,7 +2049,7 @@ GetSnapshotDataReuse(Snapshot snapshot)
* holding ProcArrayLock) exclusively). Thus the xactCompletionCount check
* ensures we would detect if the snapshot would have changed.
*
- * As the snapshot contents are the same as it was before, it is is safe
+ * As the snapshot contents are the same as it was before, it is safe
* to re-enter the snapshot's xmin into the PGPROC array. None of the rows
* visible under the snapshot could already have been removed (that'd
* require the set of running transactions to change) and it fulfills the
diff --git a/src/bin/pg_rewind/libpq_source.c b/src/bin/pg_rewind/libpq_source.c
index 86d2adcaee..ac794cf4eb 100644
--- a/src/bin/pg_rewind/libpq_source.c
+++ b/src/bin/pg_rewind/libpq_source.c
@@ -539,7 +539,7 @@ process_queued_fetch_requests(libpq_source *src)
chunkoff, rq->path, (int64) rq->offset);
/*
- * We should not receive receive more data than we requested, or
+ * We should not receive more data than we requested, or
* pg_read_binary_file() messed up. We could receive less,
* though, if the file was truncated in the source after we
* checked its size. That's OK, there should be a WAL record of
diff --git a/src/test/regress/expected/tuplesort.out b/src/test/regress/expected/tuplesort.out
index 3fc1998bf2..418f296a3f 100644
--- a/src/test/regress/expected/tuplesort.out
+++ b/src/test/regress/expected/tuplesort.out
@@ -1,7 +1,7 @@
-- only use parallelism when explicitly intending to do so
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
diff --git a/src/test/regress/sql/tuplesort.sql b/src/test/regress/sql/tuplesort.sql
index 7d7e02f02a..846484d561 100644
--- a/src/test/regress/sql/tuplesort.sql
+++ b/src/test/regress/sql/tuplesort.sql
@@ -2,7 +2,7 @@
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
--
2.17.0
--lc9FT7cWel8HagAv
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0012-doc-review-piecemeal-construction-of-partitioned-ind.patch"
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH 11/18] duplicate words
@ 2021-01-31 00:10 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Justin Pryzby @ 2021-01-31 00:10 UTC (permalink / raw)
commit 9c4f5192f69ed16c99e0d079f0b5faebd7bad212
Allow pg_rewind to use a standby server as the source system.
commit 4a252996d5fda7662b2afdf329a5c95be0fe3b01
Add tests for tuplesort.c.
commit 0a2bc5d61e713e3fe72438f020eea5fcc90b0f0b
Move per-agg and per-trans duplicate finding to the planner.
commit 623a9ba79bbdd11c5eccb30b8bd5c446130e521c
snapshot scalability: cache snapshots using a xact completion counter.
commit 2c03216d831160bedd72d45f712601b6f7d03f1c
Revamp the WAL record format.
---
src/backend/access/transam/xlogutils.c | 3 +--
src/backend/optimizer/prep/prepagg.c | 2 +-
src/backend/storage/ipc/procarray.c | 2 +-
src/bin/pg_rewind/libpq_source.c | 2 +-
src/test/regress/expected/tuplesort.out | 2 +-
src/test/regress/sql/tuplesort.sql | 2 +-
6 files changed, 6 insertions(+), 7 deletions(-)
diff --git a/src/backend/access/transam/xlogutils.c b/src/backend/access/transam/xlogutils.c
index e723253297..25d6df1659 100644
--- a/src/backend/access/transam/xlogutils.c
+++ b/src/backend/access/transam/xlogutils.c
@@ -433,8 +433,7 @@ XLogReadBufferForRedoExtended(XLogReaderState *record,
* NB: A redo function should normally not call this directly. To get a page
* to modify, use XLogReadBufferForRedoExtended instead. It is important that
* all pages modified by a WAL record are registered in the WAL records, or
- * they will be invisible to tools that that need to know which pages are
- * modified.
+ * they will be invisible to tools that need to know which pages are modified.
*/
Buffer
XLogReadBufferExtended(RelFileNode rnode, ForkNumber forknum,
diff --git a/src/backend/optimizer/prep/prepagg.c b/src/backend/optimizer/prep/prepagg.c
index 929a8ea13b..89046f9afb 100644
--- a/src/backend/optimizer/prep/prepagg.c
+++ b/src/backend/optimizer/prep/prepagg.c
@@ -71,7 +71,7 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
*
* Information about the aggregates and transition functions are collected
* in the root->agginfos and root->aggtransinfos lists. The 'aggtranstype',
- * 'aggno', and 'aggtransno' fields in are filled in in each Aggref.
+ * 'aggno', and 'aggtransno' fields of each Aggref are filled in.
*
* NOTE: This modifies the Aggrefs in the input expression in-place!
*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index cf12eda504..b9fbdcb88f 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -2049,7 +2049,7 @@ GetSnapshotDataReuse(Snapshot snapshot)
* holding ProcArrayLock) exclusively). Thus the xactCompletionCount check
* ensures we would detect if the snapshot would have changed.
*
- * As the snapshot contents are the same as it was before, it is is safe
+ * As the snapshot contents are the same as it was before, it is safe
* to re-enter the snapshot's xmin into the PGPROC array. None of the rows
* visible under the snapshot could already have been removed (that'd
* require the set of running transactions to change) and it fulfills the
diff --git a/src/bin/pg_rewind/libpq_source.c b/src/bin/pg_rewind/libpq_source.c
index 86d2adcaee..ac794cf4eb 100644
--- a/src/bin/pg_rewind/libpq_source.c
+++ b/src/bin/pg_rewind/libpq_source.c
@@ -539,7 +539,7 @@ process_queued_fetch_requests(libpq_source *src)
chunkoff, rq->path, (int64) rq->offset);
/*
- * We should not receive receive more data than we requested, or
+ * We should not receive more data than we requested, or
* pg_read_binary_file() messed up. We could receive less,
* though, if the file was truncated in the source after we
* checked its size. That's OK, there should be a WAL record of
diff --git a/src/test/regress/expected/tuplesort.out b/src/test/regress/expected/tuplesort.out
index 3fc1998bf2..418f296a3f 100644
--- a/src/test/regress/expected/tuplesort.out
+++ b/src/test/regress/expected/tuplesort.out
@@ -1,7 +1,7 @@
-- only use parallelism when explicitly intending to do so
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
diff --git a/src/test/regress/sql/tuplesort.sql b/src/test/regress/sql/tuplesort.sql
index 7d7e02f02a..846484d561 100644
--- a/src/test/regress/sql/tuplesort.sql
+++ b/src/test/regress/sql/tuplesort.sql
@@ -2,7 +2,7 @@
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
--
2.17.0
--lc9FT7cWel8HagAv
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0012-doc-review-piecemeal-construction-of-partitioned-ind.patch"
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH 11/18] duplicate words
@ 2021-01-31 00:10 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Justin Pryzby @ 2021-01-31 00:10 UTC (permalink / raw)
commit 9c4f5192f69ed16c99e0d079f0b5faebd7bad212
Allow pg_rewind to use a standby server as the source system.
commit 4a252996d5fda7662b2afdf329a5c95be0fe3b01
Add tests for tuplesort.c.
commit 0a2bc5d61e713e3fe72438f020eea5fcc90b0f0b
Move per-agg and per-trans duplicate finding to the planner.
commit 623a9ba79bbdd11c5eccb30b8bd5c446130e521c
snapshot scalability: cache snapshots using a xact completion counter.
commit 2c03216d831160bedd72d45f712601b6f7d03f1c
Revamp the WAL record format.
---
src/backend/access/transam/xlogutils.c | 3 +--
src/backend/optimizer/prep/prepagg.c | 2 +-
src/backend/storage/ipc/procarray.c | 2 +-
src/bin/pg_rewind/libpq_source.c | 2 +-
src/test/regress/expected/tuplesort.out | 2 +-
src/test/regress/sql/tuplesort.sql | 2 +-
6 files changed, 6 insertions(+), 7 deletions(-)
diff --git a/src/backend/access/transam/xlogutils.c b/src/backend/access/transam/xlogutils.c
index e723253297..25d6df1659 100644
--- a/src/backend/access/transam/xlogutils.c
+++ b/src/backend/access/transam/xlogutils.c
@@ -433,8 +433,7 @@ XLogReadBufferForRedoExtended(XLogReaderState *record,
* NB: A redo function should normally not call this directly. To get a page
* to modify, use XLogReadBufferForRedoExtended instead. It is important that
* all pages modified by a WAL record are registered in the WAL records, or
- * they will be invisible to tools that that need to know which pages are
- * modified.
+ * they will be invisible to tools that need to know which pages are modified.
*/
Buffer
XLogReadBufferExtended(RelFileNode rnode, ForkNumber forknum,
diff --git a/src/backend/optimizer/prep/prepagg.c b/src/backend/optimizer/prep/prepagg.c
index 929a8ea13b..89046f9afb 100644
--- a/src/backend/optimizer/prep/prepagg.c
+++ b/src/backend/optimizer/prep/prepagg.c
@@ -71,7 +71,7 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
*
* Information about the aggregates and transition functions are collected
* in the root->agginfos and root->aggtransinfos lists. The 'aggtranstype',
- * 'aggno', and 'aggtransno' fields in are filled in in each Aggref.
+ * 'aggno', and 'aggtransno' fields of each Aggref are filled in.
*
* NOTE: This modifies the Aggrefs in the input expression in-place!
*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index cf12eda504..b9fbdcb88f 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -2049,7 +2049,7 @@ GetSnapshotDataReuse(Snapshot snapshot)
* holding ProcArrayLock) exclusively). Thus the xactCompletionCount check
* ensures we would detect if the snapshot would have changed.
*
- * As the snapshot contents are the same as it was before, it is is safe
+ * As the snapshot contents are the same as it was before, it is safe
* to re-enter the snapshot's xmin into the PGPROC array. None of the rows
* visible under the snapshot could already have been removed (that'd
* require the set of running transactions to change) and it fulfills the
diff --git a/src/bin/pg_rewind/libpq_source.c b/src/bin/pg_rewind/libpq_source.c
index 86d2adcaee..ac794cf4eb 100644
--- a/src/bin/pg_rewind/libpq_source.c
+++ b/src/bin/pg_rewind/libpq_source.c
@@ -539,7 +539,7 @@ process_queued_fetch_requests(libpq_source *src)
chunkoff, rq->path, (int64) rq->offset);
/*
- * We should not receive receive more data than we requested, or
+ * We should not receive more data than we requested, or
* pg_read_binary_file() messed up. We could receive less,
* though, if the file was truncated in the source after we
* checked its size. That's OK, there should be a WAL record of
diff --git a/src/test/regress/expected/tuplesort.out b/src/test/regress/expected/tuplesort.out
index 3fc1998bf2..418f296a3f 100644
--- a/src/test/regress/expected/tuplesort.out
+++ b/src/test/regress/expected/tuplesort.out
@@ -1,7 +1,7 @@
-- only use parallelism when explicitly intending to do so
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
diff --git a/src/test/regress/sql/tuplesort.sql b/src/test/regress/sql/tuplesort.sql
index 7d7e02f02a..846484d561 100644
--- a/src/test/regress/sql/tuplesort.sql
+++ b/src/test/regress/sql/tuplesort.sql
@@ -2,7 +2,7 @@
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
--
2.17.0
--lc9FT7cWel8HagAv
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0012-doc-review-piecemeal-construction-of-partitioned-ind.patch"
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH 11/18] duplicate words
@ 2021-01-31 00:10 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Justin Pryzby @ 2021-01-31 00:10 UTC (permalink / raw)
commit 9c4f5192f69ed16c99e0d079f0b5faebd7bad212
Allow pg_rewind to use a standby server as the source system.
commit 4a252996d5fda7662b2afdf329a5c95be0fe3b01
Add tests for tuplesort.c.
commit 0a2bc5d61e713e3fe72438f020eea5fcc90b0f0b
Move per-agg and per-trans duplicate finding to the planner.
commit 623a9ba79bbdd11c5eccb30b8bd5c446130e521c
snapshot scalability: cache snapshots using a xact completion counter.
commit 2c03216d831160bedd72d45f712601b6f7d03f1c
Revamp the WAL record format.
---
src/backend/access/transam/xlogutils.c | 3 +--
src/backend/optimizer/prep/prepagg.c | 2 +-
src/backend/storage/ipc/procarray.c | 2 +-
src/bin/pg_rewind/libpq_source.c | 2 +-
src/test/regress/expected/tuplesort.out | 2 +-
src/test/regress/sql/tuplesort.sql | 2 +-
6 files changed, 6 insertions(+), 7 deletions(-)
diff --git a/src/backend/access/transam/xlogutils.c b/src/backend/access/transam/xlogutils.c
index e723253297..25d6df1659 100644
--- a/src/backend/access/transam/xlogutils.c
+++ b/src/backend/access/transam/xlogutils.c
@@ -433,8 +433,7 @@ XLogReadBufferForRedoExtended(XLogReaderState *record,
* NB: A redo function should normally not call this directly. To get a page
* to modify, use XLogReadBufferForRedoExtended instead. It is important that
* all pages modified by a WAL record are registered in the WAL records, or
- * they will be invisible to tools that that need to know which pages are
- * modified.
+ * they will be invisible to tools that need to know which pages are modified.
*/
Buffer
XLogReadBufferExtended(RelFileNode rnode, ForkNumber forknum,
diff --git a/src/backend/optimizer/prep/prepagg.c b/src/backend/optimizer/prep/prepagg.c
index 929a8ea13b..89046f9afb 100644
--- a/src/backend/optimizer/prep/prepagg.c
+++ b/src/backend/optimizer/prep/prepagg.c
@@ -71,7 +71,7 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
*
* Information about the aggregates and transition functions are collected
* in the root->agginfos and root->aggtransinfos lists. The 'aggtranstype',
- * 'aggno', and 'aggtransno' fields in are filled in in each Aggref.
+ * 'aggno', and 'aggtransno' fields of each Aggref are filled in.
*
* NOTE: This modifies the Aggrefs in the input expression in-place!
*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index cf12eda504..b9fbdcb88f 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -2049,7 +2049,7 @@ GetSnapshotDataReuse(Snapshot snapshot)
* holding ProcArrayLock) exclusively). Thus the xactCompletionCount check
* ensures we would detect if the snapshot would have changed.
*
- * As the snapshot contents are the same as it was before, it is is safe
+ * As the snapshot contents are the same as it was before, it is safe
* to re-enter the snapshot's xmin into the PGPROC array. None of the rows
* visible under the snapshot could already have been removed (that'd
* require the set of running transactions to change) and it fulfills the
diff --git a/src/bin/pg_rewind/libpq_source.c b/src/bin/pg_rewind/libpq_source.c
index 86d2adcaee..ac794cf4eb 100644
--- a/src/bin/pg_rewind/libpq_source.c
+++ b/src/bin/pg_rewind/libpq_source.c
@@ -539,7 +539,7 @@ process_queued_fetch_requests(libpq_source *src)
chunkoff, rq->path, (int64) rq->offset);
/*
- * We should not receive receive more data than we requested, or
+ * We should not receive more data than we requested, or
* pg_read_binary_file() messed up. We could receive less,
* though, if the file was truncated in the source after we
* checked its size. That's OK, there should be a WAL record of
diff --git a/src/test/regress/expected/tuplesort.out b/src/test/regress/expected/tuplesort.out
index 3fc1998bf2..418f296a3f 100644
--- a/src/test/regress/expected/tuplesort.out
+++ b/src/test/regress/expected/tuplesort.out
@@ -1,7 +1,7 @@
-- only use parallelism when explicitly intending to do so
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
diff --git a/src/test/regress/sql/tuplesort.sql b/src/test/regress/sql/tuplesort.sql
index 7d7e02f02a..846484d561 100644
--- a/src/test/regress/sql/tuplesort.sql
+++ b/src/test/regress/sql/tuplesort.sql
@@ -2,7 +2,7 @@
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
--
2.17.0
--lc9FT7cWel8HagAv
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0012-doc-review-piecemeal-construction-of-partitioned-ind.patch"
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH 11/18] duplicate words
@ 2021-01-31 00:10 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Justin Pryzby @ 2021-01-31 00:10 UTC (permalink / raw)
commit 9c4f5192f69ed16c99e0d079f0b5faebd7bad212
Allow pg_rewind to use a standby server as the source system.
commit 4a252996d5fda7662b2afdf329a5c95be0fe3b01
Add tests for tuplesort.c.
commit 0a2bc5d61e713e3fe72438f020eea5fcc90b0f0b
Move per-agg and per-trans duplicate finding to the planner.
commit 623a9ba79bbdd11c5eccb30b8bd5c446130e521c
snapshot scalability: cache snapshots using a xact completion counter.
commit 2c03216d831160bedd72d45f712601b6f7d03f1c
Revamp the WAL record format.
---
src/backend/access/transam/xlogutils.c | 3 +--
src/backend/optimizer/prep/prepagg.c | 2 +-
src/backend/storage/ipc/procarray.c | 2 +-
src/bin/pg_rewind/libpq_source.c | 2 +-
src/test/regress/expected/tuplesort.out | 2 +-
src/test/regress/sql/tuplesort.sql | 2 +-
6 files changed, 6 insertions(+), 7 deletions(-)
diff --git a/src/backend/access/transam/xlogutils.c b/src/backend/access/transam/xlogutils.c
index e723253297..25d6df1659 100644
--- a/src/backend/access/transam/xlogutils.c
+++ b/src/backend/access/transam/xlogutils.c
@@ -433,8 +433,7 @@ XLogReadBufferForRedoExtended(XLogReaderState *record,
* NB: A redo function should normally not call this directly. To get a page
* to modify, use XLogReadBufferForRedoExtended instead. It is important that
* all pages modified by a WAL record are registered in the WAL records, or
- * they will be invisible to tools that that need to know which pages are
- * modified.
+ * they will be invisible to tools that need to know which pages are modified.
*/
Buffer
XLogReadBufferExtended(RelFileNode rnode, ForkNumber forknum,
diff --git a/src/backend/optimizer/prep/prepagg.c b/src/backend/optimizer/prep/prepagg.c
index 929a8ea13b..89046f9afb 100644
--- a/src/backend/optimizer/prep/prepagg.c
+++ b/src/backend/optimizer/prep/prepagg.c
@@ -71,7 +71,7 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
*
* Information about the aggregates and transition functions are collected
* in the root->agginfos and root->aggtransinfos lists. The 'aggtranstype',
- * 'aggno', and 'aggtransno' fields in are filled in in each Aggref.
+ * 'aggno', and 'aggtransno' fields of each Aggref are filled in.
*
* NOTE: This modifies the Aggrefs in the input expression in-place!
*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index cf12eda504..b9fbdcb88f 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -2049,7 +2049,7 @@ GetSnapshotDataReuse(Snapshot snapshot)
* holding ProcArrayLock) exclusively). Thus the xactCompletionCount check
* ensures we would detect if the snapshot would have changed.
*
- * As the snapshot contents are the same as it was before, it is is safe
+ * As the snapshot contents are the same as it was before, it is safe
* to re-enter the snapshot's xmin into the PGPROC array. None of the rows
* visible under the snapshot could already have been removed (that'd
* require the set of running transactions to change) and it fulfills the
diff --git a/src/bin/pg_rewind/libpq_source.c b/src/bin/pg_rewind/libpq_source.c
index 86d2adcaee..ac794cf4eb 100644
--- a/src/bin/pg_rewind/libpq_source.c
+++ b/src/bin/pg_rewind/libpq_source.c
@@ -539,7 +539,7 @@ process_queued_fetch_requests(libpq_source *src)
chunkoff, rq->path, (int64) rq->offset);
/*
- * We should not receive receive more data than we requested, or
+ * We should not receive more data than we requested, or
* pg_read_binary_file() messed up. We could receive less,
* though, if the file was truncated in the source after we
* checked its size. That's OK, there should be a WAL record of
diff --git a/src/test/regress/expected/tuplesort.out b/src/test/regress/expected/tuplesort.out
index 3fc1998bf2..418f296a3f 100644
--- a/src/test/regress/expected/tuplesort.out
+++ b/src/test/regress/expected/tuplesort.out
@@ -1,7 +1,7 @@
-- only use parallelism when explicitly intending to do so
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
diff --git a/src/test/regress/sql/tuplesort.sql b/src/test/regress/sql/tuplesort.sql
index 7d7e02f02a..846484d561 100644
--- a/src/test/regress/sql/tuplesort.sql
+++ b/src/test/regress/sql/tuplesort.sql
@@ -2,7 +2,7 @@
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
--
2.17.0
--lc9FT7cWel8HagAv
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0012-doc-review-piecemeal-construction-of-partitioned-ind.patch"
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH 11/18] duplicate words
@ 2021-01-31 00:10 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Justin Pryzby @ 2021-01-31 00:10 UTC (permalink / raw)
commit 9c4f5192f69ed16c99e0d079f0b5faebd7bad212
Allow pg_rewind to use a standby server as the source system.
commit 4a252996d5fda7662b2afdf329a5c95be0fe3b01
Add tests for tuplesort.c.
commit 0a2bc5d61e713e3fe72438f020eea5fcc90b0f0b
Move per-agg and per-trans duplicate finding to the planner.
commit 623a9ba79bbdd11c5eccb30b8bd5c446130e521c
snapshot scalability: cache snapshots using a xact completion counter.
commit 2c03216d831160bedd72d45f712601b6f7d03f1c
Revamp the WAL record format.
---
src/backend/access/transam/xlogutils.c | 3 +--
src/backend/optimizer/prep/prepagg.c | 2 +-
src/backend/storage/ipc/procarray.c | 2 +-
src/bin/pg_rewind/libpq_source.c | 2 +-
src/test/regress/expected/tuplesort.out | 2 +-
src/test/regress/sql/tuplesort.sql | 2 +-
6 files changed, 6 insertions(+), 7 deletions(-)
diff --git a/src/backend/access/transam/xlogutils.c b/src/backend/access/transam/xlogutils.c
index e723253297..25d6df1659 100644
--- a/src/backend/access/transam/xlogutils.c
+++ b/src/backend/access/transam/xlogutils.c
@@ -433,8 +433,7 @@ XLogReadBufferForRedoExtended(XLogReaderState *record,
* NB: A redo function should normally not call this directly. To get a page
* to modify, use XLogReadBufferForRedoExtended instead. It is important that
* all pages modified by a WAL record are registered in the WAL records, or
- * they will be invisible to tools that that need to know which pages are
- * modified.
+ * they will be invisible to tools that need to know which pages are modified.
*/
Buffer
XLogReadBufferExtended(RelFileNode rnode, ForkNumber forknum,
diff --git a/src/backend/optimizer/prep/prepagg.c b/src/backend/optimizer/prep/prepagg.c
index 929a8ea13b..89046f9afb 100644
--- a/src/backend/optimizer/prep/prepagg.c
+++ b/src/backend/optimizer/prep/prepagg.c
@@ -71,7 +71,7 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
*
* Information about the aggregates and transition functions are collected
* in the root->agginfos and root->aggtransinfos lists. The 'aggtranstype',
- * 'aggno', and 'aggtransno' fields in are filled in in each Aggref.
+ * 'aggno', and 'aggtransno' fields of each Aggref are filled in.
*
* NOTE: This modifies the Aggrefs in the input expression in-place!
*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index cf12eda504..b9fbdcb88f 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -2049,7 +2049,7 @@ GetSnapshotDataReuse(Snapshot snapshot)
* holding ProcArrayLock) exclusively). Thus the xactCompletionCount check
* ensures we would detect if the snapshot would have changed.
*
- * As the snapshot contents are the same as it was before, it is is safe
+ * As the snapshot contents are the same as it was before, it is safe
* to re-enter the snapshot's xmin into the PGPROC array. None of the rows
* visible under the snapshot could already have been removed (that'd
* require the set of running transactions to change) and it fulfills the
diff --git a/src/bin/pg_rewind/libpq_source.c b/src/bin/pg_rewind/libpq_source.c
index 86d2adcaee..ac794cf4eb 100644
--- a/src/bin/pg_rewind/libpq_source.c
+++ b/src/bin/pg_rewind/libpq_source.c
@@ -539,7 +539,7 @@ process_queued_fetch_requests(libpq_source *src)
chunkoff, rq->path, (int64) rq->offset);
/*
- * We should not receive receive more data than we requested, or
+ * We should not receive more data than we requested, or
* pg_read_binary_file() messed up. We could receive less,
* though, if the file was truncated in the source after we
* checked its size. That's OK, there should be a WAL record of
diff --git a/src/test/regress/expected/tuplesort.out b/src/test/regress/expected/tuplesort.out
index 3fc1998bf2..418f296a3f 100644
--- a/src/test/regress/expected/tuplesort.out
+++ b/src/test/regress/expected/tuplesort.out
@@ -1,7 +1,7 @@
-- only use parallelism when explicitly intending to do so
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
diff --git a/src/test/regress/sql/tuplesort.sql b/src/test/regress/sql/tuplesort.sql
index 7d7e02f02a..846484d561 100644
--- a/src/test/regress/sql/tuplesort.sql
+++ b/src/test/regress/sql/tuplesort.sql
@@ -2,7 +2,7 @@
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
--
2.17.0
--lc9FT7cWel8HagAv
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0012-doc-review-piecemeal-construction-of-partitioned-ind.patch"
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH 11/18] duplicate words
@ 2021-01-31 00:10 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Justin Pryzby @ 2021-01-31 00:10 UTC (permalink / raw)
commit 9c4f5192f69ed16c99e0d079f0b5faebd7bad212
Allow pg_rewind to use a standby server as the source system.
commit 4a252996d5fda7662b2afdf329a5c95be0fe3b01
Add tests for tuplesort.c.
commit 0a2bc5d61e713e3fe72438f020eea5fcc90b0f0b
Move per-agg and per-trans duplicate finding to the planner.
commit 623a9ba79bbdd11c5eccb30b8bd5c446130e521c
snapshot scalability: cache snapshots using a xact completion counter.
commit 2c03216d831160bedd72d45f712601b6f7d03f1c
Revamp the WAL record format.
---
src/backend/access/transam/xlogutils.c | 3 +--
src/backend/optimizer/prep/prepagg.c | 2 +-
src/backend/storage/ipc/procarray.c | 2 +-
src/bin/pg_rewind/libpq_source.c | 2 +-
src/test/regress/expected/tuplesort.out | 2 +-
src/test/regress/sql/tuplesort.sql | 2 +-
6 files changed, 6 insertions(+), 7 deletions(-)
diff --git a/src/backend/access/transam/xlogutils.c b/src/backend/access/transam/xlogutils.c
index e723253297..25d6df1659 100644
--- a/src/backend/access/transam/xlogutils.c
+++ b/src/backend/access/transam/xlogutils.c
@@ -433,8 +433,7 @@ XLogReadBufferForRedoExtended(XLogReaderState *record,
* NB: A redo function should normally not call this directly. To get a page
* to modify, use XLogReadBufferForRedoExtended instead. It is important that
* all pages modified by a WAL record are registered in the WAL records, or
- * they will be invisible to tools that that need to know which pages are
- * modified.
+ * they will be invisible to tools that need to know which pages are modified.
*/
Buffer
XLogReadBufferExtended(RelFileNode rnode, ForkNumber forknum,
diff --git a/src/backend/optimizer/prep/prepagg.c b/src/backend/optimizer/prep/prepagg.c
index 929a8ea13b..89046f9afb 100644
--- a/src/backend/optimizer/prep/prepagg.c
+++ b/src/backend/optimizer/prep/prepagg.c
@@ -71,7 +71,7 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
*
* Information about the aggregates and transition functions are collected
* in the root->agginfos and root->aggtransinfos lists. The 'aggtranstype',
- * 'aggno', and 'aggtransno' fields in are filled in in each Aggref.
+ * 'aggno', and 'aggtransno' fields of each Aggref are filled in.
*
* NOTE: This modifies the Aggrefs in the input expression in-place!
*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index cf12eda504..b9fbdcb88f 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -2049,7 +2049,7 @@ GetSnapshotDataReuse(Snapshot snapshot)
* holding ProcArrayLock) exclusively). Thus the xactCompletionCount check
* ensures we would detect if the snapshot would have changed.
*
- * As the snapshot contents are the same as it was before, it is is safe
+ * As the snapshot contents are the same as it was before, it is safe
* to re-enter the snapshot's xmin into the PGPROC array. None of the rows
* visible under the snapshot could already have been removed (that'd
* require the set of running transactions to change) and it fulfills the
diff --git a/src/bin/pg_rewind/libpq_source.c b/src/bin/pg_rewind/libpq_source.c
index 86d2adcaee..ac794cf4eb 100644
--- a/src/bin/pg_rewind/libpq_source.c
+++ b/src/bin/pg_rewind/libpq_source.c
@@ -539,7 +539,7 @@ process_queued_fetch_requests(libpq_source *src)
chunkoff, rq->path, (int64) rq->offset);
/*
- * We should not receive receive more data than we requested, or
+ * We should not receive more data than we requested, or
* pg_read_binary_file() messed up. We could receive less,
* though, if the file was truncated in the source after we
* checked its size. That's OK, there should be a WAL record of
diff --git a/src/test/regress/expected/tuplesort.out b/src/test/regress/expected/tuplesort.out
index 3fc1998bf2..418f296a3f 100644
--- a/src/test/regress/expected/tuplesort.out
+++ b/src/test/regress/expected/tuplesort.out
@@ -1,7 +1,7 @@
-- only use parallelism when explicitly intending to do so
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
diff --git a/src/test/regress/sql/tuplesort.sql b/src/test/regress/sql/tuplesort.sql
index 7d7e02f02a..846484d561 100644
--- a/src/test/regress/sql/tuplesort.sql
+++ b/src/test/regress/sql/tuplesort.sql
@@ -2,7 +2,7 @@
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
--
2.17.0
--lc9FT7cWel8HagAv
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0012-doc-review-piecemeal-construction-of-partitioned-ind.patch"
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH 11/18] duplicate words
@ 2021-01-31 00:10 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Justin Pryzby @ 2021-01-31 00:10 UTC (permalink / raw)
commit 9c4f5192f69ed16c99e0d079f0b5faebd7bad212
Allow pg_rewind to use a standby server as the source system.
commit 4a252996d5fda7662b2afdf329a5c95be0fe3b01
Add tests for tuplesort.c.
commit 0a2bc5d61e713e3fe72438f020eea5fcc90b0f0b
Move per-agg and per-trans duplicate finding to the planner.
commit 623a9ba79bbdd11c5eccb30b8bd5c446130e521c
snapshot scalability: cache snapshots using a xact completion counter.
commit 2c03216d831160bedd72d45f712601b6f7d03f1c
Revamp the WAL record format.
---
src/backend/access/transam/xlogutils.c | 3 +--
src/backend/optimizer/prep/prepagg.c | 2 +-
src/backend/storage/ipc/procarray.c | 2 +-
src/bin/pg_rewind/libpq_source.c | 2 +-
src/test/regress/expected/tuplesort.out | 2 +-
src/test/regress/sql/tuplesort.sql | 2 +-
6 files changed, 6 insertions(+), 7 deletions(-)
diff --git a/src/backend/access/transam/xlogutils.c b/src/backend/access/transam/xlogutils.c
index e723253297..25d6df1659 100644
--- a/src/backend/access/transam/xlogutils.c
+++ b/src/backend/access/transam/xlogutils.c
@@ -433,8 +433,7 @@ XLogReadBufferForRedoExtended(XLogReaderState *record,
* NB: A redo function should normally not call this directly. To get a page
* to modify, use XLogReadBufferForRedoExtended instead. It is important that
* all pages modified by a WAL record are registered in the WAL records, or
- * they will be invisible to tools that that need to know which pages are
- * modified.
+ * they will be invisible to tools that need to know which pages are modified.
*/
Buffer
XLogReadBufferExtended(RelFileNode rnode, ForkNumber forknum,
diff --git a/src/backend/optimizer/prep/prepagg.c b/src/backend/optimizer/prep/prepagg.c
index 929a8ea13b..89046f9afb 100644
--- a/src/backend/optimizer/prep/prepagg.c
+++ b/src/backend/optimizer/prep/prepagg.c
@@ -71,7 +71,7 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
*
* Information about the aggregates and transition functions are collected
* in the root->agginfos and root->aggtransinfos lists. The 'aggtranstype',
- * 'aggno', and 'aggtransno' fields in are filled in in each Aggref.
+ * 'aggno', and 'aggtransno' fields of each Aggref are filled in.
*
* NOTE: This modifies the Aggrefs in the input expression in-place!
*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index cf12eda504..b9fbdcb88f 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -2049,7 +2049,7 @@ GetSnapshotDataReuse(Snapshot snapshot)
* holding ProcArrayLock) exclusively). Thus the xactCompletionCount check
* ensures we would detect if the snapshot would have changed.
*
- * As the snapshot contents are the same as it was before, it is is safe
+ * As the snapshot contents are the same as it was before, it is safe
* to re-enter the snapshot's xmin into the PGPROC array. None of the rows
* visible under the snapshot could already have been removed (that'd
* require the set of running transactions to change) and it fulfills the
diff --git a/src/bin/pg_rewind/libpq_source.c b/src/bin/pg_rewind/libpq_source.c
index 86d2adcaee..ac794cf4eb 100644
--- a/src/bin/pg_rewind/libpq_source.c
+++ b/src/bin/pg_rewind/libpq_source.c
@@ -539,7 +539,7 @@ process_queued_fetch_requests(libpq_source *src)
chunkoff, rq->path, (int64) rq->offset);
/*
- * We should not receive receive more data than we requested, or
+ * We should not receive more data than we requested, or
* pg_read_binary_file() messed up. We could receive less,
* though, if the file was truncated in the source after we
* checked its size. That's OK, there should be a WAL record of
diff --git a/src/test/regress/expected/tuplesort.out b/src/test/regress/expected/tuplesort.out
index 3fc1998bf2..418f296a3f 100644
--- a/src/test/regress/expected/tuplesort.out
+++ b/src/test/regress/expected/tuplesort.out
@@ -1,7 +1,7 @@
-- only use parallelism when explicitly intending to do so
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
diff --git a/src/test/regress/sql/tuplesort.sql b/src/test/regress/sql/tuplesort.sql
index 7d7e02f02a..846484d561 100644
--- a/src/test/regress/sql/tuplesort.sql
+++ b/src/test/regress/sql/tuplesort.sql
@@ -2,7 +2,7 @@
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
--
2.17.0
--lc9FT7cWel8HagAv
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0012-doc-review-piecemeal-construction-of-partitioned-ind.patch"
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH 11/18] duplicate words
@ 2021-01-31 00:10 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Justin Pryzby @ 2021-01-31 00:10 UTC (permalink / raw)
commit 9c4f5192f69ed16c99e0d079f0b5faebd7bad212
Allow pg_rewind to use a standby server as the source system.
commit 4a252996d5fda7662b2afdf329a5c95be0fe3b01
Add tests for tuplesort.c.
commit 0a2bc5d61e713e3fe72438f020eea5fcc90b0f0b
Move per-agg and per-trans duplicate finding to the planner.
commit 623a9ba79bbdd11c5eccb30b8bd5c446130e521c
snapshot scalability: cache snapshots using a xact completion counter.
commit 2c03216d831160bedd72d45f712601b6f7d03f1c
Revamp the WAL record format.
---
src/backend/access/transam/xlogutils.c | 3 +--
src/backend/optimizer/prep/prepagg.c | 2 +-
src/backend/storage/ipc/procarray.c | 2 +-
src/bin/pg_rewind/libpq_source.c | 2 +-
src/test/regress/expected/tuplesort.out | 2 +-
src/test/regress/sql/tuplesort.sql | 2 +-
6 files changed, 6 insertions(+), 7 deletions(-)
diff --git a/src/backend/access/transam/xlogutils.c b/src/backend/access/transam/xlogutils.c
index e723253297..25d6df1659 100644
--- a/src/backend/access/transam/xlogutils.c
+++ b/src/backend/access/transam/xlogutils.c
@@ -433,8 +433,7 @@ XLogReadBufferForRedoExtended(XLogReaderState *record,
* NB: A redo function should normally not call this directly. To get a page
* to modify, use XLogReadBufferForRedoExtended instead. It is important that
* all pages modified by a WAL record are registered in the WAL records, or
- * they will be invisible to tools that that need to know which pages are
- * modified.
+ * they will be invisible to tools that need to know which pages are modified.
*/
Buffer
XLogReadBufferExtended(RelFileNode rnode, ForkNumber forknum,
diff --git a/src/backend/optimizer/prep/prepagg.c b/src/backend/optimizer/prep/prepagg.c
index 929a8ea13b..89046f9afb 100644
--- a/src/backend/optimizer/prep/prepagg.c
+++ b/src/backend/optimizer/prep/prepagg.c
@@ -71,7 +71,7 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
*
* Information about the aggregates and transition functions are collected
* in the root->agginfos and root->aggtransinfos lists. The 'aggtranstype',
- * 'aggno', and 'aggtransno' fields in are filled in in each Aggref.
+ * 'aggno', and 'aggtransno' fields of each Aggref are filled in.
*
* NOTE: This modifies the Aggrefs in the input expression in-place!
*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index cf12eda504..b9fbdcb88f 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -2049,7 +2049,7 @@ GetSnapshotDataReuse(Snapshot snapshot)
* holding ProcArrayLock) exclusively). Thus the xactCompletionCount check
* ensures we would detect if the snapshot would have changed.
*
- * As the snapshot contents are the same as it was before, it is is safe
+ * As the snapshot contents are the same as it was before, it is safe
* to re-enter the snapshot's xmin into the PGPROC array. None of the rows
* visible under the snapshot could already have been removed (that'd
* require the set of running transactions to change) and it fulfills the
diff --git a/src/bin/pg_rewind/libpq_source.c b/src/bin/pg_rewind/libpq_source.c
index 86d2adcaee..ac794cf4eb 100644
--- a/src/bin/pg_rewind/libpq_source.c
+++ b/src/bin/pg_rewind/libpq_source.c
@@ -539,7 +539,7 @@ process_queued_fetch_requests(libpq_source *src)
chunkoff, rq->path, (int64) rq->offset);
/*
- * We should not receive receive more data than we requested, or
+ * We should not receive more data than we requested, or
* pg_read_binary_file() messed up. We could receive less,
* though, if the file was truncated in the source after we
* checked its size. That's OK, there should be a WAL record of
diff --git a/src/test/regress/expected/tuplesort.out b/src/test/regress/expected/tuplesort.out
index 3fc1998bf2..418f296a3f 100644
--- a/src/test/regress/expected/tuplesort.out
+++ b/src/test/regress/expected/tuplesort.out
@@ -1,7 +1,7 @@
-- only use parallelism when explicitly intending to do so
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
diff --git a/src/test/regress/sql/tuplesort.sql b/src/test/regress/sql/tuplesort.sql
index 7d7e02f02a..846484d561 100644
--- a/src/test/regress/sql/tuplesort.sql
+++ b/src/test/regress/sql/tuplesort.sql
@@ -2,7 +2,7 @@
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
--
2.17.0
--lc9FT7cWel8HagAv
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0012-doc-review-piecemeal-construction-of-partitioned-ind.patch"
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH 11/18] duplicate words
@ 2021-01-31 00:10 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Justin Pryzby @ 2021-01-31 00:10 UTC (permalink / raw)
commit 9c4f5192f69ed16c99e0d079f0b5faebd7bad212
Allow pg_rewind to use a standby server as the source system.
commit 4a252996d5fda7662b2afdf329a5c95be0fe3b01
Add tests for tuplesort.c.
commit 0a2bc5d61e713e3fe72438f020eea5fcc90b0f0b
Move per-agg and per-trans duplicate finding to the planner.
commit 623a9ba79bbdd11c5eccb30b8bd5c446130e521c
snapshot scalability: cache snapshots using a xact completion counter.
commit 2c03216d831160bedd72d45f712601b6f7d03f1c
Revamp the WAL record format.
---
src/backend/access/transam/xlogutils.c | 3 +--
src/backend/optimizer/prep/prepagg.c | 2 +-
src/backend/storage/ipc/procarray.c | 2 +-
src/bin/pg_rewind/libpq_source.c | 2 +-
src/test/regress/expected/tuplesort.out | 2 +-
src/test/regress/sql/tuplesort.sql | 2 +-
6 files changed, 6 insertions(+), 7 deletions(-)
diff --git a/src/backend/access/transam/xlogutils.c b/src/backend/access/transam/xlogutils.c
index e723253297..25d6df1659 100644
--- a/src/backend/access/transam/xlogutils.c
+++ b/src/backend/access/transam/xlogutils.c
@@ -433,8 +433,7 @@ XLogReadBufferForRedoExtended(XLogReaderState *record,
* NB: A redo function should normally not call this directly. To get a page
* to modify, use XLogReadBufferForRedoExtended instead. It is important that
* all pages modified by a WAL record are registered in the WAL records, or
- * they will be invisible to tools that that need to know which pages are
- * modified.
+ * they will be invisible to tools that need to know which pages are modified.
*/
Buffer
XLogReadBufferExtended(RelFileNode rnode, ForkNumber forknum,
diff --git a/src/backend/optimizer/prep/prepagg.c b/src/backend/optimizer/prep/prepagg.c
index 929a8ea13b..89046f9afb 100644
--- a/src/backend/optimizer/prep/prepagg.c
+++ b/src/backend/optimizer/prep/prepagg.c
@@ -71,7 +71,7 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
*
* Information about the aggregates and transition functions are collected
* in the root->agginfos and root->aggtransinfos lists. The 'aggtranstype',
- * 'aggno', and 'aggtransno' fields in are filled in in each Aggref.
+ * 'aggno', and 'aggtransno' fields of each Aggref are filled in.
*
* NOTE: This modifies the Aggrefs in the input expression in-place!
*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index cf12eda504..b9fbdcb88f 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -2049,7 +2049,7 @@ GetSnapshotDataReuse(Snapshot snapshot)
* holding ProcArrayLock) exclusively). Thus the xactCompletionCount check
* ensures we would detect if the snapshot would have changed.
*
- * As the snapshot contents are the same as it was before, it is is safe
+ * As the snapshot contents are the same as it was before, it is safe
* to re-enter the snapshot's xmin into the PGPROC array. None of the rows
* visible under the snapshot could already have been removed (that'd
* require the set of running transactions to change) and it fulfills the
diff --git a/src/bin/pg_rewind/libpq_source.c b/src/bin/pg_rewind/libpq_source.c
index 86d2adcaee..ac794cf4eb 100644
--- a/src/bin/pg_rewind/libpq_source.c
+++ b/src/bin/pg_rewind/libpq_source.c
@@ -539,7 +539,7 @@ process_queued_fetch_requests(libpq_source *src)
chunkoff, rq->path, (int64) rq->offset);
/*
- * We should not receive receive more data than we requested, or
+ * We should not receive more data than we requested, or
* pg_read_binary_file() messed up. We could receive less,
* though, if the file was truncated in the source after we
* checked its size. That's OK, there should be a WAL record of
diff --git a/src/test/regress/expected/tuplesort.out b/src/test/regress/expected/tuplesort.out
index 3fc1998bf2..418f296a3f 100644
--- a/src/test/regress/expected/tuplesort.out
+++ b/src/test/regress/expected/tuplesort.out
@@ -1,7 +1,7 @@
-- only use parallelism when explicitly intending to do so
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
diff --git a/src/test/regress/sql/tuplesort.sql b/src/test/regress/sql/tuplesort.sql
index 7d7e02f02a..846484d561 100644
--- a/src/test/regress/sql/tuplesort.sql
+++ b/src/test/regress/sql/tuplesort.sql
@@ -2,7 +2,7 @@
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
--
2.17.0
--lc9FT7cWel8HagAv
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0012-doc-review-piecemeal-construction-of-partitioned-ind.patch"
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH 11/18] duplicate words
@ 2021-01-31 00:10 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Justin Pryzby @ 2021-01-31 00:10 UTC (permalink / raw)
commit 9c4f5192f69ed16c99e0d079f0b5faebd7bad212
Allow pg_rewind to use a standby server as the source system.
commit 4a252996d5fda7662b2afdf329a5c95be0fe3b01
Add tests for tuplesort.c.
commit 0a2bc5d61e713e3fe72438f020eea5fcc90b0f0b
Move per-agg and per-trans duplicate finding to the planner.
commit 623a9ba79bbdd11c5eccb30b8bd5c446130e521c
snapshot scalability: cache snapshots using a xact completion counter.
commit 2c03216d831160bedd72d45f712601b6f7d03f1c
Revamp the WAL record format.
---
src/backend/access/transam/xlogutils.c | 3 +--
src/backend/optimizer/prep/prepagg.c | 2 +-
src/backend/storage/ipc/procarray.c | 2 +-
src/bin/pg_rewind/libpq_source.c | 2 +-
src/test/regress/expected/tuplesort.out | 2 +-
src/test/regress/sql/tuplesort.sql | 2 +-
6 files changed, 6 insertions(+), 7 deletions(-)
diff --git a/src/backend/access/transam/xlogutils.c b/src/backend/access/transam/xlogutils.c
index e723253297..25d6df1659 100644
--- a/src/backend/access/transam/xlogutils.c
+++ b/src/backend/access/transam/xlogutils.c
@@ -433,8 +433,7 @@ XLogReadBufferForRedoExtended(XLogReaderState *record,
* NB: A redo function should normally not call this directly. To get a page
* to modify, use XLogReadBufferForRedoExtended instead. It is important that
* all pages modified by a WAL record are registered in the WAL records, or
- * they will be invisible to tools that that need to know which pages are
- * modified.
+ * they will be invisible to tools that need to know which pages are modified.
*/
Buffer
XLogReadBufferExtended(RelFileNode rnode, ForkNumber forknum,
diff --git a/src/backend/optimizer/prep/prepagg.c b/src/backend/optimizer/prep/prepagg.c
index 929a8ea13b..89046f9afb 100644
--- a/src/backend/optimizer/prep/prepagg.c
+++ b/src/backend/optimizer/prep/prepagg.c
@@ -71,7 +71,7 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
*
* Information about the aggregates and transition functions are collected
* in the root->agginfos and root->aggtransinfos lists. The 'aggtranstype',
- * 'aggno', and 'aggtransno' fields in are filled in in each Aggref.
+ * 'aggno', and 'aggtransno' fields of each Aggref are filled in.
*
* NOTE: This modifies the Aggrefs in the input expression in-place!
*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index cf12eda504..b9fbdcb88f 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -2049,7 +2049,7 @@ GetSnapshotDataReuse(Snapshot snapshot)
* holding ProcArrayLock) exclusively). Thus the xactCompletionCount check
* ensures we would detect if the snapshot would have changed.
*
- * As the snapshot contents are the same as it was before, it is is safe
+ * As the snapshot contents are the same as it was before, it is safe
* to re-enter the snapshot's xmin into the PGPROC array. None of the rows
* visible under the snapshot could already have been removed (that'd
* require the set of running transactions to change) and it fulfills the
diff --git a/src/bin/pg_rewind/libpq_source.c b/src/bin/pg_rewind/libpq_source.c
index 86d2adcaee..ac794cf4eb 100644
--- a/src/bin/pg_rewind/libpq_source.c
+++ b/src/bin/pg_rewind/libpq_source.c
@@ -539,7 +539,7 @@ process_queued_fetch_requests(libpq_source *src)
chunkoff, rq->path, (int64) rq->offset);
/*
- * We should not receive receive more data than we requested, or
+ * We should not receive more data than we requested, or
* pg_read_binary_file() messed up. We could receive less,
* though, if the file was truncated in the source after we
* checked its size. That's OK, there should be a WAL record of
diff --git a/src/test/regress/expected/tuplesort.out b/src/test/regress/expected/tuplesort.out
index 3fc1998bf2..418f296a3f 100644
--- a/src/test/regress/expected/tuplesort.out
+++ b/src/test/regress/expected/tuplesort.out
@@ -1,7 +1,7 @@
-- only use parallelism when explicitly intending to do so
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
diff --git a/src/test/regress/sql/tuplesort.sql b/src/test/regress/sql/tuplesort.sql
index 7d7e02f02a..846484d561 100644
--- a/src/test/regress/sql/tuplesort.sql
+++ b/src/test/regress/sql/tuplesort.sql
@@ -2,7 +2,7 @@
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
--
2.17.0
--lc9FT7cWel8HagAv
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0012-doc-review-piecemeal-construction-of-partitioned-ind.patch"
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH 11/18] duplicate words
@ 2021-01-31 00:10 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Justin Pryzby @ 2021-01-31 00:10 UTC (permalink / raw)
commit 9c4f5192f69ed16c99e0d079f0b5faebd7bad212
Allow pg_rewind to use a standby server as the source system.
commit 4a252996d5fda7662b2afdf329a5c95be0fe3b01
Add tests for tuplesort.c.
commit 0a2bc5d61e713e3fe72438f020eea5fcc90b0f0b
Move per-agg and per-trans duplicate finding to the planner.
commit 623a9ba79bbdd11c5eccb30b8bd5c446130e521c
snapshot scalability: cache snapshots using a xact completion counter.
commit 2c03216d831160bedd72d45f712601b6f7d03f1c
Revamp the WAL record format.
---
src/backend/access/transam/xlogutils.c | 3 +--
src/backend/optimizer/prep/prepagg.c | 2 +-
src/backend/storage/ipc/procarray.c | 2 +-
src/bin/pg_rewind/libpq_source.c | 2 +-
src/test/regress/expected/tuplesort.out | 2 +-
src/test/regress/sql/tuplesort.sql | 2 +-
6 files changed, 6 insertions(+), 7 deletions(-)
diff --git a/src/backend/access/transam/xlogutils.c b/src/backend/access/transam/xlogutils.c
index e723253297..25d6df1659 100644
--- a/src/backend/access/transam/xlogutils.c
+++ b/src/backend/access/transam/xlogutils.c
@@ -433,8 +433,7 @@ XLogReadBufferForRedoExtended(XLogReaderState *record,
* NB: A redo function should normally not call this directly. To get a page
* to modify, use XLogReadBufferForRedoExtended instead. It is important that
* all pages modified by a WAL record are registered in the WAL records, or
- * they will be invisible to tools that that need to know which pages are
- * modified.
+ * they will be invisible to tools that need to know which pages are modified.
*/
Buffer
XLogReadBufferExtended(RelFileNode rnode, ForkNumber forknum,
diff --git a/src/backend/optimizer/prep/prepagg.c b/src/backend/optimizer/prep/prepagg.c
index 929a8ea13b..89046f9afb 100644
--- a/src/backend/optimizer/prep/prepagg.c
+++ b/src/backend/optimizer/prep/prepagg.c
@@ -71,7 +71,7 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
*
* Information about the aggregates and transition functions are collected
* in the root->agginfos and root->aggtransinfos lists. The 'aggtranstype',
- * 'aggno', and 'aggtransno' fields in are filled in in each Aggref.
+ * 'aggno', and 'aggtransno' fields of each Aggref are filled in.
*
* NOTE: This modifies the Aggrefs in the input expression in-place!
*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index cf12eda504..b9fbdcb88f 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -2049,7 +2049,7 @@ GetSnapshotDataReuse(Snapshot snapshot)
* holding ProcArrayLock) exclusively). Thus the xactCompletionCount check
* ensures we would detect if the snapshot would have changed.
*
- * As the snapshot contents are the same as it was before, it is is safe
+ * As the snapshot contents are the same as it was before, it is safe
* to re-enter the snapshot's xmin into the PGPROC array. None of the rows
* visible under the snapshot could already have been removed (that'd
* require the set of running transactions to change) and it fulfills the
diff --git a/src/bin/pg_rewind/libpq_source.c b/src/bin/pg_rewind/libpq_source.c
index 86d2adcaee..ac794cf4eb 100644
--- a/src/bin/pg_rewind/libpq_source.c
+++ b/src/bin/pg_rewind/libpq_source.c
@@ -539,7 +539,7 @@ process_queued_fetch_requests(libpq_source *src)
chunkoff, rq->path, (int64) rq->offset);
/*
- * We should not receive receive more data than we requested, or
+ * We should not receive more data than we requested, or
* pg_read_binary_file() messed up. We could receive less,
* though, if the file was truncated in the source after we
* checked its size. That's OK, there should be a WAL record of
diff --git a/src/test/regress/expected/tuplesort.out b/src/test/regress/expected/tuplesort.out
index 3fc1998bf2..418f296a3f 100644
--- a/src/test/regress/expected/tuplesort.out
+++ b/src/test/regress/expected/tuplesort.out
@@ -1,7 +1,7 @@
-- only use parallelism when explicitly intending to do so
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
diff --git a/src/test/regress/sql/tuplesort.sql b/src/test/regress/sql/tuplesort.sql
index 7d7e02f02a..846484d561 100644
--- a/src/test/regress/sql/tuplesort.sql
+++ b/src/test/regress/sql/tuplesort.sql
@@ -2,7 +2,7 @@
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
--
2.17.0
--lc9FT7cWel8HagAv
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0012-doc-review-piecemeal-construction-of-partitioned-ind.patch"
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH 11/18] duplicate words
@ 2021-01-31 00:10 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Justin Pryzby @ 2021-01-31 00:10 UTC (permalink / raw)
commit 9c4f5192f69ed16c99e0d079f0b5faebd7bad212
Allow pg_rewind to use a standby server as the source system.
commit 4a252996d5fda7662b2afdf329a5c95be0fe3b01
Add tests for tuplesort.c.
commit 0a2bc5d61e713e3fe72438f020eea5fcc90b0f0b
Move per-agg and per-trans duplicate finding to the planner.
commit 623a9ba79bbdd11c5eccb30b8bd5c446130e521c
snapshot scalability: cache snapshots using a xact completion counter.
commit 2c03216d831160bedd72d45f712601b6f7d03f1c
Revamp the WAL record format.
---
src/backend/access/transam/xlogutils.c | 3 +--
src/backend/optimizer/prep/prepagg.c | 2 +-
src/backend/storage/ipc/procarray.c | 2 +-
src/bin/pg_rewind/libpq_source.c | 2 +-
src/test/regress/expected/tuplesort.out | 2 +-
src/test/regress/sql/tuplesort.sql | 2 +-
6 files changed, 6 insertions(+), 7 deletions(-)
diff --git a/src/backend/access/transam/xlogutils.c b/src/backend/access/transam/xlogutils.c
index e723253297..25d6df1659 100644
--- a/src/backend/access/transam/xlogutils.c
+++ b/src/backend/access/transam/xlogutils.c
@@ -433,8 +433,7 @@ XLogReadBufferForRedoExtended(XLogReaderState *record,
* NB: A redo function should normally not call this directly. To get a page
* to modify, use XLogReadBufferForRedoExtended instead. It is important that
* all pages modified by a WAL record are registered in the WAL records, or
- * they will be invisible to tools that that need to know which pages are
- * modified.
+ * they will be invisible to tools that need to know which pages are modified.
*/
Buffer
XLogReadBufferExtended(RelFileNode rnode, ForkNumber forknum,
diff --git a/src/backend/optimizer/prep/prepagg.c b/src/backend/optimizer/prep/prepagg.c
index 929a8ea13b..89046f9afb 100644
--- a/src/backend/optimizer/prep/prepagg.c
+++ b/src/backend/optimizer/prep/prepagg.c
@@ -71,7 +71,7 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
*
* Information about the aggregates and transition functions are collected
* in the root->agginfos and root->aggtransinfos lists. The 'aggtranstype',
- * 'aggno', and 'aggtransno' fields in are filled in in each Aggref.
+ * 'aggno', and 'aggtransno' fields of each Aggref are filled in.
*
* NOTE: This modifies the Aggrefs in the input expression in-place!
*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index cf12eda504..b9fbdcb88f 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -2049,7 +2049,7 @@ GetSnapshotDataReuse(Snapshot snapshot)
* holding ProcArrayLock) exclusively). Thus the xactCompletionCount check
* ensures we would detect if the snapshot would have changed.
*
- * As the snapshot contents are the same as it was before, it is is safe
+ * As the snapshot contents are the same as it was before, it is safe
* to re-enter the snapshot's xmin into the PGPROC array. None of the rows
* visible under the snapshot could already have been removed (that'd
* require the set of running transactions to change) and it fulfills the
diff --git a/src/bin/pg_rewind/libpq_source.c b/src/bin/pg_rewind/libpq_source.c
index 86d2adcaee..ac794cf4eb 100644
--- a/src/bin/pg_rewind/libpq_source.c
+++ b/src/bin/pg_rewind/libpq_source.c
@@ -539,7 +539,7 @@ process_queued_fetch_requests(libpq_source *src)
chunkoff, rq->path, (int64) rq->offset);
/*
- * We should not receive receive more data than we requested, or
+ * We should not receive more data than we requested, or
* pg_read_binary_file() messed up. We could receive less,
* though, if the file was truncated in the source after we
* checked its size. That's OK, there should be a WAL record of
diff --git a/src/test/regress/expected/tuplesort.out b/src/test/regress/expected/tuplesort.out
index 3fc1998bf2..418f296a3f 100644
--- a/src/test/regress/expected/tuplesort.out
+++ b/src/test/regress/expected/tuplesort.out
@@ -1,7 +1,7 @@
-- only use parallelism when explicitly intending to do so
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
diff --git a/src/test/regress/sql/tuplesort.sql b/src/test/regress/sql/tuplesort.sql
index 7d7e02f02a..846484d561 100644
--- a/src/test/regress/sql/tuplesort.sql
+++ b/src/test/regress/sql/tuplesort.sql
@@ -2,7 +2,7 @@
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
--
2.17.0
--lc9FT7cWel8HagAv
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0012-doc-review-piecemeal-construction-of-partitioned-ind.patch"
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH 11/18] duplicate words
@ 2021-01-31 00:10 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Justin Pryzby @ 2021-01-31 00:10 UTC (permalink / raw)
commit 9c4f5192f69ed16c99e0d079f0b5faebd7bad212
Allow pg_rewind to use a standby server as the source system.
commit 4a252996d5fda7662b2afdf329a5c95be0fe3b01
Add tests for tuplesort.c.
commit 0a2bc5d61e713e3fe72438f020eea5fcc90b0f0b
Move per-agg and per-trans duplicate finding to the planner.
commit 623a9ba79bbdd11c5eccb30b8bd5c446130e521c
snapshot scalability: cache snapshots using a xact completion counter.
commit 2c03216d831160bedd72d45f712601b6f7d03f1c
Revamp the WAL record format.
---
src/backend/access/transam/xlogutils.c | 3 +--
src/backend/optimizer/prep/prepagg.c | 2 +-
src/backend/storage/ipc/procarray.c | 2 +-
src/bin/pg_rewind/libpq_source.c | 2 +-
src/test/regress/expected/tuplesort.out | 2 +-
src/test/regress/sql/tuplesort.sql | 2 +-
6 files changed, 6 insertions(+), 7 deletions(-)
diff --git a/src/backend/access/transam/xlogutils.c b/src/backend/access/transam/xlogutils.c
index e723253297..25d6df1659 100644
--- a/src/backend/access/transam/xlogutils.c
+++ b/src/backend/access/transam/xlogutils.c
@@ -433,8 +433,7 @@ XLogReadBufferForRedoExtended(XLogReaderState *record,
* NB: A redo function should normally not call this directly. To get a page
* to modify, use XLogReadBufferForRedoExtended instead. It is important that
* all pages modified by a WAL record are registered in the WAL records, or
- * they will be invisible to tools that that need to know which pages are
- * modified.
+ * they will be invisible to tools that need to know which pages are modified.
*/
Buffer
XLogReadBufferExtended(RelFileNode rnode, ForkNumber forknum,
diff --git a/src/backend/optimizer/prep/prepagg.c b/src/backend/optimizer/prep/prepagg.c
index 929a8ea13b..89046f9afb 100644
--- a/src/backend/optimizer/prep/prepagg.c
+++ b/src/backend/optimizer/prep/prepagg.c
@@ -71,7 +71,7 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
*
* Information about the aggregates and transition functions are collected
* in the root->agginfos and root->aggtransinfos lists. The 'aggtranstype',
- * 'aggno', and 'aggtransno' fields in are filled in in each Aggref.
+ * 'aggno', and 'aggtransno' fields of each Aggref are filled in.
*
* NOTE: This modifies the Aggrefs in the input expression in-place!
*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index cf12eda504..b9fbdcb88f 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -2049,7 +2049,7 @@ GetSnapshotDataReuse(Snapshot snapshot)
* holding ProcArrayLock) exclusively). Thus the xactCompletionCount check
* ensures we would detect if the snapshot would have changed.
*
- * As the snapshot contents are the same as it was before, it is is safe
+ * As the snapshot contents are the same as it was before, it is safe
* to re-enter the snapshot's xmin into the PGPROC array. None of the rows
* visible under the snapshot could already have been removed (that'd
* require the set of running transactions to change) and it fulfills the
diff --git a/src/bin/pg_rewind/libpq_source.c b/src/bin/pg_rewind/libpq_source.c
index 86d2adcaee..ac794cf4eb 100644
--- a/src/bin/pg_rewind/libpq_source.c
+++ b/src/bin/pg_rewind/libpq_source.c
@@ -539,7 +539,7 @@ process_queued_fetch_requests(libpq_source *src)
chunkoff, rq->path, (int64) rq->offset);
/*
- * We should not receive receive more data than we requested, or
+ * We should not receive more data than we requested, or
* pg_read_binary_file() messed up. We could receive less,
* though, if the file was truncated in the source after we
* checked its size. That's OK, there should be a WAL record of
diff --git a/src/test/regress/expected/tuplesort.out b/src/test/regress/expected/tuplesort.out
index 3fc1998bf2..418f296a3f 100644
--- a/src/test/regress/expected/tuplesort.out
+++ b/src/test/regress/expected/tuplesort.out
@@ -1,7 +1,7 @@
-- only use parallelism when explicitly intending to do so
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
diff --git a/src/test/regress/sql/tuplesort.sql b/src/test/regress/sql/tuplesort.sql
index 7d7e02f02a..846484d561 100644
--- a/src/test/regress/sql/tuplesort.sql
+++ b/src/test/regress/sql/tuplesort.sql
@@ -2,7 +2,7 @@
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
--
2.17.0
--lc9FT7cWel8HagAv
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0012-doc-review-piecemeal-construction-of-partitioned-ind.patch"
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH 11/18] duplicate words
@ 2021-01-31 00:10 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Justin Pryzby @ 2021-01-31 00:10 UTC (permalink / raw)
commit 9c4f5192f69ed16c99e0d079f0b5faebd7bad212
Allow pg_rewind to use a standby server as the source system.
commit 4a252996d5fda7662b2afdf329a5c95be0fe3b01
Add tests for tuplesort.c.
commit 0a2bc5d61e713e3fe72438f020eea5fcc90b0f0b
Move per-agg and per-trans duplicate finding to the planner.
commit 623a9ba79bbdd11c5eccb30b8bd5c446130e521c
snapshot scalability: cache snapshots using a xact completion counter.
commit 2c03216d831160bedd72d45f712601b6f7d03f1c
Revamp the WAL record format.
---
src/backend/access/transam/xlogutils.c | 3 +--
src/backend/optimizer/prep/prepagg.c | 2 +-
src/backend/storage/ipc/procarray.c | 2 +-
src/bin/pg_rewind/libpq_source.c | 2 +-
src/test/regress/expected/tuplesort.out | 2 +-
src/test/regress/sql/tuplesort.sql | 2 +-
6 files changed, 6 insertions(+), 7 deletions(-)
diff --git a/src/backend/access/transam/xlogutils.c b/src/backend/access/transam/xlogutils.c
index e723253297..25d6df1659 100644
--- a/src/backend/access/transam/xlogutils.c
+++ b/src/backend/access/transam/xlogutils.c
@@ -433,8 +433,7 @@ XLogReadBufferForRedoExtended(XLogReaderState *record,
* NB: A redo function should normally not call this directly. To get a page
* to modify, use XLogReadBufferForRedoExtended instead. It is important that
* all pages modified by a WAL record are registered in the WAL records, or
- * they will be invisible to tools that that need to know which pages are
- * modified.
+ * they will be invisible to tools that need to know which pages are modified.
*/
Buffer
XLogReadBufferExtended(RelFileNode rnode, ForkNumber forknum,
diff --git a/src/backend/optimizer/prep/prepagg.c b/src/backend/optimizer/prep/prepagg.c
index 929a8ea13b..89046f9afb 100644
--- a/src/backend/optimizer/prep/prepagg.c
+++ b/src/backend/optimizer/prep/prepagg.c
@@ -71,7 +71,7 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
*
* Information about the aggregates and transition functions are collected
* in the root->agginfos and root->aggtransinfos lists. The 'aggtranstype',
- * 'aggno', and 'aggtransno' fields in are filled in in each Aggref.
+ * 'aggno', and 'aggtransno' fields of each Aggref are filled in.
*
* NOTE: This modifies the Aggrefs in the input expression in-place!
*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index cf12eda504..b9fbdcb88f 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -2049,7 +2049,7 @@ GetSnapshotDataReuse(Snapshot snapshot)
* holding ProcArrayLock) exclusively). Thus the xactCompletionCount check
* ensures we would detect if the snapshot would have changed.
*
- * As the snapshot contents are the same as it was before, it is is safe
+ * As the snapshot contents are the same as it was before, it is safe
* to re-enter the snapshot's xmin into the PGPROC array. None of the rows
* visible under the snapshot could already have been removed (that'd
* require the set of running transactions to change) and it fulfills the
diff --git a/src/bin/pg_rewind/libpq_source.c b/src/bin/pg_rewind/libpq_source.c
index 86d2adcaee..ac794cf4eb 100644
--- a/src/bin/pg_rewind/libpq_source.c
+++ b/src/bin/pg_rewind/libpq_source.c
@@ -539,7 +539,7 @@ process_queued_fetch_requests(libpq_source *src)
chunkoff, rq->path, (int64) rq->offset);
/*
- * We should not receive receive more data than we requested, or
+ * We should not receive more data than we requested, or
* pg_read_binary_file() messed up. We could receive less,
* though, if the file was truncated in the source after we
* checked its size. That's OK, there should be a WAL record of
diff --git a/src/test/regress/expected/tuplesort.out b/src/test/regress/expected/tuplesort.out
index 3fc1998bf2..418f296a3f 100644
--- a/src/test/regress/expected/tuplesort.out
+++ b/src/test/regress/expected/tuplesort.out
@@ -1,7 +1,7 @@
-- only use parallelism when explicitly intending to do so
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
diff --git a/src/test/regress/sql/tuplesort.sql b/src/test/regress/sql/tuplesort.sql
index 7d7e02f02a..846484d561 100644
--- a/src/test/regress/sql/tuplesort.sql
+++ b/src/test/regress/sql/tuplesort.sql
@@ -2,7 +2,7 @@
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
--
2.17.0
--lc9FT7cWel8HagAv
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0012-doc-review-piecemeal-construction-of-partitioned-ind.patch"
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH 11/21] duplicate words
@ 2021-01-31 00:10 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Justin Pryzby @ 2021-01-31 00:10 UTC (permalink / raw)
commit 9c4f5192f69ed16c99e0d079f0b5faebd7bad212
Allow pg_rewind to use a standby server as the source system.
commit 4a252996d5fda7662b2afdf329a5c95be0fe3b01
Add tests for tuplesort.c.
commit 0a2bc5d61e713e3fe72438f020eea5fcc90b0f0b
Move per-agg and per-trans duplicate finding to the planner.
commit 623a9ba79bbdd11c5eccb30b8bd5c446130e521c
snapshot scalability: cache snapshots using a xact completion counter.
commit 2c03216d831160bedd72d45f712601b6f7d03f1c
Revamp the WAL record format.
---
src/backend/access/transam/xlogutils.c | 3 +--
src/backend/optimizer/prep/prepagg.c | 2 +-
src/backend/storage/ipc/procarray.c | 2 +-
src/bin/pg_rewind/libpq_source.c | 2 +-
src/test/regress/expected/tuplesort.out | 2 +-
src/test/regress/sql/tuplesort.sql | 2 +-
6 files changed, 6 insertions(+), 7 deletions(-)
diff --git a/src/backend/access/transam/xlogutils.c b/src/backend/access/transam/xlogutils.c
index e723253297..25d6df1659 100644
--- a/src/backend/access/transam/xlogutils.c
+++ b/src/backend/access/transam/xlogutils.c
@@ -433,8 +433,7 @@ XLogReadBufferForRedoExtended(XLogReaderState *record,
* NB: A redo function should normally not call this directly. To get a page
* to modify, use XLogReadBufferForRedoExtended instead. It is important that
* all pages modified by a WAL record are registered in the WAL records, or
- * they will be invisible to tools that that need to know which pages are
- * modified.
+ * they will be invisible to tools that need to know which pages are modified.
*/
Buffer
XLogReadBufferExtended(RelFileNode rnode, ForkNumber forknum,
diff --git a/src/backend/optimizer/prep/prepagg.c b/src/backend/optimizer/prep/prepagg.c
index 929a8ea13b..89046f9afb 100644
--- a/src/backend/optimizer/prep/prepagg.c
+++ b/src/backend/optimizer/prep/prepagg.c
@@ -71,7 +71,7 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
*
* Information about the aggregates and transition functions are collected
* in the root->agginfos and root->aggtransinfos lists. The 'aggtranstype',
- * 'aggno', and 'aggtransno' fields in are filled in in each Aggref.
+ * 'aggno', and 'aggtransno' fields of each Aggref are filled in.
*
* NOTE: This modifies the Aggrefs in the input expression in-place!
*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 4085891237..c1e86fb35c 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -2049,7 +2049,7 @@ GetSnapshotDataReuse(Snapshot snapshot)
* holding ProcArrayLock) exclusively). Thus the xactCompletionCount check
* ensures we would detect if the snapshot would have changed.
*
- * As the snapshot contents are the same as it was before, it is is safe
+ * As the snapshot contents are the same as it was before, it is safe
* to re-enter the snapshot's xmin into the PGPROC array. None of the rows
* visible under the snapshot could already have been removed (that'd
* require the set of running transactions to change) and it fulfills the
diff --git a/src/bin/pg_rewind/libpq_source.c b/src/bin/pg_rewind/libpq_source.c
index 86d2adcaee..ac794cf4eb 100644
--- a/src/bin/pg_rewind/libpq_source.c
+++ b/src/bin/pg_rewind/libpq_source.c
@@ -539,7 +539,7 @@ process_queued_fetch_requests(libpq_source *src)
chunkoff, rq->path, (int64) rq->offset);
/*
- * We should not receive receive more data than we requested, or
+ * We should not receive more data than we requested, or
* pg_read_binary_file() messed up. We could receive less,
* though, if the file was truncated in the source after we
* checked its size. That's OK, there should be a WAL record of
diff --git a/src/test/regress/expected/tuplesort.out b/src/test/regress/expected/tuplesort.out
index 3fc1998bf2..418f296a3f 100644
--- a/src/test/regress/expected/tuplesort.out
+++ b/src/test/regress/expected/tuplesort.out
@@ -1,7 +1,7 @@
-- only use parallelism when explicitly intending to do so
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
diff --git a/src/test/regress/sql/tuplesort.sql b/src/test/regress/sql/tuplesort.sql
index 7d7e02f02a..846484d561 100644
--- a/src/test/regress/sql/tuplesort.sql
+++ b/src/test/regress/sql/tuplesort.sql
@@ -2,7 +2,7 @@
SET max_parallel_maintenance_workers = 0;
SET max_parallel_workers = 0;
--- A table with with contents that, when sorted, triggers abbreviated
+-- A table with contents that, when sorted, triggers abbreviated
-- key aborts. One easy way to achieve that is to use uuids that all
-- have the same prefix, as abbreviated keys for uuids just use the
-- first sizeof(Datum) bytes.
--
2.17.0
--jI8keyz6grp/JLjh
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0012-doc-review-piecemeal-construction-of-partitioned-ind.patch"
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH 32/32] duplicate words
@ 2021-04-04 18:36 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Justin Pryzby @ 2021-04-04 18:36 UTC (permalink / raw)
0a1f1d3cac6baaa3744b89336673caba702f7628
---
src/include/lib/sort_template.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/include/lib/sort_template.h b/src/include/lib/sort_template.h
index 771c789ced..24d6d0006c 100644
--- a/src/include/lib/sort_template.h
+++ b/src/include/lib/sort_template.h
@@ -241,7 +241,7 @@ ST_SCOPE void ST_SORT(ST_ELEMENT_TYPE *first, size_t n
/*
* Find the median of three values. Currently, performance seems to be best
- * if the the comparator is inlined here, but the med3 function is not inlined
+ * if the comparator is inlined here, but the med3 function is not inlined
* in the qsort function.
*/
static pg_noinline ST_ELEMENT_TYPE *
--
2.17.0
--4eRLI4hEmsdu6Npr--
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH 14/19] duplicate words
@ 2022-04-06 12:24 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Justin Pryzby @ 2022-04-06 12:24 UTC (permalink / raw)
3f1ce973467a0d285961bf2f99b11d06e264e2c1n 3500ccc39b0dadd1068a03938e4b8ff562587ccc
---
src/backend/access/transam/xlogreader.c | 2 +-
src/backend/replication/basebackup_server.c | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index a5f1a648d3d..b3e37003ac5 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -300,7 +300,7 @@ XLogReleasePreviousRecord(XLogReaderState *state)
/* Release the space. */
if (unlikely(record->oversized))
{
- /* It's not in the the decode buffer, so free it to release space. */
+ /* It's not in the decode buffer, so free it to release space. */
pfree(record);
}
else
diff --git a/src/backend/replication/basebackup_server.c b/src/backend/replication/basebackup_server.c
index bc16897b33f..33595616894 100644
--- a/src/backend/replication/basebackup_server.c
+++ b/src/backend/replication/basebackup_server.c
@@ -195,7 +195,7 @@ bbsink_server_end_archive(bbsink *sink)
/*
* We intentionally don't use data_sync_elevel here, because the server
- * shouldn't PANIC just because we can't guarantee the the backup has been
+ * shouldn't PANIC just because we can't guarantee the backup has been
* written down to disk. Running recovery won't fix anything in this case
* anyway.
*/
--
2.17.1
--8X7/QrJGcKSMr1RN
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment; filename="0015-f-relpersistence.patch"
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH 04/10] duplicate words
@ 2023-01-18 04:28 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Justin Pryzby @ 2023-01-18 04:28 UTC (permalink / raw)
---
contrib/postgres_fdw/deparse.c | 2 +-
doc/src/sgml/ref/create_database.sgml | 2 +-
doc/src/sgml/ref/create_schema.sgml | 2 +-
src/include/partitioning/partdesc.h | 2 +-
src/include/port/simd.h | 2 +-
5 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/contrib/postgres_fdw/deparse.c b/contrib/postgres_fdw/deparse.c
index 8a847deb13b..09d6dd60ddc 100644
--- a/contrib/postgres_fdw/deparse.c
+++ b/contrib/postgres_fdw/deparse.c
@@ -2412,7 +2412,7 @@ deparseAnalyzeInfoSql(StringInfo buf, Relation rel)
*
* We could also do "ORDER BY random() LIMIT x", which would always pick
* the expected number of rows, but it requires sorting so it may be much
- * more expensive (particularly on large tables, which is what what the
+ * more expensive (particularly on large tables, which is what the
* remote sampling is meant to improve).
*/
void
diff --git a/doc/src/sgml/ref/create_database.sgml b/doc/src/sgml/ref/create_database.sgml
index f3df2def864..91c39c52303 100644
--- a/doc/src/sgml/ref/create_database.sgml
+++ b/doc/src/sgml/ref/create_database.sgml
@@ -89,7 +89,7 @@ CREATE DATABASE <replaceable class="parameter">name</replaceable>
The role name of the user who will own the new database,
or <literal>DEFAULT</literal> to use the default (namely, the
user executing the command). To create a database owned by another
- role, you must must be able to <literal>SET ROLE</literal> to that
+ role, you must be able to <literal>SET ROLE</literal> to that
role.
</para>
</listitem>
diff --git a/doc/src/sgml/ref/create_schema.sgml b/doc/src/sgml/ref/create_schema.sgml
index 04b0c28731e..ed69298ccc6 100644
--- a/doc/src/sgml/ref/create_schema.sgml
+++ b/doc/src/sgml/ref/create_schema.sgml
@@ -89,7 +89,7 @@ CREATE SCHEMA IF NOT EXISTS AUTHORIZATION <replaceable class="parameter">role_sp
<para>
The role name of the user who will own the new schema. If omitted,
defaults to the user executing the command. To create a schema
- owned by another role, you must must be able to
+ owned by another role, you must be able to
<literal>SET ROLE</literal> to that role.
</para>
</listitem>
diff --git a/src/include/partitioning/partdesc.h b/src/include/partitioning/partdesc.h
index 7d71fb6e931..e157eae9c1e 100644
--- a/src/include/partitioning/partdesc.h
+++ b/src/include/partitioning/partdesc.h
@@ -31,7 +31,7 @@ typedef struct PartitionDescData
int nparts; /* Number of partitions */
bool detached_exist; /* Are there any detached partitions? */
Oid *oids; /* Array of 'nparts' elements containing
- * partition OIDs in order of the their bounds */
+ * partition OIDs in order of their bounds */
bool *is_leaf; /* Array of 'nparts' elements storing whether
* the corresponding 'oids' element belongs to
* a leaf partition or not */
diff --git a/src/include/port/simd.h b/src/include/port/simd.h
index c836360d4b7..1fa6c3bc6c4 100644
--- a/src/include/port/simd.h
+++ b/src/include/port/simd.h
@@ -345,7 +345,7 @@ vector8_ssub(const Vector8 v1, const Vector8 v2)
#endif /* ! USE_NO_SIMD */
/*
- * Return a vector with all bits set in each lane where the the corresponding
+ * Return a vector with all bits set in each lane where the corresponding
* lanes in the inputs are equal.
*/
#ifndef USE_NO_SIMD
--
2.25.1
--kjpMrWxdCilgNbo1
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0005-WIP-avoid-whitespace-preceding-following-else.patch"
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Oversight in reparameterize_path_by_child leading to executor crash
@ 2023-08-23 05:37 Richard Guo <[email protected]>
1 sibling, 3 replies; 86+ messages in thread
From: Richard Guo @ 2023-08-23 05:37 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Ashutosh Bapat <[email protected]>; pgsql-hackers
On Tue, Aug 22, 2023 at 10:39 PM Tom Lane <[email protected]> wrote:
> Richard Guo <[email protected]> writes:
> > I'm wondering if we can instead adjust the 'inner_req_outer' in
> > create_nestloop_path before we perform the check to work around this
> > issue for the back branches, something like
> > + /*
> > + * Adjust the parameterization information, which refers to the
> topmost
> > + * parent.
> > + */
> > + inner_req_outer =
> > + adjust_child_relids_multilevel(root, inner_req_outer,
> > + outer_path->parent->relids,
> > +
> outer_path->parent->top_parent_relids);
>
> Mmm ... at the very least you'd need to not do that when top_parent_relids
> is unset.
Hmm, adjust_child_relids_multilevel would just return the given relids
unchanged when top_parent_relids is unset, so I think it should be safe
to call it even for non-other rel.
> ... But I think the risk/reward ratio for messing with this in the
> back branches is unattractive in any case: to fix a corner case that
> apparently nobody uses in the field, we risk breaking any number of
> mainstream parameterized-path cases. I'm content to commit the v5 patch
> (or a successor) into HEAD, but at this point I'm not sure I even want
> to risk it in v16, let alone perform delicate surgery to get it to work
> in older branches. I think we ought to go with the "tablesample scans
> can't be reparameterized" approach in v16 and before.
If we go with the "tablesample scans can't be reparameterized" approach
in the back branches, I'm a little concerned that what if we find more
cases in the futrue where we need modify RTEs for reparameterization.
So I spent some time seeking and have managed to find one: there might
be lateral references in a scan path's restriction clauses, and
currently reparameterize_path_by_child fails to adjust them.
regression=# explain (verbose, costs off)
select count(*) from prt1 t1 left join lateral
(select t1.b as t1b, t2.* from prt2 t2) s
on t1.a = s.b where s.t1b = s.a;
QUERY PLAN
----------------------------------------------------------------------
Aggregate
Output: count(*)
-> Append
-> Nested Loop
-> Seq Scan on public.prt1_p1 t1_1
Output: t1_1.a, t1_1.b
-> Index Scan using iprt2_p1_b on public.prt2_p1 t2_1
Output: t2_1.b
Index Cond: (t2_1.b = t1_1.a)
Filter: (t1.b = t2_1.a)
-> Nested Loop
-> Seq Scan on public.prt1_p2 t1_2
Output: t1_2.a, t1_2.b
-> Index Scan using iprt2_p2_b on public.prt2_p2 t2_2
Output: t2_2.b
Index Cond: (t2_2.b = t1_2.a)
Filter: (t1.b = t2_2.a)
-> Nested Loop
-> Seq Scan on public.prt1_p3 t1_3
Output: t1_3.a, t1_3.b
-> Index Scan using iprt2_p3_b on public.prt2_p3 t2_3
Output: t2_3.b
Index Cond: (t2_3.b = t1_3.a)
Filter: (t1.b = t2_3.a)
(24 rows)
The clauses in 'Filter' are not adjusted correctly. Var 't1.b' still
refers to the top parent.
For this query it would just give wrong results.
regression=# set enable_partitionwise_join to off;
SET
regression=# select count(*) from prt1 t1 left join lateral
(select t1.b as t1b, t2.* from prt2 t2) s
on t1.a = s.b where s.t1b = s.a;
count
-------
100
(1 row)
regression=# set enable_partitionwise_join to on;
SET
regression=# select count(*) from prt1 t1 left join lateral
(select t1.b as t1b, t2.* from prt2 t2) s
on t1.a = s.b where s.t1b = s.a;
count
-------
5
(1 row)
For another query it would error out during planning.
regression=# explain (verbose, costs off)
select count(*) from prt1 t1 left join lateral
(select t1.b as t1b, t2.* from prt2 t2) s
on t1.a = s.b where s.t1b = s.b;
ERROR: variable not found in subplan target list
To fix it we may need to modify RelOptInfos for Path, BitmapHeapPath,
ForeignPath and CustomPath, and modify IndexOptInfos for IndexPath. It
seems that that is not easily done without postponing reparameterization
of paths until createplan.c.
Attached is a patch which is v5 + fix for this new issue.
Thanks
Richard
Attachments:
[application/octet-stream] v7-0001-Postpone-reparameterization-of-paths-until-when-creating-plans.patch (21.1K, ../../CAMbWs4-CSR4VnZCDep3ReSoHGTA7E+3tnjF_LmHcX7yiGrkVfQ@mail.gmail.com/3-v7-0001-Postpone-reparameterization-of-paths-until-when-creating-plans.patch)
download | inline diff:
From b4963f8e484682065fab96818a9a5247a9f68737 Mon Sep 17 00:00:00 2001
From: Richard Guo <[email protected]>
Date: Wed, 23 Aug 2023 11:02:31 +0800
Subject: [PATCH v7] Postpone reparameterization of paths until when creating
plans
---
src/backend/optimizer/path/joinpath.c | 67 +++----
src/backend/optimizer/plan/createplan.c | 17 ++
src/backend/optimizer/util/pathnode.c | 200 ++++++++++++++++++-
src/include/optimizer/pathnode.h | 2 +
src/test/regress/expected/partition_join.out | 94 +++++++++
src/test/regress/sql/partition_join.sql | 21 ++
6 files changed, 358 insertions(+), 43 deletions(-)
diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c
index 821d282497..e2ecf5b14b 100644
--- a/src/backend/optimizer/path/joinpath.c
+++ b/src/backend/optimizer/path/joinpath.c
@@ -30,8 +30,9 @@
set_join_pathlist_hook_type set_join_pathlist_hook = NULL;
/*
- * Paths parameterized by the parent can be considered to be parameterized by
- * any of its child.
+ * Paths parameterized by a parent rel can be considered to be parameterized
+ * by any of its children, when we are performing partitionwise joins. These
+ * macros simplify checking for such cases. Beware multiple eval of args.
*/
#define PATH_PARAM_BY_PARENT(path, rel) \
((path)->param_info && bms_overlap(PATH_REQ_OUTER(path), \
@@ -762,6 +763,20 @@ try_nestloop_path(PlannerInfo *root,
/* If we got past that, we shouldn't have any unsafe outer-join refs */
Assert(!have_unsafe_outer_join_ref(root, outerrelids, inner_paramrels));
+ /*
+ * If the inner path is parameterized, it is parameterized by the topmost
+ * parent of the outer rel, not the outer rel itself. We will need to
+ * translate the parameterization, if this path is chosen, during
+ * create_plan(). Here we just check whether we will be able to perform
+ * the translation, and if not avoid creating a nestloop path.
+ */
+ if (PATH_PARAM_BY_PARENT(inner_path, outer_path->parent) &&
+ !path_is_reparameterizable_by_child(inner_path, outer_path->parent))
+ {
+ bms_free(required_outer);
+ return;
+ }
+
/*
* Do a precheck to quickly eliminate obviously-inferior paths. We
* calculate a cheap lower bound on the path's cost and then use
@@ -778,27 +793,6 @@ try_nestloop_path(PlannerInfo *root,
workspace.startup_cost, workspace.total_cost,
pathkeys, required_outer))
{
- /*
- * If the inner path is parameterized, it is parameterized by the
- * topmost parent of the outer rel, not the outer rel itself. Fix
- * that.
- */
- if (PATH_PARAM_BY_PARENT(inner_path, outer_path->parent))
- {
- inner_path = reparameterize_path_by_child(root, inner_path,
- outer_path->parent);
-
- /*
- * If we could not translate the path, we can't create nest loop
- * path.
- */
- if (!inner_path)
- {
- bms_free(required_outer);
- return;
- }
- }
-
add_path(joinrel, (Path *)
create_nestloop_path(root,
joinrel,
@@ -861,6 +855,17 @@ try_partial_nestloop_path(PlannerInfo *root,
return;
}
+ /*
+ * If the inner path is parameterized, it is parameterized by the topmost
+ * parent of the outer rel, not the outer rel itself. We will need to
+ * translate the parameterization, if this path is chosen, during
+ * create_plan(). Here we just check whether we will be able to perform
+ * the translation, and if not avoid creating a nestloop path.
+ */
+ if (PATH_PARAM_BY_PARENT(inner_path, outer_path->parent) &&
+ !path_is_reparameterizable_by_child(inner_path, outer_path->parent))
+ return;
+
/*
* Before creating a path, get a quick lower bound on what it is likely to
* cost. Bail out right away if it looks terrible.
@@ -870,22 +875,6 @@ try_partial_nestloop_path(PlannerInfo *root,
if (!add_partial_path_precheck(joinrel, workspace.total_cost, pathkeys))
return;
- /*
- * If the inner path is parameterized, it is parameterized by the topmost
- * parent of the outer rel, not the outer rel itself. Fix that.
- */
- if (PATH_PARAM_BY_PARENT(inner_path, outer_path->parent))
- {
- inner_path = reparameterize_path_by_child(root, inner_path,
- outer_path->parent);
-
- /*
- * If we could not translate the path, we can't create nest loop path.
- */
- if (!inner_path)
- return;
- }
-
/* Might be good enough to be worth trying, so let's try it. */
add_partial_path(joinrel, (Path *)
create_nestloop_path(root,
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 34ca6d4ac2..38bd179f4f 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -29,6 +29,7 @@
#include "optimizer/cost.h"
#include "optimizer/optimizer.h"
#include "optimizer/paramassign.h"
+#include "optimizer/pathnode.h"
#include "optimizer/paths.h"
#include "optimizer/placeholder.h"
#include "optimizer/plancat.h"
@@ -4346,6 +4347,22 @@ create_nestloop_plan(PlannerInfo *root,
List *nestParams;
Relids saveOuterRels = root->curOuterRels;
+ /*
+ * If the inner path is parameterized by the topmost parent of the outer
+ * rel rather than the outer rel itself, fix that. (Nothing happens here
+ * if it is not so parameterized.)
+ */
+ best_path->jpath.innerjoinpath =
+ reparameterize_path_by_child(root,
+ best_path->jpath.innerjoinpath,
+ best_path->jpath.outerjoinpath->parent);
+
+ /*
+ * Failure here probably means that reparameterize_path_by_child() is not
+ * in sync with path_is_reparameterizable_by_child().
+ */
+ Assert(best_path->jpath.innerjoinpath != NULL);
+
/* NestLoop can project, so no need to be picky about child tlists */
outer_plan = create_plan_recurse(root, best_path->jpath.outerjoinpath, 0);
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index 211ba65389..961e33f227 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -56,6 +56,8 @@ static int append_startup_cost_compare(const ListCell *a, const ListCell *b);
static List *reparameterize_pathlist_by_child(PlannerInfo *root,
List *pathlist,
RelOptInfo *child_rel);
+static bool pathlist_is_reparameterizable_by_child(List *pathlist,
+ RelOptInfo *child_rel);
/*****************************************************************************
@@ -2436,6 +2438,16 @@ create_nestloop_path(PlannerInfo *root,
{
NestPath *pathnode = makeNode(NestPath);
Relids inner_req_outer = PATH_REQ_OUTER(inner_path);
+ Relids outerrelids;
+
+ /*
+ * Paths are parameterized by top-level parents, so run parameterization
+ * tests on the parent relids.
+ */
+ if (outer_path->parent->top_parent_relids)
+ outerrelids = outer_path->parent->top_parent_relids;
+ else
+ outerrelids = outer_path->parent->relids;
/*
* If the inner path is parameterized by the outer, we must drop any
@@ -2445,7 +2457,7 @@ create_nestloop_path(PlannerInfo *root,
* estimates for this path. We detect such clauses by checking for serial
* number match to clauses already enforced in the inner path.
*/
- if (bms_overlap(inner_req_outer, outer_path->parent->relids))
+ if (bms_overlap(inner_req_outer, outerrelids))
{
Bitmapset *enforced_serials = get_param_path_clause_serials(inner_path);
List *jclauses = NIL;
@@ -4042,6 +4054,12 @@ reparameterize_path(PlannerInfo *root, Path *path,
*
* Currently, only a few path types are supported here, though more could be
* added at need. We return NULL if we can't reparameterize the given path.
+ *
+ * Note that this function can change referenced RTEs as well as the Path
+ * structures. Therefore, it's only safe to call during create_plan(),
+ * when we have made a final choice of which Path to use for each RTE.
+ *
+ * Keep this code in sync with path_is_reparameterizable_by_child()!
*/
Path *
reparameterize_path_by_child(PlannerInfo *root, Path *path,
@@ -4054,7 +4072,7 @@ reparameterize_path_by_child(PlannerInfo *root, Path *path,
#define ADJUST_CHILD_ATTRS(node) \
((node) = \
- (List *) adjust_appendrel_attrs_multilevel(root, (Node *) (node), \
+ (void *) adjust_appendrel_attrs_multilevel(root, (Node *) (node), \
child_rel, \
child_rel->top_parent))
@@ -4082,8 +4100,8 @@ do { \
Relids required_outer;
/*
- * If the path is not parameterized by parent of the given relation, it
- * doesn't need reparameterization.
+ * If the path is not parameterized by the parent of the given relation,
+ * it doesn't need reparameterization.
*/
if (!path->param_info ||
!bms_overlap(PATH_REQ_OUTER(path), child_rel->top_parent_relids))
@@ -4105,6 +4123,20 @@ do { \
{
case T_Path:
FLAT_COPY_PATH(new_path, path, Path);
+ ADJUST_CHILD_ATTRS(new_path->parent->baserestrictinfo);
+ if (path->pathtype == T_SampleScan)
+ {
+ Index scan_relid = path->parent->relid;
+ RangeTblEntry *rte;
+
+ /* it should be a base rel with a tablesample clause... */
+ Assert(scan_relid > 0);
+ rte = planner_rt_fetch(scan_relid, root);
+ Assert(rte->rtekind == RTE_RELATION);
+ Assert(rte->tablesample != NULL);
+
+ ADJUST_CHILD_ATTRS(rte->tablesample);
+ }
break;
case T_IndexPath:
@@ -4112,6 +4144,7 @@ do { \
IndexPath *ipath;
FLAT_COPY_PATH(ipath, path, IndexPath);
+ ADJUST_CHILD_ATTRS(ipath->indexinfo->indrestrictinfo);
ADJUST_CHILD_ATTRS(ipath->indexclauses);
new_path = (Path *) ipath;
}
@@ -4122,6 +4155,7 @@ do { \
BitmapHeapPath *bhpath;
FLAT_COPY_PATH(bhpath, path, BitmapHeapPath);
+ ADJUST_CHILD_ATTRS(bhpath->path.parent->baserestrictinfo);
REPARAMETERIZE_CHILD_PATH(bhpath->bitmapqual);
new_path = (Path *) bhpath;
}
@@ -4153,6 +4187,7 @@ do { \
ReparameterizeForeignPathByChild_function rfpc_func;
FLAT_COPY_PATH(fpath, path, ForeignPath);
+ ADJUST_CHILD_ATTRS(fpath->path.parent->baserestrictinfo);
if (fpath->fdw_outerpath)
REPARAMETERIZE_CHILD_PATH(fpath->fdw_outerpath);
if (fpath->fdw_restrictinfo)
@@ -4173,6 +4208,7 @@ do { \
CustomPath *cpath;
FLAT_COPY_PATH(cpath, path, CustomPath);
+ ADJUST_CHILD_ATTRS(cpath->path.parent->baserestrictinfo);
REPARAMETERIZE_CHILD_PATH_LIST(cpath->custom_paths);
if (cpath->custom_restrictinfo)
ADJUST_CHILD_ATTRS(cpath->custom_restrictinfo);
@@ -4335,9 +4371,145 @@ do { \
return new_path;
}
+/*
+ * path_is_reparameterizable_by_child
+ * Given a path parameterized by the parent of the given child relation,
+ * see if it can be translated to be parameterized by the child relation.
+ *
+ * This must return true if and only if reparameterize_path_by_child()
+ * would succeed on this path. Currently it's sufficient to verify that
+ * the path and all of its subpaths (if any) are of the types handled by
+ * that function. However, sub-paths that are not parameterized can be
+ * disregarded since they won't require translation.
+ */
+bool
+path_is_reparameterizable_by_child(Path *path, RelOptInfo *child_rel)
+{
+
+#define CHILD_PATH_IS_REPARAMETERIZABLE(path) \
+do { \
+ if (!path_is_reparameterizable_by_child(path, child_rel)) \
+ return false; \
+} while(0)
+
+#define CHILD_PATH_LIST_IS_REPARAMETERIZABLE(pathlist) \
+do { \
+ if (!pathlist_is_reparameterizable_by_child(pathlist, child_rel)) \
+ return false; \
+} while(0)
+
+ /*
+ * If the path is not parameterized by the parent of the given relation,
+ * it doesn't need reparameterization.
+ */
+ if (!path->param_info ||
+ !bms_overlap(PATH_REQ_OUTER(path), child_rel->top_parent_relids))
+ return true;
+
+ switch (nodeTag(path))
+ {
+ case T_Path:
+ case T_IndexPath:
+ break;
+
+ case T_BitmapHeapPath:
+ {
+ BitmapHeapPath *bhpath = (BitmapHeapPath *) path;
+
+ CHILD_PATH_IS_REPARAMETERIZABLE(bhpath->bitmapqual);
+ }
+ break;
+
+ case T_BitmapAndPath:
+ {
+ BitmapAndPath *bapath = (BitmapAndPath *) path;
+
+ CHILD_PATH_LIST_IS_REPARAMETERIZABLE(bapath->bitmapquals);
+ }
+ break;
+
+ case T_BitmapOrPath:
+ {
+ BitmapOrPath *bopath = (BitmapOrPath *) path;
+
+ CHILD_PATH_LIST_IS_REPARAMETERIZABLE(bopath->bitmapquals);
+ }
+ break;
+
+ case T_ForeignPath:
+ {
+ ForeignPath *fpath = (ForeignPath *) path;
+
+ if (fpath->fdw_outerpath)
+ CHILD_PATH_IS_REPARAMETERIZABLE(fpath->fdw_outerpath);
+ }
+ break;
+
+ case T_CustomPath:
+ {
+ CustomPath *cpath = (CustomPath *) path;
+
+ CHILD_PATH_LIST_IS_REPARAMETERIZABLE(cpath->custom_paths);
+ }
+ break;
+
+ case T_NestPath:
+ case T_MergePath:
+ case T_HashPath:
+ {
+ JoinPath *jpath = (JoinPath *) path;
+
+ CHILD_PATH_IS_REPARAMETERIZABLE(jpath->outerjoinpath);
+ CHILD_PATH_IS_REPARAMETERIZABLE(jpath->innerjoinpath);
+ }
+ break;
+
+ case T_AppendPath:
+ {
+ AppendPath *apath = (AppendPath *) path;
+
+ CHILD_PATH_LIST_IS_REPARAMETERIZABLE(apath->subpaths);
+ }
+ break;
+
+ case T_MaterialPath:
+ {
+ MaterialPath *mpath = (MaterialPath *) path;
+
+ CHILD_PATH_IS_REPARAMETERIZABLE(mpath->subpath);
+ }
+ break;
+
+ case T_MemoizePath:
+ {
+ MemoizePath *mpath = (MemoizePath *) path;
+
+ CHILD_PATH_IS_REPARAMETERIZABLE(mpath->subpath);
+ }
+ break;
+
+ case T_GatherPath:
+ {
+ GatherPath *gpath = (GatherPath *) path;
+
+ CHILD_PATH_IS_REPARAMETERIZABLE(gpath->subpath);
+ }
+ break;
+
+ default:
+
+ /* We don't know how to reparameterize this path. */
+ return false;
+ }
+
+ return true;
+}
+
/*
* reparameterize_pathlist_by_child
* Helper function to reparameterize a list of paths by given child rel.
+ *
+ * Returns NIL to indicate failure, so pathlist had better not be NIL.
*/
static List *
reparameterize_pathlist_by_child(PlannerInfo *root,
@@ -4363,3 +4535,23 @@ reparameterize_pathlist_by_child(PlannerInfo *root,
return result;
}
+
+/*
+ * pathlist_is_reparameterizable_by_child
+ * Helper function to check if a list of paths can be reparameterized.
+ */
+static bool
+pathlist_is_reparameterizable_by_child(List *pathlist, RelOptInfo *child_rel)
+{
+ ListCell *lc;
+
+ foreach(lc, pathlist)
+ {
+ Path *path = (Path *) lfirst(lc);
+
+ if (!path_is_reparameterizable_by_child(path, child_rel))
+ return false;
+ }
+
+ return true;
+}
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index 6e557bebc4..f5f8cbcfb4 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -298,6 +298,8 @@ extern Path *reparameterize_path(PlannerInfo *root, Path *path,
double loop_count);
extern Path *reparameterize_path_by_child(PlannerInfo *root, Path *path,
RelOptInfo *child_rel);
+extern bool path_is_reparameterizable_by_child(Path *path,
+ RelOptInfo *child_rel);
/*
* prototypes for relnode.c
diff --git a/src/test/regress/expected/partition_join.out b/src/test/regress/expected/partition_join.out
index 6560fe2416..e713f9fb2f 100644
--- a/src/test/regress/expected/partition_join.out
+++ b/src/test/regress/expected/partition_join.out
@@ -505,6 +505,65 @@ SELECT t1.a, ss.t2a, ss.t2c FROM prt1 t1 LEFT JOIN LATERAL
550 | |
(12 rows)
+-- lateral reference in sample scan
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt1 t1 JOIN LATERAL
+ (SELECT * FROM prt1 t2 TABLESAMPLE SYSTEM (t1.a) REPEATABLE(t1.b)) s
+ ON t1.a = s.a;
+ QUERY PLAN
+-------------------------------------------------------------
+ Append
+ -> Nested Loop
+ -> Seq Scan on prt1_p1 t1_1
+ -> Sample Scan on prt1_p1 t2_1
+ Sampling: system (t1_1.a) REPEATABLE (t1_1.b)
+ Filter: (t1_1.a = a)
+ -> Nested Loop
+ -> Seq Scan on prt1_p2 t1_2
+ -> Sample Scan on prt1_p2 t2_2
+ Sampling: system (t1_2.a) REPEATABLE (t1_2.b)
+ Filter: (t1_2.a = a)
+ -> Nested Loop
+ -> Seq Scan on prt1_p3 t1_3
+ -> Sample Scan on prt1_p3 t2_3
+ Sampling: system (t1_3.a) REPEATABLE (t1_3.b)
+ Filter: (t1_3.a = a)
+(16 rows)
+
+-- lateral reference in scan's restriction clauses
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.a;
+ QUERY PLAN
+---------------------------------------------------------------
+ Aggregate
+ -> Append
+ -> Nested Loop
+ -> Seq Scan on prt1_p1 t1_1
+ -> Index Scan using iprt2_p1_b on prt2_p1 t2_1
+ Index Cond: (b = t1_1.a)
+ Filter: (t1_1.b = a)
+ -> Nested Loop
+ -> Seq Scan on prt1_p2 t1_2
+ -> Index Scan using iprt2_p2_b on prt2_p2 t2_2
+ Index Cond: (b = t1_2.a)
+ Filter: (t1_2.b = a)
+ -> Nested Loop
+ -> Seq Scan on prt1_p3 t1_3
+ -> Index Scan using iprt2_p3_b on prt2_p3 t2_3
+ Index Cond: (b = t1_3.a)
+ Filter: (t1_3.b = a)
+(17 rows)
+
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.a;
+ count
+-------
+ 100
+(1 row)
+
-- bug with inadequate sort key representation
SET enable_partitionwise_aggregate TO true;
SET enable_hashjoin TO false;
@@ -1944,6 +2003,41 @@ SELECT * FROM prt1_l t1 LEFT JOIN LATERAL
550 | 0 | 0002 | | | | |
(12 rows)
+-- partitionwise join with lateral reference in sample scan
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt1_l t1 JOIN LATERAL
+ (SELECT * FROM prt1_l t2 TABLESAMPLE SYSTEM (t1.a) REPEATABLE(t1.b)) s ON
+ t1.a = s.a AND t1.b = s.b AND t1.c = s.c;
+ QUERY PLAN
+----------------------------------------------------------------------------------------
+ Append
+ -> Nested Loop
+ -> Seq Scan on prt1_l_p1 t1_1
+ -> Sample Scan on prt1_l_p1 t2_1
+ Sampling: system (t1_1.a) REPEATABLE (t1_1.b)
+ Filter: ((t1_1.a = a) AND (t1_1.b = b) AND ((t1_1.c)::text = (c)::text))
+ -> Nested Loop
+ -> Seq Scan on prt1_l_p2_p1 t1_2
+ -> Sample Scan on prt1_l_p2_p1 t2_2
+ Sampling: system (t1_2.a) REPEATABLE (t1_2.b)
+ Filter: ((t1_2.a = a) AND (t1_2.b = b) AND ((t1_2.c)::text = (c)::text))
+ -> Nested Loop
+ -> Seq Scan on prt1_l_p2_p2 t1_3
+ -> Sample Scan on prt1_l_p2_p2 t2_3
+ Sampling: system (t1_3.a) REPEATABLE (t1_3.b)
+ Filter: ((t1_3.a = a) AND (t1_3.b = b) AND ((t1_3.c)::text = (c)::text))
+ -> Nested Loop
+ -> Seq Scan on prt1_l_p3_p1 t1_4
+ -> Sample Scan on prt1_l_p3_p1 t2_4
+ Sampling: system (t1_4.a) REPEATABLE (t1_4.b)
+ Filter: ((t1_4.a = a) AND (t1_4.b = b) AND ((t1_4.c)::text = (c)::text))
+ -> Nested Loop
+ -> Seq Scan on prt1_l_p3_p2 t1_5
+ -> Sample Scan on prt1_l_p3_p2 t2_5
+ Sampling: system (t1_5.a) REPEATABLE (t1_5.b)
+ Filter: ((t1_5.a = a) AND (t1_5.b = b) AND ((t1_5.c)::text = (c)::text))
+(26 rows)
+
-- join with one side empty
EXPLAIN (COSTS OFF)
SELECT t1.a, t1.c, t2.b, t2.c FROM (SELECT * FROM prt1_l WHERE a = 1 AND a = 2) t1 RIGHT JOIN prt2_l t2 ON t1.a = t2.b AND t1.b = t2.a AND t1.c = t2.c;
diff --git a/src/test/regress/sql/partition_join.sql b/src/test/regress/sql/partition_join.sql
index 48daf3aee3..5ab3148dec 100644
--- a/src/test/regress/sql/partition_join.sql
+++ b/src/test/regress/sql/partition_join.sql
@@ -100,6 +100,21 @@ SELECT t1.a, ss.t2a, ss.t2c FROM prt1 t1 LEFT JOIN LATERAL
(SELECT t2.a AS t2a, t3.a AS t3a, t2.b t2b, t2.c t2c, least(t1.a,t2.a,t3.a) FROM prt1 t2 JOIN prt2 t3 ON (t2.a = t3.b)) ss
ON t1.c = ss.t2c WHERE (t1.b + coalesce(ss.t2b, 0)) = 0 ORDER BY t1.a;
+-- lateral reference in sample scan
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt1 t1 JOIN LATERAL
+ (SELECT * FROM prt1 t2 TABLESAMPLE SYSTEM (t1.a) REPEATABLE(t1.b)) s
+ ON t1.a = s.a;
+
+-- lateral reference in scan's restriction clauses
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.a;
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.a;
+
-- bug with inadequate sort key representation
SET enable_partitionwise_aggregate TO true;
SET enable_hashjoin TO false;
@@ -387,6 +402,12 @@ SELECT * FROM prt1_l t1 LEFT JOIN LATERAL
(SELECT t2.a AS t2a, t2.c AS t2c, t2.b AS t2b, t3.b AS t3b, least(t1.a,t2.a,t3.b) FROM prt1_l t2 JOIN prt2_l t3 ON (t2.a = t3.b AND t2.c = t3.c)) ss
ON t1.a = ss.t2a AND t1.c = ss.t2c WHERE t1.b = 0 ORDER BY t1.a;
+-- partitionwise join with lateral reference in sample scan
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt1_l t1 JOIN LATERAL
+ (SELECT * FROM prt1_l t2 TABLESAMPLE SYSTEM (t1.a) REPEATABLE(t1.b)) s ON
+ t1.a = s.a AND t1.b = s.b AND t1.c = s.c;
+
-- join with one side empty
EXPLAIN (COSTS OFF)
SELECT t1.a, t1.c, t2.b, t2.c FROM (SELECT * FROM prt1_l WHERE a = 1 AND a = 2) t1 RIGHT JOIN prt2_l t2 ON t1.a = t2.b AND t1.b = t2.a AND t1.c = t2.c;
--
2.31.0
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Oversight in reparameterize_path_by_child leading to executor crash
@ 2023-08-23 16:15 Ashutosh Bapat <[email protected]>
parent: Richard Guo <[email protected]>
2 siblings, 0 replies; 86+ messages in thread
From: Ashutosh Bapat @ 2023-08-23 16:15 UTC (permalink / raw)
To: Richard Guo <[email protected]>; +Cc: Tom Lane <[email protected]>; pgsql-hackers
On Wed, Aug 23, 2023 at 11:07 AM Richard Guo <[email protected]> wrote:
>
>
> On Tue, Aug 22, 2023 at 10:39 PM Tom Lane <[email protected]> wrote:
>>
>> Richard Guo <[email protected]> writes:
>> > I'm wondering if we can instead adjust the 'inner_req_outer' in
>> > create_nestloop_path before we perform the check to work around this
>> > issue for the back branches, something like
>> > + /*
>> > + * Adjust the parameterization information, which refers to the topmost
>> > + * parent.
>> > + */
>> > + inner_req_outer =
>> > + adjust_child_relids_multilevel(root, inner_req_outer,
>> > + outer_path->parent->relids,
>> > + outer_path->parent->top_parent_relids);
>>
>> Mmm ... at the very least you'd need to not do that when top_parent_relids
>> is unset.
>
>
> Hmm, adjust_child_relids_multilevel would just return the given relids
> unchanged when top_parent_relids is unset, so I think it should be safe
> to call it even for non-other rel.
>
>>
>> ... But I think the risk/reward ratio for messing with this in the
>> back branches is unattractive in any case: to fix a corner case that
>> apparently nobody uses in the field, we risk breaking any number of
>> mainstream parameterized-path cases. I'm content to commit the v5 patch
>> (or a successor) into HEAD, but at this point I'm not sure I even want
>> to risk it in v16, let alone perform delicate surgery to get it to work
>> in older branches. I think we ought to go with the "tablesample scans
>> can't be reparameterized" approach in v16 and before.
>
>
> If we go with the "tablesample scans can't be reparameterized" approach
> in the back branches, I'm a little concerned that what if we find more
> cases in the futrue where we need modify RTEs for reparameterization.
> So I spent some time seeking and have managed to find one: there might
> be lateral references in a scan path's restriction clauses, and
> currently reparameterize_path_by_child fails to adjust them.
>
> regression=# explain (verbose, costs off)
> select count(*) from prt1 t1 left join lateral
> (select t1.b as t1b, t2.* from prt2 t2) s
> on t1.a = s.b where s.t1b = s.a;
> QUERY PLAN
> ----------------------------------------------------------------------
> Aggregate
> Output: count(*)
> -> Append
> -> Nested Loop
> -> Seq Scan on public.prt1_p1 t1_1
> Output: t1_1.a, t1_1.b
> -> Index Scan using iprt2_p1_b on public.prt2_p1 t2_1
> Output: t2_1.b
> Index Cond: (t2_1.b = t1_1.a)
> Filter: (t1.b = t2_1.a)
> -> Nested Loop
> -> Seq Scan on public.prt1_p2 t1_2
> Output: t1_2.a, t1_2.b
> -> Index Scan using iprt2_p2_b on public.prt2_p2 t2_2
> Output: t2_2.b
> Index Cond: (t2_2.b = t1_2.a)
> Filter: (t1.b = t2_2.a)
> -> Nested Loop
> -> Seq Scan on public.prt1_p3 t1_3
> Output: t1_3.a, t1_3.b
> -> Index Scan using iprt2_p3_b on public.prt2_p3 t2_3
> Output: t2_3.b
> Index Cond: (t2_3.b = t1_3.a)
> Filter: (t1.b = t2_3.a)
> (24 rows)
>
> The clauses in 'Filter' are not adjusted correctly. Var 't1.b' still
> refers to the top parent.
>
> For this query it would just give wrong results.
>
> regression=# set enable_partitionwise_join to off;
> SET
> regression=# select count(*) from prt1 t1 left join lateral
> (select t1.b as t1b, t2.* from prt2 t2) s
> on t1.a = s.b where s.t1b = s.a;
> count
> -------
> 100
> (1 row)
>
> regression=# set enable_partitionwise_join to on;
> SET
> regression=# select count(*) from prt1 t1 left join lateral
> (select t1.b as t1b, t2.* from prt2 t2) s
> on t1.a = s.b where s.t1b = s.a;
> count
> -------
> 5
> (1 row)
>
>
> For another query it would error out during planning.
>
> regression=# explain (verbose, costs off)
> select count(*) from prt1 t1 left join lateral
> (select t1.b as t1b, t2.* from prt2 t2) s
> on t1.a = s.b where s.t1b = s.b;
> ERROR: variable not found in subplan target list
>
> To fix it we may need to modify RelOptInfos for Path, BitmapHeapPath,
> ForeignPath and CustomPath, and modify IndexOptInfos for IndexPath. It
> seems that that is not easily done without postponing reparameterization
> of paths until createplan.c.
Maybe I am missing something here but why aren't we copying these
restrictinfos in the path somewhere? I have a similar question for
tablesample clause as well.
--
Best Wishes,
Ashutosh Bapat
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Oversight in reparameterize_path_by_child leading to executor crash
@ 2023-08-23 17:44 Tom Lane <[email protected]>
1 sibling, 1 reply; 86+ messages in thread
From: Tom Lane @ 2023-08-23 17:44 UTC (permalink / raw)
To: Richard Guo <[email protected]>; +Cc: Ashutosh Bapat <[email protected]>; pgsql-hackers
Richard Guo <[email protected]> writes:
> If we go with the "tablesample scans can't be reparameterized" approach
> in the back branches, I'm a little concerned that what if we find more
> cases in the futrue where we need modify RTEs for reparameterization.
> So I spent some time seeking and have managed to find one: there might
> be lateral references in a scan path's restriction clauses, and
> currently reparameterize_path_by_child fails to adjust them.
Hmm, this seems completely wrong to me. By definition, such clauses
ought to be join clauses not restriction clauses, so how are we getting
into this state? IOW, I agree this is clearly buggy but I think the
bug is someplace else.
regards, tom lane
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Oversight in reparameterize_path_by_child leading to executor crash
@ 2023-08-24 02:47 Richard Guo <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Richard Guo @ 2023-08-24 02:47 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Ashutosh Bapat <[email protected]>; pgsql-hackers
On Thu, Aug 24, 2023 at 1:44 AM Tom Lane <[email protected]> wrote:
> Richard Guo <[email protected]> writes:
> > If we go with the "tablesample scans can't be reparameterized" approach
> > in the back branches, I'm a little concerned that what if we find more
> > cases in the futrue where we need modify RTEs for reparameterization.
> > So I spent some time seeking and have managed to find one: there might
> > be lateral references in a scan path's restriction clauses, and
> > currently reparameterize_path_by_child fails to adjust them.
>
> Hmm, this seems completely wrong to me. By definition, such clauses
> ought to be join clauses not restriction clauses, so how are we getting
> into this state? IOW, I agree this is clearly buggy but I think the
> bug is someplace else.
If the clause contains PHVs that syntactically belong to a rel and
meanwhile have lateral references to other rels, then it may become a
restriction clause with lateral references. Take the query shown
upthread as an example,
select count(*) from prt1 t1 left join lateral
(select t1.b as t1b, t2.* from prt2 t2) s
on t1.a = s.b where s.t1b = s.a;
The clause 's.t1b = s.a' would become 'PHV(t1.b) = t2.a' after we have
pulled up the subquery. The PHV in it syntactically belongs to 't2' and
laterally refers to 't1'. So this clause is actually a restriction
clause for rel 't2', and will be put into the baserestrictinfo of t2
rel. But it also has lateral reference to rel 't1', which we need to
adjust in reparameterize_path_by_child for partitionwise join.
Thanks
Richard
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Oversight in reparameterize_path_by_child leading to executor crash
@ 2023-09-08 07:04 Ashutosh Bapat <[email protected]>
parent: Richard Guo <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Ashutosh Bapat @ 2023-09-08 07:04 UTC (permalink / raw)
To: Richard Guo <[email protected]>; +Cc: Tom Lane <[email protected]>; pgsql-hackers
On Thu, Aug 24, 2023 at 8:17 AM Richard Guo <[email protected]> wrote:
>
>
> On Thu, Aug 24, 2023 at 1:44 AM Tom Lane <[email protected]> wrote:
>>
>> Richard Guo <[email protected]> writes:
>> > If we go with the "tablesample scans can't be reparameterized" approach
>> > in the back branches, I'm a little concerned that what if we find more
>> > cases in the futrue where we need modify RTEs for reparameterization.
>> > So I spent some time seeking and have managed to find one: there might
>> > be lateral references in a scan path's restriction clauses, and
>> > currently reparameterize_path_by_child fails to adjust them.
>>
>> Hmm, this seems completely wrong to me. By definition, such clauses
>> ought to be join clauses not restriction clauses, so how are we getting
>> into this state? IOW, I agree this is clearly buggy but I think the
>> bug is someplace else.
>
>
> If the clause contains PHVs that syntactically belong to a rel and
> meanwhile have lateral references to other rels, then it may become a
> restriction clause with lateral references. Take the query shown
> upthread as an example,
>
> select count(*) from prt1 t1 left join lateral
> (select t1.b as t1b, t2.* from prt2 t2) s
> on t1.a = s.b where s.t1b = s.a;
>
> The clause 's.t1b = s.a' would become 'PHV(t1.b) = t2.a' after we have
> pulled up the subquery. The PHV in it syntactically belongs to 't2' and
> laterally refers to 't1'. So this clause is actually a restriction
> clause for rel 't2', and will be put into the baserestrictinfo of t2
> rel. But it also has lateral reference to rel 't1', which we need to
> adjust in reparameterize_path_by_child for partitionwise join.
When the clause s.t1b = s.a is presented to distribute_qual_to_rels()
it has form PHV(t1.b) = t2.b. The PHV's ph_eval_at is 4, which is what
is returned as varno to pull_varnos(). The other Var in the caluse has
varno = 4 already so pull_varnos() returns a SINGLETON relids (b 4).
The clause is an equality clause, so it is used to create an
Equivalence class.
generate_base_implied_equalities_no_const() then constructs the same
RestrictInfo again and adds to baserestrictinfo of Rel with relid = 4
i.e. t2's baserestrictinfo. I don't know whether that's the right
thing to do. After the subquery has been pulled up, t1 and t2 can be
joined at the same level and thus the clause makes more sense as a
joininfo in both t1 as well as t2. So I tend to agree with Tom. That's
how it will move up the join tree and be evaluated at appropriate
level. But then why the query returns the right results is a mystery.
I am not sure where we are taking the original bug fix with this
investigation. Is it required to fix this problem in order to fix the
original problem OR we should commit the fix for the original problem
and then investigate this further?
--
Best Wishes,
Ashutosh Bapat
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Oversight in reparameterize_path_by_child leading to executor crash
@ 2023-09-11 02:05 Richard Guo <[email protected]>
parent: Ashutosh Bapat <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Richard Guo @ 2023-09-11 02:05 UTC (permalink / raw)
To: Ashutosh Bapat <[email protected]>; +Cc: Tom Lane <[email protected]>; pgsql-hackers
On Fri, Sep 8, 2023 at 3:04 PM Ashutosh Bapat <[email protected]>
wrote:
> When the clause s.t1b = s.a is presented to distribute_qual_to_rels()
> it has form PHV(t1.b) = t2.b. The PHV's ph_eval_at is 4, which is what
> is returned as varno to pull_varnos(). The other Var in the caluse has
> varno = 4 already so pull_varnos() returns a SINGLETON relids (b 4).
> The clause is an equality clause, so it is used to create an
> Equivalence class.
> generate_base_implied_equalities_no_const() then constructs the same
> RestrictInfo again and adds to baserestrictinfo of Rel with relid = 4
> i.e. t2's baserestrictinfo. I don't know whether that's the right
> thing to do.
Well, I think that's what PHVs are supposed to do. Quoting the README:
... Note that even with this restriction, pullup of a LATERAL
subquery can result in creating PlaceHolderVars that contain lateral
references to relations outside their syntactic scope. We still evaluate
such PHVs at their syntactic location or lower, but the presence of such a
PHV in the quals or targetlist of a plan node requires that node to appear
on the inside of a nestloop join relative to the rel(s) supplying the
lateral reference.
> I am not sure where we are taking the original bug fix with this
> investigation. Is it required to fix this problem in order to fix the
> original problem OR we should commit the fix for the original problem
> and then investigate this further?
Fair point. This seems a separate problem from the original, so I'm
okay we fix them separately.
Thanks
Richard
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Oversight in reparameterize_path_by_child leading to executor crash
@ 2023-09-20 07:18 Andrey Lepikhov <[email protected]>
parent: Richard Guo <[email protected]>
2 siblings, 0 replies; 86+ messages in thread
From: Andrey Lepikhov @ 2023-09-20 07:18 UTC (permalink / raw)
To: Richard Guo <[email protected]>; Tom Lane <[email protected]>; +Cc: Ashutosh Bapat <[email protected]>; pgsql-hackers
On 23/8/2023 12:37, Richard Guo wrote:
> If we go with the "tablesample scans can't be reparameterized" approach
> in the back branches, I'm a little concerned that what if we find more
> cases in the futrue where we need modify RTEs for reparameterization.
> So I spent some time seeking and have managed to find one: there might
> be lateral references in a scan path's restriction clauses, and
> currently reparameterize_path_by_child fails to adjust them.
It may help you somehow: in [1], we designed a feature where the
partitionwise join technique can be applied to a JOIN of partitioned and
non-partitioned tables. Unfortunately, it is out of community
discussions, but we still support it for sharding usage - it is helpful
for the implementation of 'global' tables in a distributed
configuration. And there we were stuck into the same problem with
lateral relids adjustment. So you can build a more general view of the
problem with this patch.
[1] Asymmetric partition-wise JOIN
https://www.postgresql.org/message-id/flat/CAOP8fzaVL_2SCJayLL9kj5pCA46PJOXXjuei6-3aFUV45j4LJQ%40mai...
--
regards,
Andrey Lepikhov
Postgres Professional
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Oversight in reparameterize_path_by_child leading to executor crash
@ 2023-10-13 10:18 Andrei Lepikhov <[email protected]>
parent: Richard Guo <[email protected]>
2 siblings, 1 reply; 86+ messages in thread
From: Andrei Lepikhov @ 2023-10-13 10:18 UTC (permalink / raw)
To: Richard Guo <[email protected]>; Tom Lane <[email protected]>; +Cc: Ashutosh Bapat <[email protected]>; pgsql-hackers
On 23/8/2023 12:37, Richard Guo wrote:
> To fix it we may need to modify RelOptInfos for Path, BitmapHeapPath,
> ForeignPath and CustomPath, and modify IndexOptInfos for IndexPath. It
> seems that that is not easily done without postponing reparameterization
> of paths until createplan.c.
>
> Attached is a patch which is v5 + fix for this new issue.
Having looked into the patch for a while, I couldn't answer to myself
for some stupid questions:
1. If we postpone parameterization until the plan creation, why do we
still copy the path node in the FLAT_COPY_PATH macros? Do we really need it?
2. I see big switches on path nodes. May it be time to create a
path_walker function? I recall some thread where such a suggestion was
declined, but I don't remember why.
3. Clause changing is postponed. Why does it not influence the
calc_joinrel_size_estimate? We will use statistics on the parent table
here. Am I wrong?
--
regards,
Andrey Lepikhov
Postgres Professional
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Oversight in reparameterize_path_by_child leading to executor crash
@ 2023-10-18 06:39 Richard Guo <[email protected]>
parent: Andrei Lepikhov <[email protected]>
0 siblings, 2 replies; 86+ messages in thread
From: Richard Guo @ 2023-10-18 06:39 UTC (permalink / raw)
To: Andrei Lepikhov <[email protected]>; +Cc: Tom Lane <[email protected]>; Ashutosh Bapat <[email protected]>; pgsql-hackers
On Fri, Oct 13, 2023 at 6:18 PM Andrei Lepikhov <[email protected]>
wrote:
> On 23/8/2023 12:37, Richard Guo wrote:
> > To fix it we may need to modify RelOptInfos for Path, BitmapHeapPath,
> > ForeignPath and CustomPath, and modify IndexOptInfos for IndexPath. It
> > seems that that is not easily done without postponing reparameterization
> > of paths until createplan.c.
> >
> > Attached is a patch which is v5 + fix for this new issue.
>
> Having looked into the patch for a while, I couldn't answer to myself
> for some stupid questions:
Thanks for reviewing this patch! I think these are great questions.
> 1. If we postpone parameterization until the plan creation, why do we
> still copy the path node in the FLAT_COPY_PATH macros? Do we really need
> it?
Good point. The NestPath's origin inner path should not be referenced
any more after the reparameterization, so it seems safe to adjust the
path itself, without the need of a flat-copy. I've done that in v8
patch.
> 2. I see big switches on path nodes. May it be time to create a
> path_walker function? I recall some thread where such a suggestion was
> declined, but I don't remember why.
I'm not sure. But this seems a separate topic, so maybe it's better to
discuss it separately?
> 3. Clause changing is postponed. Why does it not influence the
> calc_joinrel_size_estimate? We will use statistics on the parent table
> here. Am I wrong?
Hmm, I think the reparameterization does not change the estimated
size/costs. Quoting the comment
* The cost, number of rows, width and parallel path properties depend upon
* path->parent, which does not change during the translation.
Hi Tom, I'm wondering if you've had a chance to follow this issue. What
do you think about the proposed patch?
Thanks
Richard
Attachments:
[application/octet-stream] v8-0001-Postpone-reparameterization-of-paths-until-when-creating-plans.patch (25.0K, ../../CAMbWs48PBwe1YadzgKGW_ES=V9BZhq00BaZTOTM6Oye8n_cDNg@mail.gmail.com/3-v8-0001-Postpone-reparameterization-of-paths-until-when-creating-plans.patch)
download | inline diff:
From 87c8e01db02c6ed0df0c80db3855c4d1bf7f90a8 Mon Sep 17 00:00:00 2001
From: Richard Guo <[email protected]>
Date: Wed, 18 Oct 2023 11:18:00 +0800
Subject: [PATCH v8] Postpone reparameterization of paths until when creating
plans
---
src/backend/optimizer/path/joinpath.c | 67 +++--
src/backend/optimizer/plan/createplan.c | 17 ++
src/backend/optimizer/util/pathnode.c | 249 ++++++++++++++++---
src/include/optimizer/pathnode.h | 2 +
src/test/regress/expected/partition_join.out | 94 +++++++
src/test/regress/sql/partition_join.sql | 21 ++
6 files changed, 380 insertions(+), 70 deletions(-)
diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c
index b2916ad690..33b0cf067e 100644
--- a/src/backend/optimizer/path/joinpath.c
+++ b/src/backend/optimizer/path/joinpath.c
@@ -30,8 +30,9 @@
set_join_pathlist_hook_type set_join_pathlist_hook = NULL;
/*
- * Paths parameterized by the parent can be considered to be parameterized by
- * any of its child.
+ * Paths parameterized by a parent rel can be considered to be parameterized
+ * by any of its children, when we are performing partitionwise joins. These
+ * macros simplify checking for such cases. Beware multiple eval of args.
*/
#define PATH_PARAM_BY_PARENT(path, rel) \
((path)->param_info && bms_overlap(PATH_REQ_OUTER(path), \
@@ -765,6 +766,20 @@ try_nestloop_path(PlannerInfo *root,
/* If we got past that, we shouldn't have any unsafe outer-join refs */
Assert(!have_unsafe_outer_join_ref(root, outerrelids, inner_paramrels));
+ /*
+ * If the inner path is parameterized, it is parameterized by the topmost
+ * parent of the outer rel, not the outer rel itself. We will need to
+ * translate the parameterization, if this path is chosen, during
+ * create_plan(). Here we just check whether we will be able to perform
+ * the translation, and if not avoid creating a nestloop path.
+ */
+ if (PATH_PARAM_BY_PARENT(inner_path, outer_path->parent) &&
+ !path_is_reparameterizable_by_child(inner_path, outer_path->parent))
+ {
+ bms_free(required_outer);
+ return;
+ }
+
/*
* Do a precheck to quickly eliminate obviously-inferior paths. We
* calculate a cheap lower bound on the path's cost and then use
@@ -781,27 +796,6 @@ try_nestloop_path(PlannerInfo *root,
workspace.startup_cost, workspace.total_cost,
pathkeys, required_outer))
{
- /*
- * If the inner path is parameterized, it is parameterized by the
- * topmost parent of the outer rel, not the outer rel itself. Fix
- * that.
- */
- if (PATH_PARAM_BY_PARENT(inner_path, outer_path->parent))
- {
- inner_path = reparameterize_path_by_child(root, inner_path,
- outer_path->parent);
-
- /*
- * If we could not translate the path, we can't create nest loop
- * path.
- */
- if (!inner_path)
- {
- bms_free(required_outer);
- return;
- }
- }
-
add_path(joinrel, (Path *)
create_nestloop_path(root,
joinrel,
@@ -864,6 +858,17 @@ try_partial_nestloop_path(PlannerInfo *root,
return;
}
+ /*
+ * If the inner path is parameterized, it is parameterized by the topmost
+ * parent of the outer rel, not the outer rel itself. We will need to
+ * translate the parameterization, if this path is chosen, during
+ * create_plan(). Here we just check whether we will be able to perform
+ * the translation, and if not avoid creating a nestloop path.
+ */
+ if (PATH_PARAM_BY_PARENT(inner_path, outer_path->parent) &&
+ !path_is_reparameterizable_by_child(inner_path, outer_path->parent))
+ return;
+
/*
* Before creating a path, get a quick lower bound on what it is likely to
* cost. Bail out right away if it looks terrible.
@@ -873,22 +878,6 @@ try_partial_nestloop_path(PlannerInfo *root,
if (!add_partial_path_precheck(joinrel, workspace.total_cost, pathkeys))
return;
- /*
- * If the inner path is parameterized, it is parameterized by the topmost
- * parent of the outer rel, not the outer rel itself. Fix that.
- */
- if (PATH_PARAM_BY_PARENT(inner_path, outer_path->parent))
- {
- inner_path = reparameterize_path_by_child(root, inner_path,
- outer_path->parent);
-
- /*
- * If we could not translate the path, we can't create nest loop path.
- */
- if (!inner_path)
- return;
- }
-
/* Might be good enough to be worth trying, so let's try it. */
add_partial_path(joinrel, (Path *)
create_nestloop_path(root,
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 34ca6d4ac2..38bd179f4f 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -29,6 +29,7 @@
#include "optimizer/cost.h"
#include "optimizer/optimizer.h"
#include "optimizer/paramassign.h"
+#include "optimizer/pathnode.h"
#include "optimizer/paths.h"
#include "optimizer/placeholder.h"
#include "optimizer/plancat.h"
@@ -4346,6 +4347,22 @@ create_nestloop_plan(PlannerInfo *root,
List *nestParams;
Relids saveOuterRels = root->curOuterRels;
+ /*
+ * If the inner path is parameterized by the topmost parent of the outer
+ * rel rather than the outer rel itself, fix that. (Nothing happens here
+ * if it is not so parameterized.)
+ */
+ best_path->jpath.innerjoinpath =
+ reparameterize_path_by_child(root,
+ best_path->jpath.innerjoinpath,
+ best_path->jpath.outerjoinpath->parent);
+
+ /*
+ * Failure here probably means that reparameterize_path_by_child() is not
+ * in sync with path_is_reparameterizable_by_child().
+ */
+ Assert(best_path->jpath.innerjoinpath != NULL);
+
/* NestLoop can project, so no need to be picky about child tlists */
outer_plan = create_plan_recurse(root, best_path->jpath.outerjoinpath, 0);
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index b65323532b..376a16cacb 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -56,6 +56,8 @@ static int append_startup_cost_compare(const ListCell *a, const ListCell *b);
static List *reparameterize_pathlist_by_child(PlannerInfo *root,
List *pathlist,
RelOptInfo *child_rel);
+static bool pathlist_is_reparameterizable_by_child(List *pathlist,
+ RelOptInfo *child_rel);
/*****************************************************************************
@@ -2436,6 +2438,16 @@ create_nestloop_path(PlannerInfo *root,
{
NestPath *pathnode = makeNode(NestPath);
Relids inner_req_outer = PATH_REQ_OUTER(inner_path);
+ Relids outerrelids;
+
+ /*
+ * Paths are parameterized by top-level parents, so run parameterization
+ * tests on the parent relids.
+ */
+ if (outer_path->parent->top_parent_relids)
+ outerrelids = outer_path->parent->top_parent_relids;
+ else
+ outerrelids = outer_path->parent->relids;
/*
* If the inner path is parameterized by the outer, we must drop any
@@ -2445,7 +2457,7 @@ create_nestloop_path(PlannerInfo *root,
* estimates for this path. We detect such clauses by checking for serial
* number match to clauses already enforced in the inner path.
*/
- if (bms_overlap(inner_req_outer, outer_path->parent->relids))
+ if (bms_overlap(inner_req_outer, outerrelids))
{
Bitmapset *enforced_serials = get_param_path_clause_serials(inner_path);
List *jclauses = NIL;
@@ -4045,32 +4057,33 @@ reparameterize_path(PlannerInfo *root, Path *path,
* Given a path parameterized by the parent of the given child relation,
* translate the path to be parameterized by the given child relation.
*
- * The function creates a new path of the same type as the given path, but
- * parameterized by the given child relation. Most fields from the original
- * path can simply be flat-copied, but any expressions must be adjusted to
- * refer to the correct varnos, and any paths must be recursively
- * reparameterized. Other fields that refer to specific relids also need
- * adjustment.
+ * The function translates the given path to be parameterized by the given
+ * child relation. Most fields from the original path are not changed, but any
+ * expressions must be adjusted to refer to the correct varnos, and any paths
+ * must be recursively reparameterized. Other fields that refer to specific
+ * relids also need adjustment.
*
* The cost, number of rows, width and parallel path properties depend upon
- * path->parent, which does not change during the translation. Hence those
- * members are copied as they are.
+ * path->parent, which does not change during the translation.
*
* Currently, only a few path types are supported here, though more could be
* added at need. We return NULL if we can't reparameterize the given path.
+ *
+ * Note that this function can change referenced RTEs, RelOptInfos and
+ * IndexOptInfos as well as the Path structures. Therefore, it's only safe
+ * to call during create_plan(), when we have made a final choice of which
+ * Path to use for each RTE.
+ *
+ * Keep this code in sync with path_is_reparameterizable_by_child()!
*/
Path *
reparameterize_path_by_child(PlannerInfo *root, Path *path,
RelOptInfo *child_rel)
{
-#define FLAT_COPY_PATH(newnode, node, nodetype) \
- ( (newnode) = makeNode(nodetype), \
- memcpy((newnode), (node), sizeof(nodetype)) )
-
#define ADJUST_CHILD_ATTRS(node) \
((node) = \
- (List *) adjust_appendrel_attrs_multilevel(root, (Node *) (node), \
+ (void *) adjust_appendrel_attrs_multilevel(root, (Node *) (node), \
child_rel, \
child_rel->top_parent))
@@ -4098,15 +4111,15 @@ do { \
Relids required_outer;
/*
- * If the path is not parameterized by parent of the given relation, it
- * doesn't need reparameterization.
+ * If the path is not parameterized by the parent of the given relation,
+ * it doesn't need reparameterization.
*/
if (!path->param_info ||
!bms_overlap(PATH_REQ_OUTER(path), child_rel->top_parent_relids))
return path;
/*
- * If possible, reparameterize the given path, making a copy.
+ * If possible, reparameterize the given path.
*
* This function is currently only applied to the inner side of a nestloop
* join that is being partitioned by the partitionwise-join code. Hence,
@@ -4120,14 +4133,29 @@ do { \
switch (nodeTag(path))
{
case T_Path:
- FLAT_COPY_PATH(new_path, path, Path);
+ new_path = path;
+ ADJUST_CHILD_ATTRS(new_path->parent->baserestrictinfo);
+ if (path->pathtype == T_SampleScan)
+ {
+ Index scan_relid = path->parent->relid;
+ RangeTblEntry *rte;
+
+ /* it should be a base rel with a tablesample clause... */
+ Assert(scan_relid > 0);
+ rte = planner_rt_fetch(scan_relid, root);
+ Assert(rte->rtekind == RTE_RELATION);
+ Assert(rte->tablesample != NULL);
+
+ ADJUST_CHILD_ATTRS(rte->tablesample);
+ }
break;
case T_IndexPath:
{
IndexPath *ipath;
- FLAT_COPY_PATH(ipath, path, IndexPath);
+ ipath = (IndexPath *) path;
+ ADJUST_CHILD_ATTRS(ipath->indexinfo->indrestrictinfo);
ADJUST_CHILD_ATTRS(ipath->indexclauses);
new_path = (Path *) ipath;
}
@@ -4137,7 +4165,8 @@ do { \
{
BitmapHeapPath *bhpath;
- FLAT_COPY_PATH(bhpath, path, BitmapHeapPath);
+ bhpath = (BitmapHeapPath *) path;
+ ADJUST_CHILD_ATTRS(bhpath->path.parent->baserestrictinfo);
REPARAMETERIZE_CHILD_PATH(bhpath->bitmapqual);
new_path = (Path *) bhpath;
}
@@ -4147,7 +4176,7 @@ do { \
{
BitmapAndPath *bapath;
- FLAT_COPY_PATH(bapath, path, BitmapAndPath);
+ bapath = (BitmapAndPath *) path;
REPARAMETERIZE_CHILD_PATH_LIST(bapath->bitmapquals);
new_path = (Path *) bapath;
}
@@ -4157,7 +4186,7 @@ do { \
{
BitmapOrPath *bopath;
- FLAT_COPY_PATH(bopath, path, BitmapOrPath);
+ bopath = (BitmapOrPath *) path;
REPARAMETERIZE_CHILD_PATH_LIST(bopath->bitmapquals);
new_path = (Path *) bopath;
}
@@ -4168,7 +4197,8 @@ do { \
ForeignPath *fpath;
ReparameterizeForeignPathByChild_function rfpc_func;
- FLAT_COPY_PATH(fpath, path, ForeignPath);
+ fpath = (ForeignPath *) path;
+ ADJUST_CHILD_ATTRS(fpath->path.parent->baserestrictinfo);
if (fpath->fdw_outerpath)
REPARAMETERIZE_CHILD_PATH(fpath->fdw_outerpath);
if (fpath->fdw_restrictinfo)
@@ -4188,7 +4218,8 @@ do { \
{
CustomPath *cpath;
- FLAT_COPY_PATH(cpath, path, CustomPath);
+ cpath = (CustomPath *) path;
+ ADJUST_CHILD_ATTRS(cpath->path.parent->baserestrictinfo);
REPARAMETERIZE_CHILD_PATH_LIST(cpath->custom_paths);
if (cpath->custom_restrictinfo)
ADJUST_CHILD_ATTRS(cpath->custom_restrictinfo);
@@ -4207,7 +4238,7 @@ do { \
JoinPath *jpath;
NestPath *npath;
- FLAT_COPY_PATH(npath, path, NestPath);
+ npath = (NestPath *) path;
jpath = (JoinPath *) npath;
REPARAMETERIZE_CHILD_PATH(jpath->outerjoinpath);
@@ -4222,7 +4253,7 @@ do { \
JoinPath *jpath;
MergePath *mpath;
- FLAT_COPY_PATH(mpath, path, MergePath);
+ mpath = (MergePath *) path;
jpath = (JoinPath *) mpath;
REPARAMETERIZE_CHILD_PATH(jpath->outerjoinpath);
@@ -4238,7 +4269,7 @@ do { \
JoinPath *jpath;
HashPath *hpath;
- FLAT_COPY_PATH(hpath, path, HashPath);
+ hpath = (HashPath *) path;
jpath = (JoinPath *) hpath;
REPARAMETERIZE_CHILD_PATH(jpath->outerjoinpath);
@@ -4253,7 +4284,7 @@ do { \
{
AppendPath *apath;
- FLAT_COPY_PATH(apath, path, AppendPath);
+ apath = (AppendPath *) path;
REPARAMETERIZE_CHILD_PATH_LIST(apath->subpaths);
new_path = (Path *) apath;
}
@@ -4263,7 +4294,7 @@ do { \
{
MaterialPath *mpath;
- FLAT_COPY_PATH(mpath, path, MaterialPath);
+ mpath = (MaterialPath *) path;
REPARAMETERIZE_CHILD_PATH(mpath->subpath);
new_path = (Path *) mpath;
}
@@ -4273,7 +4304,7 @@ do { \
{
MemoizePath *mpath;
- FLAT_COPY_PATH(mpath, path, MemoizePath);
+ mpath = (MemoizePath *) path;
REPARAMETERIZE_CHILD_PATH(mpath->subpath);
ADJUST_CHILD_ATTRS(mpath->param_exprs);
new_path = (Path *) mpath;
@@ -4284,7 +4315,7 @@ do { \
{
GatherPath *gpath;
- FLAT_COPY_PATH(gpath, path, GatherPath);
+ gpath = (GatherPath *) path;
REPARAMETERIZE_CHILD_PATH(gpath->subpath);
new_path = (Path *) gpath;
}
@@ -4351,9 +4382,145 @@ do { \
return new_path;
}
+/*
+ * path_is_reparameterizable_by_child
+ * Given a path parameterized by the parent of the given child relation,
+ * see if it can be translated to be parameterized by the child relation.
+ *
+ * This must return true if and only if reparameterize_path_by_child()
+ * would succeed on this path. Currently it's sufficient to verify that
+ * the path and all of its subpaths (if any) are of the types handled by
+ * that function. However, sub-paths that are not parameterized can be
+ * disregarded since they won't require translation.
+ */
+bool
+path_is_reparameterizable_by_child(Path *path, RelOptInfo *child_rel)
+{
+
+#define CHILD_PATH_IS_REPARAMETERIZABLE(path) \
+do { \
+ if (!path_is_reparameterizable_by_child(path, child_rel)) \
+ return false; \
+} while(0)
+
+#define CHILD_PATH_LIST_IS_REPARAMETERIZABLE(pathlist) \
+do { \
+ if (!pathlist_is_reparameterizable_by_child(pathlist, child_rel)) \
+ return false; \
+} while(0)
+
+ /*
+ * If the path is not parameterized by the parent of the given relation,
+ * it doesn't need reparameterization.
+ */
+ if (!path->param_info ||
+ !bms_overlap(PATH_REQ_OUTER(path), child_rel->top_parent_relids))
+ return true;
+
+ switch (nodeTag(path))
+ {
+ case T_Path:
+ case T_IndexPath:
+ break;
+
+ case T_BitmapHeapPath:
+ {
+ BitmapHeapPath *bhpath = (BitmapHeapPath *) path;
+
+ CHILD_PATH_IS_REPARAMETERIZABLE(bhpath->bitmapqual);
+ }
+ break;
+
+ case T_BitmapAndPath:
+ {
+ BitmapAndPath *bapath = (BitmapAndPath *) path;
+
+ CHILD_PATH_LIST_IS_REPARAMETERIZABLE(bapath->bitmapquals);
+ }
+ break;
+
+ case T_BitmapOrPath:
+ {
+ BitmapOrPath *bopath = (BitmapOrPath *) path;
+
+ CHILD_PATH_LIST_IS_REPARAMETERIZABLE(bopath->bitmapquals);
+ }
+ break;
+
+ case T_ForeignPath:
+ {
+ ForeignPath *fpath = (ForeignPath *) path;
+
+ if (fpath->fdw_outerpath)
+ CHILD_PATH_IS_REPARAMETERIZABLE(fpath->fdw_outerpath);
+ }
+ break;
+
+ case T_CustomPath:
+ {
+ CustomPath *cpath = (CustomPath *) path;
+
+ CHILD_PATH_LIST_IS_REPARAMETERIZABLE(cpath->custom_paths);
+ }
+ break;
+
+ case T_NestPath:
+ case T_MergePath:
+ case T_HashPath:
+ {
+ JoinPath *jpath = (JoinPath *) path;
+
+ CHILD_PATH_IS_REPARAMETERIZABLE(jpath->outerjoinpath);
+ CHILD_PATH_IS_REPARAMETERIZABLE(jpath->innerjoinpath);
+ }
+ break;
+
+ case T_AppendPath:
+ {
+ AppendPath *apath = (AppendPath *) path;
+
+ CHILD_PATH_LIST_IS_REPARAMETERIZABLE(apath->subpaths);
+ }
+ break;
+
+ case T_MaterialPath:
+ {
+ MaterialPath *mpath = (MaterialPath *) path;
+
+ CHILD_PATH_IS_REPARAMETERIZABLE(mpath->subpath);
+ }
+ break;
+
+ case T_MemoizePath:
+ {
+ MemoizePath *mpath = (MemoizePath *) path;
+
+ CHILD_PATH_IS_REPARAMETERIZABLE(mpath->subpath);
+ }
+ break;
+
+ case T_GatherPath:
+ {
+ GatherPath *gpath = (GatherPath *) path;
+
+ CHILD_PATH_IS_REPARAMETERIZABLE(gpath->subpath);
+ }
+ break;
+
+ default:
+
+ /* We don't know how to reparameterize this path. */
+ return false;
+ }
+
+ return true;
+}
+
/*
* reparameterize_pathlist_by_child
* Helper function to reparameterize a list of paths by given child rel.
+ *
+ * Returns NIL to indicate failure, so pathlist had better not be NIL.
*/
static List *
reparameterize_pathlist_by_child(PlannerInfo *root,
@@ -4379,3 +4546,23 @@ reparameterize_pathlist_by_child(PlannerInfo *root,
return result;
}
+
+/*
+ * pathlist_is_reparameterizable_by_child
+ * Helper function to check if a list of paths can be reparameterized.
+ */
+static bool
+pathlist_is_reparameterizable_by_child(List *pathlist, RelOptInfo *child_rel)
+{
+ ListCell *lc;
+
+ foreach(lc, pathlist)
+ {
+ Path *path = (Path *) lfirst(lc);
+
+ if (!path_is_reparameterizable_by_child(path, child_rel))
+ return false;
+ }
+
+ return true;
+}
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index 6e557bebc4..f5f8cbcfb4 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -298,6 +298,8 @@ extern Path *reparameterize_path(PlannerInfo *root, Path *path,
double loop_count);
extern Path *reparameterize_path_by_child(PlannerInfo *root, Path *path,
RelOptInfo *child_rel);
+extern bool path_is_reparameterizable_by_child(Path *path,
+ RelOptInfo *child_rel);
/*
* prototypes for relnode.c
diff --git a/src/test/regress/expected/partition_join.out b/src/test/regress/expected/partition_join.out
index 6560fe2416..e713f9fb2f 100644
--- a/src/test/regress/expected/partition_join.out
+++ b/src/test/regress/expected/partition_join.out
@@ -505,6 +505,65 @@ SELECT t1.a, ss.t2a, ss.t2c FROM prt1 t1 LEFT JOIN LATERAL
550 | |
(12 rows)
+-- lateral reference in sample scan
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt1 t1 JOIN LATERAL
+ (SELECT * FROM prt1 t2 TABLESAMPLE SYSTEM (t1.a) REPEATABLE(t1.b)) s
+ ON t1.a = s.a;
+ QUERY PLAN
+-------------------------------------------------------------
+ Append
+ -> Nested Loop
+ -> Seq Scan on prt1_p1 t1_1
+ -> Sample Scan on prt1_p1 t2_1
+ Sampling: system (t1_1.a) REPEATABLE (t1_1.b)
+ Filter: (t1_1.a = a)
+ -> Nested Loop
+ -> Seq Scan on prt1_p2 t1_2
+ -> Sample Scan on prt1_p2 t2_2
+ Sampling: system (t1_2.a) REPEATABLE (t1_2.b)
+ Filter: (t1_2.a = a)
+ -> Nested Loop
+ -> Seq Scan on prt1_p3 t1_3
+ -> Sample Scan on prt1_p3 t2_3
+ Sampling: system (t1_3.a) REPEATABLE (t1_3.b)
+ Filter: (t1_3.a = a)
+(16 rows)
+
+-- lateral reference in scan's restriction clauses
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.a;
+ QUERY PLAN
+---------------------------------------------------------------
+ Aggregate
+ -> Append
+ -> Nested Loop
+ -> Seq Scan on prt1_p1 t1_1
+ -> Index Scan using iprt2_p1_b on prt2_p1 t2_1
+ Index Cond: (b = t1_1.a)
+ Filter: (t1_1.b = a)
+ -> Nested Loop
+ -> Seq Scan on prt1_p2 t1_2
+ -> Index Scan using iprt2_p2_b on prt2_p2 t2_2
+ Index Cond: (b = t1_2.a)
+ Filter: (t1_2.b = a)
+ -> Nested Loop
+ -> Seq Scan on prt1_p3 t1_3
+ -> Index Scan using iprt2_p3_b on prt2_p3 t2_3
+ Index Cond: (b = t1_3.a)
+ Filter: (t1_3.b = a)
+(17 rows)
+
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.a;
+ count
+-------
+ 100
+(1 row)
+
-- bug with inadequate sort key representation
SET enable_partitionwise_aggregate TO true;
SET enable_hashjoin TO false;
@@ -1944,6 +2003,41 @@ SELECT * FROM prt1_l t1 LEFT JOIN LATERAL
550 | 0 | 0002 | | | | |
(12 rows)
+-- partitionwise join with lateral reference in sample scan
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt1_l t1 JOIN LATERAL
+ (SELECT * FROM prt1_l t2 TABLESAMPLE SYSTEM (t1.a) REPEATABLE(t1.b)) s ON
+ t1.a = s.a AND t1.b = s.b AND t1.c = s.c;
+ QUERY PLAN
+----------------------------------------------------------------------------------------
+ Append
+ -> Nested Loop
+ -> Seq Scan on prt1_l_p1 t1_1
+ -> Sample Scan on prt1_l_p1 t2_1
+ Sampling: system (t1_1.a) REPEATABLE (t1_1.b)
+ Filter: ((t1_1.a = a) AND (t1_1.b = b) AND ((t1_1.c)::text = (c)::text))
+ -> Nested Loop
+ -> Seq Scan on prt1_l_p2_p1 t1_2
+ -> Sample Scan on prt1_l_p2_p1 t2_2
+ Sampling: system (t1_2.a) REPEATABLE (t1_2.b)
+ Filter: ((t1_2.a = a) AND (t1_2.b = b) AND ((t1_2.c)::text = (c)::text))
+ -> Nested Loop
+ -> Seq Scan on prt1_l_p2_p2 t1_3
+ -> Sample Scan on prt1_l_p2_p2 t2_3
+ Sampling: system (t1_3.a) REPEATABLE (t1_3.b)
+ Filter: ((t1_3.a = a) AND (t1_3.b = b) AND ((t1_3.c)::text = (c)::text))
+ -> Nested Loop
+ -> Seq Scan on prt1_l_p3_p1 t1_4
+ -> Sample Scan on prt1_l_p3_p1 t2_4
+ Sampling: system (t1_4.a) REPEATABLE (t1_4.b)
+ Filter: ((t1_4.a = a) AND (t1_4.b = b) AND ((t1_4.c)::text = (c)::text))
+ -> Nested Loop
+ -> Seq Scan on prt1_l_p3_p2 t1_5
+ -> Sample Scan on prt1_l_p3_p2 t2_5
+ Sampling: system (t1_5.a) REPEATABLE (t1_5.b)
+ Filter: ((t1_5.a = a) AND (t1_5.b = b) AND ((t1_5.c)::text = (c)::text))
+(26 rows)
+
-- join with one side empty
EXPLAIN (COSTS OFF)
SELECT t1.a, t1.c, t2.b, t2.c FROM (SELECT * FROM prt1_l WHERE a = 1 AND a = 2) t1 RIGHT JOIN prt2_l t2 ON t1.a = t2.b AND t1.b = t2.a AND t1.c = t2.c;
diff --git a/src/test/regress/sql/partition_join.sql b/src/test/regress/sql/partition_join.sql
index 48daf3aee3..5ab3148dec 100644
--- a/src/test/regress/sql/partition_join.sql
+++ b/src/test/regress/sql/partition_join.sql
@@ -100,6 +100,21 @@ SELECT t1.a, ss.t2a, ss.t2c FROM prt1 t1 LEFT JOIN LATERAL
(SELECT t2.a AS t2a, t3.a AS t3a, t2.b t2b, t2.c t2c, least(t1.a,t2.a,t3.a) FROM prt1 t2 JOIN prt2 t3 ON (t2.a = t3.b)) ss
ON t1.c = ss.t2c WHERE (t1.b + coalesce(ss.t2b, 0)) = 0 ORDER BY t1.a;
+-- lateral reference in sample scan
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt1 t1 JOIN LATERAL
+ (SELECT * FROM prt1 t2 TABLESAMPLE SYSTEM (t1.a) REPEATABLE(t1.b)) s
+ ON t1.a = s.a;
+
+-- lateral reference in scan's restriction clauses
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.a;
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.a;
+
-- bug with inadequate sort key representation
SET enable_partitionwise_aggregate TO true;
SET enable_hashjoin TO false;
@@ -387,6 +402,12 @@ SELECT * FROM prt1_l t1 LEFT JOIN LATERAL
(SELECT t2.a AS t2a, t2.c AS t2c, t2.b AS t2b, t3.b AS t3b, least(t1.a,t2.a,t3.b) FROM prt1_l t2 JOIN prt2_l t3 ON (t2.a = t3.b AND t2.c = t3.c)) ss
ON t1.a = ss.t2a AND t1.c = ss.t2c WHERE t1.b = 0 ORDER BY t1.a;
+-- partitionwise join with lateral reference in sample scan
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt1_l t1 JOIN LATERAL
+ (SELECT * FROM prt1_l t2 TABLESAMPLE SYSTEM (t1.a) REPEATABLE(t1.b)) s ON
+ t1.a = s.a AND t1.b = s.b AND t1.c = s.c;
+
-- join with one side empty
EXPLAIN (COSTS OFF)
SELECT t1.a, t1.c, t2.b, t2.c FROM (SELECT * FROM prt1_l WHERE a = 1 AND a = 2) t1 RIGHT JOIN prt2_l t2 ON t1.a = t2.b AND t1.b = t2.a AND t1.c = t2.c;
--
2.31.0
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Oversight in reparameterize_path_by_child leading to executor crash
@ 2023-10-18 09:41 Andrei Lepikhov <[email protected]>
parent: Richard Guo <[email protected]>
1 sibling, 0 replies; 86+ messages in thread
From: Andrei Lepikhov @ 2023-10-18 09:41 UTC (permalink / raw)
To: Richard Guo <[email protected]>; +Cc: Tom Lane <[email protected]>; Ashutosh Bapat <[email protected]>; pgsql-hackers
On 18/10/2023 13:39, Richard Guo wrote:
>
> On Fri, Oct 13, 2023 at 6:18 PM Andrei Lepikhov
> <[email protected] <mailto:[email protected]>> wrote:
>
> On 23/8/2023 12:37, Richard Guo wrote:
> > To fix it we may need to modify RelOptInfos for Path, BitmapHeapPath,
> > ForeignPath and CustomPath, and modify IndexOptInfos for
> IndexPath. It
> > seems that that is not easily done without postponing
> reparameterization
> > of paths until createplan.c.
> >
> > Attached is a patch which is v5 + fix for this new issue.
>
> Having looked into the patch for a while, I couldn't answer to myself
> for some stupid questions:
>
>
> Thanks for reviewing this patch! I think these are great questions.
>
> 1. If we postpone parameterization until the plan creation, why do we
> still copy the path node in the FLAT_COPY_PATH macros? Do we really
> need it?
>
>
> Good point. The NestPath's origin inner path should not be referenced
> any more after the reparameterization, so it seems safe to adjust the
> path itself, without the need of a flat-copy. I've done that in v8
> patch.
>
> 2. I see big switches on path nodes. May it be time to create a
> path_walker function? I recall some thread where such a suggestion was
> declined, but I don't remember why.
>
>
> I'm not sure. But this seems a separate topic, so maybe it's better to
> discuss it separately?
Agree.
> 3. Clause changing is postponed. Why does it not influence the
> calc_joinrel_size_estimate? We will use statistics on the parent table
> here. Am I wrong?
>
>
> Hmm, I think the reparameterization does not change the estimated
> size/costs. Quoting the comment
Agree. I have looked at the code and figured it out - you're right. But
it seems strange: maybe I don't understand something. Why not estimate
selectivity for parameterized clauses based on leaf partition statistic,
not the parent one?
--
regards,
Andrey Lepikhov
Postgres Professional
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Oversight in reparameterize_path_by_child leading to executor crash
@ 2023-10-19 18:52 Alena Rybakina <[email protected]>
parent: Richard Guo <[email protected]>
1 sibling, 1 reply; 86+ messages in thread
From: Alena Rybakina @ 2023-10-19 18:52 UTC (permalink / raw)
To: Richard Guo <[email protected]>; +Cc: Tom Lane <[email protected]>; Ashutosh Bapat <[email protected]>; pgsql-hackers; Andrei Lepikhov <[email protected]>
Hi!
Thank you for your work on the subject.
During review your patch I didn't understand why are you checking that
the variable is path and not new_path of type T_SamplePath (I
highlighted it)?
Path *
reparameterize_path_by_child(PlannerInfo *root, Path *path,
RelOptInfo *child_rel)
...
switch (nodeTag(path))
{
case T_Path:
new_path = path;
ADJUST_CHILD_ATTRS(new_path->parent->baserestrictinfo);
if (*path*->pathtype == T_SampleScan)
{
Is it a typo and should be new_path?
Besides, it may not be important, but reviewing your code all the time
stumbled on the statement of the comments while reading it (I
highlighted it too). This:
* path_is_reparameterizable_by_child
* Given a path parameterized by the parent of the given child
relation,
* see if it can be translated to be parameterized by the child
relation.
*
* This must return true if *and only if *reparameterize_path_by_child()
* would succeed on this path.
Maybe is it better to rewrite it simplier:
* This must return true *only if *reparameterize_path_by_child()
* would succeed on this path.
And can we add assert in reparameterize_pathlist_by_child function that
pathlist is not a NIL, because according to the comment it needs to be
added there:
Returns NIL to indicate failure, so pathlist had better not be NIL.
--
Regards,
Alena Rybakina
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Oversight in reparameterize_path_by_child leading to executor crash
@ 2023-12-06 06:51 Richard Guo <[email protected]>
parent: Alena Rybakina <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Richard Guo @ 2023-12-06 06:51 UTC (permalink / raw)
To: Alena Rybakina <[email protected]>; +Cc: Tom Lane <[email protected]>; Ashutosh Bapat <[email protected]>; pgsql-hackers; Andrei Lepikhov <[email protected]>
On Fri, Oct 20, 2023 at 2:52 AM Alena Rybakina <[email protected]>
wrote:
> Thank you for your work on the subject.
>
Thanks for taking an interest in this patch.
> During review your patch I didn't understand why are you checking that the
> variable is path and not new_path of type T_SamplePath (I highlighted it)?
>
> Is it a typo and should be new_path?
>
I don't think there is any difference: path and new_path are the same
pointer in this case.
> Besides, it may not be important, but reviewing your code all the time
> stumbled on the statement of the comments while reading it (I highlighted
> it too). This:
>
> * path_is_reparameterizable_by_child
> * Given a path parameterized by the parent of the given child
> relation,
> * see if it can be translated to be parameterized by the child
> relation.
> *
> * This must return true if *and only if *reparameterize_path_by_child()
> * would succeed on this path.
>
> Maybe is it better to rewrite it simplier:
>
> * This must return true *only if *reparameterize_path_by_child()
> * would succeed on this path.
>
I don't think so. "if and only if" is more accurate to me.
> And can we add assert in reparameterize_pathlist_by_child function that
> pathlist is not a NIL, because according to the comment it needs to be
> added there:
>
Hmm, I'm not sure, as in REPARAMETERIZE_CHILD_PATH_LIST we have already
explicitly checked that the pathlist is not NIL.
Thanks
Richard
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Oversight in reparameterize_path_by_child leading to executor crash
@ 2023-12-06 07:30 Richard Guo <[email protected]>
parent: Richard Guo <[email protected]>
0 siblings, 2 replies; 86+ messages in thread
From: Richard Guo @ 2023-12-06 07:30 UTC (permalink / raw)
To: Alena Rybakina <[email protected]>; +Cc: Tom Lane <[email protected]>; Ashutosh Bapat <[email protected]>; pgsql-hackers; Andrei Lepikhov <[email protected]>
I've self-reviewed this patch again and I think it's now in a
committable state. I'm wondering if we can mark it as 'Ready for
Committer' now, or we need more review comments/feedbacks.
To recap, this patch postpones reparameterization of paths until
createplan.c, which would help avoid building the reparameterized
expressions we might not use. More importantly, it makes it possible to
modify the expressions in RTEs (e.g. tablesample) and in RelOptInfos
(e.g. baserestrictinfo) for reparameterization. Failing to do that can
lead to executor crashes, wrong results, or planning errors, as we have
already seen.
Thanks
Richard
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Oversight in reparameterize_path_by_child leading to executor crash
@ 2023-12-08 09:38 Alena Rybakina <[email protected]>
parent: Richard Guo <[email protected]>
1 sibling, 1 reply; 86+ messages in thread
From: Alena Rybakina @ 2023-12-08 09:38 UTC (permalink / raw)
To: Richard Guo <[email protected]>; +Cc: Tom Lane <[email protected]>; Ashutosh Bapat <[email protected]>; pgsql-hackers; Andrei Lepikhov <[email protected]>
On 06.12.2023 10:30, Richard Guo wrote:
> I've self-reviewed this patch again and I think it's now in a
> committable state. I'm wondering if we can mark it as 'Ready for
> Committer' now, or we need more review comments/feedbacks.
>
> To recap, this patch postpones reparameterization of paths until
> createplan.c, which would help avoid building the reparameterized
> expressions we might not use. More importantly, it makes it possible to
> modify the expressions in RTEs (e.g. tablesample) and in RelOptInfos
> (e.g. baserestrictinfo) for reparameterization. Failing to do that can
> lead to executor crashes, wrong results, or planning errors, as we have
> already seen.
I marked it as 'Ready for Committer'. I think it is ready.
--
Regards,
Alena Rybakina
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Oversight in reparameterize_path_by_child leading to executor crash
@ 2023-12-11 03:02 Richard Guo <[email protected]>
parent: Alena Rybakina <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Richard Guo @ 2023-12-11 03:02 UTC (permalink / raw)
To: Alena Rybakina <[email protected]>; +Cc: Tom Lane <[email protected]>; Ashutosh Bapat <[email protected]>; pgsql-hackers; Andrei Lepikhov <[email protected]>
On Fri, Dec 8, 2023 at 5:39 PM Alena Rybakina <[email protected]>
wrote:
> On 06.12.2023 10:30, Richard Guo wrote:
> > I've self-reviewed this patch again and I think it's now in a
> > committable state. I'm wondering if we can mark it as 'Ready for
> > Committer' now, or we need more review comments/feedbacks.
> >
> > To recap, this patch postpones reparameterization of paths until
> > createplan.c, which would help avoid building the reparameterized
> > expressions we might not use. More importantly, it makes it possible to
> > modify the expressions in RTEs (e.g. tablesample) and in RelOptInfos
> > (e.g. baserestrictinfo) for reparameterization. Failing to do that can
> > lead to executor crashes, wrong results, or planning errors, as we have
> > already seen.
I marked it as 'Ready for Committer'. I think it is ready.
Thank you. Appreciate that.
Thanks
Richard
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Oversight in reparameterize_path_by_child leading to executor crash
@ 2023-12-13 02:55 Andrei Lepikhov <[email protected]>
parent: Richard Guo <[email protected]>
1 sibling, 1 reply; 86+ messages in thread
From: Andrei Lepikhov @ 2023-12-13 02:55 UTC (permalink / raw)
To: Richard Guo <[email protected]>; Alena Rybakina <[email protected]>; +Cc: Tom Lane <[email protected]>; Ashutosh Bapat <[email protected]>; pgsql-hackers
On 6/12/2023 14:30, Richard Guo wrote:
> I've self-reviewed this patch again and I think it's now in a
> committable state. I'm wondering if we can mark it as 'Ready for
> Committer' now, or we need more review comments/feedbacks.
>
> To recap, this patch postpones reparameterization of paths until
> createplan.c, which would help avoid building the reparameterized
> expressions we might not use. More importantly, it makes it possible to
> modify the expressions in RTEs (e.g. tablesample) and in RelOptInfos
> (e.g. baserestrictinfo) for reparameterization. Failing to do that can
> lead to executor crashes, wrong results, or planning errors, as we have
> already seen.
As I see, this patch contains the following modifications:
1. Changes in the create_nestloop_path: here, it arranges all tests to
the value of top_parent_relids, if any. It is ok for me.
2. Changes in reparameterize_path_by_child and coupled new function
path_is_reparameterizable_by_child. I compared them, and it looks good.
One thing here. You use the assertion trap in the case of inconsistency
in the behaviour of these two functions. How disastrous would it be if
someone found such inconsistency in production? Maybe better to use
elog(PANIC, ...) report?
3. ADJUST_CHILD_ATTRS() macros. Understanding the necessity of this
change, I want to say it looks a bit ugly.
4. SampleScan reparameterization part. It looks ok. It can help us in
the future with asymmetric join and something else.
5. Tests. All of them are related to partitioning and the SampleScan
issue. Maybe it is enough, but remember, this change now impacts the
parameterization feature in general.
6. I can't judge how this switch of overall approach could impact
something in the planner. I am only wary about what, from the first
reparameterization up to the plan creation, we have some inconsistency
in expressions (partitions refer to expressions with the parent relid).
What if an extension in the middle changes the parent expression?
By and large, this patch is in a good state and may be committed.
--
regards,
Andrei Lepikhov
Postgres Professional
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Oversight in reparameterize_path_by_child leading to executor crash
@ 2024-01-05 18:36 Robert Haas <[email protected]>
parent: Andrei Lepikhov <[email protected]>
0 siblings, 2 replies; 86+ messages in thread
From: Robert Haas @ 2024-01-05 18:36 UTC (permalink / raw)
To: Andrei Lepikhov <[email protected]>; +Cc: Richard Guo <[email protected]>; Alena Rybakina <[email protected]>; Tom Lane <[email protected]>; Ashutosh Bapat <[email protected]>; pgsql-hackers
On Tue, Dec 12, 2023 at 9:55 PM Andrei Lepikhov
<[email protected]> wrote:
> By and large, this patch is in a good state and may be committed.
OK, so a few people like the current form of this patch but we haven't
heard from Tom since August. Tom, any thoughts on the current
incarnation?
Richard, I think it could be useful to put a better commit message
into the patch file, describing both what problem is being fixed and
what the design of the fix is. I gather that the problem is that we
crash if the query contains a partioningwise join and also $SOMETHING,
and the solution is to move reparameterization to happen at
createplan() time but with a precheck that runs during path
generation. Presumably, that means this is more than a minimal bug
fix, because the bug could be fixed without splitting
can-it-be-reparameterized to reparameterize-it in this way. Probably
that's a performance optimization, so maybe it's worth clarifying
whether that's just an independently good idea or whether it's a part
of making the bug fix not regress performance.
I think the macro names in path_is_reparameterizable_by_child could be
better chosen. CHILD_PATH_IS_REPARAMETERIZABLE doesn't convey that the
macro will return from the calling function if not -- it looks like it
just returns a Boolean. Maybe REJECT_IF_PATH_NOT_REPARAMETERIZABLE and
REJECT_IF_PATH_LIST_NOT_REPARAMETERIZABLE or some such.
Another question here is whether we really want to back-patch all of
this or whether it might be better to, as Tom proposed previously,
back-patch a more minimal fix and leave the more invasive stuff for
master.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Oversight in reparameterize_path_by_child leading to executor crash
@ 2024-01-05 18:55 Tom Lane <[email protected]>
parent: Robert Haas <[email protected]>
1 sibling, 0 replies; 86+ messages in thread
From: Tom Lane @ 2024-01-05 18:55 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Andrei Lepikhov <[email protected]>; Richard Guo <[email protected]>; Alena Rybakina <[email protected]>; Ashutosh Bapat <[email protected]>; pgsql-hackers
Robert Haas <[email protected]> writes:
> OK, so a few people like the current form of this patch but we haven't
> heard from Tom since August. Tom, any thoughts on the current
> incarnation?
Not yet, but it is on my to-look-at list. In the meantime I concur
with your comments here.
regards, tom lane
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Oversight in reparameterize_path_by_child leading to executor crash
@ 2024-01-08 08:32 Richard Guo <[email protected]>
parent: Robert Haas <[email protected]>
1 sibling, 1 reply; 86+ messages in thread
From: Richard Guo @ 2024-01-08 08:32 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Andrei Lepikhov <[email protected]>; Alena Rybakina <[email protected]>; Tom Lane <[email protected]>; Ashutosh Bapat <[email protected]>; pgsql-hackers
Thanks for the review!
On Sat, Jan 6, 2024 at 2:36 AM Robert Haas <[email protected]> wrote:
> Richard, I think it could be useful to put a better commit message
> into the patch file, describing both what problem is being fixed and
> what the design of the fix is. I gather that the problem is that we
> crash if the query contains a partioningwise join and also $SOMETHING,
> and the solution is to move reparameterization to happen at
> createplan() time but with a precheck that runs during path
> generation. Presumably, that means this is more than a minimal bug
> fix, because the bug could be fixed without splitting
> can-it-be-reparameterized to reparameterize-it in this way. Probably
> that's a performance optimization, so maybe it's worth clarifying
> whether that's just an independently good idea or whether it's a part
> of making the bug fix not regress performance.
Thanks for the suggestion. Attached is an updated patch which is added
with a commit message that tries to explain the problem and the fix.
I think the macro names in path_is_reparameterizable_by_child could be
> better chosen. CHILD_PATH_IS_REPARAMETERIZABLE doesn't convey that the
> macro will return from the calling function if not -- it looks like it
> just returns a Boolean. Maybe REJECT_IF_PATH_NOT_REPARAMETERIZABLE and
> REJECT_IF_PATH_LIST_NOT_REPARAMETERIZABLE or some such.
Agreed.
> Another question here is whether we really want to back-patch all of
> this or whether it might be better to, as Tom proposed previously,
> back-patch a more minimal fix and leave the more invasive stuff for
> master.
Fair point. I think we can back-patch a more minimal fix, as Tom
proposed in [1], which disallows the reparameterization if the path
contains sampling info that references the outer rel. But I think we
need also to disallow the reparameterization of a path if it contains
restriction clauses that reference the outer rel, as such paths have
been found to cause incorrect results, or planning errors as in [2].
[1] https://www.postgresql.org/message-id/3163033.1692719009%40sss.pgh.pa.us
[2]
https://www.postgresql.org/message-id/CAMbWs4-CSR4VnZCDep3ReSoHGTA7E%2B3tnjF_LmHcX7yiGrkVfQ%40mail.g...
Thanks
Richard
Attachments:
[application/octet-stream] v9-0001-Postpone-reparameterization-of-paths-until-when-creating-plans.patch (26.6K, ../../CAMbWs48QmbYJ26_aULGnX4APfiynFmO8Jx49PAJNMUE6+1v_kw@mail.gmail.com/3-v9-0001-Postpone-reparameterization-of-paths-until-when-creating-plans.patch)
download | inline diff:
From 1a902299dd9c8ae729e5d17e2d923689819160e7 Mon Sep 17 00:00:00 2001
From: Richard Guo <[email protected]>
Date: Mon, 8 Jan 2024 13:47:45 +0800
Subject: [PATCH v9] Postpone reparameterization of paths until when creating
plans
When creating a nestloop join path, if the inner path is parameterized,
it is parameterized by the topmost parent of the outer rel, not the
outer rel itself. Therefore, we need to translate the parameterization
so that the inner path is parameterized by the given outer rel itself.
Currently, this is done in reparameterize_path_by_child() during
generating join paths.
However, reparameterize_path_by_child() does not perform the translation
for sample scan paths' sampling infos, or scan paths' restriction
clauses. This omission can lead to executor crashes, wrong results, or
planning errors, as we have already observed.
Please note that the sampling infos are contained in RangeTblEntries,
and the scan restriction clauses are contained in RelOptInfos or
IndexOptInfos. We cannot just modify them on the fly during generating
join paths. Doing so would break things if we end up using a
non-partitionwise join.
So this commit postpones reparameterization of the inner path until
createplan.c, where it is safe to modify referenced RangeTblEntry,
RelOptInfo or IndexOptInfo, because we have made a final choice of which
Path to use. This can only work if we know that we will be able to
perform the translation for the inner path. So we introduce a new
function path_is_reparameterizable_by_child() during generating join
paths to check whether the given path can be translated.
As an additional benefit, this commit also helps avoid building the
reparameterized expressions we might not use, resulting in saved CPU
cycles and reduced memory usage.
---
src/backend/optimizer/path/joinpath.c | 67 +++--
src/backend/optimizer/plan/createplan.c | 17 ++
src/backend/optimizer/util/pathnode.c | 249 ++++++++++++++++---
src/include/optimizer/pathnode.h | 2 +
src/test/regress/expected/partition_join.out | 94 +++++++
src/test/regress/sql/partition_join.sql | 21 ++
6 files changed, 380 insertions(+), 70 deletions(-)
diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c
index 3fd5a24fad..10147b94e0 100644
--- a/src/backend/optimizer/path/joinpath.c
+++ b/src/backend/optimizer/path/joinpath.c
@@ -30,8 +30,9 @@
set_join_pathlist_hook_type set_join_pathlist_hook = NULL;
/*
- * Paths parameterized by the parent can be considered to be parameterized by
- * any of its child.
+ * Paths parameterized by a parent rel can be considered to be parameterized
+ * by any of its children, when we are performing partitionwise joins. These
+ * macros simplify checking for such cases. Beware multiple eval of args.
*/
#define PATH_PARAM_BY_PARENT(path, rel) \
((path)->param_info && bms_overlap(PATH_REQ_OUTER(path), \
@@ -765,6 +766,20 @@ try_nestloop_path(PlannerInfo *root,
/* If we got past that, we shouldn't have any unsafe outer-join refs */
Assert(!have_unsafe_outer_join_ref(root, outerrelids, inner_paramrels));
+ /*
+ * If the inner path is parameterized, it is parameterized by the topmost
+ * parent of the outer rel, not the outer rel itself. We will need to
+ * translate the parameterization, if this path is chosen, during
+ * create_plan(). Here we just check whether we will be able to perform
+ * the translation, and if not avoid creating a nestloop path.
+ */
+ if (PATH_PARAM_BY_PARENT(inner_path, outer_path->parent) &&
+ !path_is_reparameterizable_by_child(inner_path, outer_path->parent))
+ {
+ bms_free(required_outer);
+ return;
+ }
+
/*
* Do a precheck to quickly eliminate obviously-inferior paths. We
* calculate a cheap lower bound on the path's cost and then use
@@ -781,27 +796,6 @@ try_nestloop_path(PlannerInfo *root,
workspace.startup_cost, workspace.total_cost,
pathkeys, required_outer))
{
- /*
- * If the inner path is parameterized, it is parameterized by the
- * topmost parent of the outer rel, not the outer rel itself. Fix
- * that.
- */
- if (PATH_PARAM_BY_PARENT(inner_path, outer_path->parent))
- {
- inner_path = reparameterize_path_by_child(root, inner_path,
- outer_path->parent);
-
- /*
- * If we could not translate the path, we can't create nest loop
- * path.
- */
- if (!inner_path)
- {
- bms_free(required_outer);
- return;
- }
- }
-
add_path(joinrel, (Path *)
create_nestloop_path(root,
joinrel,
@@ -864,6 +858,17 @@ try_partial_nestloop_path(PlannerInfo *root,
return;
}
+ /*
+ * If the inner path is parameterized, it is parameterized by the topmost
+ * parent of the outer rel, not the outer rel itself. We will need to
+ * translate the parameterization, if this path is chosen, during
+ * create_plan(). Here we just check whether we will be able to perform
+ * the translation, and if not avoid creating a nestloop path.
+ */
+ if (PATH_PARAM_BY_PARENT(inner_path, outer_path->parent) &&
+ !path_is_reparameterizable_by_child(inner_path, outer_path->parent))
+ return;
+
/*
* Before creating a path, get a quick lower bound on what it is likely to
* cost. Bail out right away if it looks terrible.
@@ -873,22 +878,6 @@ try_partial_nestloop_path(PlannerInfo *root,
if (!add_partial_path_precheck(joinrel, workspace.total_cost, pathkeys))
return;
- /*
- * If the inner path is parameterized, it is parameterized by the topmost
- * parent of the outer rel, not the outer rel itself. Fix that.
- */
- if (PATH_PARAM_BY_PARENT(inner_path, outer_path->parent))
- {
- inner_path = reparameterize_path_by_child(root, inner_path,
- outer_path->parent);
-
- /*
- * If we could not translate the path, we can't create nest loop path.
- */
- if (!inner_path)
- return;
- }
-
/* Might be good enough to be worth trying, so let's try it. */
add_partial_path(joinrel, (Path *)
create_nestloop_path(root,
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index ca619eab94..415fd02806 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -29,6 +29,7 @@
#include "optimizer/cost.h"
#include "optimizer/optimizer.h"
#include "optimizer/paramassign.h"
+#include "optimizer/pathnode.h"
#include "optimizer/paths.h"
#include "optimizer/placeholder.h"
#include "optimizer/plancat.h"
@@ -4346,6 +4347,22 @@ create_nestloop_plan(PlannerInfo *root,
List *nestParams;
Relids saveOuterRels = root->curOuterRels;
+ /*
+ * If the inner path is parameterized by the topmost parent of the outer
+ * rel rather than the outer rel itself, fix that. (Nothing happens here
+ * if it is not so parameterized.)
+ */
+ best_path->jpath.innerjoinpath =
+ reparameterize_path_by_child(root,
+ best_path->jpath.innerjoinpath,
+ best_path->jpath.outerjoinpath->parent);
+
+ /*
+ * Failure here probably means that reparameterize_path_by_child() is not
+ * in sync with path_is_reparameterizable_by_child().
+ */
+ Assert(best_path->jpath.innerjoinpath != NULL);
+
/* NestLoop can project, so no need to be picky about child tlists */
outer_plan = create_plan_recurse(root, best_path->jpath.outerjoinpath, 0);
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index f7ac71087a..75563d972f 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -56,6 +56,8 @@ static int append_startup_cost_compare(const ListCell *a, const ListCell *b);
static List *reparameterize_pathlist_by_child(PlannerInfo *root,
List *pathlist,
RelOptInfo *child_rel);
+static bool pathlist_is_reparameterizable_by_child(List *pathlist,
+ RelOptInfo *child_rel);
/*****************************************************************************
@@ -2436,6 +2438,16 @@ create_nestloop_path(PlannerInfo *root,
{
NestPath *pathnode = makeNode(NestPath);
Relids inner_req_outer = PATH_REQ_OUTER(inner_path);
+ Relids outerrelids;
+
+ /*
+ * Paths are parameterized by top-level parents, so run parameterization
+ * tests on the parent relids.
+ */
+ if (outer_path->parent->top_parent_relids)
+ outerrelids = outer_path->parent->top_parent_relids;
+ else
+ outerrelids = outer_path->parent->relids;
/*
* If the inner path is parameterized by the outer, we must drop any
@@ -2445,7 +2457,7 @@ create_nestloop_path(PlannerInfo *root,
* estimates for this path. We detect such clauses by checking for serial
* number match to clauses already enforced in the inner path.
*/
- if (bms_overlap(inner_req_outer, outer_path->parent->relids))
+ if (bms_overlap(inner_req_outer, outerrelids))
{
Bitmapset *enforced_serials = get_param_path_clause_serials(inner_path);
List *jclauses = NIL;
@@ -4045,32 +4057,33 @@ reparameterize_path(PlannerInfo *root, Path *path,
* Given a path parameterized by the parent of the given child relation,
* translate the path to be parameterized by the given child relation.
*
- * The function creates a new path of the same type as the given path, but
- * parameterized by the given child relation. Most fields from the original
- * path can simply be flat-copied, but any expressions must be adjusted to
- * refer to the correct varnos, and any paths must be recursively
- * reparameterized. Other fields that refer to specific relids also need
- * adjustment.
+ * The function translates the given path to be parameterized by the given
+ * child relation. Most fields from the original path are not changed, but any
+ * expressions must be adjusted to refer to the correct varnos, and any paths
+ * must be recursively reparameterized. Other fields that refer to specific
+ * relids also need adjustment.
*
* The cost, number of rows, width and parallel path properties depend upon
- * path->parent, which does not change during the translation. Hence those
- * members are copied as they are.
+ * path->parent, which does not change during the translation.
*
* Currently, only a few path types are supported here, though more could be
* added at need. We return NULL if we can't reparameterize the given path.
+ *
+ * Note that this function can change referenced RTEs, RelOptInfos and
+ * IndexOptInfos as well as the Path structures. Therefore, it's only safe
+ * to call during create_plan(), when we have made a final choice of which
+ * Path to use for each RTE/RelOptInfo/IndexOptInfo.
+ *
+ * Keep this code in sync with path_is_reparameterizable_by_child()!
*/
Path *
reparameterize_path_by_child(PlannerInfo *root, Path *path,
RelOptInfo *child_rel)
{
-#define FLAT_COPY_PATH(newnode, node, nodetype) \
- ( (newnode) = makeNode(nodetype), \
- memcpy((newnode), (node), sizeof(nodetype)) )
-
#define ADJUST_CHILD_ATTRS(node) \
((node) = \
- (List *) adjust_appendrel_attrs_multilevel(root, (Node *) (node), \
+ (void *) adjust_appendrel_attrs_multilevel(root, (Node *) (node), \
child_rel, \
child_rel->top_parent))
@@ -4098,15 +4111,15 @@ do { \
Relids required_outer;
/*
- * If the path is not parameterized by parent of the given relation, it
- * doesn't need reparameterization.
+ * If the path is not parameterized by the parent of the given relation,
+ * it doesn't need reparameterization.
*/
if (!path->param_info ||
!bms_overlap(PATH_REQ_OUTER(path), child_rel->top_parent_relids))
return path;
/*
- * If possible, reparameterize the given path, making a copy.
+ * If possible, reparameterize the given path.
*
* This function is currently only applied to the inner side of a nestloop
* join that is being partitioned by the partitionwise-join code. Hence,
@@ -4120,14 +4133,29 @@ do { \
switch (nodeTag(path))
{
case T_Path:
- FLAT_COPY_PATH(new_path, path, Path);
+ new_path = path;
+ ADJUST_CHILD_ATTRS(new_path->parent->baserestrictinfo);
+ if (path->pathtype == T_SampleScan)
+ {
+ Index scan_relid = path->parent->relid;
+ RangeTblEntry *rte;
+
+ /* it should be a base rel with a tablesample clause... */
+ Assert(scan_relid > 0);
+ rte = planner_rt_fetch(scan_relid, root);
+ Assert(rte->rtekind == RTE_RELATION);
+ Assert(rte->tablesample != NULL);
+
+ ADJUST_CHILD_ATTRS(rte->tablesample);
+ }
break;
case T_IndexPath:
{
IndexPath *ipath;
- FLAT_COPY_PATH(ipath, path, IndexPath);
+ ipath = (IndexPath *) path;
+ ADJUST_CHILD_ATTRS(ipath->indexinfo->indrestrictinfo);
ADJUST_CHILD_ATTRS(ipath->indexclauses);
new_path = (Path *) ipath;
}
@@ -4137,7 +4165,8 @@ do { \
{
BitmapHeapPath *bhpath;
- FLAT_COPY_PATH(bhpath, path, BitmapHeapPath);
+ bhpath = (BitmapHeapPath *) path;
+ ADJUST_CHILD_ATTRS(bhpath->path.parent->baserestrictinfo);
REPARAMETERIZE_CHILD_PATH(bhpath->bitmapqual);
new_path = (Path *) bhpath;
}
@@ -4147,7 +4176,7 @@ do { \
{
BitmapAndPath *bapath;
- FLAT_COPY_PATH(bapath, path, BitmapAndPath);
+ bapath = (BitmapAndPath *) path;
REPARAMETERIZE_CHILD_PATH_LIST(bapath->bitmapquals);
new_path = (Path *) bapath;
}
@@ -4157,7 +4186,7 @@ do { \
{
BitmapOrPath *bopath;
- FLAT_COPY_PATH(bopath, path, BitmapOrPath);
+ bopath = (BitmapOrPath *) path;
REPARAMETERIZE_CHILD_PATH_LIST(bopath->bitmapquals);
new_path = (Path *) bopath;
}
@@ -4168,7 +4197,8 @@ do { \
ForeignPath *fpath;
ReparameterizeForeignPathByChild_function rfpc_func;
- FLAT_COPY_PATH(fpath, path, ForeignPath);
+ fpath = (ForeignPath *) path;
+ ADJUST_CHILD_ATTRS(fpath->path.parent->baserestrictinfo);
if (fpath->fdw_outerpath)
REPARAMETERIZE_CHILD_PATH(fpath->fdw_outerpath);
if (fpath->fdw_restrictinfo)
@@ -4188,7 +4218,8 @@ do { \
{
CustomPath *cpath;
- FLAT_COPY_PATH(cpath, path, CustomPath);
+ cpath = (CustomPath *) path;
+ ADJUST_CHILD_ATTRS(cpath->path.parent->baserestrictinfo);
REPARAMETERIZE_CHILD_PATH_LIST(cpath->custom_paths);
if (cpath->custom_restrictinfo)
ADJUST_CHILD_ATTRS(cpath->custom_restrictinfo);
@@ -4207,7 +4238,7 @@ do { \
JoinPath *jpath;
NestPath *npath;
- FLAT_COPY_PATH(npath, path, NestPath);
+ npath = (NestPath *) path;
jpath = (JoinPath *) npath;
REPARAMETERIZE_CHILD_PATH(jpath->outerjoinpath);
@@ -4222,7 +4253,7 @@ do { \
JoinPath *jpath;
MergePath *mpath;
- FLAT_COPY_PATH(mpath, path, MergePath);
+ mpath = (MergePath *) path;
jpath = (JoinPath *) mpath;
REPARAMETERIZE_CHILD_PATH(jpath->outerjoinpath);
@@ -4238,7 +4269,7 @@ do { \
JoinPath *jpath;
HashPath *hpath;
- FLAT_COPY_PATH(hpath, path, HashPath);
+ hpath = (HashPath *) path;
jpath = (JoinPath *) hpath;
REPARAMETERIZE_CHILD_PATH(jpath->outerjoinpath);
@@ -4253,7 +4284,7 @@ do { \
{
AppendPath *apath;
- FLAT_COPY_PATH(apath, path, AppendPath);
+ apath = (AppendPath *) path;
REPARAMETERIZE_CHILD_PATH_LIST(apath->subpaths);
new_path = (Path *) apath;
}
@@ -4263,7 +4294,7 @@ do { \
{
MaterialPath *mpath;
- FLAT_COPY_PATH(mpath, path, MaterialPath);
+ mpath = (MaterialPath *) path;
REPARAMETERIZE_CHILD_PATH(mpath->subpath);
new_path = (Path *) mpath;
}
@@ -4273,7 +4304,7 @@ do { \
{
MemoizePath *mpath;
- FLAT_COPY_PATH(mpath, path, MemoizePath);
+ mpath = (MemoizePath *) path;
REPARAMETERIZE_CHILD_PATH(mpath->subpath);
ADJUST_CHILD_ATTRS(mpath->param_exprs);
new_path = (Path *) mpath;
@@ -4284,7 +4315,7 @@ do { \
{
GatherPath *gpath;
- FLAT_COPY_PATH(gpath, path, GatherPath);
+ gpath = (GatherPath *) path;
REPARAMETERIZE_CHILD_PATH(gpath->subpath);
new_path = (Path *) gpath;
}
@@ -4351,9 +4382,145 @@ do { \
return new_path;
}
+/*
+ * path_is_reparameterizable_by_child
+ * Given a path parameterized by the parent of the given child relation,
+ * see if it can be translated to be parameterized by the child relation.
+ *
+ * This must return true if and only if reparameterize_path_by_child()
+ * would succeed on this path. Currently it's sufficient to verify that
+ * the path and all of its subpaths (if any) are of the types handled by
+ * that function. However, sub-paths that are not parameterized can be
+ * disregarded since they won't require translation.
+ */
+bool
+path_is_reparameterizable_by_child(Path *path, RelOptInfo *child_rel)
+{
+
+#define REJECT_IF_PATH_NOT_REPARAMETERIZABLE(path) \
+do { \
+ if (!path_is_reparameterizable_by_child(path, child_rel)) \
+ return false; \
+} while(0)
+
+#define REJECT_IF_PATH_LIST_NOT_REPARAMETERIZABLE(pathlist) \
+do { \
+ if (!pathlist_is_reparameterizable_by_child(pathlist, child_rel)) \
+ return false; \
+} while(0)
+
+ /*
+ * If the path is not parameterized by the parent of the given relation,
+ * it doesn't need reparameterization.
+ */
+ if (!path->param_info ||
+ !bms_overlap(PATH_REQ_OUTER(path), child_rel->top_parent_relids))
+ return true;
+
+ switch (nodeTag(path))
+ {
+ case T_Path:
+ case T_IndexPath:
+ break;
+
+ case T_BitmapHeapPath:
+ {
+ BitmapHeapPath *bhpath = (BitmapHeapPath *) path;
+
+ REJECT_IF_PATH_NOT_REPARAMETERIZABLE(bhpath->bitmapqual);
+ }
+ break;
+
+ case T_BitmapAndPath:
+ {
+ BitmapAndPath *bapath = (BitmapAndPath *) path;
+
+ REJECT_IF_PATH_LIST_NOT_REPARAMETERIZABLE(bapath->bitmapquals);
+ }
+ break;
+
+ case T_BitmapOrPath:
+ {
+ BitmapOrPath *bopath = (BitmapOrPath *) path;
+
+ REJECT_IF_PATH_LIST_NOT_REPARAMETERIZABLE(bopath->bitmapquals);
+ }
+ break;
+
+ case T_ForeignPath:
+ {
+ ForeignPath *fpath = (ForeignPath *) path;
+
+ if (fpath->fdw_outerpath)
+ REJECT_IF_PATH_NOT_REPARAMETERIZABLE(fpath->fdw_outerpath);
+ }
+ break;
+
+ case T_CustomPath:
+ {
+ CustomPath *cpath = (CustomPath *) path;
+
+ REJECT_IF_PATH_LIST_NOT_REPARAMETERIZABLE(cpath->custom_paths);
+ }
+ break;
+
+ case T_NestPath:
+ case T_MergePath:
+ case T_HashPath:
+ {
+ JoinPath *jpath = (JoinPath *) path;
+
+ REJECT_IF_PATH_NOT_REPARAMETERIZABLE(jpath->outerjoinpath);
+ REJECT_IF_PATH_NOT_REPARAMETERIZABLE(jpath->innerjoinpath);
+ }
+ break;
+
+ case T_AppendPath:
+ {
+ AppendPath *apath = (AppendPath *) path;
+
+ REJECT_IF_PATH_LIST_NOT_REPARAMETERIZABLE(apath->subpaths);
+ }
+ break;
+
+ case T_MaterialPath:
+ {
+ MaterialPath *mpath = (MaterialPath *) path;
+
+ REJECT_IF_PATH_NOT_REPARAMETERIZABLE(mpath->subpath);
+ }
+ break;
+
+ case T_MemoizePath:
+ {
+ MemoizePath *mpath = (MemoizePath *) path;
+
+ REJECT_IF_PATH_NOT_REPARAMETERIZABLE(mpath->subpath);
+ }
+ break;
+
+ case T_GatherPath:
+ {
+ GatherPath *gpath = (GatherPath *) path;
+
+ REJECT_IF_PATH_NOT_REPARAMETERIZABLE(gpath->subpath);
+ }
+ break;
+
+ default:
+
+ /* We don't know how to reparameterize this path. */
+ return false;
+ }
+
+ return true;
+}
+
/*
* reparameterize_pathlist_by_child
* Helper function to reparameterize a list of paths by given child rel.
+ *
+ * Returns NIL to indicate failure, so pathlist had better not be NIL.
*/
static List *
reparameterize_pathlist_by_child(PlannerInfo *root,
@@ -4379,3 +4546,23 @@ reparameterize_pathlist_by_child(PlannerInfo *root,
return result;
}
+
+/*
+ * pathlist_is_reparameterizable_by_child
+ * Helper function to check if a list of paths can be reparameterized.
+ */
+static bool
+pathlist_is_reparameterizable_by_child(List *pathlist, RelOptInfo *child_rel)
+{
+ ListCell *lc;
+
+ foreach(lc, pathlist)
+ {
+ Path *path = (Path *) lfirst(lc);
+
+ if (!path_is_reparameterizable_by_child(path, child_rel))
+ return false;
+ }
+
+ return true;
+}
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index 03a0e46e70..ec55d15b4b 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -298,6 +298,8 @@ extern Path *reparameterize_path(PlannerInfo *root, Path *path,
double loop_count);
extern Path *reparameterize_path_by_child(PlannerInfo *root, Path *path,
RelOptInfo *child_rel);
+extern bool path_is_reparameterizable_by_child(Path *path,
+ RelOptInfo *child_rel);
/*
* prototypes for relnode.c
diff --git a/src/test/regress/expected/partition_join.out b/src/test/regress/expected/partition_join.out
index 6560fe2416..e713f9fb2f 100644
--- a/src/test/regress/expected/partition_join.out
+++ b/src/test/regress/expected/partition_join.out
@@ -505,6 +505,65 @@ SELECT t1.a, ss.t2a, ss.t2c FROM prt1 t1 LEFT JOIN LATERAL
550 | |
(12 rows)
+-- lateral reference in sample scan
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt1 t1 JOIN LATERAL
+ (SELECT * FROM prt1 t2 TABLESAMPLE SYSTEM (t1.a) REPEATABLE(t1.b)) s
+ ON t1.a = s.a;
+ QUERY PLAN
+-------------------------------------------------------------
+ Append
+ -> Nested Loop
+ -> Seq Scan on prt1_p1 t1_1
+ -> Sample Scan on prt1_p1 t2_1
+ Sampling: system (t1_1.a) REPEATABLE (t1_1.b)
+ Filter: (t1_1.a = a)
+ -> Nested Loop
+ -> Seq Scan on prt1_p2 t1_2
+ -> Sample Scan on prt1_p2 t2_2
+ Sampling: system (t1_2.a) REPEATABLE (t1_2.b)
+ Filter: (t1_2.a = a)
+ -> Nested Loop
+ -> Seq Scan on prt1_p3 t1_3
+ -> Sample Scan on prt1_p3 t2_3
+ Sampling: system (t1_3.a) REPEATABLE (t1_3.b)
+ Filter: (t1_3.a = a)
+(16 rows)
+
+-- lateral reference in scan's restriction clauses
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.a;
+ QUERY PLAN
+---------------------------------------------------------------
+ Aggregate
+ -> Append
+ -> Nested Loop
+ -> Seq Scan on prt1_p1 t1_1
+ -> Index Scan using iprt2_p1_b on prt2_p1 t2_1
+ Index Cond: (b = t1_1.a)
+ Filter: (t1_1.b = a)
+ -> Nested Loop
+ -> Seq Scan on prt1_p2 t1_2
+ -> Index Scan using iprt2_p2_b on prt2_p2 t2_2
+ Index Cond: (b = t1_2.a)
+ Filter: (t1_2.b = a)
+ -> Nested Loop
+ -> Seq Scan on prt1_p3 t1_3
+ -> Index Scan using iprt2_p3_b on prt2_p3 t2_3
+ Index Cond: (b = t1_3.a)
+ Filter: (t1_3.b = a)
+(17 rows)
+
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.a;
+ count
+-------
+ 100
+(1 row)
+
-- bug with inadequate sort key representation
SET enable_partitionwise_aggregate TO true;
SET enable_hashjoin TO false;
@@ -1944,6 +2003,41 @@ SELECT * FROM prt1_l t1 LEFT JOIN LATERAL
550 | 0 | 0002 | | | | |
(12 rows)
+-- partitionwise join with lateral reference in sample scan
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt1_l t1 JOIN LATERAL
+ (SELECT * FROM prt1_l t2 TABLESAMPLE SYSTEM (t1.a) REPEATABLE(t1.b)) s ON
+ t1.a = s.a AND t1.b = s.b AND t1.c = s.c;
+ QUERY PLAN
+----------------------------------------------------------------------------------------
+ Append
+ -> Nested Loop
+ -> Seq Scan on prt1_l_p1 t1_1
+ -> Sample Scan on prt1_l_p1 t2_1
+ Sampling: system (t1_1.a) REPEATABLE (t1_1.b)
+ Filter: ((t1_1.a = a) AND (t1_1.b = b) AND ((t1_1.c)::text = (c)::text))
+ -> Nested Loop
+ -> Seq Scan on prt1_l_p2_p1 t1_2
+ -> Sample Scan on prt1_l_p2_p1 t2_2
+ Sampling: system (t1_2.a) REPEATABLE (t1_2.b)
+ Filter: ((t1_2.a = a) AND (t1_2.b = b) AND ((t1_2.c)::text = (c)::text))
+ -> Nested Loop
+ -> Seq Scan on prt1_l_p2_p2 t1_3
+ -> Sample Scan on prt1_l_p2_p2 t2_3
+ Sampling: system (t1_3.a) REPEATABLE (t1_3.b)
+ Filter: ((t1_3.a = a) AND (t1_3.b = b) AND ((t1_3.c)::text = (c)::text))
+ -> Nested Loop
+ -> Seq Scan on prt1_l_p3_p1 t1_4
+ -> Sample Scan on prt1_l_p3_p1 t2_4
+ Sampling: system (t1_4.a) REPEATABLE (t1_4.b)
+ Filter: ((t1_4.a = a) AND (t1_4.b = b) AND ((t1_4.c)::text = (c)::text))
+ -> Nested Loop
+ -> Seq Scan on prt1_l_p3_p2 t1_5
+ -> Sample Scan on prt1_l_p3_p2 t2_5
+ Sampling: system (t1_5.a) REPEATABLE (t1_5.b)
+ Filter: ((t1_5.a = a) AND (t1_5.b = b) AND ((t1_5.c)::text = (c)::text))
+(26 rows)
+
-- join with one side empty
EXPLAIN (COSTS OFF)
SELECT t1.a, t1.c, t2.b, t2.c FROM (SELECT * FROM prt1_l WHERE a = 1 AND a = 2) t1 RIGHT JOIN prt2_l t2 ON t1.a = t2.b AND t1.b = t2.a AND t1.c = t2.c;
diff --git a/src/test/regress/sql/partition_join.sql b/src/test/regress/sql/partition_join.sql
index 48daf3aee3..5ab3148dec 100644
--- a/src/test/regress/sql/partition_join.sql
+++ b/src/test/regress/sql/partition_join.sql
@@ -100,6 +100,21 @@ SELECT t1.a, ss.t2a, ss.t2c FROM prt1 t1 LEFT JOIN LATERAL
(SELECT t2.a AS t2a, t3.a AS t3a, t2.b t2b, t2.c t2c, least(t1.a,t2.a,t3.a) FROM prt1 t2 JOIN prt2 t3 ON (t2.a = t3.b)) ss
ON t1.c = ss.t2c WHERE (t1.b + coalesce(ss.t2b, 0)) = 0 ORDER BY t1.a;
+-- lateral reference in sample scan
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt1 t1 JOIN LATERAL
+ (SELECT * FROM prt1 t2 TABLESAMPLE SYSTEM (t1.a) REPEATABLE(t1.b)) s
+ ON t1.a = s.a;
+
+-- lateral reference in scan's restriction clauses
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.a;
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.a;
+
-- bug with inadequate sort key representation
SET enable_partitionwise_aggregate TO true;
SET enable_hashjoin TO false;
@@ -387,6 +402,12 @@ SELECT * FROM prt1_l t1 LEFT JOIN LATERAL
(SELECT t2.a AS t2a, t2.c AS t2c, t2.b AS t2b, t3.b AS t3b, least(t1.a,t2.a,t3.b) FROM prt1_l t2 JOIN prt2_l t3 ON (t2.a = t3.b AND t2.c = t3.c)) ss
ON t1.a = ss.t2a AND t1.c = ss.t2c WHERE t1.b = 0 ORDER BY t1.a;
+-- partitionwise join with lateral reference in sample scan
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt1_l t1 JOIN LATERAL
+ (SELECT * FROM prt1_l t2 TABLESAMPLE SYSTEM (t1.a) REPEATABLE(t1.b)) s ON
+ t1.a = s.a AND t1.b = s.b AND t1.c = s.c;
+
-- join with one side empty
EXPLAIN (COSTS OFF)
SELECT t1.a, t1.c, t2.b, t2.c FROM (SELECT * FROM prt1_l WHERE a = 1 AND a = 2) t1 RIGHT JOIN prt2_l t2 ON t1.a = t2.b AND t1.b = t2.a AND t1.c = t2.c;
--
2.31.0
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Oversight in reparameterize_path_by_child leading to executor crash
@ 2024-01-15 18:30 Robert Haas <[email protected]>
parent: Richard Guo <[email protected]>
0 siblings, 2 replies; 86+ messages in thread
From: Robert Haas @ 2024-01-15 18:30 UTC (permalink / raw)
To: Richard Guo <[email protected]>; +Cc: Andrei Lepikhov <[email protected]>; Alena Rybakina <[email protected]>; Tom Lane <[email protected]>; Ashutosh Bapat <[email protected]>; pgsql-hackers
On Mon, Jan 8, 2024 at 3:32 AM Richard Guo <[email protected]> wrote:
> Thanks for the suggestion. Attached is an updated patch which is added
> with a commit message that tries to explain the problem and the fix.
This is great. The references to "the sampling infos" are a little bit
confusing to me. I'm not sure if this is referring to
TableSampleClause objects or what.
> Fair point. I think we can back-patch a more minimal fix, as Tom
> proposed in [1], which disallows the reparameterization if the path
> contains sampling info that references the outer rel. But I think we
> need also to disallow the reparameterization of a path if it contains
> restriction clauses that reference the outer rel, as such paths have
> been found to cause incorrect results, or planning errors as in [2].
Do you want to produce a patch for that, to go along with this patch for master?
I know this is on Tom's to-do list which makes me a bit reluctant to
get too involved here, because certainly he knows this code better
than I do, maybe better than anyone does, but on the other hand, we
shouldn't leave server crashes unfixed for too long, so maybe I can do
something to help at least with that part of it.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Oversight in reparameterize_path_by_child leading to executor crash
@ 2024-01-15 19:17 Tom Lane <[email protected]>
parent: Robert Haas <[email protected]>
1 sibling, 0 replies; 86+ messages in thread
From: Tom Lane @ 2024-01-15 19:17 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Richard Guo <[email protected]>; Andrei Lepikhov <[email protected]>; Alena Rybakina <[email protected]>; Ashutosh Bapat <[email protected]>; pgsql-hackers
Robert Haas <[email protected]> writes:
> I know this is on Tom's to-do list which makes me a bit reluctant to
> get too involved here, because certainly he knows this code better
> than I do, maybe better than anyone does, but on the other hand, we
> shouldn't leave server crashes unfixed for too long, so maybe I can do
> something to help at least with that part of it.
This is indeed on my to-do list, and I have every intention of
getting to it before the end of the month. But if you want to
help push things along, feel free.
regards, tom lane
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Oversight in reparameterize_path_by_child leading to executor crash
@ 2024-01-17 09:01 Richard Guo <[email protected]>
parent: Robert Haas <[email protected]>
1 sibling, 1 reply; 86+ messages in thread
From: Richard Guo @ 2024-01-17 09:01 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Andrei Lepikhov <[email protected]>; Alena Rybakina <[email protected]>; Tom Lane <[email protected]>; Ashutosh Bapat <[email protected]>; pgsql-hackers
On Tue, Jan 16, 2024 at 2:30 AM Robert Haas <[email protected]> wrote:
> On Mon, Jan 8, 2024 at 3:32 AM Richard Guo <[email protected]> wrote:
> > Thanks for the suggestion. Attached is an updated patch which is added
> > with a commit message that tries to explain the problem and the fix.
>
> This is great. The references to "the sampling infos" are a little bit
> confusing to me. I'm not sure if this is referring to
> TableSampleClause objects or what.
Yeah, it's referring to TableSampleClause objects. I've updated the
commit message to clarify that. In passing I also updated the test
cases a bit. Please see
v10-0001-Postpone-reparameterization-of-paths-until-when-creating-plans.patch
> > Fair point. I think we can back-patch a more minimal fix, as Tom
> > proposed in [1], which disallows the reparameterization if the path
> > contains sampling info that references the outer rel. But I think we
> > need also to disallow the reparameterization of a path if it contains
> > restriction clauses that reference the outer rel, as such paths have
> > been found to cause incorrect results, or planning errors as in [2].
>
> Do you want to produce a patch for that, to go along with this patch for
> master?
Sure, here it is:
v10-0001-Avoid-reparameterizing-Paths-when-it-s-not-suitable.patch
Thanks
Richard
Attachments:
[application/octet-stream] v10-0001-Postpone-reparameterization-of-paths-until-when-creating-plans.patch (30.5K, ../../CAMbWs48ZfxwRobV-QP1mt_16+2iDbfw_pdsnuzmTJntCPCdD0w@mail.gmail.com/3-v10-0001-Postpone-reparameterization-of-paths-until-when-creating-plans.patch)
download | inline diff:
From c9a3a8df98ffb3320f094a7b1ad81c48b064877c Mon Sep 17 00:00:00 2001
From: Richard Guo <[email protected]>
Date: Mon, 8 Jan 2024 13:47:45 +0800
Subject: [PATCH v10] Postpone reparameterization of paths until when creating
plans
When creating a nestloop join path, if the inner path is parameterized,
it is parameterized by the topmost parent of the outer rel, not the
outer rel itself. Therefore, we need to translate the parameterization
so that the inner path is parameterized by the given outer rel itself.
Currently, this is done in reparameterize_path_by_child() during
generating join paths.
However, reparameterize_path_by_child() does not perform the translation
for sample scan paths' sampling infos, or scan paths' restriction
clauses. This omission can lead to executor crashes, wrong results, or
planning errors, as we have already observed.
Please note that the sampling infos are contained in RangeTblEntries (as
TableSampleClause objects), and the scan restriction clauses are
contained in RelOptInfos or IndexOptInfos (as lists of RestrictInfo
objects). We cannot just modify them on the fly during generating join
paths. Doing so would break things if we end up using a
non-partitionwise join.
So this commit postpones reparameterization of the inner path until
createplan.c, where it is safe to modify referenced RangeTblEntry,
RelOptInfo or IndexOptInfo, because we have made a final choice of which
Path to use. This can only work if we know that we will be able to
perform the translation for the inner path. So we introduce a new
function path_is_reparameterizable_by_child() during generating join
paths to check whether the given path can be translated.
As an additional benefit, this commit also helps avoid building the
reparameterized expressions we might not use, resulting in saved CPU
cycles and reduced memory usage.
---
src/backend/optimizer/path/joinpath.c | 67 +++--
src/backend/optimizer/plan/createplan.c | 17 ++
src/backend/optimizer/util/pathnode.c | 249 ++++++++++++++++---
src/include/optimizer/pathnode.h | 2 +
src/test/regress/expected/partition_join.out | 168 +++++++++++++
src/test/regress/sql/partition_join.sql | 40 +++
6 files changed, 473 insertions(+), 70 deletions(-)
diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c
index e9def9d540..aecf2774de 100644
--- a/src/backend/optimizer/path/joinpath.c
+++ b/src/backend/optimizer/path/joinpath.c
@@ -30,8 +30,9 @@
set_join_pathlist_hook_type set_join_pathlist_hook = NULL;
/*
- * Paths parameterized by the parent can be considered to be parameterized by
- * any of its child.
+ * Paths parameterized by a parent rel can be considered to be parameterized
+ * by any of its children, when we are performing partitionwise joins. These
+ * macros simplify checking for such cases. Beware multiple eval of args.
*/
#define PATH_PARAM_BY_PARENT(path, rel) \
((path)->param_info && bms_overlap(PATH_REQ_OUTER(path), \
@@ -768,6 +769,20 @@ try_nestloop_path(PlannerInfo *root,
/* If we got past that, we shouldn't have any unsafe outer-join refs */
Assert(!have_unsafe_outer_join_ref(root, outerrelids, inner_paramrels));
+ /*
+ * If the inner path is parameterized, it is parameterized by the topmost
+ * parent of the outer rel, not the outer rel itself. We will need to
+ * translate the parameterization, if this path is chosen, during
+ * create_plan(). Here we just check whether we will be able to perform
+ * the translation, and if not avoid creating a nestloop path.
+ */
+ if (PATH_PARAM_BY_PARENT(inner_path, outer_path->parent) &&
+ !path_is_reparameterizable_by_child(inner_path, outer_path->parent))
+ {
+ bms_free(required_outer);
+ return;
+ }
+
/*
* Do a precheck to quickly eliminate obviously-inferior paths. We
* calculate a cheap lower bound on the path's cost and then use
@@ -784,27 +799,6 @@ try_nestloop_path(PlannerInfo *root,
workspace.startup_cost, workspace.total_cost,
pathkeys, required_outer))
{
- /*
- * If the inner path is parameterized, it is parameterized by the
- * topmost parent of the outer rel, not the outer rel itself. Fix
- * that.
- */
- if (PATH_PARAM_BY_PARENT(inner_path, outer_path->parent))
- {
- inner_path = reparameterize_path_by_child(root, inner_path,
- outer_path->parent);
-
- /*
- * If we could not translate the path, we can't create nest loop
- * path.
- */
- if (!inner_path)
- {
- bms_free(required_outer);
- return;
- }
- }
-
add_path(joinrel, (Path *)
create_nestloop_path(root,
joinrel,
@@ -867,6 +861,17 @@ try_partial_nestloop_path(PlannerInfo *root,
return;
}
+ /*
+ * If the inner path is parameterized, it is parameterized by the topmost
+ * parent of the outer rel, not the outer rel itself. We will need to
+ * translate the parameterization, if this path is chosen, during
+ * create_plan(). Here we just check whether we will be able to perform
+ * the translation, and if not avoid creating a nestloop path.
+ */
+ if (PATH_PARAM_BY_PARENT(inner_path, outer_path->parent) &&
+ !path_is_reparameterizable_by_child(inner_path, outer_path->parent))
+ return;
+
/*
* Before creating a path, get a quick lower bound on what it is likely to
* cost. Bail out right away if it looks terrible.
@@ -876,22 +881,6 @@ try_partial_nestloop_path(PlannerInfo *root,
if (!add_partial_path_precheck(joinrel, workspace.total_cost, pathkeys))
return;
- /*
- * If the inner path is parameterized, it is parameterized by the topmost
- * parent of the outer rel, not the outer rel itself. Fix that.
- */
- if (PATH_PARAM_BY_PARENT(inner_path, outer_path->parent))
- {
- inner_path = reparameterize_path_by_child(root, inner_path,
- outer_path->parent);
-
- /*
- * If we could not translate the path, we can't create nest loop path.
- */
- if (!inner_path)
- return;
- }
-
/* Might be good enough to be worth trying, so let's try it. */
add_partial_path(joinrel, (Path *)
create_nestloop_path(root,
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index ca619eab94..415fd02806 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -29,6 +29,7 @@
#include "optimizer/cost.h"
#include "optimizer/optimizer.h"
#include "optimizer/paramassign.h"
+#include "optimizer/pathnode.h"
#include "optimizer/paths.h"
#include "optimizer/placeholder.h"
#include "optimizer/plancat.h"
@@ -4346,6 +4347,22 @@ create_nestloop_plan(PlannerInfo *root,
List *nestParams;
Relids saveOuterRels = root->curOuterRels;
+ /*
+ * If the inner path is parameterized by the topmost parent of the outer
+ * rel rather than the outer rel itself, fix that. (Nothing happens here
+ * if it is not so parameterized.)
+ */
+ best_path->jpath.innerjoinpath =
+ reparameterize_path_by_child(root,
+ best_path->jpath.innerjoinpath,
+ best_path->jpath.outerjoinpath->parent);
+
+ /*
+ * Failure here probably means that reparameterize_path_by_child() is not
+ * in sync with path_is_reparameterizable_by_child().
+ */
+ Assert(best_path->jpath.innerjoinpath != NULL);
+
/* NestLoop can project, so no need to be picky about child tlists */
outer_plan = create_plan_recurse(root, best_path->jpath.outerjoinpath, 0);
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index 8dbf790e89..c4447da560 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -56,6 +56,8 @@ static int append_startup_cost_compare(const ListCell *a, const ListCell *b);
static List *reparameterize_pathlist_by_child(PlannerInfo *root,
List *pathlist,
RelOptInfo *child_rel);
+static bool pathlist_is_reparameterizable_by_child(List *pathlist,
+ RelOptInfo *child_rel);
/*****************************************************************************
@@ -2458,6 +2460,16 @@ create_nestloop_path(PlannerInfo *root,
{
NestPath *pathnode = makeNode(NestPath);
Relids inner_req_outer = PATH_REQ_OUTER(inner_path);
+ Relids outerrelids;
+
+ /*
+ * Paths are parameterized by top-level parents, so run parameterization
+ * tests on the parent relids.
+ */
+ if (outer_path->parent->top_parent_relids)
+ outerrelids = outer_path->parent->top_parent_relids;
+ else
+ outerrelids = outer_path->parent->relids;
/*
* If the inner path is parameterized by the outer, we must drop any
@@ -2467,7 +2479,7 @@ create_nestloop_path(PlannerInfo *root,
* estimates for this path. We detect such clauses by checking for serial
* number match to clauses already enforced in the inner path.
*/
- if (bms_overlap(inner_req_outer, outer_path->parent->relids))
+ if (bms_overlap(inner_req_outer, outerrelids))
{
Bitmapset *enforced_serials = get_param_path_clause_serials(inner_path);
List *jclauses = NIL;
@@ -4067,32 +4079,33 @@ reparameterize_path(PlannerInfo *root, Path *path,
* Given a path parameterized by the parent of the given child relation,
* translate the path to be parameterized by the given child relation.
*
- * The function creates a new path of the same type as the given path, but
- * parameterized by the given child relation. Most fields from the original
- * path can simply be flat-copied, but any expressions must be adjusted to
- * refer to the correct varnos, and any paths must be recursively
- * reparameterized. Other fields that refer to specific relids also need
- * adjustment.
+ * The function translates the given path to be parameterized by the given
+ * child relation. Most fields from the original path are not changed, but any
+ * expressions must be adjusted to refer to the correct varnos, and any paths
+ * must be recursively reparameterized. Other fields that refer to specific
+ * relids also need adjustment.
*
* The cost, number of rows, width and parallel path properties depend upon
- * path->parent, which does not change during the translation. Hence those
- * members are copied as they are.
+ * path->parent, which does not change during the translation.
*
* Currently, only a few path types are supported here, though more could be
* added at need. We return NULL if we can't reparameterize the given path.
+ *
+ * Note that this function can change referenced RangeTblEntries, RelOptInfos
+ * and IndexOptInfos as well as the Path structures. Therefore, it's only safe
+ * to call during create_plan(), when we have made a final choice of which Path
+ * to use for each RangeTblEntry/RelOptInfo/IndexOptInfo.
+ *
+ * Keep this code in sync with path_is_reparameterizable_by_child()!
*/
Path *
reparameterize_path_by_child(PlannerInfo *root, Path *path,
RelOptInfo *child_rel)
{
-#define FLAT_COPY_PATH(newnode, node, nodetype) \
- ( (newnode) = makeNode(nodetype), \
- memcpy((newnode), (node), sizeof(nodetype)) )
-
#define ADJUST_CHILD_ATTRS(node) \
((node) = \
- (List *) adjust_appendrel_attrs_multilevel(root, (Node *) (node), \
+ (void *) adjust_appendrel_attrs_multilevel(root, (Node *) (node), \
child_rel, \
child_rel->top_parent))
@@ -4120,15 +4133,15 @@ do { \
Relids required_outer;
/*
- * If the path is not parameterized by parent of the given relation, it
- * doesn't need reparameterization.
+ * If the path is not parameterized by the parent of the given relation,
+ * it doesn't need reparameterization.
*/
if (!path->param_info ||
!bms_overlap(PATH_REQ_OUTER(path), child_rel->top_parent_relids))
return path;
/*
- * If possible, reparameterize the given path, making a copy.
+ * If possible, reparameterize the given path.
*
* This function is currently only applied to the inner side of a nestloop
* join that is being partitioned by the partitionwise-join code. Hence,
@@ -4142,14 +4155,29 @@ do { \
switch (nodeTag(path))
{
case T_Path:
- FLAT_COPY_PATH(new_path, path, Path);
+ new_path = path;
+ ADJUST_CHILD_ATTRS(new_path->parent->baserestrictinfo);
+ if (path->pathtype == T_SampleScan)
+ {
+ Index scan_relid = path->parent->relid;
+ RangeTblEntry *rte;
+
+ /* it should be a base rel with a tablesample clause... */
+ Assert(scan_relid > 0);
+ rte = planner_rt_fetch(scan_relid, root);
+ Assert(rte->rtekind == RTE_RELATION);
+ Assert(rte->tablesample != NULL);
+
+ ADJUST_CHILD_ATTRS(rte->tablesample);
+ }
break;
case T_IndexPath:
{
IndexPath *ipath;
- FLAT_COPY_PATH(ipath, path, IndexPath);
+ ipath = (IndexPath *) path;
+ ADJUST_CHILD_ATTRS(ipath->indexinfo->indrestrictinfo);
ADJUST_CHILD_ATTRS(ipath->indexclauses);
new_path = (Path *) ipath;
}
@@ -4159,7 +4187,8 @@ do { \
{
BitmapHeapPath *bhpath;
- FLAT_COPY_PATH(bhpath, path, BitmapHeapPath);
+ bhpath = (BitmapHeapPath *) path;
+ ADJUST_CHILD_ATTRS(bhpath->path.parent->baserestrictinfo);
REPARAMETERIZE_CHILD_PATH(bhpath->bitmapqual);
new_path = (Path *) bhpath;
}
@@ -4169,7 +4198,7 @@ do { \
{
BitmapAndPath *bapath;
- FLAT_COPY_PATH(bapath, path, BitmapAndPath);
+ bapath = (BitmapAndPath *) path;
REPARAMETERIZE_CHILD_PATH_LIST(bapath->bitmapquals);
new_path = (Path *) bapath;
}
@@ -4179,7 +4208,7 @@ do { \
{
BitmapOrPath *bopath;
- FLAT_COPY_PATH(bopath, path, BitmapOrPath);
+ bopath = (BitmapOrPath *) path;
REPARAMETERIZE_CHILD_PATH_LIST(bopath->bitmapquals);
new_path = (Path *) bopath;
}
@@ -4190,7 +4219,8 @@ do { \
ForeignPath *fpath;
ReparameterizeForeignPathByChild_function rfpc_func;
- FLAT_COPY_PATH(fpath, path, ForeignPath);
+ fpath = (ForeignPath *) path;
+ ADJUST_CHILD_ATTRS(fpath->path.parent->baserestrictinfo);
if (fpath->fdw_outerpath)
REPARAMETERIZE_CHILD_PATH(fpath->fdw_outerpath);
if (fpath->fdw_restrictinfo)
@@ -4210,7 +4240,8 @@ do { \
{
CustomPath *cpath;
- FLAT_COPY_PATH(cpath, path, CustomPath);
+ cpath = (CustomPath *) path;
+ ADJUST_CHILD_ATTRS(cpath->path.parent->baserestrictinfo);
REPARAMETERIZE_CHILD_PATH_LIST(cpath->custom_paths);
if (cpath->custom_restrictinfo)
ADJUST_CHILD_ATTRS(cpath->custom_restrictinfo);
@@ -4229,7 +4260,7 @@ do { \
JoinPath *jpath;
NestPath *npath;
- FLAT_COPY_PATH(npath, path, NestPath);
+ npath = (NestPath *) path;
jpath = (JoinPath *) npath;
REPARAMETERIZE_CHILD_PATH(jpath->outerjoinpath);
@@ -4244,7 +4275,7 @@ do { \
JoinPath *jpath;
MergePath *mpath;
- FLAT_COPY_PATH(mpath, path, MergePath);
+ mpath = (MergePath *) path;
jpath = (JoinPath *) mpath;
REPARAMETERIZE_CHILD_PATH(jpath->outerjoinpath);
@@ -4260,7 +4291,7 @@ do { \
JoinPath *jpath;
HashPath *hpath;
- FLAT_COPY_PATH(hpath, path, HashPath);
+ hpath = (HashPath *) path;
jpath = (JoinPath *) hpath;
REPARAMETERIZE_CHILD_PATH(jpath->outerjoinpath);
@@ -4275,7 +4306,7 @@ do { \
{
AppendPath *apath;
- FLAT_COPY_PATH(apath, path, AppendPath);
+ apath = (AppendPath *) path;
REPARAMETERIZE_CHILD_PATH_LIST(apath->subpaths);
new_path = (Path *) apath;
}
@@ -4285,7 +4316,7 @@ do { \
{
MaterialPath *mpath;
- FLAT_COPY_PATH(mpath, path, MaterialPath);
+ mpath = (MaterialPath *) path;
REPARAMETERIZE_CHILD_PATH(mpath->subpath);
new_path = (Path *) mpath;
}
@@ -4295,7 +4326,7 @@ do { \
{
MemoizePath *mpath;
- FLAT_COPY_PATH(mpath, path, MemoizePath);
+ mpath = (MemoizePath *) path;
REPARAMETERIZE_CHILD_PATH(mpath->subpath);
ADJUST_CHILD_ATTRS(mpath->param_exprs);
new_path = (Path *) mpath;
@@ -4306,7 +4337,7 @@ do { \
{
GatherPath *gpath;
- FLAT_COPY_PATH(gpath, path, GatherPath);
+ gpath = (GatherPath *) path;
REPARAMETERIZE_CHILD_PATH(gpath->subpath);
new_path = (Path *) gpath;
}
@@ -4373,9 +4404,145 @@ do { \
return new_path;
}
+/*
+ * path_is_reparameterizable_by_child
+ * Given a path parameterized by the parent of the given child relation,
+ * see if it can be translated to be parameterized by the child relation.
+ *
+ * This must return true if and only if reparameterize_path_by_child()
+ * would succeed on this path. Currently it's sufficient to verify that
+ * the path and all of its subpaths (if any) are of the types handled by
+ * that function. However, sub-paths that are not parameterized can be
+ * disregarded since they won't require translation.
+ */
+bool
+path_is_reparameterizable_by_child(Path *path, RelOptInfo *child_rel)
+{
+
+#define REJECT_IF_PATH_NOT_REPARAMETERIZABLE(path) \
+do { \
+ if (!path_is_reparameterizable_by_child(path, child_rel)) \
+ return false; \
+} while(0)
+
+#define REJECT_IF_PATH_LIST_NOT_REPARAMETERIZABLE(pathlist) \
+do { \
+ if (!pathlist_is_reparameterizable_by_child(pathlist, child_rel)) \
+ return false; \
+} while(0)
+
+ /*
+ * If the path is not parameterized by the parent of the given relation,
+ * it doesn't need reparameterization.
+ */
+ if (!path->param_info ||
+ !bms_overlap(PATH_REQ_OUTER(path), child_rel->top_parent_relids))
+ return true;
+
+ switch (nodeTag(path))
+ {
+ case T_Path:
+ case T_IndexPath:
+ break;
+
+ case T_BitmapHeapPath:
+ {
+ BitmapHeapPath *bhpath = (BitmapHeapPath *) path;
+
+ REJECT_IF_PATH_NOT_REPARAMETERIZABLE(bhpath->bitmapqual);
+ }
+ break;
+
+ case T_BitmapAndPath:
+ {
+ BitmapAndPath *bapath = (BitmapAndPath *) path;
+
+ REJECT_IF_PATH_LIST_NOT_REPARAMETERIZABLE(bapath->bitmapquals);
+ }
+ break;
+
+ case T_BitmapOrPath:
+ {
+ BitmapOrPath *bopath = (BitmapOrPath *) path;
+
+ REJECT_IF_PATH_LIST_NOT_REPARAMETERIZABLE(bopath->bitmapquals);
+ }
+ break;
+
+ case T_ForeignPath:
+ {
+ ForeignPath *fpath = (ForeignPath *) path;
+
+ if (fpath->fdw_outerpath)
+ REJECT_IF_PATH_NOT_REPARAMETERIZABLE(fpath->fdw_outerpath);
+ }
+ break;
+
+ case T_CustomPath:
+ {
+ CustomPath *cpath = (CustomPath *) path;
+
+ REJECT_IF_PATH_LIST_NOT_REPARAMETERIZABLE(cpath->custom_paths);
+ }
+ break;
+
+ case T_NestPath:
+ case T_MergePath:
+ case T_HashPath:
+ {
+ JoinPath *jpath = (JoinPath *) path;
+
+ REJECT_IF_PATH_NOT_REPARAMETERIZABLE(jpath->outerjoinpath);
+ REJECT_IF_PATH_NOT_REPARAMETERIZABLE(jpath->innerjoinpath);
+ }
+ break;
+
+ case T_AppendPath:
+ {
+ AppendPath *apath = (AppendPath *) path;
+
+ REJECT_IF_PATH_LIST_NOT_REPARAMETERIZABLE(apath->subpaths);
+ }
+ break;
+
+ case T_MaterialPath:
+ {
+ MaterialPath *mpath = (MaterialPath *) path;
+
+ REJECT_IF_PATH_NOT_REPARAMETERIZABLE(mpath->subpath);
+ }
+ break;
+
+ case T_MemoizePath:
+ {
+ MemoizePath *mpath = (MemoizePath *) path;
+
+ REJECT_IF_PATH_NOT_REPARAMETERIZABLE(mpath->subpath);
+ }
+ break;
+
+ case T_GatherPath:
+ {
+ GatherPath *gpath = (GatherPath *) path;
+
+ REJECT_IF_PATH_NOT_REPARAMETERIZABLE(gpath->subpath);
+ }
+ break;
+
+ default:
+
+ /* We don't know how to reparameterize this path. */
+ return false;
+ }
+
+ return true;
+}
+
/*
* reparameterize_pathlist_by_child
* Helper function to reparameterize a list of paths by given child rel.
+ *
+ * Returns NIL to indicate failure, so pathlist had better not be NIL.
*/
static List *
reparameterize_pathlist_by_child(PlannerInfo *root,
@@ -4401,3 +4568,23 @@ reparameterize_pathlist_by_child(PlannerInfo *root,
return result;
}
+
+/*
+ * pathlist_is_reparameterizable_by_child
+ * Helper function to check if a list of paths can be reparameterized.
+ */
+static bool
+pathlist_is_reparameterizable_by_child(List *pathlist, RelOptInfo *child_rel)
+{
+ ListCell *lc;
+
+ foreach(lc, pathlist)
+ {
+ Path *path = (Path *) lfirst(lc);
+
+ if (!path_is_reparameterizable_by_child(path, child_rel))
+ return false;
+ }
+
+ return true;
+}
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index c43d97b48a..99c2f955aa 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -298,6 +298,8 @@ extern Path *reparameterize_path(PlannerInfo *root, Path *path,
double loop_count);
extern Path *reparameterize_path_by_child(PlannerInfo *root, Path *path,
RelOptInfo *child_rel);
+extern bool path_is_reparameterizable_by_child(Path *path,
+ RelOptInfo *child_rel);
/*
* prototypes for relnode.c
diff --git a/src/test/regress/expected/partition_join.out b/src/test/regress/expected/partition_join.out
index 6560fe2416..6d07f86b9b 100644
--- a/src/test/regress/expected/partition_join.out
+++ b/src/test/regress/expected/partition_join.out
@@ -505,6 +505,98 @@ SELECT t1.a, ss.t2a, ss.t2c FROM prt1 t1 LEFT JOIN LATERAL
550 | |
(12 rows)
+-- lateral reference in sample scan
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt1 t1 JOIN LATERAL
+ (SELECT * FROM prt1 t2 TABLESAMPLE SYSTEM (t1.a) REPEATABLE(t1.b)) s
+ ON t1.a = s.a;
+ QUERY PLAN
+-------------------------------------------------------------
+ Append
+ -> Nested Loop
+ -> Seq Scan on prt1_p1 t1_1
+ -> Sample Scan on prt1_p1 t2_1
+ Sampling: system (t1_1.a) REPEATABLE (t1_1.b)
+ Filter: (t1_1.a = a)
+ -> Nested Loop
+ -> Seq Scan on prt1_p2 t1_2
+ -> Sample Scan on prt1_p2 t2_2
+ Sampling: system (t1_2.a) REPEATABLE (t1_2.b)
+ Filter: (t1_2.a = a)
+ -> Nested Loop
+ -> Seq Scan on prt1_p3 t1_3
+ -> Sample Scan on prt1_p3 t2_3
+ Sampling: system (t1_3.a) REPEATABLE (t1_3.b)
+ Filter: (t1_3.a = a)
+(16 rows)
+
+-- lateral reference in scan's restriction clauses
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.a;
+ QUERY PLAN
+---------------------------------------------------------------
+ Aggregate
+ -> Append
+ -> Nested Loop
+ -> Seq Scan on prt1_p1 t1_1
+ -> Index Scan using iprt2_p1_b on prt2_p1 t2_1
+ Index Cond: (b = t1_1.a)
+ Filter: (t1_1.b = a)
+ -> Nested Loop
+ -> Seq Scan on prt1_p2 t1_2
+ -> Index Scan using iprt2_p2_b on prt2_p2 t2_2
+ Index Cond: (b = t1_2.a)
+ Filter: (t1_2.b = a)
+ -> Nested Loop
+ -> Seq Scan on prt1_p3 t1_3
+ -> Index Scan using iprt2_p3_b on prt2_p3 t2_3
+ Index Cond: (b = t1_3.a)
+ Filter: (t1_3.b = a)
+(17 rows)
+
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.a;
+ count
+-------
+ 100
+(1 row)
+
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.b;
+ QUERY PLAN
+--------------------------------------------------------------------
+ Aggregate
+ -> Append
+ -> Nested Loop
+ -> Seq Scan on prt1_p1 t1_1
+ -> Index Only Scan using iprt2_p1_b on prt2_p1 t2_1
+ Index Cond: (b = t1_1.a)
+ Filter: (b = t1_1.b)
+ -> Nested Loop
+ -> Seq Scan on prt1_p2 t1_2
+ -> Index Only Scan using iprt2_p2_b on prt2_p2 t2_2
+ Index Cond: (b = t1_2.a)
+ Filter: (b = t1_2.b)
+ -> Nested Loop
+ -> Seq Scan on prt1_p3 t1_3
+ -> Index Only Scan using iprt2_p3_b on prt2_p3 t2_3
+ Index Cond: (b = t1_3.a)
+ Filter: (b = t1_3.b)
+(17 rows)
+
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.b;
+ count
+-------
+ 5
+(1 row)
+
-- bug with inadequate sort key representation
SET enable_partitionwise_aggregate TO true;
SET enable_hashjoin TO false;
@@ -1944,6 +2036,82 @@ SELECT * FROM prt1_l t1 LEFT JOIN LATERAL
550 | 0 | 0002 | | | | |
(12 rows)
+-- partitionwise join with lateral reference in sample scan
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt1_l t1 JOIN LATERAL
+ (SELECT * FROM prt1_l t2 TABLESAMPLE SYSTEM (t1.a) REPEATABLE(t1.b)) s
+ ON t1.a = s.a AND t1.b = s.b AND t1.c = s.c;
+ QUERY PLAN
+----------------------------------------------------------------------------------------
+ Append
+ -> Nested Loop
+ -> Seq Scan on prt1_l_p1 t1_1
+ -> Sample Scan on prt1_l_p1 t2_1
+ Sampling: system (t1_1.a) REPEATABLE (t1_1.b)
+ Filter: ((t1_1.a = a) AND (t1_1.b = b) AND ((t1_1.c)::text = (c)::text))
+ -> Nested Loop
+ -> Seq Scan on prt1_l_p2_p1 t1_2
+ -> Sample Scan on prt1_l_p2_p1 t2_2
+ Sampling: system (t1_2.a) REPEATABLE (t1_2.b)
+ Filter: ((t1_2.a = a) AND (t1_2.b = b) AND ((t1_2.c)::text = (c)::text))
+ -> Nested Loop
+ -> Seq Scan on prt1_l_p2_p2 t1_3
+ -> Sample Scan on prt1_l_p2_p2 t2_3
+ Sampling: system (t1_3.a) REPEATABLE (t1_3.b)
+ Filter: ((t1_3.a = a) AND (t1_3.b = b) AND ((t1_3.c)::text = (c)::text))
+ -> Nested Loop
+ -> Seq Scan on prt1_l_p3_p1 t1_4
+ -> Sample Scan on prt1_l_p3_p1 t2_4
+ Sampling: system (t1_4.a) REPEATABLE (t1_4.b)
+ Filter: ((t1_4.a = a) AND (t1_4.b = b) AND ((t1_4.c)::text = (c)::text))
+ -> Nested Loop
+ -> Seq Scan on prt1_l_p3_p2 t1_5
+ -> Sample Scan on prt1_l_p3_p2 t2_5
+ Sampling: system (t1_5.a) REPEATABLE (t1_5.b)
+ Filter: ((t1_5.a = a) AND (t1_5.b = b) AND ((t1_5.c)::text = (c)::text))
+(26 rows)
+
+-- partitionwise join with lateral reference in scan's restriction clauses
+EXPLAIN (COSTS OFF)
+SELECT COUNT(*) FROM prt1_l t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2_l t2) s
+ ON t1.a = s.b AND t1.b = s.a AND t1.c = s.c
+ WHERE s.t1b = s.a;
+ QUERY PLAN
+---------------------------------------------------------------------------------------------------------------
+ Aggregate
+ -> Append
+ -> Nested Loop
+ -> Seq Scan on prt1_l_p1 t1_1
+ -> Seq Scan on prt2_l_p1 t2_1
+ Filter: ((a = t1_1.b) AND (t1_1.a = b) AND (t1_1.b = a) AND ((t1_1.c)::text = (c)::text))
+ -> Nested Loop
+ -> Seq Scan on prt1_l_p2_p1 t1_2
+ -> Seq Scan on prt2_l_p2_p1 t2_2
+ Filter: ((a = t1_2.b) AND (t1_2.a = b) AND (t1_2.b = a) AND ((t1_2.c)::text = (c)::text))
+ -> Nested Loop
+ -> Seq Scan on prt1_l_p2_p2 t1_3
+ -> Seq Scan on prt2_l_p2_p2 t2_3
+ Filter: ((a = t1_3.b) AND (t1_3.a = b) AND (t1_3.b = a) AND ((t1_3.c)::text = (c)::text))
+ -> Nested Loop
+ -> Seq Scan on prt1_l_p3_p1 t1_4
+ -> Seq Scan on prt2_l_p3_p1 t2_4
+ Filter: ((a = t1_4.b) AND (t1_4.a = b) AND (t1_4.b = a) AND ((t1_4.c)::text = (c)::text))
+ -> Nested Loop
+ -> Seq Scan on prt1_l_p3_p2 t1_5
+ -> Seq Scan on prt2_l_p3_p2 t2_5
+ Filter: ((a = t1_5.b) AND (t1_5.a = b) AND (t1_5.b = a) AND ((t1_5.c)::text = (c)::text))
+(22 rows)
+
+SELECT COUNT(*) FROM prt1_l t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2_l t2) s
+ ON t1.a = s.b AND t1.b = s.a AND t1.c = s.c
+ WHERE s.t1b = s.a;
+ count
+-------
+ 100
+(1 row)
+
-- join with one side empty
EXPLAIN (COSTS OFF)
SELECT t1.a, t1.c, t2.b, t2.c FROM (SELECT * FROM prt1_l WHERE a = 1 AND a = 2) t1 RIGHT JOIN prt2_l t2 ON t1.a = t2.b AND t1.b = t2.a AND t1.c = t2.c;
diff --git a/src/test/regress/sql/partition_join.sql b/src/test/regress/sql/partition_join.sql
index 48daf3aee3..128ce8376e 100644
--- a/src/test/regress/sql/partition_join.sql
+++ b/src/test/regress/sql/partition_join.sql
@@ -100,6 +100,29 @@ SELECT t1.a, ss.t2a, ss.t2c FROM prt1 t1 LEFT JOIN LATERAL
(SELECT t2.a AS t2a, t3.a AS t3a, t2.b t2b, t2.c t2c, least(t1.a,t2.a,t3.a) FROM prt1 t2 JOIN prt2 t3 ON (t2.a = t3.b)) ss
ON t1.c = ss.t2c WHERE (t1.b + coalesce(ss.t2b, 0)) = 0 ORDER BY t1.a;
+-- lateral reference in sample scan
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt1 t1 JOIN LATERAL
+ (SELECT * FROM prt1 t2 TABLESAMPLE SYSTEM (t1.a) REPEATABLE(t1.b)) s
+ ON t1.a = s.a;
+
+-- lateral reference in scan's restriction clauses
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.a;
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.a;
+
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.b;
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.b;
+
-- bug with inadequate sort key representation
SET enable_partitionwise_aggregate TO true;
SET enable_hashjoin TO false;
@@ -387,6 +410,23 @@ SELECT * FROM prt1_l t1 LEFT JOIN LATERAL
(SELECT t2.a AS t2a, t2.c AS t2c, t2.b AS t2b, t3.b AS t3b, least(t1.a,t2.a,t3.b) FROM prt1_l t2 JOIN prt2_l t3 ON (t2.a = t3.b AND t2.c = t3.c)) ss
ON t1.a = ss.t2a AND t1.c = ss.t2c WHERE t1.b = 0 ORDER BY t1.a;
+-- partitionwise join with lateral reference in sample scan
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt1_l t1 JOIN LATERAL
+ (SELECT * FROM prt1_l t2 TABLESAMPLE SYSTEM (t1.a) REPEATABLE(t1.b)) s
+ ON t1.a = s.a AND t1.b = s.b AND t1.c = s.c;
+
+-- partitionwise join with lateral reference in scan's restriction clauses
+EXPLAIN (COSTS OFF)
+SELECT COUNT(*) FROM prt1_l t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2_l t2) s
+ ON t1.a = s.b AND t1.b = s.a AND t1.c = s.c
+ WHERE s.t1b = s.a;
+SELECT COUNT(*) FROM prt1_l t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2_l t2) s
+ ON t1.a = s.b AND t1.b = s.a AND t1.c = s.c
+ WHERE s.t1b = s.a;
+
-- join with one side empty
EXPLAIN (COSTS OFF)
SELECT t1.a, t1.c, t2.b, t2.c FROM (SELECT * FROM prt1_l WHERE a = 1 AND a = 2) t1 RIGHT JOIN prt2_l t2 ON t1.a = t2.b AND t1.b = t2.a AND t1.c = t2.c;
--
2.31.0
[application/octet-stream] v10-0001-Avoid-reparameterizing-Paths-when-it-s-not-suitable.patch (17.6K, ../../CAMbWs48ZfxwRobV-QP1mt_16+2iDbfw_pdsnuzmTJntCPCdD0w@mail.gmail.com/4-v10-0001-Avoid-reparameterizing-Paths-when-it-s-not-suitable.patch)
download | inline diff:
From 52972c8933772115cb72933e0dc24e441c4534eb Mon Sep 17 00:00:00 2001
From: Richard Guo <[email protected]>
Date: Wed, 17 Jan 2024 14:01:27 +0800
Subject: [PATCH v10] Avoid reparameterizing Paths when it's not suitable
When creating a nestloop join path, if the inner path is parameterized,
it is parameterized by the topmost parent of the outer rel, not the
outer rel itself. Therefore, we need to translate the parameterization
so that the inner path is parameterized by the given outer rel itself.
Currently, this is done in reparameterize_path_by_child() during
generating join paths.
However, reparameterize_path_by_child() does not perform the translation
for sample scan paths' sampling infos, or scan paths' restriction
clauses. This omission can lead to executor crashes, wrong results, or
planning errors, as we have already observed.
Please note that the sampling infos are contained in RangeTblEntries (as
TableSampleClause objects), and the scan restriction clauses are
contained in RelOptInfos or IndexOptInfos (as lists of RestrictInfo
objects). We cannot just modify them on the fly during generating join
paths. Doing so would break things if we end up using a
non-partitionwise join.
So this commit chooses not to reparameterize the path if there are
lateral references to the other relation in the sampling infos or
restriction clauses associated with the path.
---
src/backend/optimizer/util/pathnode.c | 108 +++++++++++-
src/test/regress/expected/partition_join.out | 167 +++++++++++++++++++
src/test/regress/sql/partition_join.sql | 48 ++++++
3 files changed, 322 insertions(+), 1 deletion(-)
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index 2185fc35a3..f11bcd1f10 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -56,6 +56,8 @@ static int append_startup_cost_compare(const ListCell *a, const ListCell *b);
static List *reparameterize_pathlist_by_child(PlannerInfo *root,
List *pathlist,
RelOptInfo *child_rel);
+static bool contain_references_to(PlannerInfo *root, List *restrictinfo_list,
+ Relids relids);
/*****************************************************************************
@@ -4103,13 +4105,60 @@ do { \
switch (nodeTag(path))
{
case T_Path:
- FLAT_COPY_PATH(new_path, path, Path);
+ {
+ /*
+ * If the path's restriction clauses contain lateral references
+ * to the other relation, we can't reparameterize, because we
+ * must not change the RelOptInfo's contents here. (Doing so
+ * would break things if we end up using a non-partitionwise
+ * join.)
+ */
+ if (contain_references_to(root, path->parent->baserestrictinfo,
+ child_rel->top_parent_relids))
+ return NULL;
+
+ /*
+ * If it's a SampleScan with tablesample parameters referencing
+ * the other relation, we can't reparameterize, because we must
+ * not change the RTE's contents here. (Doing so would break
+ * things if we end up using a non-partitionwise join.)
+ */
+ if (path->pathtype == T_SampleScan)
+ {
+ Index scan_relid = path->parent->relid;
+ RangeTblEntry *rte;
+
+ /* it should be a base rel with a tablesample clause... */
+ Assert(scan_relid > 0);
+ rte = planner_rt_fetch(scan_relid, root);
+ Assert(rte->rtekind == RTE_RELATION);
+ Assert(rte->tablesample != NULL);
+
+ if (bms_overlap(pull_varnos(root, (Node *) rte->tablesample),
+ child_rel->top_parent_relids))
+ return NULL;
+ }
+
+ FLAT_COPY_PATH(new_path, path, Path);
+ }
break;
case T_IndexPath:
{
IndexPath *ipath;
+ /*
+ * If the path's restriction clauses contain lateral references
+ * to the other relation, we can't reparameterize, because we
+ * must not change the IndexOptInfo's contents here. (Doing so
+ * would break things if we end up using a non-partitionwise
+ * join.)
+ */
+ if (contain_references_to(root,
+ ((IndexPath *) path)->indexinfo->indrestrictinfo,
+ child_rel->top_parent_relids))
+ return NULL;
+
FLAT_COPY_PATH(ipath, path, IndexPath);
ADJUST_CHILD_ATTRS(ipath->indexclauses);
new_path = (Path *) ipath;
@@ -4120,6 +4169,17 @@ do { \
{
BitmapHeapPath *bhpath;
+ /*
+ * If the path's restriction clauses contain lateral references
+ * to the other relation, we can't reparameterize, because we
+ * must not change the RelOptInfo's contents here. (Doing so
+ * would break things if we end up using a non-partitionwise
+ * join.)
+ */
+ if (contain_references_to(root, path->parent->baserestrictinfo,
+ child_rel->top_parent_relids))
+ return NULL;
+
FLAT_COPY_PATH(bhpath, path, BitmapHeapPath);
REPARAMETERIZE_CHILD_PATH(bhpath->bitmapqual);
new_path = (Path *) bhpath;
@@ -4151,6 +4211,17 @@ do { \
ForeignPath *fpath;
ReparameterizeForeignPathByChild_function rfpc_func;
+ /*
+ * If the path's restriction clauses contain lateral references
+ * to the other relation, we can't reparameterize, because we
+ * must not change the RelOptInfo's contents here. (Doing so
+ * would break things if we end up using a non-partitionwise
+ * join.)
+ */
+ if (contain_references_to(root, path->parent->baserestrictinfo,
+ child_rel->top_parent_relids))
+ return NULL;
+
FLAT_COPY_PATH(fpath, path, ForeignPath);
if (fpath->fdw_outerpath)
REPARAMETERIZE_CHILD_PATH(fpath->fdw_outerpath);
@@ -4169,6 +4240,17 @@ do { \
{
CustomPath *cpath;
+ /*
+ * If the path's restriction clauses contain lateral references
+ * to the other relation, we can't reparameterize, because we
+ * must not change the RelOptInfo's contents here. (Doing so
+ * would break things if we end up using a non-partitionwise
+ * join.)
+ */
+ if (contain_references_to(root, path->parent->baserestrictinfo,
+ child_rel->top_parent_relids))
+ return NULL;
+
FLAT_COPY_PATH(cpath, path, CustomPath);
REPARAMETERIZE_CHILD_PATH_LIST(cpath->custom_paths);
if (cpath->methods &&
@@ -4358,3 +4440,27 @@ reparameterize_pathlist_by_child(PlannerInfo *root,
return result;
}
+
+/*
+ * contain_references_to
+ * Detect whether any Vars in the given 'restrictinfo_list' contain
+ * references to the given 'relids'.
+ */
+static bool
+contain_references_to(PlannerInfo *root, List *restrictinfo_list,
+ Relids relids)
+{
+ List *scan_clauses;
+ List *vars;
+ bool ret;
+
+ scan_clauses = extract_actual_clauses(restrictinfo_list, false);
+ vars = pull_var_clause((Node *) scan_clauses,
+ PVC_RECURSE_AGGREGATES |
+ PVC_RECURSE_WINDOWFUNCS |
+ PVC_RECURSE_PLACEHOLDERS);
+ ret = bms_overlap(pull_varnos(root, (Node *) vars), relids);
+ list_free(vars);
+
+ return ret;
+}
diff --git a/src/test/regress/expected/partition_join.out b/src/test/regress/expected/partition_join.out
index 6560fe2416..72f286b893 100644
--- a/src/test/regress/expected/partition_join.out
+++ b/src/test/regress/expected/partition_join.out
@@ -505,6 +505,99 @@ SELECT t1.a, ss.t2a, ss.t2c FROM prt1 t1 LEFT JOIN LATERAL
550 | |
(12 rows)
+SET max_parallel_workers_per_gather = 0;
+-- If there are lateral references to the other relation in sample scan, we'd
+-- fail to generate a partitionwise join.
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt1 t1 JOIN LATERAL
+ (SELECT * FROM prt1 t2 TABLESAMPLE SYSTEM (t1.a) REPEATABLE(t1.b)) s
+ ON t1.a = s.a;
+ QUERY PLAN
+---------------------------------------------------------
+ Nested Loop
+ -> Append
+ -> Seq Scan on prt1_p1 t1_1
+ -> Seq Scan on prt1_p2 t1_2
+ -> Seq Scan on prt1_p3 t1_3
+ -> Append
+ -> Sample Scan on prt1_p1 t2_1
+ Sampling: system (t1.a) REPEATABLE (t1.b)
+ Filter: (t1.a = a)
+ -> Sample Scan on prt1_p2 t2_2
+ Sampling: system (t1.a) REPEATABLE (t1.b)
+ Filter: (t1.a = a)
+ -> Sample Scan on prt1_p3 t2_3
+ Sampling: system (t1.a) REPEATABLE (t1.b)
+ Filter: (t1.a = a)
+(15 rows)
+
+-- If there are lateral references to the other relation in scan's restriction
+-- clauses, we'd fail to generate a partitionwise join.
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.a;
+ QUERY PLAN
+---------------------------------------------------------------
+ Aggregate
+ -> Nested Loop
+ -> Append
+ -> Seq Scan on prt1_p1 t1_1
+ -> Seq Scan on prt1_p2 t1_2
+ -> Seq Scan on prt1_p3 t1_3
+ -> Append
+ -> Index Scan using iprt2_p1_b on prt2_p1 t2_1
+ Index Cond: (b = t1.a)
+ Filter: (t1.b = a)
+ -> Index Scan using iprt2_p2_b on prt2_p2 t2_2
+ Index Cond: (b = t1.a)
+ Filter: (t1.b = a)
+ -> Index Scan using iprt2_p3_b on prt2_p3 t2_3
+ Index Cond: (b = t1.a)
+ Filter: (t1.b = a)
+(16 rows)
+
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.a;
+ count
+-------
+ 100
+(1 row)
+
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.b;
+ QUERY PLAN
+--------------------------------------------------------------------
+ Aggregate
+ -> Nested Loop
+ -> Append
+ -> Seq Scan on prt1_p1 t1_1
+ -> Seq Scan on prt1_p2 t1_2
+ -> Seq Scan on prt1_p3 t1_3
+ -> Append
+ -> Index Only Scan using iprt2_p1_b on prt2_p1 t2_1
+ Index Cond: (b = t1.a)
+ Filter: (b = t1.b)
+ -> Index Only Scan using iprt2_p2_b on prt2_p2 t2_2
+ Index Cond: (b = t1.a)
+ Filter: (b = t1.b)
+ -> Index Only Scan using iprt2_p3_b on prt2_p3 t2_3
+ Index Cond: (b = t1.a)
+ Filter: (b = t1.b)
+(16 rows)
+
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.b;
+ count
+-------
+ 5
+(1 row)
+
+RESET max_parallel_workers_per_gather;
-- bug with inadequate sort key representation
SET enable_partitionwise_aggregate TO true;
SET enable_hashjoin TO false;
@@ -1944,6 +2037,80 @@ SELECT * FROM prt1_l t1 LEFT JOIN LATERAL
550 | 0 | 0002 | | | | |
(12 rows)
+SET max_parallel_workers_per_gather = 0;
+-- If there are lateral references to the other relation in sample scan, we'd
+-- fail to generate a partitionwise join.
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt1_l t1 JOIN LATERAL
+ (SELECT * FROM prt1_l t2 TABLESAMPLE SYSTEM (t1.a) REPEATABLE(t1.b)) s
+ ON t1.a = s.a AND t1.b = s.b AND t1.c = s.c;
+ QUERY PLAN
+----------------------------------------------------------------------------------
+ Nested Loop
+ -> Append
+ -> Seq Scan on prt1_l_p1 t1_1
+ -> Seq Scan on prt1_l_p2_p1 t1_2
+ -> Seq Scan on prt1_l_p2_p2 t1_3
+ -> Seq Scan on prt1_l_p3_p1 t1_4
+ -> Seq Scan on prt1_l_p3_p2 t1_5
+ -> Append
+ -> Sample Scan on prt1_l_p1 t2_1
+ Sampling: system (t1.a) REPEATABLE (t1.b)
+ Filter: ((t1.a = a) AND (t1.b = b) AND ((t1.c)::text = (c)::text))
+ -> Sample Scan on prt1_l_p2_p1 t2_2
+ Sampling: system (t1.a) REPEATABLE (t1.b)
+ Filter: ((t1.a = a) AND (t1.b = b) AND ((t1.c)::text = (c)::text))
+ -> Sample Scan on prt1_l_p2_p2 t2_3
+ Sampling: system (t1.a) REPEATABLE (t1.b)
+ Filter: ((t1.a = a) AND (t1.b = b) AND ((t1.c)::text = (c)::text))
+ -> Sample Scan on prt1_l_p3_p1 t2_4
+ Sampling: system (t1.a) REPEATABLE (t1.b)
+ Filter: ((t1.a = a) AND (t1.b = b) AND ((t1.c)::text = (c)::text))
+ -> Sample Scan on prt1_l_p3_p2 t2_5
+ Sampling: system (t1.a) REPEATABLE (t1.b)
+ Filter: ((t1.a = a) AND (t1.b = b) AND ((t1.c)::text = (c)::text))
+(23 rows)
+
+-- If there are lateral references to the other relation in scan's restriction
+-- clauses, we'd fail to generate a partitionwise join.
+EXPLAIN (COSTS OFF)
+SELECT COUNT(*) FROM prt1_l t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2_l t2) s
+ ON t1.a = s.b AND t1.b = s.a AND t1.c = s.c
+ WHERE s.t1b = s.a;
+ QUERY PLAN
+-------------------------------------------------------------------------------------------------------
+ Aggregate
+ -> Nested Loop
+ -> Append
+ -> Seq Scan on prt1_l_p1 t1_1
+ -> Seq Scan on prt1_l_p2_p1 t1_2
+ -> Seq Scan on prt1_l_p2_p2 t1_3
+ -> Seq Scan on prt1_l_p3_p1 t1_4
+ -> Seq Scan on prt1_l_p3_p2 t1_5
+ -> Append
+ -> Seq Scan on prt2_l_p1 t2_1
+ Filter: ((a = t1.b) AND (t1.a = b) AND (t1.b = a) AND ((t1.c)::text = (c)::text))
+ -> Seq Scan on prt2_l_p2_p1 t2_2
+ Filter: ((a = t1.b) AND (t1.a = b) AND (t1.b = a) AND ((t1.c)::text = (c)::text))
+ -> Seq Scan on prt2_l_p2_p2 t2_3
+ Filter: ((a = t1.b) AND (t1.a = b) AND (t1.b = a) AND ((t1.c)::text = (c)::text))
+ -> Seq Scan on prt2_l_p3_p1 t2_4
+ Filter: ((a = t1.b) AND (t1.a = b) AND (t1.b = a) AND ((t1.c)::text = (c)::text))
+ -> Seq Scan on prt2_l_p3_p2 t2_5
+ Filter: ((a = t1.b) AND (t1.a = b) AND (t1.b = a) AND ((t1.c)::text = (c)::text))
+(19 rows)
+
+SELECT COUNT(*) FROM prt1_l t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2_l t2) s
+ ON t1.a = s.b AND t1.b = s.a AND t1.c = s.c
+ WHERE s.t1b = s.a;
+ count
+-------
+ 100
+(1 row)
+
+RESET max_parallel_workers_per_gather;
-- join with one side empty
EXPLAIN (COSTS OFF)
SELECT t1.a, t1.c, t2.b, t2.c FROM (SELECT * FROM prt1_l WHERE a = 1 AND a = 2) t1 RIGHT JOIN prt2_l t2 ON t1.a = t2.b AND t1.b = t2.a AND t1.c = t2.c;
diff --git a/src/test/regress/sql/partition_join.sql b/src/test/regress/sql/partition_join.sql
index 48daf3aee3..0554a6568b 100644
--- a/src/test/regress/sql/partition_join.sql
+++ b/src/test/regress/sql/partition_join.sql
@@ -100,6 +100,33 @@ SELECT t1.a, ss.t2a, ss.t2c FROM prt1 t1 LEFT JOIN LATERAL
(SELECT t2.a AS t2a, t3.a AS t3a, t2.b t2b, t2.c t2c, least(t1.a,t2.a,t3.a) FROM prt1 t2 JOIN prt2 t3 ON (t2.a = t3.b)) ss
ON t1.c = ss.t2c WHERE (t1.b + coalesce(ss.t2b, 0)) = 0 ORDER BY t1.a;
+SET max_parallel_workers_per_gather = 0;
+-- If there are lateral references to the other relation in sample scan, we'd
+-- fail to generate a partitionwise join.
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt1 t1 JOIN LATERAL
+ (SELECT * FROM prt1 t2 TABLESAMPLE SYSTEM (t1.a) REPEATABLE(t1.b)) s
+ ON t1.a = s.a;
+
+-- If there are lateral references to the other relation in scan's restriction
+-- clauses, we'd fail to generate a partitionwise join.
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.a;
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.a;
+
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.b;
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.b;
+RESET max_parallel_workers_per_gather;
+
-- bug with inadequate sort key representation
SET enable_partitionwise_aggregate TO true;
SET enable_hashjoin TO false;
@@ -387,6 +414,27 @@ SELECT * FROM prt1_l t1 LEFT JOIN LATERAL
(SELECT t2.a AS t2a, t2.c AS t2c, t2.b AS t2b, t3.b AS t3b, least(t1.a,t2.a,t3.b) FROM prt1_l t2 JOIN prt2_l t3 ON (t2.a = t3.b AND t2.c = t3.c)) ss
ON t1.a = ss.t2a AND t1.c = ss.t2c WHERE t1.b = 0 ORDER BY t1.a;
+SET max_parallel_workers_per_gather = 0;
+-- If there are lateral references to the other relation in sample scan, we'd
+-- fail to generate a partitionwise join.
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt1_l t1 JOIN LATERAL
+ (SELECT * FROM prt1_l t2 TABLESAMPLE SYSTEM (t1.a) REPEATABLE(t1.b)) s
+ ON t1.a = s.a AND t1.b = s.b AND t1.c = s.c;
+
+-- If there are lateral references to the other relation in scan's restriction
+-- clauses, we'd fail to generate a partitionwise join.
+EXPLAIN (COSTS OFF)
+SELECT COUNT(*) FROM prt1_l t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2_l t2) s
+ ON t1.a = s.b AND t1.b = s.a AND t1.c = s.c
+ WHERE s.t1b = s.a;
+SELECT COUNT(*) FROM prt1_l t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2_l t2) s
+ ON t1.a = s.b AND t1.b = s.a AND t1.c = s.c
+ WHERE s.t1b = s.a;
+RESET max_parallel_workers_per_gather;
+
-- join with one side empty
EXPLAIN (COSTS OFF)
SELECT t1.a, t1.c, t2.b, t2.c FROM (SELECT * FROM prt1_l WHERE a = 1 AND a = 2) t1 RIGHT JOIN prt2_l t2 ON t1.a = t2.b AND t1.b = t2.a AND t1.c = t2.c;
--
2.31.0
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Oversight in reparameterize_path_by_child leading to executor crash
@ 2024-01-17 09:50 Richard Guo <[email protected]>
parent: Richard Guo <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Richard Guo @ 2024-01-17 09:50 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Andrei Lepikhov <[email protected]>; Alena Rybakina <[email protected]>; Tom Lane <[email protected]>; Ashutosh Bapat <[email protected]>; pgsql-hackers
On Wed, Jan 17, 2024 at 5:01 PM Richard Guo <[email protected]> wrote:
> On Tue, Jan 16, 2024 at 2:30 AM Robert Haas <[email protected]> wrote:
>
>> On Mon, Jan 8, 2024 at 3:32 AM Richard Guo <[email protected]>
>> wrote:
>
> > Fair point. I think we can back-patch a more minimal fix, as Tom
>> > proposed in [1], which disallows the reparameterization if the path
>> > contains sampling info that references the outer rel. But I think we
>> > need also to disallow the reparameterization of a path if it contains
>> > restriction clauses that reference the outer rel, as such paths have
>> > been found to cause incorrect results, or planning errors as in [2].
>>
>> Do you want to produce a patch for that, to go along with this patch for
>> master?
>
>
> Sure, here it is:
> v10-0001-Avoid-reparameterizing-Paths-when-it-s-not-suitable.patch
>
I forgot to mention that this patch applies on v16 not master.
Thanks
Richard
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Oversight in reparameterize_path_by_child leading to executor crash
@ 2024-01-30 21:12 Tom Lane <[email protected]>
parent: Richard Guo <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Tom Lane @ 2024-01-30 21:12 UTC (permalink / raw)
To: Richard Guo <[email protected]>; +Cc: Robert Haas <[email protected]>; Andrei Lepikhov <[email protected]>; Alena Rybakina <[email protected]>; Ashutosh Bapat <[email protected]>; pgsql-hackers
Richard Guo <[email protected]> writes:
> On Wed, Jan 17, 2024 at 5:01 PM Richard Guo <[email protected]> wrote:
>> Sure, here it is:
>> v10-0001-Avoid-reparameterizing-Paths-when-it-s-not-suitable.patch
> I forgot to mention that this patch applies on v16 not master.
I spent some time looking at this patch (which seems more urgent than
the patch for master, given that we have back-branch releases coming
up). There are two things I'm not persuaded about:
* Why is it okay to just use pull_varnos on a tablesample expression,
when contain_references_to() does something different?
* Is contain_references_to() doing the right thing by specifying
PVC_RECURSE_PLACEHOLDERS? That causes it to totally ignore a
PlaceHolderVar's ph_eval_at, and I'm not convinced that's correct.
Ideally it seems to me that we want to reject a PlaceHolderVar
if either its ph_eval_at or ph_lateral overlap the other join
relation; if it was coded that way then we'd not need to recurse
into the PHV's contents. pull_varnos isn't directly amenable
to this, but I think we could use pull_var_clause with
PVC_INCLUDE_PLACEHOLDERS and then iterate through the resulting
list manually. (If this patch were meant for HEAD, I'd think
about extending the var.c code to support this usage more directly.
But as things stand, this is a one-off so I think we should just do
what we must in reparameterize_path_by_child.)
BTW, it shouldn't be necessary to write either PVC_RECURSE_AGGREGATES
or PVC_RECURSE_WINDOWFUNCS, because neither kind of node should ever
appear in a scan-level expression. I'd leave those out so that we
get an error if something unexpected happens.
regards, tom lane
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Oversight in reparameterize_path_by_child leading to executor crash
@ 2024-01-31 06:39 Richard Guo <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Richard Guo @ 2024-01-31 06:39 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Robert Haas <[email protected]>; Andrei Lepikhov <[email protected]>; Alena Rybakina <[email protected]>; Ashutosh Bapat <[email protected]>; pgsql-hackers
On Wed, Jan 31, 2024 at 5:12 AM Tom Lane <[email protected]> wrote:
> Richard Guo <[email protected]> writes:
> > On Wed, Jan 17, 2024 at 5:01 PM Richard Guo <[email protected]>
> wrote:
> >> Sure, here it is:
> >> v10-0001-Avoid-reparameterizing-Paths-when-it-s-not-suitable.patch
>
> > I forgot to mention that this patch applies on v16 not master.
>
> I spent some time looking at this patch (which seems more urgent than
> the patch for master, given that we have back-branch releases coming
> up).
Thanks for looking at this patch!
> There are two things I'm not persuaded about:
>
> * Why is it okay to just use pull_varnos on a tablesample expression,
> when contain_references_to() does something different?
Hmm, the main reason is that the expression_tree_walker() function does
not handle T_RestrictInfo nodes. So we cannot just use pull_varnos or
pull_var_clause on a restrictinfo list. So contain_references_to()
calls extract_actual_clauses() first before it calls pull_var_clause().
> * Is contain_references_to() doing the right thing by specifying
> PVC_RECURSE_PLACEHOLDERS? That causes it to totally ignore a
> PlaceHolderVar's ph_eval_at, and I'm not convinced that's correct.
I was thinking that we should recurse into the PHV's contents to see if
there are any lateral references to the other join relation. But now I
realize that checking phinfo->ph_lateral, as you suggested, might be a
better way to do that.
But I'm not sure about checking phinfo->ph_eval_at. It seems to me that
the ph_eval_at could not overlap the other join relation; otherwise the
clause would not be a restriction clause but rather a join clause.
> Ideally it seems to me that we want to reject a PlaceHolderVar
> if either its ph_eval_at or ph_lateral overlap the other join
> relation; if it was coded that way then we'd not need to recurse
> into the PHV's contents. pull_varnos isn't directly amenable
> to this, but I think we could use pull_var_clause with
> PVC_INCLUDE_PLACEHOLDERS and then iterate through the resulting
> list manually. (If this patch were meant for HEAD, I'd think
> about extending the var.c code to support this usage more directly.
> But as things stand, this is a one-off so I think we should just do
> what we must in reparameterize_path_by_child.)
Thanks for the suggestion. I've coded it this way in the v11 patch, and
left a XXX comment about checking phinfo->ph_eval_at.
> BTW, it shouldn't be necessary to write either PVC_RECURSE_AGGREGATES
> or PVC_RECURSE_WINDOWFUNCS, because neither kind of node should ever
> appear in a scan-level expression. I'd leave those out so that we
> get an error if something unexpected happens.
Good point.
Thanks
Richard
Attachments:
[application/octet-stream] v11-0001-Avoid-reparameterizing-Paths-when-it-s-not-suitable.patch (18.5K, ../../CAMbWs4-EWe9tGPQHs6Am2jRqQEnFjSK7TfNfHTBZ2fCAs9eHYg@mail.gmail.com/3-v11-0001-Avoid-reparameterizing-Paths-when-it-s-not-suitable.patch)
download | inline diff:
From 5ca2fd2b4dbbbfbe55b6cb6662df42dd731cd877 Mon Sep 17 00:00:00 2001
From: Richard Guo <[email protected]>
Date: Wed, 17 Jan 2024 14:01:27 +0800
Subject: [PATCH v11] Avoid reparameterizing Paths when it's not suitable
When creating a nestloop join path, if the inner path is parameterized,
it is parameterized by the topmost parent of the outer rel, not the
outer rel itself. Therefore, we need to translate the parameterization
so that the inner path is parameterized by the given outer rel itself.
Currently, this is done in reparameterize_path_by_child() during
generating join paths.
However, reparameterize_path_by_child() does not perform the translation
for sample scan paths' sampling infos, or scan paths' restriction
clauses. This omission can lead to executor crashes, wrong results, or
planning errors, as we have already observed.
Please note that the sampling infos are contained in RangeTblEntries (as
TableSampleClause objects), and the scan restriction clauses are
contained in RelOptInfos or IndexOptInfos (as lists of RestrictInfo
objects). We cannot just modify them on the fly during generating join
paths. Doing so would break things if we end up using a
non-partitionwise join.
So this commit chooses not to reparameterize the path if there are
lateral references to the other relation in the sampling infos or
restriction clauses associated with the path.
---
src/backend/optimizer/util/pathnode.c | 136 ++++++++++++++-
src/test/regress/expected/partition_join.out | 167 +++++++++++++++++++
src/test/regress/sql/partition_join.sql | 48 ++++++
3 files changed, 350 insertions(+), 1 deletion(-)
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index 2185fc35a3..f2fc062f5c 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -26,6 +26,7 @@
#include "optimizer/optimizer.h"
#include "optimizer/pathnode.h"
#include "optimizer/paths.h"
+#include "optimizer/placeholder.h"
#include "optimizer/planmain.h"
#include "optimizer/prep.h"
#include "optimizer/restrictinfo.h"
@@ -56,6 +57,8 @@ static int append_startup_cost_compare(const ListCell *a, const ListCell *b);
static List *reparameterize_pathlist_by_child(PlannerInfo *root,
List *pathlist,
RelOptInfo *child_rel);
+static bool contain_references_to(PlannerInfo *root, List *restrictinfo_list,
+ Relids relids);
/*****************************************************************************
@@ -4103,13 +4106,60 @@ do { \
switch (nodeTag(path))
{
case T_Path:
- FLAT_COPY_PATH(new_path, path, Path);
+ {
+ /*
+ * If the path's restriction clauses contain lateral references
+ * to the other relation, we can't reparameterize, because we
+ * must not change the RelOptInfo's contents here. (Doing so
+ * would break things if we end up using a non-partitionwise
+ * join.)
+ */
+ if (contain_references_to(root, path->parent->baserestrictinfo,
+ child_rel->top_parent_relids))
+ return NULL;
+
+ /*
+ * If it's a SampleScan with tablesample parameters referencing
+ * the other relation, we can't reparameterize, because we must
+ * not change the RTE's contents here. (Doing so would break
+ * things if we end up using a non-partitionwise join.)
+ */
+ if (path->pathtype == T_SampleScan)
+ {
+ Index scan_relid = path->parent->relid;
+ RangeTblEntry *rte;
+
+ /* it should be a base rel with a tablesample clause... */
+ Assert(scan_relid > 0);
+ rte = planner_rt_fetch(scan_relid, root);
+ Assert(rte->rtekind == RTE_RELATION);
+ Assert(rte->tablesample != NULL);
+
+ if (bms_overlap(pull_varnos(root, (Node *) rte->tablesample),
+ child_rel->top_parent_relids))
+ return NULL;
+ }
+
+ FLAT_COPY_PATH(new_path, path, Path);
+ }
break;
case T_IndexPath:
{
IndexPath *ipath;
+ /*
+ * If the path's restriction clauses contain lateral references
+ * to the other relation, we can't reparameterize, because we
+ * must not change the IndexOptInfo's contents here. (Doing so
+ * would break things if we end up using a non-partitionwise
+ * join.)
+ */
+ if (contain_references_to(root,
+ ((IndexPath *) path)->indexinfo->indrestrictinfo,
+ child_rel->top_parent_relids))
+ return NULL;
+
FLAT_COPY_PATH(ipath, path, IndexPath);
ADJUST_CHILD_ATTRS(ipath->indexclauses);
new_path = (Path *) ipath;
@@ -4120,6 +4170,17 @@ do { \
{
BitmapHeapPath *bhpath;
+ /*
+ * If the path's restriction clauses contain lateral references
+ * to the other relation, we can't reparameterize, because we
+ * must not change the RelOptInfo's contents here. (Doing so
+ * would break things if we end up using a non-partitionwise
+ * join.)
+ */
+ if (contain_references_to(root, path->parent->baserestrictinfo,
+ child_rel->top_parent_relids))
+ return NULL;
+
FLAT_COPY_PATH(bhpath, path, BitmapHeapPath);
REPARAMETERIZE_CHILD_PATH(bhpath->bitmapqual);
new_path = (Path *) bhpath;
@@ -4151,6 +4212,17 @@ do { \
ForeignPath *fpath;
ReparameterizeForeignPathByChild_function rfpc_func;
+ /*
+ * If the path's restriction clauses contain lateral references
+ * to the other relation, we can't reparameterize, because we
+ * must not change the RelOptInfo's contents here. (Doing so
+ * would break things if we end up using a non-partitionwise
+ * join.)
+ */
+ if (contain_references_to(root, path->parent->baserestrictinfo,
+ child_rel->top_parent_relids))
+ return NULL;
+
FLAT_COPY_PATH(fpath, path, ForeignPath);
if (fpath->fdw_outerpath)
REPARAMETERIZE_CHILD_PATH(fpath->fdw_outerpath);
@@ -4169,6 +4241,17 @@ do { \
{
CustomPath *cpath;
+ /*
+ * If the path's restriction clauses contain lateral references
+ * to the other relation, we can't reparameterize, because we
+ * must not change the RelOptInfo's contents here. (Doing so
+ * would break things if we end up using a non-partitionwise
+ * join.)
+ */
+ if (contain_references_to(root, path->parent->baserestrictinfo,
+ child_rel->top_parent_relids))
+ return NULL;
+
FLAT_COPY_PATH(cpath, path, CustomPath);
REPARAMETERIZE_CHILD_PATH_LIST(cpath->custom_paths);
if (cpath->methods &&
@@ -4358,3 +4441,54 @@ reparameterize_pathlist_by_child(PlannerInfo *root,
return result;
}
+
+/*
+ * contain_references_to
+ * Detect whether any PlaceHolderVars in the given 'restrictinfo_list'
+ * contain lateral references to the given 'relids'.
+ */
+static bool
+contain_references_to(PlannerInfo *root, List *restrictinfo_list,
+ Relids relids)
+{
+ List *scan_clauses;
+ List *vars;
+ ListCell *lc;
+ bool ret = false;
+
+ scan_clauses = extract_actual_clauses(restrictinfo_list, false);
+
+ /*
+ * Examine all PlaceHolderVars used in the restriction list.
+ *
+ * By omitting the relevant flags, this also gives us a cheap sanity check
+ * that no aggregates or window functions appear in the restriction list.
+ */
+ vars = pull_var_clause((Node *) scan_clauses, PVC_INCLUDE_PLACEHOLDERS);
+ foreach(lc, vars)
+ {
+ PlaceHolderVar *phv = (PlaceHolderVar *) lfirst(lc);
+ PlaceHolderInfo *phinfo;
+
+ /* Ignore any plain Vars */
+ if (!IsA(phv, PlaceHolderVar))
+ continue;
+
+ /*
+ * Check if this PHV contains lateral references to the given 'relids'.
+ *
+ * XXX do we really need to check ph_eval_at?
+ */
+ phinfo = find_placeholder_info(root, phv);
+ if (bms_overlap(phinfo->ph_eval_at, relids) ||
+ bms_overlap(phinfo->ph_lateral, relids))
+ {
+ ret = true;
+ break;
+ }
+ }
+
+ list_free(vars);
+
+ return ret;
+}
diff --git a/src/test/regress/expected/partition_join.out b/src/test/regress/expected/partition_join.out
index 6560fe2416..72f286b893 100644
--- a/src/test/regress/expected/partition_join.out
+++ b/src/test/regress/expected/partition_join.out
@@ -505,6 +505,99 @@ SELECT t1.a, ss.t2a, ss.t2c FROM prt1 t1 LEFT JOIN LATERAL
550 | |
(12 rows)
+SET max_parallel_workers_per_gather = 0;
+-- If there are lateral references to the other relation in sample scan, we'd
+-- fail to generate a partitionwise join.
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt1 t1 JOIN LATERAL
+ (SELECT * FROM prt1 t2 TABLESAMPLE SYSTEM (t1.a) REPEATABLE(t1.b)) s
+ ON t1.a = s.a;
+ QUERY PLAN
+---------------------------------------------------------
+ Nested Loop
+ -> Append
+ -> Seq Scan on prt1_p1 t1_1
+ -> Seq Scan on prt1_p2 t1_2
+ -> Seq Scan on prt1_p3 t1_3
+ -> Append
+ -> Sample Scan on prt1_p1 t2_1
+ Sampling: system (t1.a) REPEATABLE (t1.b)
+ Filter: (t1.a = a)
+ -> Sample Scan on prt1_p2 t2_2
+ Sampling: system (t1.a) REPEATABLE (t1.b)
+ Filter: (t1.a = a)
+ -> Sample Scan on prt1_p3 t2_3
+ Sampling: system (t1.a) REPEATABLE (t1.b)
+ Filter: (t1.a = a)
+(15 rows)
+
+-- If there are lateral references to the other relation in scan's restriction
+-- clauses, we'd fail to generate a partitionwise join.
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.a;
+ QUERY PLAN
+---------------------------------------------------------------
+ Aggregate
+ -> Nested Loop
+ -> Append
+ -> Seq Scan on prt1_p1 t1_1
+ -> Seq Scan on prt1_p2 t1_2
+ -> Seq Scan on prt1_p3 t1_3
+ -> Append
+ -> Index Scan using iprt2_p1_b on prt2_p1 t2_1
+ Index Cond: (b = t1.a)
+ Filter: (t1.b = a)
+ -> Index Scan using iprt2_p2_b on prt2_p2 t2_2
+ Index Cond: (b = t1.a)
+ Filter: (t1.b = a)
+ -> Index Scan using iprt2_p3_b on prt2_p3 t2_3
+ Index Cond: (b = t1.a)
+ Filter: (t1.b = a)
+(16 rows)
+
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.a;
+ count
+-------
+ 100
+(1 row)
+
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.b;
+ QUERY PLAN
+--------------------------------------------------------------------
+ Aggregate
+ -> Nested Loop
+ -> Append
+ -> Seq Scan on prt1_p1 t1_1
+ -> Seq Scan on prt1_p2 t1_2
+ -> Seq Scan on prt1_p3 t1_3
+ -> Append
+ -> Index Only Scan using iprt2_p1_b on prt2_p1 t2_1
+ Index Cond: (b = t1.a)
+ Filter: (b = t1.b)
+ -> Index Only Scan using iprt2_p2_b on prt2_p2 t2_2
+ Index Cond: (b = t1.a)
+ Filter: (b = t1.b)
+ -> Index Only Scan using iprt2_p3_b on prt2_p3 t2_3
+ Index Cond: (b = t1.a)
+ Filter: (b = t1.b)
+(16 rows)
+
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.b;
+ count
+-------
+ 5
+(1 row)
+
+RESET max_parallel_workers_per_gather;
-- bug with inadequate sort key representation
SET enable_partitionwise_aggregate TO true;
SET enable_hashjoin TO false;
@@ -1944,6 +2037,80 @@ SELECT * FROM prt1_l t1 LEFT JOIN LATERAL
550 | 0 | 0002 | | | | |
(12 rows)
+SET max_parallel_workers_per_gather = 0;
+-- If there are lateral references to the other relation in sample scan, we'd
+-- fail to generate a partitionwise join.
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt1_l t1 JOIN LATERAL
+ (SELECT * FROM prt1_l t2 TABLESAMPLE SYSTEM (t1.a) REPEATABLE(t1.b)) s
+ ON t1.a = s.a AND t1.b = s.b AND t1.c = s.c;
+ QUERY PLAN
+----------------------------------------------------------------------------------
+ Nested Loop
+ -> Append
+ -> Seq Scan on prt1_l_p1 t1_1
+ -> Seq Scan on prt1_l_p2_p1 t1_2
+ -> Seq Scan on prt1_l_p2_p2 t1_3
+ -> Seq Scan on prt1_l_p3_p1 t1_4
+ -> Seq Scan on prt1_l_p3_p2 t1_5
+ -> Append
+ -> Sample Scan on prt1_l_p1 t2_1
+ Sampling: system (t1.a) REPEATABLE (t1.b)
+ Filter: ((t1.a = a) AND (t1.b = b) AND ((t1.c)::text = (c)::text))
+ -> Sample Scan on prt1_l_p2_p1 t2_2
+ Sampling: system (t1.a) REPEATABLE (t1.b)
+ Filter: ((t1.a = a) AND (t1.b = b) AND ((t1.c)::text = (c)::text))
+ -> Sample Scan on prt1_l_p2_p2 t2_3
+ Sampling: system (t1.a) REPEATABLE (t1.b)
+ Filter: ((t1.a = a) AND (t1.b = b) AND ((t1.c)::text = (c)::text))
+ -> Sample Scan on prt1_l_p3_p1 t2_4
+ Sampling: system (t1.a) REPEATABLE (t1.b)
+ Filter: ((t1.a = a) AND (t1.b = b) AND ((t1.c)::text = (c)::text))
+ -> Sample Scan on prt1_l_p3_p2 t2_5
+ Sampling: system (t1.a) REPEATABLE (t1.b)
+ Filter: ((t1.a = a) AND (t1.b = b) AND ((t1.c)::text = (c)::text))
+(23 rows)
+
+-- If there are lateral references to the other relation in scan's restriction
+-- clauses, we'd fail to generate a partitionwise join.
+EXPLAIN (COSTS OFF)
+SELECT COUNT(*) FROM prt1_l t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2_l t2) s
+ ON t1.a = s.b AND t1.b = s.a AND t1.c = s.c
+ WHERE s.t1b = s.a;
+ QUERY PLAN
+-------------------------------------------------------------------------------------------------------
+ Aggregate
+ -> Nested Loop
+ -> Append
+ -> Seq Scan on prt1_l_p1 t1_1
+ -> Seq Scan on prt1_l_p2_p1 t1_2
+ -> Seq Scan on prt1_l_p2_p2 t1_3
+ -> Seq Scan on prt1_l_p3_p1 t1_4
+ -> Seq Scan on prt1_l_p3_p2 t1_5
+ -> Append
+ -> Seq Scan on prt2_l_p1 t2_1
+ Filter: ((a = t1.b) AND (t1.a = b) AND (t1.b = a) AND ((t1.c)::text = (c)::text))
+ -> Seq Scan on prt2_l_p2_p1 t2_2
+ Filter: ((a = t1.b) AND (t1.a = b) AND (t1.b = a) AND ((t1.c)::text = (c)::text))
+ -> Seq Scan on prt2_l_p2_p2 t2_3
+ Filter: ((a = t1.b) AND (t1.a = b) AND (t1.b = a) AND ((t1.c)::text = (c)::text))
+ -> Seq Scan on prt2_l_p3_p1 t2_4
+ Filter: ((a = t1.b) AND (t1.a = b) AND (t1.b = a) AND ((t1.c)::text = (c)::text))
+ -> Seq Scan on prt2_l_p3_p2 t2_5
+ Filter: ((a = t1.b) AND (t1.a = b) AND (t1.b = a) AND ((t1.c)::text = (c)::text))
+(19 rows)
+
+SELECT COUNT(*) FROM prt1_l t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2_l t2) s
+ ON t1.a = s.b AND t1.b = s.a AND t1.c = s.c
+ WHERE s.t1b = s.a;
+ count
+-------
+ 100
+(1 row)
+
+RESET max_parallel_workers_per_gather;
-- join with one side empty
EXPLAIN (COSTS OFF)
SELECT t1.a, t1.c, t2.b, t2.c FROM (SELECT * FROM prt1_l WHERE a = 1 AND a = 2) t1 RIGHT JOIN prt2_l t2 ON t1.a = t2.b AND t1.b = t2.a AND t1.c = t2.c;
diff --git a/src/test/regress/sql/partition_join.sql b/src/test/regress/sql/partition_join.sql
index 48daf3aee3..0554a6568b 100644
--- a/src/test/regress/sql/partition_join.sql
+++ b/src/test/regress/sql/partition_join.sql
@@ -100,6 +100,33 @@ SELECT t1.a, ss.t2a, ss.t2c FROM prt1 t1 LEFT JOIN LATERAL
(SELECT t2.a AS t2a, t3.a AS t3a, t2.b t2b, t2.c t2c, least(t1.a,t2.a,t3.a) FROM prt1 t2 JOIN prt2 t3 ON (t2.a = t3.b)) ss
ON t1.c = ss.t2c WHERE (t1.b + coalesce(ss.t2b, 0)) = 0 ORDER BY t1.a;
+SET max_parallel_workers_per_gather = 0;
+-- If there are lateral references to the other relation in sample scan, we'd
+-- fail to generate a partitionwise join.
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt1 t1 JOIN LATERAL
+ (SELECT * FROM prt1 t2 TABLESAMPLE SYSTEM (t1.a) REPEATABLE(t1.b)) s
+ ON t1.a = s.a;
+
+-- If there are lateral references to the other relation in scan's restriction
+-- clauses, we'd fail to generate a partitionwise join.
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.a;
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.a;
+
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.b;
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.b;
+RESET max_parallel_workers_per_gather;
+
-- bug with inadequate sort key representation
SET enable_partitionwise_aggregate TO true;
SET enable_hashjoin TO false;
@@ -387,6 +414,27 @@ SELECT * FROM prt1_l t1 LEFT JOIN LATERAL
(SELECT t2.a AS t2a, t2.c AS t2c, t2.b AS t2b, t3.b AS t3b, least(t1.a,t2.a,t3.b) FROM prt1_l t2 JOIN prt2_l t3 ON (t2.a = t3.b AND t2.c = t3.c)) ss
ON t1.a = ss.t2a AND t1.c = ss.t2c WHERE t1.b = 0 ORDER BY t1.a;
+SET max_parallel_workers_per_gather = 0;
+-- If there are lateral references to the other relation in sample scan, we'd
+-- fail to generate a partitionwise join.
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt1_l t1 JOIN LATERAL
+ (SELECT * FROM prt1_l t2 TABLESAMPLE SYSTEM (t1.a) REPEATABLE(t1.b)) s
+ ON t1.a = s.a AND t1.b = s.b AND t1.c = s.c;
+
+-- If there are lateral references to the other relation in scan's restriction
+-- clauses, we'd fail to generate a partitionwise join.
+EXPLAIN (COSTS OFF)
+SELECT COUNT(*) FROM prt1_l t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2_l t2) s
+ ON t1.a = s.b AND t1.b = s.a AND t1.c = s.c
+ WHERE s.t1b = s.a;
+SELECT COUNT(*) FROM prt1_l t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2_l t2) s
+ ON t1.a = s.b AND t1.b = s.a AND t1.c = s.c
+ WHERE s.t1b = s.a;
+RESET max_parallel_workers_per_gather;
+
-- join with one side empty
EXPLAIN (COSTS OFF)
SELECT t1.a, t1.c, t2.b, t2.c FROM (SELECT * FROM prt1_l WHERE a = 1 AND a = 2) t1 RIGHT JOIN prt2_l t2 ON t1.a = t2.b AND t1.b = t2.a AND t1.c = t2.c;
--
2.31.0
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Oversight in reparameterize_path_by_child leading to executor crash
@ 2024-01-31 15:21 Tom Lane <[email protected]>
parent: Richard Guo <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Tom Lane @ 2024-01-31 15:21 UTC (permalink / raw)
To: Richard Guo <[email protected]>; +Cc: Robert Haas <[email protected]>; Andrei Lepikhov <[email protected]>; Alena Rybakina <[email protected]>; Ashutosh Bapat <[email protected]>; pgsql-hackers
Richard Guo <[email protected]> writes:
> On Wed, Jan 31, 2024 at 5:12 AM Tom Lane <[email protected]> wrote:
>> * Why is it okay to just use pull_varnos on a tablesample expression,
>> when contain_references_to() does something different?
> Hmm, the main reason is that the expression_tree_walker() function does
> not handle T_RestrictInfo nodes. So we cannot just use pull_varnos or
> pull_var_clause on a restrictinfo list.
Right, the extract_actual_clauses step is not wanted for the
tablesample expression. But my point is that surely the handling of
Vars and PlaceHolderVars should be the same, which it's not, and your
v11 makes it even less so. How can it be OK to ignore Vars in the
restrictinfo case?
I think the code structure we need to end up with is a routine that
takes a RestrictInfo-free node tree (and is called directly in the
tablesample case) with a wrapper routine that strips the RestrictInfos
(for use on restriction lists).
> But I'm not sure about checking phinfo->ph_eval_at. It seems to me that
> the ph_eval_at could not overlap the other join relation; otherwise the
> clause would not be a restriction clause but rather a join clause.
At least in the tablesample case, plain Vars as well as PHVs belonging
to the other relation are definitely possible.
regards, tom lane
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Oversight in reparameterize_path_by_child leading to executor crash
@ 2024-02-01 02:54 Richard Guo <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Richard Guo @ 2024-02-01 02:54 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Robert Haas <[email protected]>; Andrei Lepikhov <[email protected]>; Alena Rybakina <[email protected]>; Ashutosh Bapat <[email protected]>; pgsql-hackers
On Wed, Jan 31, 2024 at 11:21 PM Tom Lane <[email protected]> wrote:
> Richard Guo <[email protected]> writes:
> > On Wed, Jan 31, 2024 at 5:12 AM Tom Lane <[email protected]> wrote:
> >> * Why is it okay to just use pull_varnos on a tablesample expression,
> >> when contain_references_to() does something different?
>
> > Hmm, the main reason is that the expression_tree_walker() function does
> > not handle T_RestrictInfo nodes. So we cannot just use pull_varnos or
> > pull_var_clause on a restrictinfo list.
>
> Right, the extract_actual_clauses step is not wanted for the
> tablesample expression. But my point is that surely the handling of
> Vars and PlaceHolderVars should be the same, which it's not, and your
> v11 makes it even less so. How can it be OK to ignore Vars in the
> restrictinfo case?
Hmm, in this specific scenario it seems that it's not possible to have
Vars in the restrictinfo list that laterally reference the outer join
relation; otherwise the clause containing such Vars would not be a
restriction clause but rather a join clause. So in v11 I did not check
Vars in contain_references_to().
But I agree that we'd better to handle Vars and PlaceHolderVars in the
same way.
> I think the code structure we need to end up with is a routine that
> takes a RestrictInfo-free node tree (and is called directly in the
> tablesample case) with a wrapper routine that strips the RestrictInfos
> (for use on restriction lists).
How about the attached v12 patch? But I'm a little concerned about
omitting PVC_RECURSE_AGGREGATES and PVC_RECURSE_WINDOWFUNCS in this
case. Is it possible that aggregates or window functions appear in the
tablesample expression?
Thanks
Richard
Attachments:
[application/octet-stream] v12-0001-Avoid-reparameterizing-Paths-when-it-s-not-suitable.patch (18.9K, ../../CAMbWs4-61GVu5eDk7m=jO=fAAxM-5gKy=Ngb8COZKmW-=nHLXA@mail.gmail.com/3-v12-0001-Avoid-reparameterizing-Paths-when-it-s-not-suitable.patch)
download | inline diff:
From 52db54b4d28c904682f3a72d5669e3e681241168 Mon Sep 17 00:00:00 2001
From: Richard Guo <[email protected]>
Date: Wed, 17 Jan 2024 14:01:27 +0800
Subject: [PATCH v12] Avoid reparameterizing Paths when it's not suitable
When creating a nestloop join path, if the inner path is parameterized,
it is parameterized by the topmost parent of the outer rel, not the
outer rel itself. Therefore, we need to translate the parameterization
so that the inner path is parameterized by the given outer rel itself.
Currently, this is done in reparameterize_path_by_child() during
generating join paths.
However, reparameterize_path_by_child() does not perform the translation
for sample scan paths' sampling infos, or scan paths' restriction
clauses. This omission can lead to executor crashes, wrong results, or
planning errors, as we have already observed.
Please note that the sampling infos are contained in RangeTblEntries (as
TableSampleClause objects), and the scan restriction clauses are
contained in RelOptInfos or IndexOptInfos (as lists of RestrictInfo
objects). We cannot just modify them on the fly during generating join
paths. Doing so would break things if we end up using a
non-partitionwise join.
So this commit chooses not to reparameterize the path if there are
lateral references to the other relation in the sampling infos or
restriction clauses associated with the path.
---
src/backend/optimizer/util/pathnode.c | 159 +++++++++++++++++-
src/test/regress/expected/partition_join.out | 167 +++++++++++++++++++
src/test/regress/sql/partition_join.sql | 48 ++++++
3 files changed, 373 insertions(+), 1 deletion(-)
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index 2185fc35a3..ebd3b6eec3 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -26,6 +26,7 @@
#include "optimizer/optimizer.h"
#include "optimizer/pathnode.h"
#include "optimizer/paths.h"
+#include "optimizer/placeholder.h"
#include "optimizer/planmain.h"
#include "optimizer/prep.h"
#include "optimizer/restrictinfo.h"
@@ -56,6 +57,8 @@ static int append_startup_cost_compare(const ListCell *a, const ListCell *b);
static List *reparameterize_pathlist_by_child(PlannerInfo *root,
List *pathlist,
RelOptInfo *child_rel);
+static bool contain_references_to(PlannerInfo *root, Node *clause,
+ Relids relids);
/*****************************************************************************
@@ -4103,12 +4106,67 @@ do { \
switch (nodeTag(path))
{
case T_Path:
- FLAT_COPY_PATH(new_path, path, Path);
+ {
+ List *scan_clauses;
+
+ /*
+ * If the path's restriction clauses contain lateral references
+ * to the other relation, we can't reparameterize, because we
+ * must not change the RelOptInfo's contents here. (Doing so
+ * would break things if we end up using a non-partitionwise
+ * join.)
+ */
+ scan_clauses =
+ extract_actual_clauses(path->parent->baserestrictinfo,
+ false);
+ if (contain_references_to(root, (Node *) scan_clauses,
+ child_rel->top_parent_relids))
+ return NULL;
+
+ /*
+ * If it's a SampleScan with tablesample parameters referencing
+ * the other relation, we can't reparameterize, because we must
+ * not change the RTE's contents here. (Doing so would break
+ * things if we end up using a non-partitionwise join.)
+ */
+ if (path->pathtype == T_SampleScan)
+ {
+ Index scan_relid = path->parent->relid;
+ RangeTblEntry *rte;
+
+ /* it should be a base rel with a tablesample clause... */
+ Assert(scan_relid > 0);
+ rte = planner_rt_fetch(scan_relid, root);
+ Assert(rte->rtekind == RTE_RELATION);
+ Assert(rte->tablesample != NULL);
+
+ if (contain_references_to(root, (Node *) rte->tablesample,
+ child_rel->top_parent_relids))
+ return NULL;
+ }
+
+ FLAT_COPY_PATH(new_path, path, Path);
+ }
break;
case T_IndexPath:
{
IndexPath *ipath;
+ List *scan_clauses;
+
+ /*
+ * If the path's restriction clauses contain lateral references
+ * to the other relation, we can't reparameterize, because we
+ * must not change the IndexOptInfo's contents here. (Doing so
+ * would break things if we end up using a non-partitionwise
+ * join.)
+ */
+ scan_clauses =
+ extract_actual_clauses(((IndexPath *) path)->indexinfo->indrestrictinfo,
+ false);
+ if (contain_references_to(root, (Node *) scan_clauses,
+ child_rel->top_parent_relids))
+ return NULL;
FLAT_COPY_PATH(ipath, path, IndexPath);
ADJUST_CHILD_ATTRS(ipath->indexclauses);
@@ -4119,6 +4177,21 @@ do { \
case T_BitmapHeapPath:
{
BitmapHeapPath *bhpath;
+ List *scan_clauses;
+
+ /*
+ * If the path's restriction clauses contain lateral references
+ * to the other relation, we can't reparameterize, because we
+ * must not change the RelOptInfo's contents here. (Doing so
+ * would break things if we end up using a non-partitionwise
+ * join.)
+ */
+ scan_clauses =
+ extract_actual_clauses(path->parent->baserestrictinfo,
+ false);
+ if (contain_references_to(root, (Node *) scan_clauses,
+ child_rel->top_parent_relids))
+ return NULL;
FLAT_COPY_PATH(bhpath, path, BitmapHeapPath);
REPARAMETERIZE_CHILD_PATH(bhpath->bitmapqual);
@@ -4150,6 +4223,21 @@ do { \
{
ForeignPath *fpath;
ReparameterizeForeignPathByChild_function rfpc_func;
+ List *scan_clauses;
+
+ /*
+ * If the path's restriction clauses contain lateral references
+ * to the other relation, we can't reparameterize, because we
+ * must not change the RelOptInfo's contents here. (Doing so
+ * would break things if we end up using a non-partitionwise
+ * join.)
+ */
+ scan_clauses =
+ extract_actual_clauses(path->parent->baserestrictinfo,
+ false);
+ if (contain_references_to(root, (Node *) scan_clauses,
+ child_rel->top_parent_relids))
+ return NULL;
FLAT_COPY_PATH(fpath, path, ForeignPath);
if (fpath->fdw_outerpath)
@@ -4168,6 +4256,21 @@ do { \
case T_CustomPath:
{
CustomPath *cpath;
+ List *scan_clauses;
+
+ /*
+ * If the path's restriction clauses contain lateral references
+ * to the other relation, we can't reparameterize, because we
+ * must not change the RelOptInfo's contents here. (Doing so
+ * would break things if we end up using a non-partitionwise
+ * join.)
+ */
+ scan_clauses =
+ extract_actual_clauses(path->parent->baserestrictinfo,
+ false);
+ if (contain_references_to(root, (Node *) scan_clauses,
+ child_rel->top_parent_relids))
+ return NULL;
FLAT_COPY_PATH(cpath, path, CustomPath);
REPARAMETERIZE_CHILD_PATH_LIST(cpath->custom_paths);
@@ -4358,3 +4461,57 @@ reparameterize_pathlist_by_child(PlannerInfo *root,
return result;
}
+
+/*
+ * contain_references_to
+ * Detect whether any Vars or PlaceHolderVars in the given clause contain
+ * lateral references to the given 'relids'.
+ */
+static bool
+contain_references_to(PlannerInfo *root, Node *clause, Relids relids)
+{
+ List *vars;
+ ListCell *lc;
+ bool ret = false;
+
+ /*
+ * Examine all Vars and PlaceHolderVars used in the clause.
+ *
+ * By omitting the relevant flags, this also gives us a cheap sanity check
+ * that no aggregates or window functions appear in the clause.
+ */
+ vars = pull_var_clause(clause, PVC_INCLUDE_PLACEHOLDERS);
+ foreach(lc, vars)
+ {
+ Node *node = (Node *) lfirst(lc);
+
+ if (IsA(node, Var))
+ {
+ Var *var = (Var *) node;
+
+ if (bms_is_member(var->varno, relids))
+ {
+ ret = true;
+ break;
+ }
+ }
+ else if (IsA(node, PlaceHolderVar))
+ {
+ PlaceHolderVar *phv = (PlaceHolderVar *) node;
+ PlaceHolderInfo *phinfo = find_placeholder_info(root, phv);
+
+ if (bms_overlap(phinfo->ph_eval_at, relids) ||
+ bms_overlap(phinfo->ph_lateral, relids))
+ {
+ ret = true;
+ break;
+ }
+ }
+ else
+ Assert(false);
+ }
+
+ list_free(vars);
+
+ return ret;
+}
diff --git a/src/test/regress/expected/partition_join.out b/src/test/regress/expected/partition_join.out
index 6560fe2416..72f286b893 100644
--- a/src/test/regress/expected/partition_join.out
+++ b/src/test/regress/expected/partition_join.out
@@ -505,6 +505,99 @@ SELECT t1.a, ss.t2a, ss.t2c FROM prt1 t1 LEFT JOIN LATERAL
550 | |
(12 rows)
+SET max_parallel_workers_per_gather = 0;
+-- If there are lateral references to the other relation in sample scan, we'd
+-- fail to generate a partitionwise join.
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt1 t1 JOIN LATERAL
+ (SELECT * FROM prt1 t2 TABLESAMPLE SYSTEM (t1.a) REPEATABLE(t1.b)) s
+ ON t1.a = s.a;
+ QUERY PLAN
+---------------------------------------------------------
+ Nested Loop
+ -> Append
+ -> Seq Scan on prt1_p1 t1_1
+ -> Seq Scan on prt1_p2 t1_2
+ -> Seq Scan on prt1_p3 t1_3
+ -> Append
+ -> Sample Scan on prt1_p1 t2_1
+ Sampling: system (t1.a) REPEATABLE (t1.b)
+ Filter: (t1.a = a)
+ -> Sample Scan on prt1_p2 t2_2
+ Sampling: system (t1.a) REPEATABLE (t1.b)
+ Filter: (t1.a = a)
+ -> Sample Scan on prt1_p3 t2_3
+ Sampling: system (t1.a) REPEATABLE (t1.b)
+ Filter: (t1.a = a)
+(15 rows)
+
+-- If there are lateral references to the other relation in scan's restriction
+-- clauses, we'd fail to generate a partitionwise join.
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.a;
+ QUERY PLAN
+---------------------------------------------------------------
+ Aggregate
+ -> Nested Loop
+ -> Append
+ -> Seq Scan on prt1_p1 t1_1
+ -> Seq Scan on prt1_p2 t1_2
+ -> Seq Scan on prt1_p3 t1_3
+ -> Append
+ -> Index Scan using iprt2_p1_b on prt2_p1 t2_1
+ Index Cond: (b = t1.a)
+ Filter: (t1.b = a)
+ -> Index Scan using iprt2_p2_b on prt2_p2 t2_2
+ Index Cond: (b = t1.a)
+ Filter: (t1.b = a)
+ -> Index Scan using iprt2_p3_b on prt2_p3 t2_3
+ Index Cond: (b = t1.a)
+ Filter: (t1.b = a)
+(16 rows)
+
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.a;
+ count
+-------
+ 100
+(1 row)
+
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.b;
+ QUERY PLAN
+--------------------------------------------------------------------
+ Aggregate
+ -> Nested Loop
+ -> Append
+ -> Seq Scan on prt1_p1 t1_1
+ -> Seq Scan on prt1_p2 t1_2
+ -> Seq Scan on prt1_p3 t1_3
+ -> Append
+ -> Index Only Scan using iprt2_p1_b on prt2_p1 t2_1
+ Index Cond: (b = t1.a)
+ Filter: (b = t1.b)
+ -> Index Only Scan using iprt2_p2_b on prt2_p2 t2_2
+ Index Cond: (b = t1.a)
+ Filter: (b = t1.b)
+ -> Index Only Scan using iprt2_p3_b on prt2_p3 t2_3
+ Index Cond: (b = t1.a)
+ Filter: (b = t1.b)
+(16 rows)
+
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.b;
+ count
+-------
+ 5
+(1 row)
+
+RESET max_parallel_workers_per_gather;
-- bug with inadequate sort key representation
SET enable_partitionwise_aggregate TO true;
SET enable_hashjoin TO false;
@@ -1944,6 +2037,80 @@ SELECT * FROM prt1_l t1 LEFT JOIN LATERAL
550 | 0 | 0002 | | | | |
(12 rows)
+SET max_parallel_workers_per_gather = 0;
+-- If there are lateral references to the other relation in sample scan, we'd
+-- fail to generate a partitionwise join.
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt1_l t1 JOIN LATERAL
+ (SELECT * FROM prt1_l t2 TABLESAMPLE SYSTEM (t1.a) REPEATABLE(t1.b)) s
+ ON t1.a = s.a AND t1.b = s.b AND t1.c = s.c;
+ QUERY PLAN
+----------------------------------------------------------------------------------
+ Nested Loop
+ -> Append
+ -> Seq Scan on prt1_l_p1 t1_1
+ -> Seq Scan on prt1_l_p2_p1 t1_2
+ -> Seq Scan on prt1_l_p2_p2 t1_3
+ -> Seq Scan on prt1_l_p3_p1 t1_4
+ -> Seq Scan on prt1_l_p3_p2 t1_5
+ -> Append
+ -> Sample Scan on prt1_l_p1 t2_1
+ Sampling: system (t1.a) REPEATABLE (t1.b)
+ Filter: ((t1.a = a) AND (t1.b = b) AND ((t1.c)::text = (c)::text))
+ -> Sample Scan on prt1_l_p2_p1 t2_2
+ Sampling: system (t1.a) REPEATABLE (t1.b)
+ Filter: ((t1.a = a) AND (t1.b = b) AND ((t1.c)::text = (c)::text))
+ -> Sample Scan on prt1_l_p2_p2 t2_3
+ Sampling: system (t1.a) REPEATABLE (t1.b)
+ Filter: ((t1.a = a) AND (t1.b = b) AND ((t1.c)::text = (c)::text))
+ -> Sample Scan on prt1_l_p3_p1 t2_4
+ Sampling: system (t1.a) REPEATABLE (t1.b)
+ Filter: ((t1.a = a) AND (t1.b = b) AND ((t1.c)::text = (c)::text))
+ -> Sample Scan on prt1_l_p3_p2 t2_5
+ Sampling: system (t1.a) REPEATABLE (t1.b)
+ Filter: ((t1.a = a) AND (t1.b = b) AND ((t1.c)::text = (c)::text))
+(23 rows)
+
+-- If there are lateral references to the other relation in scan's restriction
+-- clauses, we'd fail to generate a partitionwise join.
+EXPLAIN (COSTS OFF)
+SELECT COUNT(*) FROM prt1_l t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2_l t2) s
+ ON t1.a = s.b AND t1.b = s.a AND t1.c = s.c
+ WHERE s.t1b = s.a;
+ QUERY PLAN
+-------------------------------------------------------------------------------------------------------
+ Aggregate
+ -> Nested Loop
+ -> Append
+ -> Seq Scan on prt1_l_p1 t1_1
+ -> Seq Scan on prt1_l_p2_p1 t1_2
+ -> Seq Scan on prt1_l_p2_p2 t1_3
+ -> Seq Scan on prt1_l_p3_p1 t1_4
+ -> Seq Scan on prt1_l_p3_p2 t1_5
+ -> Append
+ -> Seq Scan on prt2_l_p1 t2_1
+ Filter: ((a = t1.b) AND (t1.a = b) AND (t1.b = a) AND ((t1.c)::text = (c)::text))
+ -> Seq Scan on prt2_l_p2_p1 t2_2
+ Filter: ((a = t1.b) AND (t1.a = b) AND (t1.b = a) AND ((t1.c)::text = (c)::text))
+ -> Seq Scan on prt2_l_p2_p2 t2_3
+ Filter: ((a = t1.b) AND (t1.a = b) AND (t1.b = a) AND ((t1.c)::text = (c)::text))
+ -> Seq Scan on prt2_l_p3_p1 t2_4
+ Filter: ((a = t1.b) AND (t1.a = b) AND (t1.b = a) AND ((t1.c)::text = (c)::text))
+ -> Seq Scan on prt2_l_p3_p2 t2_5
+ Filter: ((a = t1.b) AND (t1.a = b) AND (t1.b = a) AND ((t1.c)::text = (c)::text))
+(19 rows)
+
+SELECT COUNT(*) FROM prt1_l t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2_l t2) s
+ ON t1.a = s.b AND t1.b = s.a AND t1.c = s.c
+ WHERE s.t1b = s.a;
+ count
+-------
+ 100
+(1 row)
+
+RESET max_parallel_workers_per_gather;
-- join with one side empty
EXPLAIN (COSTS OFF)
SELECT t1.a, t1.c, t2.b, t2.c FROM (SELECT * FROM prt1_l WHERE a = 1 AND a = 2) t1 RIGHT JOIN prt2_l t2 ON t1.a = t2.b AND t1.b = t2.a AND t1.c = t2.c;
diff --git a/src/test/regress/sql/partition_join.sql b/src/test/regress/sql/partition_join.sql
index 48daf3aee3..0554a6568b 100644
--- a/src/test/regress/sql/partition_join.sql
+++ b/src/test/regress/sql/partition_join.sql
@@ -100,6 +100,33 @@ SELECT t1.a, ss.t2a, ss.t2c FROM prt1 t1 LEFT JOIN LATERAL
(SELECT t2.a AS t2a, t3.a AS t3a, t2.b t2b, t2.c t2c, least(t1.a,t2.a,t3.a) FROM prt1 t2 JOIN prt2 t3 ON (t2.a = t3.b)) ss
ON t1.c = ss.t2c WHERE (t1.b + coalesce(ss.t2b, 0)) = 0 ORDER BY t1.a;
+SET max_parallel_workers_per_gather = 0;
+-- If there are lateral references to the other relation in sample scan, we'd
+-- fail to generate a partitionwise join.
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt1 t1 JOIN LATERAL
+ (SELECT * FROM prt1 t2 TABLESAMPLE SYSTEM (t1.a) REPEATABLE(t1.b)) s
+ ON t1.a = s.a;
+
+-- If there are lateral references to the other relation in scan's restriction
+-- clauses, we'd fail to generate a partitionwise join.
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.a;
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.a;
+
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.b;
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.b;
+RESET max_parallel_workers_per_gather;
+
-- bug with inadequate sort key representation
SET enable_partitionwise_aggregate TO true;
SET enable_hashjoin TO false;
@@ -387,6 +414,27 @@ SELECT * FROM prt1_l t1 LEFT JOIN LATERAL
(SELECT t2.a AS t2a, t2.c AS t2c, t2.b AS t2b, t3.b AS t3b, least(t1.a,t2.a,t3.b) FROM prt1_l t2 JOIN prt2_l t3 ON (t2.a = t3.b AND t2.c = t3.c)) ss
ON t1.a = ss.t2a AND t1.c = ss.t2c WHERE t1.b = 0 ORDER BY t1.a;
+SET max_parallel_workers_per_gather = 0;
+-- If there are lateral references to the other relation in sample scan, we'd
+-- fail to generate a partitionwise join.
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt1_l t1 JOIN LATERAL
+ (SELECT * FROM prt1_l t2 TABLESAMPLE SYSTEM (t1.a) REPEATABLE(t1.b)) s
+ ON t1.a = s.a AND t1.b = s.b AND t1.c = s.c;
+
+-- If there are lateral references to the other relation in scan's restriction
+-- clauses, we'd fail to generate a partitionwise join.
+EXPLAIN (COSTS OFF)
+SELECT COUNT(*) FROM prt1_l t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2_l t2) s
+ ON t1.a = s.b AND t1.b = s.a AND t1.c = s.c
+ WHERE s.t1b = s.a;
+SELECT COUNT(*) FROM prt1_l t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2_l t2) s
+ ON t1.a = s.b AND t1.b = s.a AND t1.c = s.c
+ WHERE s.t1b = s.a;
+RESET max_parallel_workers_per_gather;
+
-- join with one side empty
EXPLAIN (COSTS OFF)
SELECT t1.a, t1.c, t2.b, t2.c FROM (SELECT * FROM prt1_l WHERE a = 1 AND a = 2) t1 RIGHT JOIN prt2_l t2 ON t1.a = t2.b AND t1.b = t2.a AND t1.c = t2.c;
--
2.31.0
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Oversight in reparameterize_path_by_child leading to executor crash
@ 2024-02-01 17:45 Tom Lane <[email protected]>
parent: Richard Guo <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Tom Lane @ 2024-02-01 17:45 UTC (permalink / raw)
To: Richard Guo <[email protected]>; +Cc: Robert Haas <[email protected]>; Andrei Lepikhov <[email protected]>; Alena Rybakina <[email protected]>; Ashutosh Bapat <[email protected]>; pgsql-hackers
Richard Guo <[email protected]> writes:
> How about the attached v12 patch? But I'm a little concerned about
> omitting PVC_RECURSE_AGGREGATES and PVC_RECURSE_WINDOWFUNCS in this
> case. Is it possible that aggregates or window functions appear in the
> tablesample expression?
No, they can't appear there; it'd make no sense to allow such things
at the relation scan level, and we don't.
regression=# select * from float8_tbl tablesample system(count(*));
ERROR: aggregate functions are not allowed in functions in FROM
LINE 1: select * from float8_tbl tablesample system(count(*));
^
regression=# select * from float8_tbl tablesample system(count(*) over ());
ERROR: window functions are not allowed in functions in FROM
LINE 1: select * from float8_tbl tablesample system(count(*) over ()...
^
I pushed v12 (with some cosmetic adjustments) into the back branches,
since we're getting close to the February release freeze. We still
need to deal with the larger fix for HEAD. Please re-post that
one so that the cfbot knows which is the patch-of-record.
regards, tom lane
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Oversight in reparameterize_path_by_child leading to executor crash
@ 2024-02-02 02:10 Richard Guo <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Richard Guo @ 2024-02-02 02:10 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Robert Haas <[email protected]>; Andrei Lepikhov <[email protected]>; Alena Rybakina <[email protected]>; Ashutosh Bapat <[email protected]>; pgsql-hackers
On Fri, Feb 2, 2024 at 1:45 AM Tom Lane <[email protected]> wrote:
> I pushed v12 (with some cosmetic adjustments) into the back branches,
> since we're getting close to the February release freeze. We still
> need to deal with the larger fix for HEAD. Please re-post that
> one so that the cfbot knows which is the patch-of-record.
Thanks for the adjustments and pushing!
Here is the patch for HEAD. I simply re-posted v10. Nothing has
changed.
Thanks
Richard
Attachments:
[application/octet-stream] v10-0001-Postpone-reparameterization-of-paths-until-when-creating-plans.patch (30.5K, ../../CAMbWs4_=AGM2JSTymqmTc57oByaPij68T+ycRmde2cSBbi7+0w@mail.gmail.com/3-v10-0001-Postpone-reparameterization-of-paths-until-when-creating-plans.patch)
download | inline diff:
From c9a3a8df98ffb3320f094a7b1ad81c48b064877c Mon Sep 17 00:00:00 2001
From: Richard Guo <[email protected]>
Date: Mon, 8 Jan 2024 13:47:45 +0800
Subject: [PATCH v10] Postpone reparameterization of paths until when creating
plans
When creating a nestloop join path, if the inner path is parameterized,
it is parameterized by the topmost parent of the outer rel, not the
outer rel itself. Therefore, we need to translate the parameterization
so that the inner path is parameterized by the given outer rel itself.
Currently, this is done in reparameterize_path_by_child() during
generating join paths.
However, reparameterize_path_by_child() does not perform the translation
for sample scan paths' sampling infos, or scan paths' restriction
clauses. This omission can lead to executor crashes, wrong results, or
planning errors, as we have already observed.
Please note that the sampling infos are contained in RangeTblEntries (as
TableSampleClause objects), and the scan restriction clauses are
contained in RelOptInfos or IndexOptInfos (as lists of RestrictInfo
objects). We cannot just modify them on the fly during generating join
paths. Doing so would break things if we end up using a
non-partitionwise join.
So this commit postpones reparameterization of the inner path until
createplan.c, where it is safe to modify referenced RangeTblEntry,
RelOptInfo or IndexOptInfo, because we have made a final choice of which
Path to use. This can only work if we know that we will be able to
perform the translation for the inner path. So we introduce a new
function path_is_reparameterizable_by_child() during generating join
paths to check whether the given path can be translated.
As an additional benefit, this commit also helps avoid building the
reparameterized expressions we might not use, resulting in saved CPU
cycles and reduced memory usage.
---
src/backend/optimizer/path/joinpath.c | 67 +++--
src/backend/optimizer/plan/createplan.c | 17 ++
src/backend/optimizer/util/pathnode.c | 249 ++++++++++++++++---
src/include/optimizer/pathnode.h | 2 +
src/test/regress/expected/partition_join.out | 168 +++++++++++++
src/test/regress/sql/partition_join.sql | 40 +++
6 files changed, 473 insertions(+), 70 deletions(-)
diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c
index e9def9d540..aecf2774de 100644
--- a/src/backend/optimizer/path/joinpath.c
+++ b/src/backend/optimizer/path/joinpath.c
@@ -30,8 +30,9 @@
set_join_pathlist_hook_type set_join_pathlist_hook = NULL;
/*
- * Paths parameterized by the parent can be considered to be parameterized by
- * any of its child.
+ * Paths parameterized by a parent rel can be considered to be parameterized
+ * by any of its children, when we are performing partitionwise joins. These
+ * macros simplify checking for such cases. Beware multiple eval of args.
*/
#define PATH_PARAM_BY_PARENT(path, rel) \
((path)->param_info && bms_overlap(PATH_REQ_OUTER(path), \
@@ -768,6 +769,20 @@ try_nestloop_path(PlannerInfo *root,
/* If we got past that, we shouldn't have any unsafe outer-join refs */
Assert(!have_unsafe_outer_join_ref(root, outerrelids, inner_paramrels));
+ /*
+ * If the inner path is parameterized, it is parameterized by the topmost
+ * parent of the outer rel, not the outer rel itself. We will need to
+ * translate the parameterization, if this path is chosen, during
+ * create_plan(). Here we just check whether we will be able to perform
+ * the translation, and if not avoid creating a nestloop path.
+ */
+ if (PATH_PARAM_BY_PARENT(inner_path, outer_path->parent) &&
+ !path_is_reparameterizable_by_child(inner_path, outer_path->parent))
+ {
+ bms_free(required_outer);
+ return;
+ }
+
/*
* Do a precheck to quickly eliminate obviously-inferior paths. We
* calculate a cheap lower bound on the path's cost and then use
@@ -784,27 +799,6 @@ try_nestloop_path(PlannerInfo *root,
workspace.startup_cost, workspace.total_cost,
pathkeys, required_outer))
{
- /*
- * If the inner path is parameterized, it is parameterized by the
- * topmost parent of the outer rel, not the outer rel itself. Fix
- * that.
- */
- if (PATH_PARAM_BY_PARENT(inner_path, outer_path->parent))
- {
- inner_path = reparameterize_path_by_child(root, inner_path,
- outer_path->parent);
-
- /*
- * If we could not translate the path, we can't create nest loop
- * path.
- */
- if (!inner_path)
- {
- bms_free(required_outer);
- return;
- }
- }
-
add_path(joinrel, (Path *)
create_nestloop_path(root,
joinrel,
@@ -867,6 +861,17 @@ try_partial_nestloop_path(PlannerInfo *root,
return;
}
+ /*
+ * If the inner path is parameterized, it is parameterized by the topmost
+ * parent of the outer rel, not the outer rel itself. We will need to
+ * translate the parameterization, if this path is chosen, during
+ * create_plan(). Here we just check whether we will be able to perform
+ * the translation, and if not avoid creating a nestloop path.
+ */
+ if (PATH_PARAM_BY_PARENT(inner_path, outer_path->parent) &&
+ !path_is_reparameterizable_by_child(inner_path, outer_path->parent))
+ return;
+
/*
* Before creating a path, get a quick lower bound on what it is likely to
* cost. Bail out right away if it looks terrible.
@@ -876,22 +881,6 @@ try_partial_nestloop_path(PlannerInfo *root,
if (!add_partial_path_precheck(joinrel, workspace.total_cost, pathkeys))
return;
- /*
- * If the inner path is parameterized, it is parameterized by the topmost
- * parent of the outer rel, not the outer rel itself. Fix that.
- */
- if (PATH_PARAM_BY_PARENT(inner_path, outer_path->parent))
- {
- inner_path = reparameterize_path_by_child(root, inner_path,
- outer_path->parent);
-
- /*
- * If we could not translate the path, we can't create nest loop path.
- */
- if (!inner_path)
- return;
- }
-
/* Might be good enough to be worth trying, so let's try it. */
add_partial_path(joinrel, (Path *)
create_nestloop_path(root,
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index ca619eab94..415fd02806 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -29,6 +29,7 @@
#include "optimizer/cost.h"
#include "optimizer/optimizer.h"
#include "optimizer/paramassign.h"
+#include "optimizer/pathnode.h"
#include "optimizer/paths.h"
#include "optimizer/placeholder.h"
#include "optimizer/plancat.h"
@@ -4346,6 +4347,22 @@ create_nestloop_plan(PlannerInfo *root,
List *nestParams;
Relids saveOuterRels = root->curOuterRels;
+ /*
+ * If the inner path is parameterized by the topmost parent of the outer
+ * rel rather than the outer rel itself, fix that. (Nothing happens here
+ * if it is not so parameterized.)
+ */
+ best_path->jpath.innerjoinpath =
+ reparameterize_path_by_child(root,
+ best_path->jpath.innerjoinpath,
+ best_path->jpath.outerjoinpath->parent);
+
+ /*
+ * Failure here probably means that reparameterize_path_by_child() is not
+ * in sync with path_is_reparameterizable_by_child().
+ */
+ Assert(best_path->jpath.innerjoinpath != NULL);
+
/* NestLoop can project, so no need to be picky about child tlists */
outer_plan = create_plan_recurse(root, best_path->jpath.outerjoinpath, 0);
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index 8dbf790e89..c4447da560 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -56,6 +56,8 @@ static int append_startup_cost_compare(const ListCell *a, const ListCell *b);
static List *reparameterize_pathlist_by_child(PlannerInfo *root,
List *pathlist,
RelOptInfo *child_rel);
+static bool pathlist_is_reparameterizable_by_child(List *pathlist,
+ RelOptInfo *child_rel);
/*****************************************************************************
@@ -2458,6 +2460,16 @@ create_nestloop_path(PlannerInfo *root,
{
NestPath *pathnode = makeNode(NestPath);
Relids inner_req_outer = PATH_REQ_OUTER(inner_path);
+ Relids outerrelids;
+
+ /*
+ * Paths are parameterized by top-level parents, so run parameterization
+ * tests on the parent relids.
+ */
+ if (outer_path->parent->top_parent_relids)
+ outerrelids = outer_path->parent->top_parent_relids;
+ else
+ outerrelids = outer_path->parent->relids;
/*
* If the inner path is parameterized by the outer, we must drop any
@@ -2467,7 +2479,7 @@ create_nestloop_path(PlannerInfo *root,
* estimates for this path. We detect such clauses by checking for serial
* number match to clauses already enforced in the inner path.
*/
- if (bms_overlap(inner_req_outer, outer_path->parent->relids))
+ if (bms_overlap(inner_req_outer, outerrelids))
{
Bitmapset *enforced_serials = get_param_path_clause_serials(inner_path);
List *jclauses = NIL;
@@ -4067,32 +4079,33 @@ reparameterize_path(PlannerInfo *root, Path *path,
* Given a path parameterized by the parent of the given child relation,
* translate the path to be parameterized by the given child relation.
*
- * The function creates a new path of the same type as the given path, but
- * parameterized by the given child relation. Most fields from the original
- * path can simply be flat-copied, but any expressions must be adjusted to
- * refer to the correct varnos, and any paths must be recursively
- * reparameterized. Other fields that refer to specific relids also need
- * adjustment.
+ * The function translates the given path to be parameterized by the given
+ * child relation. Most fields from the original path are not changed, but any
+ * expressions must be adjusted to refer to the correct varnos, and any paths
+ * must be recursively reparameterized. Other fields that refer to specific
+ * relids also need adjustment.
*
* The cost, number of rows, width and parallel path properties depend upon
- * path->parent, which does not change during the translation. Hence those
- * members are copied as they are.
+ * path->parent, which does not change during the translation.
*
* Currently, only a few path types are supported here, though more could be
* added at need. We return NULL if we can't reparameterize the given path.
+ *
+ * Note that this function can change referenced RangeTblEntries, RelOptInfos
+ * and IndexOptInfos as well as the Path structures. Therefore, it's only safe
+ * to call during create_plan(), when we have made a final choice of which Path
+ * to use for each RangeTblEntry/RelOptInfo/IndexOptInfo.
+ *
+ * Keep this code in sync with path_is_reparameterizable_by_child()!
*/
Path *
reparameterize_path_by_child(PlannerInfo *root, Path *path,
RelOptInfo *child_rel)
{
-#define FLAT_COPY_PATH(newnode, node, nodetype) \
- ( (newnode) = makeNode(nodetype), \
- memcpy((newnode), (node), sizeof(nodetype)) )
-
#define ADJUST_CHILD_ATTRS(node) \
((node) = \
- (List *) adjust_appendrel_attrs_multilevel(root, (Node *) (node), \
+ (void *) adjust_appendrel_attrs_multilevel(root, (Node *) (node), \
child_rel, \
child_rel->top_parent))
@@ -4120,15 +4133,15 @@ do { \
Relids required_outer;
/*
- * If the path is not parameterized by parent of the given relation, it
- * doesn't need reparameterization.
+ * If the path is not parameterized by the parent of the given relation,
+ * it doesn't need reparameterization.
*/
if (!path->param_info ||
!bms_overlap(PATH_REQ_OUTER(path), child_rel->top_parent_relids))
return path;
/*
- * If possible, reparameterize the given path, making a copy.
+ * If possible, reparameterize the given path.
*
* This function is currently only applied to the inner side of a nestloop
* join that is being partitioned by the partitionwise-join code. Hence,
@@ -4142,14 +4155,29 @@ do { \
switch (nodeTag(path))
{
case T_Path:
- FLAT_COPY_PATH(new_path, path, Path);
+ new_path = path;
+ ADJUST_CHILD_ATTRS(new_path->parent->baserestrictinfo);
+ if (path->pathtype == T_SampleScan)
+ {
+ Index scan_relid = path->parent->relid;
+ RangeTblEntry *rte;
+
+ /* it should be a base rel with a tablesample clause... */
+ Assert(scan_relid > 0);
+ rte = planner_rt_fetch(scan_relid, root);
+ Assert(rte->rtekind == RTE_RELATION);
+ Assert(rte->tablesample != NULL);
+
+ ADJUST_CHILD_ATTRS(rte->tablesample);
+ }
break;
case T_IndexPath:
{
IndexPath *ipath;
- FLAT_COPY_PATH(ipath, path, IndexPath);
+ ipath = (IndexPath *) path;
+ ADJUST_CHILD_ATTRS(ipath->indexinfo->indrestrictinfo);
ADJUST_CHILD_ATTRS(ipath->indexclauses);
new_path = (Path *) ipath;
}
@@ -4159,7 +4187,8 @@ do { \
{
BitmapHeapPath *bhpath;
- FLAT_COPY_PATH(bhpath, path, BitmapHeapPath);
+ bhpath = (BitmapHeapPath *) path;
+ ADJUST_CHILD_ATTRS(bhpath->path.parent->baserestrictinfo);
REPARAMETERIZE_CHILD_PATH(bhpath->bitmapqual);
new_path = (Path *) bhpath;
}
@@ -4169,7 +4198,7 @@ do { \
{
BitmapAndPath *bapath;
- FLAT_COPY_PATH(bapath, path, BitmapAndPath);
+ bapath = (BitmapAndPath *) path;
REPARAMETERIZE_CHILD_PATH_LIST(bapath->bitmapquals);
new_path = (Path *) bapath;
}
@@ -4179,7 +4208,7 @@ do { \
{
BitmapOrPath *bopath;
- FLAT_COPY_PATH(bopath, path, BitmapOrPath);
+ bopath = (BitmapOrPath *) path;
REPARAMETERIZE_CHILD_PATH_LIST(bopath->bitmapquals);
new_path = (Path *) bopath;
}
@@ -4190,7 +4219,8 @@ do { \
ForeignPath *fpath;
ReparameterizeForeignPathByChild_function rfpc_func;
- FLAT_COPY_PATH(fpath, path, ForeignPath);
+ fpath = (ForeignPath *) path;
+ ADJUST_CHILD_ATTRS(fpath->path.parent->baserestrictinfo);
if (fpath->fdw_outerpath)
REPARAMETERIZE_CHILD_PATH(fpath->fdw_outerpath);
if (fpath->fdw_restrictinfo)
@@ -4210,7 +4240,8 @@ do { \
{
CustomPath *cpath;
- FLAT_COPY_PATH(cpath, path, CustomPath);
+ cpath = (CustomPath *) path;
+ ADJUST_CHILD_ATTRS(cpath->path.parent->baserestrictinfo);
REPARAMETERIZE_CHILD_PATH_LIST(cpath->custom_paths);
if (cpath->custom_restrictinfo)
ADJUST_CHILD_ATTRS(cpath->custom_restrictinfo);
@@ -4229,7 +4260,7 @@ do { \
JoinPath *jpath;
NestPath *npath;
- FLAT_COPY_PATH(npath, path, NestPath);
+ npath = (NestPath *) path;
jpath = (JoinPath *) npath;
REPARAMETERIZE_CHILD_PATH(jpath->outerjoinpath);
@@ -4244,7 +4275,7 @@ do { \
JoinPath *jpath;
MergePath *mpath;
- FLAT_COPY_PATH(mpath, path, MergePath);
+ mpath = (MergePath *) path;
jpath = (JoinPath *) mpath;
REPARAMETERIZE_CHILD_PATH(jpath->outerjoinpath);
@@ -4260,7 +4291,7 @@ do { \
JoinPath *jpath;
HashPath *hpath;
- FLAT_COPY_PATH(hpath, path, HashPath);
+ hpath = (HashPath *) path;
jpath = (JoinPath *) hpath;
REPARAMETERIZE_CHILD_PATH(jpath->outerjoinpath);
@@ -4275,7 +4306,7 @@ do { \
{
AppendPath *apath;
- FLAT_COPY_PATH(apath, path, AppendPath);
+ apath = (AppendPath *) path;
REPARAMETERIZE_CHILD_PATH_LIST(apath->subpaths);
new_path = (Path *) apath;
}
@@ -4285,7 +4316,7 @@ do { \
{
MaterialPath *mpath;
- FLAT_COPY_PATH(mpath, path, MaterialPath);
+ mpath = (MaterialPath *) path;
REPARAMETERIZE_CHILD_PATH(mpath->subpath);
new_path = (Path *) mpath;
}
@@ -4295,7 +4326,7 @@ do { \
{
MemoizePath *mpath;
- FLAT_COPY_PATH(mpath, path, MemoizePath);
+ mpath = (MemoizePath *) path;
REPARAMETERIZE_CHILD_PATH(mpath->subpath);
ADJUST_CHILD_ATTRS(mpath->param_exprs);
new_path = (Path *) mpath;
@@ -4306,7 +4337,7 @@ do { \
{
GatherPath *gpath;
- FLAT_COPY_PATH(gpath, path, GatherPath);
+ gpath = (GatherPath *) path;
REPARAMETERIZE_CHILD_PATH(gpath->subpath);
new_path = (Path *) gpath;
}
@@ -4373,9 +4404,145 @@ do { \
return new_path;
}
+/*
+ * path_is_reparameterizable_by_child
+ * Given a path parameterized by the parent of the given child relation,
+ * see if it can be translated to be parameterized by the child relation.
+ *
+ * This must return true if and only if reparameterize_path_by_child()
+ * would succeed on this path. Currently it's sufficient to verify that
+ * the path and all of its subpaths (if any) are of the types handled by
+ * that function. However, sub-paths that are not parameterized can be
+ * disregarded since they won't require translation.
+ */
+bool
+path_is_reparameterizable_by_child(Path *path, RelOptInfo *child_rel)
+{
+
+#define REJECT_IF_PATH_NOT_REPARAMETERIZABLE(path) \
+do { \
+ if (!path_is_reparameterizable_by_child(path, child_rel)) \
+ return false; \
+} while(0)
+
+#define REJECT_IF_PATH_LIST_NOT_REPARAMETERIZABLE(pathlist) \
+do { \
+ if (!pathlist_is_reparameterizable_by_child(pathlist, child_rel)) \
+ return false; \
+} while(0)
+
+ /*
+ * If the path is not parameterized by the parent of the given relation,
+ * it doesn't need reparameterization.
+ */
+ if (!path->param_info ||
+ !bms_overlap(PATH_REQ_OUTER(path), child_rel->top_parent_relids))
+ return true;
+
+ switch (nodeTag(path))
+ {
+ case T_Path:
+ case T_IndexPath:
+ break;
+
+ case T_BitmapHeapPath:
+ {
+ BitmapHeapPath *bhpath = (BitmapHeapPath *) path;
+
+ REJECT_IF_PATH_NOT_REPARAMETERIZABLE(bhpath->bitmapqual);
+ }
+ break;
+
+ case T_BitmapAndPath:
+ {
+ BitmapAndPath *bapath = (BitmapAndPath *) path;
+
+ REJECT_IF_PATH_LIST_NOT_REPARAMETERIZABLE(bapath->bitmapquals);
+ }
+ break;
+
+ case T_BitmapOrPath:
+ {
+ BitmapOrPath *bopath = (BitmapOrPath *) path;
+
+ REJECT_IF_PATH_LIST_NOT_REPARAMETERIZABLE(bopath->bitmapquals);
+ }
+ break;
+
+ case T_ForeignPath:
+ {
+ ForeignPath *fpath = (ForeignPath *) path;
+
+ if (fpath->fdw_outerpath)
+ REJECT_IF_PATH_NOT_REPARAMETERIZABLE(fpath->fdw_outerpath);
+ }
+ break;
+
+ case T_CustomPath:
+ {
+ CustomPath *cpath = (CustomPath *) path;
+
+ REJECT_IF_PATH_LIST_NOT_REPARAMETERIZABLE(cpath->custom_paths);
+ }
+ break;
+
+ case T_NestPath:
+ case T_MergePath:
+ case T_HashPath:
+ {
+ JoinPath *jpath = (JoinPath *) path;
+
+ REJECT_IF_PATH_NOT_REPARAMETERIZABLE(jpath->outerjoinpath);
+ REJECT_IF_PATH_NOT_REPARAMETERIZABLE(jpath->innerjoinpath);
+ }
+ break;
+
+ case T_AppendPath:
+ {
+ AppendPath *apath = (AppendPath *) path;
+
+ REJECT_IF_PATH_LIST_NOT_REPARAMETERIZABLE(apath->subpaths);
+ }
+ break;
+
+ case T_MaterialPath:
+ {
+ MaterialPath *mpath = (MaterialPath *) path;
+
+ REJECT_IF_PATH_NOT_REPARAMETERIZABLE(mpath->subpath);
+ }
+ break;
+
+ case T_MemoizePath:
+ {
+ MemoizePath *mpath = (MemoizePath *) path;
+
+ REJECT_IF_PATH_NOT_REPARAMETERIZABLE(mpath->subpath);
+ }
+ break;
+
+ case T_GatherPath:
+ {
+ GatherPath *gpath = (GatherPath *) path;
+
+ REJECT_IF_PATH_NOT_REPARAMETERIZABLE(gpath->subpath);
+ }
+ break;
+
+ default:
+
+ /* We don't know how to reparameterize this path. */
+ return false;
+ }
+
+ return true;
+}
+
/*
* reparameterize_pathlist_by_child
* Helper function to reparameterize a list of paths by given child rel.
+ *
+ * Returns NIL to indicate failure, so pathlist had better not be NIL.
*/
static List *
reparameterize_pathlist_by_child(PlannerInfo *root,
@@ -4401,3 +4568,23 @@ reparameterize_pathlist_by_child(PlannerInfo *root,
return result;
}
+
+/*
+ * pathlist_is_reparameterizable_by_child
+ * Helper function to check if a list of paths can be reparameterized.
+ */
+static bool
+pathlist_is_reparameterizable_by_child(List *pathlist, RelOptInfo *child_rel)
+{
+ ListCell *lc;
+
+ foreach(lc, pathlist)
+ {
+ Path *path = (Path *) lfirst(lc);
+
+ if (!path_is_reparameterizable_by_child(path, child_rel))
+ return false;
+ }
+
+ return true;
+}
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index c43d97b48a..99c2f955aa 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -298,6 +298,8 @@ extern Path *reparameterize_path(PlannerInfo *root, Path *path,
double loop_count);
extern Path *reparameterize_path_by_child(PlannerInfo *root, Path *path,
RelOptInfo *child_rel);
+extern bool path_is_reparameterizable_by_child(Path *path,
+ RelOptInfo *child_rel);
/*
* prototypes for relnode.c
diff --git a/src/test/regress/expected/partition_join.out b/src/test/regress/expected/partition_join.out
index 6560fe2416..6d07f86b9b 100644
--- a/src/test/regress/expected/partition_join.out
+++ b/src/test/regress/expected/partition_join.out
@@ -505,6 +505,98 @@ SELECT t1.a, ss.t2a, ss.t2c FROM prt1 t1 LEFT JOIN LATERAL
550 | |
(12 rows)
+-- lateral reference in sample scan
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt1 t1 JOIN LATERAL
+ (SELECT * FROM prt1 t2 TABLESAMPLE SYSTEM (t1.a) REPEATABLE(t1.b)) s
+ ON t1.a = s.a;
+ QUERY PLAN
+-------------------------------------------------------------
+ Append
+ -> Nested Loop
+ -> Seq Scan on prt1_p1 t1_1
+ -> Sample Scan on prt1_p1 t2_1
+ Sampling: system (t1_1.a) REPEATABLE (t1_1.b)
+ Filter: (t1_1.a = a)
+ -> Nested Loop
+ -> Seq Scan on prt1_p2 t1_2
+ -> Sample Scan on prt1_p2 t2_2
+ Sampling: system (t1_2.a) REPEATABLE (t1_2.b)
+ Filter: (t1_2.a = a)
+ -> Nested Loop
+ -> Seq Scan on prt1_p3 t1_3
+ -> Sample Scan on prt1_p3 t2_3
+ Sampling: system (t1_3.a) REPEATABLE (t1_3.b)
+ Filter: (t1_3.a = a)
+(16 rows)
+
+-- lateral reference in scan's restriction clauses
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.a;
+ QUERY PLAN
+---------------------------------------------------------------
+ Aggregate
+ -> Append
+ -> Nested Loop
+ -> Seq Scan on prt1_p1 t1_1
+ -> Index Scan using iprt2_p1_b on prt2_p1 t2_1
+ Index Cond: (b = t1_1.a)
+ Filter: (t1_1.b = a)
+ -> Nested Loop
+ -> Seq Scan on prt1_p2 t1_2
+ -> Index Scan using iprt2_p2_b on prt2_p2 t2_2
+ Index Cond: (b = t1_2.a)
+ Filter: (t1_2.b = a)
+ -> Nested Loop
+ -> Seq Scan on prt1_p3 t1_3
+ -> Index Scan using iprt2_p3_b on prt2_p3 t2_3
+ Index Cond: (b = t1_3.a)
+ Filter: (t1_3.b = a)
+(17 rows)
+
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.a;
+ count
+-------
+ 100
+(1 row)
+
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.b;
+ QUERY PLAN
+--------------------------------------------------------------------
+ Aggregate
+ -> Append
+ -> Nested Loop
+ -> Seq Scan on prt1_p1 t1_1
+ -> Index Only Scan using iprt2_p1_b on prt2_p1 t2_1
+ Index Cond: (b = t1_1.a)
+ Filter: (b = t1_1.b)
+ -> Nested Loop
+ -> Seq Scan on prt1_p2 t1_2
+ -> Index Only Scan using iprt2_p2_b on prt2_p2 t2_2
+ Index Cond: (b = t1_2.a)
+ Filter: (b = t1_2.b)
+ -> Nested Loop
+ -> Seq Scan on prt1_p3 t1_3
+ -> Index Only Scan using iprt2_p3_b on prt2_p3 t2_3
+ Index Cond: (b = t1_3.a)
+ Filter: (b = t1_3.b)
+(17 rows)
+
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.b;
+ count
+-------
+ 5
+(1 row)
+
-- bug with inadequate sort key representation
SET enable_partitionwise_aggregate TO true;
SET enable_hashjoin TO false;
@@ -1944,6 +2036,82 @@ SELECT * FROM prt1_l t1 LEFT JOIN LATERAL
550 | 0 | 0002 | | | | |
(12 rows)
+-- partitionwise join with lateral reference in sample scan
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt1_l t1 JOIN LATERAL
+ (SELECT * FROM prt1_l t2 TABLESAMPLE SYSTEM (t1.a) REPEATABLE(t1.b)) s
+ ON t1.a = s.a AND t1.b = s.b AND t1.c = s.c;
+ QUERY PLAN
+----------------------------------------------------------------------------------------
+ Append
+ -> Nested Loop
+ -> Seq Scan on prt1_l_p1 t1_1
+ -> Sample Scan on prt1_l_p1 t2_1
+ Sampling: system (t1_1.a) REPEATABLE (t1_1.b)
+ Filter: ((t1_1.a = a) AND (t1_1.b = b) AND ((t1_1.c)::text = (c)::text))
+ -> Nested Loop
+ -> Seq Scan on prt1_l_p2_p1 t1_2
+ -> Sample Scan on prt1_l_p2_p1 t2_2
+ Sampling: system (t1_2.a) REPEATABLE (t1_2.b)
+ Filter: ((t1_2.a = a) AND (t1_2.b = b) AND ((t1_2.c)::text = (c)::text))
+ -> Nested Loop
+ -> Seq Scan on prt1_l_p2_p2 t1_3
+ -> Sample Scan on prt1_l_p2_p2 t2_3
+ Sampling: system (t1_3.a) REPEATABLE (t1_3.b)
+ Filter: ((t1_3.a = a) AND (t1_3.b = b) AND ((t1_3.c)::text = (c)::text))
+ -> Nested Loop
+ -> Seq Scan on prt1_l_p3_p1 t1_4
+ -> Sample Scan on prt1_l_p3_p1 t2_4
+ Sampling: system (t1_4.a) REPEATABLE (t1_4.b)
+ Filter: ((t1_4.a = a) AND (t1_4.b = b) AND ((t1_4.c)::text = (c)::text))
+ -> Nested Loop
+ -> Seq Scan on prt1_l_p3_p2 t1_5
+ -> Sample Scan on prt1_l_p3_p2 t2_5
+ Sampling: system (t1_5.a) REPEATABLE (t1_5.b)
+ Filter: ((t1_5.a = a) AND (t1_5.b = b) AND ((t1_5.c)::text = (c)::text))
+(26 rows)
+
+-- partitionwise join with lateral reference in scan's restriction clauses
+EXPLAIN (COSTS OFF)
+SELECT COUNT(*) FROM prt1_l t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2_l t2) s
+ ON t1.a = s.b AND t1.b = s.a AND t1.c = s.c
+ WHERE s.t1b = s.a;
+ QUERY PLAN
+---------------------------------------------------------------------------------------------------------------
+ Aggregate
+ -> Append
+ -> Nested Loop
+ -> Seq Scan on prt1_l_p1 t1_1
+ -> Seq Scan on prt2_l_p1 t2_1
+ Filter: ((a = t1_1.b) AND (t1_1.a = b) AND (t1_1.b = a) AND ((t1_1.c)::text = (c)::text))
+ -> Nested Loop
+ -> Seq Scan on prt1_l_p2_p1 t1_2
+ -> Seq Scan on prt2_l_p2_p1 t2_2
+ Filter: ((a = t1_2.b) AND (t1_2.a = b) AND (t1_2.b = a) AND ((t1_2.c)::text = (c)::text))
+ -> Nested Loop
+ -> Seq Scan on prt1_l_p2_p2 t1_3
+ -> Seq Scan on prt2_l_p2_p2 t2_3
+ Filter: ((a = t1_3.b) AND (t1_3.a = b) AND (t1_3.b = a) AND ((t1_3.c)::text = (c)::text))
+ -> Nested Loop
+ -> Seq Scan on prt1_l_p3_p1 t1_4
+ -> Seq Scan on prt2_l_p3_p1 t2_4
+ Filter: ((a = t1_4.b) AND (t1_4.a = b) AND (t1_4.b = a) AND ((t1_4.c)::text = (c)::text))
+ -> Nested Loop
+ -> Seq Scan on prt1_l_p3_p2 t1_5
+ -> Seq Scan on prt2_l_p3_p2 t2_5
+ Filter: ((a = t1_5.b) AND (t1_5.a = b) AND (t1_5.b = a) AND ((t1_5.c)::text = (c)::text))
+(22 rows)
+
+SELECT COUNT(*) FROM prt1_l t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2_l t2) s
+ ON t1.a = s.b AND t1.b = s.a AND t1.c = s.c
+ WHERE s.t1b = s.a;
+ count
+-------
+ 100
+(1 row)
+
-- join with one side empty
EXPLAIN (COSTS OFF)
SELECT t1.a, t1.c, t2.b, t2.c FROM (SELECT * FROM prt1_l WHERE a = 1 AND a = 2) t1 RIGHT JOIN prt2_l t2 ON t1.a = t2.b AND t1.b = t2.a AND t1.c = t2.c;
diff --git a/src/test/regress/sql/partition_join.sql b/src/test/regress/sql/partition_join.sql
index 48daf3aee3..128ce8376e 100644
--- a/src/test/regress/sql/partition_join.sql
+++ b/src/test/regress/sql/partition_join.sql
@@ -100,6 +100,29 @@ SELECT t1.a, ss.t2a, ss.t2c FROM prt1 t1 LEFT JOIN LATERAL
(SELECT t2.a AS t2a, t3.a AS t3a, t2.b t2b, t2.c t2c, least(t1.a,t2.a,t3.a) FROM prt1 t2 JOIN prt2 t3 ON (t2.a = t3.b)) ss
ON t1.c = ss.t2c WHERE (t1.b + coalesce(ss.t2b, 0)) = 0 ORDER BY t1.a;
+-- lateral reference in sample scan
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt1 t1 JOIN LATERAL
+ (SELECT * FROM prt1 t2 TABLESAMPLE SYSTEM (t1.a) REPEATABLE(t1.b)) s
+ ON t1.a = s.a;
+
+-- lateral reference in scan's restriction clauses
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.a;
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.a;
+
+EXPLAIN (COSTS OFF)
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.b;
+SELECT count(*) FROM prt1 t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2 t2) s
+ ON t1.a = s.b WHERE s.t1b = s.b;
+
-- bug with inadequate sort key representation
SET enable_partitionwise_aggregate TO true;
SET enable_hashjoin TO false;
@@ -387,6 +410,23 @@ SELECT * FROM prt1_l t1 LEFT JOIN LATERAL
(SELECT t2.a AS t2a, t2.c AS t2c, t2.b AS t2b, t3.b AS t3b, least(t1.a,t2.a,t3.b) FROM prt1_l t2 JOIN prt2_l t3 ON (t2.a = t3.b AND t2.c = t3.c)) ss
ON t1.a = ss.t2a AND t1.c = ss.t2c WHERE t1.b = 0 ORDER BY t1.a;
+-- partitionwise join with lateral reference in sample scan
+EXPLAIN (COSTS OFF)
+SELECT * FROM prt1_l t1 JOIN LATERAL
+ (SELECT * FROM prt1_l t2 TABLESAMPLE SYSTEM (t1.a) REPEATABLE(t1.b)) s
+ ON t1.a = s.a AND t1.b = s.b AND t1.c = s.c;
+
+-- partitionwise join with lateral reference in scan's restriction clauses
+EXPLAIN (COSTS OFF)
+SELECT COUNT(*) FROM prt1_l t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2_l t2) s
+ ON t1.a = s.b AND t1.b = s.a AND t1.c = s.c
+ WHERE s.t1b = s.a;
+SELECT COUNT(*) FROM prt1_l t1 LEFT JOIN LATERAL
+ (SELECT t1.b AS t1b, t2.* FROM prt2_l t2) s
+ ON t1.a = s.b AND t1.b = s.a AND t1.c = s.c
+ WHERE s.t1b = s.a;
+
-- join with one side empty
EXPLAIN (COSTS OFF)
SELECT t1.a, t1.c, t2.b, t2.c FROM (SELECT * FROM prt1_l WHERE a = 1 AND a = 2) t1 RIGHT JOIN prt2_l t2 ON t1.a = t2.b AND t1.b = t2.a AND t1.c = t2.c;
--
2.31.0
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Oversight in reparameterize_path_by_child leading to executor crash
@ 2024-03-19 18:57 Tom Lane <[email protected]>
parent: Richard Guo <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Tom Lane @ 2024-03-19 18:57 UTC (permalink / raw)
To: Richard Guo <[email protected]>; +Cc: Robert Haas <[email protected]>; Andrei Lepikhov <[email protected]>; Alena Rybakina <[email protected]>; Ashutosh Bapat <[email protected]>; pgsql-hackers
Richard Guo <[email protected]> writes:
> Here is the patch for HEAD. I simply re-posted v10. Nothing has
> changed.
I got back to this finally, and pushed it with some minor cosmetic
adjustments.
regards, tom lane
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Oversight in reparameterize_path_by_child leading to executor crash
@ 2024-03-21 11:20 Richard Guo <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Richard Guo @ 2024-03-21 11:20 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Robert Haas <[email protected]>; Andrei Lepikhov <[email protected]>; Alena Rybakina <[email protected]>; Ashutosh Bapat <[email protected]>; pgsql-hackers
On Wed, Mar 20, 2024 at 2:57 AM Tom Lane <[email protected]> wrote:
> Richard Guo <[email protected]> writes:
> > Here is the patch for HEAD. I simply re-posted v10. Nothing has
> > changed.
>
> I got back to this finally, and pushed it with some minor cosmetic
> adjustments.
Thanks for pushing!
Thanks
Richard
^ permalink raw reply [nested|flat] 86+ messages in thread
end of thread, other threads:[~2024-03-21 11:20 UTC | newest]
Thread overview: 86+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2019-06-06 04:43 Re: Should we warn against using too many partitions? David Rowley <[email protected]>
2019-06-06 05:29 ` Justin Pryzby <[email protected]>
2019-06-06 18:46 ` David Rowley <[email protected]>
2019-06-06 18:54 ` Justin Pryzby <[email protected]>
2019-06-06 19:36 ` David Rowley <[email protected]>
2019-06-06 05:46 ` Amit Langote <[email protected]>
2019-06-06 15:12 ` Alvaro Herrera <[email protected]>
2019-06-06 18:59 ` David Rowley <[email protected]>
2019-06-07 05:34 ` David Rowley <[email protected]>
2019-06-07 06:59 ` Amit Langote <[email protected]>
2019-06-08 20:29 ` David Rowley <[email protected]>
2019-06-10 08:11 ` Amit Langote <[email protected]>
2019-06-10 09:10 ` David Rowley <[email protected]>
2019-06-10 13:44 ` Alvaro Herrera <[email protected]>
2019-06-10 21:45 ` David Rowley <[email protected]>
2019-06-08 06:38 ` Justin Pryzby <[email protected]>
2019-06-09 01:15 ` David Rowley <[email protected]>
2019-06-09 04:21 ` Justin Pryzby <[email protected]>
2019-06-09 05:07 ` David Rowley <[email protected]>
2019-06-09 05:11 ` Justin Pryzby <[email protected]>
2019-06-09 05:44 ` David Rowley <[email protected]>
2019-06-10 22:11 ` Alvaro Herrera <[email protected]>
2019-06-10 23:15 ` Justin Pryzby <[email protected]>
2019-06-11 01:30 ` David Rowley <[email protected]>
2019-06-11 02:43 ` Alvaro Herrera <[email protected]>
2019-06-11 02:52 ` Amit Langote <[email protected]>
2019-06-11 20:12 ` David Rowley <[email protected]>
2019-06-12 05:48 ` Amit Langote <[email protected]>
2019-06-12 22:36 ` David Rowley <[email protected]>
2021-01-31 00:10 [PATCH 11/18] duplicate words Justin Pryzby <[email protected]>
2021-01-31 00:10 [PATCH 11/18] duplicate words Justin Pryzby <[email protected]>
2021-01-31 00:10 [PATCH 11/18] duplicate words Justin Pryzby <[email protected]>
2021-01-31 00:10 [PATCH 11/18] duplicate words Justin Pryzby <[email protected]>
2021-01-31 00:10 [PATCH 11/18] duplicate words Justin Pryzby <[email protected]>
2021-01-31 00:10 [PATCH 11/18] duplicate words Justin Pryzby <[email protected]>
2021-01-31 00:10 [PATCH 11/18] duplicate words Justin Pryzby <[email protected]>
2021-01-31 00:10 [PATCH 11/18] duplicate words Justin Pryzby <[email protected]>
2021-01-31 00:10 [PATCH 11/18] duplicate words Justin Pryzby <[email protected]>
2021-01-31 00:10 [PATCH 11/18] duplicate words Justin Pryzby <[email protected]>
2021-01-31 00:10 [PATCH 11/18] duplicate words Justin Pryzby <[email protected]>
2021-01-31 00:10 [PATCH 11/18] duplicate words Justin Pryzby <[email protected]>
2021-01-31 00:10 [PATCH 11/18] duplicate words Justin Pryzby <[email protected]>
2021-01-31 00:10 [PATCH 11/21] duplicate words Justin Pryzby <[email protected]>
2021-01-31 00:10 [PATCH 11/18] duplicate words Justin Pryzby <[email protected]>
2021-01-31 00:10 [PATCH 11/18] duplicate words Justin Pryzby <[email protected]>
2021-01-31 00:10 [PATCH 11/18] duplicate words Justin Pryzby <[email protected]>
2021-01-31 00:10 [PATCH 11/18] duplicate words Justin Pryzby <[email protected]>
2021-01-31 00:10 [PATCH 11/18] duplicate words Justin Pryzby <[email protected]>
2021-01-31 00:10 [PATCH 11/18] duplicate words Justin Pryzby <[email protected]>
2021-01-31 00:10 [PATCH 11/18] duplicate words Justin Pryzby <[email protected]>
2021-01-31 00:10 [PATCH 11/18] duplicate words Justin Pryzby <[email protected]>
2021-01-31 00:10 [PATCH 11/18] duplicate words Justin Pryzby <[email protected]>
2021-04-04 18:36 [PATCH 32/32] duplicate words Justin Pryzby <[email protected]>
2022-04-06 12:24 [PATCH 14/19] duplicate words Justin Pryzby <[email protected]>
2023-01-18 04:28 [PATCH 04/10] duplicate words Justin Pryzby <[email protected]>
2023-08-23 05:37 ` Re: Oversight in reparameterize_path_by_child leading to executor crash Richard Guo <[email protected]>
2023-08-23 16:15 ` Re: Oversight in reparameterize_path_by_child leading to executor crash Ashutosh Bapat <[email protected]>
2023-09-20 07:18 ` Re: Oversight in reparameterize_path_by_child leading to executor crash Andrey Lepikhov <[email protected]>
2023-10-13 10:18 ` Re: Oversight in reparameterize_path_by_child leading to executor crash Andrei Lepikhov <[email protected]>
2023-10-18 06:39 ` Re: Oversight in reparameterize_path_by_child leading to executor crash Richard Guo <[email protected]>
2023-10-18 09:41 ` Re: Oversight in reparameterize_path_by_child leading to executor crash Andrei Lepikhov <[email protected]>
2023-10-19 18:52 ` Re: Oversight in reparameterize_path_by_child leading to executor crash Alena Rybakina <[email protected]>
2023-12-06 06:51 ` Re: Oversight in reparameterize_path_by_child leading to executor crash Richard Guo <[email protected]>
2023-12-06 07:30 ` Re: Oversight in reparameterize_path_by_child leading to executor crash Richard Guo <[email protected]>
2023-12-08 09:38 ` Re: Oversight in reparameterize_path_by_child leading to executor crash Alena Rybakina <[email protected]>
2023-12-11 03:02 ` Re: Oversight in reparameterize_path_by_child leading to executor crash Richard Guo <[email protected]>
2023-12-13 02:55 ` Re: Oversight in reparameterize_path_by_child leading to executor crash Andrei Lepikhov <[email protected]>
2024-01-05 18:36 ` Re: Oversight in reparameterize_path_by_child leading to executor crash Robert Haas <[email protected]>
2024-01-05 18:55 ` Re: Oversight in reparameterize_path_by_child leading to executor crash Tom Lane <[email protected]>
2024-01-08 08:32 ` Re: Oversight in reparameterize_path_by_child leading to executor crash Richard Guo <[email protected]>
2024-01-15 18:30 ` Re: Oversight in reparameterize_path_by_child leading to executor crash Robert Haas <[email protected]>
2024-01-15 19:17 ` Re: Oversight in reparameterize_path_by_child leading to executor crash Tom Lane <[email protected]>
2024-01-17 09:01 ` Re: Oversight in reparameterize_path_by_child leading to executor crash Richard Guo <[email protected]>
2024-01-17 09:50 ` Re: Oversight in reparameterize_path_by_child leading to executor crash Richard Guo <[email protected]>
2024-01-30 21:12 ` Re: Oversight in reparameterize_path_by_child leading to executor crash Tom Lane <[email protected]>
2024-01-31 06:39 ` Re: Oversight in reparameterize_path_by_child leading to executor crash Richard Guo <[email protected]>
2024-01-31 15:21 ` Re: Oversight in reparameterize_path_by_child leading to executor crash Tom Lane <[email protected]>
2024-02-01 02:54 ` Re: Oversight in reparameterize_path_by_child leading to executor crash Richard Guo <[email protected]>
2024-02-01 17:45 ` Re: Oversight in reparameterize_path_by_child leading to executor crash Tom Lane <[email protected]>
2024-02-02 02:10 ` Re: Oversight in reparameterize_path_by_child leading to executor crash Richard Guo <[email protected]>
2024-03-19 18:57 ` Re: Oversight in reparameterize_path_by_child leading to executor crash Tom Lane <[email protected]>
2024-03-21 11:20 ` Re: Oversight in reparameterize_path_by_child leading to executor crash Richard Guo <[email protected]>
2023-08-23 17:44 ` Re: Oversight in reparameterize_path_by_child leading to executor crash Tom Lane <[email protected]>
2023-08-24 02:47 ` Re: Oversight in reparameterize_path_by_child leading to executor crash Richard Guo <[email protected]>
2023-09-08 07:04 ` Re: Oversight in reparameterize_path_by_child leading to executor crash Ashutosh Bapat <[email protected]>
2023-09-11 02:05 ` Re: Oversight in reparameterize_path_by_child leading to executor crash Richard Guo <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox