pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedPerformance issue after creating partitions
8+ messages / 5 participants
[nested] [flat]
* Performance issue after creating partitions
@ 2022-08-24 22:41 Teja Jakkidi <teja.jakkidi05@gmail.com>
2022-08-25 00:52 ` Re: Performance issue after creating partitions Doug Reynolds <mav@wastegate.net>
2022-08-25 02:55 ` Re: Performance issue after creating partitions Mladen Gogala <gogala.mladen@gmail.com>
2022-08-31 06:01 ` Re: Performance issue after creating partitions Olivier Gautherot <ogautherot@gautherot.net>
0 siblings, 3 replies; 8+ messages in thread
From: Teja Jakkidi @ 2022-08-24 22:41 UTC (permalink / raw)
To: pgsql-admin <pgsql-admin@lists.postgresql.org>
Hello Admin team,
We have a Postgres 13 server set up on Google Cloud. Recently we created partitions as below for the tables:
Range on ‘order_date’ column
List on ‘country_code’ column
Quarterly partitions for range.
Order date column is a time stamp column and when we are using this column as a filter in where condition : trunc(order_date) = to_date(’2022-04-01’,’YYYY-MM-DD’), the query scans all the partitions dated back from the year 2000. So, instead of going to the 2022 Q2 partition directly, the query is scanning all the partitions.
But when we remove the trunc() function and just specify order_date =‘2022-04-01 07:02:30’, this works as expected. It goes to the correct partition directly and gets the data.
Can Anyone help me on what’s happening in the first case and why all the partitions are being scanned?
For the first case, when there is a non partition table, it is behaving way better.
Thanks,
J. Teja.
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Performance issue after creating partitions
2022-08-24 22:41 Performance issue after creating partitions Teja Jakkidi <teja.jakkidi05@gmail.com>
@ 2022-08-25 00:52 ` Doug Reynolds <mav@wastegate.net>
2022-08-26 18:46 ` Re: Performance issue after creating partitions Teja Jakkidi <teja.jakkidi05@gmail.com>
2 siblings, 1 reply; 8+ messages in thread
From: Doug Reynolds @ 2022-08-25 00:52 UTC (permalink / raw)
To: Teja Jakkidi <teja.jakkidi05@gmail.com>; +Cc: pgsql-admin <pgsql-admin@lists.postgresql.org>
You need to change your query to not use the trunc. The TRUNC statement forces it to trunc every row in the table.
If you do order_date >= to_date(..) and order_date < to_date(..) + interval '1' day, it will use the partition range to narrow down to the correct partition first.
Sent from my iPhone
> On Aug 24, 2022, at 6:41 PM, Teja Jakkidi <teja.jakkidi05@gmail.com> wrote:
>
> Hello Admin team,
>
> We have a Postgres 13 server set up on Google Cloud. Recently we created partitions as below for the tables:
> Range on ‘order_date’ column
> List on ‘country_code’ column
>
> Quarterly partitions for range.
>
> Order date column is a time stamp column and when we are using this column as a filter in where condition : trunc(order_date) = to_date(’2022-04-01’,’YYYY-MM-DD’), the query scans all the partitions dated back from the year 2000. So, instead of going to the 2022 Q2 partition directly, the query is scanning all the partitions.
> But when we remove the trunc() function and just specify order_date =‘2022-04-01 07:02:30’, this works as expected. It goes to the correct partition directly and gets the data.
> Can Anyone help me on what’s happening in the first case and why all the partitions are being scanned?
> For the first case, when there is a non partition table, it is behaving way better.
>
> Thanks,
> J. Teja.
>
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Performance issue after creating partitions
2022-08-24 22:41 Performance issue after creating partitions Teja Jakkidi <teja.jakkidi05@gmail.com>
2022-08-25 00:52 ` Re: Performance issue after creating partitions Doug Reynolds <mav@wastegate.net>
@ 2022-08-26 18:46 ` Teja Jakkidi <teja.jakkidi05@gmail.com>
2022-08-30 21:21 ` Re: Performance issue after creating partitions Teja Jakkidi <teja.jakkidi05@gmail.com>
0 siblings, 1 reply; 8+ messages in thread
From: Teja Jakkidi @ 2022-08-26 18:46 UTC (permalink / raw)
To: Doug Reynolds <mav@wastegate.net>; gogala.mladen@gmail.com; +Cc: pgsql-admin <pgsql-admin@lists.postgresql.org>
Thank you, Doug and Malden for your inputs.
As suggested by Doug I tried replacing the trunc() with below command:
“ order_date >= to_date(..) and order_date < to_date(..) + interval '1' day”
What happens here is: My query has 2tables : A and B, both are having an inner join on company code and order date. We have A.Company_code, A.order_date filter in the where condition which was having trun() previously but now I replaced it with the command suggested by Doug. The company_code filter is working for both tables A and B, the query is searching only the partitions particular to that country code. However for order_date condition, the partitions for A table are being utilized correctly I.e the query is searching directly in only one partition based on that date. But for table B, it is still searching all the partitions, it is not applying the where A.order_date condition for B tables partitions while searching.
I do not understand why the A.order_date filter is not being applied to table B where as A.company_code is being applied to both tables.
Thanks and Regards,
J. Teja Sri.
> On Aug 24, 2022, at 5:52 PM, Doug Reynolds <mav@wastegate.net> wrote:
>
> You need to change your query to not use the trunc. The TRUNC statement forces it to trunc every row in the table.
>
> If you do order_date >= to_date(..) and order_date < to_date(..) + interval '1' day, it will use the partition range to narrow down to the correct partition first.
>
> Sent from my iPhone
>
>> On Aug 24, 2022, at 6:41 PM, Teja Jakkidi <teja.jakkidi05@gmail.com> wrote:
>>
>> Hello Admin team,
>>
>> We have a Postgres 13 server set up on Google Cloud. Recently we created partitions as below for the tables:
>> Range on ‘order_date’ column
>> List on ‘country_code’ column
>>
>> Quarterly partitions for range.
>>
>> Order date column is a time stamp column and when we are using this column as a filter in where condition : trunc(order_date) = to_date(’2022-04-01’,’YYYY-MM-DD’), the query scans all the partitions dated back from the year 2000. So, instead of going to the 2022 Q2 partition directly, the query is scanning all the partitions.
>> But when we remove the trunc() function and just specify order_date =‘2022-04-01 07:02:30’, this works as expected. It goes to the correct partition directly and gets the data.
>> Can Anyone help me on what’s happening in the first case and why all the partitions are being scanned?
>> For the first case, when there is a non partition table, it is behaving way better.
>>
>> Thanks,
>> J. Teja.
>>
>
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Performance issue after creating partitions
2022-08-24 22:41 Performance issue after creating partitions Teja Jakkidi <teja.jakkidi05@gmail.com>
2022-08-25 00:52 ` Re: Performance issue after creating partitions Doug Reynolds <mav@wastegate.net>
2022-08-26 18:46 ` Re: Performance issue after creating partitions Teja Jakkidi <teja.jakkidi05@gmail.com>
@ 2022-08-30 21:21 ` Teja Jakkidi <teja.jakkidi05@gmail.com>
2022-08-30 21:40 ` Re: Performance issue after creating partitions Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 8+ messages in thread
From: Teja Jakkidi @ 2022-08-30 21:21 UTC (permalink / raw)
To: Doug Reynolds <mav@wastegate.net>; gogala.mladen@gmail.com; +Cc: pgsql-admin <pgsql-admin@lists.postgresql.org>
Anyone ever encountered the below situation as what I am noticing with partitions?
Thanks,
Teja. J.
> On Aug 26, 2022, at 11:46 AM, Teja Jakkidi <teja.jakkidi05@gmail.com> wrote:
>
> Thank you, Doug and Malden for your inputs.
>
> As suggested by Doug I tried replacing the trunc() with below command:
> “ order_date >= to_date(..) and order_date < to_date(..) + interval '1' day”
>
> What happens here is: My query has 2tables : A and B, both are having an inner join on company code and order date. We have A.Company_code, A.order_date filter in the where condition which was having trun() previously but now I replaced it with the command suggested by Doug. The company_code filter is working for both tables A and B, the query is searching only the partitions particular to that country code. However for order_date condition, the partitions for A table are being utilized correctly I.e the query is searching directly in only one partition based on that date. But for table B, it is still searching all the partitions, it is not applying the where A.order_date condition for B tables partitions while searching.
> I do not understand why the A.order_date filter is not being applied to table B where as A.company_code is being applied to both tables.
>
> Thanks and Regards,
> J. Teja Sri.
>
>> On Aug 24, 2022, at 5:52 PM, Doug Reynolds <mav@wastegate.net> wrote:
>>
>> You need to change your query to not use the trunc. The TRUNC statement forces it to trunc every row in the table.
>>
>> If you do order_date >= to_date(..) and order_date < to_date(..) + interval '1' day, it will use the partition range to narrow down to the correct partition first.
>>
>> Sent from my iPhone
>>
>>>> On Aug 24, 2022, at 6:41 PM, Teja Jakkidi <teja.jakkidi05@gmail.com> wrote:
>>>
>>> Hello Admin team,
>>>
>>> We have a Postgres 13 server set up on Google Cloud. Recently we created partitions as below for the tables:
>>> Range on ‘order_date’ column
>>> List on ‘country_code’ column
>>>
>>> Quarterly partitions for range.
>>>
>>> Order date column is a time stamp column and when we are using this column as a filter in where condition : trunc(order_date) = to_date(’2022-04-01’,’YYYY-MM-DD’), the query scans all the partitions dated back from the year 2000. So, instead of going to the 2022 Q2 partition directly, the query is scanning all the partitions.
>>> But when we remove the trunc() function and just specify order_date =‘2022-04-01 07:02:30’, this works as expected. It goes to the correct partition directly and gets the data.
>>> Can Anyone help me on what’s happening in the first case and why all the partitions are being scanned?
>>> For the first case, when there is a non partition table, it is behaving way better.
>>>
>>> Thanks,
>>> J. Teja.
>>>
>>
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Performance issue after creating partitions
2022-08-24 22:41 Performance issue after creating partitions Teja Jakkidi <teja.jakkidi05@gmail.com>
2022-08-25 00:52 ` Re: Performance issue after creating partitions Doug Reynolds <mav@wastegate.net>
2022-08-26 18:46 ` Re: Performance issue after creating partitions Teja Jakkidi <teja.jakkidi05@gmail.com>
2022-08-30 21:21 ` Re: Performance issue after creating partitions Teja Jakkidi <teja.jakkidi05@gmail.com>
@ 2022-08-30 21:40 ` Tom Lane <tgl@sss.pgh.pa.us>
2022-08-31 17:59 ` Re: Performance issue after creating partitions Doug Reynolds <mav@wastegate.net>
0 siblings, 1 reply; 8+ messages in thread
From: Tom Lane @ 2022-08-30 21:40 UTC (permalink / raw)
To: Teja Jakkidi <teja.jakkidi05@gmail.com>; +Cc: Doug Reynolds <mav@wastegate.net>; gogala.mladen@gmail.com, pgsql-admin <pgsql-admin@lists.postgresql.org>
Teja Jakkidi <teja.jakkidi05@gmail.com> writes:
> Anyone ever encountered the below situation as what I am noticing with partitions?
You haven't shown us your query, so any answer would be blind speculation.
However, in the spirit of blind speculation, I'm wondering if you're
expecting those range constraints to propagate across a join. They
won't; you'd need to duplicate the conditions for the other table.
That is, if you have WHERE+JOIN/ON conditions amounting to
WHERE a.a = b.b AND b.b = constant
the planner is able to derive "a.a = constant" on the assumption of
transitivity, and use that to constrain the scan of table a (ie,
use an index, reject partitions at plan time, etc). But no such
deduction happens for
WHERE a.a = b.b AND b.b >= constant
If you want a constrained scan of a, you need to write it out:
WHERE a.a = b.b AND b.b >= constant AND a.a >= constant
regards, tom lane
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Performance issue after creating partitions
2022-08-24 22:41 Performance issue after creating partitions Teja Jakkidi <teja.jakkidi05@gmail.com>
2022-08-25 00:52 ` Re: Performance issue after creating partitions Doug Reynolds <mav@wastegate.net>
2022-08-26 18:46 ` Re: Performance issue after creating partitions Teja Jakkidi <teja.jakkidi05@gmail.com>
2022-08-30 21:21 ` Re: Performance issue after creating partitions Teja Jakkidi <teja.jakkidi05@gmail.com>
2022-08-30 21:40 ` Re: Performance issue after creating partitions Tom Lane <tgl@sss.pgh.pa.us>
@ 2022-08-31 17:59 ` Doug Reynolds <mav@wastegate.net>
0 siblings, 0 replies; 8+ messages in thread
From: Doug Reynolds @ 2022-08-31 17:59 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Teja Jakkidi <teja.jakkidi05@gmail.com>; gogala.mladen@gmail.com, pgsql-admin <pgsql-admin@lists.postgresql.org>
Tom is spot on with his suggestions.
The only thing that I'll add is that sometimes there is a mismatch with timestamp resolution, which prevents a direct inner join. In this case, you filter both partitioned tables with the greater than/less than constants, THEN complete the inner join with truncated dates. You will still have to do a full scan; however, you'll only be doing a full scan of the selected partitions.
Other than that, you need to provide the query for additional advice.
Doug
> On Aug 30, 2022, at 4:40 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> Teja Jakkidi <teja.jakkidi05@gmail.com> writes:
>> Anyone ever encountered the below situation as what I am noticing with partitions?
>
> You haven't shown us your query, so any answer would be blind speculation.
>
> However, in the spirit of blind speculation, I'm wondering if you're
> expecting those range constraints to propagate across a join. They
> won't; you'd need to duplicate the conditions for the other table.
>
> That is, if you have WHERE+JOIN/ON conditions amounting to
>
> WHERE a.a = b.b AND b.b = constant
>
> the planner is able to derive "a.a = constant" on the assumption of
> transitivity, and use that to constrain the scan of table a (ie,
> use an index, reject partitions at plan time, etc). But no such
> deduction happens for
>
> WHERE a.a = b.b AND b.b >= constant
>
> If you want a constrained scan of a, you need to write it out:
>
> WHERE a.a = b.b AND b.b >= constant AND a.a >= constant
>
> regards, tom lane
>
>
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Performance issue after creating partitions
2022-08-24 22:41 Performance issue after creating partitions Teja Jakkidi <teja.jakkidi05@gmail.com>
@ 2022-08-25 02:55 ` Mladen Gogala <gogala.mladen@gmail.com>
2 siblings, 0 replies; 8+ messages in thread
From: Mladen Gogala @ 2022-08-25 02:55 UTC (permalink / raw)
To: pgsql-admin@lists.postgresql.org
On 8/24/22 18:41, Teja Jakkidi wrote:
> trunc(order_date) = to_date(’2022-04-01’
You cannot have expression on the left side if you want partition
pruning. This is elementary mistake.
--
Mladen Gogala
Database Consultant
Tel: (347) 321-1217
https://dbwhisperer.wordpress.com
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Performance issue after creating partitions
2022-08-24 22:41 Performance issue after creating partitions Teja Jakkidi <teja.jakkidi05@gmail.com>
@ 2022-08-31 06:01 ` Olivier Gautherot <ogautherot@gautherot.net>
2 siblings, 0 replies; 8+ messages in thread
From: Olivier Gautherot @ 2022-08-31 06:01 UTC (permalink / raw)
To: Teja Jakkidi <teja.jakkidi05@gmail.com>; +Cc: pgsql-admin <pgsql-admin@lists.postgresql.org>
In this type of situation, I would have a column of type date for the
partitioning and another one for the timestamp. The additional space will
surely be marginal but the gain in performance really significant.
Another thing you could try is to add a computed index on
trunc(order_date): an index scan would move less data than a table scan.
Olivier Gautherot
Le jeu. 25 août 2022, 00:41, Teja Jakkidi <teja.jakkidi05@gmail.com> a
écrit :
> Hello Admin team,
>
> We have a Postgres 13 server set up on Google Cloud. Recently we created
> partitions as below for the tables:
> Range on ‘order_date’ column
> List on ‘country_code’ column
>
> Quarterly partitions for range.
>
> Order date column is a time stamp column and when we are using this column
> as a filter in where condition : trunc(order_date) =
> to_date(’2022-04-01’,’YYYY-MM-DD’), the query scans all the partitions
> dated back from the year 2000. So, instead of going to the 2022 Q2
> partition directly, the query is scanning all the partitions.
> But when we remove the trunc() function and just specify order_date
> =‘2022-04-01 07:02:30’, this works as expected. It goes to the correct
> partition directly and gets the data.
> Can Anyone help me on what’s happening in the first case and why all the
> partitions are being scanned?
> For the first case, when there is a non partition table, it is behaving
> way better.
>
> Thanks,
> J. Teja.
>
>
^ permalink raw reply [nested|flat] 8+ messages in thread
end of thread, other threads:[~2022-08-31 17:59 UTC | newest]
Thread overview: 8+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2022-08-24 22:41 Performance issue after creating partitions Teja Jakkidi <teja.jakkidi05@gmail.com>
2022-08-25 00:52 ` Doug Reynolds <mav@wastegate.net>
2022-08-26 18:46 ` Teja Jakkidi <teja.jakkidi05@gmail.com>
2022-08-30 21:21 ` Teja Jakkidi <teja.jakkidi05@gmail.com>
2022-08-30 21:40 ` Tom Lane <tgl@sss.pgh.pa.us>
2022-08-31 17:59 ` Doug Reynolds <mav@wastegate.net>
2022-08-25 02:55 ` Mladen Gogala <gogala.mladen@gmail.com>
2022-08-31 06:01 ` Olivier Gautherot <ogautherot@gautherot.net>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox