pg.ddx.io  pgsql-general@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Rob Sargent <robjsargent@gmail.com>
To: pgsql-general@lists.postgresql.org
Subject: Re: Query plan for "id IS NULL" on PK
Date: Tue, 14 Feb 2023 15:25:32 -0700
Message-ID: <8e622f19-bc95-eac3-188d-b03734f547f6@gmail.com> (raw)
In-Reply-To: <CALRaRxzGqDbry9r+0KY5WhpY1xB=o3v3ywrQy2BRUYqZeY5fdQ@mail.gmail.com>
References: <CALRaRxzGqDbry9r+0KY5WhpY1xB=o3v3ywrQy2BRUYqZeY5fdQ@mail.gmail.com>

On 2/14/23 15:04, Ben Chrobot wrote:
> Hello,
>
> Long time listener, first time caller.
>
> We have a large table (~470 million rows) with integer primary key id 
> (not null) on a Postgres 14.5 cluster. A third-party tool is 
> attempting to perform a SELECT-based full table copy in preparation 
> for log-based sync with a query like the following:
>
> SELECT "id", "other_column_a", "other_column_b", "created_at", 
> "updated_at"
> FROM "public"."my_large_table"
> WHERE (("id" > ? OR "id" IS NULL)) AND (("id" <= ?))
> ORDER  BY "id" LIMIT 50000;
>
> The lower bound increments by batch size (50k) while the upper bound 
> is always the `max(id)`, in our case around 575,000,000.
>
> The query plan produced is very slow as the index condition does 
> basically nothing:
>
>                         QUERY PLAN
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>  Limit  (cost=0.57..21901.46 rows=50000 width=417) (actual 
> time=1708920.675..1709198.995 rows=50000 loops=1)
>    ->  Index Scan using my_large_table_pkey on 
> my_large_table  (cost=0.57..135230792.97 rows=308733624 width=417) 
> (actual time=1708920.673..1709195.926 rows=50000 loops=1)
>          Index Cond: (id <= 575187488)
>          Filter: ((id > 193208795) OR (id IS NULL))
>          Rows Removed by Filter: 157784540
>  Planning Time: 0.186 ms
>  Execution Time: 1709200.618 ms
> (7 rows)
> Time: 1709231.721 ms (28:29.232)
>
> We cannot change the query being executed. Is there any way we can 
> make the query planner ignore `OR (id IS NULL)` (as that will never be 
> the case for the PK) and use both `id` clauses in the index condition?
>
> We have provided the vendor with the same query without the `id is 
> null` showing that it's significantly faster (see below). They have 
> informed us that addressing the null check on a not null PK is on 
> their roadmap to address but no timeline.
>
> explain analyze
> SELECT "id", "other_column_a", "other_column_b", "created_at", 
> "updated_at"
> FROM "public"."my_large_table"
> WHERE (("id" > ?)) AND (("id" <= ?))
> ORDER  BY "id" LIMIT 50000;
>
> QUERY PLAN
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
> Limit (cost=0.57..13930.19 rows=50000 width=416) (actual 
> time=2.118..400.937 rows=50000 loops=1)
> -> Index Scan using my_large_table_pkey on my_large_table 
> (cost=0.57..85429173.84 rows=306645829 width=416) (actual 
> time=2.117..398.325 rows=50000 loops=1)
> Index Cond: ((id > 193208795) AND (id <= 575187488))
> Planning Time: 0.166 ms
> Execution Time: 402.376 ms
>
> We have tried leading the planner to water with this view but it did 
> not change the slow query plan:
>
> create view my_fast_large_table as
> select *
> from my_large_table
> where id is not null;
>
> Any other tricks to try here?
>
> Thank you,
> Ben Chrobot

When will id be null in a primary key?







view thread (11+ messages)  latest in thread

Message-ID: <8e622f19-bc95-eac3-188d-b03734f547f6@gmail.com>
Permalink:  ../8e622f19-bc95-eac3-188d-b03734f547f6@gmail.com/
Also on:    postgresql.org/message-id/8e622f19-bc95-eac3-188d-b03734f547f6@gmail.com

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-general@postgresql.org
  Cc: robjsargent@gmail.com
  Subject: Re: Query plan for "id IS NULL" on PK
  In-Reply-To: <8e622f19-bc95-eac3-188d-b03734f547f6@gmail.com>
  References: <CALRaRxzGqDbry9r+0KY5WhpY1xB=o3v3ywrQy2BRUYqZeY5fdQ@mail.gmail.com> <8e622f19-bc95-eac3-188d-b03734f547f6@gmail.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox