Subject: Re: AWS forcing PG upgrade from v9.6 a disaster
To: pgsql-performance@lists.postgresql.org
References: <88dec70b-eb63-0b72-0b5d-253851ef127b@mailpen.com>
 <92cc8563-195f-8727-d887-5f2bb63f231f@mailpen.com>
 <20210601041635.GQ2082@telsasoft.com>
From: "Dean Gibson (DB Administrator)" <postgresql@mailpen.com>
Organization: UltiMeth Systems
Message-ID: <a38c45cf-3bbd-b43a-0f8a-11539c1c6dc1@mailpen.com>
Date: Tue, 1 Jun 2021 10:44:54 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101
 Thunderbird/78.10.2
MIME-Version: 1.0
In-Reply-To: <20210601041635.GQ2082@telsasoft.com>
Content-Type: multipart/alternative;
 boundary="------------A6B229792EDEF4FDBBE30E94"
Content-Language: en-US
Archived-At: 
 <https://www.postgresql.org/message-id/a38c45cf-3bbd-b43a-0f8a-11539c1c6dc1%40mailpen.com>
Precedence: bulk

This is a multi-part message in MIME format.
--------------A6B229792EDEF4FDBBE30E94
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit

On 2021-05-31 21:16, Justin Pryzby wrote:
>> Here's the FROM clause that bit me:
>>
>>     FROM lic_en
>>       JOIN govt_region   USING (territory_id, country_id)
>>       LEFT JOIN zip_code USING (territory_id, country_id, zip5)
>>       LEFT JOIN "County" USING (territory_id, country_id, fips_county);
> I'm guessing that there's a dependency/correlation between territory/country/county, and that's probably related to a misestimate causing a bad plan.
>
>> The first two JOINs are not the problem, & are in fact retained in my solution.  The problem is the third JOIN, where "fips_county" from "County" is actually matched with the corresponding field from the "zip_code" VIEW.  Works fine, if you don't mind the performance impact in v10 & above.  It has now been rewritten, to be a sub-query for an output field.  Voila ! Back to sub-second query times.
> What version of 9.6.X were you upgrading *from* ?
>
> v9.6 added selectivity estimates based on FKs, so it's not surprising if there was a plan change migrating *to* v9.6.

I originally upgraded from 9.6.20 to v12.6.  When that (otherwise 
successful) upgrade had performance problems, I upgraded the v9.6.20 
copy to v9.6.21, & tried again, with the same result.

Interestingly, on v13.2 I have now run into another (similar) 
performance issue.  I've solved it by setting the following to values I 
used with v9.x:

join_collapse_limit & from_collapse_limit = 16

geqo_threshold = 32

I pretty sure I tried those settings (on v10 & above) with the earlier 
performance problem, to no avail.  However, I now wonder what would have 
been the result if I have doubled those values before re-architecting 
some of my tables (moving from certain JOINs to specific sub-selects).


--------------A6B229792EDEF4FDBBE30E94
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit

<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body>
    <div class="moz-cite-prefix">On 2021-05-31 21:16, Justin Pryzby
      wrote:<br>
    </div>
    <blockquote type="cite"
      cite="mid:20210601041635.GQ2082@telsasoft.com">
      <blockquote type="cite">
        <pre class="moz-quote-pre" wrap="">Here's the FROM clause that bit me:

   FROM lic_en
     JOIN govt_region   USING (territory_id, country_id)
     LEFT JOIN zip_code USING (territory_id, country_id, zip5)
     LEFT JOIN "County" USING (territory_id, country_id, fips_county);
</pre>
      </blockquote>
      <pre class="moz-quote-pre" wrap="">
I'm guessing that there's a dependency/correlation between territory/country/county, and that's probably related to a misestimate causing a bad plan.

</pre>
      <blockquote type="cite">
        <pre class="moz-quote-pre" wrap="">The first two JOINs are not the problem, &amp; are in fact retained in my solution.  The problem is the third JOIN, where "fips_county" from "County" is actually matched with the corresponding field from the "zip_code" VIEW.  Works fine, if you don't mind the performance impact in v10 &amp; above.  It has now been rewritten, to be a sub-query for an output field.  Voila ! Back to sub-second query times.
</pre>
      </blockquote>
      <pre class="moz-quote-pre" wrap="">
What version of 9.6.X were you upgrading *from* ?

v9.6 added selectivity estimates based on FKs, so it's not surprising if there was a plan change migrating *to* v9.6.
</pre>
    </blockquote>
    <br>
    I originally upgraded from 9.6.20 to v12.6.  When that (otherwise
    successful) upgrade had performance problems, I upgraded the v9.6.20
    copy to v9.6.21, &amp; tried again, with the same result.<br>
    <br>
    Interestingly, on v13.2 I have now run into another (similar)
    performance issue.  I've solved it by setting the following to
    values I used with v9.x:<br>
    <br>
    join_collapse_limit &amp; from_collapse_limit = 16<br>
    <br>
    geqo_threshold = 32<br>
    <br>
    I pretty sure I tried those settings (on v10 &amp; above) with the
    earlier performance problem, to no avail.  However, I now wonder
    what would have been the result if I have doubled those values
    before re-architecting some of my tables (moving from certain JOINs
    to specific sub-selects).<br>
    <br>
  </body>
</html>

--------------A6B229792EDEF4FDBBE30E94--