Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lmn0A-0006wy-Ee for pgsql-general@arkaria.postgresql.org; Sat, 29 May 2021 00:38:54 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1lmn09-0002AQ-2L for pgsql-general@arkaria.postgresql.org; Sat, 29 May 2021 00:38:53 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lmn08-0002AI-PU for pgsql-general@lists.postgresql.org; Sat, 29 May 2021 00:38:52 +0000 Received: from aws.mailpen.com ([52.43.43.172] helo=aws2.mailpen.com) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lmn02-0004QM-AQ for pgsql-general@lists.postgresql.org; Sat, 29 May 2021 00:38:50 +0000 Received: from [172.20.70.208] (unknown [50.35.114.95]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by aws2.mailpen.com (Postfix) with ESMTPSA id EB28BC7E921 for ; Fri, 28 May 2021 17:38:43 -0700 (PDT) Subject: Re: AWS forcing PG upgrade from v9.6 a disaster To: pgsql-general@lists.postgresql.org References: <3fa6fd38-1f92-98ab-0055-3e0eb7e4e772@ultimeth.com> <2a1b34f4-73d4-585b-f4fc-3bd9b9ba248e@aklaver.com> <88255b75-e7e3-b29c-0e75-f989ea320570@mailpen.com> <70edb767-6ba0-1065-2f74-f01440e3caea@gmail.com> <2b7b9a25-e3c7-b980-a1e3-d21d8d91d34d@gmail.com> From: "Dean Gibson (DB Administrator)" Organization: UltiMeth Systems Message-ID: <988e2438-c322-c1d5-ca88-683943a80087@mailpen.com> Date: Fri, 28 May 2021 17:38:43 -0700 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.10.2 MIME-Version: 1.0 In-Reply-To: <2b7b9a25-e3c7-b980-a1e3-d21d8d91d34d@gmail.com> Content-Type: multipart/alternative; boundary="------------A23454091B10E26B1EE8A809" Content-Language: en-US List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk This is a multi-part message in MIME format. --------------A23454091B10E26B1EE8A809 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit On 2021-05-28 16:51, Ron wrote: > On 5/28/21 5:06 PM, Dean Gibson (DB Administrator) wrote: >> On 2021-05-28 12:38, Ron wrote: >>> On 5/28/21 1:40 PM, Dean Gibson (DB Administrator) wrote: >>>> On 2021-05-28 08:12, Adrian Klaver wrote: >>>>> On 5/27/21 8:41 PM, Dean Gibson (DB Administrator) wrote: >>>>>> I started to use PostgreSQL v7.3 in 2003 on my home Linux systems >>>>>> (4 at one point), gradually moving to v9.0 w/ replication in >>>>>> 2010.  In 2017 I moved my 20GB database to AWS/RDS, gradually >>>>>> upgrading to v9.6, & was entirely satisfied with the result. >>>>>> >>>>>> In March of this year, AWS announced that v9.6 was nearing end of >>>>>> support, & AWS would forcibly upgrade everyone to v12 on January >>>>>> 22, 2022, if users did not perform the upgrade earlier.  My first >>>>>> attempt was successful as far as the upgrade itself, but complex >>>>>> queries that normally ran in a couple of seconds on v9.x, were >>>>>> taking minutes in v12. >>>>> >>>>> Did you run a plain >>>>> ANALYZE(https://www.postgresql.org/docs/12/sql-analyze.html) on >>>>> the tables in the new install? >>>> >>>> After each upgrade (to 10, 11, 12, & 13), I did a "VACUUM FULL >>>> ANALYZE".  On 10 through 12, it took about 45 minutes & significant >>>> CPU activity, & temporarily doubled the size of the disk space >>>> required.  As you know, that disk space is not shrinkable under >>>> AWS's RDS.  On v13, it took 10 hours with limited CPU activity, & >>>> actually slightly less disk space required. >>> >>> Under normal conditions, VACUUM FULL is pointless on a >>> freshly-loaded database; in RDS, it's *anti-useful*. >>> >>> That's why Adrian asked if you did a plain ANALYZE. >> >> Just now did.  No change in EXPLAIN ANALYZE output. > > Did it run in less than 10 hours? > The original VACUUM FULL ANALYZE ran in 10 hours.  The plain ANALYZE ran in 88 seconds. --------------A23454091B10E26B1EE8A809 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit On 2021-05-28 16:51, Ron wrote:
On 5/28/21 5:06 PM, Dean Gibson (DB Administrator) wrote:
On 2021-05-28 12:38, Ron wrote:
On 5/28/21 1:40 PM, Dean Gibson (DB Administrator) wrote:
On 2021-05-28 08:12, Adrian Klaver wrote:
On 5/27/21 8:41 PM, Dean Gibson (DB Administrator) wrote:
I started to use PostgreSQL v7.3 in 2003 on my home Linux systems (4 at one point), gradually moving to v9.0 w/ replication in 2010.  In 2017 I moved my 20GB database to AWS/RDS, gradually upgrading to v9.6, & was entirely satisfied with the result.

In March of this year, AWS announced that v9.6 was nearing end of support, & AWS would forcibly upgrade everyone to v12 on January 22, 2022, if users did not perform the upgrade earlier.  My first attempt was successful as far as the upgrade itself, but complex queries that normally ran in a couple of seconds on v9.x, were taking minutes in v12.

Did you run a plain ANALYZE(https://www.postgresql.org/docs/12/sql-analyze.html) on the tables in the new install?

After each upgrade (to 10, 11, 12, & 13), I did a "VACUUM FULL ANALYZE".  On 10 through 12, it took about 45 minutes & significant CPU activity, & temporarily doubled the size of the disk space required.  As you know, that disk space is not shrinkable under AWS's RDS.  On v13, it took 10 hours with limited CPU activity, & actually slightly less disk space required.

Under normal conditions, VACUUM FULL is pointless on a freshly-loaded database; in RDS, it's anti-useful.

That's why Adrian asked if you did a plain ANALYZE.

Just now did.  No change in EXPLAIN ANALYZE output.

Did it run in less than 10 hours?


The original VACUUM FULL ANALYZE ran in 10 hours.  The plain ANALYZE ran in 88 seconds.
--------------A23454091B10E26B1EE8A809--