From: Tomas Vondra <tomas.vondra@2ndquadrant.com>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Andres Freund <andres@anarazel.de>
Cc: buschmann@nidsa.net
Cc: pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #16045: vacuum_db crash and illegal memory alloc after pg_upgrade from PG11 to PG12
Date: Tue, 15 Oct 2019 02:18:17 +0200
Message-ID: <20191015001817.zei45apgpu5dzngr@development> (raw)
In-Reply-To: <20191014163538.cqn6j4cl7s2mk2yz@development>
References: <20191010134303.bkwjzchjb2w66iro@development>
<30706.1570717152@sss.pgh.pa.us>
<20191010163350.2rtcjtje6tkgilqi@development>
<1542.1570738460@sss.pgh.pa.us>
<20191010204022.6hiiubm2cueu3ghk@development>
<20191013001032.lqrn4kqaovhrf77l@development>
<17717.1570991208@sss.pgh.pa.us>
<20191013203848.2eksimmpux3546wr@development>
<27596.1571062600@sss.pgh.pa.us>
<20191014163538.cqn6j4cl7s2mk2yz@development>
On Mon, Oct 14, 2019 at 06:35:38PM +0200, Tomas Vondra wrote:
> ...
>
>Aha! I forgot we copy the necessary stuff into pg_attribute. Thanks for
>clarifying, I'll polish and push the fix shortly.
>
I've pushed and backpatched the fix. Attached are similar fixes for the
existing pg_upgrade checks for pg_catalog.line and pg_catalog.unknown
types, which have the same issues with composite types and domains.
There are some additional details & examples in the commit messages.
I've kept this in two patches primarily because of backpatching - the
line fix should go back up to 9.4, the unknown is for 10.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
From 52821ea02fca502ff070508ee7be278563117509 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Tue, 15 Oct 2019 01:16:21 +0200
Subject: [PATCH 1/2] Correct the check for pg_catalog.line in pg_upgrade
The pg_upgrade check for pg_catalog.line data type when upgrading from
9.3 had a number of issues with domains and composite types. Firstly, it
detected even composite types unused in objects with storage. So for
example this was enough to trigger pg_upgrade failure:
CREATE TYPE line_composite AS (l pg_catalog.line)
On the other hand, this only happened with composite types directly on
the pg_catalog.line data type, but not with a domain. So this was not
detected
CREATE DOMAIN line_domain AS pg_catalog.line;
CREATE TYPE line_composite_2 AS (l line_domain);
unlike the first example. What's worse, we have not detected this even
when used in a table. So we missed cases like this:
CREATE TABLE t (l line_composite_2);
This fixes these false positives and false negatives by adopting the same
recursive CTE introduced by eaf900e842 for sql_identifier. Backpatch all
the way to 9.4, where the storage for pg_catalog.line data type changed.
Author: Tomas Vondra
Backpatch-to: 9.4-
Discussion: https://postgr.es/m/16045-673e8fa6b5ace196%40postgresql.org
---
src/bin/pg_upgrade/version.c | 29 ++++++++++++++++++++++++++++-
1 file changed, 28 insertions(+), 1 deletion(-)
diff --git a/src/bin/pg_upgrade/version.c b/src/bin/pg_upgrade/version.cindex 8375a46454..cfe69ea554 100644--- a/src/bin/pg_upgrade/version.c+++ b/src/bin/pg_upgrade/version.c@@ -131,14 +131,41 @@ old_9_3_check_for_line_data_type_usage(ClusterInfo *cluster)
DbInfo *active_db = &cluster->dbarr.dbs[dbnum];
PGconn *conn = connectToServer(cluster, active_db->db_name);
+ /*+ * We need the recursive CTE because the pg_catalog.line may be wrapped+ * either in a domain or composite type, or both (9.3 did not allow domains+ * on composite types, but there may be multi-level composite type).+ */
res = executeQueryOrDie(conn,
+ "WITH RECURSIVE oids AS ( "+ /* the pg_catalog.line type itself */+ " SELECT 'pg_catalog.line'::pg_catalog.regtype AS oid "+ " UNION ALL "+ " SELECT * FROM ( "+ /* domains on the type */+ " WITH x AS (SELECT oid FROM oids) "+ " SELECT t.oid FROM pg_catalog.pg_type t, x WHERE typbasetype = x.oid AND typtype = 'd' "+ " UNION "+ /* composite types containing the type */+ " SELECT t.oid FROM pg_catalog.pg_type t, pg_catalog.pg_class c, pg_catalog.pg_attribute a, x "+ " WHERE t.typtype = 'c' AND "+ " t.oid = c.reltype AND "+ " c.oid = a.attrelid AND "+ " NOT a.attisdropped AND "+ " a.atttypid = x.oid "+ " ) foo "+ ") "
"SELECT n.nspname, c.relname, a.attname "
"FROM pg_catalog.pg_class c, "
" pg_catalog.pg_namespace n, "
" pg_catalog.pg_attribute a "
"WHERE c.oid = a.attrelid AND "
" NOT a.attisdropped AND "
- " a.atttypid = 'pg_catalog.line'::pg_catalog.regtype AND "+ " a.atttypid IN (SELECT oid FROM oids) AND "+ " c.relkind IN ("+ CppAsString2(RELKIND_RELATION) ", "+ CppAsString2(RELKIND_MATVIEW) ", "+ CppAsString2(RELKIND_INDEX) ") AND "
" c.relnamespace = n.oid AND "
/* exclude possible orphaned temp tables */
" n.nspname !~ '^pg_temp_' AND "
--
2.21.0
From de98ca82812798e6b7392196fdcf34dff8a78133 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Tue, 15 Oct 2019 01:16:47 +0200
Subject: [PATCH 2/2] Correct the check for pg_catalog.unknown in pg_upgrade
The pg_upgrade check for pg_catalog.unknown type when upgrading from 9.6
had a couple of issues with domains and composite types. Firstly, it
detected even composite types unused in objects with storage. So for
example this was enough to trigger pg_upgrade failure:
CREATE TYPE unknown_composite AS (u pg_catalog.unknown)
On the other hand, this only happened with composite types directly on
the pg_catalog.unknown data type, but not with a domain. So this was not
detected
CREATE DOMAIN unknown_domain AS pg_catalog.unknown;
CREATE TYPE unknown_composite_2 AS (u unknown_domain);
unlike the first example. What's worse, we have not detected this even
when used in a table. So we missed cases like this:
CREATE TABLE t (u unknown_composite_2);
This fixes these false positives and false negatives by using the same
recursive CTE introduced by eaf900e842 for sql_identifier. Backpatch all
the way to 10, where the of pg_catalog.unknown data type was restricted.
Author: Tomas Vondra
Backpatch-to: 10-
Discussion: https://postgr.es/m/16045-673e8fa6b5ace196%40postgresql.org
---
src/bin/pg_upgrade/version.c | 21 +++++++++++++++++++--
1 file changed, 19 insertions(+), 2 deletions(-)
diff --git a/src/bin/pg_upgrade/version.c b/src/bin/pg_upgrade/version.cindex cfe69ea554..ccfb83c7eb 100644--- a/src/bin/pg_upgrade/version.c+++ b/src/bin/pg_upgrade/version.c@@ -256,16 +256,33 @@ old_9_6_check_for_unknown_data_type_usage(ClusterInfo *cluster)
PGconn *conn = connectToServer(cluster, active_db->db_name);
res = executeQueryOrDie(conn,
+ "WITH RECURSIVE oids AS ( "+ /* the sql_identifier type itself */+ " SELECT 'pg_catalog.unknown'::pg_catalog.regtype AS oid "+ " UNION ALL "+ " SELECT * FROM ( "+ /* domains on the type */+ " WITH x AS (SELECT oid FROM oids) "+ " SELECT t.oid FROM pg_catalog.pg_type t, x WHERE typbasetype = x.oid AND typtype = 'd' "+ " UNION "+ /* composite types containing the type */+ " SELECT t.oid FROM pg_catalog.pg_type t, pg_catalog.pg_class c, pg_catalog.pg_attribute a, x "+ " WHERE t.typtype = 'c' AND "+ " t.oid = c.reltype AND "+ " c.oid = a.attrelid AND "+ " NOT a.attisdropped AND "+ " a.atttypid = x.oid "+ " ) foo "+ ") "
"SELECT n.nspname, c.relname, a.attname "
"FROM pg_catalog.pg_class c, "
" pg_catalog.pg_namespace n, "
" pg_catalog.pg_attribute a "
"WHERE c.oid = a.attrelid AND "
" NOT a.attisdropped AND "
- " a.atttypid = 'pg_catalog.unknown'::pg_catalog.regtype AND "+ " a.atttypid IN (SELECT oid FROM oids) AND "
" c.relkind IN ("
CppAsString2(RELKIND_RELATION) ", "
- CppAsString2(RELKIND_COMPOSITE_TYPE) ", "
CppAsString2(RELKIND_MATVIEW) ") AND "
" c.relnamespace = n.oid AND "
/* exclude possible orphaned temp tables */
--
2.21.0
Attachments:
[text/plain] 0001-Correct-the-check-for-pg_catalog.line-in-pg_upgrade.patch (3.4K, ../20191015001817.zei45apgpu5dzngr@development/2-0001-Correct-the-check-for-pg_catalog.line-in-pg_upgrade.patch)
download | inline diff:
From 52821ea02fca502ff070508ee7be278563117509 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Tue, 15 Oct 2019 01:16:21 +0200
Subject: [PATCH 1/2] Correct the check for pg_catalog.line in pg_upgrade
The pg_upgrade check for pg_catalog.line data type when upgrading from
9.3 had a number of issues with domains and composite types. Firstly, it
detected even composite types unused in objects with storage. So for
example this was enough to trigger pg_upgrade failure:
CREATE TYPE line_composite AS (l pg_catalog.line)
On the other hand, this only happened with composite types directly on
the pg_catalog.line data type, but not with a domain. So this was not
detected
CREATE DOMAIN line_domain AS pg_catalog.line;
CREATE TYPE line_composite_2 AS (l line_domain);
unlike the first example. What's worse, we have not detected this even
when used in a table. So we missed cases like this:
CREATE TABLE t (l line_composite_2);
This fixes these false positives and false negatives by adopting the same
recursive CTE introduced by eaf900e842 for sql_identifier. Backpatch all
the way to 9.4, where the storage for pg_catalog.line data type changed.
Author: Tomas Vondra
Backpatch-to: 9.4-
Discussion: https://postgr.es/m/16045-673e8fa6b5ace196%40postgresql.org
---
src/bin/pg_upgrade/version.c | 29 ++++++++++++++++++++++++++++-
1 file changed, 28 insertions(+), 1 deletion(-)
diff --git a/src/bin/pg_upgrade/version.c b/src/bin/pg_upgrade/version.cindex 8375a46454..cfe69ea554 100644--- a/src/bin/pg_upgrade/version.c+++ b/src/bin/pg_upgrade/version.c@@ -131,14 +131,41 @@ old_9_3_check_for_line_data_type_usage(ClusterInfo *cluster)
DbInfo *active_db = &cluster->dbarr.dbs[dbnum];
PGconn *conn = connectToServer(cluster, active_db->db_name);
+ /*+ * We need the recursive CTE because the pg_catalog.line may be wrapped+ * either in a domain or composite type, or both (9.3 did not allow domains+ * on composite types, but there may be multi-level composite type).+ */
res = executeQueryOrDie(conn,
+ "WITH RECURSIVE oids AS ( "+ /* the pg_catalog.line type itself */+ " SELECT 'pg_catalog.line'::pg_catalog.regtype AS oid "+ " UNION ALL "+ " SELECT * FROM ( "+ /* domains on the type */+ " WITH x AS (SELECT oid FROM oids) "+ " SELECT t.oid FROM pg_catalog.pg_type t, x WHERE typbasetype = x.oid AND typtype = 'd' "+ " UNION "+ /* composite types containing the type */+ " SELECT t.oid FROM pg_catalog.pg_type t, pg_catalog.pg_class c, pg_catalog.pg_attribute a, x "+ " WHERE t.typtype = 'c' AND "+ " t.oid = c.reltype AND "+ " c.oid = a.attrelid AND "+ " NOT a.attisdropped AND "+ " a.atttypid = x.oid "+ " ) foo "+ ") "
"SELECT n.nspname, c.relname, a.attname "
"FROM pg_catalog.pg_class c, "
" pg_catalog.pg_namespace n, "
" pg_catalog.pg_attribute a "
"WHERE c.oid = a.attrelid AND "
" NOT a.attisdropped AND "
- " a.atttypid = 'pg_catalog.line'::pg_catalog.regtype AND "+ " a.atttypid IN (SELECT oid FROM oids) AND "+ " c.relkind IN ("+ CppAsString2(RELKIND_RELATION) ", "+ CppAsString2(RELKIND_MATVIEW) ", "+ CppAsString2(RELKIND_INDEX) ") AND "
" c.relnamespace = n.oid AND "
/* exclude possible orphaned temp tables */
" n.nspname !~ '^pg_temp_' AND "
--
2.21.0
[text/plain] 0002-Correct-the-check-for-pg_catalog.unknown-in-pg_upgra.patch (3.1K, ../20191015001817.zei45apgpu5dzngr@development/3-0002-Correct-the-check-for-pg_catalog.unknown-in-pg_upgra.patch)
download | inline diff:
From de98ca82812798e6b7392196fdcf34dff8a78133 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Tue, 15 Oct 2019 01:16:47 +0200
Subject: [PATCH 2/2] Correct the check for pg_catalog.unknown in pg_upgrade
The pg_upgrade check for pg_catalog.unknown type when upgrading from 9.6
had a couple of issues with domains and composite types. Firstly, it
detected even composite types unused in objects with storage. So for
example this was enough to trigger pg_upgrade failure:
CREATE TYPE unknown_composite AS (u pg_catalog.unknown)
On the other hand, this only happened with composite types directly on
the pg_catalog.unknown data type, but not with a domain. So this was not
detected
CREATE DOMAIN unknown_domain AS pg_catalog.unknown;
CREATE TYPE unknown_composite_2 AS (u unknown_domain);
unlike the first example. What's worse, we have not detected this even
when used in a table. So we missed cases like this:
CREATE TABLE t (u unknown_composite_2);
This fixes these false positives and false negatives by using the same
recursive CTE introduced by eaf900e842 for sql_identifier. Backpatch all
the way to 10, where the of pg_catalog.unknown data type was restricted.
Author: Tomas Vondra
Backpatch-to: 10-
Discussion: https://postgr.es/m/16045-673e8fa6b5ace196%40postgresql.org
---
src/bin/pg_upgrade/version.c | 21 +++++++++++++++++++--
1 file changed, 19 insertions(+), 2 deletions(-)
diff --git a/src/bin/pg_upgrade/version.c b/src/bin/pg_upgrade/version.cindex cfe69ea554..ccfb83c7eb 100644--- a/src/bin/pg_upgrade/version.c+++ b/src/bin/pg_upgrade/version.c@@ -256,16 +256,33 @@ old_9_6_check_for_unknown_data_type_usage(ClusterInfo *cluster)
PGconn *conn = connectToServer(cluster, active_db->db_name);
res = executeQueryOrDie(conn,
+ "WITH RECURSIVE oids AS ( "+ /* the sql_identifier type itself */+ " SELECT 'pg_catalog.unknown'::pg_catalog.regtype AS oid "+ " UNION ALL "+ " SELECT * FROM ( "+ /* domains on the type */+ " WITH x AS (SELECT oid FROM oids) "+ " SELECT t.oid FROM pg_catalog.pg_type t, x WHERE typbasetype = x.oid AND typtype = 'd' "+ " UNION "+ /* composite types containing the type */+ " SELECT t.oid FROM pg_catalog.pg_type t, pg_catalog.pg_class c, pg_catalog.pg_attribute a, x "+ " WHERE t.typtype = 'c' AND "+ " t.oid = c.reltype AND "+ " c.oid = a.attrelid AND "+ " NOT a.attisdropped AND "+ " a.atttypid = x.oid "+ " ) foo "+ ") "
"SELECT n.nspname, c.relname, a.attname "
"FROM pg_catalog.pg_class c, "
" pg_catalog.pg_namespace n, "
" pg_catalog.pg_attribute a "
"WHERE c.oid = a.attrelid AND "
" NOT a.attisdropped AND "
- " a.atttypid = 'pg_catalog.unknown'::pg_catalog.regtype AND "+ " a.atttypid IN (SELECT oid FROM oids) AND "
" c.relkind IN ("
CppAsString2(RELKIND_RELATION) ", "
- CppAsString2(RELKIND_COMPOSITE_TYPE) ", "
CppAsString2(RELKIND_MATVIEW) ") AND "
" c.relnamespace = n.oid AND "
/* exclude possible orphaned temp tables */
--
2.21.0
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: tomas.vondra@2ndquadrant.com, tgl@sss.pgh.pa.us, andres@anarazel.de, buschmann@nidsa.net, pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #16045: vacuum_db crash and illegal memory alloc after pg_upgrade from PG11 to PG12
In-Reply-To: <20191015001817.zei45apgpu5dzngr@development>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox