Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lFRll-0004I2-HF for pgsql-bugs@arkaria.postgresql.org; Fri, 26 Feb 2021 01:18:14 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1lFRlj-0006O6-0o for pgsql-bugs@arkaria.postgresql.org; Fri, 26 Feb 2021 01:18:11 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lFRlh-0006Ny-Kr for pgsql-bugs@lists.postgresql.org; Fri, 26 Feb 2021 01:18:10 +0000 Received: from out4-smtp.messagingengine.com ([66.111.4.28]) by magus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lFRlZ-0000EV-BA for pgsql-bugs@lists.postgresql.org; Fri, 26 Feb 2021 01:18:08 +0000 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 839F85C009D; Thu, 25 Feb 2021 20:17:58 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Thu, 25 Feb 2021 20:17:58 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=paquier.xyz; h= date:from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm1; bh=N+gB7IYDhWi7J0HHCWr9ztVUsQV Mtd0Q0S0mcgH4SnA=; b=0igB9vimIXCkD6qDpoO2nZdkKEF7YuliyEz7MvTfwT4 hpmFNA3TFUJpPepH44rfy0rrYU3GVsOul6eAfTqgI6Fd6BJIEsPySWg8UjG0x+E2 UW5wMUNXvx7W/xtjGjVdQrR9pGYT+b5pN5cEIpD1DPDp+tM0MFH5efWZjHNpodWB VUbHSUw0W0NVIa/jEK/WQJsQRNl2LhcA2kX0Ow0KlAbSSURw8j+JT99vsIs0DW94 gZjtb9YCApAfdbRKt1S5iOaf89aWNxHf13tCyKo/OXVWmSRga4JzP2sFs0DCwRIn TeW52KHn2zp94c4lWcE4HyHpbLhutlOxv9bmfcW8zWQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=N+gB7I YDhWi7J0HHCWr9ztVUsQVMtd0Q0S0mcgH4SnA=; b=o2TMTqAkCIWII3tEOp7A0u q6q+ocMToiqQ9a6aczS4mwiSoznr/s++CWWM1Z61xDHryNaNDjGCPQ1y2h0sr4a4 Xo+1rl8hdHaQKm4YS5se7jNbdjC7ZHkmP67ZpJu8To3CfqR3O0pyPfFD7UUJ6iig rTZm6YB3efihvrXvl/4REXlC7y/E/vRt72sDK47uMdLX8Ke4q3oFgus535g/wRUc 3Jl4cxQIELVYJFjDvl/5jqVDxrZesfbTBkqSO1CZ+Koeb/7oJjuSI/2jPuq1hH44 IjxndA4dK5r8zgl1XvVoJKjDW6heQLKEHL85j1L0Hso4rayOsQdIHv8UCY4MuhYw == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrledtgdeffecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecufghrlhcuvffnffculdejtddmnecujfgurhepfffhvf fukfhfgggtuggjsehgtderredttddvnecuhfhrohhmpefoihgthhgrvghlucfrrghquhhi vghruceomhhitghhrggvlhesphgrqhhuihgvrhdrgiihiieqnecuggftrfgrthhtvghrnh epvdegudeuhfdtueeltedtveejheehieevueeigeelteegleejleeiueeiheegvefhnecu kfhppeduuddurddutddvrddukedtrddukeehnecuvehluhhsthgvrhfuihiivgeptdenuc frrghrrghmpehmrghilhhfrhhomhepmhhitghhrggvlhesphgrqhhuihgvrhdrgiihii X-ME-Proxy: Received: from paquier.xyz (ee0822lan1.rev.em-net.ne.jp [111.102.180.185]) by mail.messagingengine.com (Postfix) with ESMTPA id C8697240054; Thu, 25 Feb 2021 20:17:54 -0500 (EST) Date: Fri, 26 Feb 2021 10:17:50 +0900 From: Michael Paquier To: Tom Lane Cc: Alexander Lakhin , pgsql-bugs@lists.postgresql.org Subject: Re: BUG #16801: Invalid memory access on WITH RECURSIVE with nested WITHs Message-ID: References: <4057243.1614179963@sss.pgh.pa.us> <31724.1614213386@sss.pgh.pa.us> <51877.1614219226@sss.pgh.pa.us> <60272.1614221283@sss.pgh.pa.us> <119550.1614266383@sss.pgh.pa.us> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="PEnGPIbP7eBncerK" Content-Disposition: inline In-Reply-To: <119550.1614266383@sss.pgh.pa.us> List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk --PEnGPIbP7eBncerK Content-Type: multipart/mixed; boundary="0ahfSXe8s1XSrqGN" Content-Disposition: inline --0ahfSXe8s1XSrqGN Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Thu, Feb 25, 2021 at 10:19:43AM -0500, Tom Lane wrote: > For me, the example I gave fails in a non-debug build. With the code as > of HEAD, I get "ERROR: stack depth limit exceeded" or a segfault. With > your patch applied, it complains about w6 not having the correct form for > a recursive CTE. Yes, same here for the test stressing makeDependencyGraphWalker(). The second test I posted for checkWellFormedRecursionWalker() passes on HEAD in a non-debug build, and triggers an assertion with debug builds. So the second test requires CLOBBER_FREED_MEMORY but not DEBUG_LIST_MEMORY_USAGE. That's not as good as the first one, but I would vote for having this second test than none to stress more the list handling when looking after invalid self-references in a CTE. This gives me the attached. I would rather have both tests in the version committed, but if you think that this addition is not necessary I won't fight hard either :) -- Michael --0ahfSXe8s1XSrqGN Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="bug-16801-fix-v2.patch" Content-Transfer-Encoding: quoted-printable diff --git a/src/backend/parser/parse_cte.c b/src/backend/parser/parse_cte.c index f4f7041ead..f46d63d451 100644 --- a/src/backend/parser/parse_cte.c +++ b/src/backend/parser/parse_cte.c @@ -730,15 +730,15 @@ makeDependencyGraphWalker(Node *node, CteState *cstat= e) * In the non-RECURSIVE case, query names are visible to the * WITH items after them and to the main query. */ - ListCell *cell1; - cstate->innerwiths =3D lcons(NIL, cstate->innerwiths); - cell1 =3D list_head(cstate->innerwiths); foreach(lc, stmt->withClause->ctes) { CommonTableExpr *cte =3D (CommonTableExpr *) lfirst(lc); + ListCell *cell1; =20 (void) makeDependencyGraphWalker(cte->ctequery, cstate); + /* note that recursion could mutate innerwiths list */ + cell1 =3D list_head(cstate->innerwiths); lfirst(cell1) =3D lappend((List *) lfirst(cell1), cte); } (void) raw_expression_tree_walker(node, @@ -1006,15 +1006,15 @@ checkWellFormedRecursionWalker(Node *node, CteState= *cstate) * In the non-RECURSIVE case, query names are visible to the * WITH items after them and to the main query. */ - ListCell *cell1; - cstate->innerwiths =3D lcons(NIL, cstate->innerwiths); - cell1 =3D list_head(cstate->innerwiths); foreach(lc, stmt->withClause->ctes) { CommonTableExpr *cte =3D (CommonTableExpr *) lfirst(lc); + ListCell *cell1; =20 (void) checkWellFormedRecursionWalker(cte->ctequery, cstate); + /* note that recursion could mutate innerwiths list */ + cell1 =3D list_head(cstate->innerwiths); lfirst(cell1) =3D lappend((List *) lfirst(cell1), cte); } checkWellFormedSelectStmt(stmt, cstate); diff --git a/src/test/regress/expected/with.out b/src/test/regress/expected= /with.out index c519a61c4f..a7a652822c 100644 --- a/src/test/regress/expected/with.out +++ b/src/test/regress/expected/with.out @@ -176,6 +176,65 @@ ERROR: operator does not exist: text + integer LINE 4: SELECT n+1 FROM t WHERE n < 10 ^ HINT: No operator matches the given name and argument types. You might ne= ed to add explicit type casts. +-- Deeply nested WITH caused a list-munging problem in v13 +-- Detection of cross-references and self-references +WITH RECURSIVE w1(c1) AS + (WITH w2(c2) AS + (WITH w3(c3) AS + (WITH w4(c4) AS + (WITH w5(c5) AS + (WITH RECURSIVE w6(c6) AS + (WITH w6(c6) AS + (WITH w8(c8) AS + (SELECT 1) + SELECT * FROM w8) + SELECT * FROM w6) + SELECT * FROM w6) + SELECT * FROM w5) + SELECT * FROM w4) + SELECT * FROM w3) + SELECT * FROM w2) +SELECT * FROM w1; + c1=20 +---- + 1 +(1 row) + +-- Detection of invalid self-references +WITH RECURSIVE outermost(x) AS ( + SELECT 1 + UNION (WITH innermost1 AS ( + SELECT 2 + UNION (WITH innermost2 AS ( + SELECT 3 + UNION (WITH innermost3 AS ( + SELECT 4 + UNION (WITH innermost4 AS ( + SELECT 5 + UNION (WITH innermost5 AS ( + SELECT 6 + UNION (WITH innermost6 AS + (SELECT 7) + SELECT * FROM innermost6)) + SELECT * FROM innermost5)) + SELECT * FROM innermost4)) + SELECT * FROM innermost3)) + SELECT * FROM innermost2)) + SELECT * FROM outermost + UNION SELECT * FROM innermost1) + ) + SELECT * FROM outermost ORDER BY 1; + x=20 +--- + 1 + 2 + 3 + 4 + 5 + 6 + 7 +(7 rows) + -- -- Some examples with a tree -- diff --git a/src/test/regress/sql/with.sql b/src/test/regress/sql/with.sql index f4ba0d8e39..85a671c630 100644 --- a/src/test/regress/sql/with.sql +++ b/src/test/regress/sql/with.sql @@ -95,6 +95,50 @@ UNION ALL ) SELECT n, pg_typeof(n) FROM t; =20 +-- Deeply nested WITH caused a list-munging problem in v13 +-- Detection of cross-references and self-references +WITH RECURSIVE w1(c1) AS + (WITH w2(c2) AS + (WITH w3(c3) AS + (WITH w4(c4) AS + (WITH w5(c5) AS + (WITH RECURSIVE w6(c6) AS + (WITH w6(c6) AS + (WITH w8(c8) AS + (SELECT 1) + SELECT * FROM w8) + SELECT * FROM w6) + SELECT * FROM w6) + SELECT * FROM w5) + SELECT * FROM w4) + SELECT * FROM w3) + SELECT * FROM w2) +SELECT * FROM w1; +-- Detection of invalid self-references +WITH RECURSIVE outermost(x) AS ( + SELECT 1 + UNION (WITH innermost1 AS ( + SELECT 2 + UNION (WITH innermost2 AS ( + SELECT 3 + UNION (WITH innermost3 AS ( + SELECT 4 + UNION (WITH innermost4 AS ( + SELECT 5 + UNION (WITH innermost5 AS ( + SELECT 6 + UNION (WITH innermost6 AS + (SELECT 7) + SELECT * FROM innermost6)) + SELECT * FROM innermost5)) + SELECT * FROM innermost4)) + SELECT * FROM innermost3)) + SELECT * FROM innermost2)) + SELECT * FROM outermost + UNION SELECT * FROM innermost1) + ) + SELECT * FROM outermost ORDER BY 1; + -- -- Some examples with a tree -- --0ahfSXe8s1XSrqGN-- --PEnGPIbP7eBncerK Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEEG72nH6vTowiyblFKnvQgOdbyQH0FAmA4TD4ACgkQnvQgOdby QH2MgBAAi60K+RII9bfxINc4cpQGiVaxyrCb23NQ7pp+KW+k3bplr5shPnWW3RRv k3vmqjcqU0twZy+1rvyAvNWjdzi1Opg6l5JCbD0qIA7xSqzKVXFu7T0WKLXeVB7J IiDwE5zhnhgoVx52OUhPuVXRn8nracX2TRZf9j/XptIOhLn9WrL0o2rmf54+phqF H19petWfrUZU9sY82WFJFqwd0GI8hTdO7OCKcx1XthrM5S7xEYCquFtQT357oJea d7kiwteJjG9+W9pHq+iQyczaYQRRxsUxFNvJf2SPPV452EOM3QdiN86gqtUAM5NK kiPaJ5iOo7U+7pjBIvoJLqL+gpprJ7nWY5hjIp97gJqiHACE/11D+dKuclwWegNN uuQk8WpHN/ps3sHFNwjF/NAhaTQcRcRzmW8T38A2ktZIKXSEhNRla16rF7roW+Te bddfF0SSLP0eYAHaDlJlgWQkL8sDW+UuSORDUpJiobqcRn9j4Xryib7m8iinE8x9 3jC14IftoYgpiOQN6q0Wqm3U8oT7unbj/W8Kn9BkkIfRLal9NeETc3Cg33umGOEI tCO9B68qLR79QdB/wDa4gzKnQrhs2P373W5kSd5giWjLmePps9EiuttzHwdRC9MN AuNdLM53WKHWWkpnGwalAoTu0Nt856NZ9S18XJLFdk81SqKXjuw= =R1KA -----END PGP SIGNATURE----- --PEnGPIbP7eBncerK--