Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lEmLC-0005AF-Hj for pgsql-bugs@arkaria.postgresql.org; Wed, 24 Feb 2021 05:04:03 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1lEmLA-0002BR-5B for pgsql-bugs@arkaria.postgresql.org; Wed, 24 Feb 2021 05:04:00 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lEmL8-00029A-Hx for pgsql-bugs@lists.postgresql.org; Wed, 24 Feb 2021 05:03:59 +0000 Received: from out5-smtp.messagingengine.com ([66.111.4.29]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lEmL1-0002we-FC for pgsql-bugs@lists.postgresql.org; Wed, 24 Feb 2021 05:03:57 +0000 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 664F55C0105; Wed, 24 Feb 2021 00:03:50 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Wed, 24 Feb 2021 00:03:50 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=paquier.xyz; h= date:from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm1; bh=A50q14PcpUir/S6BNtA5nkF2uu9 a4Pj51eKykYMS3HQ=; b=NHAJ0F2blX9stay0bvjwtXrbQeSf2fT9b73XATFBa6x EbPhD/uXyZSyGgVM1OYc1JnbQF8iQmKXD5C219z7AR9/J2suVwOedRsO9Rto3q/3 QXeUPQebgZwBQfNhERN9hZNMZxu7EWM8q1xchkDWua+RQKMGs2/tchbn9JRorEWf V6hOIL+c2uoE/QzT7hSxed/l4MP62MSxfYmusQs2RRy3yZB7r1wAM1BcITDgLjaL gJljD5miUK7TxABA5DzVvy7rnTaUPpPhHMhxAXDiyqK5TVEej2I2U2LtzuxO66sv HUe+GWjuEOc+FXuMLY97gi7b9mCwUqZ4Oxq6o5k1jzw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=A50q14 PcpUir/S6BNtA5nkF2uu9a4Pj51eKykYMS3HQ=; b=j2Qydo5BTe5caYFqZ+1zxB /dI+QfC2PHcXHn4AN/WzAq2rrzFodnvk/TeG2S+AcoYyrKVgDW/hONj19AdXGjWI +FTS4fo80yi+0/SrrpZ4X0twthJCHaeKGl7JRr7XeBrxNq0j6nNbntwV5pIvnKFI 6mpwxPaKuZ+dSjb0SOpGxIzC48QWrqn1BJP7sID1SrlM6pbvREby1cn2FBDpAMrY 1S1ztvgyyw5W79mrK1XAhgEC/+4OJCtzU6eeAqryyGskbSKKVFS/YTO9p6gxlWXJ nt84iDkfUKQWWoTh+mTuRvNstYdXKt4g8ADDy/SBycNANNAF+hCfa3TMbWCbfh8A == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrkeeigdejgecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecufghrlhcuvffnffculdejtddmnecujfgurhepfffhvf fukfhfgggtuggjsehgtdorredttddvnecuhfhrohhmpefoihgthhgrvghlucfrrghquhhi vghruceomhhitghhrggvlhesphgrqhhuihgvrhdrgiihiieqnecuggftrfgrthhtvghrnh epheegtdeuheekteegueevtdelleekgeegfeetfeejjefgkeehieekveejhedujeehnecu kfhppeduuddurddutddvrddukedtrddukeehnecuvehluhhsthgvrhfuihiivgeptdenuc frrghrrghmpehmrghilhhfrhhomhepmhhitghhrggvlhesphgrqhhuihgvrhdrgiihii X-ME-Proxy: Received: from paquier.xyz (ee0822lan1.rev.em-net.ne.jp [111.102.180.185]) by mail.messagingengine.com (Postfix) with ESMTPA id DE09D240054; Wed, 24 Feb 2021 00:03:48 -0500 (EST) Date: Wed, 24 Feb 2021 14:03:40 +0900 From: Michael Paquier To: Alexander Lakhin Cc: pgsql-bugs@lists.postgresql.org Subject: Re: BUG #16801: Invalid memory access on WITH RECURSIVE with nested WITHs Message-ID: References: <16801-393c7922143eaa4d@postgresql.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="08mqMGmhDI7AmXgD" Content-Disposition: inline In-Reply-To: List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk --08mqMGmhDI7AmXgD Content-Type: multipart/mixed; boundary="zSWldi6CowOrXAAR" Content-Disposition: inline --zSWldi6CowOrXAAR Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Feb 23, 2021 at 09:00:00AM +0300, Alexander Lakhin wrote: > I've found out that the list implementation doesn't support the > following usage pattern: > List *testList =3D NIL; > ListCell *testCell; >=20 > testList =3D lcons(NIL, testList); > testCell =3D list_head(testList); > ... > testList =3D lcons(NIL, testList); >=20 > elog(INFO, "lfirst(testCell): %p", lfirst(testCell)); // prints > 0x7f7f7f7f7f7f7f7f when compiled with -DUSE_VALGRIND >=20 > (Such list manipulation is happening in that makeDependencyGraphWalker > call.) Thanks for the reminder, I was not able to get around this issue. Anyway.. What's happening here is that the second lcons() call does new_head_cell() as testList is not NIL. This itself calls enlarge_list(), followed by wipe_mem() which invalidates the position of the list head previously stored. Oops. Coming back to the original problem, as you say there is some confusion about the list operation that we had better clarify. From what I can see cstate->innerwiths stores a List of Lists, and in each passage the code tries to build a new list appended to cstate->innerwiths. Using a ListCell to be able to track where the new list head is located is awkward on HEAD (the business with cell1), and there should be no need to keep around the reference to the first element innerwiths, as long as you save the result in a new, separate, List. The second case in checkWellFormedRecursionWalker() is equally dangerous. I have been playing with subselects and more recursions and could not trigger a problem with -DUSE_VALGRIND, but let's be safe. So attached is a patch to take care of this, with a regression test based on what has been sent upthread. This solves the issue for me. Thoughts? -- Michael --zSWldi6CowOrXAAR Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="valgrind-cte-fix.patch" Content-Transfer-Encoding: quoted-printable diff --git a/src/backend/parser/parse_cte.c b/src/backend/parser/parse_cte.c index f4f7041ead..8333fca510 100644 --- a/src/backend/parser/parse_cte.c +++ b/src/backend/parser/parse_cte.c @@ -730,16 +730,15 @@ makeDependencyGraphWalker(Node *node, CteState *cstat= e) * In the non-RECURSIVE case, query names are visible to the * WITH items after them and to the main query. */ - ListCell *cell1; + List *new_cte_list =3D NIL; =20 - cstate->innerwiths =3D lcons(NIL, cstate->innerwiths); - cell1 =3D list_head(cstate->innerwiths); + cstate->innerwiths =3D lcons(new_cte_list, cstate->innerwiths); foreach(lc, stmt->withClause->ctes) { CommonTableExpr *cte =3D (CommonTableExpr *) lfirst(lc); =20 (void) makeDependencyGraphWalker(cte->ctequery, cstate); - lfirst(cell1) =3D lappend((List *) lfirst(cell1), cte); + new_cte_list =3D lappend(new_cte_list, cte); } (void) raw_expression_tree_walker(node, makeDependencyGraphWalker, @@ -1006,16 +1005,15 @@ checkWellFormedRecursionWalker(Node *node, CteState= *cstate) * In the non-RECURSIVE case, query names are visible to the * WITH items after them and to the main query. */ - ListCell *cell1; + List *new_cte_list =3D NIL; =20 - cstate->innerwiths =3D lcons(NIL, cstate->innerwiths); - cell1 =3D list_head(cstate->innerwiths); + cstate->innerwiths =3D lcons(new_cte_list, cstate->innerwiths); foreach(lc, stmt->withClause->ctes) { CommonTableExpr *cte =3D (CommonTableExpr *) lfirst(lc); =20 (void) checkWellFormedRecursionWalker(cte->ctequery, cstate); - lfirst(cell1) =3D lappend((List *) lfirst(cell1), cte); + new_cte_list =3D lappend(new_cte_list, cte); } checkWellFormedSelectStmt(stmt, cstate); cstate->innerwiths =3D list_delete_first(cstate->innerwiths); diff --git a/src/test/regress/expected/with.out b/src/test/regress/expected= /with.out index c519a61c4f..9b04f8cba7 100644 --- a/src/test/regress/expected/with.out +++ b/src/test/regress/expected/with.out @@ -1852,6 +1852,21 @@ SELECT * FROM t; 10 (55 rows) =20 +WITH RECURSIVE j(x) AS ( + WITH outermost(x) AS ( + SELECT ( + WITH innermost as (SELECT 1) + SELECT * FROM innermost + ) + ) + SELECT * FROM outermost +) +SELECT * FROM j; + x=20 +--- + 1 +(1 row) + -- -- test WITH attached to intermediate-level set operation -- diff --git a/src/test/regress/sql/with.sql b/src/test/regress/sql/with.sql index f4ba0d8e39..72869b4bf8 100644 --- a/src/test/regress/sql/with.sql +++ b/src/test/regress/sql/with.sql @@ -871,6 +871,17 @@ WITH RECURSIVE t(j) AS ( ) SELECT * FROM t; =20 +WITH RECURSIVE j(x) AS ( + WITH outermost(x) AS ( + SELECT ( + WITH innermost as (SELECT 1) + SELECT * FROM innermost + ) + ) + SELECT * FROM outermost +) +SELECT * FROM j; + -- -- test WITH attached to intermediate-level set operation -- --zSWldi6CowOrXAAR-- --08mqMGmhDI7AmXgD Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEEG72nH6vTowiyblFKnvQgOdbyQH0FAmA13iwACgkQnvQgOdby QH1GZA/+KWQDjug/orBdx/GnK8fc2M877xPg1s2AxNPG2thyLVhksTWFa5KhfhhG LVaPftKhGFOlQMwq8avwk9bwXRS9ONmVVX0NXJkrKY7tYWb1vTF7vBrV16kNnLuf D1SK+KEoDkVWRwsPeqlffesS/mS1pwQX9SIVAFBg3QUny+d5j4+ZYCd+3ShfGrny rWzA59aJpima0xCYfbDShoTjOOUcII+fLf7xXVniM3WvA+NPg9vIF+x2YR2N0eUf Wy10eWkdFwoe/fgDQoKGH7apW2A7PEQNvjjvlM5uLajC1d3UCn3WbBSWXSZrnXFt eHBWa0xAXcgi+UVDtLoykAPWXWukuYXbce8blKggBjMXl5zdisQT6fhz0W9XmeWy 3y7jAyi6EkOGrJPCU8SecayxuTQok/5yDXqvcue5XD8n4R0eaACHLqNgBjjrOLla jpSxvmZIe1xW+c9K3x5F86VGhM/pv4OmUEupuRvZWES7eJzYUB5VXgZVelY1cKfe pHIPyehnrgCHBZrZ3pWFZpDA7brRrS/9VXist5ipC8dR99GSC2uQe1FsHzLBrH3l M92YyqcwLFQfW1JzWxdZSEMsCBglyGqndyd0O46wWo8lJy9+995NZb0BmKMtkQ/r 0mIFRPvTWmrAHB9vpWUn3FqrFQ+uyTnIF4EhO6/K/zKWUSa3LVc= =fo+E -----END PGP SIGNATURE----- --08mqMGmhDI7AmXgD--