pg.ddx.io  pgsql-bugs@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Michael Paquier <michael@paquier.xyz>
To: Alexander Lakhin <exclusion@gmail.com>
Cc: pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #16801: Invalid memory access on WITH RECURSIVE with nested WITHs
Date: Wed, 24 Feb 2021 14:03:40 +0900
Message-ID: <YDXeLBqjDeQQPGoO@paquier.xyz> (raw)
In-Reply-To: <eb358b3f-8272-d0a9-6645-983e7ad2bec4@gmail.com>
References: <16801-393c7922143eaa4d@postgresql.org>
	<X/FTDocQD3D7+aIJ@paquier.xyz>
	<eb358b3f-8272-d0a9-6645-983e7ad2bec4@gmail.com>

On Tue, Feb 23, 2021 at 09:00:00AM +0300, Alexander Lakhin wrote:
> I've found out that the list implementation doesn't support the
> following usage pattern:
> List *testList = NIL;
> ListCell *testCell;
> 
> testList = lcons(NIL, testList);
> testCell = list_head(testList);
> ...
> testList = lcons(NIL, testList);
> 
> elog(INFO, "lfirst(testCell): %p", lfirst(testCell)); // prints
> 0x7f7f7f7f7f7f7f7f when compiled with -DUSE_VALGRIND
> 
> (Such list manipulation is happening in that makeDependencyGraphWalker
> call.)

Thanks for the reminder, I was not able to get around this issue.
Anyway..  What's happening here is that the second lcons() call does
new_head_cell() as testList is not NIL.  This itself calls
enlarge_list(), followed by wipe_mem() which invalidates the position
of the list head previously stored.  Oops.

Coming back to the original problem, as you say there is some
confusion about the list operation that we had better clarify.  From
what I can see cstate->innerwiths stores a List of Lists, and in each
passage the code tries to build a new list appended to
cstate->innerwiths.  Using a ListCell to be able to track where the
new list head is located is awkward on HEAD (the business with cell1),
and there should be no need to keep around the reference to the first
element innerwiths, as long as you save the result in a new, separate,
List.

The second case in checkWellFormedRecursionWalker() is equally
dangerous.  I have been playing with subselects and more recursions
and could not trigger a problem with -DUSE_VALGRIND, but let's be
safe.

So attached is a patch to take care of this, with a regression test
based on what has been sent upthread.  This solves the issue for me.

Thoughts?
--
Michael

Attachments:

valgrind-cte-fix.patch (text/x-diff, 2.8K) download diff --git a/src/backend/parser/parse_cte.c b/src/backend/parser/parse_cte.c index f4f7041ead..8333fca510 100644 --- a/src/backend/parser/parse_cte.c +++ b/src/backend/parser/parse_cte.c @@ -730,16 +730,15 @@ makeDependencyGraphWalker(Node *node, CteState *cstate) * In the non-RECURSIVE case, query names are visible to the * WITH items after them and to the main query. */ - ListCell *cell1; + List *new_cte_list = NIL; - cstate->innerwiths = lcons(NIL, cstate->innerwiths); - cell1 = list_head(cstate->innerwiths); + cstate->innerwiths = lcons(new_cte_list, cstate->innerwiths); foreach(lc, stmt->withClause->ctes) { CommonTableExpr *cte = (CommonTableExpr *) lfirst(lc); (void) makeDependencyGraphWalker(cte->ctequery, cstate); - lfirst(cell1) = lappend((List *) lfirst(cell1), cte); + new_cte_list = lappend(new_cte_list, cte); } (void) raw_expression_tree_walker(node, makeDependencyGraphWalker, @@ -1006,16 +1005,15 @@ checkWellFormedRecursionWalker(Node *node, CteState *cstate) * In the non-RECURSIVE case, query names are visible to the * WITH items after them and to the main query. */ - ListCell *cell1; + List *new_cte_list = NIL; - cstate->innerwiths = lcons(NIL, cstate->innerwiths); - cell1 = list_head(cstate->innerwiths); + cstate->innerwiths = lcons(new_cte_list, cstate->innerwiths); foreach(lc, stmt->withClause->ctes) { CommonTableExpr *cte = (CommonTableExpr *) lfirst(lc); (void) checkWellFormedRecursionWalker(cte->ctequery, cstate); - lfirst(cell1) = lappend((List *) lfirst(cell1), cte); + new_cte_list = lappend(new_cte_list, cte); } checkWellFormedSelectStmt(stmt, cstate); cstate->innerwiths = list_delete_first(cstate->innerwiths); diff --git a/src/test/regress/expected/with.out b/src/test/regress/expected/with.out index c519a61c4f..9b04f8cba7 100644 --- a/src/test/regress/expected/with.out +++ b/src/test/regress/expected/with.out @@ -1852,6 +1852,21 @@ SELECT * FROM t; 10 (55 rows) +WITH RECURSIVE j(x) AS ( + WITH outermost(x) AS ( + SELECT ( + WITH innermost as (SELECT 1) + SELECT * FROM innermost + ) + ) + SELECT * FROM outermost +) +SELECT * FROM j; + x +--- + 1 +(1 row) + -- -- test WITH attached to intermediate-level set operation -- diff --git a/src/test/regress/sql/with.sql b/src/test/regress/sql/with.sql index f4ba0d8e39..72869b4bf8 100644 --- a/src/test/regress/sql/with.sql +++ b/src/test/regress/sql/with.sql @@ -871,6 +871,17 @@ WITH RECURSIVE t(j) AS ( ) SELECT * FROM t; +WITH RECURSIVE j(x) AS ( + WITH outermost(x) AS ( + SELECT ( + WITH innermost as (SELECT 1) + SELECT * FROM innermost + ) + ) + SELECT * FROM outermost +) +SELECT * FROM j; + -- -- test WITH attached to intermediate-level set operation --
signature.asc (application/pgp-signature, 832B) download

view thread (18+ messages)  latest in thread

Message-ID: <YDXeLBqjDeQQPGoO@paquier.xyz>
Permalink:  ../YDXeLBqjDeQQPGoO@paquier.xyz/
Also on:    postgresql.org/message-id/YDXeLBqjDeQQPGoO@paquier.xyz

 ·  · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-bugs@postgresql.org
  Cc: michael@paquier.xyz, exclusion@gmail.com, pgsql-bugs@lists.postgresql.org
  Subject: Re: BUG #16801: Invalid memory access on WITH RECURSIVE with nested WITHs
  In-Reply-To: <YDXeLBqjDeQQPGoO@paquier.xyz>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox