Message-ID: <ee8f7b0c-4a2b-4314-a9a6-b17c4c77ec57@gmail.com>
Date: Tue, 22 Jul 2025 16:01:15 +0200
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: Memory consumed by paths during partitionwise join planning
To: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>
Cc: David Rowley <dgrowleyml@gmail.com>,
 pgsql-hackers <pgsql-hackers@postgresql.org>
References: 
 <CAExHW5tUcVsBkq9qT=L5vYz4e-cwQNw=KAGJrtSyzOp3F=XacA@mail.gmail.com>
 <CAApHDvoDc4QwxMH4h_WRjJO4NK47L_gafv0oTpyjAkXKyXFz_A@mail.gmail.com>
 <CAExHW5u-X+7xj2udyXSxqPhqyLDz99o_Ev6YoPViZX5tT0Uquw@mail.gmail.com>
 <CAApHDvrc4UoP=E+zzVdQRgC0gJ37+W_zw=3wjnLkXJhvToZ50A@mail.gmail.com>
 <CAExHW5uDkMQL8SicV3_=AYcsWwMTNR8GzJo310J7sv7TMLoL6Q@mail.gmail.com>
 <CAExHW5s1ZX3HF2q9Tcmo9wxkfPp6pEdmjSySP1Whc=M5__mXrQ@mail.gmail.com>
 <CAExHW5uAdVev9CQYrdZ7w0xucqp2mYHPzcCx6EevegZVVgDbLQ@mail.gmail.com>
 <695ac356-87e6-45b3-a487-1e07b526ed52@postgrespro.ru>
 <3bcf826a-6d69-4232-b43b-d5894be8bd6f@gmail.com>
 <CAExHW5udVs3fNLgzsM+27HtW2CKBh-e_BNxQzXHfrB8ENcDUqQ@mail.gmail.com>
Content-Language: en-US
From: Andrei Lepikhov <lepihov@gmail.com>
In-Reply-To: 
 <CAExHW5udVs3fNLgzsM+27HtW2CKBh-e_BNxQzXHfrB8ENcDUqQ@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Archived-At: 
 <https://www.postgresql.org/message-id/ee8f7b0c-4a2b-4314-a9a6-b17c4c77ec57%40gmail.com>
Precedence: bulk

On 18/7/2025 13:48, Ashutosh Bapat wrote:
> On Mon, Jul 7, 2025 at 8:43 PM Andrei Lepikhov <lepihov@gmail.com> wrote:
> if (!IsA(new_path, IndexPath))
> - pfree(new_path);
> + free_path(new_path, 0, false);
> 
> Why don't we free the subpaths if they aren't referenced anymore?
During testing, I discovered that we sometimes encounter unusual cases. 
For example, imagine pathlist:
{SeqScan, IndexOnlyScan1, IndexScan2}
Someone may decide that Sort+SeqScan may be cheaper than IndexScan2. Or, 
IncrementalSort+IndexOnlyScan1 is cheaper than IndexScan2 ... And add 
this path to the same pathlist. I am unsure which exact combinations may 
arise in the planner, but without strict rules, someone may forget to 
increment the refcounter.

>> To address this complexity, I propose the following solutions:
>> 1. Localise reference management within the add_path function.
>> 2. Transition from a 'strict' approach, which removes all unused paths,
>> to a more straightforward 'pinning' method. In this approach, we would
>> simply mark a path as 'used' when someone who was added to the upper
>> path list references it. Removing less memory, we leave the code much
>> simpler.
> Yes. This was one of the ideas Tom had proposed earlier in another
> thread to manage paths better and avoid dangling pointers. May be it's
> worth starting with that first. Get rid of special handling of index
> paths and then improve the memory situation. However, even with that,
> I think we should free more paths than less.
It seems like one more trade-off: more eager cleaning means more 
resources spent.

P.S. path_walker makes sense to implement.

-- 
regards, Andrei Lepikhov