Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1iUwRA-0006pq-Dm for pgsql-bugs@arkaria.postgresql.org; Wed, 13 Nov 2019 17:28:13 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1iUwR9-0005OV-3r for pgsql-bugs@arkaria.postgresql.org; Wed, 13 Nov 2019 17:28:11 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1iUwR8-0005OO-Ox for pgsql-bugs@lists.postgresql.org; Wed, 13 Nov 2019 17:28:10 +0000 Received: from mail-wm1-x343.google.com ([2a00:1450:4864:20::343]) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1iUwR5-0007uH-Ma for pgsql-bugs@lists.postgresql.org; Wed, 13 Nov 2019 17:28:10 +0000 Received: by mail-wm1-x343.google.com with SMTP id u18so2886784wmc.3 for ; Wed, 13 Nov 2019 09:28:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=2ndquadrant-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=xXlkG3UFxIQ/D3KRCRViHryU00uq2vyc2nO9ontFYB0=; b=CxfAdjJEW940sumU5WuLu0nys4fCkdv/NUqJGMdWeqlUdF44JqQSx5KnuL/CvmNnQn r+3prD/VHJHwWYJqxD8+6ZPQxXkimcehleanYdgR3jtAjjbCmQeM/DO7QDVKzsPMsSCT rt4MJDAEpSMkpJrBwhUORGoAkn0pUeCf62XMPW5hC+x4KOwM0m8R4TWm/oBFlbZ53vQY BpAJyyOl1JuaoEhh1jN/R12rME9vZ3apsCVq6RPKE2k2X+NhGoOo6Z7fo5MynzokYm0z 2fIM02NzTfQ7h7kEktZTSyDw1DiQ6dNOmUZFBeN6bU25RLmlvx8znkedqiSnHdUjTs4J fiCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=xXlkG3UFxIQ/D3KRCRViHryU00uq2vyc2nO9ontFYB0=; b=IhHwhMU7dqWhbmpxXYvCHGrXkZa3lOtpswFeE7d2U47mCsXpUD82e4LqyJEQj5kvRX tVTNKmO4eFx0Dh3AXTIqBZaeApxNBVFswnTUiJ2CLiH3VQEolrQgO03yqA19sUb61CIS 55BN6oeX0PMM9e0gLtvmhNuwnAr8CNfT5VJQzr5Y6AuVsthOK5dYDdP4b1nucPi+bK4j 0BZlwaSU1xSzF3eG+M2uip46AZNplwzxFwcdpRCbFPe6M0JKCU9UQsNufOl/wPz1vTiU 7jqr8x4LJibsf+zsLeV2IZdGyvjZAcpgNeyCaiJpNZu6+TGacLhHAsNZi6uCwnHjVNa7 r7Ig== X-Gm-Message-State: APjAAAXZFobPR+qZF21s4EllMkAOONX9PiYJJGqShnoVogK+8m89m894 DFf6lhCagAkvFkKcfdgaKJ/gtrRqrPM= X-Google-Smtp-Source: APXvYqzoKi4xCCTW7S2HymVtH78JI+NqG7+7SvXL66QxTY1CwPeKPG7cAWpQCZ3SBgZYYekijGxI1g== X-Received: by 2002:a7b:c3ce:: with SMTP id t14mr3655569wmj.22.1573666086285; Wed, 13 Nov 2019 09:28:06 -0800 (PST) Received: from localhost (ip-86-49-253-92.net.upcbroadband.cz. [86.49.253.92]) by smtp.gmail.com with ESMTPSA id 17sm2434836wmg.19.2019.11.13.09.28.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Nov 2019 09:28:05 -0800 (PST) Date: Wed, 13 Nov 2019 18:28:03 +0100 From: Tomas Vondra To: Andres Freund Cc: ben@lantern.is, pgsql-bugs@lists.postgresql.org Subject: Re: BUG #16112: large, unexpected memory consumption Message-ID: <20191113172803.iwyjaikvhdkvftee@development> References: <16112-4448bbf55a404189@postgresql.org> <20191113145004.opxx4vuajyoadbcq@development> <20191113170528.caywphoaq45ggqbx@alap3.anarazel.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: <20191113170528.caywphoaq45ggqbx@alap3.anarazel.de> User-Agent: NeoMutt/20180716 List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk On Wed, Nov 13, 2019 at 09:05:28AM -0800, Andres Freund wrote: >Hi, > >On 2019-11-13 15:50:04 +0100, Tomas Vondra wrote: >> Yeah, I can reproduce this pretty easily. It seems like a memory leak in >> ExecMakeTableFunctionResult. a9c35cf85ca reworked FunctionCallInfo to be >> variable-length, but it gets allocated in ExecutorState context directly >> and so until the end of the query. > >Damn. Too bad this got discovered just after the point release was >wrapped :( > > >> The attached trivial patch fixes that by adding a pfree() at the end of >> the function. > >Hm. That's clearly an improvement. But I'm not quite sure it's really >the right direction. It seems like a bad idea to rely on >ExecMakeTableFunctionResult() otherwise never leaking any memory. > >It seems to we should be using memory contexts to provide a backstop >against leaks, like we normally do, instead of operating in the >per-query context. It's noteworthy that nodeProjectSet already does so. >In contrast to nodeProjectSet, I think we'd need an additional memory >context however, as we eagerly evaluate ValuePerCall expressions - and >we'd like to reset the context after each iteration. > Possibly, but I think most of the allocations already use the per-tuple context. It's just the fcinfo that seems to be allocated outside of it, so maybe we should just move it to that memory context. > >I think we also ought to use SetExprState->fcinfo, instead of allocating >a separate allocation in ExecMakeTableFunctionResult(). But that's >perhaps less important. > Maybe. > >> I wonder if we have the same issue elsewhere ... > >Quite possible - but I think in most cases we are using memory contexts >to protect against leaks like this (which is more efficient than retail >pfree'ing). > Yeah, probably. I had a quick look at some of those places (found by looking for SizeForFunctionCallInfo and palloc on the same line) and those that I looked at seemed fine. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services