Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1jNKz6-0005ZL-B8 for pgsql-bugs@arkaria.postgresql.org; Sat, 11 Apr 2020 18:36:04 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1jNKz3-0005h5-LG for pgsql-bugs@arkaria.postgresql.org; Sat, 11 Apr 2020 18:36:01 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1jNKz3-0005gy-75 for pgsql-bugs@lists.postgresql.org; Sat, 11 Apr 2020 18:36:01 +0000 Received: from mail-wm1-x342.google.com ([2a00:1450:4864:20::342]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1jNKyz-0007QP-LF for pgsql-bugs@lists.postgresql.org; Sat, 11 Apr 2020 18:35:59 +0000 Received: by mail-wm1-x342.google.com with SMTP id h2so5545960wmb.4 for ; Sat, 11 Apr 2020 11:35:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=2ndquadrant-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=Jqm1mcMcFalMjHt/WEpu0eA6Sv7ZMU1sY3Av7APO0Xc=; b=gjkvOZ0MtTVi8cR1AhY6OkGeg1UIH6PEpXBLZiIJx4yAEJ+kICNHLzLSeJfFJsYzIr u6F/sQwaSpmSz0IXqXYmt1rhukx1czlEpS0cUuN2W1ddGPwV0DU+z0PXdhpSL8uyeLrz xnhqZQ4Iss1615c1lEVthc/o+1be+/0KsYN8gmgiemGUt7Dn7tkWGQE114A5wfcTlc5/ ZmbSD1CrKVk+mwi4AzxLC2b7/wu6x6LGJygovpm8hFmVxl1HYYfdYXEdTIpqv+Z/nbvs OlNqpB3csDNRBphZHIZThngFxNuK+MGfh1A+lw8OE+sQ9unjpd9NRn5O7JfKGB8QpsI4 we5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=Jqm1mcMcFalMjHt/WEpu0eA6Sv7ZMU1sY3Av7APO0Xc=; b=WJsRebQ8ofea4VnGrF7v3kSSUxWJdGzU8WkU6l4EbXbFH35+qj1uzROGnx1GFPgdL/ GFMQNRQJf7yAz6UZ0yWWzAc8GakaeMlFbSB6h5Bc/wwaphzQzdrtIRINJPmXAxhz/q0f qt6m3V9NzfcM/1IkCWv/tjv8o5/IWhzef980fTvCv2JD9FLswByhCA1LLkdrdE0uoQ/i 7SJAWR8e+81D1Kv2AI9kTp/8qGXX9GvP5KSTsrLlV8VrBQcAi6bCox20CuzwsD+SiXLa PXR3LJ+vgAF5agRGOSE3aZXl3sF60/hKHidq6YDkJpFwvUuN+DAo05Ap8Bw+fl2jvmDX aQ4w== X-Gm-Message-State: AGi0PuZ/4PH52XIyys7oIrJD2Zcvr1nmA31kfCfGijR6rWKd/gLpuGYP 7fwuvHas8tz+3hpPBZXZTNqVi5HTNKvp7B6UHiD9QQbyXQjPyzuGbenG02upISZJOoJfVv/NZBT CAUWGw2LreClMiofghaI2qTBlKpwufdjYy1dU46aadnWOFTtI77XHV6LpXUqrquMUXg02BmnB5c UiaeO3HStK7pNVlDE= X-Google-Smtp-Source: APiQypJnqNph1n5ykspJsq/XSQRWJNPQ4p61cFDYdNDkUIjhIIQnyXSDKw3fwhMoZu5aFz3qIF1nHw== X-Received: by 2002:a1c:2e0a:: with SMTP id u10mr10741663wmu.119.1586630155708; Sat, 11 Apr 2020 11:35:55 -0700 (PDT) Received: from localhost (ip-86-49-253-141.net.upcbroadband.cz. [86.49.253.141]) by smtp.gmail.com with ESMTPSA id h3sm2837238wrm.73.2020.04.11.11.35.54 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 11 Apr 2020 11:35:54 -0700 (PDT) Date: Sat, 11 Apr 2020 20:35:52 +0200 From: Tomas Vondra To: pgsql-bugs@lists.postgresql.org Subject: Re: BUG #16112: large, unexpected memory consumption Message-ID: <20200411183552.u64s3w4nzvroiqxj@development> References: <16112-4448bbf55a404189@postgresql.org> <20191113145004.opxx4vuajyoadbcq@development> <20191113170528.caywphoaq45ggqbx@alap3.anarazel.de> <368.1573665707@sss.pgh.pa.us> <20191113174733.zwfukrx7atfh65t3@alap3.anarazel.de> <4328.1573668211@sss.pgh.pa.us> <20191115024707.y5bcsb4oo4wzovu4@alap3.anarazel.de> <20200409122154.GA90424@shadowdale.lore.is> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: <20200409122154.GA90424@shadowdale.lore.is> List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk On Thu, Apr 09, 2020 at 08:21:54AM -0400, Ben Cornett wrote: >Hi, > >Did this patch ever get applied? It is not clear to me that it did. > Hmmm, I don't think it got applied, which means we've missed yet another minor release :-( Andres, any plans to push this? regards >Thanks! > >On Thu, Nov 14, 2019 at 06:47:07PM -0800, Andres Freund wrote: >> Hi, >> >> On 2019-11-13 13:03:31 -0500, Tom Lane wrote: >> > Andres Freund writes: >> > > Thinking about it for a second longer, I don't think we'd need a new >> > > context - afaict argcontext has exactly the lifetime requirements >> > > needed. >> > >> > Hm, maybe so. That'd definitely be a better solution if it works. >> >> Here's a patch doing so. I think it'd be a good idea to rename >> argcontext into something like setcontext or such. But as I'm not that >> happy with the name, I've not yet made that change. >> >> Greetings, >> >> Andres Freund > >> diff --git i/src/backend/executor/execSRF.c w/src/backend/executor/execSRF.c >> index c8a3efc3654..85f33054265 100644 >> --- i/src/backend/executor/execSRF.c >> +++ w/src/backend/executor/execSRF.c >> @@ -114,10 +114,23 @@ ExecMakeTableFunctionResult(SetExprState *setexpr, >> ReturnSetInfo rsinfo; >> HeapTupleData tmptup; >> MemoryContext callerContext; >> - MemoryContext oldcontext; >> bool first_time = true; >> >> - callerContext = CurrentMemoryContext; >> + /* >> + * Execute per-tablefunc actions in appropriate context. >> + * >> + * The FunctionCallInfo needs to live across all the calls to a >> + * ValuePerCall function, so it can't be allocated in the per-tuple >> + * context. Similarly, the function arguments need to be evaluated in a >> + * context that is longer lived than the per-tuple context: The argument >> + * values would otherwise disappear when we reset that context in the >> + * inner loop. As the caller's CurrentMemoryContext is typically a >> + * query-lifespan context, we don't want to leak memory there. We require >> + * the caller to pass a separate memory context that can be used for this, >> + * and can be reset each time through to avoid bloat. >> + */ >> + MemoryContextReset(argContext); >> + callerContext = MemoryContextSwitchTo(argContext); >> >> funcrettype = exprType((Node *) setexpr->expr); >> >> @@ -164,20 +177,9 @@ ExecMakeTableFunctionResult(SetExprState *setexpr, >> setexpr->fcinfo->fncollation, >> NULL, (Node *) &rsinfo); >> >> - /* >> - * Evaluate the function's argument list. >> - * >> - * We can't do this in the per-tuple context: the argument values >> - * would disappear when we reset that context in the inner loop. And >> - * the caller's CurrentMemoryContext is typically a query-lifespan >> - * context, so we don't want to leak memory there. We require the >> - * caller to pass a separate memory context that can be used for this, >> - * and can be reset each time through to avoid bloat. >> - */ >> - MemoryContextReset(argContext); >> - oldcontext = MemoryContextSwitchTo(argContext); >> + /* evaluate the function's argument list */ >> + Assert(CurrentMemoryContext == argContext); >> ExecEvalFuncArgs(fcinfo, setexpr->args, econtext); >> - MemoryContextSwitchTo(oldcontext); >> >> /* >> * If function is strict, and there are any NULL arguments, skip >> @@ -217,7 +219,7 @@ ExecMakeTableFunctionResult(SetExprState *setexpr, >> CHECK_FOR_INTERRUPTS(); >> >> /* >> - * reset per-tuple memory context before each call of the function or >> + * Reset per-tuple memory context before each call of the function or >> * expression. This cleans up any local memory the function may leak >> * when called. >> */ >> @@ -257,6 +259,8 @@ ExecMakeTableFunctionResult(SetExprState *setexpr, >> */ >> if (first_time) >> { >> + MemoryContext oldcontext; >> + >> oldcontext = MemoryContextSwitchTo(econtext->ecxt_per_query_memory); >> tupstore = tuplestore_begin_heap(randomAccess, false, work_mem); >> rsinfo.setResult = tupstore; >> @@ -285,6 +289,8 @@ ExecMakeTableFunctionResult(SetExprState *setexpr, >> >> if (tupdesc == NULL) >> { >> + MemoryContext oldcontext; >> + >> /* >> * This is the first non-NULL result from the >> * function. Use the type info embedded in the >> @@ -378,15 +384,18 @@ no_function_result: >> */ >> if (rsinfo.setResult == NULL) >> { >> - MemoryContextSwitchTo(econtext->ecxt_per_query_memory); >> + MemoryContext oldcontext; >> + >> + oldcontext = MemoryContextSwitchTo(econtext->ecxt_per_query_memory); >> tupstore = tuplestore_begin_heap(randomAccess, false, work_mem); >> rsinfo.setResult = tupstore; >> + MemoryContextSwitchTo(oldcontext); >> + >> if (!returnsSet) >> { >> int natts = expectedDesc->natts; >> bool *nullflags; >> >> - MemoryContextSwitchTo(econtext->ecxt_per_tuple_memory); >> nullflags = (bool *) palloc(natts * sizeof(bool)); >> memset(nullflags, true, natts * sizeof(bool)); >> tuplestore_putvalues(tupstore, expectedDesc, NULL, nullflags); > > > -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services