pg.ddx.io  pgsql-hackers@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Tomas Vondra <tomas.vondra@enterprisedb.com>
To: Dilip Kumar <dilipbalaut@gmail.com>
To: Amit Kapila <amit.kapila16@gmail.com>
Cc: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>
Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Cc: Andres Freund <andres@anarazel.de>
Cc: Petr Jelinek <petr.jelinek@enterprisedb.com>
Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
Subject: Re: Decoding speculative insert with toast leaks memory
Date: Fri, 28 May 2021 13:46:47 +0200
Message-ID: <f1342d07-a91c-f2e8-ef15-18a8ebfdb910@enterprisedb.com> (raw)
In-Reply-To: <CAFiTN-vHmMm=Mq8VBhOO5NgKe1As914rdo0uMNxQrwELkHRFPw@mail.gmail.com>
References: <CAExHW5sPKF-Oovx_qZe4p5oM6Dvof7_P+XgsNAViug15Fm99jA@mail.gmail.com>
	<CAA4eK1LSxfzn1qGNS3n7Dy9MAmyCsgmhnxG=51N50mJVMPqxrg@mail.gmail.com>
	<CAA4eK1Kr9j1fr-=BZBeP=3-N+D-NpapOiBw37f7ceXi4Kho0Ew@mail.gmail.com>
	<CAFiTN-vkOsxSbXX=54nCD2cKpEzJ8tR8LSmwTQADEwOBqoYqig@mail.gmail.com>
	<CAA4eK1K41Kg01k5AkU-uhzkike5G4VsGNTd7arpt8kG32GZb=A@mail.gmail.com>
	<CAFiTN-vHmMm=Mq8VBhOO5NgKe1As914rdo0uMNxQrwELkHRFPw@mail.gmail.com>



On 5/27/21 6:36 AM, Dilip Kumar wrote:
> On Thu, May 27, 2021 at 9:47 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>>
>> On Thu, May 27, 2021 at 9:40 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>>
>> True, but if you do this clean-up in ReorderBufferCleanupTXN then you
>> don't need to take care at separate places. Also, toast_hash is stored
>> in txn so it appears natural to clean it up in while releasing TXN.
> 
> Make sense, basically, IMHO we will have to do in TruncateTXN and
> ReturnTXN as attached?
> 

Yeah, I've been working on a fix over the last couple days (because of a
customer issue), and I ended up with the reset in ReorderBufferReturnTXN
too - it does solve the issue in the case I've been investigating.

I'm not sure the reset in ReorderBufferTruncateTXN is correct, though.
Isn't it possible that we'll need the TOAST data after streaming part of
the transaction? After all, we're not resetting invalidations, tuplecids
and snapshot either ... And we'll eventually clean it after the streamed
transaction gets commited (ReorderBufferStreamCommit ends up calling
ReorderBufferReturnTXN too).

I wonder if there's a way to free the TOASTed data earlier, instead of
waiting until the end of the transaction (as this patch does). But I
suspect it'd be way more complex, harder to backpatch, and destroying
the hash table is a good idea anyway.

Also, I think the "if (txn->toast_hash != NULL)" checks are not needed,
because it's the first thing ReorderBufferToastReset does.



regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company





view thread (66+ messages)  latest in thread

Message-ID: <f1342d07-a91c-f2e8-ef15-18a8ebfdb910@enterprisedb.com>
Permalink:  ../f1342d07-a91c-f2e8-ef15-18a8ebfdb910@enterprisedb.com/
Also on:    postgresql.org/message-id/f1342d07-a91c-f2e8-ef15-18a8ebfdb910@enterprisedb.com

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-hackers@postgresql.org
  Cc: tomas.vondra@enterprisedb.com, dilipbalaut@gmail.com, amit.kapila16@gmail.com, ashutosh.bapat.oss@gmail.com, andres@anarazel.de, petr.jelinek@enterprisedb.com, peter.eisentraut@2ndquadrant.com
  Subject: Re: Decoding speculative insert with toast leaks memory
  In-Reply-To: <f1342d07-a91c-f2e8-ef15-18a8ebfdb910@enterprisedb.com>
  References: <CAExHW5sPKF-Oovx_qZe4p5oM6Dvof7_P+XgsNAViug15Fm99jA@mail.gmail.com> <CAA4eK1LSxfzn1qGNS3n7Dy9MAmyCsgmhnxG=51N50mJVMPqxrg@mail.gmail.com> <CAA4eK1Kr9j1fr-=BZBeP=3-N+D-NpapOiBw37f7ceXi4Kho0Ew@mail.gmail.com> <CAFiTN-vkOsxSbXX=54nCD2cKpEzJ8tR8LSmwTQADEwOBqoYqig@mail.gmail.com> <CAA4eK1K41Kg01k5AkU-uhzkike5G4VsGNTd7arpt8kG32GZb=A@mail.gmail.com> <CAFiTN-vHmMm=Mq8VBhOO5NgKe1As914rdo0uMNxQrwELkHRFPw@mail.gmail.com> <f1342d07-a91c-f2e8-ef15-18a8ebfdb910@enterprisedb.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox