Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lmaxB-0008OW-43 for pgsql-hackers@arkaria.postgresql.org; Fri, 28 May 2021 11:47:01 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1lmax7-00069k-NW for pgsql-hackers@arkaria.postgresql.org; Fri, 28 May 2021 11:46:57 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lmax7-00069b-BU for pgsql-hackers@lists.postgresql.org; Fri, 28 May 2021 11:46:57 +0000 Received: from mail-ej1-x634.google.com ([2a00:1450:4864:20::634]) by magus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1lmax2-0006JJ-CO for pgsql-hackers@lists.postgresql.org; Fri, 28 May 2021 11:46:56 +0000 Received: by mail-ej1-x634.google.com with SMTP id ci15so1527621ejc.10 for ; Fri, 28 May 2021 04:46:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=enterprisedb-com.20150623.gappssmtp.com; s=20150623; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=a6ZfqX0Hl1teHk9sdrPI8DtFKl9QozQ6gz7F+GmpD5Q=; b=hRxHojCYSD1jzGgA55vEYiQiKaeZPEAk/69ok4NAV4Z+XX8g6gs59XZB9EmFvhNMKp 16oOC2Qj9r8Jh+LwxhrwY8YImIlpA8PuC/99DF8q8rNHz4pBsFCOJsukpxmInSJH4KwY 8sYL+KeelMoF2X3HmPZ+gya7uj5Xl/P/bdd2aF1ZODi9Ovv5Z1GY92ZLsFn3zq+T1ZDq no0Hp/YyhhP24WopxBRoptysUZ0C7yn0zSWQtIeVK58S8wHsUyaTFblOCCUKEAalDnJR XAYZxGDxlp6DA8qLYGHEw9fzlfOElqrntQKQj9hVDnvbWgeOU3G15hZIwP5z2PKy0gpr Jw8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=a6ZfqX0Hl1teHk9sdrPI8DtFKl9QozQ6gz7F+GmpD5Q=; b=d5o7T03lk1CUOCkdXLs/JBKk3DG6Bo2/XqIEgGPQ0DLUrsZmKU4kghw4tVLlawugWg RhkwV9roUIyRjLiL1xAu6SGSEPh3zEUJmCgVx2apvhlbeiEnef9Ej0oTdX5GdxT0c8ZO koDzqTjfjOzX1ebtNown2bVjfuTILtaqA6rYh5FqN3rWFD8HITgaAMXffv31EJcet5tT WXw4wbYOJIJP4BWF+u74hAhPHcR/LMLqZWlGMPFNcBUMHO+EnY9GbNTKn8/uXm4kdslZ ZrX2Pxn7sDia3++tzFlyRtsn4oDvjnJoPWfWFNYNSvc9Ef9898fXtdqw1X8bTM3VKH5q u1VA== X-Gm-Message-State: AOAM532OcunEx43Ae4xWv687tuLgcz4aaCR0pN/tq09PfwfgFxrv1kf/ eQFF3grrORfQpmUhbLnERYRMBeLQE/WUJWNxD/cRVkHvYq50FExc6sAvRaykA2C8aRX6zXx+JqK ZoQP73a3suF6UR/5mM+dGmTCvsA25AUB53rwZQ9gdgLM+bHKb2c7rSRHoGKwDMmqu9GBb+qe66+ EYbKckx4DJzrJ5ObWXgYx3OFuTdY7R1DadG7FKELxJnVs2CdIrCU9r X-Google-Smtp-Source: ABdhPJznT434AJlkaZ46NGs/78HcIOQUtJVgXlv2fy+1Uij7RWyGIz2d70rVuFZRSCTtvU6P/c1zfQ== X-Received: by 2002:a17:907:1b02:: with SMTP id mp2mr8581990ejc.196.1622202410193; Fri, 28 May 2021 04:46:50 -0700 (PDT) Received: from [10.137.0.19] (ip-86-49-254-183.net.upcbroadband.cz. [86.49.254.183]) by smtp.gmail.com with ESMTPSA id q18sm2486685edd.3.2021.05.28.04.46.49 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 28 May 2021 04:46:49 -0700 (PDT) Subject: Re: Decoding speculative insert with toast leaks memory To: Dilip Kumar , Amit Kapila Cc: Ashutosh Bapat , PostgreSQL Hackers , Andres Freund , Petr Jelinek , Peter Eisentraut References: From: Tomas Vondra Message-ID: Date: Fri, 28 May 2021 13:46:47 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.10.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-CLOUD-SEC-AV-Info: enterprisedb,google_mail,monitor X-CLOUD-SEC-AV-Sent: true X-Gm-Spam: 0 X-Gm-Phishy: 0 List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On 5/27/21 6:36 AM, Dilip Kumar wrote: > On Thu, May 27, 2021 at 9:47 AM Amit Kapila wrote: >> >> On Thu, May 27, 2021 at 9:40 AM Dilip Kumar wrote: >> >> True, but if you do this clean-up in ReorderBufferCleanupTXN then you >> don't need to take care at separate places. Also, toast_hash is stored >> in txn so it appears natural to clean it up in while releasing TXN. > > Make sense, basically, IMHO we will have to do in TruncateTXN and > ReturnTXN as attached? > Yeah, I've been working on a fix over the last couple days (because of a customer issue), and I ended up with the reset in ReorderBufferReturnTXN too - it does solve the issue in the case I've been investigating. I'm not sure the reset in ReorderBufferTruncateTXN is correct, though. Isn't it possible that we'll need the TOAST data after streaming part of the transaction? After all, we're not resetting invalidations, tuplecids and snapshot either ... And we'll eventually clean it after the streamed transaction gets commited (ReorderBufferStreamCommit ends up calling ReorderBufferReturnTXN too). I wonder if there's a way to free the TOASTed data earlier, instead of waiting until the end of the transaction (as this patch does). But I suspect it'd be way more complex, harder to backpatch, and destroying the hash table is a good idea anyway. Also, I think the "if (txn->toast_hash != NULL)" checks are not needed, because it's the first thing ReorderBufferToastReset does. regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company