Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lmxsC-0003J5-Fh for pgsql-hackers@arkaria.postgresql.org; Sat, 29 May 2021 12:15:24 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1lmxsA-0000Yl-Ge for pgsql-hackers@arkaria.postgresql.org; Sat, 29 May 2021 12:15:22 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lmxsA-0000Yb-4T for pgsql-hackers@lists.postgresql.org; Sat, 29 May 2021 12:15:22 +0000 Received: from mail-ej1-x636.google.com ([2a00:1450:4864:20::636]) by magus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1lmxs2-0001dl-7I for pgsql-hackers@lists.postgresql.org; Sat, 29 May 2021 12:15:21 +0000 Received: by mail-ej1-x636.google.com with SMTP id ci15so5901844ejc.10 for ; Sat, 29 May 2021 05:15:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=enterprisedb-com.20150623.gappssmtp.com; s=20150623; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=RbsLoetxIGrkMkfFWxvnwmn60vh0zIr+/x/V1SKzRU8=; b=zj93VOGoAEh7qk1tV3gSqLp/jC83Ordt2mekMExjqx9MwYHNDc/dScRvYzKaXwFT3g r69zbpVIm+vBSk6G+klMOQTrQc0cAbv3W8Los58mNl4D/p8Qefj4StgrOkB2FPH0ffD0 iyfTxALcJTwjnXoMTYNzCb8ogOwoz4AfYbIxHPNXKGbyyvBC+fTXyR/u+01xh0ax4pPv +Ghtz53XJKEKXdj3Unw7ipfyja7mFDolyXtoX4N76cvB9olwK/2JZjb6JTsODZ8wt0p+ 1UDwP0G5ievZk1N5L50a2A1QRWCQq5D09C18RH+dL02pq/ZF/X7x9/0uZAslKZqrOMM1 uiGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=RbsLoetxIGrkMkfFWxvnwmn60vh0zIr+/x/V1SKzRU8=; b=tvyUVWndV2C/t9vHn4nXTorriDM5H4wM7KN8VYZcHiXbbGggDLBlg0GSSkKLaIYAAI TkAHgcdjw605w2A+1zYMysOfBqVrsK72wKFzrL64O3bE4kycfIjbCQvRSJZZjG8i4KY+ mQMsGZV9WluRAYqG9SX2MwWh11NARgiSwbZwLMCg8PmDa0jkXJLlgA0jC+/iqQqZz5IB obIhS+0iNNm+ndYYH3amZRgMl10TT8U9Sg2wsl6GvRK+ZJZHiaPmkoIFIwmZdp/KEE9i nwsJfiaJiaJYYnN8Gvztd1Bu3Fpy5pR6BILh4sjo7CnbTI9Wrggetaj6NWeKcIWkx2Qj hJlQ== X-Gm-Message-State: AOAM5323lxvISGqY7PfQ5H2xSx3Yr+k0vJDFkwoUGtAcfmCIqHaEuQGH 2nOeMutFzdOr5bbGnyV9FkEEAsQjWY9uHhesjIQFW15o9FKCeSLxmE2+kQc3yPVr2Ol/czlV4xB p2XyqeQQ3MYHnrw7qAdSZc5FEeFsrtGvl0Qz9dpa+HDukq2fxaSU6WDQ0GVIH5b5RNwd4hlzdiV Gk6ZTJXefSY/xept1ej38oDz/ZsDhijnZFTUML8u2IsjRQxWCxvIAK X-Google-Smtp-Source: ABdhPJzlfxWNdKlX4Zv/6LG986OPsbpqolfBppiW1J62jzHzP+4Hiw9xHvqVL7WFYR/gFQJYeITswA== X-Received: by 2002:a17:907:7848:: with SMTP id lb8mr14088438ejc.494.1622290512656; Sat, 29 May 2021 05:15:12 -0700 (PDT) Received: from [10.137.0.19] (ip-86-49-254-183.net.upcbroadband.cz. [86.49.254.183]) by smtp.gmail.com with ESMTPSA id gx23sm3601730ejb.125.2021.05.29.05.15.11 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sat, 29 May 2021 05:15:12 -0700 (PDT) Subject: Re: Decoding speculative insert with toast leaks memory To: Amit Kapila Cc: Dilip Kumar , Ashutosh Bapat , PostgreSQL Hackers , Andres Freund , Petr Jelinek , Peter Eisentraut References: From: Tomas Vondra Message-ID: <0d35243e-cf09-81a8-1dc3-0e52800732e9@enterprisedb.com> Date: Sat, 29 May 2021 14:15:10 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.10.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-CLOUD-SEC-AV-Info: enterprisedb,google_mail,monitor X-CLOUD-SEC-AV-Sent: true X-Gm-Spam: 0 X-Gm-Phishy: 0 List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On 5/29/21 6:29 AM, Amit Kapila wrote: > On Fri, May 28, 2021 at 5:16 PM Tomas Vondra > wrote: >> >> I wonder if there's a way to free the TOASTed data earlier, instead of >> waiting until the end of the transaction (as this patch does). >> > > IIUC we are anyway freeing the toasted data at the next > insert/update/delete. We can try to free at other change message types > like REORDER_BUFFER_CHANGE_MESSAGE but as you said that may make the > patch more complex, so it seems better to do the fix on the lines of > what is proposed in the patch. > +1 Even if we started doing what you mention (freeing the hash for other change types), we'd still need to do what the patch proposes because the speculative insert may be the last change in the transaction. For the other cases it works as a mitigation, so that we don't leak the memory forever. So let's get this committed, perhaps with a comment explaining that it might be possible to reset earlier if needed. regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company