Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1o9Vub-0003ZY-QU for pgsql-hackers@arkaria.postgresql.org; Thu, 07 Jul 2022 18:07:37 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1o9Vua-00038n-EU for pgsql-hackers@arkaria.postgresql.org; Thu, 07 Jul 2022 18:07:36 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1o9Vua-00033D-4r for pgsql-hackers@lists.postgresql.org; Thu, 07 Jul 2022 18:07:36 +0000 Received: from mail.thelabyrinth.net ([45.56.70.56]) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1o9VuT-0001Fm-Qg for pgsql-hackers@postgresql.org; Thu, 07 Jul 2022 18:07:35 +0000 Received: from [172.20.20.20] (c-24-126-12-194.hsd1.wv.comcast.net [24.126.12.194]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: dsteele) by mail.thelabyrinth.net (Postfix) with ESMTPSA id 6123D5466A; Thu, 7 Jul 2022 18:07:27 +0000 (UTC) Message-ID: <58b2ae42-5378-778f-e198-3dcaf5ecc47a@pgmasters.net> Date: Thu, 7 Jul 2022 14:07:26 -0400 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Thunderbird/91.11.0 Subject: Re: remove more archiving overhead Content-Language: en-US To: Nathan Bossart Cc: Robert Haas , Fujii Masao , Kyotaro Horiguchi , "pgsql-hackers@postgresql.org" References: <20220222011948.GA3850532@nathanxps13> <20220222173711.GA3852671@nathanxps13> <20220222195229.GA42988@nathanxps13> <20220224.141349.1809640551882183816.horikyota.ntt@gmail.com> <20220224175553.GA660690@nathanxps13> <20220407222327.GA1363808@nathanxps13> <768e6cb3-256d-9c0b-1797-62420ffca7ae@oss.nttdata.com> <17147b74-5465-9a7a-ae27-305d1200482c@pgmasters.net> <20220707161825.GA2205004@nathanxps13> From: David Steele In-Reply-To: <20220707161825.GA2205004@nathanxps13> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On 7/7/22 12:18, Nathan Bossart wrote: > On Thu, Jul 07, 2022 at 10:46:23AM -0400, David Steele wrote: > >> There are plenty of ways that already-archived WAL might get archived again >> and this is just one of them. > > What are some of the others? I was aware of the case that was fixed in > ff9f111, where we might try to re-archive a file with different contents, > but I'm curious what other ways you've seen this happen. On the PG side, crashes and (IIRC) immediate shutdown. In general, any failure in the archiver itself. Storage, memory, network, etc. There are plenty of ways that the file might make it to storage but postgres never gets notified, so it will retry. Any archiver that is not tolerant of this fact is not going to be very useful and this patch only makes it slightly more true. Regards, -David