Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1o9W6j-00044l-GR for pgsql-hackers@arkaria.postgresql.org; Thu, 07 Jul 2022 18:20:09 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1o9W6h-0002PG-3r for pgsql-hackers@arkaria.postgresql.org; Thu, 07 Jul 2022 18:20:07 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1o9W6g-0002P7-Pj for pgsql-hackers@lists.postgresql.org; Thu, 07 Jul 2022 18:20:06 +0000 Received: from mail-oo1-xc29.google.com ([2607:f8b0:4864:20::c29]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1o9W6b-00076I-TD for pgsql-hackers@postgresql.org; Thu, 07 Jul 2022 18:20:05 +0000 Received: by mail-oo1-xc29.google.com with SMTP id r8-20020a4a3708000000b0042f9cb12e68so134761oor.2 for ; Thu, 07 Jul 2022 11:20:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=PGuavQOT6aYGLzFRgD9kKqvXlKb6FN2bdvHXEuu28C4=; b=TLKzLu14DeKklnvv7Q4UGs/w+MF9g95S6odSkz1hUNrft0nZXttIL0JtzaRok0m1w+ 72C3hcP/EENuGXoUOqiMKTGZFK6F+xGhbUsNaJNRZWMxpQqu9XTtPbHMtHWX2pSJ5YoW JGZQjHWmwn8Yo69YsY6pPlaiU+qp9uCEy+lzVgzG4k7s4m/g9UijS9h5snxh+h/S30dj K+8DqE8yP0NCpy3UswDNSW5OsV6bVJMgcOTYJNeHLvNXWC71kg9plFHrdzPeaHhnLSyZ eZXnFok9sm+AHRQBIvxS8dJwmdFgw1I1ybJZKYOcl1iVbG6MH/F/Vwx858letp5sEf8B 4b6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=PGuavQOT6aYGLzFRgD9kKqvXlKb6FN2bdvHXEuu28C4=; b=FuWqzEBU9PmNz/WSUK9iGN2V+MiRnO6uM9CjMfys/w3isTVo+E3iTpU21nkr64Bg2B toQq1OirpTUvQ2GUF8XTkAPIBgsOr+mCIp/yO93u/hatQdstww2H4B3nZRw1nwjV4NGV wHp9dyWEYqNvvxkgMJqxz19p0cVia2F8Fsxhdnel4KTVNvW4yhqjOwmA8+0jQwiU/cQq RLsAwbly7rThsxTwM0gH/gFDGgAAHfCMkhClDdRwBy7PMBFvzg+xVPCbK8OEZWTmVYsZ L0cXm3nKB9YFoN19RzA/tWqz7HZ59ntE8qg4lSbebgsU57580wAT52gOuzptaPYnEIQ9 5yzA== X-Gm-Message-State: AJIora/Hhq9phXrCQVgt+QXgUsfLmPSvIQ9/FBxa4zSJeXrif1VUQ3Xs 1uY4Xx8KhSRBxR7gHKSKAnU= X-Google-Smtp-Source: AGRyM1uha0e3J7Bnw1bWzotswwgcae6eEvaDTbfSD9ggq963UKZtViiRqhx0jEKqJ1NI32qDcIVfNw== X-Received: by 2002:a4a:d092:0:b0:425:6c2a:12 with SMTP id i18-20020a4ad092000000b004256c2a0012mr19547834oor.96.1657218000948; Thu, 07 Jul 2022 11:20:00 -0700 (PDT) Received: from nathanxps13 ([2600:380:c031:4fe1:d651:22c1:b379:2660]) by smtp.gmail.com with ESMTPSA id d13-20020a9d72cd000000b006167fb01b8fsm17666312otk.29.2022.07.07.11.19.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Jul 2022 11:20:00 -0700 (PDT) Date: Thu, 7 Jul 2022 11:19:56 -0700 From: Nathan Bossart To: David Steele Cc: Robert Haas , Fujii Masao , Kyotaro Horiguchi , "pgsql-hackers@postgresql.org" Subject: Re: remove more archiving overhead Message-ID: <20220707181956.GA2255157@nathanxps13> References: <20220222173711.GA3852671@nathanxps13> <20220222195229.GA42988@nathanxps13> <20220224.141349.1809640551882183816.horikyota.ntt@gmail.com> <20220224175553.GA660690@nathanxps13> <20220407222327.GA1363808@nathanxps13> <768e6cb3-256d-9c0b-1797-62420ffca7ae@oss.nttdata.com> <17147b74-5465-9a7a-ae27-305d1200482c@pgmasters.net> <20220707161825.GA2205004@nathanxps13> <58b2ae42-5378-778f-e198-3dcaf5ecc47a@pgmasters.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <58b2ae42-5378-778f-e198-3dcaf5ecc47a@pgmasters.net> List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On Thu, Jul 07, 2022 at 02:07:26PM -0400, David Steele wrote: > On 7/7/22 12:18, Nathan Bossart wrote: >> On Thu, Jul 07, 2022 at 10:46:23AM -0400, David Steele wrote: >> >> > There are plenty of ways that already-archived WAL might get archived again >> > and this is just one of them. >> >> What are some of the others? I was aware of the case that was fixed in >> ff9f111, where we might try to re-archive a file with different contents, >> but I'm curious what other ways you've seen this happen. > > On the PG side, crashes and (IIRC) immediate shutdown. > > In general, any failure in the archiver itself. Storage, memory, network, > etc. There are plenty of ways that the file might make it to storage but > postgres never gets notified, so it will retry. > > Any archiver that is not tolerant of this fact is not going to be very > useful and this patch only makes it slightly more true. Ah, got it, makes sense. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com