Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1nczRi-0000SV-UE for pgsql-hackers@arkaria.postgresql.org; Sat, 09 Apr 2022 00:59:23 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1nczRf-000529-WF for pgsql-hackers@arkaria.postgresql.org; Sat, 09 Apr 2022 00:59:20 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1nczRd-000520-Uw for pgsql-hackers@lists.postgresql.org; Sat, 09 Apr 2022 00:59:19 +0000 Received: from wout2-smtp.messagingengine.com ([64.147.123.25]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1nczRa-0007oR-F7 for pgsql-hackers@lists.postgresql.org; Sat, 09 Apr 2022 00:59:16 +0000 Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.west.internal (Postfix) with ESMTP id 44CA33202104; Fri, 8 Apr 2022 20:59:12 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute5.internal (MEProxy); Fri, 08 Apr 2022 20:59:12 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=anarazel.de; h= cc:cc:content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to; s=fm1; bh=XvJlBvpuKBVHcyXBafxerifOu2Osp/XYs9pR// Oe4JE=; b=jZjRW1XGGzz8LJoOdgkzO7Xz1IXOqRRX0jtpxnlRV74PNqJgxswVRx 3J1CZOkHPe+CBRpJP9G3wjC7oIrRrH8GOyKD1GmxV0iQnmCVtNtA0tgBy5ydylCL 3H8fumNpYqa0uZLzDFS/81n0+gb06zZ/0Qr9heLiJTCJFsDuEUigNKhYAIye9tRO K2mIPv3UkY5a2h6KVffqXZk1WPj9GZ4FJCZfHCENp3MWI9s/VzRxwI2V6qCR9XXU XvAPOUXU7HloD6eq6M6MCmqoa27y+8F/MDWSPx2Rm+1S0voHYvPsNqtJSShOPtSE SOoC5kyFqsdWC+BlH4AlVleQ1peVgcOw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=XvJlBvpuKBVHcyXBa fxerifOu2Osp/XYs9pR//Oe4JE=; b=fHzVxlsgI1y4Lh/Ex30jnNvZyOJzC8F8e edy5zMA9PQVMy0z+W3aiq4SIGHNjUF4pj29UW7Q7dGtuHJAR4xfv8zvKa3SAFJsO 9mULHX6jHUUG4HpkC9n2tWYs5W2hMt8Y91Ku8hUDG473mdS/2ogMTE8Zvu2Ke+k8 eW7u3bD4E1xVmEp/rz5ExyU/7rZoXFJnUwTjALOHP6/TESjEBDY/WbuKQYeKUSRi SPKmBMBs7W8aazbslroWRH2W2y+kn1jxHsV+pCP1Je9HJYr/5M/+GNaX24ZG7eBB aoVPTreRLzhAOzfXr6KQ/NhRGWYPl3MZhJNSWpb/LbyvzZTnBumhw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddrudekuddgfeejucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvffukfhfgggtuggjsehttdertddttddvnecuhfhrohhmpeetnhgurhgv shcuhfhrvghunhguuceorghnughrvghssegrnhgrrhgriigvlhdruggvqeenucggtffrrg htthgvrhhnpedukefhkeelueegveetheelffffjeegleeuudelfeefuedtleffueejfffh ueffudenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpe grnhgurhgvshesrghnrghrrgiivghlrdguvg X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 8 Apr 2022 20:59:11 -0400 (EDT) Date: Fri, 8 Apr 2022 17:59:10 -0700 From: Andres Freund To: Tom Lane Cc: Michael Paquier , Thomas Munro , pgsql-hackers@lists.postgresql.org Subject: Re: pgsql: Add TAP test for archive_cleanup_command and recovery_end_comman Message-ID: <20220409005910.alw46xqmmgny2sgr@alap3.anarazel.de> References: <3385143.1649353230@sss.pgh.pa.us> <20220407175210.q44nnrvkovprxo2a@alap3.anarazel.de> <3386734.1649354265@sss.pgh.pa.us> <20220407181202.7xa573u6e2z5dbdz@alap3.anarazel.de> <4015413.1649454951@sss.pgh.pa.us> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4015413.1649454951@sss.pgh.pa.us> List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk Hi, On 2022-04-08 17:55:51 -0400, Tom Lane wrote: > I tried adjusting the patch so it does guarantee that (as attached), > and in two out of two tries it got past the archive_cleanup_command > failure but then hung up waiting for standby2 to promote. Adding $node_standby->safe_psql('postgres', "SELECT pg_switch_wal()"); just after $node_standby2->start; makes the tests pass here. What is that second test really testing? # Check the presence of temporary files specifically generated during # archive recovery. To ensure the presence of the temporary history # file, switch to a timeline large enough to allow a standby to recover # a history file from an archive. As this requires at least two timeline # switches, promote the existing standby first. Then create a second # standby based on the promoted one. Finally, the second standby is # promoted. Note "Then create a second standby based on the promoted one." - but that's not actually what's happening: $node_standby2->init_from_backup($node_primary, $backup_name, has_restoring => 1); It's created from the original primary, not the first standby, as the description says... Both nodes get promoted independently, in a run without valgrind: standby: 2022-04-08 17:23:42.966 PDT [2463835][startup][1/0:0][] LOG: INSERT @ 0/4000058: - XLOG/END_OF_RECOVERY: tli 2; prev tli 1; time 2022-04-08 17:23:42.96686-07 2022-04-08 17:23:42.966 PDT [2463835][startup][1/0:0][] LOG: xlog flush request 0/4000058; write 0/4000000; flush 0/4000000 standby2: 2022-04-08 17:23:43.307 PDT [2463999][startup][1/0:0][] LOG: INSERT @ 0/4000058: - XLOG/END_OF_RECOVERY: tli 3; prev tli 1; time 2022-04-08 17:23:43.307443-> 2022-04-08 17:23:43.307 PDT [2463999][startup][1/0:0][] LOG: xlog flush request 0/4000058; write 0/4000000; flush 0/4000000 except that standby2 can't choose tli 2 because it finds it used. Sure looks like something is funky with that test. But I think there's also something funky in the prefetching logic. I think it may attempt restoring during prefetching somehow, even though there's code that appears to try to prevent that? on standby2 I can see replay progress like the following: 2022-04-08 17:02:12.310 PDT [2441453][startup][1/0:0][] LOG: REDO @ 0/3024488; LSN 0/30244C8: prev 0/3024448; xid 725; len 3; blkref #0: rel 1663/5/16384, blk 4 - Heap/INSERT: off 60 flags 0x00 2022-04-08 17:02:12.311 PDT [2441453][startup][1/0:0][] DEBUG: record known xact 725 latestObservedXid 725 2022-04-08 17:02:12.311 PDT [2441453][startup][1/0:0][] CONTEXT: WAL redo at 0/3024488 for Heap/INSERT: off 60 flags 0x00; blkref #0: rel 1663/5/16384, blk 4 2022-04-08 17:02:12.312 PDT [2441453][startup][1/0:0][] DEBUG: executing restore command "cp "/home/andres/build/postgres/dev-assert/vpath/src/test/recovery/tmp_check/t_002_archiving_primary_data/archives/000000010000000000000004" "pg_wal/RECOVERYXLOG"" 2022-04-08 17:02:13.855 PDT [2441453][startup][1/0:0][] DEBUG: could not restore file "000000010000000000000004" from archive: child process exited with exit code 1 2022-04-08 17:02:13.855 PDT [2441453][startup][1/0:0][] DEBUG: could not open file "pg_wal/000000010000000000000004": No such file or directory 2022-04-08 17:02:13.856 PDT [2441453][startup][1/0:0][] LOG: REDO @ 0/30244C8; LSN 0/3024508: prev 0/3024488; xid 725; len 3; blkref #0: rel 1663/5/16384, blk 4 - Heap/INSERT: off 61 flags 0x00 2022-04-08 17:02:13.856 PDT [2441453][startup][1/0:0][] DEBUG: record known xact 725 latestObservedXid 725 2022-04-08 17:02:13.856 PDT [2441453][startup][1/0:0][] CONTEXT: WAL redo at 0/30244C8 for Heap/INSERT: off 61 flags 0x00; blkref #0: rel 1663/5/16384, blk 4 2022-04-08 17:02:13.857 PDT [2441453][startup][1/0:0][] DEBUG: executing restore command "cp "/home/andres/build/postgres/dev-assert/vpath/src/test/recovery/tmp_check/t_002_archiving_primary_data/archives/000000010000000000000004" "pg_wal/RECOVERYXLOG"" 2022-04-08 17:02:15.413 PDT [2441453][startup][1/0:0][] DEBUG: could not restore file "000000010000000000000004" from archive: child process exited with exit code 1 2022-04-08 17:02:15.413 PDT [2441453][startup][1/0:0][] DEBUG: could not open file "pg_wal/000000010000000000000004": No such file or directory 2022-04-08 17:02:15.414 PDT [2441453][startup][1/0:0][] LOG: REDO @ 0/3024508; LSN 0/3024548: prev 0/30244C8; xid 725; len 3; blkref #0: rel 1663/5/16384, blk 4 - Heap/INSERT: off 62 flags 0x00 2022-04-08 17:02:15.414 PDT [2441453][startup][1/0:0][] DEBUG: record known xact 725 latestObservedXid 725 2022-04-08 17:02:15.414 PDT [2441453][startup][1/0:0][] CONTEXT: WAL redo at 0/3024508 for Heap/INSERT: off 62 flags 0x00; blkref #0: rel 1663/5/16384, blk 4 2022-04-08 17:02:15.415 PDT [2441453][startup][1/0:0][] DEBUG: executing restore command "cp "/home/andres/build/postgres/dev-assert/vpath/src/test/recovery/tmp_check/t_002_archiving_primary_data/archives/000000010000000000000004" "pg_wal/RECOVERYXLOG"" note that we appear to wait between replaying of records, even though we apparently have WAL for the next record! And interestingly I'm not seeing the "switched WAL source from stream to archive after failure" lines I'd expect. Greetings, Andres Freund