Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1o5io1-0001EO-O8 for pgsql-hackers@arkaria.postgresql.org; Mon, 27 Jun 2022 07:05:10 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1o5inz-0001ve-Ly for pgsql-hackers@arkaria.postgresql.org; Mon, 27 Jun 2022 07:05:07 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1o5inz-0001tR-27 for pgsql-hackers@lists.postgresql.org; Mon, 27 Jun 2022 07:05:07 +0000 Received: from mail-oa1-x34.google.com ([2001:4860:4864:20::34]) by magus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1o5inv-0005DA-Gs for pgsql-hackers@postgresql.org; Mon, 27 Jun 2022 07:05:06 +0000 Received: by mail-oa1-x34.google.com with SMTP id 586e51a60fabf-f2a4c51c45so11831906fac.9 for ; Mon, 27 Jun 2022 00:05:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=leadboat.com; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to :user-agent; bh=4IiWWk7Z62jHgCdeGFUvWh41MHP0rK+eouQTZDanHkk=; b=IwWikzXBgerPjkPzc7FrAcjv7nRxYd8KsX64/Zrb7zzFYVriPl4QB781Qofd34Zel9 TLw9Wpk6xqkZ8b9DXtfvuCaZUo6qMZ+IVnh5lhO76+dgSqxQ+JmAKk3wo3GgkCawy1Ds 3bp5CUkrJnFz4gFc9J3vxNPDD3i1DxL6KA0Jk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to:user-agent; bh=4IiWWk7Z62jHgCdeGFUvWh41MHP0rK+eouQTZDanHkk=; b=S2rO9oMnmd6GH5UjesEnkfbEEMbV0lWaR6kNd79XmEPeVssyCDWALUhcv2OE3/8PJf ZocZhu6u5+Mi6MZZSg8XmV8uoDoB2CcLHDlRNysuj3+wPJzppdF5Ca+7CEzH1PVao/bQ wxlG1sjsB4HDHAzF+OASk5ckmXcbJGkqv8BaW0+6EQzITaZiDWM1wYFtElfOdj16w62G D5dzcdKJd18XGBa2/c1abpHp3ku+SCmde0xpafDPeocmQbMm1LO6ZPG1jR4bBvoSfvRD X6zUnGDrRF3BOJpMI9hOxufP9t8LfjtF0gNdahs0RP7YxEhY6Ct9R1De7fl3eWZvTFaj PIAQ== X-Gm-Message-State: AJIora9Co7Gv/qiV6ruP6oEhFx62k+O6VxawNU/rs2D0aTQBbPQKKrqp pnB4nvliNKf/x/YSpnmJlCs4DBvMECSZAA== X-Google-Smtp-Source: AGRyM1viqvvk9cP6zvkxujt/PzD1d7g1WRPjHouoIZOsyWehC1wvcjvOcgm9M++0CiaFjDc/OuYTMQ== X-Received: by 2002:a05:6870:a117:b0:101:ca1c:97be with SMTP id m23-20020a056870a11700b00101ca1c97bemr9221868oae.227.1656313501531; Mon, 27 Jun 2022 00:05:01 -0700 (PDT) Received: from rfd.leadboat.com ([2600:1702:a20:5750::2e]) by smtp.gmail.com with ESMTPSA id w10-20020a056830078a00b006060322125esm5871915ots.46.2022.06.27.00.04.59 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 27 Jun 2022 00:05:00 -0700 (PDT) Date: Mon, 27 Jun 2022 00:04:57 -0700 From: Noah Misch To: Heikki Linnakangas Cc: Tom Lane , pgsql-hackers Subject: Re: last_archived_wal is not necessary the latest WAL file (was Re: pgsql: Add test case for an archive recovery corner case.) Message-ID: <20220627070457.GA2176699@rfd.leadboat.com> References: <2842251.1644849700@sss.pgh.pa.us> <8f15e602-337e-fbb2-4b2e-e83a01ef9088@iki.fi> <831b6ab9-45d6-7ab0-ea9c-6efd4383afb8@iki.fi> <3186114.1644960507@sss.pgh.pa.us> <40de25aa-2e20-08ad-e014-edb731594a74@iki.fi> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="IS0zKkzwUGydFO0o" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <40de25aa-2e20-08ad-e014-edb731594a74@iki.fi> User-Agent: Mutt/1.5.24 (2015-08-30) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --IS0zKkzwUGydFO0o Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit On Wed, Feb 16, 2022 at 01:42:27AM +0200, Heikki Linnakangas wrote: > On 15/02/2022 23:28, Tom Lane wrote: > >Heikki Linnakangas writes: > >>That was interesting: the order that WAL segments are archived when a > >>standby is promoted is not fully deterministic. > > > >Oh, of course. > > > >>I find it a bit surprising that pg_stat_archiver.last_archived_wal is > >>not necessarily the highest-numbered segment that was archived. I > >>propose that we mention that in the docs, as in the attached patch. > > > >+1, but I think the description of that field in the pg-stat-archiver-view > >table is also pretty misleading. Maybe like > > > >- Name of the last WAL file successfully archived > >+ Name of the WAL file most recently successfully archived > > > >and similarly s/last/most recent/ for the other fields claiming > >to be "last" something. > > Makes sense, committed it that way. This has seen two failures like the following: # Failed test 'check table contents after point-in-time recovery' # at t/028_pitr_timelines.pl line 167. # got: '2' # expected: '3' # Looks like you failed 1 test of 3. sysname │ snapshot │ branch │ bfurl ──────────┼─────────────────────┼────────┼────────────────────────────────────────────────────────────────────────────────────────────── sungazer │ 2022-03-08 16:24:46 │ HEAD │ http://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=sungazer&dt=2022-03-08%2016%3A24%3A46 mylodon │ 2022-05-18 00:14:19 │ HEAD │ http://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=mylodon&dt=2022-05-18%2000%3A14%3A19 For me, it reproduces consistently with a sleep just before the startup process exits: --- a/src/backend/postmaster/startup.c +++ b/src/backend/postmaster/startup.c @@ -266,6 +266,8 @@ StartupProcessMain(void) */ StartupXLOG(); + pg_usleep(3 * 1000 * 1000); + /* * Exit normally. Exit code 0 tells postmaster that we completed recovery * successfully. A normal/passing run gets this sequence of events: checkpointer: end-of-recovery checkpoint startup process: exits postmaster reaper(): "Startup succeeded, commence normal operations" = StartArchiver() test issues its INSERT shutdown checkpoint checkpointer: exits postmaster reaper(): signal_child(PgArchPID, SIGUSR2) archiver: completes archiving, exits However, if the startup process exit is slow enough, you get: checkpointer: end-of-recovery checkpoint test issues its INSERT shutdown checkpoint checkpointer: exits postmaster reaper(): no PgArchPID, skip that part [no archiving] One can adapt the test to the server behavior by having the test wait for the archiver to start, as attached. This is sufficient to make check-world pass with the above sleep in place. I think we should also modify the PostgresNode archive_command to log a message. That lack of logging was a obstacle upthread (as seen in commit 3279cef) and again here. An alternative would be to declare that the test is right and the server is wrong. The postmaster knows how to start the checkpointer if the checkpointer is not running when the postmaster needs a shutdown checkpoint. It could start the archiver around that same area: /* Start the checkpointer if not running */ if (CheckpointerPID == 0) CheckpointerPID = StartCheckpointer(); /* And tell it to shut down */ if (CheckpointerPID != 0) { signal_child(CheckpointerPID, SIGUSR2); pmState = PM_SHUTDOWN; } Any opinions between the change-test and change-server approaches? The test is new in HEAD, but I get the same behavior in v14 and v13. In v12, node_pitr2 never exits pg_is_in_recovery(). v11 would need test code changes, which I did not attempt. Thanks, nm --IS0zKkzwUGydFO0o Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="archiver-test-wait-v1.patch" Author: Noah Misch Commit: Noah Misch diff --git a/src/test/recovery/t/028_pitr_timelines.pl b/src/test/recovery/t/028_pitr_timelines.pl index a8b12d9..bad02ed 100644 --- a/src/test/recovery/t/028_pitr_timelines.pl +++ b/src/test/recovery/t/028_pitr_timelines.pl @@ -140,6 +140,13 @@ is($result, qq{1}, "check table contents after point-in-time recovery"); # back to this timeline. $node_pitr->safe_psql('postgres', "INSERT INTO foo VALUES(3);"); +# Wait for the archiver to be running. The startup process might have yet to +# exit, in which case the postmaster has not started the archiver. If we +# stop() without an archiver, the archive will be incomplete. +$node_pitr->poll_query_until('postgres', + "SELECT true FROM pg_stat_activity WHERE backend_type = 'archiver';") + or die "Timed out while waiting for archiver to start"; + # Stop the node. This archives the last segment. $node_pitr->stop(); --IS0zKkzwUGydFO0o Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="archiver-test-logging-v1.patch" Author: Noah Misch Commit: Noah Misch diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm index c8c7bc5..12f35a7 100644 --- a/src/test/perl/PostgreSQL/Test/Cluster.pm +++ b/src/test/perl/PostgreSQL/Test/Cluster.pm @@ -1124,7 +1124,7 @@ sub enable_archiving my $copy_command = $PostgreSQL::Test::Utils::windows_os ? qq{copy "%p" "$path\\\\%f"} - : qq{cp "%p" "$path/%f"}; + : qq{echo >&2 "ARCHIVE_COMMAND %p"; cp "%p" "$path/%f"}; # Enable archive_mode and archive_command on node $self->append_conf( --IS0zKkzwUGydFO0o--