Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1w0xzw-002N3c-04 for pgsql-bugs@arkaria.postgresql.org; Fri, 13 Mar 2026 08:35:56 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.96) (envelope-from ) id 1w0xzu-002WfX-1b for pgsql-bugs@arkaria.postgresql.org; Fri, 13 Mar 2026 08:35:54 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1w0xzu-002WfP-0m for pgsql-bugs@lists.postgresql.org; Fri, 13 Mar 2026 08:35:54 +0000 Received: from mail-lf1-x132.google.com ([2a00:1450:4864:20::132]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.98.2) (envelope-from ) id 1w0xzr-00000002QrQ-3R89 for pgsql-bugs@lists.postgresql.org; Fri, 13 Mar 2026 08:35:54 +0000 Received: by mail-lf1-x132.google.com with SMTP id 2adb3069b0e04-5a133502accso2438896e87.3 for ; Fri, 13 Mar 2026 01:35:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1773390951; cv=none; d=google.com; s=arc-20240605; b=Mnn/jDb6xs/ro1+hWZcDOLddgLAssLlPVf7p7GV9mU8YpIoq5MeIg/Gelx/cIeg4wZ Fu5Ru6Uq01ggVInZS5MJ6VUg7m25WIMS8bfmCREwYDcHLSEkdyUv1TRAgctY7rGiDsvy uUngAlrRlcggs1yvdYeLONEHm0T03F2EDkhOAO55Y8x9WdNaOpugDbSmr8efqIPcfjbh vGTfCCPpMOwMByUPlN1C4UaG2T23cqPUWH2UirJTg6yO5QvkZDh7Osfxp0QCo7e+laaW l+AIKToORErbCScgtNn8F7P2dtpj85TO+C/Mwae1iu2vq6nTmbm7UbzVVNUFGnBuOtdo WBUw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:dkim-signature; bh=Dm+rnsemJFIhWTeJt22UigbMELPkzCZT/1b0RENNzSQ=; fh=vR25N3+d8dNHUTLhu+5NPbhqmVDzmSWekj9cOnxEonw=; b=XGq/YNxoyTPvJL77LSr52GLSE8ouCrWFdtDQHRXkmugJEl9tAceUUIEdYGKsAf2hRx fVIH+Ph4Llr4Dk/h3kPfx1pSltW3TrxqVFWR+QZmD+N1ssFhXS/xMSZ5OjF6BjCdr4hp tkvqK2408niIkypoJOadOHuE3EGCQ1F6OXSIZv8XN1GSylDxKEb6jvM7gHW3uSPXKcGw 2kDVfPAUX2+WzpIjcgVIBGH4uoX6AlAWkArqKbd2jemRUJP5dKal71VfIHGkN+V9GcI0 /Qi81jZANmA8lLWPpQFKNkb+zb2eyAI1tO/yBwDt7D/tabHtU/djt7JunGf1rLZmZbPh cJzg==; darn=lists.postgresql.org ARC-Authentication-Results: i=1; mx.google.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; t=1773390951; x=1773995751; darn=lists.postgresql.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=Dm+rnsemJFIhWTeJt22UigbMELPkzCZT/1b0RENNzSQ=; b=Ij/M3hC2lYyGZa+ZRccvxxRoMO5QHsnLnXhA6d0tsX2G+uDk6nbR2h1oFvmOq8DTc7 wZNF1Luov+Dy3tyYiDFzhdtn87b0MelhrxciT64Nlr1czeP/Q+JFWFm0rLvEDlZoluib g6fnDwPyWKbZn3KxjtdJJ4sBCmYz8NYGpii7cyImxZFU20PidGoFkbLPUP8c3CYml98c MAL5WRzoH9oUtYkMkHign7r28aeoCIv4k5Rnn6KVw8KjUw0E4EmmXnABh5ntE104uWvA +lofAIZC1HaAX+A4P3gsQCR8epSrWgxWPNw1gD92ewkHkWi27HJh/xAwmlLlwR7zLGBL V44w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1773390951; x=1773995751; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=Dm+rnsemJFIhWTeJt22UigbMELPkzCZT/1b0RENNzSQ=; b=pzZgFNTI2eB+ugA35lX0TYoD4y6IoJw7dfawJCCoKYwMi5FexHlU2TICw9wLf0oJOh gRrsQ1BSsL9LCbXfD5sv06eG75+5HaZ1jdPBEMc9ZJThmndVaaES/7228oQmr9Z+OfXC 0UbwyIq04m8P5Bf98K5KR9D4g5qAAOnb3Kwc22MbG9C4VOFDRZUYdXTHjXCkRrhp5Cvw cu/boydh9ZQbISxsIAz0ra253ndxszVvKirmE98FJSnVecA/c9yXe5CFvPmliWQ2MoPy Qif6h1XTUZNLGql9CAEe/5syREa3e8kNt5RjrZ9f9BTZhFGFlG+NMkxKuv0odGP2ztEU CgPQ== X-Gm-Message-State: AOJu0Yy8YlsaS07ql6pyef2FHvgtedLXhTaYVCTXNcuYU4pam26Wn+jJ 9Fde+kj+0MYW2xTzib4WoEAuj/KyZVi2scB9POpjuLZ5uG/+3EdPXosZCvtmZnLdA/kJEzOrLud yU0SA7r/It+LUYCjFVYAVILNoDr3bIeIBdO30wFzXLQ== X-Gm-Gg: ATEYQzwsUHdS5LAj05Z1T14u5H+MwFRd3LqKMjdT6lJAjw1e6+GoxCqTw9Yct4YYfyZ GFEejhWCTWDwW9E8hlZxYBh2015ANrS1jRArIQ4T/zqOKGnaiEcdJ5WICpqPl0O9sgnGMhTS0Yl k2J9FOD3FUnUCAR4Oc3uRUC9F4q7NLnhfVdLzfEi21Xyf54IC+voF1fqO1k2jmszOoaGpOjhiXg 1D6PQZqYxIQgh9I4qefjwRTHMCkqtLtQOo8eH+pn/v9WreKjIwrXyhPdyBwwYiqDX9fk9EcYS3i 3WeGxPlaU+PWujq4tjiYyBuH5sg4av31RY8kIjRm X-Received: by 2002:ac2:568c:0:b0:5a1:3f63:c7e0 with SMTP id 2adb3069b0e04-5a162b25e03mr752698e87.24.1773390950872; Fri, 13 Mar 2026 01:35:50 -0700 (PDT) MIME-Version: 1.0 References: <19432-3c569e9472f42cee@postgresql.org> <58ceb75034bf694d52f81e898df35e14122fe5c8.camel@cybertec.at> In-Reply-To: <58ceb75034bf694d52f81e898df35e14122fe5c8.camel@cybertec.at> From: Felix Hamme Date: Fri, 13 Mar 2026 09:35:39 +0100 X-Gm-Features: AaiRm52JIdohUNkGjcp4_VtH1GkCmYQ80hX4Hh13oaQeKePg7CLppWbYBYw6srI Message-ID: Subject: Re: BUG #19432: recovery fails at invalid checkpoint record To: Laurenz Albe Cc: pgsql-bugs@lists.postgresql.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk Thank you for checking, now I found what I did wrong: "mv" doesn't work as a restore_command because the same .history file is restored multiple times during recovery. I successfully recovered using a restore_command which does a cp for history files and mv for wal files. It logged this: cp /DBDATA/test/fakearchive/00000002.history pg_wal/RECOVERYHISTORY success cp /DBDATA/test/fakearchive/00000003.history pg_wal/RECOVERYHISTORY not fou= nd cp /DBDATA/test/fakearchive/00000002.history pg_wal/RECOVERYHISTORY success mv /DBDATA/test/fakearchive/000000010000000000000003 pg_wal/RECOVERYXLOG su= ccess mv /DBDATA/test/fakearchive/000000010000000000000004 pg_wal/RECOVERYXLOG su= ccess mv /DBDATA/test/fakearchive/000000020000000000000005 pg_wal/RECOVERYXLOG su= ccess mv /DBDATA/test/fakearchive/000000020000000000000006 pg_wal/RECOVERYXLOG su= ccess mv /DBDATA/test/fakearchive/000000020000000000000007 pg_wal/RECOVERYXLOG su= ccess cp /DBDATA/test/fakearchive/00000003.history pg_wal/RECOVERYHISTORY not fou= nd cp /DBDATA/test/fakearchive/00000002.history pg_wal/RECOVERYHISTORY success Is it safe in general to use mv for wal files? In other words, do the currently supported postgres versions run restore_command only once per wal file? Best regards Felix Hamme On Thu, Mar 12, 2026 at 8:29=E2=80=AFPM Laurenz Albe wrote: > > On Thu, 2026-03-12 at 16:20 +0000, PG Bug reporting form wrote: > > Hi, I'm trying to restore from a pg_basebackup at timeline 1 to a > > restore_target_time in timeline 2. > > It fails at "invalid checkpoint record", "could not locate required > > checkpoint record at 0/3000080". > > All relevant wal files are in the archive, the restore_command works an= d > > backup_label, pg_controldata, pg_waldump and 00000002.history look like > > everything should work. > > Recovering only timeline 1 works, but it fails as soon as it should pro= ceed > > in timeline 2. > > A 6.7MB tar of the basebackup and the wal archive is available at > > https://get.hidrive.com/i/PwMejRQG . This link expires on 2026-03-13, I= can > > provide a new link if needed. > > Why does this recovery fail? > > Funny. I unpacked your data directory and reduced your postgresql.auto.c= onf > to something that fits my system: > > log_min_messages =3D 'DEBUG5' > restore_command =3D 'cp /home/laurenz/hamme/fakearchive/%f %p' > recovery_target_time =3D '2026-03-11 14:51:28 UTC' > recovery_target_action =3D 'promote' > hot_standby_feedback =3D 'on' > log_destination =3D 'csvlog' > log_directory =3D '/home/laurenz/hamme/log' > logging_collector =3D 'on' > wal_level =3D 'logical' > port =3D 5433 > unix_socket_directories =3D '/home/laurenz/hamme' > max_connections =3D 300 > > Recovery worked like a charm. pg_waldump shows the checkpoint record in > 000000010000000000000003 at the correct position. > > Not sure what you did wrong. > > Yours, > Laurenz Albe