Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1oFxNl-0003dO-Sl for pgsql-bugs@arkaria.postgresql.org; Mon, 25 Jul 2022 12:40:21 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1oFxNk-00079L-OU for pgsql-bugs@arkaria.postgresql.org; Mon, 25 Jul 2022 12:40:20 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1oFxNk-00079B-G1 for pgsql-bugs@lists.postgresql.org; Mon, 25 Jul 2022 12:40:20 +0000 Received: from mail.thelabyrinth.net ([45.56.70.56]) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1oFxNf-0002Hu-Rh for pgsql-bugs@lists.postgresql.org; Mon, 25 Jul 2022 12:40:20 +0000 Received: from [172.20.20.20] (c-24-126-11-234.hsd1.wv.comcast.net [24.126.11.234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: dsteele) by mail.thelabyrinth.net (Postfix) with ESMTPSA id 2A1FE54685; Mon, 25 Jul 2022 12:40:13 +0000 (UTC) Message-ID: <4b392ab1-d4a5-5557-adf0-ae319e723da5@pgmasters.net> Date: Mon, 25 Jul 2022 08:40:12 -0400 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Thunderbird/91.11.0 Subject: Re: could not link file in wal restore lines Content-Language: en-US To: Zsolt Ero Cc: pgsql-bugs@lists.postgresql.org, Michael Paquier References: <1bbb2f44-fdf8-71ef-a8f7-24b3fadb3176@pgmasters.net> From: David Steele In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On 7/23/22 12:53, Zsolt Ero wrote: > On 23. Jul 2022 at 18:43:57, David Steele > wrote: >> On 7/23/22 12:39, Zsolt Ero wrote: >>> > On 23. Jul 2022 at 15:20:32, David Steele >> >>> > >> wrote: >>> > Zsolt, can you add --log-level-console=info to your archive_command so >>> > we can see what parameters are being passed to pgbackrest? >>> >>> archive_command = 'pgbackrest --stanza=app --log-level-console=info >>> archive-push %p' >>> >>> on the destination server (where the restore happens). There are no >>> changes in the log, only 3 lines appear once everything is done. I've >>> attached the log. Do I need to change this command on the source server? >> >> Oops, sorry, I meant the restore_command on the server where you do the >> restore. >> >> You'll need to edit this in postgresql.auto.conf after the pgbackrest >> restore but before starting PostgreSQL. > > I did that, attached the new log. From this log it looks like pgBackRest is writing to RECOVERYXLOG as I expected. The error happens relatively rarely (18) compared to the number of WAL segments fetched (1029). pgBackRest does not write the files in question, e.g. LOG: could not link file "pg_wal/000000010000015B0000002C" to "pg_wal/000000010000015B000000F4": File exists And the pg_wal directory started empty. I ran one our standard stress tests on 12.11 (recovering 1592 WAL segments) and I did not see the "could not link file" message. Your system seems to be doing recovery pretty quickly. I wonder if there is a race condition in WAL recycling? One thing you could try is to set wal_recycle = false. Regards, -David