RE: 2nd PostgreSQL server in WAL shipping cluster fails to start

public inbox for [email protected]  
help / color / mirror / Atom feed

From: Samuel VISCAPI <[email protected]>
To: Alvaro Herrera <[email protected]>
Cc: [email protected] <[email protected]>
Subject: RE: 2nd PostgreSQL server in WAL shipping cluster fails to start
Date: Thu, 30 Jan 2025 09:54:26 +0000
Message-ID: <MR2P264MB014775B1000AD1C59C5EA584FEE92@MR2P264MB0147.FRAP264.PROD.OUTLOOK.COM> (raw)
In-Reply-To: <[email protected]>
References: <MR2P264MB0147200E004CB5C9FC761ACCFE132@MR2P264MB0147.FRAP264.PROD.OUTLOOK.COM>
	<[email protected]>

Dear Alvaro and Anton,

Thank you for your quick replies, really appreciated. I ran some more tests today and the results are as follows:

Available disk space on primary server : 101GB free in the data directory. I believe some cleaning up has been done, I need to check with my colleagues about that.
Wal_sender_timeout : #wal_sender_timeout = 60s
Wal_keep_segments : not present in postgresql.conf
Networking issues : none at the moment, the primary server can ping the server in standby mode by its IP address. Name resolution doesn't work though.
Monitor replication lag between the two nodes:
- Primary node : 

SELECT pg_current_wal_lsn();
 pg_current_wal_lsn
--------------------
 18/B6C404B8

- Standby node : TODO when the node is up again

I'm tempted to give the pgbasebackup command a go: https://www.postgresql.org/docs/13/app-pgbasebackup.html

pg_basebackup -h primary_host -D /some_app/data -U postgres -v -P

@Alvaro Herrera : the archive_command is indeed set in the primary server, as follows:

archive_command = 'test ! -f /some_app/data/mnt/server/archivedir/%f && cp %p /some_app/data/mnt/server/archivedir/%f'

There are some files in the pg_wal directory but they only amount to 289MB. If I need to rebuild the replica from scratch, could I use the pg_basebackup command above for that purpose ?

And you are obviously right about the error message in French. Please find it below:

2025-01-09 09:46:35.742 CET [3147382] FATAL:  n'a pas pu recevoir des données du flux de WAL : ERREUR:  le segment demandé du journal de transaction, 000000010000000D000000D6, a déjà été supprimé
2025-01-09 09:46:40.745 CET [3147395] LOG:  démarré le flux des journaux depuis le principal à D/D6000000 sur la timeline 1
2025-01-09 09:46:40.745 CET [3147395] FATAL:  n'a pas pu recevoir des données du flux de WAL : ERREUR:  le segment demandé du journal de transaction, 000000010000000D000000D6, a déjà été supprimé
2025-01-09 09:46:45.749 CET [3147397] LOG:  démarré le flux des journaux depuis le principal à D/D6000000 sur la timeline 1
2025-01-09 09:46:45.749 CET [3147397] FATAL:  n'a pas pu recevoir des données du flux de WAL : ERREUR:  le segment demandé du journal de transaction, 000000010000000D000000D6, a déjà été supprimé
2025-01-09 09:46:50.753 CET [3147424] LOG:  démarré le flux des journaux depuis le principal à D/D6000000 sur la timeline 1
2025-01-09 09:46:50.753 CET [3147424] FATAL:  n'a pas pu recevoir des données du flux de WAL : ERREUR:  le segment demandé du journal de transaction, 000000010000000D000000D6, a déjà été supprimé

Thanks again for your help.

Best regards,

Samuel

-----Message d'origine-----
De : Alvaro Herrera <[email protected]> 
Envoyé : jeudi 9 janvier 2025 12:11
À : Samuel VISCAPI <[email protected]>
Cc : [email protected]
Objet : Re: 2nd PostgreSQL server in WAL shipping cluster fails to start

On 2025-Jan-09, Samuel VISCAPI wrote:

> Dear all,
> 
> Some years ago I set up a two PostgreSQL 13 nodes cluster on Debian 11. This cluster uses the Write-Ahead Log Shipping method. I've just been told the second server (in standby mode) is down and refuses to start again with the following error message (roughly translated from French) :
> 
> 2025-01-09 09:46:35.742 CET [3147382] FATAL:  could not receive data 
> from WAL stream: ERROR:  segment requested from transaction journal, 
> 000000010000000D000000D6, has already been removed
> 2025-01-09 09:46:40.745 CET [3147395] LOG:  journal flow started from 
> primary at D/D6000000 on timeline 1
> 2025-01-09 09:46:40.745 CET [3147395] FATAL:  could not receive data 
> from WAL stream : ERROR:  segment requested from transaction journal, 
> 000000010000000D000000D6, has already been removed
> 2025-01-09 09:46:45.749 CET [3147397] LOG: journal flow started from 
> primary at D/D6000000 on timeline 1
> 2025-01-09 09:46:45.749 CET [3147397] FATAL:  could not receive data 
> from WAL stream : ERROR:  segment requested from transaction journal, 
> 000000010000000D000000D6, has already been removed
> 2025-01-09 09:46:50.753 CET [3147424] LOG:  journal flow started from 
> primary at D/D6000000 on timeline 1
> 2025-01-09 09:46:50.753 CET [3147424] FATAL:  could not receive data 
> from WAL stream : ERROR:  segment requested from transaction journal, 
> 000000010000000D000000D6, has already been removed

This means the standby is requesting a segment that was already removed.
You may be able to find those files in a WAL archive, if you have archive_command set in the primary.  If you do, then it would work to copy those to the standby's pg_wal/ subdirectory.  If you don't have them, then the replica must be rebuilt.


Note that it's not a good idea to translate the error messages when posting -- that is, it's better to post exactly what the log file has.
For helpfulness you could post a translation for the lines separately.
But the developers can find the translated messages in the source code, if they need them ... but if you translate them yourself, there's no way to know exactly which ones they are.  Sometimes subtle differences are important cues.

-- 
Álvaro Herrera               48°01'N 7°57'E  -  https://www.EnterpriseDB.com/

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected], [email protected], [email protected]
  Subject: RE: 2nd PostgreSQL server in WAL shipping cluster fails to start
  In-Reply-To: <MR2P264MB014775B1000AD1C59C5EA584FEE92@MR2P264MB0147.FRAP264.PROD.OUTLOOK.COM>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox