Date: Tue, 21 May 2019 18:37:27 +0200
From: Tomas Vondra <tomas.vondra@2ndquadrant.com>
To: Greg Clough <Greg.Clough@ihsmarkit.com>
Cc: Mariel Cherkassky <mariel.cherkassky@gmail.com>,
	Bimal <internetuser2008@yahoo.com>,
	PostgreSQL mailing lists <pgsql-performance@postgresql.org>
Subject: Re: Trying to handle db corruption 9.6
Message-ID: <20190521163727.kdcvkpwm2n3fliou@development>
References: 
 <CA+t6e1=uKKccO4cGPdXgM4K3+fa1sY9-PE+w8V7QAHS_P1ot-g@mail.gmail.com>
 <DM6PR12MB27969B7AF5108691B55A00DEEC060@DM6PR12MB2796.namprd12.prod.outlook.com>
 <CA+t6e1=j8hwY-LCgN461-bCNB4ODxG6LNTHD9=uHko7ZAHy-fA@mail.gmail.com>
 <1909968146.4951657.1558369245203@mail.yahoo.com>
 <CA+t6e1=+Wmw8WwrMTQR0umR3ktkGiVfFUho8xeSni2Jjm4wKtg@mail.gmail.com>
 <20190520210406.5paaijqd5no6imez@development>
 <CA+t6e1k0PdBd9uwTD+BwqwfN4UNtVZht2FSEvpjS_m4Jog7xhA@mail.gmail.com>
 <20190521130714.4ncqlri3o224ptpx@development>
 <CA+t6e1m61Q_0+WwJLc5fDc0z4YwcHamEFTN2oRY+fqS2pxEUyQ@mail.gmail.com>
 <MN2PR12MB30060460ABD64B99B51549E7EC070@MN2PR12MB3006.namprd12.prod.outlook.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Disposition: inline
In-Reply-To: 
 <MN2PR12MB30060460ABD64B99B51549E7EC070@MN2PR12MB3006.namprd12.prod.outlook.com>
User-Agent: NeoMutt/20180716-1444-295967-dirty
Precedence: bulk

On Tue, May 21, 2019 at 04:03:52PM +0000, Greg Clough wrote:
>>  My restore command copy the wals from archive dir in the primary to an
>>  archive dir in the secondary(different from the pg_xlog in the
>>  secondary)
>
>I think that you're restore command puts them back into the archive, and
>then uncompresses them into pg_xlog, which is what %p represents.
>
>
>> Should I run it manually and see if the archives are copied to the
>> archive dir in the secondary or should I just copy all of them to the
>> xlog dir in the secondary ?
>
>That would be my first test, but as Thomas mentioned, you don't have any
>hint of WAL archives being restored in the postgresql.log... so it's not
>even trying.  It's not likely that archive_command is your problem at the
>moment.
>
>
>> I tried to start the secondary as a primary (I have a backup..) but I
>> still got an error (invalid checkpoint record from primary./
>> secondary). Does it means that my backup is corrupted ?
>
>I think so, but Thomas could probably confirm if all hope is lost.  Also,
>I'm not sure if there is a terminology difference but a "standby" is
>never considered a "backup".  I realise it's late in the day, but even if
>you have a correctly configured Standby you should also take backups with
>pg_basebackup, Barman, pgBackRest, etc.
>

Well, I have no idea. We still got no information about how the standby
was created, if it was ever running fine, and so on. Considering it does
not seem to be getting data from the archive, it might be the case it was
created in some strange way and never really worked. And if there really
are no log messages about the restore_command, it probably fails before
the standby even tries to execute it.

So I don't know.

>Restoring backups is where I would be heading now, as things seem
>terribly broken.
>

Right. But my impression is there are no backups ...


regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services