Subject: Re: backup manifests
From: tushar <tushar.ahuja@enterprisedb.com>
To: Robert Haas <robertmhaas@gmail.com>,
 Suraj Kharage <suraj.kharage@enterprisedb.com>
Cc: Rushabh Lathia <rushabh.lathia@gmail.com>,
 Tels <nospam-pg-abuse@bloodgate.com>, David Steele <david@pgmasters.net>,
 Andrew Dunstan <andrew.dunstan@2ndquadrant.com>,
 PostgreSQL Hackers <pgsql-hackers@postgresql.org>,
 Jeevan Chalke <jeevan.chalke@enterprisedb.com>,
 vignesh C <vignesh21@gmail.com>
References: 
 <CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com>
 <CAF1DzPVuFa26xDsKG+Vbj+zLN8P4cXJ5Gdi0w8AUQ1NBKM3=HQ@mail.gmail.com>
 <CA+TgmoZn6K4KiBvcAfYOxUwv-t9oot0ZULdPbOG5k8prpEVVZA@mail.gmail.com>
 <CAF1DzPVeJdV3q4oiCBaTs5jMoUYkqNRicAYR-DAHuHGA7ccO=w@mail.gmail.com>
 <CAF1DzPXcCgXP2awVaH4uHovPo77JdLYPAYShzwnSPMVzdUmxSQ@mail.gmail.com>
 <CA+TgmoZdJ=vBoDU1Skv52Prybp36kMvmpTDBXbmoky0owNgMiA@mail.gmail.com>
 <CAF1DzPW1CAjR7FG86b-OL+hAP2xGqmBUtv0WtRhDz+r+Mwy-Hw@mail.gmail.com>
 <CA+TgmoYY5LXfbmeCO3gpCoCJcqNj0125PO7B7ecuJjnoEdB08A@mail.gmail.com>
 <CAF1DzPXOofDsdSqgEtzpegfcvqB_1TND_h_8uQrvJuVvx-4pDA@mail.gmail.com>
 <CA+TgmobnSHW3upkS_x6D=cM9A3jT9ZA0pcH0FJitBfkyW7sFJA@mail.gmail.com>
 <CAF1DzPUAuY5HqwzBoyFssuN9eRYmTkAKYLPEika1bOTvV2W1YA@mail.gmail.com>
 <CA+TgmoZRTBiPyvQEwV79PU1ePTtSEo2UeVncrkJMbn1sU1gnRA@mail.gmail.com>
 <d97505a0-7693-3edb-e52d-07c0c1e4b3a4@enterprisedb.com>
 <8d81bcd9-fa06-a8a2-10e2-8368580e8e4f@enterprisedb.com>
Message-ID: <951f68f6-734d-3473-631d-dc84efac41bd@enterprisedb.com>
Date: Wed, 4 Mar 2020 15:26:16 +0530
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
 Thunderbird/60.9.0
MIME-Version: 1.0
In-Reply-To: <8d81bcd9-fa06-a8a2-10e2-8368580e8e4f@enterprisedb.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Content-Language: en-US
Precedence: bulk

Hi,
Another observation , if i change the ownership of a file which is under 
global/ directory
i.e

[root@tushar-ldap-docker global]# chown enterprisedb 2396

and run the pg_validatebackup command, i am getting this message -

[centos@tushar-ldap-docker bin]$ ./pg_validatebackup gggg
pg_validatebackup: * manifest_checksum = 
e8cb007bcc9c0deab6eff51cd8d9d9af6af35b86e02f3055e60e70e56737e877
pg_validatebackup: error: could not open file "global/2396": Permission 
denied
*** Error in `./pg_validatebackup': double free or corruption (!prev): 
0x0000000001850ba0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81679)[0x7fa2248e3679]
./pg_validatebackup[0x401f4c]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7fa224884505]
./pg_validatebackup[0x402049]
======= Memory map: ========
00400000-00415000 r-xp 00000000 fd:03 4044545 
/home/centos/pg13_bk_mani/edb/edbpsql/bin/pg_validatebackup
00614000-00615000 r--p 00014000 fd:03 4044545 
/home/centos/pg13_bk_mani/edb/edbpsql/bin/pg_validatebackup
00615000-00616000 rw-p 00015000 fd:03 4044545 
/home/centos/pg13_bk_mani/edb/edbpsql/bin/pg_validatebackup
017f3000-01878000 rw-p 00000000 00:00 0                                  
[heap]
7fa218000000-7fa218021000 rw-p 00000000 00:00 0
7fa218021000-7fa21c000000 ---p 00000000 00:00 0
7fa21e122000-7fa21e137000 r-xp 00000000 fd:03 141697                     
/usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fa21e137000-7fa21e336000 ---p 00015000 fd:03 141697                     
/usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fa21e336000-7fa21e337000 r--p 00014000 fd:03 141697                     
/usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fa21e337000-7fa21e338000 rw-p 00015000 fd:03 141697                     
/usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fa21e338000-7fa224862000 r--p 00000000 fd:03 266442                     
/usr/lib/locale/locale-archive
7fa224862000-7fa224a25000 r-xp 00000000 fd:03 134456                     
/usr/lib64/libc-2.17.so
7fa224a25000-7fa224c25000 ---p 001c3000 fd:03 134456                     
/usr/lib64/libc-2.17.so
7fa224c25000-7fa224c29000 r--p 001c3000 fd:03 134456                     
/usr/lib64/libc-2.17.so
7fa224c29000-7fa224c2b000 rw-p 001c7000 fd:03 134456                     
/usr/lib64/libc-2.17.so
7fa224c2b000-7fa224c30000 rw-p 00000000 00:00 0
7fa224c30000-7fa224c47000 r-xp 00000000 fd:03 134485                     
/usr/lib64/libpthread-2.17.so
7fa224c47000-7fa224e46000 ---p 00017000 fd:03 134485                     
/usr/lib64/libpthread-2.17.so
7fa224e46000-7fa224e47000 r--p 00016000 fd:03 134485                     
/usr/lib64/libpthread-2.17.so
7fa224e47000-7fa224e48000 rw-p 00017000 fd:03 134485                     
/usr/lib64/libpthread-2.17.so
7fa224e48000-7fa224e4c000 rw-p 00000000 00:00 0
7fa224e4c000-7fa224e90000 r-xp 00000000 fd:03 4044478 
/home/centos/pg13_bk_mani/edb/edbpsql/lib/libpq.so.5.13
7fa224e90000-7fa225090000 ---p 00044000 fd:03 4044478 
/home/centos/pg13_bk_mani/edb/edbpsql/lib/libpq.so.5.13
7fa225090000-7fa225093000 r--p 00044000 fd:03 4044478 
/home/centos/pg13_bk_mani/edb/edbpsql/lib/libpq.so.5.13
7fa225093000-7fa225094000 rw-p 00047000 fd:03 4044478 
/home/centos/pg13_bk_mani/edb/edbpsql/lib/libpq.so.5.13
7fa225094000-7fa2250b6000 r-xp 00000000 fd:03 130333                     
/usr/lib64/ld-2.17.so
7fa22527d000-7fa2252a2000 rw-p 00000000 00:00 0
7fa2252b3000-7fa2252b5000 rw-p 00000000 00:00 0
7fa2252b5000-7fa2252b6000 r--p 00021000 fd:03 130333                     
/usr/lib64/ld-2.17.so
7fa2252b6000-7fa2252b7000 rw-p 00022000 fd:03 130333                     
/usr/lib64/ld-2.17.so
7fa2252b7000-7fa2252b8000 rw-p 00000000 00:00 0
7ffdf354f000-7ffdf3570000 rw-p 00000000 00:00 0                          
[stack]
7ffdf3572000-7ffdf3574000 r-xp 00000000 00:00 0                          
[vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  
[vsyscall]
Aborted
[centos@tushar-ldap-docker bin]$


I am getting the error message but along with "*** Error in 
`./pg_validatebackup': double free or corruption (!prev): 
0x0000000001850ba0 ***"  messages

Is this expected ?

regards,

On 3/3/20 8:19 PM, tushar wrote:
> On 3/3/20 4:04 PM, tushar wrote:
>> Thanks Robert.  After applying all the 5 patches (v8-00*) against PG 
>> v13 (commit id -afb5465e0cfce7637066eaaaeecab30b0f23fbe3) , 
>
> There is a scenario where pg_validatebackup is not throwing an error 
> if some file deleted from pg_wal/ folder and  but later at the time of 
> restoring - we are getting an error
>
> [centos@tushar-ldap-docker bin]$ ./pg_basebackup  -D test1
>
> [centos@tushar-ldap-docker bin]$ ls test1/pg_wal/
> 000000010000000000000010  archive_status
>
> [centos@tushar-ldap-docker bin]$ rm -rf test1/pg_wal/*
>
> [centos@tushar-ldap-docker bin]$ ./pg_validatebackup test1
> pg_validatebackup: * manifest_checksum = 
> 88f1ed995c83e86252466a2c88b3e660a69cfc76c169991134b101c4f16c9df7
> pg_validatebackup: backup successfully verified
>
> [centos@tushar-ldap-docker bin]$ ./pg_ctl -D test1 start -o '-p 3333'
> waiting for server to start....2020-03-02 20:05:22.732 IST [21441] 
> LOG:  starting PostgreSQL 13devel on x86_64-pc-linux-gnu, compiled by 
> gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39), 64-bit
> 2020-03-02 20:05:22.733 IST [21441] LOG:  listening on IPv6 address 
> "::1", port 3333
> 2020-03-02 20:05:22.733 IST [21441] LOG:  listening on IPv4 address 
> "127.0.0.1", port 3333
> 2020-03-02 20:05:22.736 IST [21441] LOG:  listening on Unix socket 
> "/tmp/.s.PGSQL.3333"
> 2020-03-02 20:05:22.739 IST [21442] LOG:  database system was 
> interrupted; last known up at 2020-03-02 20:04:35 IST
> 2020-03-02 20:05:22.739 IST [21442] LOG:  creating missing WAL 
> directory "pg_wal/archive_status"
> 2020-03-02 20:05:22.886 IST [21442] LOG:  invalid checkpoint record
> 2020-03-02 20:05:22.886 IST [21442] FATAL:  could not locate required 
> checkpoint record
> 2020-03-02 20:05:22.886 IST [21442] HINT:  If you are restoring from a 
> backup, touch 
> "/home/centos/pg13_bk_mani/edb/edbpsql/bin/test1/recovery.signal" and 
> add required recovery options.
>     If you are not restoring from a backup, try removing the file 
> "/home/centos/pg13_bk_mani/edb/edbpsql/bin/test1/backup_label".
>     Be careful: removing 
> "/home/centos/pg13_bk_mani/edb/edbpsql/bin/test1/backup_label" will 
> result in a corrupt cluster if restoring from a backup.
> 2020-03-02 20:05:22.886 IST [21441] LOG:  startup process (PID 21442) 
> exited with exit code 1
> 2020-03-02 20:05:22.886 IST [21441] LOG:  aborting startup due to 
> startup process failure
> 2020-03-02 20:05:22.889 IST [21441] LOG:  database system is shut down
>  stopped waiting
> pg_ctl: could not start server
> Examine the log output.
> [centos@tushar-ldap-docker bin]$
>

-- 
regards,tushar
EnterpriseDB  https://www.enterprisedb.com/
The Enterprise PostgreSQL Company