Content-Type: multipart/mixed; boundary="------------psc8Sb5V6lNfMWorPQ1DVB3L"
Message-ID: <4980aba2-9cd4-4bd6-b2e1-e873639ab7ee@pgmasters.net>
Date: Wed, 13 Mar 2024 13:12:28 +1300
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: Add basic tests for the low-level backup method.
Content-Language: en-US
To: Michael Paquier <michael@paquier.xyz>
Cc: pgsql-hackers@lists.postgresql.org
References: <03e4cf19-fb76-4891-accc-edd429a25b21@pgmasters.net>
 <431ecc5d-8c07-4ee1-b05e-3ca536e606bb@pgmasters.net>
 <ZeAAPW8h1uxvt4Bj@paquier.xyz>
From: David Steele <david@pgmasters.net>
In-Reply-To: <ZeAAPW8h1uxvt4Bj@paquier.xyz>
Archived-At: 
 <https://www.postgresql.org/message-id/4980aba2-9cd4-4bd6-b2e1-e873639ab7ee%40pgmasters.net>
Precedence: bulk

This is a multi-part message in MIME format.
--------------psc8Sb5V6lNfMWorPQ1DVB3L
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

On 2/29/24 16:55, Michael Paquier wrote:
> On Thu, Feb 29, 2024 at 10:30:52AM +1300, David Steele wrote:
>> On 11/12/23 08:21, David Steele wrote:
>>> As promised in [1], attached are some basic tests for the low-level
>>> backup method.
>>
>> Added to the 2024-03 CF.
> 
> There is already 040_standby_failover_slots_sync.pl in recovery/ that
> uses the number of your test script.  You may want to bump it, that's
> a nit.

Renamed to 042_low_level_backup.pl.

> +unlink("$backup_dir/postmaster.pid")
> +	or BAIL_OUT("unable to unlink $backup_dir/postmaster.pid");
> +unlink("$backup_dir/postmaster.opts")
> +	or BAIL_OUT("unable to unlink $backup_dir/postmaster.opts");
> +unlink("$backup_dir/global/pg_control")
> +	or BAIL_OUT("unable to unlink $backup_dir/global/pg_control");
> 
> RELCACHE_INIT_FILENAME as well?

I'm not trying to implement the full exclusion list here, just enough to 
get the test working. Since exclusions are optional according to the 
docs I don't think we need them for a valid tests.

> +# Rather than writing out backup_label, try to recover the backup without
> +# backup_label to demonstrate that recovery will not work correctly without it,
> +# i.e. the canary table will be missing and the cluster will be corrupt. Provide
> +# only the WAL segment that recovery will think it needs.
> 
> Okay, why not.  No objections to this addition.  I am a bit surprised
> that this is not scanning some of the logs produced by the startup
> process for particular patterns.

Not sure what to look for here. There are no distinct messages for crash 
recovery. Perhaps there should be?

> +# Save backup_label into the backup directory and recover using the primary's
> +# archive. This time recovery will succeed and the canary table will be
> +# present.
> 
> Here are well, I think that we should add some log_contains() with
> pre-defined patterns to show that recovery has completed the way we
> want it with a backup_label up to the end-of-backup record.

Sure, I added a check for the new log message when recovering with a 
backup_label.

Regards,
-David
--------------psc8Sb5V6lNfMWorPQ1DVB3L
Content-Type: text/plain; charset=UTF-8; name="low-level-backup-test-v2.patch"
Content-Disposition: attachment; filename="low-level-backup-test-v2.patch"
Content-Transfer-Encoding: base64

RnJvbSAwNzY0MjlhNTc4ZWNlM2MyMTM1MjUyMjBhNDk2MWNiNWI1NmMxOTBhIE1vbiBTZXAg
MTcgMDA6MDA6MDAgMjAwMQpGcm9tOiBEYXZpZCBTdGVlbGUgPGRhdmlkQHBnbWFzdGVycy5u
ZXQ+CkRhdGU6IFdlZCwgMTMgTWFyIDIwMjQgMDA6MDE6NTUgKzAwMDAKU3ViamVjdDogQWRk
IGJhc2ljIHRlc3RzIGZvciB0aGUgbG93LWxldmVsIGJhY2t1cCBtZXRob2QuCgpUaGVyZSBh
cmUgY3VycmVudGx5IG5vIHRlc3RzIGZvciB0aGUgbG93LWxldmVsIGJhY2t1cCBtZXRob2Qu
IHBnX2JhY2t1cF9zdGFydCgpCmFuZCBwZ19iYWNrdXBfc3RvcCgpIGFyZSBjYWxsZWQgYnV0
IG5vdCBleGVyY2lzZWQgaW4gYW55IHJlYWwgZmFzaGlvbi4KClRoZXJlIGlzIGEgbG90IG1v
cmUgdGhhdCBjYW4gYmUgZG9uZSwgYnV0IHRoaXMgYXQgbGVhc3QgcHJvdmlkZXMgc29tZSBi
YXNpYwp0ZXN0cyBhbmQgcHJvdmlkZXMgYSBwbGFjZSBmb3IgZnV0dXJlIGltcHJvdmVtZW50
LgotLS0KIHNyYy90ZXN0L3JlY292ZXJ5L3QvMDQyX2xvd19sZXZlbF9iYWNrdXAucGwgfCAx
MDggKysrKysrKysrKysrKysrKysrKysKIDEgZmlsZSBjaGFuZ2VkLCAxMDggaW5zZXJ0aW9u
cygrKQogY3JlYXRlIG1vZGUgMTAwNjQ0IHNyYy90ZXN0L3JlY292ZXJ5L3QvMDQyX2xvd19s
ZXZlbF9iYWNrdXAucGwKCmRpZmYgLS1naXQgYS9zcmMvdGVzdC9yZWNvdmVyeS90LzA0Ml9s
b3dfbGV2ZWxfYmFja3VwLnBsIGIvc3JjL3Rlc3QvcmVjb3ZlcnkvdC8wNDJfbG93X2xldmVs
X2JhY2t1cC5wbApuZXcgZmlsZSBtb2RlIDEwMDY0NAppbmRleCAwMDAwMDAwMDAwLi4yMjY2
OGJkYzBkCi0tLSAvZGV2L251bGwKKysrIGIvc3JjL3Rlc3QvcmVjb3ZlcnkvdC8wNDJfbG93
X2xldmVsX2JhY2t1cC5wbApAQCAtMCwwICsxLDEwOCBAQAorCisjIENvcHlyaWdodCAoYykg
MjAyMywgUG9zdGdyZVNRTCBHbG9iYWwgRGV2ZWxvcG1lbnQgR3JvdXAKKworIyBUZXN0IGxv
dy1sZXZlbCBiYWNrdXAgbWV0aG9kIGJ5IHVzaW5nIHBnX2JhY2t1cF9zdGFydCgpIGFuZCBw
Z19iYWNrdXBfc3RvcCgpCisjIHRvIGNyZWF0ZSBiYWNrdXBzLgorCit1c2Ugc3RyaWN0Owor
dXNlIHdhcm5pbmdzOworCit1c2UgRmlsZTo6Q29weSBxdyhjb3B5KTsKK3VzZSBGaWxlOjpQ
YXRoIHF3KHJlbW92ZV90cmVlKTsKK3VzZSBQb3N0Z3JlU1FMOjpUZXN0OjpDbHVzdGVyOwor
dXNlIFBvc3RncmVTUUw6OlRlc3Q6OlV0aWxzOwordXNlIFRlc3Q6Ok1vcmU7CisKKyMgU3Rh
cnQgcHJpbWFyeSBub2RlIHdpdGggYXJjaGl2aW5nCitteSAkbm9kZV9wcmltYXJ5ID0gUG9z
dGdyZVNRTDo6VGVzdDo6Q2x1c3Rlci0+bmV3KCdwcmltYXJ5Jyk7Ciskbm9kZV9wcmltYXJ5
LT5pbml0KGhhc19hcmNoaXZpbmcgPT4gMSwgYWxsb3dzX3N0cmVhbWluZyA9PiAxKTsKKyRu
b2RlX3ByaW1hcnktPnN0YXJ0OworCisjIFN0YXJ0IGJhY2t1cAorbXkgJGJhY2t1cF9uYW1l
ID0gJ2JhY2t1cDEnOworbXkgJHBzcWwgPSAkbm9kZV9wcmltYXJ5LT5iYWNrZ3JvdW5kX3Bz
cWwoJ3Bvc3RncmVzJyk7CisKKyRwc3FsLT5xdWVyeV9zYWZlKCJTRVQgY2xpZW50X21pbl9t
ZXNzYWdlcyBUTyBXQVJOSU5HIik7CiskcHNxbC0+c2V0X3F1ZXJ5X3RpbWVyX3Jlc3RhcnQo
KTsKKyRwc3FsLT5xdWVyeV9zYWZlKCJzZWxlY3QgcGdfYmFja3VwX3N0YXJ0KCd0ZXN0IGxh
YmVsJykiKTsKKworIyBDb3B5IGZpbGVzCitteSAkYmFja3VwX2RpciA9ICRub2RlX3ByaW1h
cnktPmJhY2t1cF9kaXIoKSAuICcvJyAuICRiYWNrdXBfbmFtZTsKKworUG9zdGdyZVNRTDo6
VGVzdDo6UmVjdXJzaXZlQ29weTo6Y29weXBhdGgoCisJJG5vZGVfcHJpbWFyeS0+ZGF0YV9k
aXIoKSwgJGJhY2t1cF9kaXIpOworCisjIENsZWFudXAgc29tZSBmaWxlcy9wYXRocyB0aGF0
IHNob3VsZCBub3QgYmUgaW4gdGhlIGJhY2t1cC4gVGhlcmUgaXMgbm8KKyMgYXR0ZW1wdCB0
byBhbGwgdGhlIGV4Y2x1c2lvbnMgZG9uZSBieSBwZ19iYXNlYmFja3VwIGhlcmUsIGluIHBh
cnQgYmVjYXVzZQorIyB0aGV5IGFyZSBub3QgcmVxdWlyZWQsIGJ1dCBhbHNvIHRvIGtlZXAg
dGhlIHRlc3Qgc2ltcGxlLgorIworIyBBbHNvIHJlbW92ZSBwZ19jb250cm9sIGJlY2F1c2Ug
aXQgbmVlZHMgdG8gYmUgY29waWVkIGxhdGVyIGFuZCBpdCB3aWxsIGJlCisjIG9idmlvdXMg
aWYgdGhlIGNvcHkgZmFpbHMuCit1bmxpbmsoIiRiYWNrdXBfZGlyL3Bvc3RtYXN0ZXIucGlk
IikKKwlvciBCQUlMX09VVCgidW5hYmxlIHRvIHVubGluayAkYmFja3VwX2Rpci9wb3N0bWFz
dGVyLnBpZCIpOwordW5saW5rKCIkYmFja3VwX2Rpci9wb3N0bWFzdGVyLm9wdHMiKQorCW9y
IEJBSUxfT1VUKCJ1bmFibGUgdG8gdW5saW5rICRiYWNrdXBfZGlyL3Bvc3RtYXN0ZXIub3B0
cyIpOwordW5saW5rKCIkYmFja3VwX2Rpci9nbG9iYWwvcGdfY29udHJvbCIpCisJb3IgQkFJ
TF9PVVQoInVuYWJsZSB0byB1bmxpbmsgJGJhY2t1cF9kaXIvZ2xvYmFsL3BnX2NvbnRyb2wi
KTsKKworcmVtb3ZlX3RyZWUoIiRiYWNrdXBfZGlyL3BnX3dhbCIsIHtrZWVwX3Jvb3QgPT4g
MX0pCisJb3IgQkFJTF9PVVQoInVuYWJsZSB0byB1bmxpbmsgY29udGVudHMgb2YgJGJhY2t1
cF9kaXIvcGdfd2FsIik7CisKKyMgQ3JlYXRlIGEgdGFibGUgdGhhdCB3aWxsIGJlIHVzZWQg
dG8gdmVyaWZ5IHRoYXQgcmVjb3Zlcnkgc3RhcnRlZCBhdCB0aGUKKyMgY29ycmVjdCBsb2Nh
dGlvbiwgcmF0aGVyIHRoYW4gdGhlIGxvY2F0aW9uIHJlY29yZGVkIGluIHBnX2NvbnRyb2wK
KyRwc3FsLT5xdWVyeV9zYWZlKCJjcmVhdGUgdGFibGUgY2FuYXJ5IChpZCBpbnQpIik7CisK
KyMgQWR2YW5jZSB0aGUgY2hlY2twb2ludCBsb2NhdGlvbiBpbiBwZ19jb250cm9sIHBhc3Qg
dGhlIGxvY2F0aW9uIHdoZXJlIHRoZQorIyBiYWNrdXAgc3RhcnRlZC4gU3dpdGNoIHRoZSBX
QUwgdG8gbWFrZSBpdCByZWFsbHkgb2J2aW91cyB0aGF0IHRoZSBsb2NhdGlvbgorIyBpcyBk
aWZmZXJlbnQgYW5kIHRvIHB1dCB0aGUgY2hlY2twb2ludCBpbiBhIG5ldyBXQUwgc2VnbWVu
dC4KKyRwc3FsLT5xdWVyeV9zYWZlKCJzZWxlY3QgcGdfc3dpdGNoX3dhbCgpIik7CiskcHNx
bC0+cXVlcnlfc2FmZSgiY2hlY2twb2ludCIpOworCisjIENvcHkgcGdfY29udHJvbCBsYXN0
IHNvIGl0IGNvbnRhaW5zIHRoZSBuZXcgY2hlY2twb2ludAorY29weSgkbm9kZV9wcmltYXJ5
LT5kYXRhX2RpcigpIC4gJy9nbG9iYWwvcGdfY29udHJvbCcsCisJCSIkYmFja3VwX2Rpci9n
bG9iYWwvcGdfY29udHJvbCIpCisJb3IgQkFJTF9PVVQoInVuYWJsZSB0byBjb3B5IGdsb2Jh
bC9wZ19jb250cm9sIik7CisKKyMgU3RvcCBiYWNrdXAgYW5kIGdldCBiYWNrdXBfbGFiZWwK
K215ICRiYWNrdXBfbGFiZWwgPSAkcHNxbC0+cXVlcnlfc2FmZSgic2VsZWN0IGxhYmVsZmls
ZSBmcm9tIHBnX2JhY2t1cF9zdG9wKCkiKTsKKworIyBSYXRoZXIgdGhhbiB3cml0aW5nIG91
dCBiYWNrdXBfbGFiZWwsIHRyeSB0byByZWNvdmVyIHRoZSBiYWNrdXAgd2l0aG91dAorIyBi
YWNrdXBfbGFiZWwgdG8gZGVtb25zdHJhdGUgdGhhdCByZWNvdmVyeSB3aWxsIG5vdCB3b3Jr
IGNvcnJlY3RseSB3aXRob3V0IGl0LAorIyBpLmUuIHRoZSBjYW5hcnkgdGFibGUgd2lsbCBi
ZSBtaXNzaW5nIGFuZCB0aGUgY2x1c3RlciB3aWxsIGJlIGNvcnJ1cHQuIFByb3ZpZGUKKyMg
b25seSB0aGUgV0FMIHNlZ21lbnQgdGhhdCByZWNvdmVyeSB3aWxsIHRoaW5rIGl0IG5lZWRz
LgorIworIyBUaGUgcG9pbnQgb2YgdGhpcyB0ZXN0IGlzIHRvIGV4cGxpY2l0bHkgZGVtb25z
dHJhdGUgdGhhdCBiYWNrdXBfbGFiZWwgaXMKKyMgYmVpbmcgdXNlZCBpbiBhIGxhdGVyIHRl
c3QgdG8gZ2V0IHRoZSBjb3JyZWN0IHJlY292ZXJ5IGluZm8uCitteSAkbm9kZV9yZXBsaWNh
ID0gUG9zdGdyZVNRTDo6VGVzdDo6Q2x1c3Rlci0+bmV3KCdyZXBsaWNhX2ZhaWwnKTsKKyRu
b2RlX3JlcGxpY2EtPmluaXRfZnJvbV9iYWNrdXAoJG5vZGVfcHJpbWFyeSwgJGJhY2t1cF9u
YW1lKTsKK215ICRjYW5hcnlfcXVlcnkgPSAic2VsZWN0IGNvdW50KCopIGZyb20gcGdfY2xh
c3Mgd2hlcmUgcmVsbmFtZSA9ICdjYW5hcnknIjsKKworY29weSgkbm9kZV9wcmltYXJ5LT5h
cmNoaXZlX2RpcigpIC4gJy8wMDAwMDAwMTAwMDAwMDAwMDAwMDAwMDMnLAorCQkkbm9kZV9y
ZXBsaWNhLT5kYXRhX2RpcigpIC4gJy9wZ193YWwvMDAwMDAwMDEwMDAwMDAwMDAwMDAwMDAz
JykKKwlvciBCQUlMX09VVCgidW5hYmxlIHRvIGNvcHkgMDAwMDAwMDEwMDAwMDAwMDAwMDAw
MDAzIik7CisKKyRub2RlX3JlcGxpY2EtPnN0YXJ0OworCitvaygkbm9kZV9yZXBsaWNhLT5z
YWZlX3BzcWwoJ3Bvc3RncmVzJywgJGNhbmFyeV9xdWVyeSkgPT0gMCwKKwknY2FuYXJ5IGlz
IG1pc3NpbmcnKTsKKworJG5vZGVfcmVwbGljYS0+dGVhcmRvd25fbm9kZSgpOworJG5vZGVf
cmVwbGljYS0+Y2xlYW5fbm9kZSgpOworCisjIFNhdmUgYmFja3VwX2xhYmVsIGludG8gdGhl
IGJhY2t1cCBkaXJlY3RvcnkgYW5kIHJlY292ZXIgdXNpbmcgdGhlIHByaW1hcnkncworIyBh
cmNoaXZlLiBUaGlzIHRpbWUgcmVjb3Zlcnkgd2lsbCBzdWNjZWVkIGFuZCB0aGUgY2FuYXJ5
IHRhYmxlIHdpbGwgYmUKKyMgcHJlc2VudC4KK2FwcGVuZF90b19maWxlKCIkYmFja3VwX2Rp
ci9iYWNrdXBfbGFiZWwiLCAkYmFja3VwX2xhYmVsKTsKKworJG5vZGVfcmVwbGljYSA9IFBv
c3RncmVTUUw6OlRlc3Q6OkNsdXN0ZXItPm5ldygncmVwbGljYV9zdWNjZXNzJyk7Ciskbm9k
ZV9yZXBsaWNhLT5pbml0X2Zyb21fYmFja3VwKCRub2RlX3ByaW1hcnksICRiYWNrdXBfbmFt
ZSwKKwloYXNfcmVzdG9yaW5nID0+IDEpOworJG5vZGVfcmVwbGljYS0+c3RhcnQ7CisKK29r
KCRub2RlX3JlcGxpY2EtPmxvZ19jb250YWlucygnY29tcGxldGVkIGJhY2t1cCByZWNvdmVy
eSB3aXRoIHJlZG8gTFNOJyksCisJJ3ZlcmlmeSBiYWNrdXAgcmVjb3ZlcnkgcGVyZm9ybWVk
IHdpdGggYmFja3VwX2xhYmVsJyk7CisKK29rKCRub2RlX3JlcGxpY2EtPnNhZmVfcHNxbCgn
cG9zdGdyZXMnLCAkY2FuYXJ5X3F1ZXJ5KSA9PSAxLAorCSdjYW5hcnkgaXMgcHJlc2VudCcp
OworCitkb25lX3Rlc3RpbmcoKTsKLS0gCjIuMzQuMQoK

--------------psc8Sb5V6lNfMWorPQ1DVB3L--