Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1rsg8g-008Ynb-6a for pgsql-hackers@arkaria.postgresql.org; Fri, 05 Apr 2024 09:45:39 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1rsg8f-00ASje-8A for pgsql-hackers@arkaria.postgresql.org; Fri, 05 Apr 2024 09:45:37 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1rsg8e-00AShh-SL for pgsql-hackers@lists.postgresql.org; Fri, 05 Apr 2024 09:45:36 +0000 Received: from mail-40135.protonmail.ch ([185.70.40.135]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1rsg8a-000lW6-3u for pgsql-hackers@lists.postgresql.org; Fri, 05 Apr 2024 09:45:36 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=protonmail.com; s=protonmail3; t=1712310330; x=1712569530; bh=ZE0WpZHZ39zjzLCr1eZCBeeXJD5YAF0Oll7DKpGcRfU=; h=Date:To:From:Subject:Message-ID:Feedback-ID:From:To:Cc:Date: Subject:Reply-To:Feedback-ID:Message-ID:BIMI-Selector; b=ldlvfpUmIZxui6DZeAnbMTd1udS152KEGUop6UqlkifBkj3b3b2v/Lr+dN95pAJOt cfMDTwhngVnLjd4qhPYItYCyd4hwTGL9BwdnnXi0vm1QYYzHgVWjO5OeHUpZWs+qbU r/J6kVizW7JGGJQh1nl8KG5UX5xhnfvfZoWdRbcPLaChyy2VWdSuRUlfUmwCXoAKKb MT97BZl5bmDmg6L1/XFjWHuBixkSY1zRYZ+ysbkR2c/aZ8FcG2Kin7dS+UIz6kQ6yi 7TS6ZypSQrixi2iWIIiAzxith2d9MBU7i4M7MIn4wxQyWOrpypIkrkjLGTqYFGqJnC T1uVsNBNcN9Jg== Date: Fri, 05 Apr 2024 09:45:18 +0000 To: "pgsql-hackers@lists.postgresql.org" From: Stefan Fercot Subject: Recovery of .partial WAL segments Message-ID: Feedback-ID: 39085994:user:proton MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="b1_iJIuWeg09Oh3y43SUVo4LConPhpcCMrTyYlOi3FE2Y" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk This is a multi-part message in MIME format. --b1_iJIuWeg09Oh3y43SUVo4LConPhpcCMrTyYlOi3FE2Y Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Dear hackers, Generating a ".partial" WAL segment is pretty common nowadays (using pg_rec= eivewal or during standby promotion). However, we currently don't do anything with it unless the user manually re= moves that ".partial" extension. The 028_pitr_timelines tests are highlighting that fact: with test data bei= ng being in 000000020000000000000003 and 000000010000000000000003.partial, = a recovery following the latest timeline (2) will succeed but fail if we fo= llow the current timeline (1). By simply trying to fetch the ".partial" file in XLogFileRead, we can easil= y recover more data and also cover that (current timeline) recovery case. So, this proposed patch makes XLogFileRead try to restore ".partial" WAL ar= chives and adds a test to 028_pitr_timelines using current recovery_target_= timeline. As far as I've seen, the current pg_receivewal tests only seem to cover the= archives generation but not actually trying to recover using it. I wasn't = sure it was interesting to add such tests right now, so I didn't considered= it for this patch. Many thanks in advance for your feedback and thoughts about this, Kind Regards, -- Stefan FERCOT Data Egret (https://dataegret.com) --b1_iJIuWeg09Oh3y43SUVo4LConPhpcCMrTyYlOi3FE2Y Content-Type: text/x-patch; name=v1-0001-Make-XLogFileRead-try-to-restore-.partial-wal-arc.patch Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename=v1-0001-Make-XLogFileRead-try-to-restore-.partial-wal-arc.patch RnJvbSA4YzczMjg0YjcyMTIwZGRmMzUzN2UxNjMyZjEyNDU1NTAyZDQ3ZjZkIE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiBTdGVmYW4gRmVyY290IDxwZ3N0ZWZAZmVyY290LmJlPgpEYXRl OiBGcmksIDUgQXByIDIwMjQgMTA6NTc6MDMgKzAyMDAKU3ViamVjdDogW1BBVENIIHYxLl0gTWFr ZSBYTG9nRmlsZVJlYWQgdHJ5IHRvIHJlc3RvcmUgLnBhcnRpYWwgd2FsIGFyY2hpdmVzCgpUcnkg dG8gcmVzdG9yZSB0aGUgbm9ybWFsIGFyY2hpdmVkIHdhbCBzZWdtZW50IGZpcnN0IGFuZCwgaWYg bm90IGZvdW5kLCB0aGVuIHRyeSB0byByZXN0b3JlIHRoZSBhcmNoaXZlZCAucGFydGlhbCB3YWwg c2VnbWVudC4KVGhpcyBpcyBzYWZlIGJlY2F1c2UgdGhlIG5leHQgY29tcGxldGVkIHdhbCBzZWdt ZW50IHNob3VsZCBjb250YWluIGF0IGxlYXN0IHRoZSBzYW1lIGRhdGEuCi0tLQogc3JjL2JhY2tl bmQvYWNjZXNzL3RyYW5zYW0veGxvZ3JlY292ZXJ5LmMgfCAyMCArKysrKysrKysrLS0KIHNyYy90 ZXN0L3JlY292ZXJ5L3QvMDI4X3BpdHJfdGltZWxpbmVzLnBsIHwgMzcgKysrKysrKysrKysrKysr KysrKysrKy0KIDIgZmlsZXMgY2hhbmdlZCwgNTQgaW5zZXJ0aW9ucygrKSwgMyBkZWxldGlvbnMo LSkKCmRpZmYgLS1naXQgYS9zcmMvYmFja2VuZC9hY2Nlc3MvdHJhbnNhbS94bG9ncmVjb3Zlcnku YyBiL3NyYy9iYWNrZW5kL2FjY2Vzcy90cmFuc2FtL3hsb2dyZWNvdmVyeS5jCmluZGV4IGIyZmUy ZDA0Y2MuLjZkNTFiNmEyOTYgMTAwNjQ0Ci0tLSBhL3NyYy9iYWNrZW5kL2FjY2Vzcy90cmFuc2Ft L3hsb2dyZWNvdmVyeS5jCisrKyBiL3NyYy9iYWNrZW5kL2FjY2Vzcy90cmFuc2FtL3hsb2dyZWNv dmVyeS5jCkBAIC00MjA2LDYgKzQyMDYsOCBAQCBYTG9nRmlsZVJlYWQoWExvZ1NlZ05vIHNlZ25v LCBpbnQgZW1vZGUsIFRpbWVMaW5lSUQgdGxpLAogCWNoYXIJCWFjdGl2aXR5bXNnW01BWEZOQU1F TEVOICsgMTZdOwogCWNoYXIJCXBhdGhbTUFYUEdQQVRIXTsKIAlpbnQJCQlmZDsKKwljaGFyCSAg ICpwYXJ0aWFseGxvZ2ZuYW1lOworCWJvb2wJCXJlc3RvcmVkQXJjaGl2ZWRGaWxlOwogCiAJWExv Z0ZpbGVOYW1lKHhsb2dmbmFtZSwgdGxpLCBzZWdubywgd2FsX3NlZ21lbnRfc2l6ZSk7CiAKQEAg LTQyMTcsMTAgKzQyMTksMjQgQEAgWExvZ0ZpbGVSZWFkKFhMb2dTZWdObyBzZWdubywgaW50IGVt b2RlLCBUaW1lTGluZUlEIHRsaSwKIAkJCQkJIHhsb2dmbmFtZSk7CiAJCQlzZXRfcHNfZGlzcGxh eShhY3Rpdml0eW1zZyk7CiAKLQkJCWlmICghUmVzdG9yZUFyY2hpdmVkRmlsZShwYXRoLCB4bG9n Zm5hbWUsCisJCQkvKgorCQkJICogVHJ5IHRvIHJlc3RvcmUgdGhlIG5vcm1hbCB3YWwgc2VnbWVu dCBmaXJzdCBhbmQsIGlmIG5vdCBmb3VuZCwKKwkJCSAqIHRoZW4gdHJ5IHRvIHJlc3RvcmUgdGhl IC5wYXJ0aWFsIHdhbCBzZWdtZW50LgorCQkJICovCisKKwkJCXBhcnRpYWx4bG9nZm5hbWUgPSBw c3ByaW50ZigiJXMucGFydGlhbCIsIHhsb2dmbmFtZSk7CisKKwkJCXJlc3RvcmVkQXJjaGl2ZWRG aWxlID0gIVJlc3RvcmVBcmNoaXZlZEZpbGUocGF0aCwgeGxvZ2ZuYW1lLAorCQkJCQkJCQkJCQkJ CQkiUkVDT1ZFUllYTE9HIiwKKwkJCQkJCQkJCQkJCQkJd2FsX3NlZ21lbnRfc2l6ZSwKKwkJCQkJ CQkJCQkJCQkJSW5SZWRvKSAmJgorCQkJCSFSZXN0b3JlQXJjaGl2ZWRGaWxlKHBhdGgsIHBhcnRp YWx4bG9nZm5hbWUsCiAJCQkJCQkJCQkgIlJFQ09WRVJZWExPRyIsCiAJCQkJCQkJCQkgd2FsX3Nl Z21lbnRfc2l6ZSwKLQkJCQkJCQkJCSBJblJlZG8pKQorCQkJCQkJCQkJIEluUmVkbyk7CisKKwkJ CXBmcmVlKHBhcnRpYWx4bG9nZm5hbWUpOworCQkJaWYgKHJlc3RvcmVkQXJjaGl2ZWRGaWxlKQog CQkJCXJldHVybiAtMTsKIAkJCWJyZWFrOwogCmRpZmYgLS1naXQgYS9zcmMvdGVzdC9yZWNvdmVy eS90LzAyOF9waXRyX3RpbWVsaW5lcy5wbCBiL3NyYy90ZXN0L3JlY292ZXJ5L3QvMDI4X3BpdHJf dGltZWxpbmVzLnBsCmluZGV4IDRiN2Q4MjViNzEuLjUxMjY5NWNkMGEgMTAwNjQ0Ci0tLSBhL3Ny Yy90ZXN0L3JlY292ZXJ5L3QvMDI4X3BpdHJfdGltZWxpbmVzLnBsCisrKyBiL3NyYy90ZXN0L3Jl Y292ZXJ5L3QvMDI4X3BpdHJfdGltZWxpbmVzLnBsCkBAIC0xMTAsNyArMTEwLDcgQEAgJG5vZGVf c3RhbmRieS0+c3RvcDsKICMgc2VnbWVudCAwMDAwMDAwMjAwMDAwMDAwMDAwMDAwMDMsIGJlZm9y ZSB0aGUgdGltZWxpbmUgc3dpdGNoaW5nCiAjIHJlY29yZC4gIChUaGV5IGFyZSBhbHNvIHByZXNl bnQgaW4gdGhlCiAjIDAwMDAwMDAxMDAwMDAwMDAwMDAwMDAwMy5wYXJ0aWFsIGZpbGUsIGJ1dCAu cGFydGlhbCBmaWxlcyBhcmUgbm90Ci0jIHVzZWQgYXV0b21hdGljYWxseS4pCisjIHVzZWQgd2hl biByZWNvdmVyaW5nIGFsb25nIHRoZSBsYXRlc3QgdGltZWxpbmUgYnkgZGVmYXVsdC4pCiAKICMg Tm93IHRlc3QgUElUUiB0byB0aGUgcmVjb3ZlcnkgdGFyZ2V0LiAgSXQgc2hvdWxkIGZpbmQgdGhl IFdBTCBpbgogIyBzZWdtZW50IDAwMDAwMDAyMDAwMDAwMDAwMDAwMDAwMywgYnV0IG5vdCBmb2xs b3cgdGhlIHRpbWVsaW5lIHN3aXRjaApAQCAtMTczLDQgKzE3MywzOSBAQCAkbm9kZV9waXRyMi0+ cG9sbF9xdWVyeV91bnRpbCgncG9zdGdyZXMnLCAiU0VMRUNUIHBnX2lzX2luX3JlY292ZXJ5KCkg PSAnZic7IikKICRyZXN1bHQgPSAkbm9kZV9waXRyMi0+c2FmZV9wc3FsKCdwb3N0Z3JlcycsICJT RUxFQ1QgbWF4KGkpIEZST00gZm9vOyIpOwogaXMoJHJlc3VsdCwgcXF7M30sICJjaGVjayB0YWJs ZSBjb250ZW50cyBhZnRlciBwb2ludC1pbi10aW1lIHJlY292ZXJ5Iik7CiAKKyMgVGhlIDAwMDAw MDAxMDAwMDAwMDAwMDAwMDAwMy5wYXJ0aWFsIGZpbGUgY291bGQgaGF2ZSBiZWVuIGdlbmVyYXRl ZAorIyBieSBwZ19yZWNlaXZld2FsIHdpdGhvdXQgYW55IHN0YW5kYnkgbm9kZSBpbnZvbHZlZC4g SW4gdGhpcyBjYXNlLCB3ZQorIyB3b3VsZG4ndCBiZSBhYmxlIHRvIHJlY292ZXIgZnJvbSAwMDAw MDAwMjAwMDAwMDAwMDAwMDAwMDMuCisjIE5vdywgdGVzdCBQSVRSIHRvIHRoZSBpbml0aWFsIHJl Y292ZXJ5IHRhcmdldCBzdGF5aW5nIG9uIHRoZSBiYWNrdXAncworIyBjdXJyZW50IHRpbWVsaW5l LCB0cnlpbmcgdG8gZmV0Y2ggdGhlIGRhdGEgZnJvbSB0aGUKKyMgMDAwMDAwMDEwMDAwMDAwMDAw MDAwMDAzLnBhcnRpYWwgZmlsZS4KKworbXkgJG5vZGVfcGl0cjMgPSBQb3N0Z3JlU1FMOjpUZXN0 OjpDbHVzdGVyLT5uZXcoJ25vZGVfcGl0cjMnKTsKKyRub2RlX3BpdHIzLT5pbml0X2Zyb21fYmFj a3VwKAorCSRub2RlX3ByaW1hcnksICRiYWNrdXBfbmFtZSwKKwlzdGFuZGJ5ID0+IDAsCisJaGFz X3Jlc3RvcmluZyA9PiAxKTsKKyRub2RlX3BpdHIzLT5hcHBlbmRfY29uZigKKwkncG9zdGdyZXNx bC5jb25mJywgcXF7CityZWNvdmVyeV90YXJnZXRfbmFtZSA9ICdycCcKK3JlY292ZXJ5X3Rhcmdl dF9hY3Rpb24gPSAncHJvbW90ZScKK3JlY292ZXJ5X3RhcmdldF90aW1lbGluZSA9ICdjdXJyZW50 JworfSk7CisKK215ICRsb2dfb2Zmc2V0ID0gLXMgJG5vZGVfcGl0cjMtPmxvZ2ZpbGU7Ciskbm9k ZV9waXRyMy0+c3RhcnQ7CisKK29rKCAkbm9kZV9waXRyMy0+bG9nX2NvbnRhaW5zKAorCQkicmVz dG9yZWQgbG9nIGZpbGUgXCIwMDAwMDAwMTAwMDAwMDAwMDAwMDAwMDMucGFydGlhbFwiIGZyb20g YXJjaGl2ZSIsCisJCSRsb2dfb2Zmc2V0KSwKKwkicmVzdG9yZWQgMDAwMDAwMDEwMDAwMDAwMDAw MDAwMDAzLnBhcnRpYWwiKTsKKworIyBXYWl0IHVudGlsIHJlY292ZXJ5IGZpbmlzaGVzLgorJG5v ZGVfcGl0cjMtPnBvbGxfcXVlcnlfdW50aWwoJ3Bvc3RncmVzJywgIlNFTEVDVCBwZ19pc19pbl9y ZWNvdmVyeSgpID0gJ2YnOyIpCisgIG9yIGRpZSAiVGltZWQgb3V0IHdoaWxlIHdhaXRpbmcgZm9y IFBJVFIgcHJvbW90aW9uIjsKKworIyBDaGVjayB0aGF0IHdlIHNlZSB0aGUgZGF0YSB3ZSBleHBl Y3QuCiskcmVzdWx0ID0gJG5vZGVfcGl0cjMtPnNhZmVfcHNxbCgncG9zdGdyZXMnLCAiU0VMRUNU IG1heChpKSBGUk9NIGZvbzsiKTsKK2lzKCRyZXN1bHQsIHFxezF9LCAiY2hlY2sgdGFibGUgY29u dGVudHMgYWZ0ZXIgcG9pbnQtaW4tdGltZSByZWNvdmVyeSIpOworCiBkb25lX3Rlc3RpbmcoKTsK LS0gCjIuMzQuMQoK --b1_iJIuWeg09Oh3y43SUVo4LConPhpcCMrTyYlOi3FE2Y--