Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pwNSo-00021p-6Z for pgsql-hackers@arkaria.postgresql.org; Tue, 09 May 2023 13:33:10 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1pwNSn-0004UE-6m for pgsql-hackers@arkaria.postgresql.org; Tue, 09 May 2023 13:33:09 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pwNSm-0004U5-St for pgsql-hackers@lists.postgresql.org; Tue, 09 May 2023 13:33:08 +0000 Received: from oss.nttdata.com ([49.212.34.109]) by magus.postgresql.org with esmtps (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1pwNSj-001aJd-9s for pgsql-hackers@postgresql.org; Tue, 09 May 2023 13:33:08 +0000 Received: from oss.nttdata.com (localhost [127.0.0.1]) by oss.nttdata.com (Postfix) with ESMTPA id 369E4605CE; Tue, 9 May 2023 22:32:59 +0900 (JST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.8 at oss.nttdata.com MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=_8819b64adc77a10a25b5c44e19bc1000" Content-Transfer-Encoding: 7bit Date: Tue, 09 May 2023 22:32:59 +0900 From: torikoshia To: Michael Paquier , Kyotaro Horiguchi Cc: pgsql-hackers@postgresql.org Subject: Re: Allow pg_archivecleanup to remove backup history files In-Reply-To: References: <20230425.172948.1630351042955772893.horikyota.ntt@gmail.com> User-Agent: Roundcube Webmail/1.4.11 Message-ID: <5adce8cd5ad2269fe687d2d56a341c88@oss.nttdata.com> X-Sender: torikoshia@oss.nttdata.com List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --=_8819b64adc77a10a25b5c44e19bc1000 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII; format=flowed Horiguchi-san, Michael-san Thanks for your comments and information! Attached a patch with documentation and regression tests. On 2023-04-26 06:39, Michael Paquier wrote: > On Tue, Apr 25, 2023 at 05:29:48PM +0900, Kyotaro Horiguchi wrote: >> I thought that we have decided not to do that, but I coundn't find any >> discussion about it in the ML archive. Anyway, I think it is great >> that we have that option. > > No objections from here to make that optional. It's been argued for a > couple of times that leaving the archive history files is good for > debugging, but I can also get why one would one to clean up all the > files older than a WAL retention policy. Even if these are just few > bytes, it can be noisy for the eye to see a large accumulation of > history files mixed with the WAL segments. > -- > Michael -- Regards, -- Atsushi Torikoshi NTT DATA CORPORATION --=_8819b64adc77a10a25b5c44e19bc1000 Content-Transfer-Encoding: base64 Content-Type: text/x-diff; name=v2-0001-Allow-pg_archivecleanup-to-remove-backuphistoryfile.patch Content-Disposition: attachment; filename*0*=UTF-8''v2-0001-Allow-pg_archivecleanup-to-remove-backuphistor; filename*1*=yfile.patch; size=6086 RnJvbSA3ZDQ0MTM0ZTVkZTkzMGNlMDQ4MTkyODVhNWI3MzU5ZTM3MDcwOGQ0IE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiBBdHN1c2hpIFRvcmlrb3NoaSA8dG9yaWtvc2hpYUBvc3MubnR0 ZGF0YS5jb20+CkRhdGU6IFR1ZSwgOSBNYXkgMjAyMyAyMTo1MjowMSArMDkwMApTdWJqZWN0OiBb UEFUQ0ggdjJdIEFsbG93IHBnX2FyY2hpdmVjbGVhbnVwIHRvIHJlbW92ZSBiYWNrdXAgaGlzdG9y eSBmaWxlcwoKQWRkIG5ldyBvcHRpb24gLWIgdG8gcmVtb3ZlIGZpbGVzIGluY2x1ZGluZyBiYWNr dXAgaGlzdG9yeSBmaWxlcyBvbGRlcgp0aGFuIG9sZGVzdGtlcHR3YWxmaWxlLgoKLS0tCiBkb2Mv c3JjL3NnbWwvcmVmL3BnYXJjaGl2ZWNsZWFudXAuc2dtbCAgICAgICAgfCAxMSArKysrKwogc3Jj L2Jpbi9wZ19hcmNoaXZlY2xlYW51cC9wZ19hcmNoaXZlY2xlYW51cC5jIHwgMTEgKysrKy0KIC4u Li90LzAxMF9wZ19hcmNoaXZlY2xlYW51cC5wbCAgICAgICAgICAgICAgICB8IDQ1ICsrKysrKysr KysrKysrKysrLS0KIDMgZmlsZXMgY2hhbmdlZCwgNjIgaW5zZXJ0aW9ucygrKSwgNSBkZWxldGlv bnMoLSkKCmRpZmYgLS1naXQgYS9kb2Mvc3JjL3NnbWwvcmVmL3BnYXJjaGl2ZWNsZWFudXAuc2dt bCBiL2RvYy9zcmMvc2dtbC9yZWYvcGdhcmNoaXZlY2xlYW51cC5zZ21sCmluZGV4IDYzNWU3Yzc2 ODUuLjZjYjU2NTE1NmYgMTAwNjQ0Ci0tLSBhL2RvYy9zcmMvc2dtbC9yZWYvcGdhcmNoaXZlY2xl YW51cC5zZ21sCisrKyBiL2RvYy9zcmMvc2dtbC9yZWYvcGdhcmNoaXZlY2xlYW51cC5zZ21sCkBA IC05Myw2ICs5MywxNyBAQCBwZ19hcmNoaXZlY2xlYW51cDogIHJlbW92aW5nIGZpbGUgImFyY2hp dmUvMDAwMDAwMDEwMDAwMDAzNzAwMDAwMDBFIgogCiAgICAgPHZhcmlhYmxlbGlzdD4KIAorICAg ICA8dmFybGlzdGVudHJ5PgorICAgICAgPHRlcm0+PG9wdGlvbj4tYjwvb3B0aW9uPjwvdGVybT4K KyAgICAgIDxsaXN0aXRlbT4KKyAgICAgICA8cGFyYT4KKyAgICAgICAgIFJlbW92ZSBmaWxlcyBp bmNsdWRpbmcgYmFja3VwIGhpc3RvcnkgZmlsZXMsIHdob3NlIHN1ZmZpeCBpcyA8ZmlsZW5hbWU+ LmJhY2t1cDwvZmlsZW5hbWU+LgorICAgICAgICAgTm90ZSB0aGF0IHdoZW4gPHJlcGxhY2VhYmxl Pm9sZGVzdGtlcHR3YWxmaWxlPC9yZXBsYWNlYWJsZT4gaXMgYSBiYWNrdXAgaGlzdG9yeSBmaWxl LAorICAgICAgICAgc3BlY2lmaWVkIGZpbGUgaXMga2VwdCBhbmQgb25seSBwcmVjZWRpbmcgV0FM IGZpbGVzIGFuZCBiYWNrdXAgaGlzdG9yeSBmaWxlcyBhcmUgcmVtb3ZlZC4KKyAgICAgICA8L3Bh cmE+CisgICAgICA8L2xpc3RpdGVtPgorICAgICA8L3Zhcmxpc3RlbnRyeT4KKwogICAgICA8dmFy bGlzdGVudHJ5PgogICAgICAgPHRlcm0+PG9wdGlvbj4tZDwvb3B0aW9uPjwvdGVybT4KICAgICAg IDxsaXN0aXRlbT4KZGlmZiAtLWdpdCBhL3NyYy9iaW4vcGdfYXJjaGl2ZWNsZWFudXAvcGdfYXJj aGl2ZWNsZWFudXAuYyBiL3NyYy9iaW4vcGdfYXJjaGl2ZWNsZWFudXAvcGdfYXJjaGl2ZWNsZWFu dXAuYwppbmRleCA3NzI2ZDA1MTQ5Li5hZTZhZTc2ZjA4IDEwMDY0NAotLS0gYS9zcmMvYmluL3Bn X2FyY2hpdmVjbGVhbnVwL3BnX2FyY2hpdmVjbGVhbnVwLmMKKysrIGIvc3JjL2Jpbi9wZ19hcmNo aXZlY2xlYW51cC9wZ19hcmNoaXZlY2xlYW51cC5jCkBAIC0yMyw2ICsyMyw4IEBAIGNvbnN0IGNo YXIgKnByb2duYW1lOwogCiAvKiBPcHRpb25zIGFuZCBkZWZhdWx0cyAqLwogYm9vbAkJZHJ5cnVu ID0gZmFsc2U7CQkvKiBhcmUgd2UgcGVyZm9ybWluZyBhIGRyeS1ydW4gb3BlcmF0aW9uPyAqLwor Ym9vbAkJcmVtb3ZlQmFja3VwSGlzdG9yeUZpbGUgPSBmYWxzZTsJLyogcmVtb3ZlIGZpbGVzIGlu Y2x1ZGluZworCQkJCQkJCQkJCQkJICogYmFja3VwIGhpc3RvcnkgZmlsZXMgKi8KIGNoYXIJICAg KmFkZGl0aW9uYWxfZXh0ID0gTlVMTDsJLyogRXh0ZW5zaW9uIHRvIHJlbW92ZSBmcm9tIGZpbGVu YW1lcyAqLwogCiBjaGFyCSAgICphcmNoaXZlTG9jYXRpb247CS8qIHdoZXJlIHRvIGZpbmQgdGhl IGFyY2hpdmU/ICovCkBAIC0xMTgsNyArMTIwLDggQEAgQ2xlYW51cFByaW9yV0FMRmlsZXModm9p ZCkKIAkJCSAqIGZpbGUuIE5vdGUgdGhhdCB0aGlzIG1lYW5zIGZpbGVzIGFyZSBub3QgcmVtb3Zl ZCBpbiB0aGUgb3JkZXIKIAkJCSAqIHRoZXkgd2VyZSBvcmlnaW5hbGx5IHdyaXR0ZW4sIGluIGNh c2UgdGhpcyB3b3JyaWVzIHlvdS4KIAkJCSAqLwotCQkJaWYgKChJc1hMb2dGaWxlTmFtZSh3YWxm aWxlKSB8fCBJc1BhcnRpYWxYTG9nRmlsZU5hbWUod2FsZmlsZSkpICYmCisJCQlpZiAoKChJc1hM b2dGaWxlTmFtZSh3YWxmaWxlKSB8fCBJc1BhcnRpYWxYTG9nRmlsZU5hbWUod2FsZmlsZSkpIHx8 CisJCQkJKHJlbW92ZUJhY2t1cEhpc3RvcnlGaWxlICYmIElzQmFja3VwSGlzdG9yeUZpbGVOYW1l KHdhbGZpbGUpKSkgJiYKIAkJCQlzdHJjbXAod2FsZmlsZSArIDgsIGV4Y2x1c2l2ZUNsZWFudXBG aWxlTmFtZSArIDgpIDwgMCkKIAkJCXsKIAkJCQljaGFyCQlXQUxGaWxlUGF0aFtNQVhQR1BBVEgg KiAyXTsgLyogdGhlIGZpbGUgcGF0aApAQCAtMjUyLDYgKzI1NSw3IEBAIHVzYWdlKHZvaWQpCiAJ cHJpbnRmKF8oIlVzYWdlOlxuIikpOwogCXByaW50ZihfKCIgICVzIFtPUFRJT05dLi4uIEFSQ0hJ VkVMT0NBVElPTiBPTERFU1RLRVBUV0FMRklMRVxuIiksIHByb2duYW1lKTsKIAlwcmludGYoXygi XG5PcHRpb25zOlxuIikpOworCXByaW50ZihfKCIgIC1iICAgICAgICAgICAgIHJlbW92ZSBmaWxl cyBpbmNsdWRpbmcgYmFja3VwIGhpc3RvcnkgZmlsZXNcbiIpKTsKIAlwcmludGYoXygiICAtZCAg ICAgICAgICAgICBnZW5lcmF0ZSBkZWJ1ZyBvdXRwdXQgKHZlcmJvc2UgbW9kZSlcbiIpKTsKIAlw cmludGYoXygiICAtbiAgICAgICAgICAgICBkcnkgcnVuLCBzaG93IHRoZSBuYW1lcyBvZiB0aGUg ZmlsZXMgdGhhdCB3b3VsZCBiZSByZW1vdmVkXG4iKSk7CiAJcHJpbnRmKF8oIiAgLVYsIC0tdmVy c2lvbiAgb3V0cHV0IHZlcnNpb24gaW5mb3JtYXRpb24sIHRoZW4gZXhpdFxuIikpOwpAQCAtMjk0 LDEwICsyOTgsMTMgQEAgbWFpbihpbnQgYXJnYywgY2hhciAqKmFyZ3YpCiAJCX0KIAl9CiAKLQl3 aGlsZSAoKGMgPSBnZXRvcHQoYXJnYywgYXJndiwgImRueDoiKSkgIT0gLTEpCisJd2hpbGUgKChj ID0gZ2V0b3B0KGFyZ2MsIGFyZ3YsICJiZG54OiIpKSAhPSAtMSkKIAl7CiAJCXN3aXRjaCAoYykK IAkJeworCQkJY2FzZSAnYic6IAkJCS8qIFJlbW92ZSBiYWNrdXAgaGlzdG9yeSBmaWxlcyB0b28g Ki8KKwkJCQlyZW1vdmVCYWNrdXBIaXN0b3J5RmlsZSA9IHRydWU7CisJCQkJYnJlYWs7CiAJCQlj YXNlICdkJzoJCQkvKiBEZWJ1ZyBtb2RlICovCiAJCQkJcGdfbG9nZ2luZ19pbmNyZWFzZV92ZXJi b3NpdHkoKTsKIAkJCQlicmVhazsKZGlmZiAtLWdpdCBhL3NyYy9iaW4vcGdfYXJjaGl2ZWNsZWFu dXAvdC8wMTBfcGdfYXJjaGl2ZWNsZWFudXAucGwgYi9zcmMvYmluL3BnX2FyY2hpdmVjbGVhbnVw L3QvMDEwX3BnX2FyY2hpdmVjbGVhbnVwLnBsCmluZGV4IDc2MzIxZDEyODQuLmFmZDJmZjYyMDkg MTAwNjQ0Ci0tLSBhL3NyYy9iaW4vcGdfYXJjaGl2ZWNsZWFudXAvdC8wMTBfcGdfYXJjaGl2ZWNs ZWFudXAucGwKKysrIGIvc3JjL2Jpbi9wZ19hcmNoaXZlY2xlYW51cC90LzAxMF9wZ19hcmNoaXZl Y2xlYW51cC5wbApAQCAtMTYsOSArMTYsMTQgQEAgbXkgQHdhbGZpbGVzID0gKAogCScwMDAwMDAw MTAwMDAwMDM3MDAwMDAwMEMuZ3onLCAnMDAwMDAwMDEwMDAwMDAzNzAwMDAwMDBEJywKIAknMDAw MDAwMDEwMDAwMDAzNzAwMDAwMDBFJywgICAgJzAwMDAwMDAxMDAwMDAwMzcwMDAwMDAwRi5wYXJ0 aWFsJywpOwogCitteSBAd2FsZmlsZXNfd2l0aF9iYWNrdXBoaXN0b3J5ZmlsZSA9ICgKKwknMDAw MDAwMDIwMDAwMDAzODAwMDAwMDA3LjAwMDAwMDI4LmJhY2t1cCcsICcwMDAwMDAwMjAwMDAwMDM4 MDAwMDAwMDgnLAorCScwMDAwMDAwMjAwMDAwMDM4MDAwMDAwMDknLAorCScwMDAwMDAwMjAwMDAw MDM4MDAwMDAwMEEnLCAnMDAwMDAwMDIwMDAwMDAzODAwMDAwMDBCLjAwN0M5MzMwLmJhY2t1cCcp OworCiBzdWIgY3JlYXRlX2ZpbGVzCiB7Ci0JZm9yZWFjaCBteSAkZm4gKEB3YWxmaWxlcywgJ3Vu cmVsYXRlZF9maWxlJykKKwlmb3JlYWNoIG15ICRmbiAoQF8sICd1bnJlbGF0ZWRfZmlsZScpCiAJ ewogCQlvcGVuIG15ICRmaWxlLCAnPicsICIkdGVtcGRpci8kZm4iOwogCQlwcmludCAkZmlsZSAn Q09OVEVOVCc7CkBAIC0yNyw3ICszMiw3IEBAIHN1YiBjcmVhdGVfZmlsZXMKIAlyZXR1cm47CiB9 CiAKLWNyZWF0ZV9maWxlcygpOworY3JlYXRlX2ZpbGVzKEB3YWxmaWxlcyk7CiAKIGNvbW1hbmRf ZmFpbHNfbGlrZSgKIAlbJ3BnX2FyY2hpdmVjbGVhbnVwJ10sCkBAIC03Niw3ICs4MSw3IEBAIHN1 YiBydW5fY2hlY2sKIAogCW15ICgkc3VmZml4LCAkdGVzdF9uYW1lKSA9IEBfOwogCi0JY3JlYXRl X2ZpbGVzKCk7CisJY3JlYXRlX2ZpbGVzKEB3YWxmaWxlcyk7CiAKIAljb21tYW5kX29rKAogCQlb CkBAIC05OCw4ICsxMDMsNDIgQEAgc3ViIHJ1bl9jaGVjawogCXJldHVybjsKIH0KIAorc3ViIHJl bW92ZV9iYWNrdXBoaXN0b3J5ZmlsZV9ydW5fY2hlY2sKK3sKKwlsb2NhbCAkVGVzdDo6QnVpbGRl cjo6TGV2ZWwgPSAkVGVzdDo6QnVpbGRlcjo6TGV2ZWwgKyAxOworCisJbXkgKCRzdWZmaXgsICR0 ZXN0X25hbWUpID0gQF87CisKKwljcmVhdGVfZmlsZXMoQHdhbGZpbGVzX3dpdGhfYmFja3VwaGlz dG9yeWZpbGUpOworCisJY29tbWFuZF9vaygKKwkJWworCQkJJ3BnX2FyY2hpdmVjbGVhbnVwJywg Jy1iJywgJHRlbXBkaXIsCisJCQkkd2FsZmlsZXNfd2l0aF9iYWNrdXBoaXN0b3J5ZmlsZVsyXSAu ICRzdWZmaXgKKwkJXSwKKwkJIiR0ZXN0X25hbWU6IHJlbW92ZV9iYWNrdXBoaXN0b3J5ZmlsZV9y dW5zIik7CisKKwlvayghLWYgIiR0ZW1wZGlyLyR3YWxmaWxlc193aXRoX2JhY2t1cGhpc3Rvcnlm aWxlWzBdIiwKKwkJIiR0ZXN0X25hbWU6IGZpcnN0IC5iYWNrdXAgZmlsZSB3YXMgY2xlYW5lZCB1 cCIpOworCW9rKCEtZiAiJHRlbXBkaXIvJHdhbGZpbGVzX3dpdGhfYmFja3VwaGlzdG9yeWZpbGVb MV0iLAorCQkiJHRlc3RfbmFtZTogc2Vjb25kIG9sZGVyIFdBTCBmaWxlIHdhcyBjbGVhbmVkIHVw Iik7CisJb2soLWYgIiR0ZW1wZGlyLyR3YWxmaWxlc193aXRoX2JhY2t1cGhpc3RvcnlmaWxlWzJd IiwKKwkJIiR0ZXN0X25hbWU6IHJlc3RhcnRmaWxlIHdhcyBub3QgY2xlYW5lZCB1cCIpOworCW9r KC1mICIkdGVtcGRpci8kd2FsZmlsZXNfd2l0aF9iYWNrdXBoaXN0b3J5ZmlsZVszXSIsCisJCSIk dGVzdF9uYW1lOiBuZXdlciBXQUwgZmlsZSB3YXMgbm90IGNsZWFuZWQgdXAiKTsKKwlvaygtZiAi JHRlbXBkaXIvJHdhbGZpbGVzX3dpdGhfYmFja3VwaGlzdG9yeWZpbGVbNF0iLAorCQkiJHRlc3Rf bmFtZTogbmV3ZXIgLmJhY2t1cCBmaWxlIHdhcyBub3QgY2xlYW5lZCB1cCIpOworCW9rKC1mICIk dGVtcGRpci91bnJlbGF0ZWRfZmlsZSIsCisJCSIkdGVzdF9uYW1lOiB1bnJlbGF0ZWQgZmlsZSB3 YXMgbm90IGNsZWFuZWQgdXAiKTsKKwlyZXR1cm47Cit9CisKIHJ1bl9jaGVjaygnJywgICAgICAg ICAgICAgICAgICdwZ19hcmNoaXZlY2xlYW51cCcpOwogcnVuX2NoZWNrKCcucGFydGlhbCcsICAg ICAgICAgJ3BnX2FyY2hpdmVjbGVhbnVwIHdpdGggLnBhcnRpYWwgZmlsZScpOwogcnVuX2NoZWNr KCcuMDAwMDAwMjAuYmFja3VwJywgJ3BnX2FyY2hpdmVjbGVhbnVwIHdpdGggLmJhY2t1cCBmaWxl Jyk7CiAKK3JlbW92ZV9iYWNrdXBoaXN0b3J5ZmlsZV9ydW5fY2hlY2soJycsCQkJCSAgICdwZ19h cmNoaXZlY2xlYW51cCAtYicpOworcmVtb3ZlX2JhY2t1cGhpc3RvcnlmaWxlX3J1bl9jaGVjaygn LjAwMDAwMDIwLmJhY2t1cCcsCisJCQkJCQkJCQkncGdfYXJjaGl2ZWNsZWFudXAgLWIgd2l0aCAu YmFja3VwIGZpbGUnKTsKKwogZG9uZV90ZXN0aW5nKCk7CgpiYXNlLWNvbW1pdDogYzViN2Y2N2Zj YzhjNGEwMWM4MjY2MGViMDk5NmEzYzY5N2ZhYzI4MwotLSAKMi4yNS4xCgo= --=_8819b64adc77a10a25b5c44e19bc1000--