Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1jlDnW-000222-9N for pgsql-hackers@arkaria.postgresql.org; Tue, 16 Jun 2020 15:46:50 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1jlDnV-0007Tb-1F for pgsql-hackers@arkaria.postgresql.org; Tue, 16 Jun 2020 15:46:49 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1jlDnU-0007TT-Oz for pgsql-hackers@lists.postgresql.org; Tue, 16 Jun 2020 15:46:48 +0000 Received: from oss.nttdata.com ([49.212.34.109]) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1jlDnR-0007kq-Kc for pgsql-hackers@lists.postgresql.org; Tue, 16 Jun 2020 15:46:48 +0000 Received: from hnk.local (p2604028-ipngn12401funabasi.chiba.ocn.ne.jp [180.1.110.28]) by oss.nttdata.com (Postfix) with ESMTPSA id 7124E6020F; Wed, 17 Jun 2020 00:46:39 +0900 (JST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.102.3 at oss.nttdata.com Subject: Re: Review for GetWALAvailability() To: Kyotaro Horiguchi Cc: pgsql-hackers@lists.postgresql.org References: <8d55969f-ba37-b89a-6494-e9322ccdb35d@oss.nttdata.com> <20200615.134225.1144377461224317890.horikyota.ntt@gmail.com> <66c05b67-3396-042c-1b41-bfa6c3ddcf82@oss.nttdata.com> <20200616.140026.1624101556608079529.horikyota.ntt@gmail.com> From: Fujii Masao Message-ID: Date: Wed, 17 Jun 2020 00:46:38 +0900 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:68.0) Gecko/20100101 Thunderbird/68.9.0 MIME-Version: 1.0 In-Reply-To: <20200616.140026.1624101556608079529.horikyota.ntt@gmail.com> Content-Type: multipart/mixed; boundary="------------EC6C76FD7DD56EAB5A4B7D91" Content-Language: en-US List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk This is a multi-part message in MIME format. --------------EC6C76FD7DD56EAB5A4B7D91 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit On 2020/06/16 14:00, Kyotaro Horiguchi wrote: > At Tue, 16 Jun 2020 01:46:21 +0900, Fujii Masao wrote in >>> In short, it is known behavior but it was judged as useless to prevent >>> that. >>> That can happen when checkpointer removes up to the segment that is >>> being read by walsender. I think that that doesn't happen (or >>> happenswithin a narrow time window?) for physical replication but >>> happenes for logical replication. >>> While development, I once added walsender a code to exit for that >>> reason, but finally it is moved to InvalidateObsoleteReplicationSlots >>> as a bit defferent function. >> >> BTW, I read the code of InvalidateObsoleteReplicationSlots() and >> probably >> found some issues in it. >> >> 1. Each cycle of the "for" loop in >> InvalidateObsoleteReplicationSlots() >> emits the log message "terminating walsender ...". This means that >> if it takes more than 10ms for walsender to exit after it's signaled, >> the second and subsequent cycles would happen and output the same >> log message several times. IMO that log message should be output >> only once. > > Sounds reasonable. > >> 2. InvalidateObsoleteReplicationSlots() uses the loop to scan >> replication >> slots array and uses the "for" loop in each scan. Also it calls >> ReplicationSlotAcquire() for each "for" loop cycle, and >> ReplicationSlotAcquire() uses another loop to scan replication slots >> array. I don't think this is good design. >> >> ISTM that we can get rid of ReplicationSlotAcquire()'s loop because >> InvalidateObsoleteReplicationSlots() already know the index of the >> slot >> that we want to find. The attached patch does that. Thought? > > The inner loop is expected to run at most several times per > checkpoint, which won't be a serious problem. However, it is better if > we can get rid of that in a reasonable way. > > The attached patch changes the behavior for SAB_Block. Before the > patch, it rescans from the first slot for the same name, but with the > patch it just rechecks the same slot. The only caller of the function > with SAB_Block is ReplicationSlotDrop and I don't come up with a case > where another slot with the same name is created at different place > before the condition variable fires. But I'm not sure the change is > completely safe. Yes, that change might not be safe. So I'm thinking another approach to fix the issues. > Maybe some assertion is needed? > >> 3. There is a corner case where the termination of walsender cleans up >> the temporary replication slot while >> InvalidateObsoleteReplicationSlots() >> is sleeping on ConditionVariableTimedSleep(). In this case, >> ReplicationSlotAcquire() is called in the subsequent cycle of the >> "for" >> loop, cannot find the slot and then emits ERROR message. This leads >> to the failure of checkpoint by the checkpointer. > > Agreed. > >> To avoid this case, if SAB_Inquire is specified, >> ReplicationSlotAcquire() >> should return the special value instead of emitting ERROR even when >> it cannot find the slot. Also InvalidateObsoleteReplicationSlots() >> should >> handle that special returned value. > > I thought the same thing hearing that. While reading InvalidateObsoleteReplicationSlots() code, I found another issue. ereport(LOG, (errmsg("terminating walsender %d because replication slot \"%s\" is too far behind", wspid, NameStr(slotname)))); (void) kill(wspid, SIGTERM); wspid indicates the PID of process using the slot. That process can be a backend, for example, executing pg_replication_slot_advance(). So "walsender" in the above log message is not always correct. int wspid = ReplicationSlotAcquire(NameStr(slotname), SAB_Inquire); Why do we need to call ReplicationSlotAcquire() here and mark the slot as used by the checkpointer? Isn't it enough to check directly the slot's active_pid, instead? Maybe ReplicationSlotAcquire() is necessary because ReplicationSlotRelease() is called later? If so, why do we need to call ReplicationSlotRelease()? ISTM that we don't need to do that if the slot's active_pid is zero. No? If my understanding is right, I'd like to propose the attached patch. It introduces DeactivateReplicationSlot() and replace the "for" loop in InvalidateObsoleteReplicationSlots() with it. ReplicationSlotAcquire() and ReplicationSlotRelease() are no longer called there. Regards, -- Fujii Masao Advanced Computing Technology Center Research and Development Headquarters NTT DATA CORPORATION --------------EC6C76FD7DD56EAB5A4B7D91 Content-Type: text/plain; charset=UTF-8; x-mac-type="0"; x-mac-creator="0"; name="invalidate_obsolete_replication_slots.patch" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="invalidate_obsolete_replication_slots.patch" ZGlmZiAtLWdpdCBhL3NyYy9iYWNrZW5kL3JlcGxpY2F0aW9uL3Nsb3QuYyBiL3NyYy9iYWNr ZW5kL3JlcGxpY2F0aW9uL3Nsb3QuYwppbmRleCA1MDU0NDVmMmRjLi5iODliNmRhNzY4IDEw MDY0NAotLS0gYS9zcmMvYmFja2VuZC9yZXBsaWNhdGlvbi9zbG90LmMKKysrIGIvc3JjL2Jh Y2tlbmQvcmVwbGljYXRpb24vc2xvdC5jCkBAIC05OSw2ICs5OSw4IEBAIFJlcGxpY2F0aW9u U2xvdCAqTXlSZXBsaWNhdGlvblNsb3QgPSBOVUxMOwogaW50CQkJbWF4X3JlcGxpY2F0aW9u X3Nsb3RzID0gMDsJLyogdGhlIG1heGltdW0gbnVtYmVyIG9mIHJlcGxpY2F0aW9uCiAJCQkJ CQkJCQkJICogc2xvdHMgKi8KIAorc3RhdGljIGJvb2wgUmVwbGljYXRpb25TbG90SXNBY3Rp dmUoUmVwbGljYXRpb25TbG90ICpzbG90LCBpbnQgKmFjdGl2ZV9waWQpOworc3RhdGljIHZv aWQgRGVhY3RpdmF0ZVJlcGxpY2F0aW9uU2xvdChSZXBsaWNhdGlvblNsb3QgKnNsb3QpOwog c3RhdGljIHZvaWQgUmVwbGljYXRpb25TbG90RHJvcEFjcXVpcmVkKHZvaWQpOwogc3RhdGlj IHZvaWQgUmVwbGljYXRpb25TbG90RHJvcFB0cihSZXBsaWNhdGlvblNsb3QgKnNsb3QpOwog CkBAIC0xMDgwLDYgKzEwODIsNjEgQEAgUmVwbGljYXRpb25TbG90UmVzZXJ2ZVdhbCh2b2lk KQogCX0KIH0KIAorLyoKKyAqIElzIHRoZSBzcGVjaWZpZWQgcmVwbGljYXRpb24gc2xvdCBj dXJyZW50bHkgYWN0aXZlbHkgYmVpbmcgdXNlZD8KKyAqCisgKiBTZXQgKmFjdGl2ZV9waWQg dG8gdGhlIFBJRCBvZiB0aGUgcHJvY2VzcyB1c2luZyB0aGlzIHNsb3QgaWYgYWN0aXZlLgor ICovCitzdGF0aWMgYm9vbAorUmVwbGljYXRpb25TbG90SXNBY3RpdmUoUmVwbGljYXRpb25T bG90ICpzbG90LCBpbnQgKmFjdGl2ZV9waWQpCit7CisJaW50CQlwaWQ7CisKKwlTcGluTG9j a0FjcXVpcmUoJnNsb3QtPm11dGV4KTsKKwlwaWQgPSBzbG90LT5hY3RpdmVfcGlkOworCVNw aW5Mb2NrUmVsZWFzZSgmc2xvdC0+bXV0ZXgpOworCisJaWYgKGFjdGl2ZV9waWQgIT0gTlVM TCkKKwkJKmFjdGl2ZV9waWQgPSBwaWQ7CisKKwlyZXR1cm4gKHBpZCAhPSAwKTsKK30KKwor LyoKKyAqIERlYWN0aXZhdGUgdGhlIHNwZWNpZmllZCByZXBsaWNhdGlvbiBzbG90LgorICoK KyAqIFRlcm1pbmF0ZSB0aGUgcHJvY2VzcyB1c2luZyB0aGlzIHNsb3QgaWYgYWN0aXZlLgor ICovCitzdGF0aWMgdm9pZAorRGVhY3RpdmF0ZVJlcGxpY2F0aW9uU2xvdChSZXBsaWNhdGlv blNsb3QgKnNsb3QpCit7CisJaW50CQlhY3RpdmVfcGlkOworCWJvb2wJa2lsbGVkID0gZmFs c2U7CisKKwkvKiBRdWljayBleGl0IGlmIGFscmVhZHkgaW5hY3RpdmUgKi8KKwlpZiAoIVJl cGxpY2F0aW9uU2xvdElzQWN0aXZlKHNsb3QsICZhY3RpdmVfcGlkKSkKKwkJcmV0dXJuOwor CisJZXJlcG9ydChMT0csCisJCQkoZXJybXNnKCJ0ZXJtaW5hdGluZyB0aGUgcHJvY2VzcyAl ZCB1c2luZyByZXBsaWNhdGlvbiBzbG90IFwiJXNcIiIsCisJCQkJCWFjdGl2ZV9waWQsIE5h bWVTdHIoc2xvdC0+ZGF0YS5uYW1lKSkpKTsKKworCUNvbmRpdGlvblZhcmlhYmxlUHJlcGFy ZVRvU2xlZXAoJnNsb3QtPmFjdGl2ZV9jdik7CisJZG8KKwl7CisJCS8qCisJCSAqIFNpZ25h bCB0byB0ZXJtaW5hdGUgdGhlIHByb2Nlc3MgdXNpbmcgdGhlIHJlcGxpY2F0aW9uIHNsb3Qu CisJCSAqCisJCSAqIFRyeSB0byBzaWduYWwgZXZlcnkgMTAwbXMgdW50aWwgaXQgc3VjY2Vl ZHMuCisJCSAqLworCQlpZiAoIWtpbGxlZCAmJiBraWxsKGFjdGl2ZV9waWQsIFNJR1RFUk0p ID09IDApCisJCQlraWxsZWQgPSB0cnVlOworCQlDb25kaXRpb25WYXJpYWJsZVRpbWVkU2xl ZXAoJnNsb3QtPmFjdGl2ZV9jdiwgMTAwLAorCQkJCQkJCQkJV0FJVF9FVkVOVF9SRVBMSUNB VElPTl9TTE9UX0RST1ApOworCX0gd2hpbGUgKFJlcGxpY2F0aW9uU2xvdElzQWN0aXZlKHNs b3QsIE5VTEwpKTsKKwlDb25kaXRpb25WYXJpYWJsZUNhbmNlbFNsZWVwKCk7Cit9CisKIC8q CiAgKiBNYXJrIGFueSBzbG90IHRoYXQgcG9pbnRzIHRvIGFuIExTTiBvbGRlciB0aGFuIHRo ZSBnaXZlbiBzZWdtZW50CiAgKiBhcyBpbnZhbGlkOyBpdCByZXF1aXJlcyBXQUwgdGhhdCdz IGFib3V0IHRvIGJlIHJlbW92ZWQuCkBAIC0xMTA1LDM3ICsxMTYyLDE4IEBAIHJlc3RhcnQ6 CiAJCQljb250aW51ZTsKIAogCQlTcGluTG9ja0FjcXVpcmUoJnMtPm11dGV4KTsKLQkJaWYg KHMtPmRhdGEucmVzdGFydF9sc24gPT0gSW52YWxpZFhMb2dSZWNQdHIgfHwKLQkJCXMtPmRh dGEucmVzdGFydF9sc24gPj0gb2xkZXN0TFNOKQotCQl7Ci0JCQlTcGluTG9ja1JlbGVhc2Uo JnMtPm11dGV4KTsKLQkJCWNvbnRpbnVlOwotCQl9Ci0KIAkJc2xvdG5hbWUgPSBzLT5kYXRh Lm5hbWU7CiAJCXJlc3RhcnRfbHNuID0gcy0+ZGF0YS5yZXN0YXJ0X2xzbjsKLQogCQlTcGlu TG9ja1JlbGVhc2UoJnMtPm11dGV4KTsKKworCQlpZiAoWExvZ1JlY1B0cklzSW52YWxpZChy ZXN0YXJ0X2xzbikgfHwgcmVzdGFydF9sc24gPj0gb2xkZXN0TFNOKQorCQkJY29udGludWU7 CiAJCUxXTG9ja1JlbGVhc2UoUmVwbGljYXRpb25TbG90Q29udHJvbExvY2spOwogCi0JCWZv ciAoOzspCi0JCXsKLQkJCWludAkJCXdzcGlkID0gUmVwbGljYXRpb25TbG90QWNxdWlyZShO YW1lU3RyKHNsb3RuYW1lKSwKLQkJCQkJCQkJCQkJCQkgICBTQUJfSW5xdWlyZSk7Ci0KLQkJ CS8qIG5vIHdhbHNlbmRlcj8gc3VjY2VzcyEgKi8KLQkJCWlmICh3c3BpZCA9PSAwKQotCQkJ CWJyZWFrOwotCi0JCQllcmVwb3J0KExPRywKLQkJCQkJKGVycm1zZygidGVybWluYXRpbmcg d2Fsc2VuZGVyICVkIGJlY2F1c2UgcmVwbGljYXRpb24gc2xvdCBcIiVzXCIgaXMgdG9vIGZh ciBiZWhpbmQiLAotCQkJCQkJCXdzcGlkLCBOYW1lU3RyKHNsb3RuYW1lKSkpKTsKLQkJCSh2 b2lkKSBraWxsKHdzcGlkLCBTSUdURVJNKTsKLQotCQkJQ29uZGl0aW9uVmFyaWFibGVUaW1l ZFNsZWVwKCZzLT5hY3RpdmVfY3YsIDEwLAotCQkJCQkJCQkJCVdBSVRfRVZFTlRfUkVQTElD QVRJT05fU0xPVF9EUk9QKTsKLQkJfQotCQlDb25kaXRpb25WYXJpYWJsZUNhbmNlbFNsZWVw KCk7CisJCURlYWN0aXZhdGVSZXBsaWNhdGlvblNsb3Qocyk7CisJCVNwaW5Mb2NrQWNxdWly ZSgmcy0+bXV0ZXgpOworCQlzLT5kYXRhLnJlc3RhcnRfbHNuID0gSW52YWxpZFhMb2dSZWNQ dHI7CisJCVNwaW5Mb2NrUmVsZWFzZSgmcy0+bXV0ZXgpOwogCiAJCWVyZXBvcnQoTE9HLAog CQkJCShlcnJtc2coImludmFsaWRhdGluZyBzbG90IFwiJXNcIiBiZWNhdXNlIGl0cyByZXN0 YXJ0X2xzbiAlWC8lWCBleGNlZWRzIG1heF9zbG90X3dhbF9rZWVwX3NpemUiLApAQCAtMTE0 MywxMSArMTE4MSw2IEBAIHJlc3RhcnQ6CiAJCQkJCQkodWludDMyKSAocmVzdGFydF9sc24g Pj4gMzIpLAogCQkJCQkJKHVpbnQzMikgcmVzdGFydF9sc24pKSk7CiAKLQkJU3BpbkxvY2tB Y3F1aXJlKCZzLT5tdXRleCk7Ci0JCXMtPmRhdGEucmVzdGFydF9sc24gPSBJbnZhbGlkWExv Z1JlY1B0cjsKLQkJU3BpbkxvY2tSZWxlYXNlKCZzLT5tdXRleCk7Ci0JCVJlcGxpY2F0aW9u U2xvdFJlbGVhc2UoKTsKLQogCQkvKiBpZiB3ZSBkaWQgYW55dGhpbmcsIHN0YXJ0IGZyb20g c2NyYXRjaCAqLwogCQlDSEVDS19GT1JfSU5URVJSVVBUUygpOwogCQlnb3RvIHJlc3RhcnQ7 Cg== --------------EC6C76FD7DD56EAB5A4B7D91--