Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1jjmho-0006H4-AL for pgsql-hackers@arkaria.postgresql.org; Fri, 12 Jun 2020 16:39:00 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1jjmhn-0008RB-2H for pgsql-hackers@arkaria.postgresql.org; Fri, 12 Jun 2020 16:38:59 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1jjmhm-0008Pz-RH for pgsql-hackers@lists.postgresql.org; Fri, 12 Jun 2020 16:38:58 +0000 Received: from oss.nttdata.com ([49.212.34.109]) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1jjmhj-0001WM-K3 for pgsql-hackers@lists.postgresql.org; Fri, 12 Jun 2020 16:38:58 +0000 Received: from hnk.local (p1864124-ipbf1405funabasi.chiba.ocn.ne.jp [123.216.143.124]) by oss.nttdata.com (Postfix) with ESMTPSA id CFF8E600C2 for ; Sat, 13 Jun 2020 01:38:49 +0900 (JST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.102.3 at oss.nttdata.com To: pgsql-hackers@lists.postgresql.org From: Fujii Masao Subject: Review for GetWALAvailability() Message-ID: <8d55969f-ba37-b89a-6494-e9322ccdb35d@oss.nttdata.com> Date: Sat, 13 Jun 2020 01:38:49 +0900 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:68.0) Gecko/20100101 Thunderbird/68.9.0 MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="------------A9107EBE80AF0395B3952733" Content-Language: en-US List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk This is a multi-part message in MIME format. --------------A9107EBE80AF0395B3952733 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Hi, The document explains that "lost" value that pg_replication_slots.wal_status reports means some WAL files are definitely lost and this slot cannot be used to resume replication anymore. However, I observed "lost" value while inserting lots of records, but replication could continue normally. So I wonder if pg_replication_slots.wal_status may have a bug. wal_status is calculated in GetWALAvailability(), and probably I found some issues in it. keepSegs = ConvertToXSegs(Max(max_wal_size_mb, wal_keep_segments), wal_segment_size) + 1; max_wal_size_mb is the number of megabytes. wal_keep_segments is the number of WAL segment files. So it's strange to calculate max of them. The above should be the following? Max(ConvertToXSegs(max_wal_size_mb, wal_segment_size), wal_keep_segments) + 1 if ((max_slot_wal_keep_size_mb <= 0 || max_slot_wal_keep_size_mb >= max_wal_size_mb) && oldestSegMaxWalSize <= targetSeg) return WALAVAIL_NORMAL; This code means that wal_status reports "normal" only when max_slot_wal_keep_size is negative or larger than max_wal_size. Why is this condition necessary? The document explains "normal means that the claimed files are within max_wal_size". So whatever max_slot_wal_keep_size value is, IMO that "normal" should be reported if the WAL files claimed by the slot are within max_wal_size. Thought? Or, if that condition is really necessary, the document should be updated so that the note about the condition is added. If the WAL files claimed by the slot exceeds max_slot_wal_keep_size but any those WAL files have not been removed yet, wal_status seems to report "lost". Is this expected behavior? Per the meaning of "lost" described in the document, "lost" should be reported only when any claimed files are removed, I think. Thought? Or this behavior is expected and the document is incorrect? BTW, if we want to implement GetWALAvailability() as the document advertises, we can simplify it like the attached POC patch. Regards, -- Fujii Masao Advanced Computing Technology Center Research and Development Headquarters NTT DATA CORPORATION --------------A9107EBE80AF0395B3952733 Content-Type: text/plain; charset=UTF-8; x-mac-type="0"; x-mac-creator="0"; name="simple_getwalavailability_wip.patch" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="simple_getwalavailability_wip.patch" ZGlmZiAtLWdpdCBhL3NyYy9iYWNrZW5kL2FjY2Vzcy90cmFuc2FtL3hsb2cuYyBiL3NyYy9i YWNrZW5kL2FjY2Vzcy90cmFuc2FtL3hsb2cuYwppbmRleCA1NWNhYzE4NmRjLi4wYjljY2Ey MTczIDEwMDY0NAotLS0gYS9zcmMvYmFja2VuZC9hY2Nlc3MvdHJhbnNhbS94bG9nLmMKKysr IGIvc3JjL2JhY2tlbmQvYWNjZXNzL3RyYW5zYW0veGxvZy5jCkBAIC05NTA0LDYyICs5NTA0 LDI5IEBAIEdldFdBTEF2YWlsYWJpbGl0eShYTG9nUmVjUHRyIHRhcmdldExTTikKIAlYTG9n U2VnTm8JY3VyclNlZzsJCS8qIHNlZ2lkIG9mIGN1cnJwb3MgKi8KIAlYTG9nU2VnTm8JdGFy Z2V0U2VnOwkJLyogc2VnaWQgb2YgdGFyZ2V0TFNOICovCiAJWExvZ1NlZ05vCW9sZGVzdFNl ZzsJCS8qIGFjdHVhbCBvbGRlc3Qgc2VnaWQgKi8KLQlYTG9nU2VnTm8Jb2xkZXN0U2VnTWF4 V2FsU2l6ZTsJLyogb2xkZXN0IHNlZ2lkIGtlcHQgYnkgbWF4X3dhbF9zaXplICovCi0JWExv Z1NlZ05vCW9sZGVzdFNsb3RTZWcgPSBJbnZhbGlkWExvZ1JlY1B0cjsJLyogb2xkZXN0IHNl Z2lkIGtlcHQgYnkKLQkJCQkJCQkJCQkJCQkgKiBzbG90ICovCiAJdWludDY0CQlrZWVwU2Vn czsKIAogCS8qIHNsb3QgZG9lcyBub3QgcmVzZXJ2ZSBXQUwuIEVpdGhlciBkZWFjdGl2YXRl ZCwgb3IgaGFzIG5ldmVyIGJlZW4gYWN0aXZlICovCiAJaWYgKFhMb2dSZWNQdHJJc0ludmFs aWQodGFyZ2V0TFNOKSkKIAkJcmV0dXJuIFdBTEFWQUlMX0lOVkFMSURfTFNOOwogCi0JY3Vy cnBvcyA9IEdldFhMb2dXcml0ZVJlY1B0cigpOwotCiAJLyogY2FsY3VsYXRlIG9sZGVzdCBz ZWdtZW50IGN1cnJlbnRseSBuZWVkZWQgYnkgc2xvdHMgKi8KIAlYTEJ5dGVUb1NlZyh0YXJn ZXRMU04sIHRhcmdldFNlZywgd2FsX3NlZ21lbnRfc2l6ZSk7Ci0JS2VlcExvZ1NlZyhjdXJy cG9zLCAmb2xkZXN0U2xvdFNlZyk7CiAKLQkvKgotCSAqIEZpbmQgdGhlIG9sZGVzdCBleHRh bnQgc2VnbWVudCBmaWxlLiBXZSBnZXQgMSB1bnRpbCBjaGVja3BvaW50IHJlbW92ZXMKLQkg KiB0aGUgZmlyc3QgV0FMIHNlZ21lbnQgZmlsZSBzaW5jZSBzdGFydHVwLCB3aGljaCBjYXVz ZXMgdGhlIHN0YXR1cyBiZWluZwotCSAqIHdyb25nIHVuZGVyIGNlcnRhaW4gYWJub3JtYWwg Y29uZGl0aW9ucyBidXQgdGhhdCBkb2Vzbid0IGFjdHVhbGx5IGhhcm0uCi0JICovCi0Jb2xk ZXN0U2VnID0gWExvZ0dldExhc3RSZW1vdmVkU2Vnbm8oKSArIDE7CisJLyogRmluZCB0aGUg b2xkZXN0IGV4dGFudCBzZWdtZW50IGZpbGUgKi8KKwlvbGRlc3RTZWcgPSBYTG9nR2V0TGFz dFJlbW92ZWRTZWdubygpOwogCi0JLyogY2FsY3VsYXRlIG9sZGVzdCBzZWdtZW50IGJ5IG1h eF93YWxfc2l6ZSBhbmQgd2FsX2tlZXBfc2VnbWVudHMgKi8KKwlpZiAodGFyZ2V0U2VnIDw9 IG9sZGVzdFNlZykKKwkJcmV0dXJuIFdBTEFWQUlMX1JFTU9WRUQ7CisKKwljdXJycG9zID0g R2V0WExvZ1dyaXRlUmVjUHRyKCk7CiAJWExCeXRlVG9TZWcoY3VycnBvcywgY3VyclNlZywg d2FsX3NlZ21lbnRfc2l6ZSk7Ci0Ja2VlcFNlZ3MgPSBDb252ZXJ0VG9YU2VncyhNYXgobWF4 X3dhbF9zaXplX21iLCB3YWxfa2VlcF9zZWdtZW50cyksCi0JCQkJCQkJICB3YWxfc2VnbWVu dF9zaXplKSArIDE7CisJa2VlcFNlZ3MgPSBDb252ZXJ0VG9YU2VncyhtYXhfd2FsX3NpemVf bWIsIHdhbF9zZWdtZW50X3NpemUpOwogCi0JaWYgKGN1cnJTZWcgPiBrZWVwU2VncykKLQkJ b2xkZXN0U2VnTWF4V2FsU2l6ZSA9IGN1cnJTZWcgLSBrZWVwU2VnczsKLQllbHNlCi0JCW9s ZGVzdFNlZ01heFdhbFNpemUgPSAxOwotCi0JLyoKLQkgKiBJZiBtYXhfc2xvdF93YWxfa2Vl cF9zaXplIGhhcyBjaGFuZ2VkIGFmdGVyIHRoZSBsYXN0IGNhbGwsIHRoZSBzZWdtZW50Ci0J ICogdGhhdCB3b3VsZCBiZWVuIGtlcHQgYnkgdGhlIGN1cnJlbnQgc2V0dGluZyBtaWdodCBo YXZlIGJlZW4gbG9zdCBieSB0aGUKLQkgKiBwcmV2aW91cyBzZXR0aW5nLiBObyBwb2ludCBp biBzaG93aW5nIG5vcm1hbCBvciBrZWVwaW5nIHN0YXR1cyB2YWx1ZXMKLQkgKiBpZiB0aGUg dGFyZ2V0U2VnIGlzIGtub3duIHRvIGJlIGxvc3QuCi0JICovCi0JaWYgKHRhcmdldFNlZyA+ PSBvbGRlc3RTZWcpCi0JewotCQkvKgotCQkgKiBzaG93ICJub3JtYWwiIHdoZW4gdGFyZ2V0 U2VnIGlzIHdpdGhpbiBtYXhfd2FsX3NpemUsIGV2ZW4gaWYKLQkJICogbWF4X3Nsb3Rfd2Fs X2tlZXBfc2l6ZSBpcyBzbWFsbGVyIHRoYW4gbWF4X3dhbF9zaXplLgotCQkgKi8KLQkJaWYg KChtYXhfc2xvdF93YWxfa2VlcF9zaXplX21iIDw9IDAgfHwKLQkJCSBtYXhfc2xvdF93YWxf a2VlcF9zaXplX21iID49IG1heF93YWxfc2l6ZV9tYikgJiYKLQkJCW9sZGVzdFNlZ01heFdh bFNpemUgPD0gdGFyZ2V0U2VnKQorCWlmIChjdXJyU2VnIC0gdGFyZ2V0U2VnIDw9IGtlZXBT ZWdzKQogCQkJcmV0dXJuIFdBTEFWQUlMX05PUk1BTDsKIAotCQkvKiBiZWluZyByZXRhaW5l ZCBieSBzbG90cyAqLwotCQlpZiAob2xkZXN0U2xvdFNlZyA8PSB0YXJnZXRTZWcpCi0JCQly ZXR1cm4gV0FMQVZBSUxfUkVTRVJWRUQ7Ci0JfQotCi0JLyogRGVmaW5pdGVseSBsb3N0ICov Ci0JcmV0dXJuIFdBTEFWQUlMX1JFTU9WRUQ7CisJcmV0dXJuIFdBTEFWQUlMX1JFU0VSVkVE OwogfQogCiAK --------------A9107EBE80AF0395B3952733--