Content-Type: multipart/mixed; boundary="------------KOIAHKUdYNRQNzkdwh09azax"
Message-ID: <90980ee6-2da6-42f6-a7b0-b7bae62ae279@oss.nttdata.com>
Date: Wed, 3 Jul 2024 18:33:44 +0900
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
To: pgsql-hackers@lists.postgresql.org
Content-Language: en-US
From: Fujii Masao <masao.fujii@oss.nttdata.com>
Subject: pg_wal_summary_contents() and pg_walsummary may return different
 results on the same WAL summary file
Archived-At: <https://www.postgresql.org/message-id/90980ee6-2da6-42f6-a7b0-b7bae62ae279%40oss.nttdata.com>
Precedence: bulk

This is a multi-part message in MIME format.
--------------KOIAHKUdYNRQNzkdwh09azax
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

Hi,

I found that pg_wal_summary_contents() may miss some results that pg_walsummary returns for the same WAL summary file. Here are the steps to reproduce the issue:

-----------------------------
initdb -D data
echo "summarize_wal = on" >> data/postgresql.conf
pg_ctl -D data start
psql <<EOF
CREATE TABLE t AS SELECT n i, n j FROM generate_series(1, 1000) n;
DELETE FROM t;
CHECKPOINT;
VACUUM t;
CHECKPOINT;
SELECT foo.* FROM (SELECT * FROM pg_available_wal_summaries() ORDER BY start_lsn DESC LIMIT 1) JOIN LATERAL pg_wal_summary_contents(tli, start_lsn, end_lsn) foo ON true;
EOF
pg_walsummary -i data/pg_wal/summaries/$(ls -1 data/pg_wal/summaries/ | tail -1)
-----------------------------

In my test, pg_walsummary returned three records:

TS 1663, DB 5, REL 1259, FORK main: block 0
TS 1663, DB 5, REL 16384, FORK main: limit 0
TS 1663, DB 5, REL 16384, FORK vm: limit 0

However, pg_wal_summary_contents() returned only one record:

  relfilenode | reltablespace | reldatabase | relforknumber | relblocknumber | is_limit_block
-------------+---------------+-------------+---------------+----------------+----------------
         1259 |          1663 |           5 |             0 |              0 | f

pg_wal_summary_contents() seems to miss the summary information with "limit" that pg_walsummary reports. This appears to be a bug. The attached patch fixes this.

By the way, pg_wal_summary_contents() and pg_walsummary perform nearly the same task but are implemented in different functions. This could be the root of issues like this. In the future, it would be better to have a common function for outputting the WAL summary file that both can use.

Regards,

-- 
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION
--------------KOIAHKUdYNRQNzkdwh09azax
Content-Type: text/plain; charset=UTF-8;
 name="v1-0001-Fix-bug-in-pg_wal_summary_contents.patch"
Content-Disposition: attachment;
 filename="v1-0001-Fix-bug-in-pg_wal_summary_contents.patch"
Content-Transfer-Encoding: base64

RnJvbSA3OWExMTk3ZWEzY2RjNjliN2RkYTIxMjg3ZTYxMmIwYzlmNTMyYjU1IE1vbiBTZXAg
MTcgMDA6MDA6MDAgMjAwMQpGcm9tOiBGdWppaSBNYXNhbyA8ZnVqaWlAcG9zdGdyZXNxbC5v
cmc+CkRhdGU6IFdlZCwgMyBKdWwgMjAyNCAxNTo1MDozMCArMDkwMApTdWJqZWN0OiBbUEFU
Q0ggdjFdIEZpeCBidWcgaW4gcGdfd2FsX3N1bW1hcnlfY29udGVudHMoKS4KClByZXZpb3Vz
bHksIHBnX3dhbF9zdW1tYXJ5X2NvbnRlbnRzKCkgbWlnaHQgZmFpbCB0byByZXBvcnQgcm93
cyB3aXRoCmlzX2xpbWl0X2Jsb2NrID0gdHJ1ZSwgbGVhZGluZyB0byBkaXNjcmVwYW5jaWVz
IGJldHdlZW4KcGdfd2FsX3N1bW1hcnlfY29udGVudHMoKSBhbmQgdGhlIHBnX3dhbHN1bW1h
cnkgY29tbWFuZApvbiB0aGUgc2FtZSBXQUwgc3VtbWFyeSBmaWxlLiBUaGlzIGNvbW1pdCBy
ZXNvbHZlcyB0aGUgaXNzdWUuCi0tLQogc3JjL2JhY2tlbmQvYmFja3VwL3dhbHN1bW1hcnlm
dW5jcy5jIHwgMzQgKysrKysrKysrKysrKystLS0tLS0tLS0tLS0tLQogMSBmaWxlIGNoYW5n
ZWQsIDE3IGluc2VydGlvbnMoKyksIDE3IGRlbGV0aW9ucygtKQoKZGlmZiAtLWdpdCBhL3Ny
Yy9iYWNrZW5kL2JhY2t1cC93YWxzdW1tYXJ5ZnVuY3MuYyBiL3NyYy9iYWNrZW5kL2JhY2t1
cC93YWxzdW1tYXJ5ZnVuY3MuYwppbmRleCBmMDgyNDg4YjMzLi43ZGI3ZjEzMWNlIDEwMDY0
NAotLS0gYS9zcmMvYmFja2VuZC9iYWNrdXAvd2Fsc3VtbWFyeWZ1bmNzLmMKKysrIGIvc3Jj
L2JhY2tlbmQvYmFja3VwL3dhbHN1bW1hcnlmdW5jcy5jCkBAIC0xMTgsNiArMTE4LDIzIEBA
IHBnX3dhbF9zdW1tYXJ5X2NvbnRlbnRzKFBHX0ZVTkNUSU9OX0FSR1MpCiAJCXZhbHVlc1sy
XSA9IE9iamVjdElkR2V0RGF0dW0ocmxvY2F0b3IuZGJPaWQpOwogCQl2YWx1ZXNbM10gPSBJ
bnQxNkdldERhdHVtKChpbnQxNikgZm9ya251bSk7CiAKKwkJLyoKKwkJICogSWYgdGhlIGxp
bWl0IGJsb2NrIGlzIG5vdCBJbnZhbGlkQmxvY2tOdW1iZXIsIGVtaXQgYW4gZXh0cmEgcm93
CisJCSAqIHdpdGggdGhhdCBibG9jayBudW1iZXIgYW5kIGxpbWl0X2Jsb2NrID0gdHJ1ZS4K
KwkJICoKKwkJICogVGhlcmUgaXMgbm8gcG9pbnQgaW4gZG9pbmcgdGhpcyB3aGVuIHRoZSBs
aW1pdF9ibG9jayBpcworCQkgKiBJbnZhbGlkQmxvY2tOdW1iZXIsIGJlY2F1c2Ugbm8gYmxv
Y2sgd2l0aCB0aGF0IG51bWJlciBvciBhbnkKKwkJICogaGlnaGVyIG51bWJlciBjYW4gZXZl
ciBleGlzdC4KKwkJICovCisJCWlmIChCbG9ja051bWJlcklzVmFsaWQobGltaXRfYmxvY2sp
KQorCQl7CisJCQl2YWx1ZXNbNF0gPSBJbnQ2NEdldERhdHVtKChpbnQ2NCkgbGltaXRfYmxv
Y2spOworCQkJdmFsdWVzWzVdID0gQm9vbEdldERhdHVtKHRydWUpOworCisJCQl0dXBsZSA9
IGhlYXBfZm9ybV90dXBsZShyc2ktPnNldERlc2MsIHZhbHVlcywgbnVsbHMpOworCQkJdHVw
bGVzdG9yZV9wdXR0dXBsZShyc2ktPnNldFJlc3VsdCwgdHVwbGUpOworCQl9CisKIAkJLyog
TG9vcCBvdmVyIGJsb2NrcyB3aXRoaW4gdGhlIGN1cnJlbnQgcmVsYXRpb24gZm9yay4gKi8K
IAkJd2hpbGUgKDEpCiAJCXsKQEAgLTE0MywyMyArMTYwLDYgQEAgcGdfd2FsX3N1bW1hcnlf
Y29udGVudHMoUEdfRlVOQ1RJT05fQVJHUykKIAkJCQl0dXBsZSA9IGhlYXBfZm9ybV90dXBs
ZShyc2ktPnNldERlc2MsIHZhbHVlcywgbnVsbHMpOwogCQkJCXR1cGxlc3RvcmVfcHV0dHVw
bGUocnNpLT5zZXRSZXN1bHQsIHR1cGxlKTsKIAkJCX0KLQotCQkJLyoKLQkJCSAqIElmIHRo
ZSBsaW1pdCBibG9jayBpcyBub3QgSW52YWxpZEJsb2NrTnVtYmVyLCBlbWl0IGFuIGV4dHJh
IHJvdwotCQkJICogd2l0aCB0aGF0IGJsb2NrIG51bWJlciBhbmQgbGltaXRfYmxvY2sgPSB0
cnVlLgotCQkJICoKLQkJCSAqIFRoZXJlIGlzIG5vIHBvaW50IGluIGRvaW5nIHRoaXMgd2hl
biB0aGUgbGltaXRfYmxvY2sgaXMKLQkJCSAqIEludmFsaWRCbG9ja051bWJlciwgYmVjYXVz
ZSBubyBibG9jayB3aXRoIHRoYXQgbnVtYmVyIG9yIGFueQotCQkJICogaGlnaGVyIG51bWJl
ciBjYW4gZXZlciBleGlzdC4KLQkJCSAqLwotCQkJaWYgKEJsb2NrTnVtYmVySXNWYWxpZChs
aW1pdF9ibG9jaykpCi0JCQl7Ci0JCQkJdmFsdWVzWzRdID0gSW50NjRHZXREYXR1bSgoaW50
NjQpIGxpbWl0X2Jsb2NrKTsKLQkJCQl2YWx1ZXNbNV0gPSBCb29sR2V0RGF0dW0odHJ1ZSk7
Ci0KLQkJCQl0dXBsZSA9IGhlYXBfZm9ybV90dXBsZShyc2ktPnNldERlc2MsIHZhbHVlcywg
bnVsbHMpOwotCQkJCXR1cGxlc3RvcmVfcHV0dHVwbGUocnNpLT5zZXRSZXN1bHQsIHR1cGxl
KTsKLQkJCX0KIAkJfQogCX0KIAotLSAKMi40NS4xCgo=

--------------KOIAHKUdYNRQNzkdwh09azax--