public inbox for [email protected]
help / color / mirror / Atom feedFrom: ls7777 <[email protected]>
To: [email protected] <[email protected]>
Subject: Patch for migration of the pg_commit_ts directory
Date: Thu, 03 Apr 2025 23:27:32 +0500
Message-ID: <[email protected]> (raw)
------==--bound.1245643.r-production-main-61.klg.yp-c.yandex.net
Content-Transfer-Encoding: 8bit
Content-Type: text/html; charset=utf-8
<div><div><div><div>Good afternoon!</div><div>During pg_upgrade migration, the pg_commit_ts directory is not copied when track_commit_timestamp = on is set. There is a patch in the attachment that fixes this.</div></div><div> </div><div>Contents & Purpose</div><div>==================</div><div>The patch copies the pg_commit_ts directory of the old cluster and updates the Latest checkpoints oldestCommitTsXid and Latest checkpoint's newestCommitTsXid for the new cluster. If required.</div><div> </div><div>Initial Run</div><div>===========</div><div>The patch can be applied on the master branch.</div><div>I don't understand what tests are needed for such a patch. The patch is very simple.</div><div> Performance</div><div>===========</div><div>The pg_upgrade operation time increases by the time the pg_commit_ts catalog is copied.</div><div> </div><div> </div></div></div>
------==--bound.1245643.r-production-main-61.klg.yp-c.yandex.net
Content-Disposition: attachment;
filename="v1-0001-Migration-of-the-pg_commit_ts-directory.patch"
Content-Transfer-Encoding: base64
Content-Type: text/x-diff;
name="v1-0001-Migration-of-the-pg_commit_ts-directory.patch"
RnJvbSA1ZmNiYWNkODcwMDI3YWZlOWQ1OWJlYzY5NmZhZGZhZTU3MjdkYjViIE1vbiBTZXAgMTcg
MDA6MDA6MDAgMjAwMQpGcm9tOiBTZXJnZXkgTGV2aW4gPGxzNzc3N0B5YW5kZXgucnU+CkRhdGU6
IFRodSwgMyBBcHIgMjAyNSAyMTo0NDowMSArMDUwMApTdWJqZWN0OiBbUEFUQ0ggdjFdIE1pZ3Jh
dGlvbiBvZiB0aGUgcGdfY29tbWl0X3RzIGRpcmVjdG9yeQoKLS0tCiBzcmMvYmluL3BnX3VwZ3Jh
ZGUvY29udHJvbGRhdGEuYyB8IDIzICsrKysrKysrKysrKysrKysrKysrKystCiBzcmMvYmluL3Bn
X3VwZ3JhZGUvcGdfdXBncmFkZS5jICB8ICA2ICsrKystLQogc3JjL2Jpbi9wZ191cGdyYWRlL3Bn
X3VwZ3JhZGUuaCAgfCAgMiArKwogMyBmaWxlcyBjaGFuZ2VkLCAyOCBpbnNlcnRpb25zKCspLCAz
IGRlbGV0aW9ucygtKQoKZGlmZiAtLWdpdCBhL3NyYy9iaW4vcGdfdXBncmFkZS9jb250cm9sZGF0
YS5jIGIvc3JjL2Jpbi9wZ191cGdyYWRlL2NvbnRyb2xkYXRhLmMKaW5kZXggNDdlZTI3ZWM4MzUu
LjBkYTVlZTQyMjVlIDEwMDY0NAotLS0gYS9zcmMvYmluL3BnX3VwZ3JhZGUvY29udHJvbGRhdGEu
YworKysgYi9zcmMvYmluL3BnX3VwZ3JhZGUvY29udHJvbGRhdGEuYwpAQCAtMjA3LDcgKzIwNyw4
IEBAIGdldF9jb250cm9sX2RhdGEoQ2x1c3RlckluZm8gKmNsdXN0ZXIpCiAJCWNsdXN0ZXItPmNv
bnRyb2xkYXRhLmRhdGFfY2hlY2tzdW1fdmVyc2lvbiA9IDA7CiAJCWdvdF9kYXRhX2NoZWNrc3Vt
X3ZlcnNpb24gPSB0cnVlOwogCX0KLQorCWNsdXN0ZXItPmNvbnRyb2xkYXRhLmNoa3BudF9vbGRz
dENvbW1pdFRzeGlkID0gMDsKKwljbHVzdGVyLT5jb250cm9sZGF0YS5jaGtwbnRfbmV3c3RDb21t
aXRUc3hpZCA9IDA7CiAJLyogd2UgaGF2ZSB0aGUgcmVzdWx0IG9mIGNtZCBpbiAib3V0cHV0Ii4g
c28gcGFyc2UgaXQgbGluZSBieSBsaW5lIG5vdyAqLwogCXdoaWxlIChmZ2V0cyhidWZpbiwgc2l6
ZW9mKGJ1ZmluKSwgb3V0cHV0KSkKIAl7CkBAIC0zMjAsNiArMzIxLDI2IEBAIGdldF9jb250cm9s
X2RhdGEoQ2x1c3RlckluZm8gKmNsdXN0ZXIpCiAJCQljbHVzdGVyLT5jb250cm9sZGF0YS5jaGtw
bnRfbnh0bXVsdGkgPSBzdHIydWludChwKTsKIAkJCWdvdF9tdWx0aSA9IHRydWU7CiAJCX0KKwkJ
ZWxzZSBpZiAoKHAgPSBzdHJzdHIoYnVmaW4sICJMYXRlc3QgY2hlY2twb2ludCdzIG9sZGVzdENv
bW1pdFRzWGlkOiIpKSAhPSBOVUxMKQorCQl7CisJCQlwID0gc3RyY2hyKHAsICc6Jyk7CisKKwkJ
CWlmIChwID09IE5VTEwgfHwgc3RybGVuKHApIDw9IDEpCisJCQkJcGdfZmF0YWwoIiVkOiBjb250
cm9sZGF0YSByZXRyaWV2YWwgcHJvYmxlbSIsIF9fTElORV9fKTsKKworCQkJcCsrOwkJCQkvKiBy
ZW1vdmUgJzonIGNoYXIgKi8KKwkJCWNsdXN0ZXItPmNvbnRyb2xkYXRhLmNoa3BudF9vbGRzdENv
bW1pdFRzeGlkID0gc3RyMnVpbnQocCk7CisJCX0KKwkJZWxzZSBpZiAoKHAgPSBzdHJzdHIoYnVm
aW4sICJMYXRlc3QgY2hlY2twb2ludCdzIG5ld2VzdENvbW1pdFRzWGlkOiIpKSAhPSBOVUxMKQor
CQl7CisJCQlwID0gc3RyY2hyKHAsICc6Jyk7CisKKwkJCWlmIChwID09IE5VTEwgfHwgc3RybGVu
KHApIDw9IDEpCisJCQkJcGdfZmF0YWwoIiVkOiBjb250cm9sZGF0YSByZXRyaWV2YWwgcHJvYmxl
bSIsIF9fTElORV9fKTsKKworCQkJcCsrOwkJCQkvKiByZW1vdmUgJzonIGNoYXIgKi8KKwkJCWNs
dXN0ZXItPmNvbnRyb2xkYXRhLmNoa3BudF9uZXdzdENvbW1pdFRzeGlkID0gc3RyMnVpbnQocCk7
CisJCX0KIAkJZWxzZSBpZiAoKHAgPSBzdHJzdHIoYnVmaW4sICJMYXRlc3QgY2hlY2twb2ludCdz
IG9sZGVzdFhJRDoiKSkgIT0gTlVMTCkKIAkJewogCQkJcCA9IHN0cmNocihwLCAnOicpOwpkaWZm
IC0tZ2l0IGEvc3JjL2Jpbi9wZ191cGdyYWRlL3BnX3VwZ3JhZGUuYyBiL3NyYy9iaW4vcGdfdXBn
cmFkZS9wZ191cGdyYWRlLmMKaW5kZXggOTI5NWU0NmFlZDMuLjc0NmY4YTZiMDJiIDEwMDY0NAot
LS0gYS9zcmMvYmluL3BnX3VwZ3JhZGUvcGdfdXBncmFkZS5jCisrKyBiL3NyYy9iaW4vcGdfdXBn
cmFkZS9wZ191cGdyYWRlLmMKQEAgLTc1MSw2ICs3NTEsOCBAQCBjb3B5X3hhY3RfeGxvZ194aWQo
dm9pZCkKIAkJCQkJICAicGdfY2xvZyIgOiAicGdfeGFjdCIsCiAJCQkJCSAgR0VUX01BSk9SX1ZF
UlNJT04obmV3X2NsdXN0ZXIubWFqb3JfdmVyc2lvbikgPD0gOTA2ID8KIAkJCQkJICAicGdfY2xv
ZyIgOiAicGdfeGFjdCIpOworCWlmIChvbGRfY2x1c3Rlci5jb250cm9sZGF0YS5jaGtwbnRfb2xk
c3RDb21taXRUc3hpZCA+IDApCisJCQkJCSAgIGNvcHlfc3ViZGlyX2ZpbGVzKCJwZ19jb21taXRf
dHMiLCAicGdfY29tbWl0X3RzIik7CiAKIAlwcmVwX3N0YXR1cygiU2V0dGluZyBvbGRlc3QgWElE
IGZvciBuZXcgY2x1c3RlciIpOwogCWV4ZWNfcHJvZyhVVElMSVRZX0xPR19GSUxFLCBOVUxMLCB0
cnVlLCB0cnVlLApAQCAtNzczLDggKzc3NSw4IEBAIGNvcHlfeGFjdF94bG9nX3hpZCh2b2lkKQog
CWV4ZWNfcHJvZyhVVElMSVRZX0xPR19GSUxFLCBOVUxMLCB0cnVlLCB0cnVlLAogCQkJICAiXCIl
cy9wZ19yZXNldHdhbFwiIC1mIC1jICV1LCV1IFwiJXNcIiIsCiAJCQkgIG5ld19jbHVzdGVyLmJp
bmRpciwKLQkJCSAgb2xkX2NsdXN0ZXIuY29udHJvbGRhdGEuY2hrcG50X254dHhpZCwKLQkJCSAg
b2xkX2NsdXN0ZXIuY29udHJvbGRhdGEuY2hrcG50X254dHhpZCwKKwkJCSAgb2xkX2NsdXN0ZXIu
Y29udHJvbGRhdGEuY2hrcG50X29sZHN0Q29tbWl0VHN4aWQgPT0gMCA/IG9sZF9jbHVzdGVyLmNv
bnRyb2xkYXRhLmNoa3BudF9ueHR4aWQgOiBvbGRfY2x1c3Rlci5jb250cm9sZGF0YS5jaGtwbnRf
b2xkc3RDb21taXRUc3hpZCwKKwkJCSAgb2xkX2NsdXN0ZXIuY29udHJvbGRhdGEuY2hrcG50X25l
d3N0Q29tbWl0VHN4aWQgPT0gMCA/IG9sZF9jbHVzdGVyLmNvbnRyb2xkYXRhLmNoa3BudF9ueHR4
aWQgOiBvbGRfY2x1c3Rlci5jb250cm9sZGF0YS5jaGtwbnRfbmV3c3RDb21taXRUc3hpZCwKIAkJ
CSAgbmV3X2NsdXN0ZXIucGdkYXRhKTsKIAljaGVja19vaygpOwogCmRpZmYgLS1naXQgYS9zcmMv
YmluL3BnX3VwZ3JhZGUvcGdfdXBncmFkZS5oIGIvc3JjL2Jpbi9wZ191cGdyYWRlL3BnX3VwZ3Jh
ZGUuaAppbmRleCA2OWM5NjViYjdkMC4uYjU1ZTJjMmVhODQgMTAwNjQ0Ci0tLSBhL3NyYy9iaW4v
cGdfdXBncmFkZS9wZ191cGdyYWRlLmgKKysrIGIvc3JjL2Jpbi9wZ191cGdyYWRlL3BnX3VwZ3Jh
ZGUuaApAQCAtMjM4LDYgKzIzOCw4IEBAIHR5cGVkZWYgc3RydWN0CiAJdWludDMyCQljaGtwbnRf
bnh0bXhvZmY7CiAJdWludDMyCQljaGtwbnRfb2xkc3RNdWx0aTsKIAl1aW50MzIJCWNoa3BudF9v
bGRzdHhpZDsKKwl1aW50MzIJCWNoa3BudF9vbGRzdENvbW1pdFRzeGlkOworCXVpbnQzMgkJY2hr
cG50X25ld3N0Q29tbWl0VHN4aWQ7CiAJdWludDMyCQlhbGlnbjsKIAl1aW50MzIJCWJsb2Nrc3o7
CiAJdWludDMyCQlsYXJnZXN6OwotLSAKMi40Mi4yCgo=
------==--bound.1245643.r-production-main-61.klg.yp-c.yandex.net--
Attachments:
[text/x-diff] v1-0001-Migration-of-the-pg_commit_ts-directory.patch (3.4K, ../[email protected]/2-v1-0001-Migration-of-the-pg_commit_ts-directory.patch)
download | inline diff:
From 5fcbacd870027afe9d59bec696fadfae5727db5b Mon Sep 17 00:00:00 2001
From: Sergey Levin <[email protected]>
Date: Thu, 3 Apr 2025 21:44:01 +0500
Subject: [PATCH v1] Migration of the pg_commit_ts directory
---
src/bin/pg_upgrade/controldata.c | 23 ++++++++++++++++++++++-
src/bin/pg_upgrade/pg_upgrade.c | 6 ++++--
src/bin/pg_upgrade/pg_upgrade.h | 2 ++
3 files changed, 28 insertions(+), 3 deletions(-)
diff --git a/src/bin/pg_upgrade/controldata.c b/src/bin/pg_upgrade/controldata.c
index 47ee27ec835..0da5ee4225e 100644
--- a/src/bin/pg_upgrade/controldata.c
+++ b/src/bin/pg_upgrade/controldata.c
@@ -207,7 +207,8 @@ get_control_data(ClusterInfo *cluster)
cluster->controldata.data_checksum_version = 0;
got_data_checksum_version = true;
}
-
+ cluster->controldata.chkpnt_oldstCommitTsxid = 0;
+ cluster->controldata.chkpnt_newstCommitTsxid = 0;
/* we have the result of cmd in "output". so parse it line by line now */
while (fgets(bufin, sizeof(bufin), output))
{
@@ -320,6 +321,26 @@ get_control_data(ClusterInfo *cluster)
cluster->controldata.chkpnt_nxtmulti = str2uint(p);
got_multi = true;
}
+ else if ((p = strstr(bufin, "Latest checkpoint's oldestCommitTsXid:")) != NULL)
+ {
+ p = strchr(p, ':');
+
+ if (p == NULL || strlen(p) <= 1)
+ pg_fatal("%d: controldata retrieval problem", __LINE__);
+
+ p++; /* remove ':' char */
+ cluster->controldata.chkpnt_oldstCommitTsxid = str2uint(p);
+ }
+ else if ((p = strstr(bufin, "Latest checkpoint's newestCommitTsXid:")) != NULL)
+ {
+ p = strchr(p, ':');
+
+ if (p == NULL || strlen(p) <= 1)
+ pg_fatal("%d: controldata retrieval problem", __LINE__);
+
+ p++; /* remove ':' char */
+ cluster->controldata.chkpnt_newstCommitTsxid = str2uint(p);
+ }
else if ((p = strstr(bufin, "Latest checkpoint's oldestXID:")) != NULL)
{
p = strchr(p, ':');
diff --git a/src/bin/pg_upgrade/pg_upgrade.c b/src/bin/pg_upgrade/pg_upgrade.c
index 9295e46aed3..746f8a6b02b 100644
--- a/src/bin/pg_upgrade/pg_upgrade.c
+++ b/src/bin/pg_upgrade/pg_upgrade.c
@@ -751,6 +751,8 @@ copy_xact_xlog_xid(void)
"pg_clog" : "pg_xact",
GET_MAJOR_VERSION(new_cluster.major_version) <= 906 ?
"pg_clog" : "pg_xact");
+ if (old_cluster.controldata.chkpnt_oldstCommitTsxid > 0)
+ copy_subdir_files("pg_commit_ts", "pg_commit_ts");
prep_status("Setting oldest XID for new cluster");
exec_prog(UTILITY_LOG_FILE, NULL, true, true,
@@ -773,8 +775,8 @@ copy_xact_xlog_xid(void)
exec_prog(UTILITY_LOG_FILE, NULL, true, true,
"\"%s/pg_resetwal\" -f -c %u,%u \"%s\"",
new_cluster.bindir,
- old_cluster.controldata.chkpnt_nxtxid,
- old_cluster.controldata.chkpnt_nxtxid,
+ old_cluster.controldata.chkpnt_oldstCommitTsxid == 0 ? old_cluster.controldata.chkpnt_nxtxid : old_cluster.controldata.chkpnt_oldstCommitTsxid,
+ old_cluster.controldata.chkpnt_newstCommitTsxid == 0 ? old_cluster.controldata.chkpnt_nxtxid : old_cluster.controldata.chkpnt_newstCommitTsxid,
new_cluster.pgdata);
check_ok();
diff --git a/src/bin/pg_upgrade/pg_upgrade.h b/src/bin/pg_upgrade/pg_upgrade.h
index 69c965bb7d0..b55e2c2ea84 100644
--- a/src/bin/pg_upgrade/pg_upgrade.h
+++ b/src/bin/pg_upgrade/pg_upgrade.h
@@ -238,6 +238,8 @@ typedef struct
uint32 chkpnt_nxtmxoff;
uint32 chkpnt_oldstMulti;
uint32 chkpnt_oldstxid;
+ uint32 chkpnt_oldstCommitTsxid;
+ uint32 chkpnt_newstCommitTsxid;
uint32 align;
uint32 blocksz;
uint32 largesz;
--
2.42.2
view thread (13+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected]
Subject: Re: Patch for migration of the pg_commit_ts directory
In-Reply-To: <[email protected]>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox