Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1wSc3D-003IFW-0G for pgsql-hackers@arkaria.postgresql.org; Thu, 28 May 2026 14:49:35 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.96) (envelope-from ) id 1wSc39-00CS9g-0x for pgsql-hackers@arkaria.postgresql.org; Thu, 28 May 2026 14:49:32 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1wSc38-00CS9X-1R for pgsql-hackers@lists.postgresql.org; Thu, 28 May 2026 14:49:31 +0000 Received: from mail-yx1-xb12a.google.com ([2607:f8b0:4864:20::b12a]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.98.2) (envelope-from ) id 1wSc37-00000001BEn-01Ga for pgsql-hackers@lists.postgresql.org; Thu, 28 May 2026 14:49:30 +0000 Received: by mail-yx1-xb12a.google.com with SMTP id 956f58d0204a3-660390acd71so1061197d50.1 for ; Thu, 28 May 2026 07:49:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1779979768; cv=none; d=google.com; s=arc-20240605; b=A+QNUHf5VMwD/NDxoecFua2w6BEH2smI6/suyCisu3wQkCzlrsnfIR+jj2oplhLduf 1NKZiUZohlUgub6c/izDdyH37ep2o3n9ruuAVrDeEP1EuMghWGElDA2C/zrKUMZADgd/ 9iN77Ti1TN+rD3Vet4WkqjF0QXh0JPZTzxeVWVS4pmpsTt8EA47u/7TMDxsHPa099MmV fw0vzifrZPeQGN4VEOm5yZ+OrburMEQZwlNkOA5tzBk47LnUGkz6Lh5HfIp0F6ZrV1EU MRP0MEwkx4yDtrws2rf+qUFw/ixCJpnT5TB5BSucMGQmeoqi2DEhtP0Z5M3I62r5Jl+U 8USQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:dkim-signature; bh=Rmz+pOOU5p/MfJJu4hZdyAA6/oEjhJETQ0fIPc8fZa8=; fh=h5/PGq5bii4eHcM3/QqoM2w8Mi7OX6OLbPoARjsI6B4=; b=Y5PQkBrWsWDoaNx7pkB4zPNA5/w0fG8sqKjKgvqFa9R8yR1JubnzPdS/5v5V3jVm9c 9T0htB6oXJo37kAaaPHhvRAbTLjwc83Vj7PqGOYXVXyCVhjL6kFh8vTbJs6WiajQc7ng nwfatAw+M3PkA6wvPxX/32b/T1qoniZRqL16CIBgtoic125qmzEFEKb1P8H4DTRewejN vUly6FhGw5FyGwoP15opSRypK0AzhokfOQTddKi/CKmguNa3NSIuMP0SmT6RR5Rn7Y2B SHSMTtdArlajKibvVu02sxz9d7Datl27LwsgO+9b95owX4OgCeTmu2QUvBOTjG3zuyEF XzUg==; darn=lists.postgresql.org ARC-Authentication-Results: i=1; mx.google.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1779979768; x=1780584568; darn=lists.postgresql.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=Rmz+pOOU5p/MfJJu4hZdyAA6/oEjhJETQ0fIPc8fZa8=; b=RJ1x/LQ02HsBtW3zqfDinQJaKzXYfcEY/EDr3ZuRQD2D9vF23X7g0kH0nJOwHEbH2r UIdTsZyjYD+aYzhCqTbg9X2VA6i0Uw88lqDsAfHZ149L7ffjzZi8X6RdJXlre4SjX0zx ZW2Jf+4JmB2RrqWUrEMT8mgCU80693rLHxUeb0a/nTGlTF4O++8tDK9xMg2lSkxnmuzo P+S/jLnDJ4KU2COq7ZHt9aN3X6MmuusdzfgF9j28lrlz5lP9Nnen0e6wtBMv/MGBFWzy b6nAzV+YZ5N/uCWMt4Mvwt+BQ7diygje8lQWiUOS+XTMnPqEyYHAyI2XHQMp+oBDcoFx f2tw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1779979768; x=1780584568; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=Rmz+pOOU5p/MfJJu4hZdyAA6/oEjhJETQ0fIPc8fZa8=; b=WDLSu4BhmndjTC+5ubd2jKDTT0CkvvSEW72sw2xvqUcsLH6SaXHV1LR3UGAF0ul0qV pMpPTli1dzuVCMPHsczikasCztqN/Z8Cdz/biLXed7Spqw04lV8w95+dYm5SyUYBiSQp E7q1TxYrSamR2AY+oH+DMIO3/vvg4kSHvHPpOo8cs0gEDBgVPRMmfsaRf0WY5RYVxFVG dS9zk6Xp+nJMy13cDXlg4Q0aY42dKdSv29zxFQjt7gKP16lm0Nj1iMvp8D/P7jJtgHNn 1qgFPynw5Ucm9nP8gKCoLYxTn4ibaDOt2X5egZbRHRI9yI3Py7WEdQzDIo8hHLNFX+j5 MnoA== X-Forwarded-Encrypted: i=1; AFNElJ/cSfYFnSQeYyk5HMpze4e6xSkgvP4tvhqCemfcJflCIYRlp/uNS8Jn1CCvN2HPO1b97uh03eL63XBLBsbN@lists.postgresql.org X-Gm-Message-State: AOJu0YyVpmGYQKdK3YTmLtuV7Rp/fXWX5Y7eiOLcTnCEwAQ0G1DZTIm0 cwF19BAMgQS+T76HYDvXH+SQeMa4gw4GxLNWZw4yP0mfLqDCuU4YMzKEXOuCUwUr+RESovVbJ9y 2bpsP6UhrXGz+KJCHLYSYk6y9jPJ8j+U= X-Gm-Gg: Acq92OEzey7kbWjPlOSiqWcQwud6nrxGLgR/0Z4IPNo7Djz6ndzdtyV3jSGJyy2i6Or piov61VA2cIDLm0wNblI7w7HQJbY8Cal7y2xHWkQj56LxJyBPec26+DHPYf+6nr0VpuZrl5axeS OUTBQ/zRdUWDQOQSDA7UhWNtw+VKpW6g99ligOggQEwzDR5Jpy/dun9Hz32aRofTc2PMrCI4tI7 gR+j0N3fLKkjJ/xyaFwaDop/zzWDeDS6hzOzXd3/4oCxP3qdbSJqjfHG7zE6/ya+unHi75Nk2p1 k943fEXvi8wUM1C+5g9nM+WluX1qAtznzFeRxtyrdTGoTH4PFbHibnnZLkpn8g== X-Received: by 2002:a05:690e:1181:b0:654:6450:19f7 with SMTP id 956f58d0204a3-6604471b0bdmr1199083d50.0.1779979768056; Thu, 28 May 2026 07:49:28 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: vignesh C Date: Thu, 28 May 2026 20:19:14 +0530 X-Gm-Features: AVHnY4J23Pj_lNxSJ7yF8X_By20S_LnZPDrTBcKOBqRaoYJpNEF_KRkqz_GEC3c Message-ID: Subject: Re: Proposal: Conflict log history table for Logical Replication To: shveta malik Cc: Nisha Moond , Peter Smith , Dilip Kumar , Amit Kapila , Masahiko Sawada , Bharath Rupireddy , PostgreSQL Hackers , shveta malik Content-Type: multipart/mixed; boundary="000000000000c883010652e1d5b2" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --000000000000c883010652e1d5b2 Content-Type: text/plain; charset="UTF-8" On Wed, 27 May 2026 at 14:04, vignesh C wrote: > > > I have fixed the rest of the comments. The attached v41 version patch > has the changes for the same. Additionally the comments from [1] have > also been fixed. I was evaluating whether the existing pg_upgrade changes for conflict log tables can handle the addition of new columns in a future release. To validate this, I performed the following: Added two new columns to the conflict log table: v20_new_col1 TEXT v20_new_col2 TEXT These changes are present in patch '0001'. For adding new columns during binary upgrade, the following version-specific logic is required in 'pg_dump': ALTER TABLE pg_conflict.pg_conflict_log_for_subid_oid ADD COLUMN v20_new_col1 TEXT; ALTER TABLE pg_conflict.pg_conflict_log_for_subid_oid ADD COLUMN v20_new_col2 TEXT; These changes are included in patch '0001'. One important point here is that when 'ALTER TABLE ... ADD COLUMN' is run, the server does not rewrite existing rows on disk. Instead, it only updates the system catalog with the new column metadata. While selecting data from the table, the server handles this as follows: 1. Deform what is physically present - 'slot_deform_heap_tuple()' reads the raw tuple bytes from disk, but only up to 't_natts', which is the number of columns recorded in the tuple header at the time that row was inserted. It stops there because the tuple has no physical data for columns added later. 2. Fill in what is missing - After deforming the tuple, if the number of populated columns is still less than the number of columns requested by the query, it calls 'slot_getmissingattrs()' to cover the gap. Since the new columns were added with no default value, 'slot_getmissingattrs()' sets: tts_isnull[attnum] = true; This is how NULL is returned for the newly added columns in existing rows. These changes were tested on a new server with the v40 version patch + '0001' patch. 1. Pre-upgrade state using v40 version patches Simulated conflicts using a setup where the schema does not include the new columns: postgres=# select * from pg_conflict.pg_conflict_log_for_subid_16396 ; .... (4 rows) 2. Upgrade using 'pg_upgrade' The upgrade was performed on a cluster initialized with patches v40 + '0001', and it completed successfully. Post-upgrade verification: postgres=# select conflict_type, v20_new_col1, v20_new_col2 from pg_conflict.pg_conflict_log_for_subid_16396 ; conflict_type | v20_new_col1 | v20_new_col2 ---------------+--------------+-------------- insert_exists | | insert_exists | | insert_exists | | insert_exists | | (4 rows) Existing rows were preserved, and the newly added columns are visible and populated with NULLs, as expected. 3. Post-upgrade conflict insertion After starting the old publisher again to continue generating conflicts: postgres=# select conflict_type, v20_new_col1, v20_new_col2 from pg_conflict.pg_conflict_log_for_subid_16396 ; conflict_type | v20_new_col1 | v20_new_col2 ---------------+--------------+-------------- insert_exists | | insert_exists | | insert_exists | | insert_exists | | insert_exists | v20_new_col1 | v20_new_col2 insert_exists | v20_new_col1 | v20_new_col2 insert_exists | v20_new_col1 | v20_new_col2 (7 rows) New conflicts are inserted successfully, and the newly added columns are correctly populated for new entries. Based on this testing, the current 'pg_upgrade' framework, along with the additional dump-time adjustments, appears sufficient to support schema evolution of conflict log tables, specifically for adding new columns in future releases. Thoughts? Regards, Vignesh --000000000000c883010652e1d5b2 Content-Type: application/octet-stream; name="0001-Add-new-columns-to-CLT-and-add-upgrade-changes.patch" Content-Disposition: attachment; filename="0001-Add-new-columns-to-CLT-and-add-upgrade-changes.patch" Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: f_mpplx2t60 RnJvbSBiZWRkNDg2OGNmODExODkzYTIyMjIzNzgxMjRiNzE0NmUyM2FkOGYxIE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiBWaWduZXNoIEMgPHZpZ25lc2gyMUBnbWFpbC5jb20+CkRhdGU6 IFR1ZSwgMjYgTWF5IDIwMjYgMTQ6NDE6MDMgKzA1MzAKU3ViamVjdDogW1BBVENIXSBBZGQgbmV3 IGNvbHVtbnMgdG8gQ0xUIGFuZCBhZGQgdXBncmFkZSBjaGFuZ2VzCgpBZGQgbmV3IGNvbHVtbnMg djIwX25ld19jb2wxIGFuZCB2MjBfbmV3X2NvbDIgb2YgdHlwZSBURVhUCnRvIGNvbmZsaWN0IGxv ZyB0YWJsZXMuCkFsc28gdXBkYXRlIHBnX2R1bXAgYmluYXJ5IHVwZ3JhZGUgbG9naWMgdG8gYWRk IHRoZXNlIGNvbHVtbnMKZHVyaW5nIHBnX3VwZ3JhZGUuCi0tLQogc3JjL2JhY2tlbmQvcmVwbGlj YXRpb24vbG9naWNhbC9jb25mbGljdC5jIHwgIDggKysrKysrLS0KIHNyYy9iaW4vcGdfZHVtcC9w Z19kdW1wLmMgICAgICAgICAgICAgICAgICB8IDE1ICsrKysrKysrKysrKysrKwogMiBmaWxlcyBj aGFuZ2VkLCAyMSBpbnNlcnRpb25zKCspLCAyIGRlbGV0aW9ucygtKQoKZGlmZiAtLWdpdCBhL3Ny Yy9iYWNrZW5kL3JlcGxpY2F0aW9uL2xvZ2ljYWwvY29uZmxpY3QuYyBiL3NyYy9iYWNrZW5kL3Jl cGxpY2F0aW9uL2xvZ2ljYWwvY29uZmxpY3QuYwppbmRleCBhZGY0OWJkYTdhNy4uODRkOTFmZWY5 NDggMTAwNjQ0Ci0tLSBhL3NyYy9iYWNrZW5kL3JlcGxpY2F0aW9uL2xvZ2ljYWwvY29uZmxpY3Qu YworKysgYi9zcmMvYmFja2VuZC9yZXBsaWNhdGlvbi9sb2dpY2FsL2NvbmZsaWN0LmMKQEAgLTcy LDcgKzcyLDkgQEAgc3RhdGljIGNvbnN0IENvbmZsaWN0TG9nQ29sdW1uRGVmIENvbmZsaWN0TG9n U2NoZW1hW10gPSB7CiAJeyAuYXR0bmFtZSA9ICJyZW1vdGVfb3JpZ2luIiwgICAgLmF0dHR5cGlk ID0gVEVYVE9JRCB9LAogCXsgLmF0dG5hbWUgPSAicmVtb3RlX3R1cGxlIiwgICAgIC5hdHR0eXBp ZCA9IEpTT05PSUQgfSwKIAl7IC5hdHRuYW1lID0gInJlcGxpY2FfaWRlbnRpdHkiLCAuYXR0dHlw aWQgPSBKU09OT0lEIH0sCi0JeyAuYXR0bmFtZSA9ICJsb2NhbF9jb25mbGljdHMiLCAgLmF0dHR5 cGlkID0gSlNPTkFSUkFZT0lEIH0KKwl7IC5hdHRuYW1lID0gImxvY2FsX2NvbmZsaWN0cyIsICAu YXR0dHlwaWQgPSBKU09OQVJSQVlPSUQgfSwKKwl7IC5hdHRuYW1lID0gInYyMF9uZXdfY29sMSIs ICAgICAuYXR0dHlwaWQgPSBURVhUT0lEIH0sCisJeyAuYXR0bmFtZSA9ICJ2MjBfbmV3X2NvbDIi LCAgICAgLmF0dHR5cGlkID0gVEVYVE9JRCB9LAogfTsKIAogI2RlZmluZSBOVU1fQ09ORkxJQ1Rf QVRUUlMgbGVuZ3Rob2YoQ29uZmxpY3RMb2dTY2hlbWEpCkBAIC0xMzQ3LDEwICsxMzQ5LDEyIEBA IHByZXBhcmVfY29uZmxpY3RfbG9nX3R1cGxlKEVTdGF0ZSAqZXN0YXRlLCBSZWxhdGlvbiByZWws CiAJZWxzZQogCQludWxsc1thdHRubysrXSA9IHRydWU7CiAKLQl2YWx1ZXNbYXR0bm9dID0gYnVp bGRfbG9jYWxfY29uZmxpY3RzX2pzb25fYXJyYXkoZXN0YXRlLCByZWwsCisJdmFsdWVzW2F0dG5v KytdID0gYnVpbGRfbG9jYWxfY29uZmxpY3RzX2pzb25fYXJyYXkoZXN0YXRlLCByZWwsCiAJCQkJ CQkJCQkJCQkJIGNvbmZsaWN0X3R5cGUsCiAJCQkJCQkJCQkJCQkJIGNvbmZsaWN0dHVwbGVzKTsK IAorCXZhbHVlc1thdHRubysrXSA9IENTdHJpbmdHZXRUZXh0RGF0dW0oInYyMF9uZXdfY29sMSIp OworCXZhbHVlc1thdHRub10gPSBDU3RyaW5nR2V0VGV4dERhdHVtKCJ2MjBfbmV3X2NvbDIiKTsK IAlBc3NlcnQoYXR0bm8gKyAxID09IE5VTV9DT05GTElDVF9BVFRSUyk7CiAKIAlvbGRjdHggPSBN ZW1vcnlDb250ZXh0U3dpdGNoVG8oQXBwbHlDb250ZXh0KTsKZGlmZiAtLWdpdCBhL3NyYy9iaW4v cGdfZHVtcC9wZ19kdW1wLmMgYi9zcmMvYmluL3BnX2R1bXAvcGdfZHVtcC5jCmluZGV4IGYyNjAw MzU4NzlkLi4xODQ0YTlkMjBhNyAxMDA2NDQKLS0tIGEvc3JjL2Jpbi9wZ19kdW1wL3BnX2R1bXAu YworKysgYi9zcmMvYmluL3BnX2R1bXAvcGdfZHVtcC5jCkBAIC01NzY1LDYgKzU3NjUsMjEgQEAg ZHVtcFN1YnNjcmlwdGlvbihBcmNoaXZlICpmb3V0LCBjb25zdCBTdWJzY3JpcHRpb25JbmZvICpz dWJpbmZvKQogCQkJCQkJICBxc3VibmFtZSwKIAkJCQkJCSAgc3ViaW5mby0+c3ViY29uZmxpY3Rs b2dkZXN0KTsKIAorCWlmIChkb3B0LT5iaW5hcnlfdXBncmFkZSAmJiBmb3V0LT5yZW1vdGVWZXJz aW9uID49IDE5MDAwMCAmJgorCQkocGdfc3RyY2FzZWNtcChzdWJpbmZvLT5zdWJjb25mbGljdGxv Z2Rlc3QsICJsb2ciKSAhPSAwKSkKKwl7CisJCWFwcGVuZFBRRXhwQnVmZmVyU3RyKHF1ZXJ5LAor CQkJCQkJICAJICJcblxuU0VUIGFsbG93X3N5c3RlbV90YWJsZV9tb2RzID0gb247XG4iKTsKKwkJ YXBwZW5kUFFFeHBCdWZmZXIocXVlcnksCisJCQkJCQkgICJcblxuQUxURVIgVEFCTEUgcGdfY29u ZmxpY3QucGdfY29uZmxpY3RfbG9nX2Zvcl9zdWJpZF8lZCBBREQgQ09MVU1OIHYyMF9uZXdfY29s MSBURVhUO1xuIiwKKwkJCQkJCSAgc3ViaW5mby0+ZG9iai5jYXRJZC5vaWQpOworCQlhcHBlbmRQ UUV4cEJ1ZmZlcihxdWVyeSwKKwkJCQkJCSAgIlxuXG5BTFRFUiBUQUJMRSBwZ19jb25mbGljdC5w Z19jb25mbGljdF9sb2dfZm9yX3N1YmlkXyVkIEFERCBDT0xVTU4gdjIwX25ld19jb2wyIFRFWFQ7 XG4iLAorCQkJCQkJICBzdWJpbmZvLT5kb2JqLmNhdElkLm9pZCk7CQkJCQkJICAKKwkJYXBwZW5k UFFFeHBCdWZmZXJTdHIocXVlcnksCisJCQkJCQkgIAkgIlxuXG5TRVQgYWxsb3dfc3lzdGVtX3Rh YmxlX21vZHMgPSBvZmY7XG4iKTsKKwl9CisKIAkvKgogCSAqIEluIGJpbmFyeS11cGdyYWRlIG1v ZGUsIHdlIGFsbG93IHRoZSByZXBsaWNhdGlvbiB0byBjb250aW51ZSBhZnRlciB0aGUKIAkgKiB1 cGdyYWRlLgotLSAKMi41My4wCgo= --000000000000c883010652e1d5b2--