Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1qFHdo-0005SR-5W for pgsql-bugs@arkaria.postgresql.org; Fri, 30 Jun 2023 17:10:40 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1qFHdm-0002A4-TI for pgsql-bugs@arkaria.postgresql.org; Fri, 30 Jun 2023 17:10:38 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1qFHdm-00029t-Jp for pgsql-bugs@lists.postgresql.org; Fri, 30 Jun 2023 17:10:38 +0000 Received: from sonic310-20.consmr.mail.sg3.yahoo.com ([106.10.244.140]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1qFHdj-001Oin-1z for pgsql-bugs@lists.postgresql.org; Fri, 30 Jun 2023 17:10:37 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1688145029; bh=cRxLR4VqO1J2qvrNaRz0MRG/FrP8zY67jQwwrdgSuXo=; h=Date:From:To:Subject:References:From:Subject:Reply-To; b=tMoVt/+8BtVYnrrN38XHJ8MvncLnVlTBK1Qxl0QxnBNN/Rwq+wTMi0aOX8fiWCFsUiWw1quov9RcIGTEqoCQrNHuHAK2CZa/iEkSZEndcJjrFfhTH24qVoaMi7pyhIfTjC98epWCCCFD/FWNpXnovKwzAZymO1Xp//wSOcQecm6Wy10CbYsHCVMnonREGxvhM/BX5xo3v51EVwoj4HQIFknkevG1SzWXuGbh9IJWm6SPmtQbSjeUD9FWshXZHOkZvPy2F+oLX1QHwrm7q2Pg0DC0ni6K6rYn04AIm8P7vwhDMrFgt4OSZO+SArvgO6mcHVFJIR/EcjZ0DzTWwdmKGw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1688145029; bh=dKbh6k+kbCixSS9ULonz6f+ky4sDiEqe8ZDqynx01ax=; h=X-Sonic-MF:Date:From:To:Subject:From:Subject; b=oKJgGvpk7cFkDbP8uVJFT3MCYv7VpokwOUO0V06YDaBq65X5JOVf5olqpUAZt01nkX9YOWMRGeUdWN/q0X/cQvB+/Ch4jJiUr4/vag/gxRzeidV5D7KEOykVSZVFg96ZlFHyweKJB+MpYC66OqVR5oIY/t0pt0sT1s8Lfnh1qZ3CspOtamfspxaDdzdsuG8JlztgguYNd0ogLiaXizFMON9Zboh/pZs4weONvZVAIxNR1f3wfKzthu1ctvUNVJgE6Dc6YlVEpOJCGEdwFl49PUg0wstMEE52Nzf7/qxfjEQbG2ClncwBK8s3ZHrP2FNH73O57UvP9lX78tZNYRHj8g== X-YMail-OSG: kOxe.PQVM1kKdxs7D_uSC1K3RY1LR2E0xQjga_Q1EEhlTWF2GW2eGDfhrf3wvg_ mhE0188uASTHgBfvgISlGbNnoAhVlwiQpddCCVs_xx5Is3ebUjI3CEF46ZLwx43WD2wnOnpYEfeW k5Ffan0BjTKXubZzGJUf7w.nP0xMNWZKk0mZEECrV6utUGhfulYY5cGb7tG2ZNQ4OkSxFrJbvzrU o8hQ7KKVki2s4XYOfmcJXkKaypT03_WtAAo0W0mXl8inaMeORDbRmaosGQTkucfokkQ_5g1Kswkt xKJyELOAtbdxCh5Vlpubkfpa88s.Qqm2X42vvYk0Q_u1EJmg3noKq.wHL_UIe2dJVrvLXt9bhy6Z O5kRswo4d3IwcGpWzypYx7QGzeRFVF8MkLJLrN9kihwvL4ZJ86xmSCn5lGw06uz0T6FLMK6LLzgO OBxpxRQEoLzoThbtFo7u65a9QEIIcGnNAF_UygoMuxvBHX.yxYCisPJb0XTqmaGdsg2jtw_Gwl_D nEACwyTP6EY84ebyzzEdNgF7.L446LkHowGD8tDA6wFm1NCJ3hg13HGrcli1KEqqPBglBf9BwGrT 8HnDo.CZvxJEgXXo5.9uTPHU8CazDrleJHEurFTPeKqi0eulkPB6FQmYlNYTjzY6IX_hc_37QnvX bP.oC4nuxiI8Fp7OS7Swa2V30E6PZfD3YFDm2XOK7RviYD7ibpZHDdmE602v1U_oOVMpxIC8uAC6 8Q41ss2sWlDV6cHP8Z6zW1gc3wzy1RJvMyseY8TGkPFIQlYHMM81u8zXMXgZ8Vk9qL9yFLURkGO. nOLP2s6FUKE0bAz5wqz59gMpu4pdYC2yGp2InPGSkxHToJ1Jk.tgVenIcjjlW9SFHI8WdN4zjmxN gm0sm3mr3tkN5ZOEaztHiGB2Ys_jtttH0HdFctviZxezSXZwHYCph.xUNdW0B1VtAPt1lhj0Qobr NvrXhtxLUvUhNTeaTaakgMSCdTG..WipGlHiXYdupgWeH7scaWb9BBtR48oifREYcOvrCmxss0Is WO4EkCrJBAjjAsc5Rs9o5wUqpDGHABnGPfX9f9H_uf4V1l.DcLRUh8xBDs712vS28F7cvxeQGP4l cvxLIbNH19HTbfQvtx29jeRyUL8jyAm1mo5LtBXark2uUJ_.nPNkqcFEWfDf7tv9LgF61oWHM3x7 FSMke3hylnayxhiMpvNLoGqfAgUUBn1qvnWP_O6QwzO0DymTGJ18suCJ1KitB_LETRcse45VsD4d vYVvU1HVOgOZCAftj4_tFLN4UjkUcjRmZ_uuBq2x2Zg0QgknF1RM.Htz_Nd2G2arOqHSIFplW__7 ThgeiGMTMmoiCyLE8om3wWCzylxPRcLPY0.tM670c943e8RdYpa2_2Xpy9q2rFbqoczf7hdrv8gU UKQGdAF9SjQyKjChpuAuZyig4mTvWJH9PqOVseEkRqBQ04_ANvhQoddzyIpn0uZJSJoPSbHqIf3t CsyeO0ZiGP12P0Odb2Q9Ntb9P1bMrYmLJ9QsWui1t3DQo9EnkdolY54bNEZ4.fo.8JZie1a_tx8w GxoBLR1PwHOLpxqBJciQp1NPgN9sNO_YVlU47a7Np25mMaUZmz.NDlk4WITpOgCXA_Wa5LN_wIdy 7bng1UnMG5YLyGfb0PvjbeYne2QZsfJmgkV_sV3SGU9XScOjCVp1N5SfnlEao2CsjEK8v6AXyXRT 8IUrUpH_622.fDZ.DoSnvluAzys4apSEWSGLVzfLGMv_8nspocQtnPPh_SDAEgDkebTG1tTso61r RbSgdwxkt54fUj4Ui9m42P06fBT8T18GSMcSkpn2EEemuoK4Yi83Oxd5frsXTsqfcvd.FtaB5Kt8 RT9nEwsPmW0.hEFU7sytWwiVuuiCjNgOX8Z.eU8k5UM0Qc.CAWdMOP.Ruy40R9C9RZg4HDt3j4DV 4h7V_0N39QbGiuhMZoxRKAMcAsZBofn8Y4VWz9rcdAVKfhnJBRzlnhgG30nYx2IAVmKruPuGFe7e HHLK0RFO9M1EgJRSm4j7aNLc_K4PmPFHIXPbwNjqBd8Ijr7ZZoHpDxW6x0smTqxXGSegRU3z1HlQ F6m554HHr7N3D49vLKwoAprEKkUBbThlixkt4GIbupVa7gkz79lsFSQ7UBIvriL74ZWf74l.rp6r oNbSa1VwkRoDgBFK9_yfjm9yii3cqYrHfRO5ty02Ro1pu40HgRWzvC.h4dX4w X-Sonic-MF: X-Sonic-ID: 6be2db09-ecea-4524-a728-653d2a5eef70 Received: from sonic.gate.mail.ne1.yahoo.com by sonic310.consmr.mail.sg3.yahoo.com with HTTP; Fri, 30 Jun 2023 17:10:29 +0000 Date: Fri, 30 Jun 2023 17:10:25 +0000 (UTC) From: Chandy G To: "pgsql-bugs@lists.postgresql.org" Message-ID: <780628120.613465.1688145025701@mail.yahoo.com> Subject: Toasted column values during replication. (Enhancement) MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_613464_1260155346.1688145025700" References: <780628120.613465.1688145025701.ref@mail.yahoo.com> X-Mailer: WebService/1.1.21612 YMailNorrin Content-Length: 4411 List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk ------=_Part_613464_1260155346.1688145025700 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable =C2=A0 We are running into the postgres toasted values showing up in replic= ation slots. We are using debezium and have gone through the following 1. Various options for handling toasted values during replication at the ap= plication level.2. Postgres option to turn on replica identitiy to full. While these are solutions to the problem, each of them comes with their own= downside of=20 =C2=A0(a) custom implementation needed for each consumer.=C2=A0(b) replica = identity to full - wal log file size increase.=C2=A0 worst case: all tables= have columns have toastable columns.=20 Wanted to understand if there is any scope for enhancement / improvements i= n postgres itself to handle such scenarios. Enhancement request: =C2=A0Can the postgres internal component (wal-sender = / *) interpret this special toasted-col-value sitting in the wal-file -> re= place it with the actual value and send it to the replication slot. I presu= me, today the wal-sender=C2=A0 does not have this support and it just sends= whatever data that the wal log has.=C2=A0 So if the wal-sender - can inter= cept such toasted values and translate it to actual values before being sen= t - that would solve the problem without every consumer having to implement= a solution to handle toasted value. This would also help prevent the unwar= ranted increase of the wal-log if replica-identity to full is set as a w/a. Thanks. ------=_Part_613464_1260155346.1688145025700 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 7bit
  We are running into the postgres toasted values showing up in replication slots. We are using debezium and have gone through the following

1. Various options for handling toasted values during replication at the application level.
2. Postgres option to turn on replica identitiy to full.

While these are solutions to the problem, each of them comes with their own downside of
 (a) custom implementation needed for each consumer.
 (b) replica identity to full - wal log file size increase.  worst case: all tables have columns have toastable columns.


Wanted to understand if there is any scope for enhancement / improvements in postgres itself to handle such scenarios.

Enhancement request:  Can the postgres internal component (wal-sender / *) interpret this special toasted-col-value sitting in the wal-file -> replace it with the actual value and send it to the replication slot. I presume, today the wal-sender  does not have this support and it just sends whatever data that the wal log has.  So if the wal-sender - can intercept such toasted values and translate it to actual values before being sent - that would solve the problem without every consumer having to implement a solution to handle toasted value. This would also help prevent the unwarranted increase of the wal-log if replica-identity to full is set as a w/a.

Thanks.
------=_Part_613464_1260155346.1688145025700--