MIME-Version: 1.0
References: 
 <CAGfChW62f5NTNbLsqO-6_CrmKPqBEQtWPcPDafu8pCwZznk=xw@mail.gmail.com>
 <CALj2ACXYGQ2nYF0LOYi5u7xBqH+MEb8eWsTEvJrx_QrC+107Zw@mail.gmail.com>
 <CAA4eK1+qVztg-noRzsHnAqrPgwNLb=YZC4Ri9EeUS6sdBdkfJw@mail.gmail.com>
In-Reply-To: 
 <CAA4eK1+qVztg-noRzsHnAqrPgwNLb=YZC4Ri9EeUS6sdBdkfJw@mail.gmail.com>
From: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Date: Thu, 18 Jan 2024 16:44:00 +0530
Message-ID: 
 <CALj2ACVGxXpPi822ESNyD2zz3xhuAqwjCYpVLeN9DUFMU-1pdQ@mail.gmail.com>
Subject: Re: Reduce useless changes before reassembly during logical
 replication
To: Amit Kapila <amit.kapila16@gmail.com>
Cc: li jie <ggysxcq@gmail.com>,
	PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: 
 <https://www.postgresql.org/message-id/CALj2ACVGxXpPi822ESNyD2zz3xhuAqwjCYpVLeN9DUFMU-1pdQ%40mail.gmail.com>
Precedence: bulk

On Thu, Jan 18, 2024 at 2:47=E2=80=AFPM Amit Kapila <amit.kapila16@gmail.co=
m> wrote:
>
> On Thu, Jan 18, 2024 at 12:12=E2=80=AFPM Bharath Rupireddy
> <bharath.rupireddyforpostgres@gmail.com> wrote:
> >
> > On Wed, Jan 17, 2024 at 11:45=E2=80=AFAM li jie <ggysxcq@gmail.com> wro=
te:
> > >
> > > Hi hackers,
> > >
> > > During logical replication, if there is a large write transaction, so=
me
> > > spill files will be written to disk, depending on the setting of
> > > logical_decoding_work_mem.
> > >
> > > This behavior can effectively avoid OOM, but if the transaction
> > > generates a lot of change before commit, a large number of files may
> > > fill the disk. For example, you can update a TB-level table.
> > >
> > > However, I found an inelegant phenomenon. If the modified large table=
 is not
> > > published, its changes will also be written with a large number of sp=
ill files.
> > > Look at an example below:
> >
> > Thanks. I agree that decoding and queuing the changes of unpublished
> > tables' data into reorder buffer is an unnecessary task for walsender.
> > It takes processing efforts (CPU overhead), consumes disk space and
> > uses memory configured via logical_decoding_work_mem for a replication
> > connection inefficiently.
> >
>
> This is all true but note that in successful cases (where the table is
> published) all the work done by FilterByTable(accessing caches,
> transaction-related stuff) can add noticeable overhead as anyway we do
> that later in pgoutput_change().

Right. Overhead for published tables need to be studied. A possible
way is to mark the checks performed in
FilterByTable/filter_by_table_cb and skip the same checks in
pgoutput_change. I'm not sure if this works without any issues though.

--=20
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com