public inbox for [email protected]  

help / color / mirror / Atom feed

From: Bharath Rupireddy <[email protected]>
To: Amit Kapila <[email protected]>
Cc: li jie <[email protected]>
Cc: PostgreSQL Hackers <[email protected]>
Subject: Re: Reduce useless changes before reassembly during logical replication
Date: Thu, 18 Jan 2024 16:44:00 +0530
Message-ID: <CALj2ACVGxXpPi822ESNyD2zz3xhuAqwjCYpVLeN9DUFMU-1pdQ@mail.gmail.com> (raw)
In-Reply-To: <CAA4eK1+qVztg-noRzsHnAqrPgwNLb=YZC4Ri9EeUS6sdBdkfJw@mail.gmail.com>
References: <CAGfChW62f5NTNbLsqO-6_CrmKPqBEQtWPcPDafu8pCwZznk=xw@mail.gmail.com>
	<CALj2ACXYGQ2nYF0LOYi5u7xBqH+MEb8eWsTEvJrx_QrC+107Zw@mail.gmail.com>
	<CAA4eK1+qVztg-noRzsHnAqrPgwNLb=YZC4Ri9EeUS6sdBdkfJw@mail.gmail.com>

On Thu, Jan 18, 2024 at 2:47 PM Amit Kapila <[email protected]> wrote:
>
> On Thu, Jan 18, 2024 at 12:12 PM Bharath Rupireddy
> <[email protected]> wrote:
> >
> > On Wed, Jan 17, 2024 at 11:45 AM li jie <[email protected]> wrote:
> > >
> > > Hi hackers,
> > >
> > > During logical replication, if there is a large write transaction, some
> > > spill files will be written to disk, depending on the setting of
> > > logical_decoding_work_mem.
> > >
> > > This behavior can effectively avoid OOM, but if the transaction
> > > generates a lot of change before commit, a large number of files may
> > > fill the disk. For example, you can update a TB-level table.
> > >
> > > However, I found an inelegant phenomenon. If the modified large table is not
> > > published, its changes will also be written with a large number of spill files.
> > > Look at an example below:
> >
> > Thanks. I agree that decoding and queuing the changes of unpublished
> > tables' data into reorder buffer is an unnecessary task for walsender.
> > It takes processing efforts (CPU overhead), consumes disk space and
> > uses memory configured via logical_decoding_work_mem for a replication
> > connection inefficiently.
> >
>
> This is all true but note that in successful cases (where the table is
> published) all the work done by FilterByTable(accessing caches,
> transaction-related stuff) can add noticeable overhead as anyway we do
> that later in pgoutput_change().

Right. Overhead for published tables need to be studied. A possible
way is to mark the checks performed in
FilterByTable/filter_by_table_cb and skip the same checks in
pgoutput_change. I'm not sure if this works without any issues though.

-- 
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

view thread (8+ messages)  latest in thread

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected], [email protected], [email protected], [email protected]
  Subject: Re: Reduce useless changes before reassembly during logical replication
  In-Reply-To: <CALj2ACVGxXpPi822ESNyD2zz3xhuAqwjCYpVLeN9DUFMU-1pdQ@mail.gmail.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox