MIME-Version: 1.0
References: 
 <OSBPR01MB48887CA8F40C8D984A6DC00CED199@OSBPR01MB4888.jpnprd01.prod.outlook.com>
 <CAD21AoAixHnpjG-TtnSejJ2Dv1VsrzGr3oVPSRFhYjz3Z8_XZA@mail.gmail.com>
 <OSBPR01MB4888D9048D3D7E7C3D035D54EDEF9@OSBPR01MB4888.jpnprd01.prod.outlook.com>
 <CAD21AoCbj6YBNmdG9gAXf=dPE+bdY95ENwkh4q5wPL0q=qMzVw@mail.gmail.com>
 <CAA4eK1+tOV-+ssGjj1zq+nAL8a9LfPsxbtyupZGvZ0U7nV0A7g@mail.gmail.com>
 <CAD21AoAO0004L1kv9+uArrhT7d=vFj=2ZUKFbUfpZnHJku-37g@mail.gmail.com>
 <CAA4eK1+CfMrJeA8TR__uFMQVZk50L+OGUEW-8h5G0hD_erfaSw@mail.gmail.com>
 <CAD21AoDtn1bY4CYkBDXMgxpyBQZTAh45HMTXfAjQ3nY_G_=7Mg@mail.gmail.com>
 <CAA4eK1KNqNnWq=wep5OALjqRRKbqp3pRjnqBJi2CPfMYJMUjrQ@mail.gmail.com>
 <CAD21AoA6X7UtbewgRVetcwh2tuAp=umR5n_bE69ZBj3mN6R31Q@mail.gmail.com>
 <CAA4eK1KMT8biciVqTBoZ9gYV-Gf297JFeNhJaxZNmFrZL8m2jA@mail.gmail.com>
 <TY2PR01MB48905729E8CB81805CCF64C0EDCE9@TY2PR01MB4890.jpnprd01.prod.outlook.com>
 <TY2PR01MB489065DB7B8066936087C5E4EDD59@TY2PR01MB4890.jpnprd01.prod.outlook.com>
 <OSBPR01MB4888507DD52192F0CDF9A49DEDDA9@OSBPR01MB4888.jpnprd01.prod.outlook.com>
 <TY2PR01MB489042618A073A2FC7BF62C0EDA29@TY2PR01MB4890.jpnprd01.prod.outlook.com>
 <CAA4eK1JEVoTJNMu94vVdUoak-uefo9V=XJ5bQQ+o47uawDFftg@mail.gmail.com>
 <TY2PR01MB48902C1278A49542EDBB377FEDA89@TY2PR01MB4890.jpnprd01.prod.outlook.com>
 <CAA4eK1+NT3u-hcYzgrOn2JaEQ=V3EvRxOwvY4Vs5GxR6TgQpcA@mail.gmail.com>
 <CAD21AoCnuZu1NA+xF3ZEHikuEKKzVm-NdPM=MZFJ3KZ3jvN+uw@mail.gmail.com>
 <CAA4eK1L3dPeUNu740mu-5znsvXpx2PkRRZ=w1Msf58o=EWSnnQ@mail.gmail.com>
 <CAD21AoA3QUBaMvSk_RBwkAMqEqqLVUF20gp9mCg62xuJG2nwMw@mail.gmail.com>
 <CAA4eK1KzZxCOPqGPiRj0s=xxtgyVv+6Wm-yRoEccnmMV489ZNw@mail.gmail.com>
 <CAD21AoAZchTropRqk2KWVNTDSTLnwVWRGnkyY_KQYvydL42ZKQ@mail.gmail.com>
In-Reply-To: 
 <CAD21AoAZchTropRqk2KWVNTDSTLnwVWRGnkyY_KQYvydL42ZKQ@mail.gmail.com>
From: Amit Kapila <amit.kapila16@gmail.com>
Date: Thu, 30 Sep 2021 08:42:26 +0530
Message-ID: 
 <CAA4eK1K3=jnAHP4UHHLacOAezZTZUcYuNwrKQ21GJENPDYYAaw@mail.gmail.com>
Subject: Re: Failed transaction statistics to measure the logical replication
 progress
To: Masahiko Sawada <sawada.mshk@gmail.com>
Cc: "osumi.takamichi@fujitsu.com" <osumi.takamichi@fujitsu.com>,
	"pgsql-hackers@lists.postgresql.org" <pgsql-hackers@lists.postgresql.org>
Content-Type: text/plain; charset="UTF-8"
Archived-At: 
 <https://www.postgresql.org/message-id/CAA4eK1K3%3DjnAHP4UHHLacOAezZTZUcYuNwrKQ21GJENPDYYAaw%40mail.gmail.com>
Precedence: bulk

On Thu, Sep 30, 2021 at 8:18 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Wed, Sep 29, 2021 at 8:55 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Wed, Sep 29, 2021 at 4:49 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> > > On Tue, Sep 28, 2021 at 7:04 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > >
> > > > On Tue, Sep 28, 2021 at 11:35 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > > >
> > > > > On Tue, Sep 28, 2021 at 1:54 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > > >
> > > > > > >
> > > > > > > Then, if, we proceed in this direction,
> > > > > > > the place to implement those stats
> > > > > > > would be on the LogicalRepWorker struct, instead ?
> > > > > > >
> > > > > >
> > > > > > Or, we can make existing stats persistent and then add these stats on
> > > > > > top of it. Sawada-San, do you have any thoughts on this matter?
> > > > >
> > > > > I think that making existing stats including received_lsn and
> > > > > last_msg_receipt_time persistent by using stats collector could cause
> > > > > massive reporting messages. We can report these messages with a
> > > > > certain interval to reduce the amount of messages but we will end up
> > > > > seeing old stats on the view.
> > > > >
> > > >
> > > > Can't we keep the current and new stats both in-memory and persist on
> > > > disk? So, the persistent stats data will be used to fill the in-memory
> > > > counters after restarting of workers, otherwise, we will always refer
> > > > to in-memory values.
> > >
> > > Interesting. Probably we can have apply workers and table sync workers
> > > send their statistics to the stats collector at exit (before the stats
> > > collector shutting down)? And the startup process will restore them at
> > > restart?
> > >
> >
> > I think we do need to send at the exit but we should probably send at
> > some other regular interval as well to avoid losing all the stats
> > after the crash-restart situation.
>
> But we clear all statistics collected by the stats collector during
> crash recovery.
>

Right. So, maybe if we do what you are suggesting is sufficient.

-- 
With Regards,
Amit Kapila.