public inbox for [email protected]
help / color / mirror / Atom feedFrom: Amit Kapila <[email protected]>
To: [email protected] <[email protected]>
Cc: Masahiko Sawada <[email protected]>
Cc: [email protected] <[email protected]>
Subject: Re: Failed transaction statistics to measure the logical replication progress
Date: Wed, 29 Sep 2021 16:21:21 +0530
Message-ID: <CAA4eK1LSKWM6-5KfQfWOsaqTmnu+uaqab4GzcxxzNc-R3u3RSw@mail.gmail.com> (raw)
In-Reply-To: <OSBPR01MB4888B9F62644397398240DC3EDA99@OSBPR01MB4888.jpnprd01.prod.outlook.com>
References: <OSBPR01MB48887CA8F40C8D984A6DC00CED199@OSBPR01MB4888.jpnprd01.prod.outlook.com>
<CAD21AoAixHnpjG-TtnSejJ2Dv1VsrzGr3oVPSRFhYjz3Z8_XZA@mail.gmail.com>
<OSBPR01MB4888D9048D3D7E7C3D035D54EDEF9@OSBPR01MB4888.jpnprd01.prod.outlook.com>
<CAD21AoCbj6YBNmdG9gAXf=dPE+bdY95ENwkh4q5wPL0q=qMzVw@mail.gmail.com>
<CAA4eK1+tOV-+ssGjj1zq+nAL8a9LfPsxbtyupZGvZ0U7nV0A7g@mail.gmail.com>
<CAD21AoAO0004L1kv9+uArrhT7d=vFj=2ZUKFbUfpZnHJku-37g@mail.gmail.com>
<CAA4eK1+CfMrJeA8TR__uFMQVZk50L+OGUEW-8h5G0hD_erfaSw@mail.gmail.com>
<CAD21AoDtn1bY4CYkBDXMgxpyBQZTAh45HMTXfAjQ3nY_G_=7Mg@mail.gmail.com>
<CAA4eK1KNqNnWq=wep5OALjqRRKbqp3pRjnqBJi2CPfMYJMUjrQ@mail.gmail.com>
<CAD21AoA6X7UtbewgRVetcwh2tuAp=umR5n_bE69ZBj3mN6R31Q@mail.gmail.com>
<CAA4eK1KMT8biciVqTBoZ9gYV-Gf297JFeNhJaxZNmFrZL8m2jA@mail.gmail.com>
<TY2PR01MB48905729E8CB81805CCF64C0EDCE9@TY2PR01MB4890.jpnprd01.prod.outlook.com>
<TY2PR01MB489065DB7B8066936087C5E4EDD59@TY2PR01MB4890.jpnprd01.prod.outlook.com>
<OSBPR01MB4888507DD52192F0CDF9A49DEDDA9@OSBPR01MB4888.jpnprd01.prod.outlook.com>
<TY2PR01MB489042618A073A2FC7BF62C0EDA29@TY2PR01MB4890.jpnprd01.prod.outlook.com>
<CAA4eK1JEVoTJNMu94vVdUoak-uefo9V=XJ5bQQ+o47uawDFftg@mail.gmail.com>
<TY2PR01MB48902C1278A49542EDBB377FEDA89@TY2PR01MB4890.jpnprd01.prod.outlook.com>
<CAA4eK1+NT3u-hcYzgrOn2JaEQ=V3EvRxOwvY4Vs5GxR6TgQpcA@mail.gmail.com>
<CAD21AoCnuZu1NA+xF3ZEHikuEKKzVm-NdPM=MZFJ3KZ3jvN+uw@mail.gmail.com>
<CAA4eK1L3dPeUNu740mu-5znsvXpx2PkRRZ=w1Msf58o=EWSnnQ@mail.gmail.com>
<OSBPR01MB4888B9F62644397398240DC3EDA99@OSBPR01MB4888.jpnprd01.prod.outlook.com>
On Wed, Sep 29, 2021 at 11:35 AM [email protected]
<[email protected]> wrote:
>
> Hi,
>
>
> Thank you, Amit-san and Sawada-san for the discussion.
> On Tuesday, September 28, 2021 7:05 PM Amit Kapila <[email protected]> wrote:
> > > Another idea could be to have a separate view, say
> > > pg_stat_subscription_xact but I'm not sure it's a better idea.
> > >
> >
> > Yeah, that is another idea but I am afraid that having three different
> > views for subscription stats will be too much. I think it would be
> > better if we can display these additional stats via the existing view
> > pg_stat_subscription or the new view pg_stat_subscription_errors (or
> > whatever name we want to give it).
> pg_stat_subscription_errors specializes in showing an error record.
> So, it would be awkward to combine it with other normal xact stats.
>
>
> > > > > Then, if, we proceed in this direction, the place to implement
> > > > > those stats would be on the LogicalRepWorker struct, instead ?
> > > > >
> > > >
> > > > Or, we can make existing stats persistent and then add these stats
> > > > on top of it. Sawada-San, do you have any thoughts on this matter?
> > >
> > > I think that making existing stats including received_lsn and
> > > last_msg_receipt_time persistent by using stats collector could cause
> > > massive reporting messages. We can report these messages with a
> > > certain interval to reduce the amount of messages but we will end up
> > > seeing old stats on the view.
> > >
> >
> > Can't we keep the current and new stats both in-memory and persist on disk?
> > So, the persistent stats data will be used to fill the in-memory counters after
> > restarting of workers, otherwise, we will always refer to in-memory values.
> I felt this isn't impossible.
> When we have to update the values of the xact stats is
> the end of message apply for COMMIT, COMMIT PREPARED, STREAM_ABORT and etc
> or the time when an error happens during apply. Then, if we want,
> we can update xact stats values at such moments accordingly.
> I'm thinking that we will have a hash table whose key is a pair of subid + relid
> and entry is a proposed stats structure and update the entry,
> depending on the above timings.
>
Are you thinking of a separate hash table then what we are going to
create for Sawada-San's patch related to error stats? Isn't it
possible to have stats in the same hash table and same file?
> Here, one thing a bit unclear to me is
> whether we should move existing stats of pg_stat_subscription
> (such as last_lsn and reply_lsn) to the hash entry or not.
>
I think we should move it to hash entry. I think that is an
improvement over what we have now because now after restart those
stats gets lost.
--
With Regards,
Amit Kapila.
view thread (113+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected]
Subject: Re: Failed transaction statistics to measure the logical replication progress
In-Reply-To: <CAA4eK1LSKWM6-5KfQfWOsaqTmnu+uaqab4GzcxxzNc-R3u3RSw@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox