public inbox for [email protected]
help / color / mirror / Atom feedFrom: Masahiko Sawada <[email protected]>
To: [email protected] <[email protected]>
Cc: Amit Kapila <[email protected]>
Cc: shveta malik <[email protected]>
Cc: Peter Smith <[email protected]>
Cc: [email protected] <[email protected]>
Cc: [email protected] <[email protected]>
Cc: PostgreSQL Hackers <[email protected]>
Cc: Dilip Kumar <[email protected]>
Subject: Re: Perform streaming logical transactions by background workers and parallel apply
Date: Fri, 13 Jan 2023 14:43:21 +0900
Message-ID: <CAD21AoDBQ5hVi4hLmiU1D1JTMWtdj9NU5N4__c9Aqs8tUp0dsQ@mail.gmail.com> (raw)
In-Reply-To: <OS0PR01MB5716C663C85687E76672327094FD9@OS0PR01MB5716.jpnprd01.prod.outlook.com>
References: <CAA4eK1LGmZsevrqJra0V4O8oBU_eKyzm2VMpSAYQaDgC6n4fkA@mail.gmail.com>
<CAFiTN-uhnJkrUsQByBHMK268T-GEx_D8DQ6b2T+aW6RiU75pbQ@mail.gmail.com>
<OS0PR01MB571621ED532C2D7C3E01625894FB9@OS0PR01MB5716.jpnprd01.prod.outlook.com>
<CAFiTN-t5+_zTf86EzgCwObs1ED-P_5Ab1KMw340d3oPY_+Cwpw@mail.gmail.com>
<OS3PR01MB5718AE486227CE3ACB844C5E94F89@OS3PR01MB5718.jpnprd01.prod.outlook.com>
<CAFiTN-sgQ-K9BnMWKLR7Hm36QNjS7ZOBF6hsXJmQz5ChC0Desw@mail.gmail.com>
<CAA4eK1+P9WOFBupaTZCzYtranSH=8TSUYV=kk5xQJ-VNKnvgmQ@mail.gmail.com>
<OS0PR01MB571685A645E509267AC5F30894F99@OS0PR01MB5716.jpnprd01.prod.outlook.com>
<OS0PR01MB57168A5A1C31BC316CB63D3C94F99@OS0PR01MB5716.jpnprd01.prod.outlook.com>
<CAA4eK1KK7Xty7SaCReY_b5moPURowMGOsfvhOb6xQ1EAaJAh6A@mail.gmail.com>
<OS0PR01MB5716863B23DBAD186F64ADF194FF9@OS0PR01MB5716.jpnprd01.prod.outlook.com>
<CAHut+PvA10Bp9Jaw9OS2+puKHr7ry_xB3Tf2-bbv5gyxD5E_gw@mail.gmail.com>
<CAA4eK1KYUbnthSPyo4VjnhMygB0c1DZtp0XC-V2-GSETQ743ww@mail.gmail.com>
<CAJpy0uC0JamKT6pTtux3NDhcqSMJnDoQXZxx32pq=2yyxQ1c5Q@mail.gmail.com>
<CAA4eK1KueUT_vr1cg43MpvRcaVvW03i6zDECQNAwFBK5PhXOVA@mail.gmail.com>
<OS0PR01MB5716C663C85687E76672327094FD9@OS0PR01MB5716.jpnprd01.prod.outlook.com>
On Thu, Jan 12, 2023 at 9:34 PM [email protected]
<[email protected]> wrote:
>
> On Thursday, January 12, 2023 7:08 PM Amit Kapila <[email protected]> wrote:
> >
> > On Thu, Jan 12, 2023 at 4:21 PM shveta malik <[email protected]> wrote:
> > >
> > > On Thu, Jan 12, 2023 at 10:34 AM Amit Kapila <[email protected]>
> > wrote:
> > > >
> > > > On Thu, Jan 12, 2023 at 9:54 AM Peter Smith <[email protected]>
> > wrote:
> > > > >
> > > > >
> > > > > doc/src/sgml/monitoring.sgml
> > > > >
> > > > > 5. pg_stat_subscription
> > > > >
> > > > > @@ -3198,11 +3198,22 @@ SELECT pid, wait_event_type, wait_event
> > > > > FROM pg_stat_activity WHERE wait_event i
> > > > >
> > > > > <row>
> > > > > <entry role="catalog_table_entry"><para
> > > > > role="column_definition">
> > > > > + <structfield>apply_leader_pid</structfield>
> > <type>integer</type>
> > > > > + </para>
> > > > > + <para>
> > > > > + Process ID of the leader apply worker, if this process is a apply
> > > > > + parallel worker. NULL if this process is a leader apply worker or a
> > > > > + synchronization worker.
> > > > > + </para></entry>
> > > > > + </row>
> > > > > +
> > > > > + <row>
> > > > > + <entry role="catalog_table_entry"><para
> > > > > + role="column_definition">
> > > > > <structfield>relid</structfield> <type>oid</type>
> > > > > </para>
> > > > > <para>
> > > > > OID of the relation that the worker is synchronizing; null for the
> > > > > - main apply worker
> > > > > + main apply worker and the parallel apply worker
> > > > > </para></entry>
> > > > > </row>
> > > > >
> > > > > 5a.
> > > > >
> > > > > (Same as general comment #1 about terminology)
> > > > >
> > > > > "apply_leader_pid" --> "leader_apply_pid"
> > > > >
> > > >
> > > > How about naming this as just leader_pid? I think it could be
> > > > helpful in the future if we decide to parallelize initial sync (aka
> > > > parallel
> > > > copy) because then we could use this for the leader PID of parallel
> > > > sync workers as well.
> > > >
> > > > --
> > >
> > > I still prefer leader_apply_pid.
> > > leader_pid does not tell which 'operation' it belongs to. 'apply'
> > > gives the clarity that it is apply related process.
> > >
> >
> > But then do you suggest that tomorrow if we allow parallel sync workers then
> > we have a separate column leader_sync_pid? I think that doesn't sound like a
> > good idea and moreover one can refer to docs for clarification.
>
> I agree that leader_pid would be better not only for future parallel copy sync feature,
> but also it's more consistent with the leader_pid column in pg_stat_activity.
>
> And here is the version patch which addressed Peter's comments and renamed all
> the related stuff to leader_pid.
Here are two comments on v79-0003 patch.
+ /* Force to serialize messages if stream_serialize_threshold
is reached. */
+ if (stream_serialize_threshold != -1 &&
+ (stream_serialize_threshold == 0 ||
+ stream_serialize_threshold < parallel_stream_nchunks))
+ {
+ parallel_stream_nchunks = 0;
+ return false;
+ }
I think it would be better if we show the log message ""logical
replication apply worker will serialize the remaining changes of
remote transaction %u to a file" even in stream_serialize_threshold
case.
IIUC parallel_stream_nchunks won't be reset if pa_send_data() failed
due to the timeout.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
view thread (105+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: Perform streaming logical transactions by background workers and parallel apply
In-Reply-To: <CAD21AoDBQ5hVi4hLmiU1D1JTMWtdj9NU5N4__c9Aqs8tUp0dsQ@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox