public inbox for [email protected]
help / color / mirror / Atom feedFrom: Dilip Kumar <[email protected]>
To: [email protected] <[email protected]>
Cc: Amit Kapila <[email protected]>
Cc: Masahiko Sawada <[email protected]>
Cc: [email protected] <[email protected]>
Cc: Peter Smith <[email protected]>
Cc: [email protected] <[email protected]>
Cc: PostgreSQL Hackers <[email protected]>
Subject: Re: Perform streaming logical transactions by background workers and parallel apply
Date: Thu, 5 Jan 2023 13:51:53 +0530
Message-ID: <CAFiTN-sTYk=h75Jn1a7ee+5hOcdQFjKGDvF_0NWQQXmoBv4A+A@mail.gmail.com> (raw)
In-Reply-To: <OS0PR01MB5716C2CC59810717EC9601EE94FA9@OS0PR01MB5716.jpnprd01.prod.outlook.com>
References: <CAA4eK1KdVRffYEYiv9=4tVWhP=Jkfh1h6FM0A1R0NEe8cLT8+g@mail.gmail.com>
<CAA4eK1JU4J3-Z=fcMcyQw-iG-DNr=u6w_g-iM_4xwg1pKupkFQ@mail.gmail.com>
<OS0PR01MB57163693482C5470C41914AF94EA9@OS0PR01MB5716.jpnprd01.prod.outlook.com>
<CAHut+PuYJz-26QWMSRN_HwiJNfYG4ZxEV3RGE-VyZtXXeFm+cw@mail.gmail.com>
<OS0PR01MB5716C1C1D8244BA86311193894EB9@OS0PR01MB5716.jpnprd01.prod.outlook.com>
<CAA4eK1LrADDNXKRVYox4Q+zSFQvtiQu-7piyd_PCmVjg1gsN2w@mail.gmail.com>
<OS0PR01MB57169EF567017861EACBD10D94E99@OS0PR01MB5716.jpnprd01.prod.outlook.com>
<OS0PR01MB571623EFD8903C93424EFADC94E99@OS0PR01MB5716.jpnprd01.prod.outlook.com>
<OS0PR01MB57166C4E79559C9A5570838C94EC9@OS0PR01MB5716.jpnprd01.prod.outlook.com>
<CAA4eK1+NV9BDPjh5XB9PnZYNdnQht8+2JGRcerJpSUXtYPtQsg@mail.gmail.com>
<OS3PR01MB6275B61076717E4CE9E079D19EED9@OS3PR01MB6275.jpnprd01.prod.outlook.com>
<CAA4eK1LXch0EDB9RCDiqqmM3S9_cam=svMMK08i8Fmspms=6Xg@mail.gmail.com>
<OS3PR01MB62756CC6FFBDD431F2878ACE9EF29@OS3PR01MB6275.jpnprd01.prod.outlook.com>
<CAA4eK1Konv5gm2bZhej_CZqotr=qhUHnhhiJixKLRmCMsg+Egw@mail.gmail.com>
<OS3PR01MB627586DA752A9E07F02FB7609EF09@OS3PR01MB6275.jpnprd01.prod.outlook.com>
<CAA4eK1+63gaq1ruaUTvhM6-GRWi0waZT7X+TJL2tx+dWzisveQ@mail.gmail.com>
<OS3PR01MB6275D4B22266412FEEBDB3AE9EF49@OS3PR01MB6275.jpnprd01.prod.outlook.com>
<CAD21AoC=PCkeURevhZdNjcKHYws=paPcotgG1fAnq9g8GdwGXw@mail.gmail.com>
<OS0PR01MB5716087C537FA8311F208B6094F59@OS0PR01MB5716.jpnprd01.prod.outlook.com>
<CAFiTN-vy5aD2qEe2g8ssh+n=iyusFm84FxPsUyWOMeQdvbN3ow@mail.gmail.com>
<CAA4eK1LdtevYSW_Lk5MJJeCMWqQtUgKCtPkX4FNKVS-ptEBGxw@mail.gmail.com>
<CAFiTN-u_hFNEB6iQTtY9sLV_0fYX87OGH5BWbjTALYgfoHnxhw@mail.gmail.com>
<OS0PR01MB5716C2CC59810717EC9601EE94FA9@OS0PR01MB5716.jpnprd01.prod.outlook.com>
On Thu, Jan 5, 2023 at 9:07 AM [email protected]
<[email protected]> wrote:
>
> On Wednesday, January 4, 2023 9:29 PM Dilip Kumar <[email protected]> wrote:
> > I think this looks good to me.
>
> Thanks for the comments.
> Attach the new version patch set which changed the comments as suggested.
Thanks for the updated patch, while testing this I see one strange
behavior which seems like bug to me, here is the step to reproduce
1. start 2 servers(config: logical_decoding_work_mem=64kB)
./pg_ctl -D data/ -c -l pub_logs start
./pg_ctl -D data1/ -c -l sub_logs start
2. Publisher:
create table t(a int PRIMARY KEY ,b text);
CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS
'select array_agg(md5(g::text))::text from generate_series(1, 256) g';
create publication test_pub for table t
with(PUBLISH='insert,delete,update,truncate');
alter table t replica identity FULL ;
insert into t values (generate_series(1,2000),large_val()) ON CONFLICT
(a) DO UPDATE SET a=EXCLUDED.a*300;
3. Subscription Server:
create table t(a int,b text);
create subscription test_sub CONNECTION 'host=localhost port=5432
dbname=postgres' PUBLICATION test_pub WITH ( slot_name =
test_slot_sub1,streaming=parallel);
4. Publication Server:
begin ;
savepoint a;
delete from t;
savepoint b;
insert into t values (generate_series(1,5000),large_val()) ON CONFLICT
(a) DO UPDATE SET a=EXCLUDED.a*30000; -- (while executing this start
publisher in 2-3 secs)
Restart the publication server, while the transaction is still in an
uncommitted state.
./pg_ctl -D data/ -c -l pub_logs stop -mi
./pg_ctl -D data/ -c -l pub_logs start -mi
after this, the parallel apply worker stuck in waiting on stream lock
forever (even after 10 mins) -- see below, from subscriber logs I can
see one of the parallel apply worker [75677] started but never
finished [no error], after that I have performed more operation [same
insert] which got applied by new parallel apply worked which got
started and finished within 1 second.
dilipku+ 75660 1 0 13:39 ? 00:00:00
/home/dilipkumar/work/PG/install/bin/postgres -D data
dilipku+ 75661 75660 0 13:39 ? 00:00:00 postgres: checkpointer
dilipku+ 75662 75660 0 13:39 ? 00:00:00 postgres: background writer
dilipku+ 75664 75660 0 13:39 ? 00:00:00 postgres: walwriter
dilipku+ 75665 75660 0 13:39 ? 00:00:00 postgres: autovacuum launcher
dilipku+ 75666 75660 0 13:39 ? 00:00:00 postgres: logical
replication launcher
dilipku+ 75675 75595 0 13:39 ? 00:00:00 postgres: logical
replication apply worker for subscription 16389
dilipku+ 75676 75660 0 13:39 ? 00:00:00 postgres: walsender
dilipkumar postgres ::1(42192) START_REPLICATION
dilipku+ 75677 75595 0 13:39 ? 00:00:00 postgres: logical
replication parallel apply worker for subscription 16389 waiting
Subscriber logs:
2023-01-05 13:39:07.261 IST [75595] LOG: background worker "logical
replication worker" (PID 75649) exited with exit code 1
2023-01-05 13:39:12.272 IST [75675] LOG: logical replication apply
worker for subscription "test_sub" has started
2023-01-05 13:39:12.307 IST [75677] LOG: logical replication parallel
apply worker for subscription "test_sub" has started
2023-01-05 13:43:31.003 IST [75596] LOG: checkpoint starting: time
2023-01-05 13:46:32.045 IST [76337] LOG: logical replication parallel
apply worker for subscription "test_sub" has started
2023-01-05 13:46:35.214 IST [76337] LOG: logical replication parallel
apply worker for subscription "test_sub" has finished
2023-01-05 13:46:50.241 IST [76384] LOG: logical replication parallel
apply worker for subscription "test_sub" has started
2023-01-05 13:46:53.676 IST [76384] LOG: logical replication parallel
apply worker for subscription "test_sub" has finished
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
view thread (7+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: Perform streaming logical transactions by background workers and parallel apply
In-Reply-To: <CAFiTN-sTYk=h75Jn1a7ee+5hOcdQFjKGDvF_0NWQQXmoBv4A+A@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox