Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

pg.ddx.io  pgsql-hackers@postgresql.org mailing list archive  
help / color / mirror / Atom feed

From: Erik Rijkers <er@xs4all.nl>
To: Dilip Kumar <dilipbalaut@gmail.com>
Cc: Kuntal Ghosh <kuntalghosh.2007@gmail.com>
Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>
Cc: Amit Kapila <amit.kapila16@gmail.com>
Cc: Michael Paquier <michael@paquier.xyz>
Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
Cc: PostgreSQL Hackers <pgsql-hackers@postgresql.org>
Subject: Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
Date: Sat, 18 Apr 2020 11:07:48 +0200
Message-ID: <bcc76806992e913a6862e64d633a166a@xs4all.nl> (raw)
In-Reply-To: <36881eec90df541999a4e0a4b2eec60d@xs4all.nl>
References: <CAFiTN-sn0Bd4Q72Cn9t8-mZp1sxWRv2SmUoeR_7J05SqNTR--g@mail.gmail.com>
	<CAA4eK1KyrP6m53FQV5v0gysEnaTHPdc2xYT0KeNN+fhOcedRwA@mail.gmail.com>
	<CAFiTN-uBd3p50s=kfEiSbg93_Fnf_VU+haS4FqBZAm7PmbX69g@mail.gmail.com>
	<CAA4eK1LkLU5frY4kZGO38d8m6R504H+GbfkwsQLAp7sQ-4oqbw@mail.gmail.com>
	<20200329005905.ooxq25qyjwn2mvxz@development>
	<CAA4eK1LvUDNaCJUZEh5BhV31U6OnNWP=OCUmyZM2k4JGx5CXvQ@mail.gmail.com>
	<20200329153105.mdryxzz562tg65pk@development>
	<CAA4eK1JHxMaeQhvfpaBpUCLoeg3=RbbQQ_h0O0WwOU2kssRFsg@mail.gmail.com>
	<20200330152758.fgp62lwb2ug7cl2a@development>
	<CAA4eK1JMdBkYQF-fjSesb713Fib_v+ihFO-ZdDWpio8CQ7Yr_A@mail.gmail.com>
	<20200408005905.misvqmjk5c7wd5vr@development>
	<CAFiTN-uF4O+Uh_dQX1Um1xJYSKCBUrEB2o-etumUch8F3cEhGA@mail.gmail.com>
	<CAGz5QCJZx_V96e2SrJ9RnuNHk=kbb4uKFsXaymP_Wykma1h5_g@mail.gmail.com>
	<CAFiTN-tWk1mztXSEyfm=kHbJsRUOkXrGs4frk0qWpwKDyuD57A@mail.gmail.com>
	<CAGz5QC+DMNxD8wJyxBtJR7c2krJTa1m_vF6XDz--EVfcgF+qzg@mail.gmail.com>
	<CAFiTN-tLpNYVU7++teYq5gCYn8doOBawLB2pFSCLHV_c0OqsnQ@mail.gmail.com>
	<CAGz5QCLJK0QxV3jBcnEWd+cjz9en22=adQNFvmqpkCmR37wvZg@mail.gmail.com>
	<CAFiTN-vy3_KA-+kTff1=UvKasp0r1BDScDuJv5hN2SEw1PHn_A@mail.gmail.com>
	<f7f89dd6fb80780c2855b43a8d096ba2@xs4all.nl>
	<CAFiTN-tz+jjGHk4dkW=v==fJQZidobB7vp=_yhrT_ij0OLkiBA@mail.gmail.com>
	<36881eec90df541999a4e0a4b2eec60d@xs4all.nl>

On 2020-04-16 11:46, Erik Rijkers wrote:
> On 2020-04-16 11:33, Dilip Kumar wrote:
>> On Tue, Apr 14, 2020 at 9:14 PM Erik Rijkers <er@xs4all.nl> wrote:
>>> 
>>> On 2020-04-14 12:10, Dilip Kumar wrote:
>>> 
>>> > v14-0001-Immediately-WAL-log-assignments.patch                 +
>>> > v14-0002-Issue-individual-invalidations-with.patch             +
>>> > v14-0003-Extend-the-output-plugin-API-with-stream-methods.patch+
>>> > v14-0004-Gracefully-handle-concurrent-aborts-of-uncommitt.patch+
>>> > v14-0005-Implement-streaming-mode-in-ReorderBuffer.patch       +
>>> > v14-0006-Add-support-for-streaming-to-built-in-replicatio.patch+
>>> > v14-0007-Track-statistics-for-streaming.patch                  +
>>> > v14-0008-Enable-streaming-for-all-subscription-TAP-tests.patch +
>>> > v14-0009-Add-TAP-test-for-streaming-vs.-DDL.patch              +
>>> > v14-0010-Bugfix-handling-of-incomplete-toast-tuple.patch
>>> 
>>> applied on top of 8128b0c (a few hours ago)
>> 
> 
> I've added your new patch
> 
> [bugfix_replica_identity_full_on_subscriber.patch]
> 
> on top of all those above but the crash (apparently the same crash)
> that I had earlier still occurs (and pretty soon).
> 
> server process (PID 1721) was terminated by signal 11: Segmentation 
> fault
> 
> I'll try to isolate it better and get a stacktrace
> 
> 
>> Hi Erik,
>> 
>> While setting up the cascading replication I have hit one issue on
>> base code[1].  After fixing that I have got one crash with streaming
>> on patch.  I am not sure whether you are facing any of these 2 issues
>> or any other issue.  If your issue is not any of these then plese
>> share the callstack and steps to reproduce.

I figured out a few things about this. Attached is a bash script 
test.sh, to reproduce:

There is a variable  CRASH_IT  that determines whether the whole thing 
will fail (with a segmentation fault) or not.  As attached it has  
CRASH_IT=0 and does not crash.  When you change that to CRASH_IT=1, then 
it will crash.  It turns out that this just depends on a short wait 
state (3 seconds, on my machine) between setting up de replication, and 
the running of pgbench.  It's possible that on very fast machines maybe 
it does not occur; we've had such difference between hardware before. 
This is a i5-3330S.

It deletes files so look it over before you run it.  It may also depend 
on some of my local set-up but I guess that should be easily fixed.

Can you let me know if you can reproduce the problem with this?

thanks,

Erik Rijkers



>> 
>> [1]
>> https://www.postgresql.org/message-id/CAFiTN-u64S5bUiPL1q5kwpHNd0hRnf1OE-bzxNiOs5zo84i51w%40mail.gma...
>> 
>> 
>> --
>> Regards,
>> Dilip Kumar
>> EnterpriseDB: http://www.enterprisedb.com

view thread (536+ messages)  latest in thread

Message-ID: <bcc76806992e913a6862e64d633a166a@xs4all.nl>
Permalink:  ../bcc76806992e913a6862e64d633a166a@xs4all.nl/
Also on:    postgresql.org/message-id/bcc76806992e913a6862e64d633a166a@xs4all.nl

 ·

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-hackers@postgresql.org
  Cc: er@xs4all.nl, dilipbalaut@gmail.com, kuntalghosh.2007@gmail.com, tomas.vondra@2ndquadrant.com, amit.kapila16@gmail.com, michael@paquier.xyz, peter.eisentraut@2ndquadrant.com
  Subject: Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
  In-Reply-To: <bcc76806992e913a6862e64d633a166a@xs4all.nl>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox