MIME-Version: 1.0
References: 
 <CAA5-nLARN7-3SLU_QUxfy510pmrYK6JJb=bk3hcgemAM_pAv+w@mail.gmail.com>
 <CAA4eK1JLerufRV4Zr4h-vW4Fjz6LJjd-QZxVywpFrtfvZhXVWg@mail.gmail.com>
 <CAA5-nLC0NKo1VsKgf+jEEH=5jGAN2pyCivyG59yX3bQcLSL4tg@mail.gmail.com>
 <CAA4eK1KHQfnhmdnLZQrRoAW3dYOPqXKtT5C54HoBno3MCtH4wA@mail.gmail.com>
 <CAA5-nLCH-XbCknSx2nVMtCByj8pv3jyRFQxL0qj4FLwby4TyNg@mail.gmail.com>
 <CAA4eK1KrJP=oMOu5VJKn1uZjn+Xq110AGUxJpBrp3jN7vWXVPg@mail.gmail.com>
 <CAA5-nLA8C0LZdVDdPDNeWcJYGzGVV3nJ7Ni0PdSyHJNxi628ww@mail.gmail.com>
 <CAA4eK1LeN1V85i2ZfU2cOj5vJjpEVSvJ6LOfAu-u7mfQrW=v1Q@mail.gmail.com>
 <CAA5-nLDhnKRwsPF8z5+ej4D9P9WTfVuMeVsigxuO=+sjf0ruyQ@mail.gmail.com>
In-Reply-To: 
 <CAA5-nLDhnKRwsPF8z5+ej4D9P9WTfVuMeVsigxuO=+sjf0ruyQ@mail.gmail.com>
From: Amit Kapila <amit.kapila16@gmail.com>
Date: Tue, 21 Sep 2021 15:22:29 +0530
Message-ID: 
 <CAA4eK1JYB2eLNTs48szQ8oJXW9GLcrxT0yWETQS0gF2sVBxMYA@mail.gmail.com>
Subject: Re: Logical replication timeout problem
To: Fabrice Chapuis <fabrice636861@gmail.com>
Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Content-Type: text/plain; charset="UTF-8"
Archived-At: 
 <https://www.postgresql.org/message-id/CAA4eK1JYB2eLNTs48szQ8oJXW9GLcrxT0yWETQS0gF2sVBxMYA%40mail.gmail.com>
Precedence: bulk

On Tue, Sep 21, 2021 at 1:52 PM Fabrice Chapuis <fabrice636861@gmail.com> wrote:
>
> If I understand, the instruction to send keep alive by the wal sender has not been reached in the for loop, for what reason?
> ...
> * Check for replication timeout. */
>   WalSndCheckTimeOut();
>
> /* Send keepalive if the time has come */
>   WalSndKeepaliveIfNecessary();
> ...
>

Are you sure that these functions have not been called? Or the case is
that these are called but due to some reason the keep-alive is not
sent? IIUC, these are called after processing each WAL record so not
sure how is it possible in your case that these are not reached?

> The data load is performed on a table which is not replicated, I do not understand why the whole transaction linked to an insert is copied to snap files given that table does not take part of the logical replication.
>

It is because we don't know till the end of the transaction (where we
start sending the data) whether the table will be replicated or not. I
think specifically for this purpose the new 'streaming' feature
introduced in PG-14 will help us to avoid writing data of such tables
to snap/spill files. See 'streaming' option in Create Subscription
docs [1].

> We are going to do a test by modifying parameters wal_sender_timeout/wal_receiver_timeout from 1' to 5'. The problem is that these parameters are global and changing them will also impact the physical replication.
>

Do you mean you are planning to change from 1 minute to 5 minutes? I
agree with the global nature of parameters and I think your approach
to finding out the root cause is good here because otherwise, under
some similar or more heavy workload, it might lead to the same
situation.

> Concerning the walsender timeout, when the worker is started again after a timeout, it will trigger a new walsender associated with it.
>

Right, I know that but I was curious to know if the walsender has
exited before walreceiver.

[1] - https://www.postgresql.org/docs/devel/sql-createsubscription.html

-- 
With Regards,
Amit Kapila.