public inbox for [email protected]  

help / color / mirror / Atom feed

From: RECHTÉ Marc <[email protected]>
To: Hayato Kuroda (Fujitsu) <[email protected]>
Cc: [email protected]
Subject: Re: Logical replication timeout
Date: Mon, 23 Dec 2024 09:31:09 +0000 (GMT)
Message-ID: <[email protected]> (raw)
In-Reply-To: <OSCPR01MB1496698CA14BD0DE49261819AF5022@OSCPR01MB14966.jpnprd01.prod.outlook.com>
References: <[email protected]>
	<OSCPR01MB1496698CA14BD0DE49261819AF5022@OSCPR01MB14966.jpnprd01.prod.outlook.com>

> Can you enable the parameter "streaming" to on on your system [1]? It allows to
> stream the in-progress transactions to the subscriber side. I feel this can avoid
> the case that there are many .spill files on the publisher side.

> Another approach is to tune the logical_decoding_work_mem parameter [2].
> This specifies the maximum amount of memory used by the logical decoding, and
> some changes are spilled when it exceeds the limitation. Naively, this setting
> can reduce the number of files.


> [1]: https://www.postgresql.org/docs/14/sql-createsubscription.html
> [2]: https://www.postgresql.org/docs/14/runtime-config-resource.html#GUC-LOGICAL-DECODING-WORK-MEM

> Best regards,
> Hayato Kuroda
> FUJITSU LIMITED

Dear Hayato,

Thanks for your suggestions that were both already tested. In our (real) case (a single transaction with 12 millions sub-transactions):

1) setting the subscription as streaming, just delay a bit the spill file surge. It does not prevent the creation of spill files.

2) we set logical_decoding_work_mem to 20GB, which probably also delayed the problem, but did not solve it.

The real problem is spill file deletions that can take days in this particular case !

Marc

view thread (3+ messages)  latest in thread

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected], [email protected], [email protected]
  Subject: Re: Logical replication timeout
  In-Reply-To: <[email protected]>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox