From: "wangw.fnst@fujitsu.com" <wangw.fnst@fujitsu.com>
To: Euler Taveira <euler@eulerto.com>, Amit Kapila <amit.kapila16@gmail.com>
CC: Masahiko Sawada <sawada.mshk@gmail.com>, "kuroda.hayato@fujitsu.com"
	<kuroda.hayato@fujitsu.com>, Peter Smith <smithpb2250@gmail.com>, Fabrice
 Chapuis <fabrice636861@gmail.com>, Simon Riggs
	<simon.riggs@enterprisedb.com>, Petr Jelinek <petr.jelinek@enterprisedb.com>,
	"tanghy.fnst@fujitsu.com" <tanghy.fnst@fujitsu.com>, PostgreSQL Hackers
	<pgsql-hackers@lists.postgresql.org>, Ajin Cherian <itsajin@gmail.com>
Subject: RE: Logical replication timeout problem
Thread-Topic: Logical replication timeout problem
Thread-Index:
 AQHXq6q0fGO4Dh5+k0GIgFVk++toyquoBbGAgABGZICAAnlLAIAB+ycAgAATswCAAElQAIAA8bIAgAAdFACAABkigIAAYZsAgAEivICAAHkiAIABaZWAgAAlYwCACggKAIAM/F4AgDYaB4CAANP8AIAAsSuAgD7B+oCAAUduAIAJeV2AgA4SawCABpEzgIABUlCAgAGGygCAAD8kAIABVFSAgAAdkoCAB/p7AIABQC/ggABIXACAAVeYMIAAOpQAgABCZQCAASy+AIAFyaWQgBRitNCAD7afAIAGWNIQgAHoyACAB8R6cIAAOXWAgAKPYVCAAgvSAIABg4EAgAXSvjCAAG1LgIABNYgAgABJ9YCACr/LsIAAvEGAgAEaMACAADb0gIAAGyqAgAEizbCAAF5QgIAEXb0AgAFV6cCAA7VegIABO8mggAAP0wCAAEKDEIAEKtoAgABG3JCAAUgqsIAB+P7QgAATMACAAcuPAIAA5KGAgAAG6ACAAAhZgIAAE+YAgAfzQhCAAAcGgIAAKseAgABKWDCAARZyAIAGWo9ggANqW4CAAa0sgIAF49+w
Date: Mon, 18 Apr 2022 06:19:15 +0000
Message-ID:
 <OS3PR01MB62754D7C91CE80B3A68FFC1A9EF39@OS3PR01MB6275.jpnprd01.prod.outlook.com>
References:
 <CAA4eK1+fQjndoBOFUn9Wy0hhm3MLyUWEpcT9O7iuCELktfdBiQ@mail.gmail.com>
 <CAA4eK1LGnaPuWs2M4sDfpd6JQZjoh4DGAsgUvNW=Or8i9z6K8w@mail.gmail.com>
 <OS3PR01MB62756599B78FA7D4C908109A9E179@OS3PR01MB6275.jpnprd01.prod.outlook.com>
 <CAA4eK1JKZrOfCLMVZYfUnm_wfxucDjOzeGwQy3hJ_U9Y3AV05Q@mail.gmail.com>
 <OS3PR01MB627542044A4C16511E63D5DA9E1A9@OS3PR01MB6275.jpnprd01.prod.outlook.com>
 <CAD21AoCEhZ6D1VtqvxK1vO0=NnJd6phWbijt-w=M5hwka=-c1A@mail.gmail.com>
 <OS3PR01MB62750C20AD7AF4F13541FC109E1A9@OS3PR01MB6275.jpnprd01.prod.outlook.com>
 <TYAPR01MB586636517DED745651D6E75AF51D9@TYAPR01MB5866.jpnprd01.prod.outlook.com>
 <OS3PR01MB6275C64F264662E84D2FB7AE9E1D9@OS3PR01MB6275.jpnprd01.prod.outlook.com>
 <OS3PR01MB6275F0E4E27F5522847589ED9E1E9@OS3PR01MB6275.jpnprd01.prod.outlook.com>
 <OS3PR01MB6275E0C2B4D9E488AD7CBA209E1F9@OS3PR01MB6275.jpnprd01.prod.outlook.com>
 <CAA4eK1JUgO9F5fZb5P8quMYfDbD7SBNApsTsGLf6vYNNetpWTA@mail.gmail.com>
 <CAD21AoCQi33mjd=bM0Jhkk3QaV-fHVB_soA7iQmTor2A-1Cx3Q@mail.gmail.com>
 <0271e197-2cfe-4627-9c11-47535cc7fdf3@www.fastmail.com>
 <CAA4eK1+wSvfDVUSe-4o6APFdO8=9G-HnZJ+01_LQOxqLmhqe5Q@mail.gmail.com>
 <8cc4f31f-240a-4590-94a8-a50bdf4f4d1c@www.fastmail.com>
 <CAA4eK1+HNmgtWRm2U7EG2T0yWbAvhqSDDFBmKX5+vSgBsRmY=w@mail.gmail.com>
 <OS3PR01MB6275FD2CE0850FC66512D8F99EE79@OS3PR01MB6275.jpnprd01.prod.outlook.com>
 <CAA4eK1+A2aesu8YS9Y8bFBCbOUw_UkU1Nj_-V6rfXBiJgCxqKg@mail.gmail.com>
 <CAA4eK1L-MU+YfRUD0J1wBu6CJ2FEaSVh=Kr3PjjyshprUpvkeg@mail.gmail.com>
 <OS3PR01MB627516D7A3A2ED91EEAA510C9EE79@OS3PR01MB6275.jpnprd01.prod.outlook.com>
 <CAA4eK1L8jDBGqdZU0Fk6LDSpdqE+3wg2AwkgdxwiCqmXHc6aoQ@mail.gmail.com>
 <OS3PR01MB62755D216245199554DDC8DB9EEA9@OS3PR01MB6275.jpnprd01.prod.outlook.com>
 <CAA4eK1Kj=-5opwBstZtu2CBDMtoQPmDD4c=X5arqx54wq52FNw@mail.gmail.com>
 <ef68bb18-37f9-4303-998e-a15d2fdb2563@www.fastmail.com>
In-Reply-To: <ef68bb18-37f9-4303-998e-a15d2fdb2563@www.fastmail.com>
Accept-Language: ja-JP, en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0:
 =?iso-2022-jp?B?WnZxSzVhUHJRVTZyekk1SXFaa0R4STlSd3F4NkYxeE1hNkpreDY3V2dU?=
 =?iso-2022-jp?B?U2MxSVI2R1pUZitoSVFHVXlJNjhhU0hJK3BoTW1ka1NMbEc5WXlxa0M0?=
 =?iso-2022-jp?B?eWg2MHRmVzEzVm9lcGhrTUdERFc1NUQvWWR5RDM2am4xbUQ5QUlNNlVV?=
 =?iso-2022-jp?B?MjY0cnVBNGZqcFlWZ3g0bTcwdm5FN3g0ZHk1bkdkREZGcWp1eTdxcGtj?=
 =?iso-2022-jp?B?UjNXdWkxamI2Z3BtZ01UMnJFQzRHK2dmd3E4UGlFMHl1WFh2cGVLTVZH?=
 =?iso-2022-jp?B?aVkrd2pKdGxGaE96NUxnVEg4U2lmMU94ODNUdTk1YVBseWkyZGs2YUxE?=
 =?iso-2022-jp?B?cS9qR2U4dFFLM3hOTWdzS2NzeGRZa202eUVCQm12c1Z0S0VMbU1hVjBq?=
 =?iso-2022-jp?B?L0ZmbmM1eXArY2Y2VTUxNHNuZmt1Q2ZnaTIxbnRGd2ZFbldxUUJlYVR5?=
 =?iso-2022-jp?B?UTl0S0hCbytzeW8rVUVkbWdPQmV5WSs1c1g3TGY2QUVmK3JXYUVEUlF3?=
 =?iso-2022-jp?B?Q0VuOHNGYnFGZDZXVnlncWduMGtmT2FVZlFyM2taOTBIb0lJVXpGaVBQ?=
 =?iso-2022-jp?B?czZjTmUraWs0ZmsvL2JyZk1MRFg5eTZUNG1wUzNKc2FacURPeGd4enAv?=
 =?iso-2022-jp?B?enYrbklJbGpkNzlSSnJYVS9LQWU3TEtGUDFtbWxac1V0RFZwblZMajN4?=
 =?iso-2022-jp?B?SUpzVUpxQ2t2RUlnRDgwczY4L25pZjdmSFhPSWM0MXdONVU3NmhOSDQ1?=
 =?iso-2022-jp?B?Vk1rSUZjTFBkcERiV05KZ1hhUU1GWEh5NjloUkQ0d0NzNzk1dU1HTjdJ?=
 =?iso-2022-jp?B?bGQwMmttYXI2QjI2Qm9SNjFOOHVyQXJMSXpYT0oreW83ODFianN6VnJV?=
 =?iso-2022-jp?B?c2JtNGlRb0NKQ0c5Q2ZVZ0J0dlZudkd0UnFjMnNmSUc4M1U5cjVmQmU5?=
 =?iso-2022-jp?B?RytEQ3Q0QmNvTTMzRWI4QnJvbUdWU3pvN01VMk81NU4rQWxxRU5HclJa?=
 =?iso-2022-jp?B?d2h3bGU2ajRTZ2R1ZjdVME1BeEo2YTB4K0cxZjJ3WEFQTnIyY2ZGWTV2?=
 =?iso-2022-jp?B?VWlHaW1wTkpmSFJ5MXlRSkU0RlN2bnpObUF2dm5ZNlFMdUZuMCtQYjlz?=
 =?iso-2022-jp?B?SGY5YnBRSDBzS25hazIyVEVhZHZUUURNUzlMNWVwNE5DNHUzM2JJajdl?=
 =?iso-2022-jp?B?UEZiRTQyY3pQcGJXdlhWUktGSVpMZlB0akRJZXRXSm1ZcFp5WlhiWW5L?=
 =?iso-2022-jp?B?ZWhoOCtvaUNXci81T2ZVenppNHRpajZTTUpKV3pGUXN0OFJnckhBMjBy?=
 =?iso-2022-jp?B?d0NGWHNzZHBXWVpGYjl0VEYrNVNIZmN3OTY0Z1FuVXhRd0NBZnoyTEFs?=
 =?iso-2022-jp?B?RE1QbTQwUGkzbWlBUk45U0ozT0NxTVcxaUs0UG9pMVJaRG9NeDNHQjBP?=
 =?iso-2022-jp?B?ei9qemhEUEhGdUUyaC9WSmprWVpqSFVHMFE5N2NMVTRLdnEvOHdYMXRC?=
 =?iso-2022-jp?B?QWxhKzNEb3pHQ2dMd1ZZL2o4OFlaWkYxSTRSZ25YTkxvSmJncGpOOVVy?=
 =?iso-2022-jp?B?QkRpNmdqZzlmSC9yc1JPNGFFbmczRzdQU0t1ZnBjVzAwVitxaloyeXhl?=
 =?iso-2022-jp?B?RWlRTGYrNFJpUFhzNUJsMWljZEVnMHplQWxsZ0xLa1NvWXQxeWFRVURW?=
 =?iso-2022-jp?B?Vk0vWE9rU1lxV1NjK0p2YmFzQ1dLVnJRWHg3Z1F2NThqQVBhdzRsY2Vo?=
 =?iso-2022-jp?B?d1FSd3c4aEpxaFE2cStVdnY4a0lDUlpTbXdIdUJaYnIyMTgvZFBRSlhr?=
 =?iso-2022-jp?B?T3NxRFphM1BqQTh6S3IxSTluVDBUY2NxVTRmd1ZBZ3JTV0R3Rzkwa3px?=
 =?iso-2022-jp?B?aGtHM1h4YUpaZ3kxQkMwNVJHTElMaUQ5TC9Ka0NPSTFnUWtsYmFMNW5i?=
 =?iso-2022-jp?B?bEpQeENWaE12ZTV4d2I0QWVKbGQxYy9GR2o3REwwc2diVHREblJRUUpM?=
 =?iso-2022-jp?B?aWdzNStjc3NOaGlESkFpNytpZlBJTldWUS95TVNLOWNZQzBNZFNjTHZT?=
 =?iso-2022-jp?B?VVMxV2s5UVNmd2piUzV6YnY0b05Ca3JWdDlFWjl6S0pGbFdObGN2VHht?=
 =?iso-2022-jp?B?TXpqcXB2SUN3dVI3Mm1PKzJTRGRGM0t6Ym83ekw3ak9GUS8wMnVGdjZP?=
 =?iso-2022-jp?B?dmN1VG9GcnBGRS9nbCtZbHNqOTQwbzlrUmFBeG8wQmlLamZmMmdIVWgy?=
 =?iso-2022-jp?B?WlE5aktXL0QxQkczdWFhVThWbmVqazd4SVBNejg1KzJDVE5XVVhuaElp?=
 =?iso-2022-jp?B?OXdWUjZJenFKTFVHc3dnWm5QL0VOMkxwdWdUUzhpTCszdEw0RXJQcDJF?=
 =?iso-2022-jp?B?c1BWMlFtMWxoRlhYcFZxV3lVQlBqZFNSZjVUOERCd3UrbDJqTmRRS2Uv?=
 =?iso-2022-jp?B?ZG9KekxwUEhlL2k4NXpVWWhFNktUSENDU2xUeU1nRDFqd2JRZlZYekd6?=
 =?iso-2022-jp?B?TmJYYzgxMnh0VWJMZmgzcS9leWtwVlFDNFFWdz09?=
Content-Type: text/plain; charset="iso-2022-jp"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: fujitsu.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: OS3PR01MB6275.jpnprd01.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 6d564575-4c08-4819-9a04-08da21035d4a
X-MS-Exchange-CrossTenant-originalarrivaltime: 18 Apr 2022 06:19:15.7479
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: a19f121d-81e1-4858-a9d8-736e267fd4c7
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: eY/H1NGRxLab8TdkFK4kNvrIZnqW2uKjFwb97/fkqIEPqNcZEY6TW7XypNne8vbXdoisp41MgQRt1TCapVqveg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: TYCPR01MB5757
List-Id: <pgsql-hackers.lists.postgresql.org>
List-Help: <https://lists.postgresql.org/manage/>
List-Subscribe: <https://lists.postgresql.org/manage/>
List-Post: <mailto:pgsql-hackers@lists.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@lists.postgresql.org>
List-Archive: <https://www.postgresql.org/list/pgsql-hackers>
Archived-At: <https://www.postgresql.org/message-id/OS3PR01MB62754D7C91CE80B3A68FFC1A9EF39%40OS3PR01MB6275.jpnprd01.prod.outlook.com>
Precedence: bulk

On Thur, Apr 14, 2022 at 8:21 PM Euler Taveira <euler@eulerto.com> wrote:
>
Thanks for your comments.

> + * For a large transaction, if we don't send any change to the downstrea=
m for a
> + * long time then it can timeout. This can happen when all or most of th=
e
> + * changes are either not published or got filtered out.
>=20
> We should probable mention that "long time" is wal_receiver_timeout on
> subscriber.
Improve as suggested.
Add "(exceeds the wal_receiver_timeout of standby)" to explain what "long t=
ime"
means.

> +    * change as that can have overhead. Testing reveals that there is no
> +    * noticeable overhead in doing it after continuously processing 100 =
or so
> +    * changes.
>=20
> Tests revealed that ...
Improve as suggested.

> +    * We don't have a mechanism to get the ack for any LSN other than en=
d xact
> +    * lsn from the downstream. So, we track lag only for end xact lsn's.
>=20
> s/lsn/LSN/ and s/lsn's/LSNs/
>=20
> I would say "end of transaction LSN".
Improve as suggested.

> + * If too many changes are processed then try to send a keepalive messag=
e to
> + * receiver to avoid timeouts.
>=20
> In logical replication, if too many changes are processed then try to sen=
d a
> keepalive message. It might avoid a timeout in the subscriber.
Improve as suggested.

Kindly have a look at new patch shared in [1].

[1] - https://www.postgresql.org/message-id/OS3PR01MB627561344A2C7ECF68E41D=
7E9EF39%40OS3PR01MB6275.jpnprd01.prod.outlook.com

Regards,
Wang wei