pg.ddx.io  pgsql-bugs@postgresql.org mailing list archive  

help / color / mirror / Atom feed
From: Gilles Darold <gilles@darold.net>
To: Alex Richman <alexrichman@onesignal.com>
To: Amit Kapila <amit.kapila16@gmail.com>
Cc: wangw.fnst@fujitsu.com <wangw.fnst@fujitsu.com>
Cc: pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Cc: Niels Stevens <niels.stevens@onesignal.com>
Subject: Re: Logical Replica ReorderBuffer Size Accounting Issues
Date: Mon, 27 Feb 2023 22:34:59 +0000
Message-ID: <83b1108f-9552-3ebb-35fc-e00ac9ff0cf1@darold.net> (raw)
In-Reply-To: <CAMnUB3o_=q4FYZ5yqCgy=TQUk=ce49RNr7CJD4EGP_=KuYrzFA@mail.gmail.com>
References: <CAMnUB3oYugXCBLSkih+qNsWQPciEwos6g_AMbnz_peNoxfHwyw@mail.gmail.com>
	<CAA4eK1JjxNFGkDHLSSecWWD3nP+1KE4M=4G-AX2D2S+K_=m09w@mail.gmail.com>
	<CAMnUB3qVvWazGHxtC1emxEaTnU28N+CznabVRMbqfNQq9kf9fw@mail.gmail.com>
	<OS3PR01MB6275677D0DAEC1132FD0AC7F9EFE9@OS3PR01MB6275.jpnprd01.prod.outlook.com>
	<CAMnUB3oSCM0BNK=o0_3PwHc1oJepgex_=mNAD5oY7rEjR8958w@mail.gmail.com>
	<OS3PR01MB627580022656E8A41B31B63A9EFF9@OS3PR01MB6275.jpnprd01.prod.outlook.com>
	<CAMnUB3pGWcUL08fWB4QmO0+2yNBBckXq=ndyLoGAU+V_2WQaCg@mail.gmail.com>
	<OS3PR01MB6275A699D18DEF195D8A81729EFD9@OS3PR01MB6275.jpnprd01.prod.outlook.com>
	<CAMnUB3pwknqoe5s-bGuRD8nX1bWkZRbFF=jWNLTWbm_etFigkA@mail.gmail.com>
	<OS3PR01MB6275A7E5323601D59D18DB979EC29@OS3PR01MB6275.jpnprd01.prod.outlook.com>
	<CAA4eK1KdwDrLW2tE-jawqEADy8w=6nHtEwMWeno12WnZ+xEByQ@mail.gmail.com>
	<CAMnUB3pmEZnky2Z+e5tfOCnLcLr_ti4xXQ4Kpm=SkaSziC9Z9g@mail.gmail.com>
	<CAMnUB3o_=q4FYZ5yqCgy=TQUk=ce49RNr7CJD4EGP_=KuYrzFA@mail.gmail.com>

Le 16/02/2023 à 19:08, Alex Richman a écrit :
> Hi all,
>
> Looping back to say we updated to 15.2 and are still seeing this 
> issue, though it is less prevalent.
>
> Thanks,
> - Alex.
>
> On Wed, 18 Jan 2023 at 11:16, Alex Richman <alexrichman@onesignal.com> 
> wrote:
>
>
>
>     On Wed, 18 Jan 2023 at 10:10, Amit Kapila
>     <amit.kapila16@gmail.com> wrote:
>
>         Alex,
>         Do we see this problem with small tuples as well? I see from your
>         earlier email that tuple size is ~800 bytes in the production
>         environment. It is possible that after commit 1b0d9aa4 such
>         kind of
>         problems are not there with small tuple sizes but that commit
>         happened
>         in PG15 whereas your production environment might be on a prior
>         release.
>
>
>     Hi Amit,
>
>     Our prod environment is also on 15.1, which is where we first saw
>     the issue, so I'm afraid the issue still seems to be present here.
>
>     Looping back on the earlier discussion, we applied the malloc
>     patch from [1] ([2]) to a prod server, which also fixes the issue
>     there.  Attached is a graph of the last 48 hours of memory usage,
>     the ~200GB spikes are instances of the LR walsender memory issue.
>     After patch is applied (blue mark), baseline memory drops and we
>     no longer see the spikes.  Per-process memory stats corroborate
>     that the LR walsender memory is now never more than a few MB RSS
>     per process.
>
>     Thanks,
>     - Alex.
>
>     [1]
>     https://www.postgresql.org/message-id/CAMnUB3pwknqoe5s-bGuRD8nX1bWkZRbFF%3DjWNLTWbm_etFigkA%40mail.g...
>     [2]
>     https://gist.github.com/alex-richman-onesignal/4ad147b37eaab99f41a150b51899a564
>

Hi,


Amit / Wang Wei do you have a better patch to fix this bug or should we 
use the one provided by Alex? I have tried some other fixes but it's a 
fail, you may have better understanding of the problem and the way to 
fix it.


Best regards

-- 
Gilles Darold

view thread (24+ messages)  latest in thread

Message-ID: <83b1108f-9552-3ebb-35fc-e00ac9ff0cf1@darold.net>
Permalink:  ../83b1108f-9552-3ebb-35fc-e00ac9ff0cf1@darold.net/
Also on:    postgresql.org/message-id/83b1108f-9552-3ebb-35fc-e00ac9ff0cf1@darold.net

 · 

reply
Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-bugs@postgresql.org
  Cc: gilles@darold.net, alexrichman@onesignal.com, amit.kapila16@gmail.com, wangw.fnst@fujitsu.com, pgsql-bugs@lists.postgresql.org, niels.stevens@onesignal.com
  Subject: Re: Logical Replica ReorderBuffer Size Accounting Issues
  In-Reply-To: <83b1108f-9552-3ebb-35fc-e00ac9ff0cf1@darold.net>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox