Content-Type: multipart/alternative;
 boundary="------------POGO54IrrAePrDbhyu5N5teB"
Message-ID: <83b1108f-9552-3ebb-35fc-e00ac9ff0cf1@darold.net>
Date: Mon, 27 Feb 2023 22:34:59 +0000
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.7.1
To: Alex Richman <alexrichman@onesignal.com>,
 Amit Kapila <amit.kapila16@gmail.com>
Cc: "wangw.fnst@fujitsu.com" <wangw.fnst@fujitsu.com>,
 "pgsql-bugs@lists.postgresql.org" <pgsql-bugs@lists.postgresql.org>,
 Niels Stevens <niels.stevens@onesignal.com>
References: 
 <CAMnUB3oYugXCBLSkih+qNsWQPciEwos6g_AMbnz_peNoxfHwyw@mail.gmail.com>
 <CAA4eK1JjxNFGkDHLSSecWWD3nP+1KE4M=4G-AX2D2S+K_=m09w@mail.gmail.com>
 <CAMnUB3qVvWazGHxtC1emxEaTnU28N+CznabVRMbqfNQq9kf9fw@mail.gmail.com>
 <OS3PR01MB6275677D0DAEC1132FD0AC7F9EFE9@OS3PR01MB6275.jpnprd01.prod.outlook.com>
 <CAMnUB3oSCM0BNK=o0_3PwHc1oJepgex_=mNAD5oY7rEjR8958w@mail.gmail.com>
 <OS3PR01MB627580022656E8A41B31B63A9EFF9@OS3PR01MB6275.jpnprd01.prod.outlook.com>
 <CAMnUB3pGWcUL08fWB4QmO0+2yNBBckXq=ndyLoGAU+V_2WQaCg@mail.gmail.com>
 <OS3PR01MB6275A699D18DEF195D8A81729EFD9@OS3PR01MB6275.jpnprd01.prod.outlook.com>
 <CAMnUB3pwknqoe5s-bGuRD8nX1bWkZRbFF=jWNLTWbm_etFigkA@mail.gmail.com>
 <OS3PR01MB6275A7E5323601D59D18DB979EC29@OS3PR01MB6275.jpnprd01.prod.outlook.com>
 <CAA4eK1KdwDrLW2tE-jawqEADy8w=6nHtEwMWeno12WnZ+xEByQ@mail.gmail.com>
 <CAMnUB3pmEZnky2Z+e5tfOCnLcLr_ti4xXQ4Kpm=SkaSziC9Z9g@mail.gmail.com>
 <CAMnUB3o_=q4FYZ5yqCgy=TQUk=ce49RNr7CJD4EGP_=KuYrzFA@mail.gmail.com>
Content-Language: fr
From: Gilles Darold <gilles@darold.net>
Subject: Re: Logical Replica ReorderBuffer Size Accounting Issues
In-Reply-To: 
 <CAMnUB3o_=q4FYZ5yqCgy=TQUk=ce49RNr7CJD4EGP_=KuYrzFA@mail.gmail.com>
Archived-At: 
 <https://www.postgresql.org/message-id/83b1108f-9552-3ebb-35fc-e00ac9ff0cf1%40darold.net>
Precedence: bulk

This is a multi-part message in MIME format.
--------------POGO54IrrAePrDbhyu5N5teB
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

Le 16/02/2023 à 19:08, Alex Richman a écrit :
> Hi all,
>
> Looping back to say we updated to 15.2 and are still seeing this 
> issue, though it is less prevalent.
>
> Thanks,
> - Alex.
>
> On Wed, 18 Jan 2023 at 11:16, Alex Richman <alexrichman@onesignal.com> 
> wrote:
>
>
>
>     On Wed, 18 Jan 2023 at 10:10, Amit Kapila
>     <amit.kapila16@gmail.com> wrote:
>
>         Alex,
>         Do we see this problem with small tuples as well? I see from your
>         earlier email that tuple size is ~800 bytes in the production
>         environment. It is possible that after commit 1b0d9aa4 such
>         kind of
>         problems are not there with small tuple sizes but that commit
>         happened
>         in PG15 whereas your production environment might be on a prior
>         release.
>
>
>     Hi Amit,
>
>     Our prod environment is also on 15.1, which is where we first saw
>     the issue, so I'm afraid the issue still seems to be present here.
>
>     Looping back on the earlier discussion, we applied the malloc
>     patch from [1] ([2]) to a prod server, which also fixes the issue
>     there.  Attached is a graph of the last 48 hours of memory usage,
>     the ~200GB spikes are instances of the LR walsender memory issue.
>     After patch is applied (blue mark), baseline memory drops and we
>     no longer see the spikes.  Per-process memory stats corroborate
>     that the LR walsender memory is now never more than a few MB RSS
>     per process.
>
>     Thanks,
>     - Alex.
>
>     [1]
>     https://www.postgresql.org/message-id/CAMnUB3pwknqoe5s-bGuRD8nX1bWkZRbFF%3DjWNLTWbm_etFigkA%40mail.gmail.com
>     [2]
>     https://gist.github.com/alex-richman-onesignal/4ad147b37eaab99f41a150b51899a564
>

Hi,


Amit / Wang Wei do you have a better patch to fix this bug or should we 
use the one provided by Alex? I have tried some other fixes but it's a 
fail, you may have better understanding of the problem and the way to 
fix it.


Best regards

-- 
Gilles Darold

--------------POGO54IrrAePrDbhyu5N5teB
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit

<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body>
    <div class="moz-cite-prefix">Le 16/02/2023 à 19:08, Alex Richman a
      écrit :<br>
    </div>
    <blockquote type="cite"
cite="mid:CAMnUB3o_=q4FYZ5yqCgy=TQUk=ce49RNr7CJD4EGP_=KuYrzFA@mail.gmail.com">
      <meta http-equiv="content-type" content="text/html; charset=UTF-8">
      <div dir="ltr">Hi all,
        <div><br>
        </div>
        <div>Looping back to say we updated to 15.2 and are still seeing
          this issue, though it is less prevalent.</div>
        <div><br>
        </div>
        <div>Thanks,</div>
        <div>- Alex.</div>
      </div>
      <br>
      <div class="gmail_quote">
        <div dir="ltr" class="gmail_attr">On Wed, 18 Jan 2023 at 11:16,
          Alex Richman &lt;<a href="mailto:alexrichman@onesignal.com"
            moz-do-not-send="true" class="moz-txt-link-freetext">alexrichman@onesignal.com</a>&gt;
          wrote:<br>
        </div>
        <blockquote class="gmail_quote" style="margin:0px 0px 0px
          0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
          <div dir="ltr">
            <div dir="ltr"><br>
            </div>
            <br>
            <div class="gmail_quote">
              <div dir="ltr" class="gmail_attr">On Wed, 18 Jan 2023 at
                10:10, Amit Kapila &lt;<a
                  href="mailto:amit.kapila16@gmail.com" target="_blank"
                  moz-do-not-send="true" class="moz-txt-link-freetext">amit.kapila16@gmail.com</a>&gt;
                wrote:<br>
              </div>
              <blockquote class="gmail_quote" style="margin:0px 0px 0px
                0.8ex;border-left:1px solid
                rgb(204,204,204);padding-left:1ex">Alex,<br>
                Do we see this problem with small tuples as well? I see
                from your<br>
                earlier email that tuple size is ~800 bytes in the
                production<br>
                environment. It is possible that after commit 1b0d9aa4
                such kind of<br>
                problems are not there with small tuple sizes but that
                commit happened<br>
                in PG15 whereas your production environment might be on
                a prior<br>
                release.<br>
              </blockquote>
              <div><br>
              </div>
              <div>Hi Amit,</div>
              <div><br>
              </div>
              <div>Our prod environment is also on 15.1, which is where
                we first saw the issue, so I'm afraid the issue still
                seems to be present here.</div>
              <div><br>
              </div>
              <div>Looping back on the earlier discussion, we applied
                the malloc patch from [1] ([2]) to a prod server, which
                also fixes the issue there.  Attached is a graph of the
                last 48 hours of memory usage, the ~200GB spikes are
                instances of the LR walsender memory issue.</div>
              <div>After patch is applied (blue mark), baseline memory
                drops and we no longer see the spikes.  Per-process
                memory stats corroborate that the LR walsender memory is
                now never more than a few MB RSS per process.</div>
              <div><br>
              </div>
              <div>Thanks,</div>
              <div>- Alex.</div>
              <div><br>
              </div>
              <div>[1] <a
href="https://www.postgresql.org/message-id/CAMnUB3pwknqoe5s-bGuRD8nX1bWkZRbFF%3DjWNLTWbm_etFigkA%40mail.gmail.com"
                  target="_blank" moz-do-not-send="true"
                  class="moz-txt-link-freetext">https://www.postgresql.org/message-id/CAMnUB3pwknqoe5s-bGuRD8nX1bWkZRbFF%3DjWNLTWbm_etFigkA%40mail.gmail.com</a></div>
              <div>[2] <a
href="https://gist.github.com/alex-richman-onesignal/4ad147b37eaab99f41a150b51899a564"
                  target="_blank" moz-do-not-send="true"
                  class="moz-txt-link-freetext">https://gist.github.com/alex-richman-onesignal/4ad147b37eaab99f41a150b51899a564</a></div>
            </div>
          </div>
        </blockquote>
      </div>
    </blockquote>
    <p><br>
    </p>
    <p>Hi,</p>
    <p><br>
    </p>
    <p>Amit / Wang Wei do you have a better patch to fix this bug or
      should we use the one provided by Alex? I have tried some other
      fixes but it's a fail, you may have better understanding of the
      problem and the way to fix it.</p>
    <p><br>
    </p>
    <p>Best regards<br>
    </p>
    <pre class="moz-signature" cols="72">-- 
Gilles Darold
</pre>
  </body>
</html>

--------------POGO54IrrAePrDbhyu5N5teB--