From: Tom Lane <tgl@sss.pgh.pa.us>
To: Amit Kapila <amit.kapila16@gmail.com>
cc: Noah Misch <noah@leadboat.com>, Amit Khandekar <amitdkhan.pg@gmail.com>,
        Alvaro Herrera from 2ndQuadrant <alvherre@alvh.no-ip.org>,
        Andres Freund <andres@anarazel.de>,
        =?UTF-8?Q?Juan_Jos=C3=A9_Santamar=C3=ADa_Flecha?=
 <juanjo.santamaria@gmail.com>,
        PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>,
        Robert Haas <robertmhaas@gmail.com>,
        Thomas Munro <thomas.munro@gmail.com>,
        Tomas Vondra <tomas.vondra@2ndquadrant.com>
Subject: Re: logical decoding : exceeded maxAllocatedDescs for .spill files
In-reply-to: 
 <CAA4eK1JitJgbOv-ie3VCXW5A74j-v0b8S55YMunwmkFDZ_4NCg@mail.gmail.com>
References: 
 <CAA4eK1+fogKCWvFzkEbSn5O7HQ-Xo+RKOtFf8=+tfqOzxd21dg@mail.gmail.com>
 <CAJ3gD9dV9SBCoMOJDZPDrB9-W74UpMvEeL+DBqg2Yg2BSmQstQ@mail.gmail.com>
 <CAA4eK1JdK+-uz_fXOo+p6no7jxvthjmxDkqyqC2bmKvHXiRUjw@mail.gmail.com>
 <CAJ3gD9edQpM6XkNGaaQ1eN9698-DOiQbF8UoseVXAQPwT0Se7A@mail.gmail.com>
 <CAA4eK1JSEn=UuTiYiAwqAWWEmLvMtnPMQ2+my0X8diEgtG6LqA@mail.gmail.com>
 <CAA4eK1LMDx6vK8Kdw8WUeW1MjToN2xVffL2kvtHvZg17=Y6QQg@mail.gmail.com>
 <CAA4eK1LJqMuXoCLuxkTr1HidbR8DkgRrVC7jHWDyXT=FD2gt6Q@mail.gmail.com>
 <CAJ3gD9dFYJnOxUsR3MeJ=3hJKZk5XqAx-6fYUNXt6jJOK3QFMQ@mail.gmail.com>
 <20200104185148.GA2270238@rfd.leadboat.com>
 <CAJ3gD9diTB8rqyUWo=jZrSQxvW7o-gGaO6uoF5racneH3+FYOg@mail.gmail.com>
 <20200109053704.GA2502006@rfd.leadboat.com> <22498.1578548741@sss.pgh.pa.us>
 <CAA4eK1JitJgbOv-ie3VCXW5A74j-v0b8S55YMunwmkFDZ_4NCg@mail.gmail.com>
Comments: In-reply-to Amit Kapila <amit.kapila16@gmail.com>
	message dated "Thu, 09 Jan 2020 11:47:46 +0530"
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-ID: <14738.1578613901.1@sss.pgh.pa.us>
Content-Transfer-Encoding: quoted-printable
Date: Thu, 09 Jan 2020 18:51:41 -0500
Message-ID: <14739.1578613901@sss.pgh.pa.us>
Precedence: bulk

Amit Kapila <amit.kapila16@gmail.com> writes:
> On Thu, Jan 9, 2020 at 11:15 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Noah Misch <noah@leadboat.com> writes:
>>> Even so, a web search for "extend_brk" led to the answer.  By default,=
 32-bit
>>> AIX binaries get only 256M of RAM for stack and sbrk.  The new regress=
ion test
>>> used more than that, hence this crash.

>> Hm, so
>> (1) Why did we get a crash and not some more-decipherable out-of-resour=
ces
>> error?  Can we improve that experience?
>> (2) Should we be dialing back the resource consumption of this test?

> In HEAD, we have a guc variable 'logical_decoding_work_mem' by which
> we can control the memory usage of changes and we have used that, but
> for back branches, we don't have such a control.

I poked into this a bit more by running the src/test/recovery tests under
restrictive ulimit settings.  I used

ulimit -s 1024
ulimit -v 250000

(At least on my 64-bit RHEL6 box, reducing ulimit -v much below this
causes initdb to fail, apparently because the post-bootstrap process
tries to load all our tsearch and encoding conversion shlibs at once,
and it hasn't got enough VM space to do so.  Someday we may have to
improve that.)

I did not manage to duplicate Noah's crash this way.  What I see in
the v10 branch is that the new 006_logical_decoding.pl test fails,
but with a clean "out of memory" error.  The memory map dump that
that produces fingers the culprit pretty unambiguously:

...
          ReorderBuffer: 223302560 total in 26995 blocks; 7056 free (3 chu=
nks); 223295504 used
            ReorderBufferByXid: 24576 total in 2 blocks; 11888 free (3 chu=
nks); 12688 used
            Slab: TXN: 8192 total in 1 blocks; 5208 free (21 chunks); 2984=
 used
            Slab: Change: 2170880 total in 265 blocks; 2800 free (35 chunk=
s); 2168080 used
...
Grand total: 226714720 bytes in 27327 blocks; 590888 free (785 chunks); 22=
6123832 used

The test case is only inserting 50K fairly-short rows, so this seems
like an unreasonable amount of memory to be consuming for that; and
even if you think it's reasonable, it clearly isn't going to scale
to large production transactions.

Now, the good news is that v11 and later get through
006_logical_decoding.pl just fine under the same restriction.
So we did something in v11 to fix this excessive memory consumption.
However, unless we're willing to back-port whatever that was, this
test case is clearly consuming excessive resources for the v10 branch.

We're not out of the woods either.  I also observe that v12 and HEAD
fall over, under these same test conditions, with a stack-overflow
error in the 012_subtransactions.pl test.  This seems to be due to
somebody's decision to use a heavily recursive function to generate a
bunch of subtransactions.  Is there a good reason for hs_subxids() to
use recursion instead of a loop?  If there is, what's the value of
using 201 levels rather than, say, 10?

Anyway it remains unclear why Noah's machine got a crash instead of
something more user-friendly.  But the reason why it's only in the
v10 branch seems non-mysterious.

			regards, tom lane