MIME-Version: 1.0
References: <7b0a51d6-0d9d-7366-3a1a-f74397a02f55@dunslane.net>
 <3892d996-0ec8-3a56-1a6c-b397a3c31516@dunslane.net> <CAOYmi+mZ6ZVDHsP6Fcx+3vGYRSY+7fXmQP7wNuk1KTOVtwK2Pw@mail.gmail.com>
 <CAOYmi+mZER=D1gEDOt7Td8tB214eHFbViYvDvEjF4uiJ3bFv=A@mail.gmail.com>
 <3efe8333-3285-1d9d-5ad3-0f9784d5c1c4@dunslane.net> <CAOYmi+kLQmvpGzMsry67B5=1JYhm04rv8J7QX-qh3Xy-kceGJw@mail.gmail.com>
 <1778cdac-bb93-5bbd-1505-5684e95b0094@dunslane.net> <CAOYmi+nHV55Uhz+o-HKq0GNiWn2L5gMcuyRQEz_fqpGY=pFxKA@mail.gmail.com>
 <CAOYmi+=DAS_SpWM8Qd-C2Cv0yWUkENbsHcjEYt4vdCAp_BG_Jw@mail.gmail.com>
 <1b30469a-9d29-c021-f28a-74e248760aa4@dunslane.net> <ec7d0be8-79f3-05ea-d5ed-323e496556dc@dunslane.net>
 <CAOYmi+kOY47od5NuB+379XSnAfTknocLkYV6qz+Gk1XRPwG1iQ@mail.gmail.com>
 <CAOYmi+kNPw+3hjjQarC8bm-iDa=CRmTB1TMb35zbydRLK+7+1A@mail.gmail.com>
 <682c8fff-355c-a04f-57ac-81055c4ccda8@dunslane.net> <70705cf3-c875-d489-c6a9-7dd61f1dc1af@dunslane.net>
 <CAOYmi+n1dZEMCptykQS7Jd4A_bMyebAD9LuFXM4NaeaYKwH3ng@mail.gmail.com>
 <694fd815-0414-0f09-98b3-db28d77fa37f@dunslane.net> <CAOYmi+kgckFsBNOBsrekDipq88G56pVndVXKcA7qMWYx9feChQ@mail.gmail.com>
 <CAD5tBcLjFnTnAvUpQg-K5sJa3ECnEj8CQZ-AYrH1k+F=42D+Vg@mail.gmail.com>
 <CAOYmi+n1fxCt3QGz9g=GfY1+J6__vVqsAud=HYwpUrfZM_VJHw@mail.gmail.com>
 <CAD5tBcLi2ffZkktV2qrsKSBykE-N8CiYgrfbv0vZ-F7=xLFeqw@mail.gmail.com> <CAD5tBcJg-Ja3CcY8N+NtziFN8TL8JRWv97s=-siT=Oep2+JGjw@mail.gmail.com>
In-Reply-To: <CAD5tBcJg-Ja3CcY8N+NtziFN8TL8JRWv97s=-siT=Oep2+JGjw@mail.gmail.com>
From: Jacob Champion <jacob.champion@enterprisedb.com>
Date: Wed, 20 Mar 2024 10:09:33 -0700
Message-ID: <CAOYmi+nY=rF6dJCzaOuA3d-3FbwXCcecOs_S1NutexFA3dRXAw@mail.gmail.com>
Subject: Re: WIP Incremental JSON Parser
To: Andrew Dunstan <andrew@dunslane.net>
Cc: Robert Haas <robertmhaas@gmail.com>, Peter Smith <smithpb2250@gmail.com>, 
	PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: <https://www.postgresql.org/message-id/CAOYmi%2BnY%3DrF6dJCzaOuA3d-3FbwXCcecOs_S1NutexFA3dRXAw%40mail.gmail.com>
Precedence: bulk

On Tue, Mar 19, 2024 at 3:07=E2=80=AFPM Andrew Dunstan <andrew@dunslane.net=
> wrote:
> On Mon, Mar 18, 2024 at 3:35=E2=80=AFPM Jacob Champion <jacob.champion@en=
terprisedb.com> wrote:
>> With the incremental parser, I think prev_token_terminator is not
>> likely to be safe to use except in very specific circumstances, since
>> it could be pointing into a stale chunk. Some documentation around how
>> to use that safely in a semantic action would be good.
>
> Quite right. It's not safe. Should we ensure it's set to something like N=
ULL or -1?

Nulling it out seems reasonable.

> Also, where do you think we should put a warning about it?

I was thinking in the doc comment for JsonLexContext.

> It also removes the frontend exits I had. In the case of stack depth, we =
follow the example of the RD parser and only check stack depth for backend =
code. In the case of the check that the lexer is set up for incremental par=
sing, the exit is replaced by an Assert. That means your test for an over-n=
ested array doesn't work any more, so I have commented it out.

Hm, okay. We really ought to fix the recursive parser, but that's for
a separate thread. (Probably OAuth.) The ideal behavior IMO would be
for the caller to configure a maximum depth in the JsonLexContext.

Note that the repalloc will eventually still exit() if the pstack gets
too big; is that a concern? Alternatively, could unbounded heap growth
be a problem for a superuser? I guess the scalars themselves aren't
capped for length...

On Wed, Mar 20, 2024 at 12:19=E2=80=AFAM Andrew Dunstan <andrew@dunslane.ne=
t> wrote:
> On second thoughts, I think it might be better if we invent a new error r=
eturn code for a lexer mode mismatch.

+1

--Jacob