Subject: Re: how to gate experimental features (SQL/PGQ)

From: Tom Lane <[email-masked]>
Date: Tue, 13 Jan 2026 12:24:46 -0500

Jacob Champion <[email-masked]> writes:
> On Tue, Jan 13, 2026 at 7:17 AM Andres Freund <[email-masked]> wrote:
>> I don't even know how you could implement 3) realistically. We have zero
>> infrastructure for making e.g. parser, keyword list etc change due to defines
>> compile time.

> Is that an architecturally unsolvable thing, or is it a simple matter
> of programming? Would it be nice to have said infrastructure?

You'd have to throw out flex and bison and build some sort of
extensible parser.  That has some attraction to me personally
(I worked on such systems decades ago at HP), but it's fairly
hard to justify the amount of effort that would be needed to
get there.  It might well be slower than a flex/bison parser,
and/or have poorer detection of grammar inconsistencies, either
of which would be bad for our usage.

> But overall, I think I'd rather work with an environment where we
> occasionally say "hey, this clearly-labeled experimental feature was
> not baked enough, pull it back now" as opposed to occasionally saying
> "hey, we never got this one feature everyone wants because we'll never
> be practically sure that it's 'right' enough without user testing".

This seems workable for some kinds of features and less so for others.
As an example, forcing initdb post-release seems like a nonstarter
even for "experimental" features, so anything that affects initial
catalog contents is still going to be a problem.

It strikes me that we do have a mechanism that could be used to cope
with catalog changes, which is to package the new catalog objects as
an extension.  Then ALTER EXTENSION UPDATE could be used to migrate
from FooBar Draft-2 to Draft-4.  But the tricky bit would be to
replace the extension with in-core objects once we decide it's
completely baked.  Our previous attempts to do that sort of thing
have been painful.  It might be better just to say that you only
get to change experimental catalog entries once a year in major
releases.  (Slow progress is still better than no progress.)

			regards, tom lane

---

From: Andrew Dunstan <[email-masked]>
Date: Thu, 15 Jan 2026 12:05:44 -0500

On 2026-01-13 Tu 12:24 PM, Tom Lane wrote:
> Jacob Champion<[email-masked]> writes:
>> On Tue, Jan 13, 2026 at 7:17 AM Andres Freund<[email-masked]> wrote:

Maybe, but maybe not. ISTR that gcc abandoned use of bison for their C 
compiler a long time ago, and that gnat's Ada compiler was hand cut from 
the get go.

SQL is a different kettle of fish, of course - it dwarfs C and Ada in 
complexity.

Not saying I think this would be a good thing or not. I agree that the 
effort required would be huge and the benefit might be modest, but it is 
feasible IMHO.

cheers

andrew

--
Andrew Dunstan
EDB:https://www.enterprisedb.com

---

From: Andres Freund <[email-masked]>
Date: Thu, 15 Jan 2026 12:23:52 -0500

Hi,

Sorry for the duplicate email, I somehow corrupted the last one.

On 2026-01-15 12:05:44 -0500, Andrew Dunstan wrote:
> compiler a long time ago, and that gnat's Ada compiler was hand cut from the
> get go.
> 

The handwritten parser is also used for C++ which is *quite* a bit more
complicated than C, and probably roughly on-par with SQL.

I think the situation for C like languages is a bit different than with SQL
though, because typically the compiler specific syntax extensions are much
more modest than with SQL. Due to the different compilers for C like language,
it's probably easier to find grammar / language issues than for something like
postgres SQL dialect, which differs substantially from other SQL
implementations and thus only has a single parser.

That said, I do suspect we might eventually want to just give up on the
"parser generator as a sanity check" aspect and go with a hand written
recursive descent parser. For speed, extensibility and better error
recovery/messages.

Greetings,

Andres Freund

---

From: Tom Lane <[email-masked]>
Date: Thu, 15 Jan 2026 12:31:35 -0500

> SQL is a different kettle of fish, of course - it dwarfs C and Ada in 

Yeah.  And we have additional problems besides the grammar being far
larger than it is for those projects:

* It's a moving target (to a much greater degree than C, anyway).

* A lot of our code is written by people who are not parser experts.

So it gives me great comfort that bison will complain if you hand it
an ambiguous or unsatisfiable grammar.  If we went over to a
handwritten parser, I'd have next to no faith in it not being buggy.
There's good reasons why people put so much effort into parser
generators back in the day.

			regards, tom lane