MIME-Version: 1.0
References: <5D2C9081-5DFE-4E27-AB14-7358238EA1BC@gmail.com>
 <B1F56C92-9B12-41A0-A6C1-F4BDA3949237@gmail.com>
In-Reply-To: <B1F56C92-9B12-41A0-A6C1-F4BDA3949237@gmail.com>
From: Ewan Young <kdbase.hack@gmail.com>
Date: Fri, 10 Jul 2026 10:44:12 +0800
Message-ID: 
 <CAON2xHNYC3kw=Y37Gv1TgfNjEPvTR5JWx0xwRj2UWu-=Ae9V9w@mail.gmail.com>
Subject: Re: Fix duplicate detection for null-treatment window functions
To: Chao Li <li.evan.chao@gmail.com>
Cc: PostgreSQL-development <pgsql-hackers@lists.postgresql.org>,
 Tatsuo Ishii <ishii@postgresql.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: 
 <https://www.postgresql.org/message-id/CAON2xHNYC3kw%3DY37Gv1TgfNjEPvTR5JWx0xwRj2UWu-%3DAe9V9w%40mail.gmail.com>
Precedence: bulk

Hi Chao,


On Thu, Jul 9, 2026 at 8:35=E2=80=AFPM Chao Li <li.evan.chao@gmail.com> wro=
te:
>
>
>
> > On Jul 7, 2026, at 08:14, Chao Li <li.evan.chao@gmail.com> wrote:
> >
> > Hi,
> >
> > I just spotted an oversight from =E2=80=9C[25a30bbd4] Add IGNORE NULLS/=
RESPECT NULLS option to Window functions=E2=80=9D.
> >
> > In ExecInitWindowAgg(), there is logic to detect duplicate functions:
> > ```
> > if (i <=3D wfuncno && wfunc->ignore_nulls =3D=3D perfunc[i].ignore_null=
s)
> > {
> > /* Found a match to an existing entry, so just mark it */
> > wfuncstate->wfuncno =3D i;
> > continue;
> > }
> > ```
> >
> > However, when appending function info to perfunc, ignore_nulls is not c=
opied:
> > ```
> > /* Fill in the perfuncstate data */
> > perfuncstate->wfuncstate =3D wfuncstate;
> > perfuncstate->wfunc =3D wfunc;
> > perfuncstate->numArguments =3D list_length(wfuncstate->args);
> > perfuncstate->winCollation =3D wfunc->inputcollid;
> > ```
> >
> > As a result, wfunc->ignore_nulls =3D=3D perfunc[i].ignore_nulls can nev=
er be true for duplicate IGNORE NULLS or explicit RESPECT NULLS calls. This=
 means duplicate detection doesn't work for those calls. This bug is easy t=
o prove by adding temporary logs, and the fix is straightforward: copy igno=
re_nulls when filling in the perfuncstate data.
> >
> > This is a simple repro:
> >
> > Without the fix:
> > ```
> > evantest=3D# WITH t(x) AS (VALUES (NULL::int), (1), (2))
> > evantest-# SELECT first_value(x) IGNORE NULLS OVER w AS a,
> > evantest-#        first_value(x) IGNORE NULLS OVER w AS b
> > evantest-# FROM t
> > evantest-# WINDOW w AS (ORDER BY x NULLS FIRST
> > evantest(#              ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED =
FOLLOWING);
> > INFO:  WindowAgg duplicate cache miss: no previous match, ignore_nulls =
1
> > INFO:  WindowAgg duplicate cache miss: matched wfuncno 0, ignore_nulls =
1, cached ignore_nulls 0
> > a | b
> > ---+---
> > 1 | 1
> > 1 | 1
> > 1 | 1
> > (3 rows)
> > ```
> >
> > With the fix:
> > ```
> > evantest=3D# WITH t(x) AS (VALUES (NULL::int), (1), (2))
> > evantest-# SELECT first_value(x) IGNORE NULLS OVER w AS a,
> > evantest-#        first_value(x) IGNORE NULLS OVER w AS b
> > evantest-# FROM t
> > evantest-# WINDOW w AS (ORDER BY x NULLS FIRST
> > evantest(#              ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED =
FOLLOWING);
> > INFO:  WindowAgg duplicate cache miss: no previous match, ignore_nulls =
1
> > INFO:  WindowAgg duplicate cache hit: existing wfuncno 0, ignore_nulls =
1
> > a | b
> > ---+---
> > 1 | 1
> > 1 | 1
> > 1 | 1
> > (3 rows)
> > ```
> >
> > See the attached patch for details. The actual fix is only one line. Th=
e INFO logs above were produced with temporary debug logs added around the =
duplicate-function lookup, those logs are not part of the proposed fix. I l=
eft those logs in the patch with TODO comments only so reviewers can see th=
e behavior before and after the fix. They should be removed before pushing.
> >
> > Best regards,
> > --
> > Chao Li (Evan)
> > HighGo Software Co., Ltd.
> > https://www.highgo.com/
> >
> >
> >
> >
> > <v1-0001-Fix-duplicate-detection-for-null-treatment-window.patch>
>
> I realized that leaving the temp log code in the patch is not friendly to=
 the CF test. So, splitting the temp log part into a diff file.

Nice catch, and v2 does restore the intended de-duplication.

While reviewing it, though, I think the check the patch repairs is actually
redundant, and the real issue is that the code (both the original commit an=
d
the patch) keeps a shadow copy of ignore_nulls that has to be maintained by
hand.

WindowFunc.ignore_nulls is a plain scalar field with no pg_node_attr, so
equal() already compares it -- the generated _equalWindowFunc() has:

    COMPARE_SCALAR_FIELD(winagg);
    COMPARE_SCALAR_FIELD(ignore_nulls);
    COMPARE_LOCATION_FIELD(location);

That means the equal() call in the dedup loop already distinguishes two
WindowFuncs that differ only in null treatment:

    for (i =3D 0; i <=3D wfuncno; i++)
    {
        if (equal(wfunc, perfunc[i].wfunc) &&
            !contain_volatile_functions((Node *) wfunc))
            break;
    }
    if (i <=3D wfuncno && wfunc->ignore_nulls =3D=3D perfunc[i].ignore_null=
s)

If ignore_nulls differs, equal() returns false, the loop never breaks on
that entry, and we never reach the extra term. If equal() matches, then
ignore_nulls necessarily matched too. So "&& wfunc->ignore_nulls =3D=3D
perfunc[i].ignore_nulls" can never change the outcome, and
WindowStatePerFuncData.ignore_nulls exists only to feed it -- it is read
only there and, before this patch, written nowhere (which is exactly why it
was always 0).

So rather than populating the field, I'd suggest dropping the redundant ter=
m
and the field, and letting equal() do the work:

and the field, and letting equal() do the work:

-   if (i <=3D wfuncno && wfunc->ignore_nulls =3D=3D perfunc[i].ignore_null=
s)
+   if (i <=3D wfuncno)
    {
        /* Found a match to an existing entry, so just mark it */
        wfuncstate->wfuncno =3D i;
        continue;
    }

plus removing the ignore_nulls member from WindowStatePerFuncData and
trimming the now-stale "which needs the same ignore_nulls value" comment.
That fixes the same bug while removing the duplicated state that caused it,
so it can't silently drift again.

The one argument for an explicit check is defensiveness: if someone later
tags ignore_nulls with a pg_node_attr that excludes it from equal(), the
loop would start collapsing functions with different null treatment. If
that's a worry, the robust form compares the stored node directly instead o=
f
a shadow copy, and still needs no separate field:

    if (i <=3D wfuncno &&
        wfunc->ignore_nulls =3D=3D perfunc[i].wfunc->ignore_nulls)

Given ignore_nulls has to stay significant to equal() anyway (two calls wit=
h
different null treatment really are different functions), I'd lean toward
just removing the check.

Happy to send a patch along these lines if you agree.

>
> Best regards,
> --
> Chao Li (Evan)
> HighGo Software Co., Ltd.
> https://www.highgo.com/
>
>
>
>

--=20
Regards,
Ewan Young