pg.ddx.io pgsql-bugs@postgresql.org mailing list archive
help / color / mirror / Atom feedPostgres 16.1 - Bug: cache entry already complete
14+ messages / 4 participants
[nested] [flat]
* Postgres 16.1 - Bug: cache entry already complete
@ 2024-01-02 15:36 Amadeo Gallardo <amadeo@ruddr.io>
2024-01-02 17:31 ` Re: Postgres 16.1 - Bug: cache entry already complete Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 14+ messages in thread
From: Amadeo Gallardo @ 2024-01-02 15:36 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org
Hi,
Upgrading from Postgres 14 to Postgres 16.1 introduced the following error
when running a query:
nativeError: error: select "date", coalesce(sum(hours), 0) as "hours" from
"allocation_date" where "allocation_date"."project_id" = $1 and
"allocation_date"."date" > $2 and "allocation_date"."hours" > $3 group by
"date" order by "date" asc - cache entry already complete
Note that `allocation_date` is a fairly complex view. Not quite sure what
causes the "cache entry already complete", but it consistently failed in
both the MacOS and Windows installations.
Maybe it's a regression of:
https://www.postgresql.org/message-id/CAApHDvrkXT%2Bzhrusz7xdRBS3jZR%3DkB32AgBc4cnFCtxTaxQHvA%40mail...
There's also a comment here:
https://www.postgresql.org/about/news/postgresql-weekly-news-may-23-2021-2217/
Which points to:
https://www.postgresql.org/message-id/CAOxo6X+dy-V58iEPFgst8ahPKEU+38NZzUuc+a7wDBZd4TrHMQ@mail.gmail...
Anything I can do to help you reproduce this?
Thanks,
--
*Amadeo Gallardo*
email: amadeo@ruddr.io
mobile: 786.961.0106
web: www.ruddr.io
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: Postgres 16.1 - Bug: cache entry already complete
2024-01-02 15:36 Postgres 16.1 - Bug: cache entry already complete Amadeo Gallardo <amadeo@ruddr.io>
@ 2024-01-02 17:31 ` Tom Lane <tgl@sss.pgh.pa.us>
2024-01-02 19:51 ` Re: Postgres 16.1 - Bug: cache entry already complete Amadeo Gallardo <amadeo@ruddr.io>
0 siblings, 1 reply; 14+ messages in thread
From: Tom Lane @ 2024-01-02 17:31 UTC (permalink / raw)
To: Amadeo Gallardo <amadeo@ruddr.io>; +Cc: pgsql-bugs@lists.postgresql.org, David Rowley <dgrowleyml@gmail.com>
Amadeo Gallardo <amadeo@ruddr.io> writes:
> Upgrading from Postgres 14 to Postgres 16.1 introduced the following error
> when running a query:
> nativeError: error: select "date", coalesce(sum(hours), 0) as "hours" from
> "allocation_date" where "allocation_date"."project_id" = $1 and
> "allocation_date"."date" > $2 and "allocation_date"."hours" > $3 group by
> "date" order by "date" asc - cache entry already complete
> Note that `allocation_date` is a fairly complex view. Not quite sure what
> causes the "cache entry already complete", but it consistently failed in
> both the MacOS and Windows installations.
The only occurrence of that error string is in nodeMemoize.c, so
I suppose you've found a bug in Memoize. A bit odd that it doesn't
manifest in v14 where Memoize came in. But anyway, can you reduce
this to a self-contained test case?
regards, tom lane
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: Postgres 16.1 - Bug: cache entry already complete
2024-01-02 15:36 Postgres 16.1 - Bug: cache entry already complete Amadeo Gallardo <amadeo@ruddr.io>
2024-01-02 17:31 ` Re: Postgres 16.1 - Bug: cache entry already complete Tom Lane <tgl@sss.pgh.pa.us>
@ 2024-01-02 19:51 ` Amadeo Gallardo <amadeo@ruddr.io>
2024-01-02 23:48 ` Re: Postgres 16.1 - Bug: cache entry already complete Amadeo Gallardo <amadeo@ruddr.io>
0 siblings, 1 reply; 14+ messages in thread
From: Amadeo Gallardo @ 2024-01-02 19:51 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-bugs@lists.postgresql.org, David Rowley <dgrowleyml@gmail.com>
Hi Tom,
Thanks for reaching out. So far, I was able to narrow it down to a problem
when doing a union between two tables, one of them having a varchar column
and the other one having a text column. Adding an explicit cast seems to
have fixed it, but not really sure why yet. I will try to create a simple
repro scenario.
Thanks,
On Tue, Jan 2, 2024 at 2:31 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Amadeo Gallardo <amadeo@ruddr.io> writes:
> > Upgrading from Postgres 14 to Postgres 16.1 introduced the following
> error
> > when running a query:
>
> > nativeError: error: select "date", coalesce(sum(hours), 0) as "hours"
> from
> > "allocation_date" where "allocation_date"."project_id" = $1 and
> > "allocation_date"."date" > $2 and "allocation_date"."hours" > $3 group by
> > "date" order by "date" asc - cache entry already complete
>
> > Note that `allocation_date` is a fairly complex view. Not quite sure what
> > causes the "cache entry already complete", but it consistently failed in
> > both the MacOS and Windows installations.
>
> The only occurrence of that error string is in nodeMemoize.c, so
> I suppose you've found a bug in Memoize. A bit odd that it doesn't
> manifest in v14 where Memoize came in. But anyway, can you reduce
> this to a self-contained test case?
>
> regards, tom lane
>
--
*Amadeo Gallardo*
email: amadeo@ruddr.io
mobile: 786.961.0106
web: www.ruddr.io
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: Postgres 16.1 - Bug: cache entry already complete
2024-01-02 15:36 Postgres 16.1 - Bug: cache entry already complete Amadeo Gallardo <amadeo@ruddr.io>
2024-01-02 17:31 ` Re: Postgres 16.1 - Bug: cache entry already complete Tom Lane <tgl@sss.pgh.pa.us>
2024-01-02 19:51 ` Re: Postgres 16.1 - Bug: cache entry already complete Amadeo Gallardo <amadeo@ruddr.io>
@ 2024-01-02 23:48 ` Amadeo Gallardo <amadeo@ruddr.io>
2024-01-02 23:59 ` Re: Postgres 16.1 - Bug: cache entry already complete David Rowley <dgrowleyml@gmail.com>
2024-01-03 00:05 ` Re: Postgres 16.1 - Bug: cache entry already complete Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 2 replies; 14+ messages in thread
From: Amadeo Gallardo @ 2024-01-02 23:48 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-bugs@lists.postgresql.org, David Rowley <dgrowleyml@gmail.com>
Hi,
Attached is a sample script that I used to reproduce the error. I had to
add some specific sample data and force-disable certain planner settings,
as otherwise the problem did not occur.
Additionally, here's a trace of the error:
org.postgresql.util.PSQLException: ERROR: cache entry already complete
> at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2676)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2366)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:356)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:496)
> at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:413)
> at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:333)
> at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:319)
> at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:295)
> at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:290)
Hopefully this helps to narrow down the issue.
Thanks
On Tue, Jan 2, 2024 at 4:51 PM Amadeo Gallardo <amadeo@ruddr.io> wrote:
> Hi Tom,
>
> Thanks for reaching out. So far, I was able to narrow it down to a problem
> when doing a union between two tables, one of them having a varchar column
> and the other one having a text column. Adding an explicit cast seems to
> have fixed it, but not really sure why yet. I will try to create a simple
> repro scenario.
>
> Thanks,
>
> On Tue, Jan 2, 2024 at 2:31 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
>> Amadeo Gallardo <amadeo@ruddr.io> writes:
>> > Upgrading from Postgres 14 to Postgres 16.1 introduced the following
>> error
>> > when running a query:
>>
>> > nativeError: error: select "date", coalesce(sum(hours), 0) as "hours"
>> from
>> > "allocation_date" where "allocation_date"."project_id" = $1 and
>> > "allocation_date"."date" > $2 and "allocation_date"."hours" > $3 group
>> by
>> > "date" order by "date" asc - cache entry already complete
>>
>> > Note that `allocation_date` is a fairly complex view. Not quite sure
>> what
>> > causes the "cache entry already complete", but it consistently failed in
>> > both the MacOS and Windows installations.
>>
>> The only occurrence of that error string is in nodeMemoize.c, so
>> I suppose you've found a bug in Memoize. A bit odd that it doesn't
>> manifest in v14 where Memoize came in. But anyway, can you reduce
>> this to a self-contained test case?
>>
>> regards, tom lane
>>
>
>
> --
> *Amadeo Gallardo*
> email: amadeo@ruddr.io
> mobile: 786.961.0106
> web: www.ruddr.io
>
--
*Amadeo Gallardo*
email: amadeo@ruddr.io
mobile: 786.961.0106
web: www.ruddr.io
Attachments:
cache-entry-already-complete.zip (application/x-zip-compressed, 3.4K) download
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: Postgres 16.1 - Bug: cache entry already complete
2024-01-02 15:36 Postgres 16.1 - Bug: cache entry already complete Amadeo Gallardo <amadeo@ruddr.io>
2024-01-02 17:31 ` Re: Postgres 16.1 - Bug: cache entry already complete Tom Lane <tgl@sss.pgh.pa.us>
2024-01-02 19:51 ` Re: Postgres 16.1 - Bug: cache entry already complete Amadeo Gallardo <amadeo@ruddr.io>
2024-01-02 23:48 ` Re: Postgres 16.1 - Bug: cache entry already complete Amadeo Gallardo <amadeo@ruddr.io>
@ 2024-01-02 23:59 ` David Rowley <dgrowleyml@gmail.com>
2024-01-03 02:36 ` Re: Postgres 16.1 - Bug: cache entry already complete Richard Guo <guofenglinux@gmail.com>
1 sibling, 1 reply; 14+ messages in thread
From: David Rowley @ 2024-01-02 23:59 UTC (permalink / raw)
To: Amadeo Gallardo <amadeo@ruddr.io>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; pgsql-bugs@lists.postgresql.org
On Wed, 3 Jan 2024 at 12:49, Amadeo Gallardo <amadeo@ruddr.io> wrote:
> Attached is a sample script that I used to reproduce the error. I had to add some specific sample data and force-disable certain planner settings, as otherwise the problem did not occur.
> Additionally, here's a trace of the error:
Thanks for the reproducer. I can recreate this using your script
after adding an ANALYZE after the INSERTs. Without ndistinct
statistics, the planner won't risk using Memoize.
I'll investigate the cache complete error.
David
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: Postgres 16.1 - Bug: cache entry already complete
2024-01-02 15:36 Postgres 16.1 - Bug: cache entry already complete Amadeo Gallardo <amadeo@ruddr.io>
2024-01-02 17:31 ` Re: Postgres 16.1 - Bug: cache entry already complete Tom Lane <tgl@sss.pgh.pa.us>
2024-01-02 19:51 ` Re: Postgres 16.1 - Bug: cache entry already complete Amadeo Gallardo <amadeo@ruddr.io>
2024-01-02 23:48 ` Re: Postgres 16.1 - Bug: cache entry already complete Amadeo Gallardo <amadeo@ruddr.io>
2024-01-02 23:59 ` Re: Postgres 16.1 - Bug: cache entry already complete David Rowley <dgrowleyml@gmail.com>
@ 2024-01-03 02:36 ` Richard Guo <guofenglinux@gmail.com>
2024-01-03 03:05 ` Re: Postgres 16.1 - Bug: cache entry already complete Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 14+ messages in thread
From: Richard Guo @ 2024-01-03 02:36 UTC (permalink / raw)
To: David Rowley <dgrowleyml@gmail.com>; +Cc: Amadeo Gallardo <amadeo@ruddr.io>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-bugs@lists.postgresql.org
On Wed, Jan 3, 2024 at 7:59 AM David Rowley <dgrowleyml@gmail.com> wrote:
> On Wed, 3 Jan 2024 at 12:49, Amadeo Gallardo <amadeo@ruddr.io> wrote:
> > Attached is a sample script that I used to reproduce the error. I had to
> add some specific sample data and force-disable certain planner settings,
> as otherwise the problem did not occur.
> > Additionally, here's a trace of the error:
>
> Thanks for the reproducer. I can recreate this using your script
> after adding an ANALYZE after the INSERTs. Without ndistinct
> statistics, the planner won't risk using Memoize.
It seems that the repro script was not sent to the mailing list. I
cannot see it.
Thanks
Richard
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: Postgres 16.1 - Bug: cache entry already complete
2024-01-02 15:36 Postgres 16.1 - Bug: cache entry already complete Amadeo Gallardo <amadeo@ruddr.io>
2024-01-02 17:31 ` Re: Postgres 16.1 - Bug: cache entry already complete Tom Lane <tgl@sss.pgh.pa.us>
2024-01-02 19:51 ` Re: Postgres 16.1 - Bug: cache entry already complete Amadeo Gallardo <amadeo@ruddr.io>
2024-01-02 23:48 ` Re: Postgres 16.1 - Bug: cache entry already complete Amadeo Gallardo <amadeo@ruddr.io>
2024-01-02 23:59 ` Re: Postgres 16.1 - Bug: cache entry already complete David Rowley <dgrowleyml@gmail.com>
2024-01-03 02:36 ` Re: Postgres 16.1 - Bug: cache entry already complete Richard Guo <guofenglinux@gmail.com>
@ 2024-01-03 03:05 ` Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 14+ messages in thread
From: Tom Lane @ 2024-01-03 03:05 UTC (permalink / raw)
To: Richard Guo <guofenglinux@gmail.com>; +Cc: David Rowley <dgrowleyml@gmail.com>; Amadeo Gallardo <amadeo@ruddr.io>; pgsql-bugs@lists.postgresql.org
Richard Guo <guofenglinux@gmail.com> writes:
> It seems that the repro script was not sent to the mailing list. I
> cannot see it.
Awaiting moderator approval, I suppose.
regards, tom lane
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: Postgres 16.1 - Bug: cache entry already complete
2024-01-02 15:36 Postgres 16.1 - Bug: cache entry already complete Amadeo Gallardo <amadeo@ruddr.io>
2024-01-02 17:31 ` Re: Postgres 16.1 - Bug: cache entry already complete Tom Lane <tgl@sss.pgh.pa.us>
2024-01-02 19:51 ` Re: Postgres 16.1 - Bug: cache entry already complete Amadeo Gallardo <amadeo@ruddr.io>
2024-01-02 23:48 ` Re: Postgres 16.1 - Bug: cache entry already complete Amadeo Gallardo <amadeo@ruddr.io>
@ 2024-01-03 00:05 ` Tom Lane <tgl@sss.pgh.pa.us>
2024-01-03 00:40 ` Re: Postgres 16.1 - Bug: cache entry already complete David Rowley <dgrowleyml@gmail.com>
1 sibling, 1 reply; 14+ messages in thread
From: Tom Lane @ 2024-01-03 00:05 UTC (permalink / raw)
To: Amadeo Gallardo <amadeo@ruddr.io>; +Cc: pgsql-bugs@lists.postgresql.org, David Rowley <dgrowleyml@gmail.com>
Amadeo Gallardo <amadeo@ruddr.io> writes:
> Attached is a sample script that I used to reproduce the error. I had to
> add some specific sample data and force-disable certain planner settings,
> as otherwise the problem did not occur.
Thanks! EXPLAIN shows
QUERY PLAN
------------------------------------------------------------------------------------------------
Nested Loop Left Join (cost=0.46..82.87 rows=315 width=1)
Join Filter: (t3.t2_id = t2.id)
-> Nested Loop Left Join (cost=0.30..32.29 rows=315 width=32)
-> Nested Loop Left Join (cost=0.15..17.29 rows=315 width=32)
-> Seq Scan on t4 (cost=0.00..6.15 rows=315 width=32)
-> Memoize (cost=0.15..0.19 rows=1 width=16)
Cache Key: t4.t2_id
Cache Mode: logical
-> Index Only Scan using t2_pkey on t2 (cost=0.14..0.18 rows=1 width=16)
Index Cond: (id = t4.t2_id)
-> Memoize (cost=0.15..0.20 rows=1 width=16)
Cache Key: t4.t1_id
Cache Mode: logical
-> Index Only Scan using t1_pkey on t1 (cost=0.14..0.19 rows=1 width=16)
Index Cond: (id = t4.t1_id)
-> Memoize (cost=0.16..0.55 rows=1 width=33)
Cache Key: t1.id, t1.id
Cache Mode: logical
-> Index Scan using t3_t1_id_index on t3 (cost=0.15..0.54 rows=1 width=33)
Index Cond: (t1_id = t1.id)
(20 rows)
It's that last Memoize node with the duplicate cache keys that is
failing. (It's unclear to me whether having duplicate cache keys
is supposed to be a valid situation, and even less clear whether
that has any direct connection to the failure.) What I guess
is happening is that the single-row flag is getting set because
t1.id is a primary key, despite the fact that it's coming from
inside a join that could result in duplicates.
regards, tom lane
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: Postgres 16.1 - Bug: cache entry already complete
2024-01-02 15:36 Postgres 16.1 - Bug: cache entry already complete Amadeo Gallardo <amadeo@ruddr.io>
2024-01-02 17:31 ` Re: Postgres 16.1 - Bug: cache entry already complete Tom Lane <tgl@sss.pgh.pa.us>
2024-01-02 19:51 ` Re: Postgres 16.1 - Bug: cache entry already complete Amadeo Gallardo <amadeo@ruddr.io>
2024-01-02 23:48 ` Re: Postgres 16.1 - Bug: cache entry already complete Amadeo Gallardo <amadeo@ruddr.io>
2024-01-03 00:05 ` Re: Postgres 16.1 - Bug: cache entry already complete Tom Lane <tgl@sss.pgh.pa.us>
@ 2024-01-03 00:40 ` David Rowley <dgrowleyml@gmail.com>
2024-01-03 01:47 ` Re: Postgres 16.1 - Bug: cache entry already complete David Rowley <dgrowleyml@gmail.com>
0 siblings, 1 reply; 14+ messages in thread
From: David Rowley @ 2024-01-03 00:40 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Amadeo Gallardo <amadeo@ruddr.io>; pgsql-bugs@lists.postgresql.org
On Wed, 3 Jan 2024 at 13:05, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> QUERY PLAN
> ------------------------------------------------------------------------------------------------
> Nested Loop Left Join (cost=0.46..82.87 rows=315 width=1)
> Join Filter: (t3.t2_id = t2.id)
> -> Nested Loop Left Join (cost=0.30..32.29 rows=315 width=32)
> -> Nested Loop Left Join (cost=0.15..17.29 rows=315 width=32)
> -> Seq Scan on t4 (cost=0.00..6.15 rows=315 width=32)
> -> Memoize (cost=0.15..0.19 rows=1 width=16)
> Cache Key: t4.t2_id
> Cache Mode: logical
> -> Index Only Scan using t2_pkey on t2 (cost=0.14..0.18 rows=1 width=16)
> Index Cond: (id = t4.t2_id)
> -> Memoize (cost=0.15..0.20 rows=1 width=16)
> Cache Key: t4.t1_id
> Cache Mode: logical
> -> Index Only Scan using t1_pkey on t1 (cost=0.14..0.19 rows=1 width=16)
> Index Cond: (id = t4.t1_id)
> -> Memoize (cost=0.16..0.55 rows=1 width=33)
> Cache Key: t1.id, t1.id
> Cache Mode: logical
> -> Index Scan using t3_t1_id_index on t3 (cost=0.15..0.54 rows=1 width=33)
> Index Cond: (t1_id = t1.id)
> (20 rows)
>
> It's that last Memoize node with the duplicate cache keys that is
> failing. (It's unclear to me whether having duplicate cache keys
> is supposed to be a valid situation, and even less clear whether
> that has any direct connection to the failure.) What I guess
> is happening is that the single-row flag is getting set because
> t1.id is a primary key, despite the fact that it's coming from
> inside a join that could result in duplicates.
The problem is that singlerow is set entirely based on innerunique.
That's wrong because in this case, the unique properties include the
Join Filter, not just the parameterized qual.
I think the fix should be in get_memoize_path(). What we pass to
create_memoize_path() for the "singlerow" parameter needs to be more
than just extra->inner_unique. The Join Filter must also be empty.
David
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: Postgres 16.1 - Bug: cache entry already complete
2024-01-02 15:36 Postgres 16.1 - Bug: cache entry already complete Amadeo Gallardo <amadeo@ruddr.io>
2024-01-02 17:31 ` Re: Postgres 16.1 - Bug: cache entry already complete Tom Lane <tgl@sss.pgh.pa.us>
2024-01-02 19:51 ` Re: Postgres 16.1 - Bug: cache entry already complete Amadeo Gallardo <amadeo@ruddr.io>
2024-01-02 23:48 ` Re: Postgres 16.1 - Bug: cache entry already complete Amadeo Gallardo <amadeo@ruddr.io>
2024-01-03 00:05 ` Re: Postgres 16.1 - Bug: cache entry already complete Tom Lane <tgl@sss.pgh.pa.us>
2024-01-03 00:40 ` Re: Postgres 16.1 - Bug: cache entry already complete David Rowley <dgrowleyml@gmail.com>
@ 2024-01-03 01:47 ` David Rowley <dgrowleyml@gmail.com>
2024-01-03 11:01 ` Re: Postgres 16.1 - Bug: cache entry already complete Richard Guo <guofenglinux@gmail.com>
0 siblings, 1 reply; 14+ messages in thread
From: David Rowley @ 2024-01-03 01:47 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Amadeo Gallardo <amadeo@ruddr.io>; pgsql-bugs@lists.postgresql.org
On Wed, 3 Jan 2024 at 13:40, David Rowley <dgrowleyml@gmail.com> wrote:
> The problem is that singlerow is set entirely based on innerunique.
> That's wrong because in this case, the unique properties include the
> Join Filter, not just the parameterized qual.
>
> I think the fix should be in get_memoize_path(). What we pass to
> create_memoize_path() for the "singlerow" parameter needs to be more
> than just extra->inner_unique. The Join Filter must also be empty.
It looks like we only figure out what will become the "Join Filter" in
create_nestloop_path(), which is slightly too late as we create the
MemoizePath before creating the NestLoopPath.
In the attached, I drafted up some code that duplicates the logic in
create_nestloop_path() to determine if there's going to be a
joinrestrictinfo (Join Filter) and to set MemoizePath.singlerow to
false if there is going to be one.
My concern with that is that the Unique Join optimisation will cause
execution to skip to the next outer row and that will leave us no
means of marking the Memoize cache entry as complete. In
singlerow==false Memoize nodes, we only mark the cache as complete
when we read the inner node to completion. The unique join
optimisation means we often don't do that due to skipping to the next
outer row on finding the first inner match.
Basically, what I'm saying is that Memoize is going to result in many
more cache misses due to incomplete cache entries. Maybe we should
have get_memoize_path() return NULL for this case so that we don't
Memoize when there's a Join Filter and extra->inner_unique is set to
true.
With the attached, I see:
QUERY PLAN
------------------------------------------------------------------------------------------
Nested Loop Left Join (actual rows=315 loops=1)
Join Filter: (t3.t2_id = t2.id)
Rows Removed by Join Filter: 1007
-> Nested Loop Left Join (actual rows=315 loops=1)
-> Nested Loop Left Join (actual rows=315 loops=1)
-> Seq Scan on t4 (actual rows=315 loops=1)
-> Memoize (actual rows=1 loops=315)
Cache Key: t4.t2_id
Cache Mode: logical
SingleRow: true
Hits: 296 Misses: 19 Evictions: 0 Overflows: 0
Memory Usage: 3kB
-> Index Only Scan using t2_pkey on t2 (actual
rows=1 loops=19)
Index Cond: (id = t4.t2_id)
Heap Fetches: 18
-> Memoize (actual rows=1 loops=315)
Cache Key: t4.t1_id
Cache Mode: logical
SingleRow: true
Hits: 276 Misses: 39 Evictions: 0 Overflows: 0
Memory Usage: 5kB
-> Index Only Scan using t1_pkey on t1 (actual rows=1 loops=39)
Index Cond: (id = t4.t1_id)
Heap Fetches: 38
-> Memoize (actual rows=4 loops=315)
Cache Key: t1.id, t1.id
Cache Mode: logical
SingleRow: false
Hits: 199 Misses: 116 Evictions: 0 Overflows: 0 Memory Usage: 15kB
-> Index Scan using t3_t1_id_index on t3 (actual rows=3 loops=116)
Index Cond: (t1_id = t1.id)
Planning Time: 0.322 ms
Execution Time: 3654.894 ms
(31 rows)
(I just added SingleRow to explain to make it easier to see what's going on)
Notice the Cache Misses is 116 for the problem Memoize node. There
are only 77 distinct values for t1_id in t3. So we're certainly
classing cache entries as invalid due to them being complete==false.
David
Attachments:
memoize_singlerow.patch (text/plain, 1.7K) download
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index f1d71bc54e..bfafb4cf7f 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -3198,6 +3198,8 @@ show_memoize_info(MemoizeState *mstate, List *ancestors, ExplainState *es)
appendStringInfo(es->str, "Cache Mode: %s\n", mstate->binary_mode ? "binary" : "logical");
}
+ ExplainPropertyBool("SingleRow", mstate->singlerow, es);
+
pfree(keystr.data);
if (!es->analyze)
diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c
index b2916ad690..09d7b71cab 100644
--- a/src/backend/optimizer/path/joinpath.c
+++ b/src/backend/optimizer/path/joinpath.c
@@ -682,12 +682,45 @@ get_memoize_path(PlannerInfo *root, RelOptInfo *innerrel,
&hash_operators,
&binary_mode))
{
+ bool singlerow;
+
+ if (extra->inner_unique)
+ {
+ Relids inner_req_outer = PATH_REQ_OUTER(inner_path);
+
+ singlerow = true;
+
+ if (bms_overlap(inner_req_outer, outer_path->parent->relids))
+ {
+
+ Bitmapset *enforced_serials;
+ ListCell *lc;
+
+ enforced_serials = get_param_path_clause_serials(inner_path);
+
+ foreach (lc, extra->restrictlist)
+ {
+ RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc);
+
+ if (!bms_is_member(rinfo->rinfo_serial, enforced_serials))
+ {
+ singlerow = false;
+ break;
+ }
+ }
+ }
+ }
+ else
+ {
+ singlerow = false;
+ }
+
return (Path *) create_memoize_path(root,
innerrel,
inner_path,
param_exprs,
hash_operators,
- extra->inner_unique,
+ singlerow,
binary_mode,
outer_path->rows);
}
* Re: Postgres 16.1 - Bug: cache entry already complete
2024-01-02 15:36 Postgres 16.1 - Bug: cache entry already complete Amadeo Gallardo <amadeo@ruddr.io>
2024-01-02 17:31 ` Re: Postgres 16.1 - Bug: cache entry already complete Tom Lane <tgl@sss.pgh.pa.us>
2024-01-02 19:51 ` Re: Postgres 16.1 - Bug: cache entry already complete Amadeo Gallardo <amadeo@ruddr.io>
2024-01-02 23:48 ` Re: Postgres 16.1 - Bug: cache entry already complete Amadeo Gallardo <amadeo@ruddr.io>
2024-01-03 00:05 ` Re: Postgres 16.1 - Bug: cache entry already complete Tom Lane <tgl@sss.pgh.pa.us>
2024-01-03 00:40 ` Re: Postgres 16.1 - Bug: cache entry already complete David Rowley <dgrowleyml@gmail.com>
2024-01-03 01:47 ` Re: Postgres 16.1 - Bug: cache entry already complete David Rowley <dgrowleyml@gmail.com>
@ 2024-01-03 11:01 ` Richard Guo <guofenglinux@gmail.com>
2024-01-04 12:04 ` Re: Postgres 16.1 - Bug: cache entry already complete David Rowley <dgrowleyml@gmail.com>
2024-01-22 09:49 ` Re: Postgres 16.1 - Bug: cache entry already complete David Rowley <dgrowleyml@gmail.com>
0 siblings, 2 replies; 14+ messages in thread
From: Richard Guo @ 2024-01-03 11:01 UTC (permalink / raw)
To: David Rowley <dgrowleyml@gmail.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Amadeo Gallardo <amadeo@ruddr.io>; pgsql-bugs@lists.postgresql.org
On Wed, Jan 3, 2024 at 9:48 AM David Rowley <dgrowleyml@gmail.com> wrote:
> My concern with that is that the Unique Join optimisation will cause
> execution to skip to the next outer row and that will leave us no
> means of marking the Memoize cache entry as complete. In
> singlerow==false Memoize nodes, we only mark the cache as complete
> when we read the inner node to completion. The unique join
> optimisation means we often don't do that due to skipping to the next
> outer row on finding the first inner match.
>
> Basically, what I'm saying is that Memoize is going to result in many
> more cache misses due to incomplete cache entries. Maybe we should
> have get_memoize_path() return NULL for this case so that we don't
> Memoize when there's a Join Filter and extra->inner_unique is set to
> true.
I agree that we should just avoid making a memoize path in this case.
And I think we already have this logic in get_memoize_path():
if (extra->inner_unique &&
(inner_path->param_info == NULL ||
list_length(inner_path->param_info->ppi_clauses) <
list_length(extra->restrictlist)))
return NULL;
But why does the code above fail to capture the problematic query? It
turns out that inner_path->param_info->ppi_clauses contains cloned
clauses, i.e. multiple versions of the same clause in order to cope with
outer join identity 3. And this causes the above code to give incorrect
conclusions about the length of 'ppi_clauses'. And this also explains
why this issue only occurs in v16.
As a clause's serial number is unique within the current PlannerInfo
context, and multiple clones of the same clause have the same serial
number, it seems to me that it's more correct to calculate the length of
ppi_clauses by:
bms_num_members(inner_path->param_info->ppi_serials)
So I think maybe we can fix this issue with the attached patch.
BTW, the SingleRow in explain seems quite handy. I wonder if we can
keep it.
Thanks
Richard
Attachments:
v2-0001-memoize-singlerow.patch (application/octet-stream, 3.2K) download
From 2ad681e8483e51897ba052a391537ab6bdc40f9c Mon Sep 17 00:00:00 2001
From: Richard Guo <guofenglinux@gmail.com>
Date: Wed, 3 Jan 2024 18:52:20 +0800
Subject: [PATCH v2] memoize singlerow
---
src/backend/optimizer/path/joinpath.c | 2 +-
src/test/regress/expected/memoize.out | 27 +++++++++++++++++++++++++++
src/test/regress/sql/memoize.sql | 16 ++++++++++++++++
3 files changed, 44 insertions(+), 1 deletion(-)
diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c
index b2916ad690..c961146526 100644
--- a/src/backend/optimizer/path/joinpath.c
+++ b/src/backend/optimizer/path/joinpath.c
@@ -635,7 +635,7 @@ get_memoize_path(PlannerInfo *root, RelOptInfo *innerrel,
*/
if (extra->inner_unique &&
(inner_path->param_info == NULL ||
- list_length(inner_path->param_info->ppi_clauses) <
+ bms_num_members(inner_path->param_info->ppi_serials) <
list_length(extra->restrictlist)))
return NULL;
diff --git a/src/test/regress/expected/memoize.out b/src/test/regress/expected/memoize.out
index f5202430f8..08df8941f8 100644
--- a/src/test/regress/expected/memoize.out
+++ b/src/test/regress/expected/memoize.out
@@ -348,3 +348,30 @@ RESET max_parallel_workers_per_gather;
RESET parallel_tuple_cost;
RESET parallel_setup_cost;
RESET min_parallel_table_scan_size;
+-- Test unique joins when not the entire join condition is parameterized
+CREATE TABLE tbl_singlerow(a int primary key, b int);
+INSERT INTO tbl_singlerow SELECT i, i FROM generate_series(1,10)i;
+ANALYZE tbl_singlerow;
+-- Ensure we do not get a Memoize plan
+EXPLAIN (COSTS OFF)
+SELECT * FROM tbl_singlerow t1
+ LEFT JOIN tbl_singlerow t2 ON TRUE
+ LEFT JOIN tbl_singlerow t3 ON TRUE
+ LEFT JOIN
+ LATERAL (SELECT t3.a AS t3a, * FROM tbl_singlerow t4) t4
+ ON t4.a = t3.b AND t4.b = t2.b;
+ QUERY PLAN
+------------------------------------------------------
+ Nested Loop Left Join
+ Join Filter: ((t4.a = t3.b) AND (t4.b = t2.b))
+ -> Nested Loop Left Join
+ -> Nested Loop Left Join
+ -> Seq Scan on tbl_singlerow t1
+ -> Materialize
+ -> Seq Scan on tbl_singlerow t2
+ -> Materialize
+ -> Seq Scan on tbl_singlerow t3
+ -> Seq Scan on tbl_singlerow t4
+(10 rows)
+
+DROP TABLE tbl_singlerow;
diff --git a/src/test/regress/sql/memoize.sql b/src/test/regress/sql/memoize.sql
index 29ab1ea62d..44b52e730e 100644
--- a/src/test/regress/sql/memoize.sql
+++ b/src/test/regress/sql/memoize.sql
@@ -181,3 +181,19 @@ RESET max_parallel_workers_per_gather;
RESET parallel_tuple_cost;
RESET parallel_setup_cost;
RESET min_parallel_table_scan_size;
+
+-- Test unique joins when not the entire join condition is parameterized
+CREATE TABLE tbl_singlerow(a int primary key, b int);
+INSERT INTO tbl_singlerow SELECT i, i FROM generate_series(1,10)i;
+ANALYZE tbl_singlerow;
+
+-- Ensure we do not get a Memoize plan
+EXPLAIN (COSTS OFF)
+SELECT * FROM tbl_singlerow t1
+ LEFT JOIN tbl_singlerow t2 ON TRUE
+ LEFT JOIN tbl_singlerow t3 ON TRUE
+ LEFT JOIN
+ LATERAL (SELECT t3.a AS t3a, * FROM tbl_singlerow t4) t4
+ ON t4.a = t3.b AND t4.b = t2.b;
+
+DROP TABLE tbl_singlerow;
--
2.31.0
* Re: Postgres 16.1 - Bug: cache entry already complete
2024-01-02 15:36 Postgres 16.1 - Bug: cache entry already complete Amadeo Gallardo <amadeo@ruddr.io>
2024-01-02 17:31 ` Re: Postgres 16.1 - Bug: cache entry already complete Tom Lane <tgl@sss.pgh.pa.us>
2024-01-02 19:51 ` Re: Postgres 16.1 - Bug: cache entry already complete Amadeo Gallardo <amadeo@ruddr.io>
2024-01-02 23:48 ` Re: Postgres 16.1 - Bug: cache entry already complete Amadeo Gallardo <amadeo@ruddr.io>
2024-01-03 00:05 ` Re: Postgres 16.1 - Bug: cache entry already complete Tom Lane <tgl@sss.pgh.pa.us>
2024-01-03 00:40 ` Re: Postgres 16.1 - Bug: cache entry already complete David Rowley <dgrowleyml@gmail.com>
2024-01-03 01:47 ` Re: Postgres 16.1 - Bug: cache entry already complete David Rowley <dgrowleyml@gmail.com>
2024-01-03 11:01 ` Re: Postgres 16.1 - Bug: cache entry already complete Richard Guo <guofenglinux@gmail.com>
@ 2024-01-04 12:04 ` David Rowley <dgrowleyml@gmail.com>
2024-01-11 03:26 ` Re: Postgres 16.1 - Bug: cache entry already complete Richard Guo <guofenglinux@gmail.com>
1 sibling, 1 reply; 14+ messages in thread
From: David Rowley @ 2024-01-04 12:04 UTC (permalink / raw)
To: Richard Guo <guofenglinux@gmail.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Amadeo Gallardo <amadeo@ruddr.io>; pgsql-bugs@lists.postgresql.org
On Thu, 4 Jan 2024 at 00:01, Richard Guo <guofenglinux@gmail.com> wrote:
> if (extra->inner_unique &&
> (inner_path->param_info == NULL ||
> list_length(inner_path->param_info->ppi_clauses) <
> list_length(extra->restrictlist)))
> return NULL;
I see. So it worked fine until Tom's " Make Vars be outer-join-aware." stuff.
9e215378d fixed this already and looks like I came to the same
conclusion about the cache completion issue back then too.
> But why does the code above fail to capture the problematic query? It
> turns out that inner_path->param_info->ppi_clauses contains cloned
> clauses, i.e. multiple versions of the same clause in order to cope with
> outer join identity 3. And this causes the above code to give incorrect
> conclusions about the length of 'ppi_clauses'. And this also explains
> why this issue only occurs in v16.
>
> As a clause's serial number is unique within the current PlannerInfo
> context, and multiple clones of the same clause have the same serial
> number, it seems to me that it's more correct to calculate the length of
> ppi_clauses by:
>
> bms_num_members(inner_path->param_info->ppi_serials)
>
> So I think maybe we can fix this issue with the attached patch.
hmm, I'd need to study what Tom's been changing around this. I don't
have a good handle on which qual lists can have duplicated quals.
In [1], you can see I took the inspiration for that fix from
generate_mergejoin_paths(). I don't yet know if it's just ppi_clauses
that can have duplicates or if there's a similar hazard in the
list_length() checks in generate_mergejoin_paths()
> BTW, the SingleRow in explain seems quite handy. I wonder if we can
> keep it.
If the fix is to just disable Memoize when the ppi_clauses don't cover
the entire join condition, then the Inner Unique from the parent
nested loop tells us this. Isn't that enough? The reason I added
SingleRow was because in the patch I posted, I'd made it so SingleRow
could be false in cases where Inner Unique was true.
David
[1] https://postgr.es/m/CAApHDvrWsfc3naVQZxS0efU%3DvJOA7dG3NV7fGhkgo2%3DJ38OEpg%40mail.gmail.com
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: Postgres 16.1 - Bug: cache entry already complete
2024-01-02 15:36 Postgres 16.1 - Bug: cache entry already complete Amadeo Gallardo <amadeo@ruddr.io>
2024-01-02 17:31 ` Re: Postgres 16.1 - Bug: cache entry already complete Tom Lane <tgl@sss.pgh.pa.us>
2024-01-02 19:51 ` Re: Postgres 16.1 - Bug: cache entry already complete Amadeo Gallardo <amadeo@ruddr.io>
2024-01-02 23:48 ` Re: Postgres 16.1 - Bug: cache entry already complete Amadeo Gallardo <amadeo@ruddr.io>
2024-01-03 00:05 ` Re: Postgres 16.1 - Bug: cache entry already complete Tom Lane <tgl@sss.pgh.pa.us>
2024-01-03 00:40 ` Re: Postgres 16.1 - Bug: cache entry already complete David Rowley <dgrowleyml@gmail.com>
2024-01-03 01:47 ` Re: Postgres 16.1 - Bug: cache entry already complete David Rowley <dgrowleyml@gmail.com>
2024-01-03 11:01 ` Re: Postgres 16.1 - Bug: cache entry already complete Richard Guo <guofenglinux@gmail.com>
2024-01-04 12:04 ` Re: Postgres 16.1 - Bug: cache entry already complete David Rowley <dgrowleyml@gmail.com>
@ 2024-01-11 03:26 ` Richard Guo <guofenglinux@gmail.com>
0 siblings, 0 replies; 14+ messages in thread
From: Richard Guo @ 2024-01-11 03:26 UTC (permalink / raw)
To: David Rowley <dgrowleyml@gmail.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Amadeo Gallardo <amadeo@ruddr.io>; pgsql-bugs@lists.postgresql.org
On Thu, Jan 4, 2024 at 8:04 PM David Rowley <dgrowleyml@gmail.com> wrote:
> On Thu, 4 Jan 2024 at 00:01, Richard Guo <guofenglinux@gmail.com> wrote:
> > if (extra->inner_unique &&
> > (inner_path->param_info == NULL ||
> > list_length(inner_path->param_info->ppi_clauses) <
> > list_length(extra->restrictlist)))
> > return NULL;
>
> I see. So it worked fine until Tom's " Make Vars be outer-join-aware."
> stuff.
>
> 9e215378d fixed this already and looks like I came to the same
> conclusion about the cache completion issue back then too.
Exactly.
> > But why does the code above fail to capture the problematic query? It
> > turns out that inner_path->param_info->ppi_clauses contains cloned
> > clauses, i.e. multiple versions of the same clause in order to cope with
> > outer join identity 3. And this causes the above code to give incorrect
> > conclusions about the length of 'ppi_clauses'. And this also explains
> > why this issue only occurs in v16.
> >
> > As a clause's serial number is unique within the current PlannerInfo
> > context, and multiple clones of the same clause have the same serial
> > number, it seems to me that it's more correct to calculate the length of
> > ppi_clauses by:
> >
> > bms_num_members(inner_path->param_info->ppi_serials)
> >
> > So I think maybe we can fix this issue with the attached patch.
>
> hmm, I'd need to study what Tom's been changing around this. I don't
> have a good handle on which qual lists can have duplicated quals.
>
> In [1], you can see I took the inspiration for that fix from
> generate_mergejoin_paths(). I don't yet know if it's just ppi_clauses
> that can have duplicates or if there's a similar hazard in the
> list_length() checks in generate_mergejoin_paths()
AFAIU we may have clone clauses in RelOptInfo.joininfo to cope with
commuted-left-join cases per identity 3. When we build ParamPathInfo
for parameterized paths we may move join clauses to ppi_clauses when
possible, so ParamPathInfo.ppi_clauses may also have clone clauses.
When we construct restrict list for a join relation from the joininfo
lists of the relations it joins, only one of the clone clauses would be
chosen to be put in the restrict list if they are restriction clauses,
thanks to the check against required_relids and incompatible_relids.
if (rinfo->has_clone || rinfo->is_clone)
{
Assert(!RINFO_IS_PUSHED_DOWN(rinfo, joinrel->relids));
if (!bms_is_subset(rinfo->required_relids, both_input_relids))
continue;
if (bms_overlap(rinfo->incompatible_relids, both_input_relids))
continue;
}
The mergeclause_list examined in generate_mergejoin_paths() is extracted
from joinrel's restrictlist, so it will not have multiple versions for
one clause. Therefore, I think we're good with the list_length() checks
in generate_mergejoin_paths().
> > BTW, the SingleRow in explain seems quite handy. I wonder if we can
> > keep it.
>
> If the fix is to just disable Memoize when the ppi_clauses don't cover
> the entire join condition, then the Inner Unique from the parent
> nested loop tells us this. Isn't that enough? The reason I added
> SingleRow was because in the patch I posted, I'd made it so SingleRow
> could be false in cases where Inner Unique was true.
Ah yes, you're right.
Thanks
Richard
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: Postgres 16.1 - Bug: cache entry already complete
2024-01-02 15:36 Postgres 16.1 - Bug: cache entry already complete Amadeo Gallardo <amadeo@ruddr.io>
2024-01-02 17:31 ` Re: Postgres 16.1 - Bug: cache entry already complete Tom Lane <tgl@sss.pgh.pa.us>
2024-01-02 19:51 ` Re: Postgres 16.1 - Bug: cache entry already complete Amadeo Gallardo <amadeo@ruddr.io>
2024-01-02 23:48 ` Re: Postgres 16.1 - Bug: cache entry already complete Amadeo Gallardo <amadeo@ruddr.io>
2024-01-03 00:05 ` Re: Postgres 16.1 - Bug: cache entry already complete Tom Lane <tgl@sss.pgh.pa.us>
2024-01-03 00:40 ` Re: Postgres 16.1 - Bug: cache entry already complete David Rowley <dgrowleyml@gmail.com>
2024-01-03 01:47 ` Re: Postgres 16.1 - Bug: cache entry already complete David Rowley <dgrowleyml@gmail.com>
2024-01-03 11:01 ` Re: Postgres 16.1 - Bug: cache entry already complete Richard Guo <guofenglinux@gmail.com>
@ 2024-01-22 09:49 ` David Rowley <dgrowleyml@gmail.com>
1 sibling, 0 replies; 14+ messages in thread
From: David Rowley @ 2024-01-22 09:49 UTC (permalink / raw)
To: Richard Guo <guofenglinux@gmail.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Amadeo Gallardo <amadeo@ruddr.io>; pgsql-bugs@lists.postgresql.org
On Thu, 4 Jan 2024 at 00:01, Richard Guo <guofenglinux@gmail.com> wrote:
> As a clause's serial number is unique within the current PlannerInfo
> context, and multiple clones of the same clause have the same serial
> number, it seems to me that it's more correct to calculate the length of
> ppi_clauses by:
>
> bms_num_members(inner_path->param_info->ppi_serials)
>
> So I think maybe we can fix this issue with the attached patch.
Thanks for the patch. I looked at this again and believe the
bms_num_member() check is a good fix.
I considered if Amadeo's case is worth including as a test and I
concluded that having that as a test won't necessarily guard us
against getting this error again if there are still bugs in this area.
David
^ permalink raw reply [nested|flat] 14+ messages in thread
end of thread, other threads:[~2024-01-22 09:49 UTC | newest]
Thread overview: 14+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2024-01-02 15:36 Postgres 16.1 - Bug: cache entry already complete Amadeo Gallardo <amadeo@ruddr.io>
2024-01-02 17:31 ` Tom Lane <tgl@sss.pgh.pa.us>
2024-01-02 19:51 ` Amadeo Gallardo <amadeo@ruddr.io>
2024-01-02 23:48 ` Amadeo Gallardo <amadeo@ruddr.io>
2024-01-02 23:59 ` David Rowley <dgrowleyml@gmail.com>
2024-01-03 02:36 ` Richard Guo <guofenglinux@gmail.com>
2024-01-03 03:05 ` Tom Lane <tgl@sss.pgh.pa.us>
2024-01-03 00:05 ` Tom Lane <tgl@sss.pgh.pa.us>
2024-01-03 00:40 ` David Rowley <dgrowleyml@gmail.com>
2024-01-03 01:47 ` David Rowley <dgrowleyml@gmail.com>
2024-01-03 11:01 ` Richard Guo <guofenglinux@gmail.com>
2024-01-04 12:04 ` David Rowley <dgrowleyml@gmail.com>
2024-01-11 03:26 ` Richard Guo <guofenglinux@gmail.com>
2024-01-22 09:49 ` David Rowley <dgrowleyml@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox