public inbox for [email protected]
help / color / mirror / Atom feedRe: [PATCH] Log PostgreSQL version number on startup
24+ messages / 6 participants
[nested] [flat]
* Re: [PATCH] Log PostgreSQL version number on startup
@ 2019-01-05 14:53 Peter Eisentraut <[email protected]>
0 siblings, 1 reply; 24+ messages in thread
From: Peter Eisentraut @ 2019-01-05 14:53 UTC (permalink / raw)
To: Christoph Berg <[email protected]>; pgsql-hackers
On 21/11/2018 15:46, Christoph Berg wrote:
> A startup looks like this:
>
> 2018-11-21 15:19:47.259 CET [24453] LOG: listening on IPv6 address "::1", port 5431
> 2018-11-21 15:19:47.259 CET [24453] LOG: listening on IPv4 address "127.0.0.1", port 5431
> 2018-11-21 15:19:47.315 CET [24453] LOG: listening on Unix socket "/tmp/.s.PGSQL.5431"
> 2018-11-21 15:19:47.394 CET [24453] LOG: starting PostgreSQL 12devel on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.2.0-9) 8.2.0, 64-bit
> 2018-11-21 15:19:47.426 CET [24454] LOG: database system was shut down at 2018-11-21 15:15:35 CET
> 2018-11-21 15:19:47.460 CET [24453] LOG: database system is ready to accept connections
>
> (I'd rather put the start message before the listening messages, but I
> think the startup message should be logged via logging_collector, and
> listening is logged before the log file is opened.)
Why don't we start the logging collector before opening the sockets?
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 24+ messages in thread
* Re: [PATCH] Log PostgreSQL version number on startup
@ 2019-01-16 19:58 Peter Eisentraut <[email protected]>
parent: Peter Eisentraut <[email protected]>
0 siblings, 1 reply; 24+ messages in thread
From: Peter Eisentraut @ 2019-01-16 19:58 UTC (permalink / raw)
To: Christoph Berg <[email protected]>; pgsql-hackers
On 05/01/2019 15:53, Peter Eisentraut wrote:
> On 21/11/2018 15:46, Christoph Berg wrote:
>> A startup looks like this:
>>
>> 2018-11-21 15:19:47.259 CET [24453] LOG: listening on IPv6 address "::1", port 5431
>> 2018-11-21 15:19:47.259 CET [24453] LOG: listening on IPv4 address "127.0.0.1", port 5431
>> 2018-11-21 15:19:47.315 CET [24453] LOG: listening on Unix socket "/tmp/.s.PGSQL.5431"
>> 2018-11-21 15:19:47.394 CET [24453] LOG: starting PostgreSQL 12devel on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.2.0-9) 8.2.0, 64-bit
>> 2018-11-21 15:19:47.426 CET [24454] LOG: database system was shut down at 2018-11-21 15:15:35 CET
>> 2018-11-21 15:19:47.460 CET [24453] LOG: database system is ready to accept connections
>>
>> (I'd rather put the start message before the listening messages, but I
>> think the startup message should be logged via logging_collector, and
>> listening is logged before the log file is opened.)
>
> Why don't we start the logging collector before opening the sockets?
Specifically, something like the attached.
This keeps the dynamic module loading before the logging collector
start, so we see those error messages on stderr, but then the setting up
of the sockets would get logged.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 24+ messages in thread
* Re: [PATCH] Log PostgreSQL version number on startup
@ 2019-01-29 15:46 Christoph Berg <[email protected]>
parent: Peter Eisentraut <[email protected]>
0 siblings, 1 reply; 24+ messages in thread
From: Christoph Berg @ 2019-01-29 15:46 UTC (permalink / raw)
To: Peter Eisentraut <[email protected]>; +Cc: pgsql-hackers
Re: Peter Eisentraut 2019-01-16 <[email protected]>
> > Why don't we start the logging collector before opening the sockets?
>
> Specifically, something like the attached.
>
> This keeps the dynamic module loading before the logging collector
> start, so we see those error messages on stderr, but then the setting up
> of the sockets would get logged.
This works nicely, so +1.
I'm attaching your patch as 0001 and my rebased one on top of it as
0002.
Christoph
--
Senior Berater, Tel.: +49 2166 9901 187
credativ GmbH, HRB Mönchengladbach 12080, USt-ID-Nummer: DE204566209
Trompeterallee 108, 41189 Mönchengladbach
Geschäftsführung: Dr. Michael Meskes, Jörg Folz, Sascha Heuer
Unser Umgang mit personenbezogenen Daten unterliegt
folgenden Bestimmungen: https://www.credativ.de/datenschutz
^ permalink raw reply [nested|flat] 24+ messages in thread
* Re: [PATCH] Log PostgreSQL version number on startup
@ 2019-01-30 22:29 Peter Eisentraut <[email protected]>
parent: Christoph Berg <[email protected]>
0 siblings, 0 replies; 24+ messages in thread
From: Peter Eisentraut @ 2019-01-30 22:29 UTC (permalink / raw)
To: Christoph Berg <[email protected]>; pgsql-hackers
On 29/01/2019 16:46, Christoph Berg wrote:
> Re: Peter Eisentraut 2019-01-16 <[email protected]>
>>> Why don't we start the logging collector before opening the sockets?
>>
>> Specifically, something like the attached.
>>
>> This keeps the dynamic module loading before the logging collector
>> start, so we see those error messages on stderr, but then the setting up
>> of the sockets would get logged.
>
> This works nicely, so +1.
>
> I'm attaching your patch as 0001 and my rebased one on top of it as
> 0002.
committed
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 24+ messages in thread
* [PATCH] ArchiveOpts structure
@ 2019-02-01 11:16 Alvaro Herrera <[email protected]>
0 siblings, 0 replies; 24+ messages in thread
From: Alvaro Herrera @ 2019-02-01 11:16 UTC (permalink / raw)
ArchiveEntry function has some number of arguments, that can be
considered as optional. Refactor out arguments for ArchiveEntry into
ArchiveOpts structure, to make it more flexible for changes.
Author: Dmitry Dolgov
Discussion: https://postgr.es/m/CA+q6zcXRxPE+qp6oerQWJ3zS061WPOhdxeMrdc-Yf-2V5vsrEw@mail.gmail.com
---
src/bin/pg_dump/pg_backup_archiver.c | 113 ++---
src/bin/pg_dump/pg_backup_archiver.h | 31 +-
src/bin/pg_dump/pg_dump.c | 952 +++++++++++++++++------------------
3 files changed, 521 insertions(+), 575 deletions(-)
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 2c2f6fb4a9..a3e530bac7 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -77,7 +77,7 @@ static ArchiveHandle *_allocAH(const char *FileSpec, const ArchiveFormat fmt,
static void _getObjectDescription(PQExpBuffer buf, TocEntry *te,
ArchiveHandle *AH);
static void _printTocEntry(ArchiveHandle *AH, TocEntry *te, bool isData);
-static char *replace_line_endings(const char *str);
+static char *sanitize_line(const char *str, bool want_hyphen);
static void _doSetFixedOutputState(ArchiveHandle *AH);
static void _doSetSessionAuth(ArchiveHandle *AH, const char *user);
static void _reconnectToDB(ArchiveHandle *AH, const char *dbname);
@@ -1066,17 +1066,8 @@ WriteData(Archive *AHX, const void *data, size_t dLen)
/* Public */
TocEntry *
-ArchiveEntry(Archive *AHX,
- CatalogId catalogId, DumpId dumpId,
- const char *tag,
- const char *namespace,
- const char *tablespace,
- const char *owner,
- const char *desc, teSection section,
- const char *defn,
- const char *dropStmt, const char *copyStmt,
- const DumpId *deps, int nDeps,
- DataDumperPtr dumpFn, void *dumpArg)
+ArchiveEntry(Archive *AHX, CatalogId catalogId,
+ DumpId dumpId, ArchiveOpts *opts)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
TocEntry *newToc;
@@ -1094,22 +1085,22 @@ ArchiveEntry(Archive *AHX,
newToc->catalogId = catalogId;
newToc->dumpId = dumpId;
- newToc->section = section;
+ newToc->section = opts->section;
- newToc->tag = pg_strdup(tag);
- newToc->namespace = namespace ? pg_strdup(namespace) : NULL;
- newToc->tablespace = tablespace ? pg_strdup(tablespace) : NULL;
- newToc->owner = pg_strdup(owner);
- newToc->desc = pg_strdup(desc);
- newToc->defn = pg_strdup(defn);
- newToc->dropStmt = pg_strdup(dropStmt);
- newToc->copyStmt = copyStmt ? pg_strdup(copyStmt) : NULL;
+ newToc->tag = pg_strdup(opts->tag);
+ newToc->namespace = opts->namespace ? pg_strdup(opts->namespace) : NULL;
+ newToc->tablespace = opts->tablespace ? pg_strdup(opts->tablespace) : NULL;
+ newToc->owner = opts->owner ? pg_strdup(opts->owner) : NULL;
+ newToc->desc = pg_strdup(opts->description);
+ newToc->defn = opts->createStmt ? pg_strdup(opts->createStmt) : NULL;
+ newToc->dropStmt = opts->dropStmt ? pg_strdup(opts->dropStmt) : NULL;
+ newToc->copyStmt = opts->copyStmt ? pg_strdup(opts->copyStmt) : NULL;
- if (nDeps > 0)
+ if (opts->nDeps > 0)
{
- newToc->dependencies = (DumpId *) pg_malloc(nDeps * sizeof(DumpId));
- memcpy(newToc->dependencies, deps, nDeps * sizeof(DumpId));
- newToc->nDeps = nDeps;
+ newToc->dependencies = (DumpId *) pg_malloc(opts->nDeps * sizeof(DumpId));
+ memcpy(newToc->dependencies, opts->deps, opts->nDeps * sizeof(DumpId));
+ newToc->nDeps = opts->nDeps;
}
else
{
@@ -1117,9 +1108,9 @@ ArchiveEntry(Archive *AHX,
newToc->nDeps = 0;
}
- newToc->dataDumper = dumpFn;
- newToc->dataDumperArg = dumpArg;
- newToc->hadDumper = dumpFn ? true : false;
+ newToc->dataDumper = opts->dumpFn;
+ newToc->dataDumperArg = opts->dumpArg;
+ newToc->hadDumper = opts->dumpFn ? true : false;
newToc->formatData = NULL;
newToc->dataLength = 0;
@@ -1152,7 +1143,7 @@ PrintTOCSummary(Archive *AHX)
ahprintf(AH, ";\n; Archive created at %s\n", stamp_str);
ahprintf(AH, "; dbname: %s\n; TOC Entries: %d\n; Compression: %d\n",
- replace_line_endings(AH->archdbname),
+ sanitize_line(AH->archdbname, false),
AH->tocCount, AH->compression);
switch (AH->format)
@@ -1197,21 +1188,10 @@ PrintTOCSummary(Archive *AHX)
char *sanitized_owner;
/*
- * As in _printTocEntry(), sanitize strings that might contain
- * newlines, to ensure that each logical output line is in fact
- * one physical output line. This prevents confusion when the
- * file is read by "pg_restore -L". Note that we currently don't
- * bother to quote names, meaning that the name fields aren't
- * automatically parseable. "pg_restore -L" doesn't care because
- * it only examines the dumpId field, but someday we might want to
- * try harder.
*/
- sanitized_name = replace_line_endings(te->tag);
- if (te->namespace)
- sanitized_schema = replace_line_endings(te->namespace);
- else
- sanitized_schema = pg_strdup("-");
- sanitized_owner = replace_line_endings(te->owner);
+ sanitized_name = sanitize_line(te->tag, false);
+ sanitized_schema = sanitize_line(te->namespace, true);
+ sanitized_owner = sanitize_line(te->owner, false);
ahprintf(AH, "%d; %u %u %s %s %s %s\n", te->dumpId,
te->catalogId.tableoid, te->catalogId.oid,
@@ -3577,21 +3557,9 @@ _printTocEntry(ArchiveHandle *AH, TocEntry *te, bool isData)
}
}
- /*
- * Zap any line endings embedded in user-supplied fields, to prevent
- * corruption of the dump (which could, in the worst case, present an
- * SQL injection vulnerability if someone were to incautiously load a
- * dump containing objects with maliciously crafted names).
- */
- sanitized_name = replace_line_endings(te->tag);
- if (te->namespace)
- sanitized_schema = replace_line_endings(te->namespace);
- else
- sanitized_schema = pg_strdup("-");
- if (!ropt->noOwner)
- sanitized_owner = replace_line_endings(te->owner);
- else
- sanitized_owner = pg_strdup("-");
+ sanitized_name = sanitize_line(te->tag, false);
+ sanitized_schema = sanitize_line(te->namespace, true);
+ sanitized_owner = sanitize_line(ropt->noOwner ? NULL : te->owner, true);
ahprintf(AH, "-- %sName: %s; Type: %s; Schema: %s; Owner: %s",
pfx, sanitized_name, te->desc, sanitized_schema,
@@ -3605,7 +3573,7 @@ _printTocEntry(ArchiveHandle *AH, TocEntry *te, bool isData)
{
char *sanitized_tablespace;
- sanitized_tablespace = replace_line_endings(te->tablespace);
+ sanitized_tablespace = sanitize_line(te->tablespace, false);
ahprintf(AH, "; Tablespace: %s", sanitized_tablespace);
free(sanitized_tablespace);
}
@@ -3629,7 +3597,7 @@ _printTocEntry(ArchiveHandle *AH, TocEntry *te, bool isData)
}
else
{
- if (strlen(te->defn) > 0)
+ if (te->defn && strlen(te->defn) > 0)
ahprintf(AH, "%s\n\n", te->defn);
}
@@ -3640,7 +3608,8 @@ _printTocEntry(ArchiveHandle *AH, TocEntry *te, bool isData)
* with DROP commands must appear in one list or the other.
*/
if (!ropt->noOwner && !ropt->use_setsessauth &&
- strlen(te->owner) > 0 && strlen(te->dropStmt) > 0)
+ te->owner && strlen(te->owner) > 0 &&
+ te->dropStmt && strlen(te->dropStmt) > 0)
{
if (strcmp(te->desc, "AGGREGATE") == 0 ||
strcmp(te->desc, "BLOB") == 0 ||
@@ -3713,16 +3682,30 @@ _printTocEntry(ArchiveHandle *AH, TocEntry *te, bool isData)
}
/*
- * Sanitize a string to be included in an SQL comment or TOC listing,
- * by replacing any newlines with spaces.
- * The result is a freshly malloc'd string.
+ * Sanitize a string to be included in an SQL comment or TOC listing, by
+ * replacing any newlines with spaces. This ensures each logical output line
+ * is in fact one physical output line, to prevent corruption of the dump
+ * (which could, in the worst case, present an SQL injection vulnerability
+ * if someone were to incautiously load a dump containing objects with
+ * maliciously crafted names).
+ *
+ * The result is a freshly malloc'd string. If the input string is NULL,
+ * return a malloc'ed empty string, unless want_hyphen, in which case return a
+ * malloc'ed hyphen.
+ *
+ * Note that we currently don't bother to quote names, meaning that the name
+ * fields aren't automatically parseable. "pg_restore -L" doesn't care because
+ * it only examines the dumpId field, but someday we might want to try harder.
*/
static char *
-replace_line_endings(const char *str)
+sanitize_line(const char *str, bool want_hyphen)
{
char *result;
char *s;
+ if (!str)
+ return pg_strdup(want_hyphen ? "-" : "");
+
result = pg_strdup(str);
for (s = result; *s != '\0'; s++)
diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h
index 306d2ceba9..d10321be18 100644
--- a/src/bin/pg_dump/pg_backup_archiver.h
+++ b/src/bin/pg_dump/pg_backup_archiver.h
@@ -405,17 +405,28 @@ extern void on_exit_close_archive(Archive *AHX);
extern void warn_or_exit_horribly(ArchiveHandle *AH, const char *modulename, const char *fmt,...) pg_attribute_printf(3, 4);
+typedef struct
+{
+ const char *tag;
+ const char *namespace;
+ const char *tablespace;
+ const char *owner;
+ const char *description;
+ teSection section;
+ const char *createStmt;
+ const char *dropStmt;
+ const char *copyStmt;
+ const DumpId *deps;
+ int nDeps;
+ DataDumperPtr dumpFn;
+ void *dumpArg;
+} ArchiveOpts;
+
+#define ARCHIVE_OPTS(...) &(ArchiveOpts){__VA_ARGS__}
+
/* Called to add a TOC entry */
-extern TocEntry *ArchiveEntry(Archive *AHX,
- CatalogId catalogId, DumpId dumpId,
- const char *tag,
- const char *namespace, const char *tablespace,
- const char *owner,
- const char *desc, teSection section,
- const char *defn,
- const char *dropStmt, const char *copyStmt,
- const DumpId *deps, int nDeps,
- DataDumperPtr dumpFn, void *dumpArg);
+extern TocEntry *ArchiveEntry(Archive *AHX, CatalogId catalogId,
+ DumpId dumpId, ArchiveOpts *opts);
extern void WriteTOC(ArchiveHandle *AH);
extern void ReadTOC(ArchiveHandle *AH);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 2b1a94733b..247bb6eefa 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -2135,12 +2135,16 @@ dumpTableData(Archive *fout, TableDataInfo *tdinfo)
TocEntry *te;
te = ArchiveEntry(fout, tdinfo->dobj.catId, tdinfo->dobj.dumpId,
- tbinfo->dobj.name, tbinfo->dobj.namespace->dobj.name,
- NULL, tbinfo->rolname,
- "TABLE DATA", SECTION_DATA,
- "", "", copyStmt,
- &(tbinfo->dobj.dumpId), 1,
- dumpFn, tdinfo);
+ ARCHIVE_OPTS(.tag = tbinfo->dobj.name,
+ .namespace = tbinfo->dobj.namespace->dobj.name,
+ .owner = tbinfo->rolname,
+ .description = "TABLE DATA",
+ .section = SECTION_DATA,
+ .copyStmt = copyStmt,
+ .deps = &(tbinfo->dobj.dumpId),
+ .nDeps = 1,
+ .dumpFn = dumpFn,
+ .dumpArg = tdinfo));
/*
* Set the TocEntry's dataLength in case we are doing a parallel dump
@@ -2184,19 +2188,14 @@ refreshMatViewData(Archive *fout, TableDataInfo *tdinfo)
ArchiveEntry(fout,
tdinfo->dobj.catId, /* catalog ID */
tdinfo->dobj.dumpId, /* dump ID */
- tbinfo->dobj.name, /* Name */
- tbinfo->dobj.namespace->dobj.name, /* Namespace */
- NULL, /* Tablespace */
- tbinfo->rolname, /* Owner */
- "MATERIALIZED VIEW DATA", /* Desc */
- SECTION_POST_DATA, /* Section */
- q->data, /* Create */
- "", /* Del */
- NULL, /* Copy */
- tdinfo->dobj.dependencies, /* Deps */
- tdinfo->dobj.nDeps, /* # Deps */
- NULL, /* Dumper */
- NULL); /* Dumper Arg */
+ ARCHIVE_OPTS(.tag = tbinfo->dobj.name,
+ .namespace = tbinfo->dobj.namespace->dobj.name,
+ .owner = tbinfo->rolname,
+ .description = "MATERIALIZED VIEW DATA",
+ .section = SECTION_POST_DATA,
+ .createStmt = q->data,
+ .deps = tdinfo->dobj.dependencies,
+ .nDeps = tdinfo->dobj.nDeps));
destroyPQExpBuffer(q);
}
@@ -2722,19 +2721,12 @@ dumpDatabase(Archive *fout)
ArchiveEntry(fout,
dbCatId, /* catalog ID */
dbDumpId, /* dump ID */
- datname, /* Name */
- NULL, /* Namespace */
- NULL, /* Tablespace */
- dba, /* Owner */
- "DATABASE", /* Desc */
- SECTION_PRE_DATA, /* Section */
- creaQry->data, /* Create */
- delQry->data, /* Del */
- NULL, /* Copy */
- NULL, /* Deps */
- 0, /* # Deps */
- NULL, /* Dumper */
- NULL); /* Dumper Arg */
+ ARCHIVE_OPTS(.tag = datname,
+ .owner = dba,
+ .description = "DATABASE",
+ .section = SECTION_PRE_DATA,
+ .createStmt = creaQry->data,
+ .dropStmt = delQry->data));
/* Compute correct tag for archive entry */
appendPQExpBuffer(labelq, "DATABASE %s", qdatname);
@@ -2762,11 +2754,13 @@ dumpDatabase(Archive *fout)
appendPQExpBufferStr(dbQry, ";\n");
ArchiveEntry(fout, nilCatalogId, createDumpId(),
- labelq->data, NULL, NULL, dba,
- "COMMENT", SECTION_NONE,
- dbQry->data, "", NULL,
- &(dbDumpId), 1,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = labelq->data,
+ .owner = dba,
+ .description = "COMMENT",
+ .section = SECTION_NONE,
+ .createStmt = dbQry->data,
+ .deps = &(dbDumpId),
+ .nDeps = 1));
}
}
else
@@ -2789,11 +2783,13 @@ dumpDatabase(Archive *fout)
emitShSecLabels(conn, shres, seclabelQry, "DATABASE", datname);
if (seclabelQry->len > 0)
ArchiveEntry(fout, nilCatalogId, createDumpId(),
- labelq->data, NULL, NULL, dba,
- "SECURITY LABEL", SECTION_NONE,
- seclabelQry->data, "", NULL,
- &(dbDumpId), 1,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = labelq->data,
+ .owner = dba,
+ .description = "SECURITY LABEL",
+ .section = SECTION_NONE,
+ .createStmt = seclabelQry->data,
+ .deps = &(dbDumpId),
+ .nDeps = 1));
destroyPQExpBuffer(seclabelQry);
PQclear(shres);
}
@@ -2859,11 +2855,13 @@ dumpDatabase(Archive *fout)
if (creaQry->len > 0)
ArchiveEntry(fout, nilCatalogId, createDumpId(),
- datname, NULL, NULL, dba,
- "DATABASE PROPERTIES", SECTION_PRE_DATA,
- creaQry->data, delQry->data, NULL,
- &(dbDumpId), 1,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = datname,
+ .owner = dba,
+ .description = "DATABASE PROPERTIES",
+ .section = SECTION_PRE_DATA,
+ .createStmt = creaQry->data,
+ .dropStmt = delQry->data,
+ .deps = &dbDumpId));
/*
* pg_largeobject comes from the old system intact, so set its
@@ -2904,11 +2902,10 @@ dumpDatabase(Archive *fout)
atooid(PQgetvalue(lo_res, 0, i_relminmxid)),
LargeObjectRelationId);
ArchiveEntry(fout, nilCatalogId, createDumpId(),
- "pg_largeobject", NULL, NULL, "",
- "pg_largeobject", SECTION_PRE_DATA,
- loOutQry->data, "", NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = "pg_largeobject",
+ .description = "pg_largeobject",
+ .section = SECTION_PRE_DATA,
+ .createStmt = loOutQry->data));
PQclear(lo_res);
@@ -3014,11 +3011,10 @@ dumpEncoding(Archive *AH)
appendPQExpBufferStr(qry, ";\n");
ArchiveEntry(AH, nilCatalogId, createDumpId(),
- "ENCODING", NULL, NULL, "",
- "ENCODING", SECTION_PRE_DATA,
- qry->data, "", NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = "ENCODING",
+ .description = "ENCODING",
+ .section = SECTION_PRE_DATA,
+ .createStmt = qry->data));
destroyPQExpBuffer(qry);
}
@@ -3041,11 +3037,10 @@ dumpStdStrings(Archive *AH)
stdstrings);
ArchiveEntry(AH, nilCatalogId, createDumpId(),
- "STDSTRINGS", NULL, NULL, "",
- "STDSTRINGS", SECTION_PRE_DATA,
- qry->data, "", NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = "STDSTRINGS",
+ .description = "STDSTRINGS",
+ .section = SECTION_PRE_DATA,
+ .createStmt = qry->data));
destroyPQExpBuffer(qry);
}
@@ -3097,11 +3092,10 @@ dumpSearchPath(Archive *AH)
write_msg(NULL, "saving search_path = %s\n", path->data);
ArchiveEntry(AH, nilCatalogId, createDumpId(),
- "SEARCHPATH", NULL, NULL, "",
- "SEARCHPATH", SECTION_PRE_DATA,
- qry->data, "", NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = "SEARCHPATH",
+ .description = "SEARCHPATH",
+ .section = SECTION_PRE_DATA,
+ .createStmt = qry->data));
/* Also save it in AH->searchpath, in case we're doing plain text dump */
AH->searchpath = pg_strdup(qry->data);
@@ -3273,13 +3267,12 @@ dumpBlob(Archive *fout, BlobInfo *binfo)
if (binfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, binfo->dobj.catId, binfo->dobj.dumpId,
- binfo->dobj.name,
- NULL, NULL,
- binfo->rolname,
- "BLOB", SECTION_PRE_DATA,
- cquery->data, dquery->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = binfo->dobj.name,
+ .owner = binfo->rolname,
+ .description = "BLOB",
+ .section = SECTION_PRE_DATA,
+ .createStmt = cquery->data,
+ .dropStmt = dquery->data));
/* Dump comment if any */
if (binfo->dobj.dump & DUMP_COMPONENT_COMMENT)
@@ -3577,14 +3570,14 @@ dumpPolicy(Archive *fout, PolicyInfo *polinfo)
*/
if (polinfo->dobj.dump & DUMP_COMPONENT_POLICY)
ArchiveEntry(fout, polinfo->dobj.catId, polinfo->dobj.dumpId,
- polinfo->dobj.name,
- polinfo->dobj.namespace->dobj.name,
- NULL,
- tbinfo->rolname,
- "ROW SECURITY", SECTION_POST_DATA,
- query->data, "", NULL,
- &(tbinfo->dobj.dumpId), 1,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = polinfo->dobj.name,
+ .namespace = polinfo->dobj.namespace->dobj.name,
+ .owner = tbinfo->rolname,
+ .description = "ROW SECURITY",
+ .section = SECTION_POST_DATA,
+ .createStmt = query->data,
+ .deps = &(tbinfo->dobj.dumpId),
+ .nDeps = 1));
destroyPQExpBuffer(query);
return;
@@ -3633,14 +3626,13 @@ dumpPolicy(Archive *fout, PolicyInfo *polinfo)
if (polinfo->dobj.dump & DUMP_COMPONENT_POLICY)
ArchiveEntry(fout, polinfo->dobj.catId, polinfo->dobj.dumpId,
- tag,
- polinfo->dobj.namespace->dobj.name,
- NULL,
- tbinfo->rolname,
- "POLICY", SECTION_POST_DATA,
- query->data, delqry->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = tag,
+ .namespace = polinfo->dobj.namespace->dobj.name,
+ .owner = tbinfo->rolname,
+ .description = "POLICY",
+ .section = SECTION_POST_DATA,
+ .createStmt = query->data,
+ .dropStmt = delqry->data));
free(tag);
destroyPQExpBuffer(query);
@@ -3807,14 +3799,12 @@ dumpPublication(Archive *fout, PublicationInfo *pubinfo)
appendPQExpBufferStr(query, "');\n");
ArchiveEntry(fout, pubinfo->dobj.catId, pubinfo->dobj.dumpId,
- pubinfo->dobj.name,
- NULL,
- NULL,
- pubinfo->rolname,
- "PUBLICATION", SECTION_POST_DATA,
- query->data, delq->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = pubinfo->dobj.name,
+ .owner = pubinfo->rolname,
+ .description = "PUBLICATION",
+ .section = SECTION_POST_DATA,
+ .createStmt = query->data,
+ .dropStmt = delq->data));
if (pubinfo->dobj.dump & DUMP_COMPONENT_COMMENT)
dumpComment(fout, "PUBLICATION", qpubname,
@@ -3950,14 +3940,11 @@ dumpPublicationTable(Archive *fout, PublicationRelInfo *pubrinfo)
* done by table drop.
*/
ArchiveEntry(fout, pubrinfo->dobj.catId, pubrinfo->dobj.dumpId,
- tag,
- tbinfo->dobj.namespace->dobj.name,
- NULL,
- "",
- "PUBLICATION TABLE", SECTION_POST_DATA,
- query->data, "", NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = tag,
+ .namespace = tbinfo->dobj.namespace->dobj.name,
+ .description = "PUBLICATION TABLE",
+ .section = SECTION_POST_DATA,
+ .createStmt = query->data));
free(tag);
destroyPQExpBuffer(query);
@@ -4143,14 +4130,12 @@ dumpSubscription(Archive *fout, SubscriptionInfo *subinfo)
appendPQExpBufferStr(query, ");\n");
ArchiveEntry(fout, subinfo->dobj.catId, subinfo->dobj.dumpId,
- subinfo->dobj.name,
- NULL,
- NULL,
- subinfo->rolname,
- "SUBSCRIPTION", SECTION_POST_DATA,
- query->data, delq->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = subinfo->dobj.name,
+ .owner = subinfo->rolname,
+ .description = "SUBSCRIPTION",
+ .section = SECTION_POST_DATA,
+ .createStmt = query->data,
+ .dropStmt = delq->data));
if (subinfo->dobj.dump & DUMP_COMPONENT_COMMENT)
dumpComment(fout, "SUBSCRIPTION", qsubname,
@@ -9377,11 +9362,14 @@ dumpComment(Archive *fout, const char *type, const char *name,
* post-data.
*/
ArchiveEntry(fout, nilCatalogId, createDumpId(),
- tag->data, namespace, NULL, owner,
- "COMMENT", SECTION_NONE,
- query->data, "", NULL,
- &(dumpId), 1,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = tag->data,
+ .namespace = namespace,
+ .owner = owner,
+ .description = "COMMENT",
+ .section = SECTION_NONE,
+ .createStmt = query->data,
+ .deps = &dumpId,
+ .nDeps = 1));
destroyPQExpBuffer(query);
destroyPQExpBuffer(tag);
@@ -9443,13 +9431,14 @@ dumpTableComment(Archive *fout, TableInfo *tbinfo,
appendPQExpBufferStr(query, ";\n");
ArchiveEntry(fout, nilCatalogId, createDumpId(),
- tag->data,
- tbinfo->dobj.namespace->dobj.name,
- NULL, tbinfo->rolname,
- "COMMENT", SECTION_NONE,
- query->data, "", NULL,
- &(tbinfo->dobj.dumpId), 1,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = tag->data,
+ .namespace = tbinfo->dobj.namespace->dobj.name,
+ .owner = tbinfo->rolname,
+ .description = "COMMENT",
+ .section = SECTION_NONE,
+ .createStmt = query->data,
+ .deps = &(tbinfo->dobj.dumpId),
+ .nDeps = 1));
}
else if (objsubid > 0 && objsubid <= tbinfo->numatts)
{
@@ -9467,13 +9456,14 @@ dumpTableComment(Archive *fout, TableInfo *tbinfo,
appendPQExpBufferStr(query, ";\n");
ArchiveEntry(fout, nilCatalogId, createDumpId(),
- tag->data,
- tbinfo->dobj.namespace->dobj.name,
- NULL, tbinfo->rolname,
- "COMMENT", SECTION_NONE,
- query->data, "", NULL,
- &(tbinfo->dobj.dumpId), 1,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = tag->data,
+ .namespace = tbinfo->dobj.namespace->dobj.name,
+ .owner = tbinfo->rolname,
+ .description = "COMMENT",
+ .section = SECTION_NONE,
+ .createStmt = query->data,
+ .deps = &(tbinfo->dobj.dumpId),
+ .nDeps = 1));
}
comments++;
@@ -9750,11 +9740,10 @@ dumpDumpableObject(Archive *fout, DumpableObject *dobj)
TocEntry *te;
te = ArchiveEntry(fout, dobj->catId, dobj->dumpId,
- dobj->name, NULL, NULL, "",
- "BLOBS", SECTION_DATA,
- "", "", NULL,
- NULL, 0,
- dumpBlobs, NULL);
+ ARCHIVE_OPTS(.tag = dobj->name,
+ .description = "BLOBS",
+ .section = SECTION_DATA,
+ .dumpFn = dumpBlobs));
/*
* Set the TocEntry's dataLength in case we are doing a
@@ -9822,13 +9811,12 @@ dumpNamespace(Archive *fout, NamespaceInfo *nspinfo)
if (nspinfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, nspinfo->dobj.catId, nspinfo->dobj.dumpId,
- nspinfo->dobj.name,
- NULL, NULL,
- nspinfo->rolname,
- "SCHEMA", SECTION_PRE_DATA,
- q->data, delq->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = nspinfo->dobj.name,
+ .owner = nspinfo->rolname,
+ .description = "SCHEMA",
+ .section = SECTION_PRE_DATA,
+ .createStmt = q->data,
+ .dropStmt = delq->data));
/* Dump Schema Comments and Security Labels */
if (nspinfo->dobj.dump & DUMP_COMPONENT_COMMENT)
@@ -9958,13 +9946,11 @@ dumpExtension(Archive *fout, ExtensionInfo *extinfo)
if (extinfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, extinfo->dobj.catId, extinfo->dobj.dumpId,
- extinfo->dobj.name,
- NULL, NULL,
- "",
- "EXTENSION", SECTION_PRE_DATA,
- q->data, delq->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = extinfo->dobj.name,
+ .description = "EXTENSION",
+ .section = SECTION_PRE_DATA,
+ .createStmt = q->data,
+ .dropStmt = delq->data));
/* Dump Extension Comments and Security Labels */
if (extinfo->dobj.dump & DUMP_COMPONENT_COMMENT)
@@ -10108,14 +10094,13 @@ dumpEnumType(Archive *fout, TypeInfo *tyinfo)
if (tyinfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, tyinfo->dobj.catId, tyinfo->dobj.dumpId,
- tyinfo->dobj.name,
- tyinfo->dobj.namespace->dobj.name,
- NULL,
- tyinfo->rolname,
- "TYPE", SECTION_PRE_DATA,
- q->data, delq->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = tyinfo->dobj.name,
+ .namespace = tyinfo->dobj.namespace->dobj.name,
+ .owner = tyinfo->rolname,
+ .description = "TYPE",
+ .section = SECTION_PRE_DATA,
+ .createStmt = q->data,
+ .dropStmt = delq->data));
/* Dump Type Comments and Security Labels */
if (tyinfo->dobj.dump & DUMP_COMPONENT_COMMENT)
@@ -10235,14 +10220,13 @@ dumpRangeType(Archive *fout, TypeInfo *tyinfo)
if (tyinfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, tyinfo->dobj.catId, tyinfo->dobj.dumpId,
- tyinfo->dobj.name,
- tyinfo->dobj.namespace->dobj.name,
- NULL,
- tyinfo->rolname,
- "TYPE", SECTION_PRE_DATA,
- q->data, delq->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = tyinfo->dobj.name,
+ .namespace = tyinfo->dobj.namespace->dobj.name,
+ .owner = tyinfo->rolname,
+ .description = "TYPE",
+ .section = SECTION_PRE_DATA,
+ .createStmt = q->data,
+ .dropStmt = delq->data));
/* Dump Type Comments and Security Labels */
if (tyinfo->dobj.dump & DUMP_COMPONENT_COMMENT)
@@ -10308,14 +10292,13 @@ dumpUndefinedType(Archive *fout, TypeInfo *tyinfo)
if (tyinfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, tyinfo->dobj.catId, tyinfo->dobj.dumpId,
- tyinfo->dobj.name,
- tyinfo->dobj.namespace->dobj.name,
- NULL,
- tyinfo->rolname,
- "TYPE", SECTION_PRE_DATA,
- q->data, delq->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = tyinfo->dobj.name,
+ .namespace = tyinfo->dobj.namespace->dobj.name,
+ .owner = tyinfo->rolname,
+ .description = "TYPE",
+ .section = SECTION_PRE_DATA,
+ .createStmt = q->data,
+ .dropStmt = delq->data));
/* Dump Type Comments and Security Labels */
if (tyinfo->dobj.dump & DUMP_COMPONENT_COMMENT)
@@ -10590,14 +10573,13 @@ dumpBaseType(Archive *fout, TypeInfo *tyinfo)
if (tyinfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, tyinfo->dobj.catId, tyinfo->dobj.dumpId,
- tyinfo->dobj.name,
- tyinfo->dobj.namespace->dobj.name,
- NULL,
- tyinfo->rolname,
- "TYPE", SECTION_PRE_DATA,
- q->data, delq->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = tyinfo->dobj.name,
+ .namespace = tyinfo->dobj.namespace->dobj.name,
+ .owner = tyinfo->rolname,
+ .description = "TYPE",
+ .section = SECTION_PRE_DATA,
+ .createStmt = q->data,
+ .dropStmt = delq->data));
/* Dump Type Comments and Security Labels */
if (tyinfo->dobj.dump & DUMP_COMPONENT_COMMENT)
@@ -10747,14 +10729,13 @@ dumpDomain(Archive *fout, TypeInfo *tyinfo)
if (tyinfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, tyinfo->dobj.catId, tyinfo->dobj.dumpId,
- tyinfo->dobj.name,
- tyinfo->dobj.namespace->dobj.name,
- NULL,
- tyinfo->rolname,
- "DOMAIN", SECTION_PRE_DATA,
- q->data, delq->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = tyinfo->dobj.name,
+ .namespace = tyinfo->dobj.namespace->dobj.name,
+ .owner = tyinfo->rolname,
+ .description = "DOMAIN",
+ .section = SECTION_PRE_DATA,
+ .createStmt = q->data,
+ .dropStmt = delq->data));
/* Dump Domain Comments and Security Labels */
if (tyinfo->dobj.dump & DUMP_COMPONENT_COMMENT)
@@ -10969,14 +10950,13 @@ dumpCompositeType(Archive *fout, TypeInfo *tyinfo)
if (tyinfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, tyinfo->dobj.catId, tyinfo->dobj.dumpId,
- tyinfo->dobj.name,
- tyinfo->dobj.namespace->dobj.name,
- NULL,
- tyinfo->rolname,
- "TYPE", SECTION_PRE_DATA,
- q->data, delq->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = tyinfo->dobj.name,
+ .namespace = tyinfo->dobj.namespace->dobj.name,
+ .owner = tyinfo->rolname,
+ .description = "TYPE",
+ .section = SECTION_PRE_DATA,
+ .createStmt = q->data,
+ .dropStmt = delq->data));
/* Dump Type Comments and Security Labels */
@@ -11105,13 +11085,14 @@ dumpCompositeTypeColComments(Archive *fout, TypeInfo *tyinfo)
appendPQExpBufferStr(query, ";\n");
ArchiveEntry(fout, nilCatalogId, createDumpId(),
- target->data,
- tyinfo->dobj.namespace->dobj.name,
- NULL, tyinfo->rolname,
- "COMMENT", SECTION_NONE,
- query->data, "", NULL,
- &(tyinfo->dobj.dumpId), 1,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = target->data,
+ .namespace = tyinfo->dobj.namespace->dobj.name,
+ .owner = tyinfo->rolname,
+ .description = "COMMENT",
+ .section = SECTION_NONE,
+ .createStmt = query->data,
+ .deps = &(tyinfo->dobj.dumpId),
+ .nDeps = 1));
}
comments++;
@@ -11160,14 +11141,12 @@ dumpShellType(Archive *fout, ShellTypeInfo *stinfo)
if (stinfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, stinfo->dobj.catId, stinfo->dobj.dumpId,
- stinfo->dobj.name,
- stinfo->dobj.namespace->dobj.name,
- NULL,
- stinfo->baseType->rolname,
- "SHELL TYPE", SECTION_PRE_DATA,
- q->data, "", NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = stinfo->dobj.name,
+ .namespace = stinfo->dobj.namespace->dobj.name,
+ .owner = stinfo->baseType->rolname,
+ .description = "SHELL TYPE",
+ .section = SECTION_PRE_DATA,
+ .createStmt = q->data));
destroyPQExpBuffer(q);
}
@@ -11272,12 +11251,13 @@ dumpProcLang(Archive *fout, ProcLangInfo *plang)
if (plang->dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, plang->dobj.catId, plang->dobj.dumpId,
- plang->dobj.name,
- NULL, NULL, plang->lanowner,
- "PROCEDURAL LANGUAGE", SECTION_PRE_DATA,
- defqry->data, delqry->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = plang->dobj.name,
+ .owner = plang->lanowner,
+ .description = "PROCEDURAL LANGUAGE",
+ .section = SECTION_PRE_DATA,
+ .createStmt = defqry->data,
+ .dropStmt = delqry->data,
+ ));
/* Dump Proc Lang Comments and Security Labels */
if (plang->dobj.dump & DUMP_COMPONENT_COMMENT)
@@ -11942,14 +11922,13 @@ dumpFunc(Archive *fout, FuncInfo *finfo)
if (finfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, finfo->dobj.catId, finfo->dobj.dumpId,
- funcsig_tag,
- finfo->dobj.namespace->dobj.name,
- NULL,
- finfo->rolname,
- keyword, SECTION_PRE_DATA,
- q->data, delqry->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = funcsig_tag,
+ .namespace = finfo->dobj.namespace->dobj.name,
+ .owner = finfo->rolname,
+ .description = keyword,
+ .section = SECTION_PRE_DATA,
+ .createStmt = q->data,
+ .dropStmt = delqry->data));
/* Dump Function Comments and Security Labels */
if (finfo->dobj.dump & DUMP_COMPONENT_COMMENT)
@@ -12077,12 +12056,11 @@ dumpCast(Archive *fout, CastInfo *cast)
if (cast->dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, cast->dobj.catId, cast->dobj.dumpId,
- labelq->data,
- NULL, NULL, "",
- "CAST", SECTION_PRE_DATA,
- defqry->data, delqry->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = labelq->data,
+ .description = "CAST",
+ .section = SECTION_PRE_DATA,
+ .createStmt = defqry->data,
+ .dropStmt = delqry->data));
/* Dump Cast Comments */
if (cast->dobj.dump & DUMP_COMPONENT_COMMENT)
@@ -12205,12 +12183,13 @@ dumpTransform(Archive *fout, TransformInfo *transform)
if (transform->dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, transform->dobj.catId, transform->dobj.dumpId,
- labelq->data,
- NULL, NULL, "",
- "TRANSFORM", SECTION_PRE_DATA,
- defqry->data, delqry->data, NULL,
- transform->dobj.dependencies, transform->dobj.nDeps,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = labelq->data,
+ .description = "TRANSFORM",
+ .section = SECTION_PRE_DATA,
+ .createStmt = defqry->data,
+ .dropStmt = delqry->data,
+ .deps = transform->dobj.dependencies,
+ .nDeps = transform->dobj.nDeps));
/* Dump Transform Comments */
if (transform->dobj.dump & DUMP_COMPONENT_COMMENT)
@@ -12418,14 +12397,13 @@ dumpOpr(Archive *fout, OprInfo *oprinfo)
if (oprinfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, oprinfo->dobj.catId, oprinfo->dobj.dumpId,
- oprinfo->dobj.name,
- oprinfo->dobj.namespace->dobj.name,
- NULL,
- oprinfo->rolname,
- "OPERATOR", SECTION_PRE_DATA,
- q->data, delq->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = oprinfo->dobj.name,
+ .namespace = oprinfo->dobj.namespace->dobj.name,
+ .owner = oprinfo->rolname,
+ .description = "OPERATOR",
+ .section = SECTION_PRE_DATA,
+ .createStmt = q->data,
+ .dropStmt = delq->data));
/* Dump Operator Comments */
if (oprinfo->dobj.dump & DUMP_COMPONENT_COMMENT)
@@ -12588,14 +12566,11 @@ dumpAccessMethod(Archive *fout, AccessMethodInfo *aminfo)
if (aminfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, aminfo->dobj.catId, aminfo->dobj.dumpId,
- aminfo->dobj.name,
- NULL,
- NULL,
- "",
- "ACCESS METHOD", SECTION_PRE_DATA,
- q->data, delq->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = aminfo->dobj.name,
+ .description = "ACCESS METHOD",
+ .section = SECTION_PRE_DATA,
+ .createStmt = q->data,
+ .dropStmt = delq->data));
/* Dump Access Method Comments */
if (aminfo->dobj.dump & DUMP_COMPONENT_COMMENT)
@@ -12954,14 +12929,13 @@ dumpOpclass(Archive *fout, OpclassInfo *opcinfo)
if (opcinfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, opcinfo->dobj.catId, opcinfo->dobj.dumpId,
- opcinfo->dobj.name,
- opcinfo->dobj.namespace->dobj.name,
- NULL,
- opcinfo->rolname,
- "OPERATOR CLASS", SECTION_PRE_DATA,
- q->data, delq->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = opcinfo->dobj.name,
+ .namespace = opcinfo->dobj.namespace->dobj.name,
+ .owner = opcinfo->rolname,
+ .description = "OPERATOR CLASS",
+ .section = SECTION_PRE_DATA,
+ .createStmt = q->data,
+ .dropStmt = delq->data));
/* Dump Operator Class Comments */
if (opcinfo->dobj.dump & DUMP_COMPONENT_COMMENT)
@@ -13221,14 +13195,13 @@ dumpOpfamily(Archive *fout, OpfamilyInfo *opfinfo)
if (opfinfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, opfinfo->dobj.catId, opfinfo->dobj.dumpId,
- opfinfo->dobj.name,
- opfinfo->dobj.namespace->dobj.name,
- NULL,
- opfinfo->rolname,
- "OPERATOR FAMILY", SECTION_PRE_DATA,
- q->data, delq->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = opfinfo->dobj.name,
+ .namespace = opfinfo->dobj.namespace->dobj.name,
+ .owner = opfinfo->rolname,
+ .description = "OPERATOR FAMILY",
+ .section = SECTION_PRE_DATA,
+ .createStmt = q->data,
+ .dropStmt = delq->data));
/* Dump Operator Family Comments */
if (opfinfo->dobj.dump & DUMP_COMPONENT_COMMENT)
@@ -13364,14 +13337,13 @@ dumpCollation(Archive *fout, CollInfo *collinfo)
if (collinfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, collinfo->dobj.catId, collinfo->dobj.dumpId,
- collinfo->dobj.name,
- collinfo->dobj.namespace->dobj.name,
- NULL,
- collinfo->rolname,
- "COLLATION", SECTION_PRE_DATA,
- q->data, delq->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = collinfo->dobj.name,
+ .namespace = collinfo->dobj.namespace->dobj.name,
+ .owner = collinfo->rolname,
+ .description = "COLLATION",
+ .section = SECTION_PRE_DATA,
+ .createStmt = q->data,
+ .dropStmt = delq->data));
/* Dump Collation Comments */
if (collinfo->dobj.dump & DUMP_COMPONENT_COMMENT)
@@ -13459,14 +13431,13 @@ dumpConversion(Archive *fout, ConvInfo *convinfo)
if (convinfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, convinfo->dobj.catId, convinfo->dobj.dumpId,
- convinfo->dobj.name,
- convinfo->dobj.namespace->dobj.name,
- NULL,
- convinfo->rolname,
- "CONVERSION", SECTION_PRE_DATA,
- q->data, delq->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = convinfo->dobj.name,
+ .namespace = convinfo->dobj.namespace->dobj.name,
+ .owner = convinfo->rolname,
+ .description = "CONVERSION",
+ .section = SECTION_PRE_DATA,
+ .createStmt = q->data,
+ .dropStmt = delq->data));
/* Dump Conversion Comments */
if (convinfo->dobj.dump & DUMP_COMPONENT_COMMENT)
@@ -13948,14 +13919,13 @@ dumpAgg(Archive *fout, AggInfo *agginfo)
if (agginfo->aggfn.dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, agginfo->aggfn.dobj.catId,
agginfo->aggfn.dobj.dumpId,
- aggsig_tag,
- agginfo->aggfn.dobj.namespace->dobj.name,
- NULL,
- agginfo->aggfn.rolname,
- "AGGREGATE", SECTION_PRE_DATA,
- q->data, delq->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = aggsig_tag,
+ .namespace = agginfo->aggfn.dobj.namespace->dobj.name,
+ .owner = agginfo->aggfn.rolname,
+ .description = "AGGREGATE",
+ .section = SECTION_PRE_DATA,
+ .createStmt = q->data,
+ .dropStmt = delq->data));
/* Dump Aggregate Comments */
if (agginfo->aggfn.dobj.dump & DUMP_COMPONENT_COMMENT)
@@ -14046,14 +14016,12 @@ dumpTSParser(Archive *fout, TSParserInfo *prsinfo)
if (prsinfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, prsinfo->dobj.catId, prsinfo->dobj.dumpId,
- prsinfo->dobj.name,
- prsinfo->dobj.namespace->dobj.name,
- NULL,
- "",
- "TEXT SEARCH PARSER", SECTION_PRE_DATA,
- q->data, delq->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = prsinfo->dobj.name,
+ .namespace = prsinfo->dobj.namespace->dobj.name,
+ .description = "TEXT SEARCH PARSER",
+ .section = SECTION_PRE_DATA,
+ .createStmt = q->data,
+ .dropStmt = delq->data));
/* Dump Parser Comments */
if (prsinfo->dobj.dump & DUMP_COMPONENT_COMMENT)
@@ -14126,14 +14094,13 @@ dumpTSDictionary(Archive *fout, TSDictInfo *dictinfo)
if (dictinfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, dictinfo->dobj.catId, dictinfo->dobj.dumpId,
- dictinfo->dobj.name,
- dictinfo->dobj.namespace->dobj.name,
- NULL,
- dictinfo->rolname,
- "TEXT SEARCH DICTIONARY", SECTION_PRE_DATA,
- q->data, delq->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = dictinfo->dobj.name,
+ .namespace = dictinfo->dobj.namespace->dobj.name,
+ .owner = dictinfo->rolname,
+ .description = "TEXT SEARCH DICTIONARY",
+ .section = SECTION_PRE_DATA,
+ .createStmt = q->data,
+ .dropStmt = delq->data));
/* Dump Dictionary Comments */
if (dictinfo->dobj.dump & DUMP_COMPONENT_COMMENT)
@@ -14187,14 +14154,12 @@ dumpTSTemplate(Archive *fout, TSTemplateInfo *tmplinfo)
if (tmplinfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, tmplinfo->dobj.catId, tmplinfo->dobj.dumpId,
- tmplinfo->dobj.name,
- tmplinfo->dobj.namespace->dobj.name,
- NULL,
- "",
- "TEXT SEARCH TEMPLATE", SECTION_PRE_DATA,
- q->data, delq->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = tmplinfo->dobj.name,
+ .namespace = tmplinfo->dobj.namespace->dobj.name,
+ .description = "TEXT SEARCH TEMPLATE",
+ .section = SECTION_PRE_DATA,
+ .createStmt = q->data,
+ .dropStmt = delq->data));
/* Dump Template Comments */
if (tmplinfo->dobj.dump & DUMP_COMPONENT_COMMENT)
@@ -14307,14 +14272,13 @@ dumpTSConfig(Archive *fout, TSConfigInfo *cfginfo)
if (cfginfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, cfginfo->dobj.catId, cfginfo->dobj.dumpId,
- cfginfo->dobj.name,
- cfginfo->dobj.namespace->dobj.name,
- NULL,
- cfginfo->rolname,
- "TEXT SEARCH CONFIGURATION", SECTION_PRE_DATA,
- q->data, delq->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = cfginfo->dobj.name,
+ .namespace = cfginfo->dobj.namespace->dobj.name,
+ .owner = cfginfo->rolname,
+ .description = "TEXT SEARCH CONFIGURATION",
+ .section = SECTION_PRE_DATA,
+ .createStmt = q->data,
+ .dropStmt = delq->data));
/* Dump Configuration Comments */
if (cfginfo->dobj.dump & DUMP_COMPONENT_COMMENT)
@@ -14373,14 +14337,12 @@ dumpForeignDataWrapper(Archive *fout, FdwInfo *fdwinfo)
if (fdwinfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, fdwinfo->dobj.catId, fdwinfo->dobj.dumpId,
- fdwinfo->dobj.name,
- NULL,
- NULL,
- fdwinfo->rolname,
- "FOREIGN DATA WRAPPER", SECTION_PRE_DATA,
- q->data, delq->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = fdwinfo->dobj.name,
+ .owner = fdwinfo->rolname,
+ .description = "FOREIGN DATA WRAPPER",
+ .section = SECTION_PRE_DATA,
+ .createStmt = q->data,
+ .dropStmt = delq->data));
/* Dump Foreign Data Wrapper Comments */
if (fdwinfo->dobj.dump & DUMP_COMPONENT_COMMENT)
@@ -14464,14 +14426,12 @@ dumpForeignServer(Archive *fout, ForeignServerInfo *srvinfo)
if (srvinfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, srvinfo->dobj.catId, srvinfo->dobj.dumpId,
- srvinfo->dobj.name,
- NULL,
- NULL,
- srvinfo->rolname,
- "SERVER", SECTION_PRE_DATA,
- q->data, delq->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = srvinfo->dobj.name,
+ .owner = srvinfo->rolname,
+ .description = "SERVER",
+ .section = SECTION_PRE_DATA,
+ .createStmt = q->data,
+ .dropStmt = delq->data));
/* Dump Foreign Server Comments */
if (srvinfo->dobj.dump & DUMP_COMPONENT_COMMENT)
@@ -14582,14 +14542,13 @@ dumpUserMappings(Archive *fout,
usename, servername);
ArchiveEntry(fout, nilCatalogId, createDumpId(),
- tag->data,
- namespace,
- NULL,
- owner,
- "USER MAPPING", SECTION_PRE_DATA,
- q->data, delq->data, NULL,
- &dumpId, 1,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = tag->data,
+ .namespace = namespace,
+ .owner = owner,
+ .description = "USER MAPPING",
+ .section = SECTION_PRE_DATA,
+ .createStmt = q->data,
+ .dropStmt = delq->data));
}
PQclear(res);
@@ -14661,14 +14620,13 @@ dumpDefaultACL(Archive *fout, DefaultACLInfo *daclinfo)
if (daclinfo->dobj.dump & DUMP_COMPONENT_ACL)
ArchiveEntry(fout, daclinfo->dobj.catId, daclinfo->dobj.dumpId,
- tag->data,
- daclinfo->dobj.namespace ? daclinfo->dobj.namespace->dobj.name : NULL,
- NULL,
- daclinfo->defaclrole,
- "DEFAULT ACL", SECTION_POST_DATA,
- q->data, "", NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = tag->data,
+ .namespace = daclinfo->dobj.namespace ?
+ daclinfo->dobj.namespace->dobj.name : NULL,
+ .owner = daclinfo->defaclrole,
+ .description = "DEFAULT ACL",
+ .section = SECTION_POST_DATA,
+ .createStmt = q->data));
destroyPQExpBuffer(tag);
destroyPQExpBuffer(q);
@@ -14760,13 +14718,14 @@ dumpACL(Archive *fout, CatalogId objCatId, DumpId objDumpId,
appendPQExpBuffer(tag, "%s %s", type, name);
ArchiveEntry(fout, nilCatalogId, createDumpId(),
- tag->data, nspname,
- NULL,
- owner ? owner : "",
- "ACL", SECTION_NONE,
- sql->data, "", NULL,
- &(objDumpId), 1,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = tag->data,
+ .namespace = nspname,
+ .owner = owner,
+ .description = "ACL",
+ .section = SECTION_NONE,
+ .createStmt = sql->data,
+ .deps = &(objDumpId),
+ .nDeps = 1));
destroyPQExpBuffer(tag);
}
@@ -14848,11 +14807,14 @@ dumpSecLabel(Archive *fout, const char *type, const char *name,
appendPQExpBuffer(tag, "%s %s", type, name);
ArchiveEntry(fout, nilCatalogId, createDumpId(),
- tag->data, namespace, NULL, owner,
- "SECURITY LABEL", SECTION_NONE,
- query->data, "", NULL,
- &(dumpId), 1,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = tag->data,
+ .namespace = namespace,
+ .owner = owner,
+ .description = "SECURITY LABEL",
+ .section = SECTION_NONE,
+ .createStmt = query->data,
+ .deps = &(dumpId),
+ .nDeps = 1));
destroyPQExpBuffer(tag);
}
@@ -14928,13 +14890,14 @@ dumpTableSecLabel(Archive *fout, TableInfo *tbinfo, const char *reltypename)
appendPQExpBuffer(target, "%s %s", reltypename,
fmtId(tbinfo->dobj.name));
ArchiveEntry(fout, nilCatalogId, createDumpId(),
- target->data,
- tbinfo->dobj.namespace->dobj.name,
- NULL, tbinfo->rolname,
- "SECURITY LABEL", SECTION_NONE,
- query->data, "", NULL,
- &(tbinfo->dobj.dumpId), 1,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = target->data,
+ .namespace = tbinfo->dobj.namespace->dobj.name,
+ .owner = tbinfo->rolname,
+ .description = "SECURITY LABEL",
+ .section = SECTION_NONE,
+ .createStmt = query->data,
+ .deps = &(tbinfo->dobj.dumpId),
+ .nDeps = 1));
}
destroyPQExpBuffer(query);
destroyPQExpBuffer(target);
@@ -16012,16 +15975,16 @@ dumpTableSchema(Archive *fout, TableInfo *tbinfo)
if (tbinfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, tbinfo->dobj.catId, tbinfo->dobj.dumpId,
- tbinfo->dobj.name,
- tbinfo->dobj.namespace->dobj.name,
- (tbinfo->relkind == RELKIND_VIEW) ? NULL : tbinfo->reltablespace,
- tbinfo->rolname,
- reltypename,
- tbinfo->postponed_def ?
- SECTION_POST_DATA : SECTION_PRE_DATA,
- q->data, delq->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = tbinfo->dobj.name,
+ .namespace = tbinfo->dobj.namespace->dobj.name,
+ .tablespace = (tbinfo->relkind == RELKIND_VIEW) ?
+ NULL : tbinfo->reltablespace,
+ .owner = tbinfo->rolname,
+ .description = reltypename,
+ .section = tbinfo->postponed_def ?
+ SECTION_POST_DATA : SECTION_PRE_DATA,
+ .createStmt = q->data,
+ .dropStmt = delq->data));
/* Dump Table Comments */
@@ -16092,14 +16055,13 @@ dumpAttrDef(Archive *fout, AttrDefInfo *adinfo)
if (adinfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, adinfo->dobj.catId, adinfo->dobj.dumpId,
- tag,
- tbinfo->dobj.namespace->dobj.name,
- NULL,
- tbinfo->rolname,
- "DEFAULT", SECTION_PRE_DATA,
- q->data, delq->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = tag,
+ .namespace = tbinfo->dobj.namespace->dobj.name,
+ .owner = tbinfo->rolname,
+ .description = "DEFAULT",
+ .section = SECTION_PRE_DATA,
+ .createStmt = q->data,
+ .dropStmt = delq->data));
free(tag);
destroyPQExpBuffer(q);
@@ -16241,14 +16203,14 @@ dumpIndex(Archive *fout, IndxInfo *indxinfo)
if (indxinfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, indxinfo->dobj.catId, indxinfo->dobj.dumpId,
- indxinfo->dobj.name,
- tbinfo->dobj.namespace->dobj.name,
- indxinfo->tablespace,
- tbinfo->rolname,
- "INDEX", SECTION_POST_DATA,
- q->data, delq->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = indxinfo->dobj.name,
+ .namespace = tbinfo->dobj.namespace->dobj.name,
+ .tablespace = indxinfo->tablespace,
+ .owner = tbinfo->rolname,
+ .description = "INDEX",
+ .section = SECTION_POST_DATA,
+ .createStmt = q->data,
+ .dropStmt = delq->data));
if (indstatcolsarray)
free(indstatcolsarray);
@@ -16290,14 +16252,11 @@ dumpIndexAttach(Archive *fout, IndexAttachInfo *attachinfo)
fmtQualifiedDumpable(attachinfo->partitionIdx));
ArchiveEntry(fout, attachinfo->dobj.catId, attachinfo->dobj.dumpId,
- attachinfo->dobj.name,
- attachinfo->dobj.namespace->dobj.name,
- NULL,
- "",
- "INDEX ATTACH", SECTION_POST_DATA,
- q->data, "", NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = attachinfo->dobj.name,
+ .namespace = attachinfo->dobj.namespace->dobj.name,
+ .description = "INDEX ATTACH",
+ .section = SECTION_POST_DATA,
+ .createStmt = q->data));
destroyPQExpBuffer(q);
}
@@ -16345,14 +16304,13 @@ dumpStatisticsExt(Archive *fout, StatsExtInfo *statsextinfo)
if (statsextinfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, statsextinfo->dobj.catId,
statsextinfo->dobj.dumpId,
- statsextinfo->dobj.name,
- statsextinfo->dobj.namespace->dobj.name,
- NULL,
- statsextinfo->rolname,
- "STATISTICS", SECTION_POST_DATA,
- q->data, delq->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = statsextinfo->dobj.name,
+ .namespace = statsextinfo->dobj.namespace->dobj.name,
+ .owner = statsextinfo->rolname,
+ .description = "STATISTICS",
+ .section = SECTION_POST_DATA,
+ .createStmt = q->data,
+ .dropStmt = delq->data));
/* Dump Statistics Comments */
if (statsextinfo->dobj.dump & DUMP_COMPONENT_COMMENT)
@@ -16506,14 +16464,14 @@ dumpConstraint(Archive *fout, ConstraintInfo *coninfo)
if (coninfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, coninfo->dobj.catId, coninfo->dobj.dumpId,
- tag,
- tbinfo->dobj.namespace->dobj.name,
- indxinfo->tablespace,
- tbinfo->rolname,
- "CONSTRAINT", SECTION_POST_DATA,
- q->data, delq->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = tag,
+ .namespace = tbinfo->dobj.namespace->dobj.name,
+ .tablespace = indxinfo->tablespace,
+ .owner = tbinfo->rolname,
+ .description = "CONSTRAINT",
+ .section = SECTION_POST_DATA,
+ .createStmt = q->data,
+ .dropStmt = delq->data));
}
else if (coninfo->contype == 'f')
{
@@ -16546,14 +16504,13 @@ dumpConstraint(Archive *fout, ConstraintInfo *coninfo)
if (coninfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, coninfo->dobj.catId, coninfo->dobj.dumpId,
- tag,
- tbinfo->dobj.namespace->dobj.name,
- NULL,
- tbinfo->rolname,
- "FK CONSTRAINT", SECTION_POST_DATA,
- q->data, delq->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = tag,
+ .namespace = tbinfo->dobj.namespace->dobj.name,
+ .owner = tbinfo->rolname,
+ .description = "FK CONSTRAINT",
+ .section = SECTION_POST_DATA,
+ .createStmt = q->data,
+ .dropStmt = delq->data));
}
else if (coninfo->contype == 'c' && tbinfo)
{
@@ -16578,14 +16535,13 @@ dumpConstraint(Archive *fout, ConstraintInfo *coninfo)
if (coninfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, coninfo->dobj.catId, coninfo->dobj.dumpId,
- tag,
- tbinfo->dobj.namespace->dobj.name,
- NULL,
- tbinfo->rolname,
- "CHECK CONSTRAINT", SECTION_POST_DATA,
- q->data, delq->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = tag,
+ .namespace = tbinfo->dobj.namespace->dobj.name,
+ .owner = tbinfo->rolname,
+ .description = "CHECK CONSTRAINT",
+ .section = SECTION_POST_DATA,
+ .createStmt = q->data,
+ .dropStmt = delq->data));
}
}
else if (coninfo->contype == 'c' && tbinfo == NULL)
@@ -16611,14 +16567,13 @@ dumpConstraint(Archive *fout, ConstraintInfo *coninfo)
if (coninfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, coninfo->dobj.catId, coninfo->dobj.dumpId,
- tag,
- tyinfo->dobj.namespace->dobj.name,
- NULL,
- tyinfo->rolname,
- "CHECK CONSTRAINT", SECTION_POST_DATA,
- q->data, delq->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = tag,
+ .namespace = tyinfo->dobj.namespace->dobj.name,
+ .owner = tyinfo->rolname,
+ .description = "CHECK CONSTRAINT",
+ .section = SECTION_POST_DATA,
+ .createStmt = q->data,
+ .dropStmt = delq->data));
}
}
else
@@ -16885,14 +16840,13 @@ dumpSequence(Archive *fout, TableInfo *tbinfo)
if (tbinfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, tbinfo->dobj.catId, tbinfo->dobj.dumpId,
- tbinfo->dobj.name,
- tbinfo->dobj.namespace->dobj.name,
- NULL,
- tbinfo->rolname,
- "SEQUENCE", SECTION_PRE_DATA,
- query->data, delqry->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = tbinfo->dobj.name,
+ .namespace = tbinfo->dobj.namespace->dobj.name,
+ .owner = tbinfo->rolname,
+ .description = "SEQUENCE",
+ .section = SECTION_PRE_DATA,
+ .createStmt = query->data,
+ .dropStmt = delqry->data));
/*
* If the sequence is owned by a table column, emit the ALTER for it as a
@@ -16926,14 +16880,14 @@ dumpSequence(Archive *fout, TableInfo *tbinfo)
if (tbinfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, nilCatalogId, createDumpId(),
- tbinfo->dobj.name,
- tbinfo->dobj.namespace->dobj.name,
- NULL,
- tbinfo->rolname,
- "SEQUENCE OWNED BY", SECTION_PRE_DATA,
- query->data, "", NULL,
- &(tbinfo->dobj.dumpId), 1,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = tbinfo->dobj.name,
+ .namespace = tbinfo->dobj.namespace->dobj.name,
+ .owner = tbinfo->rolname,
+ .description = "SEQUENCE OWNED BY",
+ .section = SECTION_PRE_DATA,
+ .createStmt = query->data,
+ .deps = &(tbinfo->dobj.dumpId),
+ .nDeps = 1));
}
}
@@ -16994,14 +16948,14 @@ dumpSequenceData(Archive *fout, TableDataInfo *tdinfo)
if (tdinfo->dobj.dump & DUMP_COMPONENT_DATA)
ArchiveEntry(fout, nilCatalogId, createDumpId(),
- tbinfo->dobj.name,
- tbinfo->dobj.namespace->dobj.name,
- NULL,
- tbinfo->rolname,
- "SEQUENCE SET", SECTION_DATA,
- query->data, "", NULL,
- &(tbinfo->dobj.dumpId), 1,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = tbinfo->dobj.name,
+ .namespace = tbinfo->dobj.namespace->dobj.name,
+ .owner = tbinfo->rolname,
+ .description = "SEQUENCE SET",
+ .section = SECTION_DATA,
+ .createStmt = query->data,
+ .deps = &(tbinfo->dobj.dumpId),
+ .nDeps = 1));
PQclear(res);
@@ -17193,14 +17147,13 @@ dumpTrigger(Archive *fout, TriggerInfo *tginfo)
if (tginfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, tginfo->dobj.catId, tginfo->dobj.dumpId,
- tag,
- tbinfo->dobj.namespace->dobj.name,
- NULL,
- tbinfo->rolname,
- "TRIGGER", SECTION_POST_DATA,
- query->data, delqry->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = tag,
+ .namespace = tbinfo->dobj.namespace->dobj.name,
+ .owner = tbinfo->rolname,
+ .description = "TRIGGER",
+ .section = SECTION_POST_DATA,
+ .createStmt = query->data,
+ .dropStmt = delqry->data));
if (tginfo->dobj.dump & DUMP_COMPONENT_COMMENT)
dumpComment(fout, trigprefix->data, qtabname,
@@ -17282,12 +17235,12 @@ dumpEventTrigger(Archive *fout, EventTriggerInfo *evtinfo)
if (evtinfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, evtinfo->dobj.catId, evtinfo->dobj.dumpId,
- evtinfo->dobj.name, NULL, NULL,
- evtinfo->evtowner,
- "EVENT TRIGGER", SECTION_POST_DATA,
- query->data, delqry->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = evtinfo->dobj.name,
+ .owner = evtinfo->evtowner,
+ .description = "EVENT TRIGGER",
+ .section = SECTION_POST_DATA,
+ .createStmt = query->data,
+ .dropStmt = delqry->data));
if (evtinfo->dobj.dump & DUMP_COMPONENT_COMMENT)
dumpComment(fout, "EVENT TRIGGER", qevtname,
@@ -17440,14 +17393,13 @@ dumpRule(Archive *fout, RuleInfo *rinfo)
if (rinfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
ArchiveEntry(fout, rinfo->dobj.catId, rinfo->dobj.dumpId,
- tag,
- tbinfo->dobj.namespace->dobj.name,
- NULL,
- tbinfo->rolname,
- "RULE", SECTION_POST_DATA,
- cmd->data, delcmd->data, NULL,
- NULL, 0,
- NULL, NULL);
+ ARCHIVE_OPTS(.tag = tag,
+ .namespace = tbinfo->dobj.namespace->dobj.name,
+ .owner = tbinfo->rolname,
+ .description = "RULE",
+ .section = SECTION_POST_DATA,
+ .createStmt = cmd->data,
+ .dropStmt = delcmd->data));
/* Dump rule comments */
if (rinfo->dobj.dump & DUMP_COMPONENT_COMMENT)
--
2.11.0
--iiy4stvy6qmt4ip6--
^ permalink raw reply [nested|flat] 24+ messages in thread
* [PATCH v29 07/11] Add DISTINCT support for IVM
@ 2023-05-31 10:08 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 24+ messages in thread
From: Yugo Nagata @ 2023-05-31 10:08 UTC (permalink / raw)
When IMMV is created with DISTINCT, multiplicity of tuples is
counted and stored in "__ivm_count__" column, which is a hidden
column of IMMV. The value in __ivm_count__ is updated when IMMV
is maintained incrementally. A tuple in IMMV can be removed if
and only if the count becomes zero.
---
src/backend/commands/createas.c | 141 ++++++++++++++++++++------
src/backend/commands/indexcmds.c | 40 ++++++++
src/backend/commands/matview.c | 148 ++++++++++++++++++++++++++--
src/backend/commands/tablecmds.c | 9 ++
src/backend/parser/parse_relation.c | 18 +++-
src/backend/rewrite/rewriteDefine.c | 3 +-
src/include/commands/createas.h | 2 +
src/include/nodes/parsenodes.h | 1 +
8 files changed, 317 insertions(+), 45 deletions(-)
diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c
index 415f110516..076f35ee6b 100644
--- a/src/backend/commands/createas.c
+++ b/src/backend/commands/createas.c
@@ -53,6 +53,7 @@
#include "parser/parser.h"
#include "parser/parsetree.h"
#include "parser/parse_clause.h"
+#include "parser/parse_func.h"
#include "rewrite/rewriteHandler.h"
#include "storage/smgr.h"
#include "tcop/tcopprot.h"
@@ -309,6 +310,9 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt,
errhint("functions must be marked IMMUTABLE")));
check_ivm_restriction((Node *) query);
+
+ /* For IMMV, we need to rewrite matview query */
+ query = rewriteQueryForIMMV(query, into->colNames);
}
if (into->skipData)
@@ -413,6 +417,49 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt,
return address;
}
+/*
+ * rewriteQueryForIMMV -- rewrite view definition query for IMMV
+ *
+ * count(*) is added for counting distinct tuples in views.
+ */
+Query *
+rewriteQueryForIMMV(Query *query, List *colNames)
+{
+ Query *rewritten;
+
+ Node *node;
+ ParseState *pstate = make_parsestate(NULL);
+ FuncCall *fn;
+
+ rewritten = copyObject(query);
+ pstate->p_expr_kind = EXPR_KIND_SELECT_TARGET;
+
+ /*
+ * Convert DISTINCT to GROUP BY and add count(*) for counting distinct
+ * tuples in views.
+ */
+ if (rewritten->distinctClause)
+ {
+ TargetEntry *tle;
+
+ rewritten->groupClause = transformDistinctClause(NULL, &rewritten->targetList, rewritten->sortClause, false);
+
+ fn = makeFuncCall(SystemFuncName("count"), NIL, COERCE_EXPLICIT_CALL, -1);
+ fn->agg_star = true;
+
+ node = ParseFuncOrColumn(pstate, fn->funcname, NIL, NULL, fn, false, -1);
+
+ tle = makeTargetEntry((Expr *) node,
+ list_length(rewritten->targetList) + 1,
+ pstrdup("__ivm_count__"),
+ false);
+ rewritten->targetList = lappend(rewritten->targetList, tle);
+ rewritten->hasAggs = true;
+ }
+
+ return rewritten;
+}
+
/*
* GetIntoRelEFlags --- compute executor flags needed for CREATE TABLE AS
*
@@ -536,7 +583,8 @@ intorel_startup(DestReceiver *self, int operation, TupleDesc typeinfo)
ColumnDef *col;
char *colname;
- if (lc)
+ /* Don't override hidden columns added for IVM */
+ if (lc && !isIvmName(NameStr(attribute->attname)))
{
colname = strVal(lfirst(lc));
lc = lnext(into->colNames, lc);
@@ -940,10 +988,6 @@ check_ivm_restriction_walker(Node *node, void *context)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("LIMIT/OFFSET clause is not supported on incrementally maintainable materialized view")));
- if (qry->distinctClause)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("DISTINCT is not supported on incrementally maintainable materialized view")));
if (qry->hasDistinctOn)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -1090,12 +1134,18 @@ CreateIndexOnIMMV(Query *query, Relation matviewRel)
char idxname[NAMEDATALEN];
List *indexoidlist = RelationGetIndexList(matviewRel);
ListCell *indexoidscan;
- Bitmapset *key_attnos;
snprintf(idxname, sizeof(idxname), "%s_index", RelationGetRelationName(matviewRel));
index = makeNode(IndexStmt);
+ /*
+ * We consider null values not distinct to make sure that views with DISTINCT
+ * or GROUP BY don't contain multiple NULL rows when NULL is inserted to
+ * a base table concurrently.
+ */
+ index->nulls_not_distinct = true;
+
index->unique = true;
index->primary = false;
index->isconstraint = false;
@@ -1122,41 +1172,68 @@ CreateIndexOnIMMV(Query *query, Relation matviewRel)
index->concurrent = false;
index->if_not_exists = false;
- /* create index on the base tables' primary key columns */
- key_attnos = get_primary_key_attnos_from_query(query, &constraintList);
- if (key_attnos)
+ if (query->distinctClause)
{
+ /* create unique constraint on all columns */
foreach(lc, query->targetList)
{
TargetEntry *tle = (TargetEntry *) lfirst(lc);
Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, tle->resno - 1);
-
- if (bms_is_member(tle->resno - FirstLowInvalidHeapAttributeNumber, key_attnos))
- {
- IndexElem *iparam;
-
- iparam = makeNode(IndexElem);
- iparam->name = pstrdup(NameStr(attr->attname));
- iparam->expr = NULL;
- iparam->indexcolname = NULL;
- iparam->collation = NIL;
- iparam->opclass = NIL;
- iparam->opclassopts = NIL;
- iparam->ordering = SORTBY_DEFAULT;
- iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
- index->indexParams = lappend(index->indexParams, iparam);
- }
+ IndexElem *iparam;
+
+ iparam = makeNode(IndexElem);
+ iparam->name = pstrdup(NameStr(attr->attname));
+ iparam->expr = NULL;
+ iparam->indexcolname = NULL;
+ iparam->collation = NIL;
+ iparam->opclass = NIL;
+ iparam->opclassopts = NIL;
+ iparam->ordering = SORTBY_DEFAULT;
+ iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
+ index->indexParams = lappend(index->indexParams, iparam);
}
}
else
{
- /* create no index, just notice that an appropriate index is necessary for efficient IVM */
- ereport(NOTICE,
- (errmsg("could not create an index on materialized view \"%s\" automatically",
- RelationGetRelationName(matviewRel)),
- errdetail("This target list does not have all the primary key columns. "),
- errhint("Create an index on the materialized view for efficient incremental maintenance.")));
- return;
+ Bitmapset *key_attnos;
+
+ /* create index on the base tables' primary key columns */
+ key_attnos = get_primary_key_attnos_from_query(query, &constraintList);
+ if (key_attnos)
+ {
+ foreach(lc, query->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);
+ Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, tle->resno - 1);
+
+ if (bms_is_member(tle->resno - FirstLowInvalidHeapAttributeNumber, key_attnos))
+ {
+ IndexElem *iparam;
+
+ iparam = makeNode(IndexElem);
+ iparam->name = pstrdup(NameStr(attr->attname));
+ iparam->expr = NULL;
+ iparam->indexcolname = NULL;
+ iparam->collation = NIL;
+ iparam->opclass = NIL;
+ iparam->opclassopts = NIL;
+ iparam->ordering = SORTBY_DEFAULT;
+ iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
+ index->indexParams = lappend(index->indexParams, iparam);
+ }
+ }
+ }
+ else
+ {
+ /* create no index, just notice that an appropriate index is necessary for efficient IVM */
+ ereport(NOTICE,
+ (errmsg("could not create an index on materialized view \"%s\" automatically",
+ RelationGetRelationName(matviewRel)),
+ errdetail("This target list does not have all the primary key columns, "
+ "or this view does not contain DISTINCT clause."),
+ errhint("Create an index on the materialized view for efficient incremental maintenance.")));
+ return;
+ }
}
/* If we have a compatible index, we don't need to create another. */
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index ab8b81b302..4811a1c8df 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -38,6 +38,7 @@
#include "commands/dbcommands.h"
#include "commands/defrem.h"
#include "commands/event_trigger.h"
+#include "commands/matview.h"
#include "commands/progress.h"
#include "commands/tablecmds.h"
#include "commands/tablespace.h"
@@ -1104,6 +1105,45 @@ DefineIndex(Oid tableId,
safe_index = indexInfo->ii_Expressions == NIL &&
indexInfo->ii_Predicate == NIL;
+ /*
+ * We disallow unique indexes on IVM columns of IMMVs.
+ */
+ if (RelationIsIVM(rel) && stmt->unique)
+ {
+ for (int i = 0; i < indexInfo->ii_NumIndexKeyAttrs; i++)
+ {
+ AttrNumber attno = indexInfo->ii_IndexAttrNumbers[i];
+ if (attno > 0)
+ {
+ char *name = NameStr(TupleDescAttr(rel->rd_att, attno - 1)->attname);
+ if (name && isIvmName(name))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("unique index creation on IVM columns is not supported")));
+ }
+ }
+
+ if (indexInfo->ii_Expressions)
+ {
+ Bitmapset *indexattrs = NULL;
+ int varno = -1;
+
+ pull_varattnos((Node *) indexInfo->ii_Expressions, 1, &indexattrs);
+
+ while ((varno = bms_next_member(indexattrs, varno)) >= 0)
+ {
+ int attno = varno + FirstLowInvalidHeapAttributeNumber;
+ char *name = NameStr(TupleDescAttr(rel->rd_att, attno - 1)->attname);
+ if (name && isIvmName(name))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("unique index creation on IVM columns is not supported")));
+ }
+
+ }
+ }
+
+
/*
* Report index creation if appropriate (delay this till after most of the
* error checks)
diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c
index 39305f3c49..aa518f20ef 100644
--- a/src/backend/commands/matview.c
+++ b/src/backend/commands/matview.c
@@ -152,11 +152,15 @@ static Query *rewrite_query_for_postupdate_state(Query *query, MV_TriggerTable *
static void apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *new_tuplestores,
TupleDesc tupdesc_old, TupleDesc tupdesc_new,
- Query *query);
+ Query *query, bool use_count, char *count_colname);
static void apply_old_delta(const char *matviewname, const char *deltaname_old,
List *keys);
+static void apply_old_delta_with_count(const char *matviewname, const char *deltaname_old,
+ List *keys, const char *count_colname);
static void apply_new_delta(const char *matviewname, const char *deltaname_new,
StringInfo target_list);
+static void apply_new_delta_with_count(const char *matviewname, const char* deltaname_new,
+ List *keys, StringInfo target_list, const char* count_colname);
static char *get_matching_condition_string(List *keys);
static void generate_equal(StringInfo querybuf, Oid opttype,
const char *leftop, const char *rightop);
@@ -271,6 +275,7 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
Oid matviewOid;
Relation matviewRel;
Query *dataQuery;
+ Query *viewQuery;
Oid tableSpace;
Oid relowner;
Oid OIDNewHeap;
@@ -330,8 +335,13 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
"CONCURRENTLY", "WITH NO DATA")));
- dataQuery = get_matview_query(matviewRel);
+ viewQuery = get_matview_query(matviewRel);
+ /* For IMMV, we need to rewrite matview query */
+ if (!stmt->skipData && RelationIsIVM(matviewRel))
+ dataQuery = rewriteQueryForIMMV(viewQuery,NIL);
+ else
+ dataQuery = viewQuery;
/*
* Check that there is a unique index with no WHERE clause on one or more
@@ -511,8 +521,8 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
if (!stmt->skipData && RelationIsIVM(matviewRel) && !oldPopulated)
{
- CreateIndexOnIMMV(dataQuery, matviewRel);
- CreateIvmTriggersOnBaseTables(dataQuery, matviewOid);
+ CreateIndexOnIMMV(viewQuery, matviewRel);
+ CreateIvmTriggersOnBaseTables(viewQuery, matviewOid);
}
table_close(matviewRel, NoLock);
@@ -1512,6 +1522,13 @@ IVM_immediate_maintenance(PG_FUNCTION_ARGS)
int rte_index = lfirst_int(lc2);
TupleDesc tupdesc_old;
TupleDesc tupdesc_new;
+ bool use_count = false;
+ char *count_colname = NULL;
+
+ count_colname = pstrdup("__ivm_count__");
+
+ if (query->distinctClause)
+ use_count = true;
/* calculate delta tables */
calc_delta(table, rte_index, rewritten, dest_old, dest_new,
@@ -1524,7 +1541,8 @@ IVM_immediate_maintenance(PG_FUNCTION_ARGS)
{
/* apply the delta tables to the materialized view */
apply_delta(matviewOid, old_tuplestore, new_tuplestore,
- tupdesc_old, tupdesc_new, query);
+ tupdesc_old, tupdesc_new, query, use_count,
+ count_colname);
}
PG_CATCH();
{
@@ -1997,7 +2015,7 @@ rewrite_query_for_postupdate_state(Query *query, MV_TriggerTable *table, int rte
static void
apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *new_tuplestores,
TupleDesc tupdesc_old, TupleDesc tupdesc_new,
- Query *query)
+ Query *query, bool use_count, char *count_colname)
{
StringInfoData querybuf;
StringInfoData target_list_buf;
@@ -2073,7 +2091,12 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
if (rc != SPI_OK_REL_REGISTER)
elog(ERROR, "SPI_register failed");
- apply_old_delta(matviewname, OLD_DELTA_ENRNAME, keys);
+ if (use_count)
+ /* apply old delta and get rows to be recalculated */
+ apply_old_delta_with_count(matviewname, OLD_DELTA_ENRNAME,
+ keys, count_colname);
+ else
+ apply_old_delta(matviewname, OLD_DELTA_ENRNAME, keys);
}
/* For tuple insertion */
@@ -2095,7 +2118,11 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
elog(ERROR, "SPI_register failed");
/* apply new delta */
- apply_new_delta(matviewname, NEW_DELTA_ENRNAME, &target_list_buf);
+ if (use_count)
+ apply_new_delta_with_count(matviewname, NEW_DELTA_ENRNAME,
+ keys, &target_list_buf, count_colname);
+ else
+ apply_new_delta(matviewname, NEW_DELTA_ENRNAME, &target_list_buf);
}
/* We're done maintaining the materialized view. */
@@ -2108,6 +2135,51 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
elog(ERROR, "SPI_finish failed");
}
+/*
+ * apply_old_delta_with_count
+ *
+ * Execute a query for applying a delta table given by deltname_old
+ * which contains tuples to be deleted from to a materialized view given by
+ * matviewname. This is used when counting is required, that is, the view
+ * has aggregate or distinct.
+ */
+static void
+apply_old_delta_with_count(const char *matviewname, const char *deltaname_old,
+ List *keys, const char *count_colname)
+{
+ StringInfoData querybuf;
+ char *match_cond;
+
+ /* build WHERE condition for searching tuples to be deleted */
+ match_cond = get_matching_condition_string(keys);
+
+ /* Search for matching tuples from the view and update or delete if found. */
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH t AS (" /* collecting tid of target tuples in the view */
+ "SELECT diff.%s, " /* count column */
+ "(diff.%s OPERATOR(pg_catalog.=) mv.%s) AS for_dlt, "
+ "mv.ctid "
+ "FROM %s AS mv, %s AS diff "
+ "WHERE %s" /* tuple matching condition */
+ "), updt AS (" /* update a tuple if this is not to be deleted */
+ "UPDATE %s AS mv SET %s = mv.%s OPERATOR(pg_catalog.-) t.%s "
+ "FROM t WHERE mv.ctid OPERATOR(pg_catalog.=) t.ctid AND NOT for_dlt "
+ ")"
+ /* delete a tuple if this is to be deleted */
+ "DELETE FROM %s AS mv USING t "
+ "WHERE mv.ctid OPERATOR(pg_catalog.=) t.ctid AND for_dlt",
+ count_colname,
+ count_colname, count_colname,
+ matviewname, deltaname_old,
+ match_cond,
+ matviewname, count_colname, count_colname, count_colname,
+ matviewname);
+
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_DELETE)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+}
+
/*
* apply_old_delta
*
@@ -2157,6 +2229,66 @@ apply_old_delta(const char *matviewname, const char *deltaname_old,
elog(ERROR, "SPI_exec failed: %s", querybuf.data);
}
+/*
+ * apply_new_delta_with_count
+ *
+ * Execute a query for applying a delta table given by deltname_new
+ * which contains tuples to be inserted into a materialized view given by
+ * matviewname. This is used when counting is required, that is, the view
+ * has aggregate or distinct. Also, when a table in EXISTS sub queries
+ * is modified.
+ */
+static void
+apply_new_delta_with_count(const char *matviewname, const char* deltaname_new,
+ List *keys, StringInfo target_list, const char* count_colname)
+{
+ StringInfoData querybuf;
+ StringInfoData returning_keys;
+ ListCell *lc;
+ char *match_cond = "";
+
+ /* build WHERE condition for searching tuples to be updated */
+ match_cond = get_matching_condition_string(keys);
+
+ /* build string of keys list */
+ initStringInfo(&returning_keys);
+ if (keys)
+ {
+ foreach (lc, keys)
+ {
+ Form_pg_attribute attr = (Form_pg_attribute) lfirst(lc);
+ char *resname = NameStr(attr->attname);
+ appendStringInfo(&returning_keys, "%s", quote_qualified_identifier("mv", resname));
+ if (lnext(keys, lc))
+ appendStringInfo(&returning_keys, ", ");
+ }
+ }
+ else
+ appendStringInfo(&returning_keys, "NULL");
+
+ /* Search for matching tuples from the view and update if found or insert if not. */
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH updt AS (" /* update a tuple if this exists in the view */
+ "UPDATE %s AS mv SET %s = mv.%s OPERATOR(pg_catalog.+) diff.%s "
+ "FROM %s AS diff "
+ "WHERE %s " /* tuple matching condition */
+ "RETURNING %s" /* returning keys of updated tuples */
+ ") INSERT INTO %s (%s) " /* insert a new tuple if this doesn't exist */
+ "SELECT %s FROM %s AS diff "
+ "WHERE NOT EXISTS (SELECT 1 FROM updt AS mv WHERE %s);",
+ matviewname, count_colname, count_colname, count_colname,
+ deltaname_new,
+ match_cond,
+ returning_keys.data,
+ matviewname, target_list->data,
+ target_list->data, deltaname_new,
+ match_cond);
+
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+}
+
/*
* apply_new_delta
*
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 47c900445c..adbd768e0d 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -53,6 +53,7 @@
#include "commands/cluster.h"
#include "commands/comment.h"
#include "commands/defrem.h"
+#include "commands/matview.h"
#include "commands/event_trigger.h"
#include "commands/policy.h"
#include "commands/sequence.h"
@@ -3673,6 +3674,14 @@ renameatt_internal(Oid myrelid,
targetrelation = relation_open(myrelid, AccessExclusiveLock);
renameatt_check(myrelid, RelationGetForm(targetrelation), recursing);
+ /*
+ * Don't rename IVM columns.
+ */
+ if (RelationIsIVM(targetrelation) && isIvmName(oldattname))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("IVM column can not be renamed")));
+
/*
* if the 'recurse' flag is set then we are supposed to rename this
* attribute in all classes that inherit from 'relname' (as well as in
diff --git a/src/backend/parser/parse_relation.c b/src/backend/parser/parse_relation.c
index 864ea9b0d5..c257440414 100644
--- a/src/backend/parser/parse_relation.c
+++ b/src/backend/parser/parse_relation.c
@@ -36,6 +36,7 @@
#include "utils/rel.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
+#include "commands/matview.h"
/*
@@ -97,7 +98,7 @@ static void expandTupleDesc(TupleDesc tupdesc, Alias *eref,
int count, int offset,
int rtindex, int sublevels_up,
int location, bool include_dropped,
- List **colnames, List **colvars);
+ List **colnames, List **colvars, bool is_ivm);
static int specialAttNum(const char *attname);
static bool rte_visible_if_lateral(ParseState *pstate, RangeTblEntry *rte);
static bool rte_visible_if_qualified(ParseState *pstate, RangeTblEntry *rte);
@@ -1502,6 +1503,7 @@ addRangeTableEntry(ParseState *pstate,
rte->relid = RelationGetRelid(rel);
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -1587,6 +1589,7 @@ addRangeTableEntryForRelation(ParseState *pstate,
rte->relid = RelationGetRelid(rel);
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -2758,7 +2761,7 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
expandTupleDesc(tupdesc, rte->eref,
rtfunc->funccolcount, atts_done,
rtindex, sublevels_up, location,
- include_dropped, colnames, colvars);
+ include_dropped, colnames, colvars, false);
}
else if (functypclass == TYPEFUNC_SCALAR)
{
@@ -3026,7 +3029,7 @@ expandRelation(Oid relid, Alias *eref, int rtindex, int sublevels_up,
expandTupleDesc(rel->rd_att, eref, rel->rd_att->natts, 0,
rtindex, sublevels_up,
location, include_dropped,
- colnames, colvars);
+ colnames, colvars, RelationIsIVM(rel));
relation_close(rel, AccessShareLock);
}
@@ -3043,7 +3046,7 @@ static void
expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
int rtindex, int sublevels_up,
int location, bool include_dropped,
- List **colnames, List **colvars)
+ List **colnames, List **colvars, bool is_ivm)
{
ListCell *aliascell;
int varattno;
@@ -3056,6 +3059,9 @@ expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
{
Form_pg_attribute attr = TupleDescAttr(tupdesc, varattno);
+ if (is_ivm && isIvmName(NameStr(attr->attname)) && !MatViewIncrementalMaintenanceIsEnabled())
+ continue;
+
if (attr->attisdropped)
{
if (include_dropped)
@@ -3218,6 +3224,10 @@ expandNSItemAttrs(ParseState *pstate, ParseNamespaceItem *nsitem,
Var *varnode = (Var *) lfirst(var);
TargetEntry *te;
+ /* if transform * into columnlist with IMMV, remove IVM columns */
+ if (rte->relisivm && isIvmName(label) && !MatViewIncrementalMaintenanceIsEnabled())
+ continue;
+
te = makeTargetEntry((Expr *) varnode,
(AttrNumber) pstate->p_next_resno++,
label,
diff --git a/src/backend/rewrite/rewriteDefine.c b/src/backend/rewrite/rewriteDefine.c
index e36fc72e1e..f6dc7ba202 100644
--- a/src/backend/rewrite/rewriteDefine.c
+++ b/src/backend/rewrite/rewriteDefine.c
@@ -621,7 +621,8 @@ checkRuleResultList(List *targetList, TupleDesc resultDesc, bool isSelect,
attr->atttypmod))));
}
- if (i != resultDesc->natts)
+ /* No check for materialized views since this could have special columns for IVM */
+ if ((!isSelect || requireColumnNameMatch) && i != resultDesc->natts)
ereport(ERROR,
(errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
isSelect ?
diff --git a/src/include/commands/createas.h b/src/include/commands/createas.h
index 09a64fa2e5..76a7873ebf 100644
--- a/src/include/commands/createas.h
+++ b/src/include/commands/createas.h
@@ -29,6 +29,8 @@ extern ObjectAddress ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *st
extern void CreateIvmTriggersOnBaseTables(Query *qry, Oid matviewOid);
extern void CreateIndexOnIMMV(Query *query, Relation matviewRel);
+extern Query *rewriteQueryForIMMV(Query *query, List *colNames);
+
extern int GetIntoRelEFlags(IntoClause *intoClause);
extern DestReceiver *CreateIntoRelDestReceiver(IntoClause *intoClause);
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index fef4c714b8..1a2b8fa09e 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -1073,6 +1073,7 @@ typedef struct RangeTblEntry
int rellockmode; /* lock level that query requires on the rel */
struct TableSampleClause *tablesample; /* sampling info, or NULL */
Index perminfoindex;
+ bool relisivm;
/*
* Fields valid for a subquery RTE (else NULL):
--
2.25.1
--Multipart=_Mon__28_Aug_2023_11_52_52_+0900_hj6L5h176QaSGtg7
Content-Type: text/x-diff;
name="v29-0008-Add-aggregates-support-in-IVM.patch"
Content-Disposition: attachment;
filename="v29-0008-Add-aggregates-support-in-IVM.patch"
Content-Transfer-Encoding: quoted-printable
^ permalink raw reply [nested|flat] 24+ messages in thread
* [PATCH v29 07/11] Add DISTINCT support for IVM
@ 2023-05-31 10:08 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 24+ messages in thread
From: Yugo Nagata @ 2023-05-31 10:08 UTC (permalink / raw)
When IMMV is created with DISTINCT, multiplicity of tuples is
counted and stored in "__ivm_count__" column, which is a hidden
column of IMMV. The value in __ivm_count__ is updated when IMMV
is maintained incrementally. A tuple in IMMV can be removed if
and only if the count becomes zero.
---
src/backend/commands/createas.c | 141 ++++++++++++++++++++------
src/backend/commands/indexcmds.c | 40 ++++++++
src/backend/commands/matview.c | 148 ++++++++++++++++++++++++++--
src/backend/commands/tablecmds.c | 9 ++
src/backend/parser/parse_relation.c | 18 +++-
src/backend/rewrite/rewriteDefine.c | 3 +-
src/include/commands/createas.h | 2 +
src/include/nodes/parsenodes.h | 1 +
8 files changed, 317 insertions(+), 45 deletions(-)
diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c
index 415f110516..076f35ee6b 100644
--- a/src/backend/commands/createas.c
+++ b/src/backend/commands/createas.c
@@ -53,6 +53,7 @@
#include "parser/parser.h"
#include "parser/parsetree.h"
#include "parser/parse_clause.h"
+#include "parser/parse_func.h"
#include "rewrite/rewriteHandler.h"
#include "storage/smgr.h"
#include "tcop/tcopprot.h"
@@ -309,6 +310,9 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt,
errhint("functions must be marked IMMUTABLE")));
check_ivm_restriction((Node *) query);
+
+ /* For IMMV, we need to rewrite matview query */
+ query = rewriteQueryForIMMV(query, into->colNames);
}
if (into->skipData)
@@ -413,6 +417,49 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt,
return address;
}
+/*
+ * rewriteQueryForIMMV -- rewrite view definition query for IMMV
+ *
+ * count(*) is added for counting distinct tuples in views.
+ */
+Query *
+rewriteQueryForIMMV(Query *query, List *colNames)
+{
+ Query *rewritten;
+
+ Node *node;
+ ParseState *pstate = make_parsestate(NULL);
+ FuncCall *fn;
+
+ rewritten = copyObject(query);
+ pstate->p_expr_kind = EXPR_KIND_SELECT_TARGET;
+
+ /*
+ * Convert DISTINCT to GROUP BY and add count(*) for counting distinct
+ * tuples in views.
+ */
+ if (rewritten->distinctClause)
+ {
+ TargetEntry *tle;
+
+ rewritten->groupClause = transformDistinctClause(NULL, &rewritten->targetList, rewritten->sortClause, false);
+
+ fn = makeFuncCall(SystemFuncName("count"), NIL, COERCE_EXPLICIT_CALL, -1);
+ fn->agg_star = true;
+
+ node = ParseFuncOrColumn(pstate, fn->funcname, NIL, NULL, fn, false, -1);
+
+ tle = makeTargetEntry((Expr *) node,
+ list_length(rewritten->targetList) + 1,
+ pstrdup("__ivm_count__"),
+ false);
+ rewritten->targetList = lappend(rewritten->targetList, tle);
+ rewritten->hasAggs = true;
+ }
+
+ return rewritten;
+}
+
/*
* GetIntoRelEFlags --- compute executor flags needed for CREATE TABLE AS
*
@@ -536,7 +583,8 @@ intorel_startup(DestReceiver *self, int operation, TupleDesc typeinfo)
ColumnDef *col;
char *colname;
- if (lc)
+ /* Don't override hidden columns added for IVM */
+ if (lc && !isIvmName(NameStr(attribute->attname)))
{
colname = strVal(lfirst(lc));
lc = lnext(into->colNames, lc);
@@ -940,10 +988,6 @@ check_ivm_restriction_walker(Node *node, void *context)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("LIMIT/OFFSET clause is not supported on incrementally maintainable materialized view")));
- if (qry->distinctClause)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("DISTINCT is not supported on incrementally maintainable materialized view")));
if (qry->hasDistinctOn)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -1090,12 +1134,18 @@ CreateIndexOnIMMV(Query *query, Relation matviewRel)
char idxname[NAMEDATALEN];
List *indexoidlist = RelationGetIndexList(matviewRel);
ListCell *indexoidscan;
- Bitmapset *key_attnos;
snprintf(idxname, sizeof(idxname), "%s_index", RelationGetRelationName(matviewRel));
index = makeNode(IndexStmt);
+ /*
+ * We consider null values not distinct to make sure that views with DISTINCT
+ * or GROUP BY don't contain multiple NULL rows when NULL is inserted to
+ * a base table concurrently.
+ */
+ index->nulls_not_distinct = true;
+
index->unique = true;
index->primary = false;
index->isconstraint = false;
@@ -1122,41 +1172,68 @@ CreateIndexOnIMMV(Query *query, Relation matviewRel)
index->concurrent = false;
index->if_not_exists = false;
- /* create index on the base tables' primary key columns */
- key_attnos = get_primary_key_attnos_from_query(query, &constraintList);
- if (key_attnos)
+ if (query->distinctClause)
{
+ /* create unique constraint on all columns */
foreach(lc, query->targetList)
{
TargetEntry *tle = (TargetEntry *) lfirst(lc);
Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, tle->resno - 1);
-
- if (bms_is_member(tle->resno - FirstLowInvalidHeapAttributeNumber, key_attnos))
- {
- IndexElem *iparam;
-
- iparam = makeNode(IndexElem);
- iparam->name = pstrdup(NameStr(attr->attname));
- iparam->expr = NULL;
- iparam->indexcolname = NULL;
- iparam->collation = NIL;
- iparam->opclass = NIL;
- iparam->opclassopts = NIL;
- iparam->ordering = SORTBY_DEFAULT;
- iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
- index->indexParams = lappend(index->indexParams, iparam);
- }
+ IndexElem *iparam;
+
+ iparam = makeNode(IndexElem);
+ iparam->name = pstrdup(NameStr(attr->attname));
+ iparam->expr = NULL;
+ iparam->indexcolname = NULL;
+ iparam->collation = NIL;
+ iparam->opclass = NIL;
+ iparam->opclassopts = NIL;
+ iparam->ordering = SORTBY_DEFAULT;
+ iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
+ index->indexParams = lappend(index->indexParams, iparam);
}
}
else
{
- /* create no index, just notice that an appropriate index is necessary for efficient IVM */
- ereport(NOTICE,
- (errmsg("could not create an index on materialized view \"%s\" automatically",
- RelationGetRelationName(matviewRel)),
- errdetail("This target list does not have all the primary key columns. "),
- errhint("Create an index on the materialized view for efficient incremental maintenance.")));
- return;
+ Bitmapset *key_attnos;
+
+ /* create index on the base tables' primary key columns */
+ key_attnos = get_primary_key_attnos_from_query(query, &constraintList);
+ if (key_attnos)
+ {
+ foreach(lc, query->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);
+ Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, tle->resno - 1);
+
+ if (bms_is_member(tle->resno - FirstLowInvalidHeapAttributeNumber, key_attnos))
+ {
+ IndexElem *iparam;
+
+ iparam = makeNode(IndexElem);
+ iparam->name = pstrdup(NameStr(attr->attname));
+ iparam->expr = NULL;
+ iparam->indexcolname = NULL;
+ iparam->collation = NIL;
+ iparam->opclass = NIL;
+ iparam->opclassopts = NIL;
+ iparam->ordering = SORTBY_DEFAULT;
+ iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
+ index->indexParams = lappend(index->indexParams, iparam);
+ }
+ }
+ }
+ else
+ {
+ /* create no index, just notice that an appropriate index is necessary for efficient IVM */
+ ereport(NOTICE,
+ (errmsg("could not create an index on materialized view \"%s\" automatically",
+ RelationGetRelationName(matviewRel)),
+ errdetail("This target list does not have all the primary key columns, "
+ "or this view does not contain DISTINCT clause."),
+ errhint("Create an index on the materialized view for efficient incremental maintenance.")));
+ return;
+ }
}
/* If we have a compatible index, we don't need to create another. */
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index ab8b81b302..4811a1c8df 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -38,6 +38,7 @@
#include "commands/dbcommands.h"
#include "commands/defrem.h"
#include "commands/event_trigger.h"
+#include "commands/matview.h"
#include "commands/progress.h"
#include "commands/tablecmds.h"
#include "commands/tablespace.h"
@@ -1104,6 +1105,45 @@ DefineIndex(Oid tableId,
safe_index = indexInfo->ii_Expressions == NIL &&
indexInfo->ii_Predicate == NIL;
+ /*
+ * We disallow unique indexes on IVM columns of IMMVs.
+ */
+ if (RelationIsIVM(rel) && stmt->unique)
+ {
+ for (int i = 0; i < indexInfo->ii_NumIndexKeyAttrs; i++)
+ {
+ AttrNumber attno = indexInfo->ii_IndexAttrNumbers[i];
+ if (attno > 0)
+ {
+ char *name = NameStr(TupleDescAttr(rel->rd_att, attno - 1)->attname);
+ if (name && isIvmName(name))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("unique index creation on IVM columns is not supported")));
+ }
+ }
+
+ if (indexInfo->ii_Expressions)
+ {
+ Bitmapset *indexattrs = NULL;
+ int varno = -1;
+
+ pull_varattnos((Node *) indexInfo->ii_Expressions, 1, &indexattrs);
+
+ while ((varno = bms_next_member(indexattrs, varno)) >= 0)
+ {
+ int attno = varno + FirstLowInvalidHeapAttributeNumber;
+ char *name = NameStr(TupleDescAttr(rel->rd_att, attno - 1)->attname);
+ if (name && isIvmName(name))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("unique index creation on IVM columns is not supported")));
+ }
+
+ }
+ }
+
+
/*
* Report index creation if appropriate (delay this till after most of the
* error checks)
diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c
index 39305f3c49..aa518f20ef 100644
--- a/src/backend/commands/matview.c
+++ b/src/backend/commands/matview.c
@@ -152,11 +152,15 @@ static Query *rewrite_query_for_postupdate_state(Query *query, MV_TriggerTable *
static void apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *new_tuplestores,
TupleDesc tupdesc_old, TupleDesc tupdesc_new,
- Query *query);
+ Query *query, bool use_count, char *count_colname);
static void apply_old_delta(const char *matviewname, const char *deltaname_old,
List *keys);
+static void apply_old_delta_with_count(const char *matviewname, const char *deltaname_old,
+ List *keys, const char *count_colname);
static void apply_new_delta(const char *matviewname, const char *deltaname_new,
StringInfo target_list);
+static void apply_new_delta_with_count(const char *matviewname, const char* deltaname_new,
+ List *keys, StringInfo target_list, const char* count_colname);
static char *get_matching_condition_string(List *keys);
static void generate_equal(StringInfo querybuf, Oid opttype,
const char *leftop, const char *rightop);
@@ -271,6 +275,7 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
Oid matviewOid;
Relation matviewRel;
Query *dataQuery;
+ Query *viewQuery;
Oid tableSpace;
Oid relowner;
Oid OIDNewHeap;
@@ -330,8 +335,13 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
"CONCURRENTLY", "WITH NO DATA")));
- dataQuery = get_matview_query(matviewRel);
+ viewQuery = get_matview_query(matviewRel);
+ /* For IMMV, we need to rewrite matview query */
+ if (!stmt->skipData && RelationIsIVM(matviewRel))
+ dataQuery = rewriteQueryForIMMV(viewQuery,NIL);
+ else
+ dataQuery = viewQuery;
/*
* Check that there is a unique index with no WHERE clause on one or more
@@ -511,8 +521,8 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
if (!stmt->skipData && RelationIsIVM(matviewRel) && !oldPopulated)
{
- CreateIndexOnIMMV(dataQuery, matviewRel);
- CreateIvmTriggersOnBaseTables(dataQuery, matviewOid);
+ CreateIndexOnIMMV(viewQuery, matviewRel);
+ CreateIvmTriggersOnBaseTables(viewQuery, matviewOid);
}
table_close(matviewRel, NoLock);
@@ -1512,6 +1522,13 @@ IVM_immediate_maintenance(PG_FUNCTION_ARGS)
int rte_index = lfirst_int(lc2);
TupleDesc tupdesc_old;
TupleDesc tupdesc_new;
+ bool use_count = false;
+ char *count_colname = NULL;
+
+ count_colname = pstrdup("__ivm_count__");
+
+ if (query->distinctClause)
+ use_count = true;
/* calculate delta tables */
calc_delta(table, rte_index, rewritten, dest_old, dest_new,
@@ -1524,7 +1541,8 @@ IVM_immediate_maintenance(PG_FUNCTION_ARGS)
{
/* apply the delta tables to the materialized view */
apply_delta(matviewOid, old_tuplestore, new_tuplestore,
- tupdesc_old, tupdesc_new, query);
+ tupdesc_old, tupdesc_new, query, use_count,
+ count_colname);
}
PG_CATCH();
{
@@ -1997,7 +2015,7 @@ rewrite_query_for_postupdate_state(Query *query, MV_TriggerTable *table, int rte
static void
apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *new_tuplestores,
TupleDesc tupdesc_old, TupleDesc tupdesc_new,
- Query *query)
+ Query *query, bool use_count, char *count_colname)
{
StringInfoData querybuf;
StringInfoData target_list_buf;
@@ -2073,7 +2091,12 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
if (rc != SPI_OK_REL_REGISTER)
elog(ERROR, "SPI_register failed");
- apply_old_delta(matviewname, OLD_DELTA_ENRNAME, keys);
+ if (use_count)
+ /* apply old delta and get rows to be recalculated */
+ apply_old_delta_with_count(matviewname, OLD_DELTA_ENRNAME,
+ keys, count_colname);
+ else
+ apply_old_delta(matviewname, OLD_DELTA_ENRNAME, keys);
}
/* For tuple insertion */
@@ -2095,7 +2118,11 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
elog(ERROR, "SPI_register failed");
/* apply new delta */
- apply_new_delta(matviewname, NEW_DELTA_ENRNAME, &target_list_buf);
+ if (use_count)
+ apply_new_delta_with_count(matviewname, NEW_DELTA_ENRNAME,
+ keys, &target_list_buf, count_colname);
+ else
+ apply_new_delta(matviewname, NEW_DELTA_ENRNAME, &target_list_buf);
}
/* We're done maintaining the materialized view. */
@@ -2108,6 +2135,51 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
elog(ERROR, "SPI_finish failed");
}
+/*
+ * apply_old_delta_with_count
+ *
+ * Execute a query for applying a delta table given by deltname_old
+ * which contains tuples to be deleted from to a materialized view given by
+ * matviewname. This is used when counting is required, that is, the view
+ * has aggregate or distinct.
+ */
+static void
+apply_old_delta_with_count(const char *matviewname, const char *deltaname_old,
+ List *keys, const char *count_colname)
+{
+ StringInfoData querybuf;
+ char *match_cond;
+
+ /* build WHERE condition for searching tuples to be deleted */
+ match_cond = get_matching_condition_string(keys);
+
+ /* Search for matching tuples from the view and update or delete if found. */
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH t AS (" /* collecting tid of target tuples in the view */
+ "SELECT diff.%s, " /* count column */
+ "(diff.%s OPERATOR(pg_catalog.=) mv.%s) AS for_dlt, "
+ "mv.ctid "
+ "FROM %s AS mv, %s AS diff "
+ "WHERE %s" /* tuple matching condition */
+ "), updt AS (" /* update a tuple if this is not to be deleted */
+ "UPDATE %s AS mv SET %s = mv.%s OPERATOR(pg_catalog.-) t.%s "
+ "FROM t WHERE mv.ctid OPERATOR(pg_catalog.=) t.ctid AND NOT for_dlt "
+ ")"
+ /* delete a tuple if this is to be deleted */
+ "DELETE FROM %s AS mv USING t "
+ "WHERE mv.ctid OPERATOR(pg_catalog.=) t.ctid AND for_dlt",
+ count_colname,
+ count_colname, count_colname,
+ matviewname, deltaname_old,
+ match_cond,
+ matviewname, count_colname, count_colname, count_colname,
+ matviewname);
+
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_DELETE)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+}
+
/*
* apply_old_delta
*
@@ -2157,6 +2229,66 @@ apply_old_delta(const char *matviewname, const char *deltaname_old,
elog(ERROR, "SPI_exec failed: %s", querybuf.data);
}
+/*
+ * apply_new_delta_with_count
+ *
+ * Execute a query for applying a delta table given by deltname_new
+ * which contains tuples to be inserted into a materialized view given by
+ * matviewname. This is used when counting is required, that is, the view
+ * has aggregate or distinct. Also, when a table in EXISTS sub queries
+ * is modified.
+ */
+static void
+apply_new_delta_with_count(const char *matviewname, const char* deltaname_new,
+ List *keys, StringInfo target_list, const char* count_colname)
+{
+ StringInfoData querybuf;
+ StringInfoData returning_keys;
+ ListCell *lc;
+ char *match_cond = "";
+
+ /* build WHERE condition for searching tuples to be updated */
+ match_cond = get_matching_condition_string(keys);
+
+ /* build string of keys list */
+ initStringInfo(&returning_keys);
+ if (keys)
+ {
+ foreach (lc, keys)
+ {
+ Form_pg_attribute attr = (Form_pg_attribute) lfirst(lc);
+ char *resname = NameStr(attr->attname);
+ appendStringInfo(&returning_keys, "%s", quote_qualified_identifier("mv", resname));
+ if (lnext(keys, lc))
+ appendStringInfo(&returning_keys, ", ");
+ }
+ }
+ else
+ appendStringInfo(&returning_keys, "NULL");
+
+ /* Search for matching tuples from the view and update if found or insert if not. */
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH updt AS (" /* update a tuple if this exists in the view */
+ "UPDATE %s AS mv SET %s = mv.%s OPERATOR(pg_catalog.+) diff.%s "
+ "FROM %s AS diff "
+ "WHERE %s " /* tuple matching condition */
+ "RETURNING %s" /* returning keys of updated tuples */
+ ") INSERT INTO %s (%s) " /* insert a new tuple if this doesn't exist */
+ "SELECT %s FROM %s AS diff "
+ "WHERE NOT EXISTS (SELECT 1 FROM updt AS mv WHERE %s);",
+ matviewname, count_colname, count_colname, count_colname,
+ deltaname_new,
+ match_cond,
+ returning_keys.data,
+ matviewname, target_list->data,
+ target_list->data, deltaname_new,
+ match_cond);
+
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+}
+
/*
* apply_new_delta
*
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 47c900445c..adbd768e0d 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -53,6 +53,7 @@
#include "commands/cluster.h"
#include "commands/comment.h"
#include "commands/defrem.h"
+#include "commands/matview.h"
#include "commands/event_trigger.h"
#include "commands/policy.h"
#include "commands/sequence.h"
@@ -3673,6 +3674,14 @@ renameatt_internal(Oid myrelid,
targetrelation = relation_open(myrelid, AccessExclusiveLock);
renameatt_check(myrelid, RelationGetForm(targetrelation), recursing);
+ /*
+ * Don't rename IVM columns.
+ */
+ if (RelationIsIVM(targetrelation) && isIvmName(oldattname))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("IVM column can not be renamed")));
+
/*
* if the 'recurse' flag is set then we are supposed to rename this
* attribute in all classes that inherit from 'relname' (as well as in
diff --git a/src/backend/parser/parse_relation.c b/src/backend/parser/parse_relation.c
index 864ea9b0d5..c257440414 100644
--- a/src/backend/parser/parse_relation.c
+++ b/src/backend/parser/parse_relation.c
@@ -36,6 +36,7 @@
#include "utils/rel.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
+#include "commands/matview.h"
/*
@@ -97,7 +98,7 @@ static void expandTupleDesc(TupleDesc tupdesc, Alias *eref,
int count, int offset,
int rtindex, int sublevels_up,
int location, bool include_dropped,
- List **colnames, List **colvars);
+ List **colnames, List **colvars, bool is_ivm);
static int specialAttNum(const char *attname);
static bool rte_visible_if_lateral(ParseState *pstate, RangeTblEntry *rte);
static bool rte_visible_if_qualified(ParseState *pstate, RangeTblEntry *rte);
@@ -1502,6 +1503,7 @@ addRangeTableEntry(ParseState *pstate,
rte->relid = RelationGetRelid(rel);
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -1587,6 +1589,7 @@ addRangeTableEntryForRelation(ParseState *pstate,
rte->relid = RelationGetRelid(rel);
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -2758,7 +2761,7 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
expandTupleDesc(tupdesc, rte->eref,
rtfunc->funccolcount, atts_done,
rtindex, sublevels_up, location,
- include_dropped, colnames, colvars);
+ include_dropped, colnames, colvars, false);
}
else if (functypclass == TYPEFUNC_SCALAR)
{
@@ -3026,7 +3029,7 @@ expandRelation(Oid relid, Alias *eref, int rtindex, int sublevels_up,
expandTupleDesc(rel->rd_att, eref, rel->rd_att->natts, 0,
rtindex, sublevels_up,
location, include_dropped,
- colnames, colvars);
+ colnames, colvars, RelationIsIVM(rel));
relation_close(rel, AccessShareLock);
}
@@ -3043,7 +3046,7 @@ static void
expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
int rtindex, int sublevels_up,
int location, bool include_dropped,
- List **colnames, List **colvars)
+ List **colnames, List **colvars, bool is_ivm)
{
ListCell *aliascell;
int varattno;
@@ -3056,6 +3059,9 @@ expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
{
Form_pg_attribute attr = TupleDescAttr(tupdesc, varattno);
+ if (is_ivm && isIvmName(NameStr(attr->attname)) && !MatViewIncrementalMaintenanceIsEnabled())
+ continue;
+
if (attr->attisdropped)
{
if (include_dropped)
@@ -3218,6 +3224,10 @@ expandNSItemAttrs(ParseState *pstate, ParseNamespaceItem *nsitem,
Var *varnode = (Var *) lfirst(var);
TargetEntry *te;
+ /* if transform * into columnlist with IMMV, remove IVM columns */
+ if (rte->relisivm && isIvmName(label) && !MatViewIncrementalMaintenanceIsEnabled())
+ continue;
+
te = makeTargetEntry((Expr *) varnode,
(AttrNumber) pstate->p_next_resno++,
label,
diff --git a/src/backend/rewrite/rewriteDefine.c b/src/backend/rewrite/rewriteDefine.c
index e36fc72e1e..f6dc7ba202 100644
--- a/src/backend/rewrite/rewriteDefine.c
+++ b/src/backend/rewrite/rewriteDefine.c
@@ -621,7 +621,8 @@ checkRuleResultList(List *targetList, TupleDesc resultDesc, bool isSelect,
attr->atttypmod))));
}
- if (i != resultDesc->natts)
+ /* No check for materialized views since this could have special columns for IVM */
+ if ((!isSelect || requireColumnNameMatch) && i != resultDesc->natts)
ereport(ERROR,
(errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
isSelect ?
diff --git a/src/include/commands/createas.h b/src/include/commands/createas.h
index 09a64fa2e5..76a7873ebf 100644
--- a/src/include/commands/createas.h
+++ b/src/include/commands/createas.h
@@ -29,6 +29,8 @@ extern ObjectAddress ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *st
extern void CreateIvmTriggersOnBaseTables(Query *qry, Oid matviewOid);
extern void CreateIndexOnIMMV(Query *query, Relation matviewRel);
+extern Query *rewriteQueryForIMMV(Query *query, List *colNames);
+
extern int GetIntoRelEFlags(IntoClause *intoClause);
extern DestReceiver *CreateIntoRelDestReceiver(IntoClause *intoClause);
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index fef4c714b8..1a2b8fa09e 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -1073,6 +1073,7 @@ typedef struct RangeTblEntry
int rellockmode; /* lock level that query requires on the rel */
struct TableSampleClause *tablesample; /* sampling info, or NULL */
Index perminfoindex;
+ bool relisivm;
/*
* Fields valid for a subquery RTE (else NULL):
--
2.25.1
--Multipart=_Mon__28_Aug_2023_16_05_30_+0900_b1OvQD_3A3ZMTGvj
Content-Type: text/x-diff;
name="v29-0008-Add-aggregates-support-in-IVM.patch"
Content-Disposition: attachment;
filename="v29-0008-Add-aggregates-support-in-IVM.patch"
Content-Transfer-Encoding: quoted-printable
^ permalink raw reply [nested|flat] 24+ messages in thread
* [PATCH v37 07/11] Add DISTINCT support for IVM
@ 2023-05-31 10:08 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 24+ messages in thread
From: Yugo Nagata @ 2023-05-31 10:08 UTC (permalink / raw)
When IMMV is created with DISTINCT, multiplicity of tuples is
counted and stored in "__ivm_count__" column, which is a hidden
column of IMMV. The value in __ivm_count__ is updated when IMMV
is maintained incrementally. A tuple in IMMV can be removed if
and only if the count becomes zero.
---
src/backend/commands/createas.c | 141 +++++++++++++++++++++------
src/backend/commands/indexcmds.c | 40 ++++++++
src/backend/commands/matview.c | 145 ++++++++++++++++++++++++++--
src/backend/commands/tablecmds.c | 9 ++
src/backend/nodes/outfuncs.c | 1 +
src/backend/nodes/readfuncs.c | 1 +
src/backend/parser/parse_relation.c | 18 +++-
src/backend/rewrite/rewriteDefine.c | 3 +-
src/include/commands/createas.h | 2 +
src/include/nodes/parsenodes.h | 2 +
10 files changed, 319 insertions(+), 43 deletions(-)
diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c
index a499688b79d..cd8db0059f9 100644
--- a/src/backend/commands/createas.c
+++ b/src/backend/commands/createas.c
@@ -55,6 +55,7 @@
#include "parser/parser.h"
#include "parser/parsetree.h"
#include "parser/parse_clause.h"
+#include "parser/parse_func.h"
#include "rewrite/rewriteHandler.h"
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
@@ -307,6 +308,9 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt,
errhint("functions must be marked IMMUTABLE")));
check_ivm_restriction((Node *) query);
+
+ /* For IMMV, we need to rewrite matview query */
+ query = rewriteQueryForIMMV(query, into->colNames);
}
if (into->skipData)
@@ -416,6 +420,49 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt,
return address;
}
+/*
+ * rewriteQueryForIMMV -- rewrite view definition query for IMMV
+ *
+ * count(*) is added for counting distinct tuples in views.
+ */
+Query *
+rewriteQueryForIMMV(Query *query, List *colNames)
+{
+ Query *rewritten;
+
+ Node *node;
+ ParseState *pstate = make_parsestate(NULL);
+ FuncCall *fn;
+
+ rewritten = copyObject(query);
+ pstate->p_expr_kind = EXPR_KIND_SELECT_TARGET;
+
+ /*
+ * Convert DISTINCT to GROUP BY and add count(*) for counting distinct
+ * tuples in views.
+ */
+ if (rewritten->distinctClause)
+ {
+ TargetEntry *tle;
+
+ rewritten->groupClause = transformDistinctClause(NULL, &rewritten->targetList, rewritten->sortClause, false);
+
+ fn = makeFuncCall(SystemFuncName("count"), NIL, COERCE_EXPLICIT_CALL, -1);
+ fn->agg_star = true;
+
+ node = ParseFuncOrColumn(pstate, fn->funcname, NIL, NULL, fn, false, -1);
+
+ tle = makeTargetEntry((Expr *) node,
+ list_length(rewritten->targetList) + 1,
+ pstrdup("__ivm_count__"),
+ false);
+ rewritten->targetList = lappend(rewritten->targetList, tle);
+ rewritten->hasAggs = true;
+ }
+
+ return rewritten;
+}
+
/*
* GetIntoRelEFlags --- compute executor flags needed for CREATE TABLE AS
*
@@ -539,7 +586,8 @@ intorel_startup(DestReceiver *self, int operation, TupleDesc typeinfo)
ColumnDef *col;
char *colname;
- if (lc)
+ /* Don't override hidden columns added for IVM */
+ if (lc && !isIvmName(NameStr(attribute->attname)))
{
colname = strVal(lfirst(lc));
lc = lnext(into->colNames, lc);
@@ -943,10 +991,6 @@ check_ivm_restriction_walker(Node *node, void *context)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("LIMIT/OFFSET clause is not supported on incrementally maintainable materialized view")));
- if (qry->distinctClause)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("DISTINCT is not supported on incrementally maintainable materialized view")));
if (qry->hasDistinctOn)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -1108,12 +1152,18 @@ CreateIndexOnIMMV(Query *query, Relation matviewRel)
char idxname[NAMEDATALEN];
List *indexoidlist = RelationGetIndexList(matviewRel);
ListCell *indexoidscan;
- Bitmapset *key_attnos;
snprintf(idxname, sizeof(idxname), "%s_index", RelationGetRelationName(matviewRel));
index = makeNode(IndexStmt);
+ /*
+ * We consider null values not distinct to make sure that views with DISTINCT
+ * or GROUP BY don't contain multiple NULL rows when NULL is inserted to
+ * a base table concurrently.
+ */
+ index->nulls_not_distinct = true;
+
index->unique = true;
index->primary = false;
index->isconstraint = false;
@@ -1140,41 +1190,68 @@ CreateIndexOnIMMV(Query *query, Relation matviewRel)
index->concurrent = false;
index->if_not_exists = false;
- /* create index on the base tables' primary key columns */
- key_attnos = get_primary_key_attnos_from_query(query, &constraintList);
- if (key_attnos)
+ if (query->distinctClause)
{
+ /* create unique constraint on all columns */
foreach(lc, query->targetList)
{
TargetEntry *tle = (TargetEntry *) lfirst(lc);
Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, tle->resno - 1);
-
- if (bms_is_member(tle->resno - FirstLowInvalidHeapAttributeNumber, key_attnos))
- {
- IndexElem *iparam;
-
- iparam = makeNode(IndexElem);
- iparam->name = pstrdup(NameStr(attr->attname));
- iparam->expr = NULL;
- iparam->indexcolname = NULL;
- iparam->collation = NIL;
- iparam->opclass = NIL;
- iparam->opclassopts = NIL;
- iparam->ordering = SORTBY_DEFAULT;
- iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
- index->indexParams = lappend(index->indexParams, iparam);
- }
+ IndexElem *iparam;
+
+ iparam = makeNode(IndexElem);
+ iparam->name = pstrdup(NameStr(attr->attname));
+ iparam->expr = NULL;
+ iparam->indexcolname = NULL;
+ iparam->collation = NIL;
+ iparam->opclass = NIL;
+ iparam->opclassopts = NIL;
+ iparam->ordering = SORTBY_DEFAULT;
+ iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
+ index->indexParams = lappend(index->indexParams, iparam);
}
}
else
{
- /* create no index, just notice that an appropriate index is necessary for efficient IVM */
- ereport(NOTICE,
- (errmsg("could not create an index on materialized view \"%s\" automatically",
- RelationGetRelationName(matviewRel)),
- errdetail("This target list does not have all the primary key columns. "),
- errhint("Create an index on the materialized view for efficient incremental maintenance.")));
- return;
+ Bitmapset *key_attnos;
+
+ /* create index on the base tables' primary key columns */
+ key_attnos = get_primary_key_attnos_from_query(query, &constraintList);
+ if (key_attnos)
+ {
+ foreach(lc, query->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);
+ Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, tle->resno - 1);
+
+ if (bms_is_member(tle->resno - FirstLowInvalidHeapAttributeNumber, key_attnos))
+ {
+ IndexElem *iparam;
+
+ iparam = makeNode(IndexElem);
+ iparam->name = pstrdup(NameStr(attr->attname));
+ iparam->expr = NULL;
+ iparam->indexcolname = NULL;
+ iparam->collation = NIL;
+ iparam->opclass = NIL;
+ iparam->opclassopts = NIL;
+ iparam->ordering = SORTBY_DEFAULT;
+ iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
+ index->indexParams = lappend(index->indexParams, iparam);
+ }
+ }
+ }
+ else
+ {
+ /* create no index, just notice that an appropriate index is necessary for efficient IVM */
+ ereport(NOTICE,
+ (errmsg("could not create an index on materialized view \"%s\" automatically",
+ RelationGetRelationName(matviewRel)),
+ errdetail("This target list does not have all the primary key columns, "
+ "or this view does not contain DISTINCT clause."),
+ errhint("Create an index on the materialized view for efficient incremental maintenance.")));
+ return;
+ }
}
/* If we have a compatible index, we don't need to create another. */
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 9ab74c8df0a..1f95e6b229f 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -41,6 +41,7 @@
#include "commands/comment.h"
#include "commands/defrem.h"
#include "commands/event_trigger.h"
+#include "commands/matview.h"
#include "commands/progress.h"
#include "commands/tablecmds.h"
#include "commands/tablespace.h"
@@ -1179,6 +1180,45 @@ DefineIndex(ParseState *pstate,
safe_index = indexInfo->ii_Expressions == NIL &&
indexInfo->ii_Predicate == NIL;
+ /*
+ * We disallow unique indexes on IVM columns of IMMVs.
+ */
+ if (RelationIsIVM(rel) && stmt->unique)
+ {
+ for (int i = 0; i < indexInfo->ii_NumIndexKeyAttrs; i++)
+ {
+ AttrNumber attno = indexInfo->ii_IndexAttrNumbers[i];
+ if (attno > 0)
+ {
+ char *name = NameStr(TupleDescAttr(rel->rd_att, attno - 1)->attname);
+ if (name && isIvmName(name))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("unique index creation on IVM columns is not supported")));
+ }
+ }
+
+ if (indexInfo->ii_Expressions)
+ {
+ Bitmapset *indexattrs = NULL;
+ int varno = -1;
+
+ pull_varattnos((Node *) indexInfo->ii_Expressions, 1, &indexattrs);
+
+ while ((varno = bms_next_member(indexattrs, varno)) >= 0)
+ {
+ int attno = varno + FirstLowInvalidHeapAttributeNumber;
+ char *name = NameStr(TupleDescAttr(rel->rd_att, attno - 1)->attname);
+ if (name && isIvmName(name))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("unique index creation on IVM columns is not supported")));
+ }
+
+ }
+ }
+
+
/*
* Report index creation if appropriate (delay this till after most of the
* error checks)
diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c
index eaf39f80cd3..a2746ca9265 100644
--- a/src/backend/commands/matview.c
+++ b/src/backend/commands/matview.c
@@ -193,11 +193,15 @@ static Query *rewrite_query_for_postupdate_state(Query *query, MV_TriggerTable *
static void apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *new_tuplestores,
TupleDesc tupdesc_old, TupleDesc tupdesc_new,
- Query *query);
+ Query *query, bool use_count, char *count_colname);
static void apply_old_delta(const char *matviewname, const char *deltaname_old,
List *keys);
+static void apply_old_delta_with_count(const char *matviewname, const char *deltaname_old,
+ List *keys, const char *count_colname);
static void apply_new_delta(const char *matviewname, const char *deltaname_new,
StringInfo target_list);
+static void apply_new_delta_with_count(const char *matviewname, const char* deltaname_new,
+ List *keys, StringInfo target_list, const char* count_colname);
static char *get_matching_condition_string(List *keys);
static void generate_equal(StringInfo querybuf, Oid opttype,
const char *leftop, const char *rightop);
@@ -346,6 +350,7 @@ RefreshMatViewByOid(Oid matviewOid, bool is_create, bool skipData,
QueryCompletion *qc)
{
Relation matviewRel;
+ Query *viewQuery;
Query *dataQuery;
Oid tableSpace;
Oid relowner;
@@ -394,7 +399,13 @@ RefreshMatViewByOid(Oid matviewOid, bool is_create, bool skipData,
errmsg("%s options %s and %s cannot be used together",
"REFRESH", "CONCURRENTLY", "WITH NO DATA")));
- dataQuery = get_matview_query(matviewRel);
+ viewQuery = get_matview_query(matviewRel);
+
+ /* For IMMV, we need to rewrite matview query */
+ if (!skipData && RelationIsIVM(matviewRel))
+ dataQuery = rewriteQueryForIMMV(viewQuery,NIL);
+ else
+ dataQuery = viewQuery;
/*
* Check that there is a unique index with no WHERE clause on one or more
@@ -1688,6 +1699,13 @@ IVM_immediate_maintenance(PG_FUNCTION_ARGS)
int rte_index = lfirst_int(lc2);
TupleDesc tupdesc_old;
TupleDesc tupdesc_new;
+ bool use_count = false;
+ char *count_colname = NULL;
+
+ count_colname = pstrdup("__ivm_count__");
+
+ if (query->distinctClause)
+ use_count = true;
/* calculate delta tables */
calc_delta(table, rte_index, rewritten, dest_old, dest_new,
@@ -1700,7 +1718,8 @@ IVM_immediate_maintenance(PG_FUNCTION_ARGS)
{
/* apply the delta tables to the materialized view */
apply_delta(matviewOid, old_tuplestore, new_tuplestore,
- tupdesc_old, tupdesc_new, query);
+ tupdesc_old, tupdesc_new, query, use_count,
+ count_colname);
}
PG_CATCH();
{
@@ -2238,7 +2257,7 @@ rewrite_query_for_postupdate_state(Query *query, MV_TriggerTable *table, int rte
static void
apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *new_tuplestores,
TupleDesc tupdesc_old, TupleDesc tupdesc_new,
- Query *query)
+ Query *query, bool use_count, char *count_colname)
{
StringInfoData querybuf;
StringInfoData target_list_buf;
@@ -2314,7 +2333,12 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
if (rc != SPI_OK_REL_REGISTER)
elog(ERROR, "SPI_register failed");
- apply_old_delta(matviewname, OLD_DELTA_ENRNAME, keys);
+ if (use_count)
+ /* apply old delta and get rows to be recalculated */
+ apply_old_delta_with_count(matviewname, OLD_DELTA_ENRNAME,
+ keys, count_colname);
+ else
+ apply_old_delta(matviewname, OLD_DELTA_ENRNAME, keys);
}
/* For tuple insertion */
@@ -2336,7 +2360,11 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
elog(ERROR, "SPI_register failed");
/* apply new delta */
- apply_new_delta(matviewname, NEW_DELTA_ENRNAME, &target_list_buf);
+ if (use_count)
+ apply_new_delta_with_count(matviewname, NEW_DELTA_ENRNAME,
+ keys, &target_list_buf, count_colname);
+ else
+ apply_new_delta(matviewname, NEW_DELTA_ENRNAME, &target_list_buf);
}
/* We're done maintaining the materialized view. */
@@ -2349,6 +2377,51 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
elog(ERROR, "SPI_finish failed");
}
+/*
+ * apply_old_delta_with_count
+ *
+ * Execute a query for applying a delta table given by deltname_old
+ * which contains tuples to be deleted from to a materialized view given by
+ * matviewname. This is used when counting is required, that is, the view
+ * has aggregate or distinct.
+ */
+static void
+apply_old_delta_with_count(const char *matviewname, const char *deltaname_old,
+ List *keys, const char *count_colname)
+{
+ StringInfoData querybuf;
+ char *match_cond;
+
+ /* build WHERE condition for searching tuples to be deleted */
+ match_cond = get_matching_condition_string(keys);
+
+ /* Search for matching tuples from the view and update or delete if found. */
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH t AS (" /* collecting tid of target tuples in the view */
+ "SELECT diff.%s, " /* count column */
+ "(diff.%s OPERATOR(pg_catalog.=) mv.%s) AS for_dlt, "
+ "mv.ctid "
+ "FROM %s AS mv, %s AS diff "
+ "WHERE %s" /* tuple matching condition */
+ "), updt AS (" /* update a tuple if this is not to be deleted */
+ "UPDATE %s AS mv SET %s = mv.%s OPERATOR(pg_catalog.-) t.%s "
+ "FROM t WHERE mv.ctid OPERATOR(pg_catalog.=) t.ctid AND NOT for_dlt "
+ ")"
+ /* delete a tuple if this is to be deleted */
+ "DELETE FROM %s AS mv USING t "
+ "WHERE mv.ctid OPERATOR(pg_catalog.=) t.ctid AND for_dlt",
+ count_colname,
+ count_colname, count_colname,
+ matviewname, deltaname_old,
+ match_cond,
+ matviewname, count_colname, count_colname, count_colname,
+ matviewname);
+
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_DELETE)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+}
+
/*
* apply_old_delta
*
@@ -2398,6 +2471,66 @@ apply_old_delta(const char *matviewname, const char *deltaname_old,
elog(ERROR, "SPI_exec failed: %s", querybuf.data);
}
+/*
+ * apply_new_delta_with_count
+ *
+ * Execute a query for applying a delta table given by deltname_new
+ * which contains tuples to be inserted into a materialized view given by
+ * matviewname. This is used when counting is required, that is, the view
+ * has aggregate or distinct. Also, when a table in EXISTS sub queries
+ * is modified.
+ */
+static void
+apply_new_delta_with_count(const char *matviewname, const char* deltaname_new,
+ List *keys, StringInfo target_list, const char* count_colname)
+{
+ StringInfoData querybuf;
+ StringInfoData returning_keys;
+ ListCell *lc;
+ char *match_cond = "";
+
+ /* build WHERE condition for searching tuples to be updated */
+ match_cond = get_matching_condition_string(keys);
+
+ /* build string of keys list */
+ initStringInfo(&returning_keys);
+ if (keys)
+ {
+ foreach (lc, keys)
+ {
+ Form_pg_attribute attr = (Form_pg_attribute) lfirst(lc);
+ char *resname = NameStr(attr->attname);
+ appendStringInfo(&returning_keys, "%s", quote_qualified_identifier("mv", resname));
+ if (lnext(keys, lc))
+ appendStringInfo(&returning_keys, ", ");
+ }
+ }
+ else
+ appendStringInfo(&returning_keys, "NULL");
+
+ /* Search for matching tuples from the view and update if found or insert if not. */
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH updt AS (" /* update a tuple if this exists in the view */
+ "UPDATE %s AS mv SET %s = mv.%s OPERATOR(pg_catalog.+) diff.%s "
+ "FROM %s AS diff "
+ "WHERE %s " /* tuple matching condition */
+ "RETURNING %s" /* returning keys of updated tuples */
+ ") INSERT INTO %s (%s) " /* insert a new tuple if this doesn't exist */
+ "SELECT %s FROM %s AS diff "
+ "WHERE NOT EXISTS (SELECT 1 FROM updt AS mv WHERE %s);",
+ matviewname, count_colname, count_colname, count_colname,
+ deltaname_new,
+ match_cond,
+ returning_keys.data,
+ matviewname, target_list->data,
+ target_list->data, deltaname_new,
+ match_cond);
+
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+}
+
/*
* apply_new_delta
*
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 99bc709959a..ee65292deb0 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -60,6 +60,7 @@
#include "catalog/toasting.h"
#include "commands/comment.h"
#include "commands/defrem.h"
+#include "commands/matview.h"
#include "commands/event_trigger.h"
#include "commands/extension.h"
#include "commands/matview.h"
@@ -3923,6 +3924,14 @@ renameatt_internal(Oid myrelid,
targetrelation = relation_open(myrelid, AccessExclusiveLock);
renameatt_check(myrelid, RelationGetForm(targetrelation), recursing);
+ /*
+ * Don't rename IVM columns.
+ */
+ if (RelationIsIVM(targetrelation) && isIvmName(oldattname))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("IVM column can not be renamed")));
+
/*
* if the 'recurse' flag is set then we are supposed to rename this
* attribute in all classes that inherit from 'relname' (as well as in
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 953c5797c5d..0f228eca259 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -516,6 +516,7 @@ _outRangeTblEntry(StringInfo str, const RangeTblEntry *node)
WRITE_INT_FIELD(rellockmode);
WRITE_UINT_FIELD(perminfoindex);
WRITE_NODE_FIELD(tablesample);
+ WRITE_BOOL_FIELD(relisivm);
break;
case RTE_SUBQUERY:
WRITE_NODE_FIELD(subquery);
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index b6b2ce6c792..dba4d8ff2de 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -365,6 +365,7 @@ _readRangeTblEntry(void)
READ_INT_FIELD(rellockmode);
READ_UINT_FIELD(perminfoindex);
READ_NODE_FIELD(tablesample);
+ READ_BOOL_FIELD(relisivm);
break;
case RTE_SUBQUERY:
READ_NODE_FIELD(subquery);
diff --git a/src/backend/parser/parse_relation.c b/src/backend/parser/parse_relation.c
index 43460e4a5a5..dc048c21bed 100644
--- a/src/backend/parser/parse_relation.c
+++ b/src/backend/parser/parse_relation.c
@@ -33,6 +33,7 @@
#include "utils/lsyscache.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
+#include "commands/matview.h"
/*
@@ -96,7 +97,7 @@ static void expandTupleDesc(TupleDesc tupdesc, Alias *eref,
int rtindex, int sublevels_up,
VarReturningType returning_type,
int location, bool include_dropped,
- List **colnames, List **colvars);
+ List **colnames, List **colvars, bool is_ivm);
static int specialAttNum(const char *attname);
static bool rte_visible_if_lateral(ParseState *pstate, RangeTblEntry *rte);
static bool rte_visible_if_qualified(ParseState *pstate, RangeTblEntry *rte);
@@ -1512,6 +1513,7 @@ addRangeTableEntry(ParseState *pstate,
rte->inh = inh;
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -1593,6 +1595,7 @@ addRangeTableEntryForRelation(ParseState *pstate,
rte->inh = inh;
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -2935,7 +2938,7 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
rtfunc->funccolcount, atts_done,
rtindex, sublevels_up,
returning_type, location,
- include_dropped, colnames, colvars);
+ include_dropped, colnames, colvars, false);
}
else if (functypclass == TYPEFUNC_SCALAR)
{
@@ -3211,7 +3214,7 @@ expandRelation(Oid relid, Alias *eref, int rtindex, int sublevels_up,
expandTupleDesc(rel->rd_att, eref, rel->rd_att->natts, 0,
rtindex, sublevels_up, returning_type,
location, include_dropped,
- colnames, colvars);
+ colnames, colvars, RelationIsIVM(rel));
relation_close(rel, AccessShareLock);
}
@@ -3229,7 +3232,7 @@ expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
int rtindex, int sublevels_up,
VarReturningType returning_type,
int location, bool include_dropped,
- List **colnames, List **colvars)
+ List **colnames, List **colvars, bool is_ivm)
{
ListCell *aliascell;
int varattno;
@@ -3242,6 +3245,9 @@ expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
{
Form_pg_attribute attr = TupleDescAttr(tupdesc, varattno);
+ if (is_ivm && isIvmName(NameStr(attr->attname)) && !MatViewIncrementalMaintenanceIsEnabled())
+ continue;
+
if (attr->attisdropped)
{
if (include_dropped)
@@ -3406,6 +3412,10 @@ expandNSItemAttrs(ParseState *pstate, ParseNamespaceItem *nsitem,
Var *varnode = (Var *) lfirst(var);
TargetEntry *te;
+ /* if transform * into columnlist with IMMV, remove IVM columns */
+ if (rte->relisivm && isIvmName(label) && !MatViewIncrementalMaintenanceIsEnabled())
+ continue;
+
te = makeTargetEntry((Expr *) varnode,
(AttrNumber) pstate->p_next_resno++,
label,
diff --git a/src/backend/rewrite/rewriteDefine.c b/src/backend/rewrite/rewriteDefine.c
index 6a223fbeaa4..74ac631299c 100644
--- a/src/backend/rewrite/rewriteDefine.c
+++ b/src/backend/rewrite/rewriteDefine.c
@@ -614,7 +614,8 @@ checkRuleResultList(List *targetList, TupleDesc resultDesc, bool isSelect,
attr->atttypmod))));
}
- if (i != resultDesc->natts)
+ /* No check for materialized views since this could have special columns for IVM */
+ if ((!isSelect || requireColumnNameMatch) && i != resultDesc->natts)
ereport(ERROR,
(errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
isSelect ?
diff --git a/src/include/commands/createas.h b/src/include/commands/createas.h
index c286ebcd70e..bfd0249b10d 100644
--- a/src/include/commands/createas.h
+++ b/src/include/commands/createas.h
@@ -29,6 +29,8 @@ extern ObjectAddress ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *st
extern void CreateIvmTriggersOnBaseTables(Query *qry, Oid matviewOid);
extern void CreateIndexOnIMMV(Query *query, Relation matviewRel);
+extern Query *rewriteQueryForIMMV(Query *query, List *colNames);
+
extern int GetIntoRelEFlags(IntoClause *intoClause);
extern DestReceiver *CreateIntoRelDestReceiver(IntoClause *intoClause);
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 91377a6cde3..59e689a1fac 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -1207,6 +1207,8 @@ typedef struct RangeTblEntry
Index perminfoindex pg_node_attr(query_jumble_ignore);
/* sampling info, or NULL */
struct TableSampleClause *tablesample;
+ /* incrementally maintainable materialized view? */
+ bool relisivm;
/*
* Fields valid for a subquery RTE (else NULL):
--
2.43.0
--Multipart=_Fri__29_May_2026_23_14_17_+0900_Te0o73X2VqYK57Gd
Content-Type: text/x-diff;
name="v37-0006-Add-Incremental-View-Maintenance-support.patch"
Content-Disposition: attachment;
filename="v37-0006-Add-Incremental-View-Maintenance-support.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 24+ messages in thread
* [PATCH v38 07/11] Add DISTINCT support for IVM
@ 2023-05-31 10:08 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 24+ messages in thread
From: Yugo Nagata @ 2023-05-31 10:08 UTC (permalink / raw)
When IMMV is created with DISTINCT, multiplicity of tuples is
counted and stored in "__ivm_count__" column, which is a hidden
column of IMMV. The value in __ivm_count__ is updated when IMMV
is maintained incrementally. A tuple in IMMV can be removed if
and only if the count becomes zero.
---
src/backend/commands/createas.c | 141 +++++++++++++++++++++------
src/backend/commands/indexcmds.c | 40 ++++++++
src/backend/commands/matview.c | 145 ++++++++++++++++++++++++++--
src/backend/commands/tablecmds.c | 9 ++
src/backend/nodes/outfuncs.c | 1 +
src/backend/nodes/readfuncs.c | 1 +
src/backend/parser/parse_relation.c | 18 +++-
src/backend/rewrite/rewriteDefine.c | 3 +-
src/include/commands/createas.h | 2 +
src/include/nodes/parsenodes.h | 2 +
10 files changed, 319 insertions(+), 43 deletions(-)
diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c
index a499688b79d..cd8db0059f9 100644
--- a/src/backend/commands/createas.c
+++ b/src/backend/commands/createas.c
@@ -55,6 +55,7 @@
#include "parser/parser.h"
#include "parser/parsetree.h"
#include "parser/parse_clause.h"
+#include "parser/parse_func.h"
#include "rewrite/rewriteHandler.h"
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
@@ -307,6 +308,9 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt,
errhint("functions must be marked IMMUTABLE")));
check_ivm_restriction((Node *) query);
+
+ /* For IMMV, we need to rewrite matview query */
+ query = rewriteQueryForIMMV(query, into->colNames);
}
if (into->skipData)
@@ -416,6 +420,49 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt,
return address;
}
+/*
+ * rewriteQueryForIMMV -- rewrite view definition query for IMMV
+ *
+ * count(*) is added for counting distinct tuples in views.
+ */
+Query *
+rewriteQueryForIMMV(Query *query, List *colNames)
+{
+ Query *rewritten;
+
+ Node *node;
+ ParseState *pstate = make_parsestate(NULL);
+ FuncCall *fn;
+
+ rewritten = copyObject(query);
+ pstate->p_expr_kind = EXPR_KIND_SELECT_TARGET;
+
+ /*
+ * Convert DISTINCT to GROUP BY and add count(*) for counting distinct
+ * tuples in views.
+ */
+ if (rewritten->distinctClause)
+ {
+ TargetEntry *tle;
+
+ rewritten->groupClause = transformDistinctClause(NULL, &rewritten->targetList, rewritten->sortClause, false);
+
+ fn = makeFuncCall(SystemFuncName("count"), NIL, COERCE_EXPLICIT_CALL, -1);
+ fn->agg_star = true;
+
+ node = ParseFuncOrColumn(pstate, fn->funcname, NIL, NULL, fn, false, -1);
+
+ tle = makeTargetEntry((Expr *) node,
+ list_length(rewritten->targetList) + 1,
+ pstrdup("__ivm_count__"),
+ false);
+ rewritten->targetList = lappend(rewritten->targetList, tle);
+ rewritten->hasAggs = true;
+ }
+
+ return rewritten;
+}
+
/*
* GetIntoRelEFlags --- compute executor flags needed for CREATE TABLE AS
*
@@ -539,7 +586,8 @@ intorel_startup(DestReceiver *self, int operation, TupleDesc typeinfo)
ColumnDef *col;
char *colname;
- if (lc)
+ /* Don't override hidden columns added for IVM */
+ if (lc && !isIvmName(NameStr(attribute->attname)))
{
colname = strVal(lfirst(lc));
lc = lnext(into->colNames, lc);
@@ -943,10 +991,6 @@ check_ivm_restriction_walker(Node *node, void *context)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("LIMIT/OFFSET clause is not supported on incrementally maintainable materialized view")));
- if (qry->distinctClause)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("DISTINCT is not supported on incrementally maintainable materialized view")));
if (qry->hasDistinctOn)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -1108,12 +1152,18 @@ CreateIndexOnIMMV(Query *query, Relation matviewRel)
char idxname[NAMEDATALEN];
List *indexoidlist = RelationGetIndexList(matviewRel);
ListCell *indexoidscan;
- Bitmapset *key_attnos;
snprintf(idxname, sizeof(idxname), "%s_index", RelationGetRelationName(matviewRel));
index = makeNode(IndexStmt);
+ /*
+ * We consider null values not distinct to make sure that views with DISTINCT
+ * or GROUP BY don't contain multiple NULL rows when NULL is inserted to
+ * a base table concurrently.
+ */
+ index->nulls_not_distinct = true;
+
index->unique = true;
index->primary = false;
index->isconstraint = false;
@@ -1140,41 +1190,68 @@ CreateIndexOnIMMV(Query *query, Relation matviewRel)
index->concurrent = false;
index->if_not_exists = false;
- /* create index on the base tables' primary key columns */
- key_attnos = get_primary_key_attnos_from_query(query, &constraintList);
- if (key_attnos)
+ if (query->distinctClause)
{
+ /* create unique constraint on all columns */
foreach(lc, query->targetList)
{
TargetEntry *tle = (TargetEntry *) lfirst(lc);
Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, tle->resno - 1);
-
- if (bms_is_member(tle->resno - FirstLowInvalidHeapAttributeNumber, key_attnos))
- {
- IndexElem *iparam;
-
- iparam = makeNode(IndexElem);
- iparam->name = pstrdup(NameStr(attr->attname));
- iparam->expr = NULL;
- iparam->indexcolname = NULL;
- iparam->collation = NIL;
- iparam->opclass = NIL;
- iparam->opclassopts = NIL;
- iparam->ordering = SORTBY_DEFAULT;
- iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
- index->indexParams = lappend(index->indexParams, iparam);
- }
+ IndexElem *iparam;
+
+ iparam = makeNode(IndexElem);
+ iparam->name = pstrdup(NameStr(attr->attname));
+ iparam->expr = NULL;
+ iparam->indexcolname = NULL;
+ iparam->collation = NIL;
+ iparam->opclass = NIL;
+ iparam->opclassopts = NIL;
+ iparam->ordering = SORTBY_DEFAULT;
+ iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
+ index->indexParams = lappend(index->indexParams, iparam);
}
}
else
{
- /* create no index, just notice that an appropriate index is necessary for efficient IVM */
- ereport(NOTICE,
- (errmsg("could not create an index on materialized view \"%s\" automatically",
- RelationGetRelationName(matviewRel)),
- errdetail("This target list does not have all the primary key columns. "),
- errhint("Create an index on the materialized view for efficient incremental maintenance.")));
- return;
+ Bitmapset *key_attnos;
+
+ /* create index on the base tables' primary key columns */
+ key_attnos = get_primary_key_attnos_from_query(query, &constraintList);
+ if (key_attnos)
+ {
+ foreach(lc, query->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);
+ Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, tle->resno - 1);
+
+ if (bms_is_member(tle->resno - FirstLowInvalidHeapAttributeNumber, key_attnos))
+ {
+ IndexElem *iparam;
+
+ iparam = makeNode(IndexElem);
+ iparam->name = pstrdup(NameStr(attr->attname));
+ iparam->expr = NULL;
+ iparam->indexcolname = NULL;
+ iparam->collation = NIL;
+ iparam->opclass = NIL;
+ iparam->opclassopts = NIL;
+ iparam->ordering = SORTBY_DEFAULT;
+ iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
+ index->indexParams = lappend(index->indexParams, iparam);
+ }
+ }
+ }
+ else
+ {
+ /* create no index, just notice that an appropriate index is necessary for efficient IVM */
+ ereport(NOTICE,
+ (errmsg("could not create an index on materialized view \"%s\" automatically",
+ RelationGetRelationName(matviewRel)),
+ errdetail("This target list does not have all the primary key columns, "
+ "or this view does not contain DISTINCT clause."),
+ errhint("Create an index on the materialized view for efficient incremental maintenance.")));
+ return;
+ }
}
/* If we have a compatible index, we don't need to create another. */
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 9ab74c8df0a..1f95e6b229f 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -41,6 +41,7 @@
#include "commands/comment.h"
#include "commands/defrem.h"
#include "commands/event_trigger.h"
+#include "commands/matview.h"
#include "commands/progress.h"
#include "commands/tablecmds.h"
#include "commands/tablespace.h"
@@ -1179,6 +1180,45 @@ DefineIndex(ParseState *pstate,
safe_index = indexInfo->ii_Expressions == NIL &&
indexInfo->ii_Predicate == NIL;
+ /*
+ * We disallow unique indexes on IVM columns of IMMVs.
+ */
+ if (RelationIsIVM(rel) && stmt->unique)
+ {
+ for (int i = 0; i < indexInfo->ii_NumIndexKeyAttrs; i++)
+ {
+ AttrNumber attno = indexInfo->ii_IndexAttrNumbers[i];
+ if (attno > 0)
+ {
+ char *name = NameStr(TupleDescAttr(rel->rd_att, attno - 1)->attname);
+ if (name && isIvmName(name))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("unique index creation on IVM columns is not supported")));
+ }
+ }
+
+ if (indexInfo->ii_Expressions)
+ {
+ Bitmapset *indexattrs = NULL;
+ int varno = -1;
+
+ pull_varattnos((Node *) indexInfo->ii_Expressions, 1, &indexattrs);
+
+ while ((varno = bms_next_member(indexattrs, varno)) >= 0)
+ {
+ int attno = varno + FirstLowInvalidHeapAttributeNumber;
+ char *name = NameStr(TupleDescAttr(rel->rd_att, attno - 1)->attname);
+ if (name && isIvmName(name))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("unique index creation on IVM columns is not supported")));
+ }
+
+ }
+ }
+
+
/*
* Report index creation if appropriate (delay this till after most of the
* error checks)
diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c
index eaf39f80cd3..a2746ca9265 100644
--- a/src/backend/commands/matview.c
+++ b/src/backend/commands/matview.c
@@ -193,11 +193,15 @@ static Query *rewrite_query_for_postupdate_state(Query *query, MV_TriggerTable *
static void apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *new_tuplestores,
TupleDesc tupdesc_old, TupleDesc tupdesc_new,
- Query *query);
+ Query *query, bool use_count, char *count_colname);
static void apply_old_delta(const char *matviewname, const char *deltaname_old,
List *keys);
+static void apply_old_delta_with_count(const char *matviewname, const char *deltaname_old,
+ List *keys, const char *count_colname);
static void apply_new_delta(const char *matviewname, const char *deltaname_new,
StringInfo target_list);
+static void apply_new_delta_with_count(const char *matviewname, const char* deltaname_new,
+ List *keys, StringInfo target_list, const char* count_colname);
static char *get_matching_condition_string(List *keys);
static void generate_equal(StringInfo querybuf, Oid opttype,
const char *leftop, const char *rightop);
@@ -346,6 +350,7 @@ RefreshMatViewByOid(Oid matviewOid, bool is_create, bool skipData,
QueryCompletion *qc)
{
Relation matviewRel;
+ Query *viewQuery;
Query *dataQuery;
Oid tableSpace;
Oid relowner;
@@ -394,7 +399,13 @@ RefreshMatViewByOid(Oid matviewOid, bool is_create, bool skipData,
errmsg("%s options %s and %s cannot be used together",
"REFRESH", "CONCURRENTLY", "WITH NO DATA")));
- dataQuery = get_matview_query(matviewRel);
+ viewQuery = get_matview_query(matviewRel);
+
+ /* For IMMV, we need to rewrite matview query */
+ if (!skipData && RelationIsIVM(matviewRel))
+ dataQuery = rewriteQueryForIMMV(viewQuery,NIL);
+ else
+ dataQuery = viewQuery;
/*
* Check that there is a unique index with no WHERE clause on one or more
@@ -1688,6 +1699,13 @@ IVM_immediate_maintenance(PG_FUNCTION_ARGS)
int rte_index = lfirst_int(lc2);
TupleDesc tupdesc_old;
TupleDesc tupdesc_new;
+ bool use_count = false;
+ char *count_colname = NULL;
+
+ count_colname = pstrdup("__ivm_count__");
+
+ if (query->distinctClause)
+ use_count = true;
/* calculate delta tables */
calc_delta(table, rte_index, rewritten, dest_old, dest_new,
@@ -1700,7 +1718,8 @@ IVM_immediate_maintenance(PG_FUNCTION_ARGS)
{
/* apply the delta tables to the materialized view */
apply_delta(matviewOid, old_tuplestore, new_tuplestore,
- tupdesc_old, tupdesc_new, query);
+ tupdesc_old, tupdesc_new, query, use_count,
+ count_colname);
}
PG_CATCH();
{
@@ -2238,7 +2257,7 @@ rewrite_query_for_postupdate_state(Query *query, MV_TriggerTable *table, int rte
static void
apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *new_tuplestores,
TupleDesc tupdesc_old, TupleDesc tupdesc_new,
- Query *query)
+ Query *query, bool use_count, char *count_colname)
{
StringInfoData querybuf;
StringInfoData target_list_buf;
@@ -2314,7 +2333,12 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
if (rc != SPI_OK_REL_REGISTER)
elog(ERROR, "SPI_register failed");
- apply_old_delta(matviewname, OLD_DELTA_ENRNAME, keys);
+ if (use_count)
+ /* apply old delta and get rows to be recalculated */
+ apply_old_delta_with_count(matviewname, OLD_DELTA_ENRNAME,
+ keys, count_colname);
+ else
+ apply_old_delta(matviewname, OLD_DELTA_ENRNAME, keys);
}
/* For tuple insertion */
@@ -2336,7 +2360,11 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
elog(ERROR, "SPI_register failed");
/* apply new delta */
- apply_new_delta(matviewname, NEW_DELTA_ENRNAME, &target_list_buf);
+ if (use_count)
+ apply_new_delta_with_count(matviewname, NEW_DELTA_ENRNAME,
+ keys, &target_list_buf, count_colname);
+ else
+ apply_new_delta(matviewname, NEW_DELTA_ENRNAME, &target_list_buf);
}
/* We're done maintaining the materialized view. */
@@ -2349,6 +2377,51 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
elog(ERROR, "SPI_finish failed");
}
+/*
+ * apply_old_delta_with_count
+ *
+ * Execute a query for applying a delta table given by deltname_old
+ * which contains tuples to be deleted from to a materialized view given by
+ * matviewname. This is used when counting is required, that is, the view
+ * has aggregate or distinct.
+ */
+static void
+apply_old_delta_with_count(const char *matviewname, const char *deltaname_old,
+ List *keys, const char *count_colname)
+{
+ StringInfoData querybuf;
+ char *match_cond;
+
+ /* build WHERE condition for searching tuples to be deleted */
+ match_cond = get_matching_condition_string(keys);
+
+ /* Search for matching tuples from the view and update or delete if found. */
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH t AS (" /* collecting tid of target tuples in the view */
+ "SELECT diff.%s, " /* count column */
+ "(diff.%s OPERATOR(pg_catalog.=) mv.%s) AS for_dlt, "
+ "mv.ctid "
+ "FROM %s AS mv, %s AS diff "
+ "WHERE %s" /* tuple matching condition */
+ "), updt AS (" /* update a tuple if this is not to be deleted */
+ "UPDATE %s AS mv SET %s = mv.%s OPERATOR(pg_catalog.-) t.%s "
+ "FROM t WHERE mv.ctid OPERATOR(pg_catalog.=) t.ctid AND NOT for_dlt "
+ ")"
+ /* delete a tuple if this is to be deleted */
+ "DELETE FROM %s AS mv USING t "
+ "WHERE mv.ctid OPERATOR(pg_catalog.=) t.ctid AND for_dlt",
+ count_colname,
+ count_colname, count_colname,
+ matviewname, deltaname_old,
+ match_cond,
+ matviewname, count_colname, count_colname, count_colname,
+ matviewname);
+
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_DELETE)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+}
+
/*
* apply_old_delta
*
@@ -2398,6 +2471,66 @@ apply_old_delta(const char *matviewname, const char *deltaname_old,
elog(ERROR, "SPI_exec failed: %s", querybuf.data);
}
+/*
+ * apply_new_delta_with_count
+ *
+ * Execute a query for applying a delta table given by deltname_new
+ * which contains tuples to be inserted into a materialized view given by
+ * matviewname. This is used when counting is required, that is, the view
+ * has aggregate or distinct. Also, when a table in EXISTS sub queries
+ * is modified.
+ */
+static void
+apply_new_delta_with_count(const char *matviewname, const char* deltaname_new,
+ List *keys, StringInfo target_list, const char* count_colname)
+{
+ StringInfoData querybuf;
+ StringInfoData returning_keys;
+ ListCell *lc;
+ char *match_cond = "";
+
+ /* build WHERE condition for searching tuples to be updated */
+ match_cond = get_matching_condition_string(keys);
+
+ /* build string of keys list */
+ initStringInfo(&returning_keys);
+ if (keys)
+ {
+ foreach (lc, keys)
+ {
+ Form_pg_attribute attr = (Form_pg_attribute) lfirst(lc);
+ char *resname = NameStr(attr->attname);
+ appendStringInfo(&returning_keys, "%s", quote_qualified_identifier("mv", resname));
+ if (lnext(keys, lc))
+ appendStringInfo(&returning_keys, ", ");
+ }
+ }
+ else
+ appendStringInfo(&returning_keys, "NULL");
+
+ /* Search for matching tuples from the view and update if found or insert if not. */
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH updt AS (" /* update a tuple if this exists in the view */
+ "UPDATE %s AS mv SET %s = mv.%s OPERATOR(pg_catalog.+) diff.%s "
+ "FROM %s AS diff "
+ "WHERE %s " /* tuple matching condition */
+ "RETURNING %s" /* returning keys of updated tuples */
+ ") INSERT INTO %s (%s) " /* insert a new tuple if this doesn't exist */
+ "SELECT %s FROM %s AS diff "
+ "WHERE NOT EXISTS (SELECT 1 FROM updt AS mv WHERE %s);",
+ matviewname, count_colname, count_colname, count_colname,
+ deltaname_new,
+ match_cond,
+ returning_keys.data,
+ matviewname, target_list->data,
+ target_list->data, deltaname_new,
+ match_cond);
+
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+}
+
/*
* apply_new_delta
*
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 920b62d254d..f62765f0a89 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -60,6 +60,7 @@
#include "catalog/toasting.h"
#include "commands/comment.h"
#include "commands/defrem.h"
+#include "commands/matview.h"
#include "commands/event_trigger.h"
#include "commands/extension.h"
#include "commands/matview.h"
@@ -3930,6 +3931,14 @@ renameatt_internal(Oid myrelid,
targetrelation = relation_open(myrelid, AccessExclusiveLock);
renameatt_check(myrelid, RelationGetForm(targetrelation), recursing);
+ /*
+ * Don't rename IVM columns.
+ */
+ if (RelationIsIVM(targetrelation) && isIvmName(oldattname))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("IVM column can not be renamed")));
+
/*
* if the 'recurse' flag is set then we are supposed to rename this
* attribute in all classes that inherit from 'relname' (as well as in
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 953c5797c5d..0f228eca259 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -516,6 +516,7 @@ _outRangeTblEntry(StringInfo str, const RangeTblEntry *node)
WRITE_INT_FIELD(rellockmode);
WRITE_UINT_FIELD(perminfoindex);
WRITE_NODE_FIELD(tablesample);
+ WRITE_BOOL_FIELD(relisivm);
break;
case RTE_SUBQUERY:
WRITE_NODE_FIELD(subquery);
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index b6b2ce6c792..dba4d8ff2de 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -365,6 +365,7 @@ _readRangeTblEntry(void)
READ_INT_FIELD(rellockmode);
READ_UINT_FIELD(perminfoindex);
READ_NODE_FIELD(tablesample);
+ READ_BOOL_FIELD(relisivm);
break;
case RTE_SUBQUERY:
READ_NODE_FIELD(subquery);
diff --git a/src/backend/parser/parse_relation.c b/src/backend/parser/parse_relation.c
index ced210cd206..5392e9f15b2 100644
--- a/src/backend/parser/parse_relation.c
+++ b/src/backend/parser/parse_relation.c
@@ -33,6 +33,7 @@
#include "utils/lsyscache.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
+#include "commands/matview.h"
/*
@@ -96,7 +97,7 @@ static void expandTupleDesc(TupleDesc tupdesc, Alias *eref,
int rtindex, int sublevels_up,
VarReturningType returning_type,
int location, bool include_dropped,
- List **colnames, List **colvars);
+ List **colnames, List **colvars, bool is_ivm);
static int specialAttNum(const char *attname);
static bool rte_visible_if_lateral(ParseState *pstate, RangeTblEntry *rte);
static bool rte_visible_if_qualified(ParseState *pstate, RangeTblEntry *rte);
@@ -1544,6 +1545,7 @@ addRangeTableEntry(ParseState *pstate,
rte->inh = inh;
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -1625,6 +1627,7 @@ addRangeTableEntryForRelation(ParseState *pstate,
rte->inh = inh;
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -2967,7 +2970,7 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
rtfunc->funccolcount, atts_done,
rtindex, sublevels_up,
returning_type, location,
- include_dropped, colnames, colvars);
+ include_dropped, colnames, colvars, false);
}
else if (functypclass == TYPEFUNC_SCALAR)
{
@@ -3243,7 +3246,7 @@ expandRelation(Oid relid, Alias *eref, int rtindex, int sublevels_up,
expandTupleDesc(rel->rd_att, eref, rel->rd_att->natts, 0,
rtindex, sublevels_up, returning_type,
location, include_dropped,
- colnames, colvars);
+ colnames, colvars, RelationIsIVM(rel));
relation_close(rel, AccessShareLock);
}
@@ -3261,7 +3264,7 @@ expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
int rtindex, int sublevels_up,
VarReturningType returning_type,
int location, bool include_dropped,
- List **colnames, List **colvars)
+ List **colnames, List **colvars, bool is_ivm)
{
ListCell *aliascell;
int varattno;
@@ -3274,6 +3277,9 @@ expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
{
Form_pg_attribute attr = TupleDescAttr(tupdesc, varattno);
+ if (is_ivm && isIvmName(NameStr(attr->attname)) && !MatViewIncrementalMaintenanceIsEnabled())
+ continue;
+
if (attr->attisdropped)
{
if (include_dropped)
@@ -3438,6 +3444,10 @@ expandNSItemAttrs(ParseState *pstate, ParseNamespaceItem *nsitem,
Var *varnode = (Var *) lfirst(var);
TargetEntry *te;
+ /* if transform * into columnlist with IMMV, remove IVM columns */
+ if (rte->relisivm && isIvmName(label) && !MatViewIncrementalMaintenanceIsEnabled())
+ continue;
+
te = makeTargetEntry((Expr *) varnode,
(AttrNumber) pstate->p_next_resno++,
label,
diff --git a/src/backend/rewrite/rewriteDefine.c b/src/backend/rewrite/rewriteDefine.c
index 6a223fbeaa4..74ac631299c 100644
--- a/src/backend/rewrite/rewriteDefine.c
+++ b/src/backend/rewrite/rewriteDefine.c
@@ -614,7 +614,8 @@ checkRuleResultList(List *targetList, TupleDesc resultDesc, bool isSelect,
attr->atttypmod))));
}
- if (i != resultDesc->natts)
+ /* No check for materialized views since this could have special columns for IVM */
+ if ((!isSelect || requireColumnNameMatch) && i != resultDesc->natts)
ereport(ERROR,
(errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
isSelect ?
diff --git a/src/include/commands/createas.h b/src/include/commands/createas.h
index c286ebcd70e..bfd0249b10d 100644
--- a/src/include/commands/createas.h
+++ b/src/include/commands/createas.h
@@ -29,6 +29,8 @@ extern ObjectAddress ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *st
extern void CreateIvmTriggersOnBaseTables(Query *qry, Oid matviewOid);
extern void CreateIndexOnIMMV(Query *query, Relation matviewRel);
+extern Query *rewriteQueryForIMMV(Query *query, List *colNames);
+
extern int GetIntoRelEFlags(IntoClause *intoClause);
extern DestReceiver *CreateIntoRelDestReceiver(IntoClause *intoClause);
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 4133c404a6b..cbcef860061 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -1209,6 +1209,8 @@ typedef struct RangeTblEntry
Index perminfoindex pg_node_attr(query_jumble_ignore);
/* sampling info, or NULL */
struct TableSampleClause *tablesample;
+ /* incrementally maintainable materialized view? */
+ bool relisivm;
/*
* Fields valid for a subquery RTE (else NULL):
--
2.43.0
--Multipart=_Wed__1_Jul_2026_00_04_01_+0900_OVSy2WWK_9aByzDJ
Content-Type: text/x-diff;
name="v38-0006-Add-Incremental-View-Maintenance-support.patch"
Content-Disposition: attachment;
filename="v38-0006-Add-Incremental-View-Maintenance-support.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 24+ messages in thread
* [PATCH v30 07/11] Add DISTINCT support for IVM
@ 2023-05-31 10:08 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 24+ messages in thread
From: Yugo Nagata @ 2023-05-31 10:08 UTC (permalink / raw)
When IMMV is created with DISTINCT, multiplicity of tuples is
counted and stored in "__ivm_count__" column, which is a hidden
column of IMMV. The value in __ivm_count__ is updated when IMMV
is maintained incrementally. A tuple in IMMV can be removed if
and only if the count becomes zero.
---
src/backend/commands/createas.c | 141 ++++++++++++++++++++------
src/backend/commands/indexcmds.c | 40 ++++++++
src/backend/commands/matview.c | 148 ++++++++++++++++++++++++++--
src/backend/commands/tablecmds.c | 9 ++
src/backend/parser/parse_relation.c | 18 +++-
src/backend/rewrite/rewriteDefine.c | 3 +-
src/include/commands/createas.h | 2 +
src/include/nodes/parsenodes.h | 1 +
8 files changed, 317 insertions(+), 45 deletions(-)
diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c
index 08262100ea..cce44278fa 100644
--- a/src/backend/commands/createas.c
+++ b/src/backend/commands/createas.c
@@ -53,6 +53,7 @@
#include "parser/parser.h"
#include "parser/parsetree.h"
#include "parser/parse_clause.h"
+#include "parser/parse_func.h"
#include "rewrite/rewriteHandler.h"
#include "storage/smgr.h"
#include "tcop/tcopprot.h"
@@ -309,6 +310,9 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt,
errhint("functions must be marked IMMUTABLE")));
check_ivm_restriction((Node *) query);
+
+ /* For IMMV, we need to rewrite matview query */
+ query = rewriteQueryForIMMV(query, into->colNames);
}
if (into->skipData)
@@ -413,6 +417,49 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt,
return address;
}
+/*
+ * rewriteQueryForIMMV -- rewrite view definition query for IMMV
+ *
+ * count(*) is added for counting distinct tuples in views.
+ */
+Query *
+rewriteQueryForIMMV(Query *query, List *colNames)
+{
+ Query *rewritten;
+
+ Node *node;
+ ParseState *pstate = make_parsestate(NULL);
+ FuncCall *fn;
+
+ rewritten = copyObject(query);
+ pstate->p_expr_kind = EXPR_KIND_SELECT_TARGET;
+
+ /*
+ * Convert DISTINCT to GROUP BY and add count(*) for counting distinct
+ * tuples in views.
+ */
+ if (rewritten->distinctClause)
+ {
+ TargetEntry *tle;
+
+ rewritten->groupClause = transformDistinctClause(NULL, &rewritten->targetList, rewritten->sortClause, false);
+
+ fn = makeFuncCall(SystemFuncName("count"), NIL, COERCE_EXPLICIT_CALL, -1);
+ fn->agg_star = true;
+
+ node = ParseFuncOrColumn(pstate, fn->funcname, NIL, NULL, fn, false, -1);
+
+ tle = makeTargetEntry((Expr *) node,
+ list_length(rewritten->targetList) + 1,
+ pstrdup("__ivm_count__"),
+ false);
+ rewritten->targetList = lappend(rewritten->targetList, tle);
+ rewritten->hasAggs = true;
+ }
+
+ return rewritten;
+}
+
/*
* GetIntoRelEFlags --- compute executor flags needed for CREATE TABLE AS
*
@@ -536,7 +583,8 @@ intorel_startup(DestReceiver *self, int operation, TupleDesc typeinfo)
ColumnDef *col;
char *colname;
- if (lc)
+ /* Don't override hidden columns added for IVM */
+ if (lc && !isIvmName(NameStr(attribute->attname)))
{
colname = strVal(lfirst(lc));
lc = lnext(into->colNames, lc);
@@ -940,10 +988,6 @@ check_ivm_restriction_walker(Node *node, void *context)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("LIMIT/OFFSET clause is not supported on incrementally maintainable materialized view")));
- if (qry->distinctClause)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("DISTINCT is not supported on incrementally maintainable materialized view")));
if (qry->hasDistinctOn)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -1090,12 +1134,18 @@ CreateIndexOnIMMV(Query *query, Relation matviewRel)
char idxname[NAMEDATALEN];
List *indexoidlist = RelationGetIndexList(matviewRel);
ListCell *indexoidscan;
- Bitmapset *key_attnos;
snprintf(idxname, sizeof(idxname), "%s_index", RelationGetRelationName(matviewRel));
index = makeNode(IndexStmt);
+ /*
+ * We consider null values not distinct to make sure that views with DISTINCT
+ * or GROUP BY don't contain multiple NULL rows when NULL is inserted to
+ * a base table concurrently.
+ */
+ index->nulls_not_distinct = true;
+
index->unique = true;
index->primary = false;
index->isconstraint = false;
@@ -1122,41 +1172,68 @@ CreateIndexOnIMMV(Query *query, Relation matviewRel)
index->concurrent = false;
index->if_not_exists = false;
- /* create index on the base tables' primary key columns */
- key_attnos = get_primary_key_attnos_from_query(query, &constraintList);
- if (key_attnos)
+ if (query->distinctClause)
{
+ /* create unique constraint on all columns */
foreach(lc, query->targetList)
{
TargetEntry *tle = (TargetEntry *) lfirst(lc);
Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, tle->resno - 1);
-
- if (bms_is_member(tle->resno - FirstLowInvalidHeapAttributeNumber, key_attnos))
- {
- IndexElem *iparam;
-
- iparam = makeNode(IndexElem);
- iparam->name = pstrdup(NameStr(attr->attname));
- iparam->expr = NULL;
- iparam->indexcolname = NULL;
- iparam->collation = NIL;
- iparam->opclass = NIL;
- iparam->opclassopts = NIL;
- iparam->ordering = SORTBY_DEFAULT;
- iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
- index->indexParams = lappend(index->indexParams, iparam);
- }
+ IndexElem *iparam;
+
+ iparam = makeNode(IndexElem);
+ iparam->name = pstrdup(NameStr(attr->attname));
+ iparam->expr = NULL;
+ iparam->indexcolname = NULL;
+ iparam->collation = NIL;
+ iparam->opclass = NIL;
+ iparam->opclassopts = NIL;
+ iparam->ordering = SORTBY_DEFAULT;
+ iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
+ index->indexParams = lappend(index->indexParams, iparam);
}
}
else
{
- /* create no index, just notice that an appropriate index is necessary for efficient IVM */
- ereport(NOTICE,
- (errmsg("could not create an index on materialized view \"%s\" automatically",
- RelationGetRelationName(matviewRel)),
- errdetail("This target list does not have all the primary key columns. "),
- errhint("Create an index on the materialized view for efficient incremental maintenance.")));
- return;
+ Bitmapset *key_attnos;
+
+ /* create index on the base tables' primary key columns */
+ key_attnos = get_primary_key_attnos_from_query(query, &constraintList);
+ if (key_attnos)
+ {
+ foreach(lc, query->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);
+ Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, tle->resno - 1);
+
+ if (bms_is_member(tle->resno - FirstLowInvalidHeapAttributeNumber, key_attnos))
+ {
+ IndexElem *iparam;
+
+ iparam = makeNode(IndexElem);
+ iparam->name = pstrdup(NameStr(attr->attname));
+ iparam->expr = NULL;
+ iparam->indexcolname = NULL;
+ iparam->collation = NIL;
+ iparam->opclass = NIL;
+ iparam->opclassopts = NIL;
+ iparam->ordering = SORTBY_DEFAULT;
+ iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
+ index->indexParams = lappend(index->indexParams, iparam);
+ }
+ }
+ }
+ else
+ {
+ /* create no index, just notice that an appropriate index is necessary for efficient IVM */
+ ereport(NOTICE,
+ (errmsg("could not create an index on materialized view \"%s\" automatically",
+ RelationGetRelationName(matviewRel)),
+ errdetail("This target list does not have all the primary key columns, "
+ "or this view does not contain DISTINCT clause."),
+ errhint("Create an index on the materialized view for efficient incremental maintenance.")));
+ return;
+ }
}
/* If we have a compatible index, we don't need to create another. */
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 7a87626f5f..d661fbbb48 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -39,6 +39,7 @@
#include "commands/dbcommands.h"
#include "commands/defrem.h"
#include "commands/event_trigger.h"
+#include "commands/matview.h"
#include "commands/progress.h"
#include "commands/tablecmds.h"
#include "commands/tablespace.h"
@@ -1129,6 +1130,45 @@ DefineIndex(Oid tableId,
safe_index = indexInfo->ii_Expressions == NIL &&
indexInfo->ii_Predicate == NIL;
+ /*
+ * We disallow unique indexes on IVM columns of IMMVs.
+ */
+ if (RelationIsIVM(rel) && stmt->unique)
+ {
+ for (int i = 0; i < indexInfo->ii_NumIndexKeyAttrs; i++)
+ {
+ AttrNumber attno = indexInfo->ii_IndexAttrNumbers[i];
+ if (attno > 0)
+ {
+ char *name = NameStr(TupleDescAttr(rel->rd_att, attno - 1)->attname);
+ if (name && isIvmName(name))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("unique index creation on IVM columns is not supported")));
+ }
+ }
+
+ if (indexInfo->ii_Expressions)
+ {
+ Bitmapset *indexattrs = NULL;
+ int varno = -1;
+
+ pull_varattnos((Node *) indexInfo->ii_Expressions, 1, &indexattrs);
+
+ while ((varno = bms_next_member(indexattrs, varno)) >= 0)
+ {
+ int attno = varno + FirstLowInvalidHeapAttributeNumber;
+ char *name = NameStr(TupleDescAttr(rel->rd_att, attno - 1)->attname);
+ if (name && isIvmName(name))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("unique index creation on IVM columns is not supported")));
+ }
+
+ }
+ }
+
+
/*
* Report index creation if appropriate (delay this till after most of the
* error checks)
diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c
index a821992a37..dbcbc79fff 100644
--- a/src/backend/commands/matview.c
+++ b/src/backend/commands/matview.c
@@ -152,11 +152,15 @@ static Query *rewrite_query_for_postupdate_state(Query *query, MV_TriggerTable *
static void apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *new_tuplestores,
TupleDesc tupdesc_old, TupleDesc tupdesc_new,
- Query *query);
+ Query *query, bool use_count, char *count_colname);
static void apply_old_delta(const char *matviewname, const char *deltaname_old,
List *keys);
+static void apply_old_delta_with_count(const char *matviewname, const char *deltaname_old,
+ List *keys, const char *count_colname);
static void apply_new_delta(const char *matviewname, const char *deltaname_new,
StringInfo target_list);
+static void apply_new_delta_with_count(const char *matviewname, const char* deltaname_new,
+ List *keys, StringInfo target_list, const char* count_colname);
static char *get_matching_condition_string(List *keys);
static void generate_equal(StringInfo querybuf, Oid opttype,
const char *leftop, const char *rightop);
@@ -271,6 +275,7 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
Oid matviewOid;
Relation matviewRel;
Query *dataQuery;
+ Query *viewQuery;
Oid tableSpace;
Oid relowner;
Oid OIDNewHeap;
@@ -330,8 +335,13 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
"CONCURRENTLY", "WITH NO DATA")));
- dataQuery = get_matview_query(matviewRel);
+ viewQuery = get_matview_query(matviewRel);
+ /* For IMMV, we need to rewrite matview query */
+ if (!stmt->skipData && RelationIsIVM(matviewRel))
+ dataQuery = rewriteQueryForIMMV(viewQuery,NIL);
+ else
+ dataQuery = viewQuery;
/*
* Check that there is a unique index with no WHERE clause on one or more
@@ -511,8 +521,8 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
if (!stmt->skipData && RelationIsIVM(matviewRel) && !oldPopulated)
{
- CreateIndexOnIMMV(dataQuery, matviewRel);
- CreateIvmTriggersOnBaseTables(dataQuery, matviewOid);
+ CreateIndexOnIMMV(viewQuery, matviewRel);
+ CreateIvmTriggersOnBaseTables(viewQuery, matviewOid);
}
table_close(matviewRel, NoLock);
@@ -1534,6 +1544,13 @@ IVM_immediate_maintenance(PG_FUNCTION_ARGS)
int rte_index = lfirst_int(lc2);
TupleDesc tupdesc_old;
TupleDesc tupdesc_new;
+ bool use_count = false;
+ char *count_colname = NULL;
+
+ count_colname = pstrdup("__ivm_count__");
+
+ if (query->distinctClause)
+ use_count = true;
/* calculate delta tables */
calc_delta(table, rte_index, rewritten, dest_old, dest_new,
@@ -1546,7 +1563,8 @@ IVM_immediate_maintenance(PG_FUNCTION_ARGS)
{
/* apply the delta tables to the materialized view */
apply_delta(matviewOid, old_tuplestore, new_tuplestore,
- tupdesc_old, tupdesc_new, query);
+ tupdesc_old, tupdesc_new, query, use_count,
+ count_colname);
}
PG_CATCH();
{
@@ -2019,7 +2037,7 @@ rewrite_query_for_postupdate_state(Query *query, MV_TriggerTable *table, int rte
static void
apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *new_tuplestores,
TupleDesc tupdesc_old, TupleDesc tupdesc_new,
- Query *query)
+ Query *query, bool use_count, char *count_colname)
{
StringInfoData querybuf;
StringInfoData target_list_buf;
@@ -2095,7 +2113,12 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
if (rc != SPI_OK_REL_REGISTER)
elog(ERROR, "SPI_register failed");
- apply_old_delta(matviewname, OLD_DELTA_ENRNAME, keys);
+ if (use_count)
+ /* apply old delta and get rows to be recalculated */
+ apply_old_delta_with_count(matviewname, OLD_DELTA_ENRNAME,
+ keys, count_colname);
+ else
+ apply_old_delta(matviewname, OLD_DELTA_ENRNAME, keys);
}
/* For tuple insertion */
@@ -2117,7 +2140,11 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
elog(ERROR, "SPI_register failed");
/* apply new delta */
- apply_new_delta(matviewname, NEW_DELTA_ENRNAME, &target_list_buf);
+ if (use_count)
+ apply_new_delta_with_count(matviewname, NEW_DELTA_ENRNAME,
+ keys, &target_list_buf, count_colname);
+ else
+ apply_new_delta(matviewname, NEW_DELTA_ENRNAME, &target_list_buf);
}
/* We're done maintaining the materialized view. */
@@ -2130,6 +2157,51 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
elog(ERROR, "SPI_finish failed");
}
+/*
+ * apply_old_delta_with_count
+ *
+ * Execute a query for applying a delta table given by deltname_old
+ * which contains tuples to be deleted from to a materialized view given by
+ * matviewname. This is used when counting is required, that is, the view
+ * has aggregate or distinct.
+ */
+static void
+apply_old_delta_with_count(const char *matviewname, const char *deltaname_old,
+ List *keys, const char *count_colname)
+{
+ StringInfoData querybuf;
+ char *match_cond;
+
+ /* build WHERE condition for searching tuples to be deleted */
+ match_cond = get_matching_condition_string(keys);
+
+ /* Search for matching tuples from the view and update or delete if found. */
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH t AS (" /* collecting tid of target tuples in the view */
+ "SELECT diff.%s, " /* count column */
+ "(diff.%s OPERATOR(pg_catalog.=) mv.%s) AS for_dlt, "
+ "mv.ctid "
+ "FROM %s AS mv, %s AS diff "
+ "WHERE %s" /* tuple matching condition */
+ "), updt AS (" /* update a tuple if this is not to be deleted */
+ "UPDATE %s AS mv SET %s = mv.%s OPERATOR(pg_catalog.-) t.%s "
+ "FROM t WHERE mv.ctid OPERATOR(pg_catalog.=) t.ctid AND NOT for_dlt "
+ ")"
+ /* delete a tuple if this is to be deleted */
+ "DELETE FROM %s AS mv USING t "
+ "WHERE mv.ctid OPERATOR(pg_catalog.=) t.ctid AND for_dlt",
+ count_colname,
+ count_colname, count_colname,
+ matviewname, deltaname_old,
+ match_cond,
+ matviewname, count_colname, count_colname, count_colname,
+ matviewname);
+
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_DELETE)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+}
+
/*
* apply_old_delta
*
@@ -2179,6 +2251,66 @@ apply_old_delta(const char *matviewname, const char *deltaname_old,
elog(ERROR, "SPI_exec failed: %s", querybuf.data);
}
+/*
+ * apply_new_delta_with_count
+ *
+ * Execute a query for applying a delta table given by deltname_new
+ * which contains tuples to be inserted into a materialized view given by
+ * matviewname. This is used when counting is required, that is, the view
+ * has aggregate or distinct. Also, when a table in EXISTS sub queries
+ * is modified.
+ */
+static void
+apply_new_delta_with_count(const char *matviewname, const char* deltaname_new,
+ List *keys, StringInfo target_list, const char* count_colname)
+{
+ StringInfoData querybuf;
+ StringInfoData returning_keys;
+ ListCell *lc;
+ char *match_cond = "";
+
+ /* build WHERE condition for searching tuples to be updated */
+ match_cond = get_matching_condition_string(keys);
+
+ /* build string of keys list */
+ initStringInfo(&returning_keys);
+ if (keys)
+ {
+ foreach (lc, keys)
+ {
+ Form_pg_attribute attr = (Form_pg_attribute) lfirst(lc);
+ char *resname = NameStr(attr->attname);
+ appendStringInfo(&returning_keys, "%s", quote_qualified_identifier("mv", resname));
+ if (lnext(keys, lc))
+ appendStringInfo(&returning_keys, ", ");
+ }
+ }
+ else
+ appendStringInfo(&returning_keys, "NULL");
+
+ /* Search for matching tuples from the view and update if found or insert if not. */
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH updt AS (" /* update a tuple if this exists in the view */
+ "UPDATE %s AS mv SET %s = mv.%s OPERATOR(pg_catalog.+) diff.%s "
+ "FROM %s AS diff "
+ "WHERE %s " /* tuple matching condition */
+ "RETURNING %s" /* returning keys of updated tuples */
+ ") INSERT INTO %s (%s) " /* insert a new tuple if this doesn't exist */
+ "SELECT %s FROM %s AS diff "
+ "WHERE NOT EXISTS (SELECT 1 FROM updt AS mv WHERE %s);",
+ matviewname, count_colname, count_colname, count_colname,
+ deltaname_new,
+ match_cond,
+ returning_keys.data,
+ matviewname, target_list->data,
+ target_list->data, deltaname_new,
+ match_cond);
+
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+}
+
/*
* apply_new_delta
*
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index f798794556..02a688acb7 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -53,6 +53,7 @@
#include "commands/cluster.h"
#include "commands/comment.h"
#include "commands/defrem.h"
+#include "commands/matview.h"
#include "commands/event_trigger.h"
#include "commands/policy.h"
#include "commands/sequence.h"
@@ -3784,6 +3785,14 @@ renameatt_internal(Oid myrelid,
targetrelation = relation_open(myrelid, AccessExclusiveLock);
renameatt_check(myrelid, RelationGetForm(targetrelation), recursing);
+ /*
+ * Don't rename IVM columns.
+ */
+ if (RelationIsIVM(targetrelation) && isIvmName(oldattname))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("IVM column can not be renamed")));
+
/*
* if the 'recurse' flag is set then we are supposed to rename this
* attribute in all classes that inherit from 'relname' (as well as in
diff --git a/src/backend/parser/parse_relation.c b/src/backend/parser/parse_relation.c
index 34a0ec5901..d5b45c587d 100644
--- a/src/backend/parser/parse_relation.c
+++ b/src/backend/parser/parse_relation.c
@@ -36,6 +36,7 @@
#include "utils/rel.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
+#include "commands/matview.h"
/*
@@ -97,7 +98,7 @@ static void expandTupleDesc(TupleDesc tupdesc, Alias *eref,
int count, int offset,
int rtindex, int sublevels_up,
int location, bool include_dropped,
- List **colnames, List **colvars);
+ List **colnames, List **colvars, bool is_ivm);
static int specialAttNum(const char *attname);
static bool rte_visible_if_lateral(ParseState *pstate, RangeTblEntry *rte);
static bool rte_visible_if_qualified(ParseState *pstate, RangeTblEntry *rte);
@@ -1502,6 +1503,7 @@ addRangeTableEntry(ParseState *pstate,
rte->relid = RelationGetRelid(rel);
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -1587,6 +1589,7 @@ addRangeTableEntryForRelation(ParseState *pstate,
rte->relid = RelationGetRelid(rel);
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -2758,7 +2761,7 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
expandTupleDesc(tupdesc, rte->eref,
rtfunc->funccolcount, atts_done,
rtindex, sublevels_up, location,
- include_dropped, colnames, colvars);
+ include_dropped, colnames, colvars, false);
}
else if (functypclass == TYPEFUNC_SCALAR)
{
@@ -3026,7 +3029,7 @@ expandRelation(Oid relid, Alias *eref, int rtindex, int sublevels_up,
expandTupleDesc(rel->rd_att, eref, rel->rd_att->natts, 0,
rtindex, sublevels_up,
location, include_dropped,
- colnames, colvars);
+ colnames, colvars, RelationIsIVM(rel));
relation_close(rel, AccessShareLock);
}
@@ -3043,7 +3046,7 @@ static void
expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
int rtindex, int sublevels_up,
int location, bool include_dropped,
- List **colnames, List **colvars)
+ List **colnames, List **colvars, bool is_ivm)
{
ListCell *aliascell;
int varattno;
@@ -3056,6 +3059,9 @@ expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
{
Form_pg_attribute attr = TupleDescAttr(tupdesc, varattno);
+ if (is_ivm && isIvmName(NameStr(attr->attname)) && !MatViewIncrementalMaintenanceIsEnabled())
+ continue;
+
if (attr->attisdropped)
{
if (include_dropped)
@@ -3218,6 +3224,10 @@ expandNSItemAttrs(ParseState *pstate, ParseNamespaceItem *nsitem,
Var *varnode = (Var *) lfirst(var);
TargetEntry *te;
+ /* if transform * into columnlist with IMMV, remove IVM columns */
+ if (rte->relisivm && isIvmName(label) && !MatViewIncrementalMaintenanceIsEnabled())
+ continue;
+
te = makeTargetEntry((Expr *) varnode,
(AttrNumber) pstate->p_next_resno++,
label,
diff --git a/src/backend/rewrite/rewriteDefine.c b/src/backend/rewrite/rewriteDefine.c
index b449244a53..b9d161d6a3 100644
--- a/src/backend/rewrite/rewriteDefine.c
+++ b/src/backend/rewrite/rewriteDefine.c
@@ -621,7 +621,8 @@ checkRuleResultList(List *targetList, TupleDesc resultDesc, bool isSelect,
attr->atttypmod))));
}
- if (i != resultDesc->natts)
+ /* No check for materialized views since this could have special columns for IVM */
+ if ((!isSelect || requireColumnNameMatch) && i != resultDesc->natts)
ereport(ERROR,
(errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
isSelect ?
diff --git a/src/include/commands/createas.h b/src/include/commands/createas.h
index 396ad1bb4c..6b47e66bfd 100644
--- a/src/include/commands/createas.h
+++ b/src/include/commands/createas.h
@@ -29,6 +29,8 @@ extern ObjectAddress ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *st
extern void CreateIvmTriggersOnBaseTables(Query *qry, Oid matviewOid);
extern void CreateIndexOnIMMV(Query *query, Relation matviewRel);
+extern Query *rewriteQueryForIMMV(Query *query, List *colNames);
+
extern int GetIntoRelEFlags(IntoClause *intoClause);
extern DestReceiver *CreateIntoRelDestReceiver(IntoClause *intoClause);
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index d60e148ff2..fa3a95897c 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -1074,6 +1074,7 @@ typedef struct RangeTblEntry
int rellockmode; /* lock level that query requires on the rel */
struct TableSampleClause *tablesample; /* sampling info, or NULL */
Index perminfoindex;
+ bool relisivm;
/*
* Fields valid for a subquery RTE (else NULL):
--
2.25.1
--Multipart=_Mon__4_Mar_2024_11_58_46_+0900_UaponF/qQhQrVCFt
Content-Type: text/x-diff;
name="v30-0008-Add-aggregates-support-in-IVM.patch"
Content-Disposition: attachment;
filename="v30-0008-Add-aggregates-support-in-IVM.patch"
Content-Transfer-Encoding: quoted-printable
^ permalink raw reply [nested|flat] 24+ messages in thread
* [PATCH v30 07/11] Add DISTINCT support for IVM
@ 2023-05-31 10:08 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 24+ messages in thread
From: Yugo Nagata @ 2023-05-31 10:08 UTC (permalink / raw)
When IMMV is created with DISTINCT, multiplicity of tuples is
counted and stored in "__ivm_count__" column, which is a hidden
column of IMMV. The value in __ivm_count__ is updated when IMMV
is maintained incrementally. A tuple in IMMV can be removed if
and only if the count becomes zero.
---
src/backend/commands/createas.c | 141 ++++++++++++++++++++------
src/backend/commands/indexcmds.c | 40 ++++++++
src/backend/commands/matview.c | 148 ++++++++++++++++++++++++++--
src/backend/commands/tablecmds.c | 9 ++
src/backend/parser/parse_relation.c | 18 +++-
src/backend/rewrite/rewriteDefine.c | 3 +-
src/include/commands/createas.h | 2 +
src/include/nodes/parsenodes.h | 1 +
8 files changed, 317 insertions(+), 45 deletions(-)
diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c
index 08262100ea..cce44278fa 100644
--- a/src/backend/commands/createas.c
+++ b/src/backend/commands/createas.c
@@ -53,6 +53,7 @@
#include "parser/parser.h"
#include "parser/parsetree.h"
#include "parser/parse_clause.h"
+#include "parser/parse_func.h"
#include "rewrite/rewriteHandler.h"
#include "storage/smgr.h"
#include "tcop/tcopprot.h"
@@ -309,6 +310,9 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt,
errhint("functions must be marked IMMUTABLE")));
check_ivm_restriction((Node *) query);
+
+ /* For IMMV, we need to rewrite matview query */
+ query = rewriteQueryForIMMV(query, into->colNames);
}
if (into->skipData)
@@ -413,6 +417,49 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt,
return address;
}
+/*
+ * rewriteQueryForIMMV -- rewrite view definition query for IMMV
+ *
+ * count(*) is added for counting distinct tuples in views.
+ */
+Query *
+rewriteQueryForIMMV(Query *query, List *colNames)
+{
+ Query *rewritten;
+
+ Node *node;
+ ParseState *pstate = make_parsestate(NULL);
+ FuncCall *fn;
+
+ rewritten = copyObject(query);
+ pstate->p_expr_kind = EXPR_KIND_SELECT_TARGET;
+
+ /*
+ * Convert DISTINCT to GROUP BY and add count(*) for counting distinct
+ * tuples in views.
+ */
+ if (rewritten->distinctClause)
+ {
+ TargetEntry *tle;
+
+ rewritten->groupClause = transformDistinctClause(NULL, &rewritten->targetList, rewritten->sortClause, false);
+
+ fn = makeFuncCall(SystemFuncName("count"), NIL, COERCE_EXPLICIT_CALL, -1);
+ fn->agg_star = true;
+
+ node = ParseFuncOrColumn(pstate, fn->funcname, NIL, NULL, fn, false, -1);
+
+ tle = makeTargetEntry((Expr *) node,
+ list_length(rewritten->targetList) + 1,
+ pstrdup("__ivm_count__"),
+ false);
+ rewritten->targetList = lappend(rewritten->targetList, tle);
+ rewritten->hasAggs = true;
+ }
+
+ return rewritten;
+}
+
/*
* GetIntoRelEFlags --- compute executor flags needed for CREATE TABLE AS
*
@@ -536,7 +583,8 @@ intorel_startup(DestReceiver *self, int operation, TupleDesc typeinfo)
ColumnDef *col;
char *colname;
- if (lc)
+ /* Don't override hidden columns added for IVM */
+ if (lc && !isIvmName(NameStr(attribute->attname)))
{
colname = strVal(lfirst(lc));
lc = lnext(into->colNames, lc);
@@ -940,10 +988,6 @@ check_ivm_restriction_walker(Node *node, void *context)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("LIMIT/OFFSET clause is not supported on incrementally maintainable materialized view")));
- if (qry->distinctClause)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("DISTINCT is not supported on incrementally maintainable materialized view")));
if (qry->hasDistinctOn)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -1090,12 +1134,18 @@ CreateIndexOnIMMV(Query *query, Relation matviewRel)
char idxname[NAMEDATALEN];
List *indexoidlist = RelationGetIndexList(matviewRel);
ListCell *indexoidscan;
- Bitmapset *key_attnos;
snprintf(idxname, sizeof(idxname), "%s_index", RelationGetRelationName(matviewRel));
index = makeNode(IndexStmt);
+ /*
+ * We consider null values not distinct to make sure that views with DISTINCT
+ * or GROUP BY don't contain multiple NULL rows when NULL is inserted to
+ * a base table concurrently.
+ */
+ index->nulls_not_distinct = true;
+
index->unique = true;
index->primary = false;
index->isconstraint = false;
@@ -1122,41 +1172,68 @@ CreateIndexOnIMMV(Query *query, Relation matviewRel)
index->concurrent = false;
index->if_not_exists = false;
- /* create index on the base tables' primary key columns */
- key_attnos = get_primary_key_attnos_from_query(query, &constraintList);
- if (key_attnos)
+ if (query->distinctClause)
{
+ /* create unique constraint on all columns */
foreach(lc, query->targetList)
{
TargetEntry *tle = (TargetEntry *) lfirst(lc);
Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, tle->resno - 1);
-
- if (bms_is_member(tle->resno - FirstLowInvalidHeapAttributeNumber, key_attnos))
- {
- IndexElem *iparam;
-
- iparam = makeNode(IndexElem);
- iparam->name = pstrdup(NameStr(attr->attname));
- iparam->expr = NULL;
- iparam->indexcolname = NULL;
- iparam->collation = NIL;
- iparam->opclass = NIL;
- iparam->opclassopts = NIL;
- iparam->ordering = SORTBY_DEFAULT;
- iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
- index->indexParams = lappend(index->indexParams, iparam);
- }
+ IndexElem *iparam;
+
+ iparam = makeNode(IndexElem);
+ iparam->name = pstrdup(NameStr(attr->attname));
+ iparam->expr = NULL;
+ iparam->indexcolname = NULL;
+ iparam->collation = NIL;
+ iparam->opclass = NIL;
+ iparam->opclassopts = NIL;
+ iparam->ordering = SORTBY_DEFAULT;
+ iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
+ index->indexParams = lappend(index->indexParams, iparam);
}
}
else
{
- /* create no index, just notice that an appropriate index is necessary for efficient IVM */
- ereport(NOTICE,
- (errmsg("could not create an index on materialized view \"%s\" automatically",
- RelationGetRelationName(matviewRel)),
- errdetail("This target list does not have all the primary key columns. "),
- errhint("Create an index on the materialized view for efficient incremental maintenance.")));
- return;
+ Bitmapset *key_attnos;
+
+ /* create index on the base tables' primary key columns */
+ key_attnos = get_primary_key_attnos_from_query(query, &constraintList);
+ if (key_attnos)
+ {
+ foreach(lc, query->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);
+ Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, tle->resno - 1);
+
+ if (bms_is_member(tle->resno - FirstLowInvalidHeapAttributeNumber, key_attnos))
+ {
+ IndexElem *iparam;
+
+ iparam = makeNode(IndexElem);
+ iparam->name = pstrdup(NameStr(attr->attname));
+ iparam->expr = NULL;
+ iparam->indexcolname = NULL;
+ iparam->collation = NIL;
+ iparam->opclass = NIL;
+ iparam->opclassopts = NIL;
+ iparam->ordering = SORTBY_DEFAULT;
+ iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
+ index->indexParams = lappend(index->indexParams, iparam);
+ }
+ }
+ }
+ else
+ {
+ /* create no index, just notice that an appropriate index is necessary for efficient IVM */
+ ereport(NOTICE,
+ (errmsg("could not create an index on materialized view \"%s\" automatically",
+ RelationGetRelationName(matviewRel)),
+ errdetail("This target list does not have all the primary key columns, "
+ "or this view does not contain DISTINCT clause."),
+ errhint("Create an index on the materialized view for efficient incremental maintenance.")));
+ return;
+ }
}
/* If we have a compatible index, we don't need to create another. */
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 7a87626f5f..d661fbbb48 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -39,6 +39,7 @@
#include "commands/dbcommands.h"
#include "commands/defrem.h"
#include "commands/event_trigger.h"
+#include "commands/matview.h"
#include "commands/progress.h"
#include "commands/tablecmds.h"
#include "commands/tablespace.h"
@@ -1129,6 +1130,45 @@ DefineIndex(Oid tableId,
safe_index = indexInfo->ii_Expressions == NIL &&
indexInfo->ii_Predicate == NIL;
+ /*
+ * We disallow unique indexes on IVM columns of IMMVs.
+ */
+ if (RelationIsIVM(rel) && stmt->unique)
+ {
+ for (int i = 0; i < indexInfo->ii_NumIndexKeyAttrs; i++)
+ {
+ AttrNumber attno = indexInfo->ii_IndexAttrNumbers[i];
+ if (attno > 0)
+ {
+ char *name = NameStr(TupleDescAttr(rel->rd_att, attno - 1)->attname);
+ if (name && isIvmName(name))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("unique index creation on IVM columns is not supported")));
+ }
+ }
+
+ if (indexInfo->ii_Expressions)
+ {
+ Bitmapset *indexattrs = NULL;
+ int varno = -1;
+
+ pull_varattnos((Node *) indexInfo->ii_Expressions, 1, &indexattrs);
+
+ while ((varno = bms_next_member(indexattrs, varno)) >= 0)
+ {
+ int attno = varno + FirstLowInvalidHeapAttributeNumber;
+ char *name = NameStr(TupleDescAttr(rel->rd_att, attno - 1)->attname);
+ if (name && isIvmName(name))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("unique index creation on IVM columns is not supported")));
+ }
+
+ }
+ }
+
+
/*
* Report index creation if appropriate (delay this till after most of the
* error checks)
diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c
index a821992a37..dbcbc79fff 100644
--- a/src/backend/commands/matview.c
+++ b/src/backend/commands/matview.c
@@ -152,11 +152,15 @@ static Query *rewrite_query_for_postupdate_state(Query *query, MV_TriggerTable *
static void apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *new_tuplestores,
TupleDesc tupdesc_old, TupleDesc tupdesc_new,
- Query *query);
+ Query *query, bool use_count, char *count_colname);
static void apply_old_delta(const char *matviewname, const char *deltaname_old,
List *keys);
+static void apply_old_delta_with_count(const char *matviewname, const char *deltaname_old,
+ List *keys, const char *count_colname);
static void apply_new_delta(const char *matviewname, const char *deltaname_new,
StringInfo target_list);
+static void apply_new_delta_with_count(const char *matviewname, const char* deltaname_new,
+ List *keys, StringInfo target_list, const char* count_colname);
static char *get_matching_condition_string(List *keys);
static void generate_equal(StringInfo querybuf, Oid opttype,
const char *leftop, const char *rightop);
@@ -271,6 +275,7 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
Oid matviewOid;
Relation matviewRel;
Query *dataQuery;
+ Query *viewQuery;
Oid tableSpace;
Oid relowner;
Oid OIDNewHeap;
@@ -330,8 +335,13 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
"CONCURRENTLY", "WITH NO DATA")));
- dataQuery = get_matview_query(matviewRel);
+ viewQuery = get_matview_query(matviewRel);
+ /* For IMMV, we need to rewrite matview query */
+ if (!stmt->skipData && RelationIsIVM(matviewRel))
+ dataQuery = rewriteQueryForIMMV(viewQuery,NIL);
+ else
+ dataQuery = viewQuery;
/*
* Check that there is a unique index with no WHERE clause on one or more
@@ -511,8 +521,8 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
if (!stmt->skipData && RelationIsIVM(matviewRel) && !oldPopulated)
{
- CreateIndexOnIMMV(dataQuery, matviewRel);
- CreateIvmTriggersOnBaseTables(dataQuery, matviewOid);
+ CreateIndexOnIMMV(viewQuery, matviewRel);
+ CreateIvmTriggersOnBaseTables(viewQuery, matviewOid);
}
table_close(matviewRel, NoLock);
@@ -1534,6 +1544,13 @@ IVM_immediate_maintenance(PG_FUNCTION_ARGS)
int rte_index = lfirst_int(lc2);
TupleDesc tupdesc_old;
TupleDesc tupdesc_new;
+ bool use_count = false;
+ char *count_colname = NULL;
+
+ count_colname = pstrdup("__ivm_count__");
+
+ if (query->distinctClause)
+ use_count = true;
/* calculate delta tables */
calc_delta(table, rte_index, rewritten, dest_old, dest_new,
@@ -1546,7 +1563,8 @@ IVM_immediate_maintenance(PG_FUNCTION_ARGS)
{
/* apply the delta tables to the materialized view */
apply_delta(matviewOid, old_tuplestore, new_tuplestore,
- tupdesc_old, tupdesc_new, query);
+ tupdesc_old, tupdesc_new, query, use_count,
+ count_colname);
}
PG_CATCH();
{
@@ -2019,7 +2037,7 @@ rewrite_query_for_postupdate_state(Query *query, MV_TriggerTable *table, int rte
static void
apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *new_tuplestores,
TupleDesc tupdesc_old, TupleDesc tupdesc_new,
- Query *query)
+ Query *query, bool use_count, char *count_colname)
{
StringInfoData querybuf;
StringInfoData target_list_buf;
@@ -2095,7 +2113,12 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
if (rc != SPI_OK_REL_REGISTER)
elog(ERROR, "SPI_register failed");
- apply_old_delta(matviewname, OLD_DELTA_ENRNAME, keys);
+ if (use_count)
+ /* apply old delta and get rows to be recalculated */
+ apply_old_delta_with_count(matviewname, OLD_DELTA_ENRNAME,
+ keys, count_colname);
+ else
+ apply_old_delta(matviewname, OLD_DELTA_ENRNAME, keys);
}
/* For tuple insertion */
@@ -2117,7 +2140,11 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
elog(ERROR, "SPI_register failed");
/* apply new delta */
- apply_new_delta(matviewname, NEW_DELTA_ENRNAME, &target_list_buf);
+ if (use_count)
+ apply_new_delta_with_count(matviewname, NEW_DELTA_ENRNAME,
+ keys, &target_list_buf, count_colname);
+ else
+ apply_new_delta(matviewname, NEW_DELTA_ENRNAME, &target_list_buf);
}
/* We're done maintaining the materialized view. */
@@ -2130,6 +2157,51 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
elog(ERROR, "SPI_finish failed");
}
+/*
+ * apply_old_delta_with_count
+ *
+ * Execute a query for applying a delta table given by deltname_old
+ * which contains tuples to be deleted from to a materialized view given by
+ * matviewname. This is used when counting is required, that is, the view
+ * has aggregate or distinct.
+ */
+static void
+apply_old_delta_with_count(const char *matviewname, const char *deltaname_old,
+ List *keys, const char *count_colname)
+{
+ StringInfoData querybuf;
+ char *match_cond;
+
+ /* build WHERE condition for searching tuples to be deleted */
+ match_cond = get_matching_condition_string(keys);
+
+ /* Search for matching tuples from the view and update or delete if found. */
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH t AS (" /* collecting tid of target tuples in the view */
+ "SELECT diff.%s, " /* count column */
+ "(diff.%s OPERATOR(pg_catalog.=) mv.%s) AS for_dlt, "
+ "mv.ctid "
+ "FROM %s AS mv, %s AS diff "
+ "WHERE %s" /* tuple matching condition */
+ "), updt AS (" /* update a tuple if this is not to be deleted */
+ "UPDATE %s AS mv SET %s = mv.%s OPERATOR(pg_catalog.-) t.%s "
+ "FROM t WHERE mv.ctid OPERATOR(pg_catalog.=) t.ctid AND NOT for_dlt "
+ ")"
+ /* delete a tuple if this is to be deleted */
+ "DELETE FROM %s AS mv USING t "
+ "WHERE mv.ctid OPERATOR(pg_catalog.=) t.ctid AND for_dlt",
+ count_colname,
+ count_colname, count_colname,
+ matviewname, deltaname_old,
+ match_cond,
+ matviewname, count_colname, count_colname, count_colname,
+ matviewname);
+
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_DELETE)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+}
+
/*
* apply_old_delta
*
@@ -2179,6 +2251,66 @@ apply_old_delta(const char *matviewname, const char *deltaname_old,
elog(ERROR, "SPI_exec failed: %s", querybuf.data);
}
+/*
+ * apply_new_delta_with_count
+ *
+ * Execute a query for applying a delta table given by deltname_new
+ * which contains tuples to be inserted into a materialized view given by
+ * matviewname. This is used when counting is required, that is, the view
+ * has aggregate or distinct. Also, when a table in EXISTS sub queries
+ * is modified.
+ */
+static void
+apply_new_delta_with_count(const char *matviewname, const char* deltaname_new,
+ List *keys, StringInfo target_list, const char* count_colname)
+{
+ StringInfoData querybuf;
+ StringInfoData returning_keys;
+ ListCell *lc;
+ char *match_cond = "";
+
+ /* build WHERE condition for searching tuples to be updated */
+ match_cond = get_matching_condition_string(keys);
+
+ /* build string of keys list */
+ initStringInfo(&returning_keys);
+ if (keys)
+ {
+ foreach (lc, keys)
+ {
+ Form_pg_attribute attr = (Form_pg_attribute) lfirst(lc);
+ char *resname = NameStr(attr->attname);
+ appendStringInfo(&returning_keys, "%s", quote_qualified_identifier("mv", resname));
+ if (lnext(keys, lc))
+ appendStringInfo(&returning_keys, ", ");
+ }
+ }
+ else
+ appendStringInfo(&returning_keys, "NULL");
+
+ /* Search for matching tuples from the view and update if found or insert if not. */
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH updt AS (" /* update a tuple if this exists in the view */
+ "UPDATE %s AS mv SET %s = mv.%s OPERATOR(pg_catalog.+) diff.%s "
+ "FROM %s AS diff "
+ "WHERE %s " /* tuple matching condition */
+ "RETURNING %s" /* returning keys of updated tuples */
+ ") INSERT INTO %s (%s) " /* insert a new tuple if this doesn't exist */
+ "SELECT %s FROM %s AS diff "
+ "WHERE NOT EXISTS (SELECT 1 FROM updt AS mv WHERE %s);",
+ matviewname, count_colname, count_colname, count_colname,
+ deltaname_new,
+ match_cond,
+ returning_keys.data,
+ matviewname, target_list->data,
+ target_list->data, deltaname_new,
+ match_cond);
+
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+}
+
/*
* apply_new_delta
*
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index f798794556..02a688acb7 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -53,6 +53,7 @@
#include "commands/cluster.h"
#include "commands/comment.h"
#include "commands/defrem.h"
+#include "commands/matview.h"
#include "commands/event_trigger.h"
#include "commands/policy.h"
#include "commands/sequence.h"
@@ -3784,6 +3785,14 @@ renameatt_internal(Oid myrelid,
targetrelation = relation_open(myrelid, AccessExclusiveLock);
renameatt_check(myrelid, RelationGetForm(targetrelation), recursing);
+ /*
+ * Don't rename IVM columns.
+ */
+ if (RelationIsIVM(targetrelation) && isIvmName(oldattname))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("IVM column can not be renamed")));
+
/*
* if the 'recurse' flag is set then we are supposed to rename this
* attribute in all classes that inherit from 'relname' (as well as in
diff --git a/src/backend/parser/parse_relation.c b/src/backend/parser/parse_relation.c
index 34a0ec5901..d5b45c587d 100644
--- a/src/backend/parser/parse_relation.c
+++ b/src/backend/parser/parse_relation.c
@@ -36,6 +36,7 @@
#include "utils/rel.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
+#include "commands/matview.h"
/*
@@ -97,7 +98,7 @@ static void expandTupleDesc(TupleDesc tupdesc, Alias *eref,
int count, int offset,
int rtindex, int sublevels_up,
int location, bool include_dropped,
- List **colnames, List **colvars);
+ List **colnames, List **colvars, bool is_ivm);
static int specialAttNum(const char *attname);
static bool rte_visible_if_lateral(ParseState *pstate, RangeTblEntry *rte);
static bool rte_visible_if_qualified(ParseState *pstate, RangeTblEntry *rte);
@@ -1502,6 +1503,7 @@ addRangeTableEntry(ParseState *pstate,
rte->relid = RelationGetRelid(rel);
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -1587,6 +1589,7 @@ addRangeTableEntryForRelation(ParseState *pstate,
rte->relid = RelationGetRelid(rel);
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -2758,7 +2761,7 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
expandTupleDesc(tupdesc, rte->eref,
rtfunc->funccolcount, atts_done,
rtindex, sublevels_up, location,
- include_dropped, colnames, colvars);
+ include_dropped, colnames, colvars, false);
}
else if (functypclass == TYPEFUNC_SCALAR)
{
@@ -3026,7 +3029,7 @@ expandRelation(Oid relid, Alias *eref, int rtindex, int sublevels_up,
expandTupleDesc(rel->rd_att, eref, rel->rd_att->natts, 0,
rtindex, sublevels_up,
location, include_dropped,
- colnames, colvars);
+ colnames, colvars, RelationIsIVM(rel));
relation_close(rel, AccessShareLock);
}
@@ -3043,7 +3046,7 @@ static void
expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
int rtindex, int sublevels_up,
int location, bool include_dropped,
- List **colnames, List **colvars)
+ List **colnames, List **colvars, bool is_ivm)
{
ListCell *aliascell;
int varattno;
@@ -3056,6 +3059,9 @@ expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
{
Form_pg_attribute attr = TupleDescAttr(tupdesc, varattno);
+ if (is_ivm && isIvmName(NameStr(attr->attname)) && !MatViewIncrementalMaintenanceIsEnabled())
+ continue;
+
if (attr->attisdropped)
{
if (include_dropped)
@@ -3218,6 +3224,10 @@ expandNSItemAttrs(ParseState *pstate, ParseNamespaceItem *nsitem,
Var *varnode = (Var *) lfirst(var);
TargetEntry *te;
+ /* if transform * into columnlist with IMMV, remove IVM columns */
+ if (rte->relisivm && isIvmName(label) && !MatViewIncrementalMaintenanceIsEnabled())
+ continue;
+
te = makeTargetEntry((Expr *) varnode,
(AttrNumber) pstate->p_next_resno++,
label,
diff --git a/src/backend/rewrite/rewriteDefine.c b/src/backend/rewrite/rewriteDefine.c
index b449244a53..b9d161d6a3 100644
--- a/src/backend/rewrite/rewriteDefine.c
+++ b/src/backend/rewrite/rewriteDefine.c
@@ -621,7 +621,8 @@ checkRuleResultList(List *targetList, TupleDesc resultDesc, bool isSelect,
attr->atttypmod))));
}
- if (i != resultDesc->natts)
+ /* No check for materialized views since this could have special columns for IVM */
+ if ((!isSelect || requireColumnNameMatch) && i != resultDesc->natts)
ereport(ERROR,
(errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
isSelect ?
diff --git a/src/include/commands/createas.h b/src/include/commands/createas.h
index 396ad1bb4c..6b47e66bfd 100644
--- a/src/include/commands/createas.h
+++ b/src/include/commands/createas.h
@@ -29,6 +29,8 @@ extern ObjectAddress ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *st
extern void CreateIvmTriggersOnBaseTables(Query *qry, Oid matviewOid);
extern void CreateIndexOnIMMV(Query *query, Relation matviewRel);
+extern Query *rewriteQueryForIMMV(Query *query, List *colNames);
+
extern int GetIntoRelEFlags(IntoClause *intoClause);
extern DestReceiver *CreateIntoRelDestReceiver(IntoClause *intoClause);
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index d60e148ff2..fa3a95897c 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -1074,6 +1074,7 @@ typedef struct RangeTblEntry
int rellockmode; /* lock level that query requires on the rel */
struct TableSampleClause *tablesample; /* sampling info, or NULL */
Index perminfoindex;
+ bool relisivm;
/*
* Fields valid for a subquery RTE (else NULL):
--
2.25.1
--Multipart=_Mon__4_Mar_2024_11_58_46_+0900_UaponF/qQhQrVCFt
Content-Type: text/x-diff;
name="v30-0008-Add-aggregates-support-in-IVM.patch"
Content-Disposition: attachment;
filename="v30-0008-Add-aggregates-support-in-IVM.patch"
Content-Transfer-Encoding: quoted-printable
^ permalink raw reply [nested|flat] 24+ messages in thread
* [PATCH v30 07/11] Add DISTINCT support for IVM
@ 2023-05-31 10:08 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 24+ messages in thread
From: Yugo Nagata @ 2023-05-31 10:08 UTC (permalink / raw)
When IMMV is created with DISTINCT, multiplicity of tuples is
counted and stored in "__ivm_count__" column, which is a hidden
column of IMMV. The value in __ivm_count__ is updated when IMMV
is maintained incrementally. A tuple in IMMV can be removed if
and only if the count becomes zero.
---
src/backend/commands/createas.c | 141 ++++++++++++++++++++------
src/backend/commands/indexcmds.c | 40 ++++++++
src/backend/commands/matview.c | 148 ++++++++++++++++++++++++++--
src/backend/commands/tablecmds.c | 9 ++
src/backend/parser/parse_relation.c | 18 +++-
src/backend/rewrite/rewriteDefine.c | 3 +-
src/include/commands/createas.h | 2 +
src/include/nodes/parsenodes.h | 1 +
8 files changed, 317 insertions(+), 45 deletions(-)
diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c
index 08262100ea..cce44278fa 100644
--- a/src/backend/commands/createas.c
+++ b/src/backend/commands/createas.c
@@ -53,6 +53,7 @@
#include "parser/parser.h"
#include "parser/parsetree.h"
#include "parser/parse_clause.h"
+#include "parser/parse_func.h"
#include "rewrite/rewriteHandler.h"
#include "storage/smgr.h"
#include "tcop/tcopprot.h"
@@ -309,6 +310,9 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt,
errhint("functions must be marked IMMUTABLE")));
check_ivm_restriction((Node *) query);
+
+ /* For IMMV, we need to rewrite matview query */
+ query = rewriteQueryForIMMV(query, into->colNames);
}
if (into->skipData)
@@ -413,6 +417,49 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt,
return address;
}
+/*
+ * rewriteQueryForIMMV -- rewrite view definition query for IMMV
+ *
+ * count(*) is added for counting distinct tuples in views.
+ */
+Query *
+rewriteQueryForIMMV(Query *query, List *colNames)
+{
+ Query *rewritten;
+
+ Node *node;
+ ParseState *pstate = make_parsestate(NULL);
+ FuncCall *fn;
+
+ rewritten = copyObject(query);
+ pstate->p_expr_kind = EXPR_KIND_SELECT_TARGET;
+
+ /*
+ * Convert DISTINCT to GROUP BY and add count(*) for counting distinct
+ * tuples in views.
+ */
+ if (rewritten->distinctClause)
+ {
+ TargetEntry *tle;
+
+ rewritten->groupClause = transformDistinctClause(NULL, &rewritten->targetList, rewritten->sortClause, false);
+
+ fn = makeFuncCall(SystemFuncName("count"), NIL, COERCE_EXPLICIT_CALL, -1);
+ fn->agg_star = true;
+
+ node = ParseFuncOrColumn(pstate, fn->funcname, NIL, NULL, fn, false, -1);
+
+ tle = makeTargetEntry((Expr *) node,
+ list_length(rewritten->targetList) + 1,
+ pstrdup("__ivm_count__"),
+ false);
+ rewritten->targetList = lappend(rewritten->targetList, tle);
+ rewritten->hasAggs = true;
+ }
+
+ return rewritten;
+}
+
/*
* GetIntoRelEFlags --- compute executor flags needed for CREATE TABLE AS
*
@@ -536,7 +583,8 @@ intorel_startup(DestReceiver *self, int operation, TupleDesc typeinfo)
ColumnDef *col;
char *colname;
- if (lc)
+ /* Don't override hidden columns added for IVM */
+ if (lc && !isIvmName(NameStr(attribute->attname)))
{
colname = strVal(lfirst(lc));
lc = lnext(into->colNames, lc);
@@ -940,10 +988,6 @@ check_ivm_restriction_walker(Node *node, void *context)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("LIMIT/OFFSET clause is not supported on incrementally maintainable materialized view")));
- if (qry->distinctClause)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("DISTINCT is not supported on incrementally maintainable materialized view")));
if (qry->hasDistinctOn)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -1090,12 +1134,18 @@ CreateIndexOnIMMV(Query *query, Relation matviewRel)
char idxname[NAMEDATALEN];
List *indexoidlist = RelationGetIndexList(matviewRel);
ListCell *indexoidscan;
- Bitmapset *key_attnos;
snprintf(idxname, sizeof(idxname), "%s_index", RelationGetRelationName(matviewRel));
index = makeNode(IndexStmt);
+ /*
+ * We consider null values not distinct to make sure that views with DISTINCT
+ * or GROUP BY don't contain multiple NULL rows when NULL is inserted to
+ * a base table concurrently.
+ */
+ index->nulls_not_distinct = true;
+
index->unique = true;
index->primary = false;
index->isconstraint = false;
@@ -1122,41 +1172,68 @@ CreateIndexOnIMMV(Query *query, Relation matviewRel)
index->concurrent = false;
index->if_not_exists = false;
- /* create index on the base tables' primary key columns */
- key_attnos = get_primary_key_attnos_from_query(query, &constraintList);
- if (key_attnos)
+ if (query->distinctClause)
{
+ /* create unique constraint on all columns */
foreach(lc, query->targetList)
{
TargetEntry *tle = (TargetEntry *) lfirst(lc);
Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, tle->resno - 1);
-
- if (bms_is_member(tle->resno - FirstLowInvalidHeapAttributeNumber, key_attnos))
- {
- IndexElem *iparam;
-
- iparam = makeNode(IndexElem);
- iparam->name = pstrdup(NameStr(attr->attname));
- iparam->expr = NULL;
- iparam->indexcolname = NULL;
- iparam->collation = NIL;
- iparam->opclass = NIL;
- iparam->opclassopts = NIL;
- iparam->ordering = SORTBY_DEFAULT;
- iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
- index->indexParams = lappend(index->indexParams, iparam);
- }
+ IndexElem *iparam;
+
+ iparam = makeNode(IndexElem);
+ iparam->name = pstrdup(NameStr(attr->attname));
+ iparam->expr = NULL;
+ iparam->indexcolname = NULL;
+ iparam->collation = NIL;
+ iparam->opclass = NIL;
+ iparam->opclassopts = NIL;
+ iparam->ordering = SORTBY_DEFAULT;
+ iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
+ index->indexParams = lappend(index->indexParams, iparam);
}
}
else
{
- /* create no index, just notice that an appropriate index is necessary for efficient IVM */
- ereport(NOTICE,
- (errmsg("could not create an index on materialized view \"%s\" automatically",
- RelationGetRelationName(matviewRel)),
- errdetail("This target list does not have all the primary key columns. "),
- errhint("Create an index on the materialized view for efficient incremental maintenance.")));
- return;
+ Bitmapset *key_attnos;
+
+ /* create index on the base tables' primary key columns */
+ key_attnos = get_primary_key_attnos_from_query(query, &constraintList);
+ if (key_attnos)
+ {
+ foreach(lc, query->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);
+ Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, tle->resno - 1);
+
+ if (bms_is_member(tle->resno - FirstLowInvalidHeapAttributeNumber, key_attnos))
+ {
+ IndexElem *iparam;
+
+ iparam = makeNode(IndexElem);
+ iparam->name = pstrdup(NameStr(attr->attname));
+ iparam->expr = NULL;
+ iparam->indexcolname = NULL;
+ iparam->collation = NIL;
+ iparam->opclass = NIL;
+ iparam->opclassopts = NIL;
+ iparam->ordering = SORTBY_DEFAULT;
+ iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
+ index->indexParams = lappend(index->indexParams, iparam);
+ }
+ }
+ }
+ else
+ {
+ /* create no index, just notice that an appropriate index is necessary for efficient IVM */
+ ereport(NOTICE,
+ (errmsg("could not create an index on materialized view \"%s\" automatically",
+ RelationGetRelationName(matviewRel)),
+ errdetail("This target list does not have all the primary key columns, "
+ "or this view does not contain DISTINCT clause."),
+ errhint("Create an index on the materialized view for efficient incremental maintenance.")));
+ return;
+ }
}
/* If we have a compatible index, we don't need to create another. */
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 7a87626f5f..d661fbbb48 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -39,6 +39,7 @@
#include "commands/dbcommands.h"
#include "commands/defrem.h"
#include "commands/event_trigger.h"
+#include "commands/matview.h"
#include "commands/progress.h"
#include "commands/tablecmds.h"
#include "commands/tablespace.h"
@@ -1129,6 +1130,45 @@ DefineIndex(Oid tableId,
safe_index = indexInfo->ii_Expressions == NIL &&
indexInfo->ii_Predicate == NIL;
+ /*
+ * We disallow unique indexes on IVM columns of IMMVs.
+ */
+ if (RelationIsIVM(rel) && stmt->unique)
+ {
+ for (int i = 0; i < indexInfo->ii_NumIndexKeyAttrs; i++)
+ {
+ AttrNumber attno = indexInfo->ii_IndexAttrNumbers[i];
+ if (attno > 0)
+ {
+ char *name = NameStr(TupleDescAttr(rel->rd_att, attno - 1)->attname);
+ if (name && isIvmName(name))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("unique index creation on IVM columns is not supported")));
+ }
+ }
+
+ if (indexInfo->ii_Expressions)
+ {
+ Bitmapset *indexattrs = NULL;
+ int varno = -1;
+
+ pull_varattnos((Node *) indexInfo->ii_Expressions, 1, &indexattrs);
+
+ while ((varno = bms_next_member(indexattrs, varno)) >= 0)
+ {
+ int attno = varno + FirstLowInvalidHeapAttributeNumber;
+ char *name = NameStr(TupleDescAttr(rel->rd_att, attno - 1)->attname);
+ if (name && isIvmName(name))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("unique index creation on IVM columns is not supported")));
+ }
+
+ }
+ }
+
+
/*
* Report index creation if appropriate (delay this till after most of the
* error checks)
diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c
index a821992a37..dbcbc79fff 100644
--- a/src/backend/commands/matview.c
+++ b/src/backend/commands/matview.c
@@ -152,11 +152,15 @@ static Query *rewrite_query_for_postupdate_state(Query *query, MV_TriggerTable *
static void apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *new_tuplestores,
TupleDesc tupdesc_old, TupleDesc tupdesc_new,
- Query *query);
+ Query *query, bool use_count, char *count_colname);
static void apply_old_delta(const char *matviewname, const char *deltaname_old,
List *keys);
+static void apply_old_delta_with_count(const char *matviewname, const char *deltaname_old,
+ List *keys, const char *count_colname);
static void apply_new_delta(const char *matviewname, const char *deltaname_new,
StringInfo target_list);
+static void apply_new_delta_with_count(const char *matviewname, const char* deltaname_new,
+ List *keys, StringInfo target_list, const char* count_colname);
static char *get_matching_condition_string(List *keys);
static void generate_equal(StringInfo querybuf, Oid opttype,
const char *leftop, const char *rightop);
@@ -271,6 +275,7 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
Oid matviewOid;
Relation matviewRel;
Query *dataQuery;
+ Query *viewQuery;
Oid tableSpace;
Oid relowner;
Oid OIDNewHeap;
@@ -330,8 +335,13 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
"CONCURRENTLY", "WITH NO DATA")));
- dataQuery = get_matview_query(matviewRel);
+ viewQuery = get_matview_query(matviewRel);
+ /* For IMMV, we need to rewrite matview query */
+ if (!stmt->skipData && RelationIsIVM(matviewRel))
+ dataQuery = rewriteQueryForIMMV(viewQuery,NIL);
+ else
+ dataQuery = viewQuery;
/*
* Check that there is a unique index with no WHERE clause on one or more
@@ -511,8 +521,8 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
if (!stmt->skipData && RelationIsIVM(matviewRel) && !oldPopulated)
{
- CreateIndexOnIMMV(dataQuery, matviewRel);
- CreateIvmTriggersOnBaseTables(dataQuery, matviewOid);
+ CreateIndexOnIMMV(viewQuery, matviewRel);
+ CreateIvmTriggersOnBaseTables(viewQuery, matviewOid);
}
table_close(matviewRel, NoLock);
@@ -1534,6 +1544,13 @@ IVM_immediate_maintenance(PG_FUNCTION_ARGS)
int rte_index = lfirst_int(lc2);
TupleDesc tupdesc_old;
TupleDesc tupdesc_new;
+ bool use_count = false;
+ char *count_colname = NULL;
+
+ count_colname = pstrdup("__ivm_count__");
+
+ if (query->distinctClause)
+ use_count = true;
/* calculate delta tables */
calc_delta(table, rte_index, rewritten, dest_old, dest_new,
@@ -1546,7 +1563,8 @@ IVM_immediate_maintenance(PG_FUNCTION_ARGS)
{
/* apply the delta tables to the materialized view */
apply_delta(matviewOid, old_tuplestore, new_tuplestore,
- tupdesc_old, tupdesc_new, query);
+ tupdesc_old, tupdesc_new, query, use_count,
+ count_colname);
}
PG_CATCH();
{
@@ -2019,7 +2037,7 @@ rewrite_query_for_postupdate_state(Query *query, MV_TriggerTable *table, int rte
static void
apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *new_tuplestores,
TupleDesc tupdesc_old, TupleDesc tupdesc_new,
- Query *query)
+ Query *query, bool use_count, char *count_colname)
{
StringInfoData querybuf;
StringInfoData target_list_buf;
@@ -2095,7 +2113,12 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
if (rc != SPI_OK_REL_REGISTER)
elog(ERROR, "SPI_register failed");
- apply_old_delta(matviewname, OLD_DELTA_ENRNAME, keys);
+ if (use_count)
+ /* apply old delta and get rows to be recalculated */
+ apply_old_delta_with_count(matviewname, OLD_DELTA_ENRNAME,
+ keys, count_colname);
+ else
+ apply_old_delta(matviewname, OLD_DELTA_ENRNAME, keys);
}
/* For tuple insertion */
@@ -2117,7 +2140,11 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
elog(ERROR, "SPI_register failed");
/* apply new delta */
- apply_new_delta(matviewname, NEW_DELTA_ENRNAME, &target_list_buf);
+ if (use_count)
+ apply_new_delta_with_count(matviewname, NEW_DELTA_ENRNAME,
+ keys, &target_list_buf, count_colname);
+ else
+ apply_new_delta(matviewname, NEW_DELTA_ENRNAME, &target_list_buf);
}
/* We're done maintaining the materialized view. */
@@ -2130,6 +2157,51 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
elog(ERROR, "SPI_finish failed");
}
+/*
+ * apply_old_delta_with_count
+ *
+ * Execute a query for applying a delta table given by deltname_old
+ * which contains tuples to be deleted from to a materialized view given by
+ * matviewname. This is used when counting is required, that is, the view
+ * has aggregate or distinct.
+ */
+static void
+apply_old_delta_with_count(const char *matviewname, const char *deltaname_old,
+ List *keys, const char *count_colname)
+{
+ StringInfoData querybuf;
+ char *match_cond;
+
+ /* build WHERE condition for searching tuples to be deleted */
+ match_cond = get_matching_condition_string(keys);
+
+ /* Search for matching tuples from the view and update or delete if found. */
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH t AS (" /* collecting tid of target tuples in the view */
+ "SELECT diff.%s, " /* count column */
+ "(diff.%s OPERATOR(pg_catalog.=) mv.%s) AS for_dlt, "
+ "mv.ctid "
+ "FROM %s AS mv, %s AS diff "
+ "WHERE %s" /* tuple matching condition */
+ "), updt AS (" /* update a tuple if this is not to be deleted */
+ "UPDATE %s AS mv SET %s = mv.%s OPERATOR(pg_catalog.-) t.%s "
+ "FROM t WHERE mv.ctid OPERATOR(pg_catalog.=) t.ctid AND NOT for_dlt "
+ ")"
+ /* delete a tuple if this is to be deleted */
+ "DELETE FROM %s AS mv USING t "
+ "WHERE mv.ctid OPERATOR(pg_catalog.=) t.ctid AND for_dlt",
+ count_colname,
+ count_colname, count_colname,
+ matviewname, deltaname_old,
+ match_cond,
+ matviewname, count_colname, count_colname, count_colname,
+ matviewname);
+
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_DELETE)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+}
+
/*
* apply_old_delta
*
@@ -2179,6 +2251,66 @@ apply_old_delta(const char *matviewname, const char *deltaname_old,
elog(ERROR, "SPI_exec failed: %s", querybuf.data);
}
+/*
+ * apply_new_delta_with_count
+ *
+ * Execute a query for applying a delta table given by deltname_new
+ * which contains tuples to be inserted into a materialized view given by
+ * matviewname. This is used when counting is required, that is, the view
+ * has aggregate or distinct. Also, when a table in EXISTS sub queries
+ * is modified.
+ */
+static void
+apply_new_delta_with_count(const char *matviewname, const char* deltaname_new,
+ List *keys, StringInfo target_list, const char* count_colname)
+{
+ StringInfoData querybuf;
+ StringInfoData returning_keys;
+ ListCell *lc;
+ char *match_cond = "";
+
+ /* build WHERE condition for searching tuples to be updated */
+ match_cond = get_matching_condition_string(keys);
+
+ /* build string of keys list */
+ initStringInfo(&returning_keys);
+ if (keys)
+ {
+ foreach (lc, keys)
+ {
+ Form_pg_attribute attr = (Form_pg_attribute) lfirst(lc);
+ char *resname = NameStr(attr->attname);
+ appendStringInfo(&returning_keys, "%s", quote_qualified_identifier("mv", resname));
+ if (lnext(keys, lc))
+ appendStringInfo(&returning_keys, ", ");
+ }
+ }
+ else
+ appendStringInfo(&returning_keys, "NULL");
+
+ /* Search for matching tuples from the view and update if found or insert if not. */
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH updt AS (" /* update a tuple if this exists in the view */
+ "UPDATE %s AS mv SET %s = mv.%s OPERATOR(pg_catalog.+) diff.%s "
+ "FROM %s AS diff "
+ "WHERE %s " /* tuple matching condition */
+ "RETURNING %s" /* returning keys of updated tuples */
+ ") INSERT INTO %s (%s) " /* insert a new tuple if this doesn't exist */
+ "SELECT %s FROM %s AS diff "
+ "WHERE NOT EXISTS (SELECT 1 FROM updt AS mv WHERE %s);",
+ matviewname, count_colname, count_colname, count_colname,
+ deltaname_new,
+ match_cond,
+ returning_keys.data,
+ matviewname, target_list->data,
+ target_list->data, deltaname_new,
+ match_cond);
+
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+}
+
/*
* apply_new_delta
*
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index f798794556..02a688acb7 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -53,6 +53,7 @@
#include "commands/cluster.h"
#include "commands/comment.h"
#include "commands/defrem.h"
+#include "commands/matview.h"
#include "commands/event_trigger.h"
#include "commands/policy.h"
#include "commands/sequence.h"
@@ -3784,6 +3785,14 @@ renameatt_internal(Oid myrelid,
targetrelation = relation_open(myrelid, AccessExclusiveLock);
renameatt_check(myrelid, RelationGetForm(targetrelation), recursing);
+ /*
+ * Don't rename IVM columns.
+ */
+ if (RelationIsIVM(targetrelation) && isIvmName(oldattname))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("IVM column can not be renamed")));
+
/*
* if the 'recurse' flag is set then we are supposed to rename this
* attribute in all classes that inherit from 'relname' (as well as in
diff --git a/src/backend/parser/parse_relation.c b/src/backend/parser/parse_relation.c
index 34a0ec5901..d5b45c587d 100644
--- a/src/backend/parser/parse_relation.c
+++ b/src/backend/parser/parse_relation.c
@@ -36,6 +36,7 @@
#include "utils/rel.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
+#include "commands/matview.h"
/*
@@ -97,7 +98,7 @@ static void expandTupleDesc(TupleDesc tupdesc, Alias *eref,
int count, int offset,
int rtindex, int sublevels_up,
int location, bool include_dropped,
- List **colnames, List **colvars);
+ List **colnames, List **colvars, bool is_ivm);
static int specialAttNum(const char *attname);
static bool rte_visible_if_lateral(ParseState *pstate, RangeTblEntry *rte);
static bool rte_visible_if_qualified(ParseState *pstate, RangeTblEntry *rte);
@@ -1502,6 +1503,7 @@ addRangeTableEntry(ParseState *pstate,
rte->relid = RelationGetRelid(rel);
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -1587,6 +1589,7 @@ addRangeTableEntryForRelation(ParseState *pstate,
rte->relid = RelationGetRelid(rel);
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -2758,7 +2761,7 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
expandTupleDesc(tupdesc, rte->eref,
rtfunc->funccolcount, atts_done,
rtindex, sublevels_up, location,
- include_dropped, colnames, colvars);
+ include_dropped, colnames, colvars, false);
}
else if (functypclass == TYPEFUNC_SCALAR)
{
@@ -3026,7 +3029,7 @@ expandRelation(Oid relid, Alias *eref, int rtindex, int sublevels_up,
expandTupleDesc(rel->rd_att, eref, rel->rd_att->natts, 0,
rtindex, sublevels_up,
location, include_dropped,
- colnames, colvars);
+ colnames, colvars, RelationIsIVM(rel));
relation_close(rel, AccessShareLock);
}
@@ -3043,7 +3046,7 @@ static void
expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
int rtindex, int sublevels_up,
int location, bool include_dropped,
- List **colnames, List **colvars)
+ List **colnames, List **colvars, bool is_ivm)
{
ListCell *aliascell;
int varattno;
@@ -3056,6 +3059,9 @@ expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
{
Form_pg_attribute attr = TupleDescAttr(tupdesc, varattno);
+ if (is_ivm && isIvmName(NameStr(attr->attname)) && !MatViewIncrementalMaintenanceIsEnabled())
+ continue;
+
if (attr->attisdropped)
{
if (include_dropped)
@@ -3218,6 +3224,10 @@ expandNSItemAttrs(ParseState *pstate, ParseNamespaceItem *nsitem,
Var *varnode = (Var *) lfirst(var);
TargetEntry *te;
+ /* if transform * into columnlist with IMMV, remove IVM columns */
+ if (rte->relisivm && isIvmName(label) && !MatViewIncrementalMaintenanceIsEnabled())
+ continue;
+
te = makeTargetEntry((Expr *) varnode,
(AttrNumber) pstate->p_next_resno++,
label,
diff --git a/src/backend/rewrite/rewriteDefine.c b/src/backend/rewrite/rewriteDefine.c
index b449244a53..b9d161d6a3 100644
--- a/src/backend/rewrite/rewriteDefine.c
+++ b/src/backend/rewrite/rewriteDefine.c
@@ -621,7 +621,8 @@ checkRuleResultList(List *targetList, TupleDesc resultDesc, bool isSelect,
attr->atttypmod))));
}
- if (i != resultDesc->natts)
+ /* No check for materialized views since this could have special columns for IVM */
+ if ((!isSelect || requireColumnNameMatch) && i != resultDesc->natts)
ereport(ERROR,
(errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
isSelect ?
diff --git a/src/include/commands/createas.h b/src/include/commands/createas.h
index 396ad1bb4c..6b47e66bfd 100644
--- a/src/include/commands/createas.h
+++ b/src/include/commands/createas.h
@@ -29,6 +29,8 @@ extern ObjectAddress ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *st
extern void CreateIvmTriggersOnBaseTables(Query *qry, Oid matviewOid);
extern void CreateIndexOnIMMV(Query *query, Relation matviewRel);
+extern Query *rewriteQueryForIMMV(Query *query, List *colNames);
+
extern int GetIntoRelEFlags(IntoClause *intoClause);
extern DestReceiver *CreateIntoRelDestReceiver(IntoClause *intoClause);
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index d60e148ff2..fa3a95897c 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -1074,6 +1074,7 @@ typedef struct RangeTblEntry
int rellockmode; /* lock level that query requires on the rel */
struct TableSampleClause *tablesample; /* sampling info, or NULL */
Index perminfoindex;
+ bool relisivm;
/*
* Fields valid for a subquery RTE (else NULL):
--
2.25.1
--Multipart=_Mon__4_Mar_2024_11_58_46_+0900_UaponF/qQhQrVCFt
Content-Type: text/x-diff;
name="v30-0008-Add-aggregates-support-in-IVM.patch"
Content-Disposition: attachment;
filename="v30-0008-Add-aggregates-support-in-IVM.patch"
Content-Transfer-Encoding: quoted-printable
^ permalink raw reply [nested|flat] 24+ messages in thread
* [PATCH v31 07/11] Add DISTINCT support for IVM
@ 2023-05-31 10:08 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 24+ messages in thread
From: Yugo Nagata @ 2023-05-31 10:08 UTC (permalink / raw)
When IMMV is created with DISTINCT, multiplicity of tuples is
counted and stored in "__ivm_count__" column, which is a hidden
column of IMMV. The value in __ivm_count__ is updated when IMMV
is maintained incrementally. A tuple in IMMV can be removed if
and only if the count becomes zero.
---
src/backend/commands/createas.c | 141 ++++++++++++++++++++------
src/backend/commands/indexcmds.c | 40 ++++++++
src/backend/commands/matview.c | 148 ++++++++++++++++++++++++++--
src/backend/commands/tablecmds.c | 9 ++
src/backend/nodes/outfuncs.c | 1 +
src/backend/nodes/readfuncs.c | 1 +
src/backend/parser/parse_relation.c | 18 +++-
src/backend/rewrite/rewriteDefine.c | 3 +-
src/include/commands/createas.h | 2 +
src/include/nodes/parsenodes.h | 2 +
10 files changed, 320 insertions(+), 45 deletions(-)
diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c
index 04a5ee9e37..8f2bd5203e 100644
--- a/src/backend/commands/createas.c
+++ b/src/backend/commands/createas.c
@@ -50,6 +50,7 @@
#include "parser/parser.h"
#include "parser/parsetree.h"
#include "parser/parse_clause.h"
+#include "parser/parse_func.h"
#include "rewrite/rewriteHandler.h"
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
@@ -305,6 +306,9 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt,
errhint("functions must be marked IMMUTABLE")));
check_ivm_restriction((Node *) query);
+
+ /* For IMMV, we need to rewrite matview query */
+ query = rewriteQueryForIMMV(query, into->colNames);
}
if (into->skipData)
@@ -409,6 +413,49 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt,
return address;
}
+/*
+ * rewriteQueryForIMMV -- rewrite view definition query for IMMV
+ *
+ * count(*) is added for counting distinct tuples in views.
+ */
+Query *
+rewriteQueryForIMMV(Query *query, List *colNames)
+{
+ Query *rewritten;
+
+ Node *node;
+ ParseState *pstate = make_parsestate(NULL);
+ FuncCall *fn;
+
+ rewritten = copyObject(query);
+ pstate->p_expr_kind = EXPR_KIND_SELECT_TARGET;
+
+ /*
+ * Convert DISTINCT to GROUP BY and add count(*) for counting distinct
+ * tuples in views.
+ */
+ if (rewritten->distinctClause)
+ {
+ TargetEntry *tle;
+
+ rewritten->groupClause = transformDistinctClause(NULL, &rewritten->targetList, rewritten->sortClause, false);
+
+ fn = makeFuncCall(SystemFuncName("count"), NIL, COERCE_EXPLICIT_CALL, -1);
+ fn->agg_star = true;
+
+ node = ParseFuncOrColumn(pstate, fn->funcname, NIL, NULL, fn, false, -1);
+
+ tle = makeTargetEntry((Expr *) node,
+ list_length(rewritten->targetList) + 1,
+ pstrdup("__ivm_count__"),
+ false);
+ rewritten->targetList = lappend(rewritten->targetList, tle);
+ rewritten->hasAggs = true;
+ }
+
+ return rewritten;
+}
+
/*
* GetIntoRelEFlags --- compute executor flags needed for CREATE TABLE AS
*
@@ -532,7 +579,8 @@ intorel_startup(DestReceiver *self, int operation, TupleDesc typeinfo)
ColumnDef *col;
char *colname;
- if (lc)
+ /* Don't override hidden columns added for IVM */
+ if (lc && !isIvmName(NameStr(attribute->attname)))
{
colname = strVal(lfirst(lc));
lc = lnext(into->colNames, lc);
@@ -936,10 +984,6 @@ check_ivm_restriction_walker(Node *node, void *context)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("LIMIT/OFFSET clause is not supported on incrementally maintainable materialized view")));
- if (qry->distinctClause)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("DISTINCT is not supported on incrementally maintainable materialized view")));
if (qry->hasDistinctOn)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -1086,12 +1130,18 @@ CreateIndexOnIMMV(Query *query, Relation matviewRel)
char idxname[NAMEDATALEN];
List *indexoidlist = RelationGetIndexList(matviewRel);
ListCell *indexoidscan;
- Bitmapset *key_attnos;
snprintf(idxname, sizeof(idxname), "%s_index", RelationGetRelationName(matviewRel));
index = makeNode(IndexStmt);
+ /*
+ * We consider null values not distinct to make sure that views with DISTINCT
+ * or GROUP BY don't contain multiple NULL rows when NULL is inserted to
+ * a base table concurrently.
+ */
+ index->nulls_not_distinct = true;
+
index->unique = true;
index->primary = false;
index->isconstraint = false;
@@ -1118,41 +1168,68 @@ CreateIndexOnIMMV(Query *query, Relation matviewRel)
index->concurrent = false;
index->if_not_exists = false;
- /* create index on the base tables' primary key columns */
- key_attnos = get_primary_key_attnos_from_query(query, &constraintList);
- if (key_attnos)
+ if (query->distinctClause)
{
+ /* create unique constraint on all columns */
foreach(lc, query->targetList)
{
TargetEntry *tle = (TargetEntry *) lfirst(lc);
Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, tle->resno - 1);
-
- if (bms_is_member(tle->resno - FirstLowInvalidHeapAttributeNumber, key_attnos))
- {
- IndexElem *iparam;
-
- iparam = makeNode(IndexElem);
- iparam->name = pstrdup(NameStr(attr->attname));
- iparam->expr = NULL;
- iparam->indexcolname = NULL;
- iparam->collation = NIL;
- iparam->opclass = NIL;
- iparam->opclassopts = NIL;
- iparam->ordering = SORTBY_DEFAULT;
- iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
- index->indexParams = lappend(index->indexParams, iparam);
- }
+ IndexElem *iparam;
+
+ iparam = makeNode(IndexElem);
+ iparam->name = pstrdup(NameStr(attr->attname));
+ iparam->expr = NULL;
+ iparam->indexcolname = NULL;
+ iparam->collation = NIL;
+ iparam->opclass = NIL;
+ iparam->opclassopts = NIL;
+ iparam->ordering = SORTBY_DEFAULT;
+ iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
+ index->indexParams = lappend(index->indexParams, iparam);
}
}
else
{
- /* create no index, just notice that an appropriate index is necessary for efficient IVM */
- ereport(NOTICE,
- (errmsg("could not create an index on materialized view \"%s\" automatically",
- RelationGetRelationName(matviewRel)),
- errdetail("This target list does not have all the primary key columns. "),
- errhint("Create an index on the materialized view for efficient incremental maintenance.")));
- return;
+ Bitmapset *key_attnos;
+
+ /* create index on the base tables' primary key columns */
+ key_attnos = get_primary_key_attnos_from_query(query, &constraintList);
+ if (key_attnos)
+ {
+ foreach(lc, query->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);
+ Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, tle->resno - 1);
+
+ if (bms_is_member(tle->resno - FirstLowInvalidHeapAttributeNumber, key_attnos))
+ {
+ IndexElem *iparam;
+
+ iparam = makeNode(IndexElem);
+ iparam->name = pstrdup(NameStr(attr->attname));
+ iparam->expr = NULL;
+ iparam->indexcolname = NULL;
+ iparam->collation = NIL;
+ iparam->opclass = NIL;
+ iparam->opclassopts = NIL;
+ iparam->ordering = SORTBY_DEFAULT;
+ iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
+ index->indexParams = lappend(index->indexParams, iparam);
+ }
+ }
+ }
+ else
+ {
+ /* create no index, just notice that an appropriate index is necessary for efficient IVM */
+ ereport(NOTICE,
+ (errmsg("could not create an index on materialized view \"%s\" automatically",
+ RelationGetRelationName(matviewRel)),
+ errdetail("This target list does not have all the primary key columns, "
+ "or this view does not contain DISTINCT clause."),
+ errhint("Create an index on the materialized view for efficient incremental maintenance.")));
+ return;
+ }
}
/* If we have a compatible index, we don't need to create another. */
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index d9016ef487..fb5265e6c3 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -41,6 +41,7 @@
#include "commands/dbcommands.h"
#include "commands/defrem.h"
#include "commands/event_trigger.h"
+#include "commands/matview.h"
#include "commands/progress.h"
#include "commands/tablecmds.h"
#include "commands/tablespace.h"
@@ -1132,6 +1133,45 @@ DefineIndex(Oid tableId,
safe_index = indexInfo->ii_Expressions == NIL &&
indexInfo->ii_Predicate == NIL;
+ /*
+ * We disallow unique indexes on IVM columns of IMMVs.
+ */
+ if (RelationIsIVM(rel) && stmt->unique)
+ {
+ for (int i = 0; i < indexInfo->ii_NumIndexKeyAttrs; i++)
+ {
+ AttrNumber attno = indexInfo->ii_IndexAttrNumbers[i];
+ if (attno > 0)
+ {
+ char *name = NameStr(TupleDescAttr(rel->rd_att, attno - 1)->attname);
+ if (name && isIvmName(name))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("unique index creation on IVM columns is not supported")));
+ }
+ }
+
+ if (indexInfo->ii_Expressions)
+ {
+ Bitmapset *indexattrs = NULL;
+ int varno = -1;
+
+ pull_varattnos((Node *) indexInfo->ii_Expressions, 1, &indexattrs);
+
+ while ((varno = bms_next_member(indexattrs, varno)) >= 0)
+ {
+ int attno = varno + FirstLowInvalidHeapAttributeNumber;
+ char *name = NameStr(TupleDescAttr(rel->rd_att, attno - 1)->attname);
+ if (name && isIvmName(name))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("unique index creation on IVM columns is not supported")));
+ }
+
+ }
+ }
+
+
/*
* Report index creation if appropriate (delay this till after most of the
* error checks)
diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c
index 1061c37b2c..f2e8aa02a3 100644
--- a/src/backend/commands/matview.c
+++ b/src/backend/commands/matview.c
@@ -148,11 +148,15 @@ static Query *rewrite_query_for_postupdate_state(Query *query, MV_TriggerTable *
static void apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *new_tuplestores,
TupleDesc tupdesc_old, TupleDesc tupdesc_new,
- Query *query);
+ Query *query, bool use_count, char *count_colname);
static void apply_old_delta(const char *matviewname, const char *deltaname_old,
List *keys);
+static void apply_old_delta_with_count(const char *matviewname, const char *deltaname_old,
+ List *keys, const char *count_colname);
static void apply_new_delta(const char *matviewname, const char *deltaname_new,
StringInfo target_list);
+static void apply_new_delta_with_count(const char *matviewname, const char* deltaname_new,
+ List *keys, StringInfo target_list, const char* count_colname);
static char *get_matching_condition_string(List *keys);
static void generate_equal(StringInfo querybuf, Oid opttype,
const char *leftop, const char *rightop);
@@ -267,6 +271,7 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
Oid matviewOid;
Relation matviewRel;
Query *dataQuery;
+ Query *viewQuery;
Oid tableSpace;
Oid relowner;
Oid OIDNewHeap;
@@ -329,8 +334,13 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
"CONCURRENTLY", "WITH NO DATA")));
- dataQuery = get_matview_query(matviewRel);
+ viewQuery = get_matview_query(matviewRel);
+ /* For IMMV, we need to rewrite matview query */
+ if (!stmt->skipData && RelationIsIVM(matviewRel))
+ dataQuery = rewriteQueryForIMMV(viewQuery,NIL);
+ else
+ dataQuery = viewQuery;
/*
* Check that there is a unique index with no WHERE clause on one or more
@@ -510,8 +520,8 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
if (!stmt->skipData && RelationIsIVM(matviewRel) && !oldPopulated)
{
- CreateIndexOnIMMV(dataQuery, matviewRel);
- CreateIvmTriggersOnBaseTables(dataQuery, matviewOid);
+ CreateIndexOnIMMV(viewQuery, matviewRel);
+ CreateIvmTriggersOnBaseTables(viewQuery, matviewOid);
}
table_close(matviewRel, NoLock);
@@ -1533,6 +1543,13 @@ IVM_immediate_maintenance(PG_FUNCTION_ARGS)
int rte_index = lfirst_int(lc2);
TupleDesc tupdesc_old;
TupleDesc tupdesc_new;
+ bool use_count = false;
+ char *count_colname = NULL;
+
+ count_colname = pstrdup("__ivm_count__");
+
+ if (query->distinctClause)
+ use_count = true;
/* calculate delta tables */
calc_delta(table, rte_index, rewritten, dest_old, dest_new,
@@ -1545,7 +1562,8 @@ IVM_immediate_maintenance(PG_FUNCTION_ARGS)
{
/* apply the delta tables to the materialized view */
apply_delta(matviewOid, old_tuplestore, new_tuplestore,
- tupdesc_old, tupdesc_new, query);
+ tupdesc_old, tupdesc_new, query, use_count,
+ count_colname);
}
PG_CATCH();
{
@@ -2018,7 +2036,7 @@ rewrite_query_for_postupdate_state(Query *query, MV_TriggerTable *table, int rte
static void
apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *new_tuplestores,
TupleDesc tupdesc_old, TupleDesc tupdesc_new,
- Query *query)
+ Query *query, bool use_count, char *count_colname)
{
StringInfoData querybuf;
StringInfoData target_list_buf;
@@ -2094,7 +2112,12 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
if (rc != SPI_OK_REL_REGISTER)
elog(ERROR, "SPI_register failed");
- apply_old_delta(matviewname, OLD_DELTA_ENRNAME, keys);
+ if (use_count)
+ /* apply old delta and get rows to be recalculated */
+ apply_old_delta_with_count(matviewname, OLD_DELTA_ENRNAME,
+ keys, count_colname);
+ else
+ apply_old_delta(matviewname, OLD_DELTA_ENRNAME, keys);
}
/* For tuple insertion */
@@ -2116,7 +2139,11 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
elog(ERROR, "SPI_register failed");
/* apply new delta */
- apply_new_delta(matviewname, NEW_DELTA_ENRNAME, &target_list_buf);
+ if (use_count)
+ apply_new_delta_with_count(matviewname, NEW_DELTA_ENRNAME,
+ keys, &target_list_buf, count_colname);
+ else
+ apply_new_delta(matviewname, NEW_DELTA_ENRNAME, &target_list_buf);
}
/* We're done maintaining the materialized view. */
@@ -2129,6 +2156,51 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
elog(ERROR, "SPI_finish failed");
}
+/*
+ * apply_old_delta_with_count
+ *
+ * Execute a query for applying a delta table given by deltname_old
+ * which contains tuples to be deleted from to a materialized view given by
+ * matviewname. This is used when counting is required, that is, the view
+ * has aggregate or distinct.
+ */
+static void
+apply_old_delta_with_count(const char *matviewname, const char *deltaname_old,
+ List *keys, const char *count_colname)
+{
+ StringInfoData querybuf;
+ char *match_cond;
+
+ /* build WHERE condition for searching tuples to be deleted */
+ match_cond = get_matching_condition_string(keys);
+
+ /* Search for matching tuples from the view and update or delete if found. */
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH t AS (" /* collecting tid of target tuples in the view */
+ "SELECT diff.%s, " /* count column */
+ "(diff.%s OPERATOR(pg_catalog.=) mv.%s) AS for_dlt, "
+ "mv.ctid "
+ "FROM %s AS mv, %s AS diff "
+ "WHERE %s" /* tuple matching condition */
+ "), updt AS (" /* update a tuple if this is not to be deleted */
+ "UPDATE %s AS mv SET %s = mv.%s OPERATOR(pg_catalog.-) t.%s "
+ "FROM t WHERE mv.ctid OPERATOR(pg_catalog.=) t.ctid AND NOT for_dlt "
+ ")"
+ /* delete a tuple if this is to be deleted */
+ "DELETE FROM %s AS mv USING t "
+ "WHERE mv.ctid OPERATOR(pg_catalog.=) t.ctid AND for_dlt",
+ count_colname,
+ count_colname, count_colname,
+ matviewname, deltaname_old,
+ match_cond,
+ matviewname, count_colname, count_colname, count_colname,
+ matviewname);
+
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_DELETE)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+}
+
/*
* apply_old_delta
*
@@ -2178,6 +2250,66 @@ apply_old_delta(const char *matviewname, const char *deltaname_old,
elog(ERROR, "SPI_exec failed: %s", querybuf.data);
}
+/*
+ * apply_new_delta_with_count
+ *
+ * Execute a query for applying a delta table given by deltname_new
+ * which contains tuples to be inserted into a materialized view given by
+ * matviewname. This is used when counting is required, that is, the view
+ * has aggregate or distinct. Also, when a table in EXISTS sub queries
+ * is modified.
+ */
+static void
+apply_new_delta_with_count(const char *matviewname, const char* deltaname_new,
+ List *keys, StringInfo target_list, const char* count_colname)
+{
+ StringInfoData querybuf;
+ StringInfoData returning_keys;
+ ListCell *lc;
+ char *match_cond = "";
+
+ /* build WHERE condition for searching tuples to be updated */
+ match_cond = get_matching_condition_string(keys);
+
+ /* build string of keys list */
+ initStringInfo(&returning_keys);
+ if (keys)
+ {
+ foreach (lc, keys)
+ {
+ Form_pg_attribute attr = (Form_pg_attribute) lfirst(lc);
+ char *resname = NameStr(attr->attname);
+ appendStringInfo(&returning_keys, "%s", quote_qualified_identifier("mv", resname));
+ if (lnext(keys, lc))
+ appendStringInfo(&returning_keys, ", ");
+ }
+ }
+ else
+ appendStringInfo(&returning_keys, "NULL");
+
+ /* Search for matching tuples from the view and update if found or insert if not. */
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH updt AS (" /* update a tuple if this exists in the view */
+ "UPDATE %s AS mv SET %s = mv.%s OPERATOR(pg_catalog.+) diff.%s "
+ "FROM %s AS diff "
+ "WHERE %s " /* tuple matching condition */
+ "RETURNING %s" /* returning keys of updated tuples */
+ ") INSERT INTO %s (%s) " /* insert a new tuple if this doesn't exist */
+ "SELECT %s FROM %s AS diff "
+ "WHERE NOT EXISTS (SELECT 1 FROM updt AS mv WHERE %s);",
+ matviewname, count_colname, count_colname, count_colname,
+ deltaname_new,
+ match_cond,
+ returning_keys.data,
+ matviewname, target_list->data,
+ target_list->data, deltaname_new,
+ match_cond);
+
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+}
+
/*
* apply_new_delta
*
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 6741e721ae..dad09b9b0b 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -56,6 +56,7 @@
#include "commands/cluster.h"
#include "commands/comment.h"
#include "commands/defrem.h"
+#include "commands/matview.h"
#include "commands/event_trigger.h"
#include "commands/sequence.h"
#include "commands/tablecmds.h"
@@ -3789,6 +3790,14 @@ renameatt_internal(Oid myrelid,
targetrelation = relation_open(myrelid, AccessExclusiveLock);
renameatt_check(myrelid, RelationGetForm(targetrelation), recursing);
+ /*
+ * Don't rename IVM columns.
+ */
+ if (RelationIsIVM(targetrelation) && isIvmName(oldattname))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("IVM column can not be renamed")));
+
/*
* if the 'recurse' flag is set then we are supposed to rename this
* attribute in all classes that inherit from 'relname' (as well as in
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 3337b77ae6..c191f70a6f 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -510,6 +510,7 @@ _outRangeTblEntry(StringInfo str, const RangeTblEntry *node)
WRITE_INT_FIELD(rellockmode);
WRITE_UINT_FIELD(perminfoindex);
WRITE_NODE_FIELD(tablesample);
+ WRITE_BOOL_FIELD(relisivm);
break;
case RTE_SUBQUERY:
WRITE_NODE_FIELD(subquery);
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index c4d01a441a..ffcab8cda2 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -361,6 +361,7 @@ _readRangeTblEntry(void)
READ_INT_FIELD(rellockmode);
READ_UINT_FIELD(perminfoindex);
READ_NODE_FIELD(tablesample);
+ READ_BOOL_FIELD(relisivm);
break;
case RTE_SUBQUERY:
READ_NODE_FIELD(subquery);
diff --git a/src/backend/parser/parse_relation.c b/src/backend/parser/parse_relation.c
index 427b7325db..65aecc96a7 100644
--- a/src/backend/parser/parse_relation.c
+++ b/src/backend/parser/parse_relation.c
@@ -36,6 +36,7 @@
#include "utils/rel.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
+#include "commands/matview.h"
/*
@@ -97,7 +98,7 @@ static void expandTupleDesc(TupleDesc tupdesc, Alias *eref,
int count, int offset,
int rtindex, int sublevels_up,
int location, bool include_dropped,
- List **colnames, List **colvars);
+ List **colnames, List **colvars, bool is_ivm);
static int specialAttNum(const char *attname);
static bool rte_visible_if_lateral(ParseState *pstate, RangeTblEntry *rte);
static bool rte_visible_if_qualified(ParseState *pstate, RangeTblEntry *rte);
@@ -1503,6 +1504,7 @@ addRangeTableEntry(ParseState *pstate,
rte->inh = inh;
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -1588,6 +1590,7 @@ addRangeTableEntryForRelation(ParseState *pstate,
rte->inh = inh;
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -2752,7 +2755,7 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
expandTupleDesc(tupdesc, rte->eref,
rtfunc->funccolcount, atts_done,
rtindex, sublevels_up, location,
- include_dropped, colnames, colvars);
+ include_dropped, colnames, colvars, false);
}
else if (functypclass == TYPEFUNC_SCALAR)
{
@@ -3020,7 +3023,7 @@ expandRelation(Oid relid, Alias *eref, int rtindex, int sublevels_up,
expandTupleDesc(rel->rd_att, eref, rel->rd_att->natts, 0,
rtindex, sublevels_up,
location, include_dropped,
- colnames, colvars);
+ colnames, colvars, RelationIsIVM(rel));
relation_close(rel, AccessShareLock);
}
@@ -3037,7 +3040,7 @@ static void
expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
int rtindex, int sublevels_up,
int location, bool include_dropped,
- List **colnames, List **colvars)
+ List **colnames, List **colvars, bool is_ivm)
{
ListCell *aliascell;
int varattno;
@@ -3050,6 +3053,9 @@ expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
{
Form_pg_attribute attr = TupleDescAttr(tupdesc, varattno);
+ if (is_ivm && isIvmName(NameStr(attr->attname)) && !MatViewIncrementalMaintenanceIsEnabled())
+ continue;
+
if (attr->attisdropped)
{
if (include_dropped)
@@ -3212,6 +3218,10 @@ expandNSItemAttrs(ParseState *pstate, ParseNamespaceItem *nsitem,
Var *varnode = (Var *) lfirst(var);
TargetEntry *te;
+ /* if transform * into columnlist with IMMV, remove IVM columns */
+ if (rte->relisivm && isIvmName(label) && !MatViewIncrementalMaintenanceIsEnabled())
+ continue;
+
te = makeTargetEntry((Expr *) varnode,
(AttrNumber) pstate->p_next_resno++,
label,
diff --git a/src/backend/rewrite/rewriteDefine.c b/src/backend/rewrite/rewriteDefine.c
index 6cc9a8d8bf..5d22dbcfcf 100644
--- a/src/backend/rewrite/rewriteDefine.c
+++ b/src/backend/rewrite/rewriteDefine.c
@@ -614,7 +614,8 @@ checkRuleResultList(List *targetList, TupleDesc resultDesc, bool isSelect,
attr->atttypmod))));
}
- if (i != resultDesc->natts)
+ /* No check for materialized views since this could have special columns for IVM */
+ if ((!isSelect || requireColumnNameMatch) && i != resultDesc->natts)
ereport(ERROR,
(errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
isSelect ?
diff --git a/src/include/commands/createas.h b/src/include/commands/createas.h
index 396ad1bb4c..6b47e66bfd 100644
--- a/src/include/commands/createas.h
+++ b/src/include/commands/createas.h
@@ -29,6 +29,8 @@ extern ObjectAddress ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *st
extern void CreateIvmTriggersOnBaseTables(Query *qry, Oid matviewOid);
extern void CreateIndexOnIMMV(Query *query, Relation matviewRel);
+extern Query *rewriteQueryForIMMV(Query *query, List *colNames);
+
extern int GetIntoRelEFlags(IntoClause *intoClause);
extern DestReceiver *CreateIntoRelDestReceiver(IntoClause *intoClause);
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index b89baef95d..bfa48d659e 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -1089,6 +1089,8 @@ typedef struct RangeTblEntry
Index perminfoindex pg_node_attr(query_jumble_ignore);
/* sampling info, or NULL */
struct TableSampleClause *tablesample;
+ /* incrementally maintainable materialized view? */
+ bool relisivm;
/*
* Fields valid for a subquery RTE (else NULL):
--
2.25.1
--Multipart=_Fri__29_Mar_2024_23_47_00_+0900_KGpmmDOIs1266Ib1
Content-Type: text/x-diff;
name="v31-0008-Add-aggregates-support-in-IVM.patch"
Content-Disposition: attachment;
filename="v31-0008-Add-aggregates-support-in-IVM.patch"
Content-Transfer-Encoding: quoted-printable
^ permalink raw reply [nested|flat] 24+ messages in thread
* [PATCH v32 07/11] Add DISTINCT support for IVM
@ 2023-05-31 10:08 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 24+ messages in thread
From: Yugo Nagata @ 2023-05-31 10:08 UTC (permalink / raw)
When IMMV is created with DISTINCT, multiplicity of tuples is
counted and stored in "__ivm_count__" column, which is a hidden
column of IMMV. The value in __ivm_count__ is updated when IMMV
is maintained incrementally. A tuple in IMMV can be removed if
and only if the count becomes zero.
---
src/backend/commands/createas.c | 141 ++++++++++++++++++++------
src/backend/commands/indexcmds.c | 40 ++++++++
src/backend/commands/matview.c | 148 ++++++++++++++++++++++++++--
src/backend/commands/tablecmds.c | 9 ++
src/backend/nodes/outfuncs.c | 1 +
src/backend/nodes/readfuncs.c | 1 +
src/backend/parser/parse_relation.c | 18 +++-
src/backend/rewrite/rewriteDefine.c | 3 +-
src/include/commands/createas.h | 2 +
src/include/nodes/parsenodes.h | 2 +
10 files changed, 320 insertions(+), 45 deletions(-)
diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c
index e9846c8d0f..299c5a133c 100644
--- a/src/backend/commands/createas.c
+++ b/src/backend/commands/createas.c
@@ -50,6 +50,7 @@
#include "parser/parser.h"
#include "parser/parsetree.h"
#include "parser/parse_clause.h"
+#include "parser/parse_func.h"
#include "rewrite/rewriteHandler.h"
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
@@ -305,6 +306,9 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt,
errhint("functions must be marked IMMUTABLE")));
check_ivm_restriction((Node *) query);
+
+ /* For IMMV, we need to rewrite matview query */
+ query = rewriteQueryForIMMV(query, into->colNames);
}
if (into->skipData)
@@ -409,6 +413,49 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt,
return address;
}
+/*
+ * rewriteQueryForIMMV -- rewrite view definition query for IMMV
+ *
+ * count(*) is added for counting distinct tuples in views.
+ */
+Query *
+rewriteQueryForIMMV(Query *query, List *colNames)
+{
+ Query *rewritten;
+
+ Node *node;
+ ParseState *pstate = make_parsestate(NULL);
+ FuncCall *fn;
+
+ rewritten = copyObject(query);
+ pstate->p_expr_kind = EXPR_KIND_SELECT_TARGET;
+
+ /*
+ * Convert DISTINCT to GROUP BY and add count(*) for counting distinct
+ * tuples in views.
+ */
+ if (rewritten->distinctClause)
+ {
+ TargetEntry *tle;
+
+ rewritten->groupClause = transformDistinctClause(NULL, &rewritten->targetList, rewritten->sortClause, false);
+
+ fn = makeFuncCall(SystemFuncName("count"), NIL, COERCE_EXPLICIT_CALL, -1);
+ fn->agg_star = true;
+
+ node = ParseFuncOrColumn(pstate, fn->funcname, NIL, NULL, fn, false, -1);
+
+ tle = makeTargetEntry((Expr *) node,
+ list_length(rewritten->targetList) + 1,
+ pstrdup("__ivm_count__"),
+ false);
+ rewritten->targetList = lappend(rewritten->targetList, tle);
+ rewritten->hasAggs = true;
+ }
+
+ return rewritten;
+}
+
/*
* GetIntoRelEFlags --- compute executor flags needed for CREATE TABLE AS
*
@@ -532,7 +579,8 @@ intorel_startup(DestReceiver *self, int operation, TupleDesc typeinfo)
ColumnDef *col;
char *colname;
- if (lc)
+ /* Don't override hidden columns added for IVM */
+ if (lc && !isIvmName(NameStr(attribute->attname)))
{
colname = strVal(lfirst(lc));
lc = lnext(into->colNames, lc);
@@ -938,10 +986,6 @@ check_ivm_restriction_walker(Node *node, void *context)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("LIMIT/OFFSET clause is not supported on incrementally maintainable materialized view")));
- if (qry->distinctClause)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("DISTINCT is not supported on incrementally maintainable materialized view")));
if (qry->hasDistinctOn)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -1088,12 +1132,18 @@ CreateIndexOnIMMV(Query *query, Relation matviewRel)
char idxname[NAMEDATALEN];
List *indexoidlist = RelationGetIndexList(matviewRel);
ListCell *indexoidscan;
- Bitmapset *key_attnos;
snprintf(idxname, sizeof(idxname), "%s_index", RelationGetRelationName(matviewRel));
index = makeNode(IndexStmt);
+ /*
+ * We consider null values not distinct to make sure that views with DISTINCT
+ * or GROUP BY don't contain multiple NULL rows when NULL is inserted to
+ * a base table concurrently.
+ */
+ index->nulls_not_distinct = true;
+
index->unique = true;
index->primary = false;
index->isconstraint = false;
@@ -1120,41 +1170,68 @@ CreateIndexOnIMMV(Query *query, Relation matviewRel)
index->concurrent = false;
index->if_not_exists = false;
- /* create index on the base tables' primary key columns */
- key_attnos = get_primary_key_attnos_from_query(query, &constraintList);
- if (key_attnos)
+ if (query->distinctClause)
{
+ /* create unique constraint on all columns */
foreach(lc, query->targetList)
{
TargetEntry *tle = (TargetEntry *) lfirst(lc);
Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, tle->resno - 1);
-
- if (bms_is_member(tle->resno - FirstLowInvalidHeapAttributeNumber, key_attnos))
- {
- IndexElem *iparam;
-
- iparam = makeNode(IndexElem);
- iparam->name = pstrdup(NameStr(attr->attname));
- iparam->expr = NULL;
- iparam->indexcolname = NULL;
- iparam->collation = NIL;
- iparam->opclass = NIL;
- iparam->opclassopts = NIL;
- iparam->ordering = SORTBY_DEFAULT;
- iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
- index->indexParams = lappend(index->indexParams, iparam);
- }
+ IndexElem *iparam;
+
+ iparam = makeNode(IndexElem);
+ iparam->name = pstrdup(NameStr(attr->attname));
+ iparam->expr = NULL;
+ iparam->indexcolname = NULL;
+ iparam->collation = NIL;
+ iparam->opclass = NIL;
+ iparam->opclassopts = NIL;
+ iparam->ordering = SORTBY_DEFAULT;
+ iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
+ index->indexParams = lappend(index->indexParams, iparam);
}
}
else
{
- /* create no index, just notice that an appropriate index is necessary for efficient IVM */
- ereport(NOTICE,
- (errmsg("could not create an index on materialized view \"%s\" automatically",
- RelationGetRelationName(matviewRel)),
- errdetail("This target list does not have all the primary key columns. "),
- errhint("Create an index on the materialized view for efficient incremental maintenance.")));
- return;
+ Bitmapset *key_attnos;
+
+ /* create index on the base tables' primary key columns */
+ key_attnos = get_primary_key_attnos_from_query(query, &constraintList);
+ if (key_attnos)
+ {
+ foreach(lc, query->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);
+ Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, tle->resno - 1);
+
+ if (bms_is_member(tle->resno - FirstLowInvalidHeapAttributeNumber, key_attnos))
+ {
+ IndexElem *iparam;
+
+ iparam = makeNode(IndexElem);
+ iparam->name = pstrdup(NameStr(attr->attname));
+ iparam->expr = NULL;
+ iparam->indexcolname = NULL;
+ iparam->collation = NIL;
+ iparam->opclass = NIL;
+ iparam->opclassopts = NIL;
+ iparam->ordering = SORTBY_DEFAULT;
+ iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
+ index->indexParams = lappend(index->indexParams, iparam);
+ }
+ }
+ }
+ else
+ {
+ /* create no index, just notice that an appropriate index is necessary for efficient IVM */
+ ereport(NOTICE,
+ (errmsg("could not create an index on materialized view \"%s\" automatically",
+ RelationGetRelationName(matviewRel)),
+ errdetail("This target list does not have all the primary key columns, "
+ "or this view does not contain DISTINCT clause."),
+ errhint("Create an index on the materialized view for efficient incremental maintenance.")));
+ return;
+ }
}
/* If we have a compatible index, we don't need to create another. */
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index d9016ef487..fb5265e6c3 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -41,6 +41,7 @@
#include "commands/dbcommands.h"
#include "commands/defrem.h"
#include "commands/event_trigger.h"
+#include "commands/matview.h"
#include "commands/progress.h"
#include "commands/tablecmds.h"
#include "commands/tablespace.h"
@@ -1132,6 +1133,45 @@ DefineIndex(Oid tableId,
safe_index = indexInfo->ii_Expressions == NIL &&
indexInfo->ii_Predicate == NIL;
+ /*
+ * We disallow unique indexes on IVM columns of IMMVs.
+ */
+ if (RelationIsIVM(rel) && stmt->unique)
+ {
+ for (int i = 0; i < indexInfo->ii_NumIndexKeyAttrs; i++)
+ {
+ AttrNumber attno = indexInfo->ii_IndexAttrNumbers[i];
+ if (attno > 0)
+ {
+ char *name = NameStr(TupleDescAttr(rel->rd_att, attno - 1)->attname);
+ if (name && isIvmName(name))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("unique index creation on IVM columns is not supported")));
+ }
+ }
+
+ if (indexInfo->ii_Expressions)
+ {
+ Bitmapset *indexattrs = NULL;
+ int varno = -1;
+
+ pull_varattnos((Node *) indexInfo->ii_Expressions, 1, &indexattrs);
+
+ while ((varno = bms_next_member(indexattrs, varno)) >= 0)
+ {
+ int attno = varno + FirstLowInvalidHeapAttributeNumber;
+ char *name = NameStr(TupleDescAttr(rel->rd_att, attno - 1)->attname);
+ if (name && isIvmName(name))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("unique index creation on IVM columns is not supported")));
+ }
+
+ }
+ }
+
+
/*
* Report index creation if appropriate (delay this till after most of the
* error checks)
diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c
index 78a5dd1df9..0064e10966 100644
--- a/src/backend/commands/matview.c
+++ b/src/backend/commands/matview.c
@@ -148,11 +148,15 @@ static Query *rewrite_query_for_postupdate_state(Query *query, MV_TriggerTable *
static void apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *new_tuplestores,
TupleDesc tupdesc_old, TupleDesc tupdesc_new,
- Query *query);
+ Query *query, bool use_count, char *count_colname);
static void apply_old_delta(const char *matviewname, const char *deltaname_old,
List *keys);
+static void apply_old_delta_with_count(const char *matviewname, const char *deltaname_old,
+ List *keys, const char *count_colname);
static void apply_new_delta(const char *matviewname, const char *deltaname_new,
StringInfo target_list);
+static void apply_new_delta_with_count(const char *matviewname, const char* deltaname_new,
+ List *keys, StringInfo target_list, const char* count_colname);
static char *get_matching_condition_string(List *keys);
static void generate_equal(StringInfo querybuf, Oid opttype,
const char *leftop, const char *rightop);
@@ -267,6 +271,7 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
Oid matviewOid;
Relation matviewRel;
Query *dataQuery;
+ Query *viewQuery;
Oid tableSpace;
Oid relowner;
Oid OIDNewHeap;
@@ -329,8 +334,13 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
"CONCURRENTLY", "WITH NO DATA")));
- dataQuery = get_matview_query(matviewRel);
+ viewQuery = get_matview_query(matviewRel);
+ /* For IMMV, we need to rewrite matview query */
+ if (!stmt->skipData && RelationIsIVM(matviewRel))
+ dataQuery = rewriteQueryForIMMV(viewQuery,NIL);
+ else
+ dataQuery = viewQuery;
/*
* Check that there is a unique index with no WHERE clause on one or more
@@ -510,8 +520,8 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
if (!stmt->skipData && RelationIsIVM(matviewRel) && !oldPopulated)
{
- CreateIndexOnIMMV(dataQuery, matviewRel);
- CreateIvmTriggersOnBaseTables(dataQuery, matviewOid);
+ CreateIndexOnIMMV(viewQuery, matviewRel);
+ CreateIvmTriggersOnBaseTables(viewQuery, matviewOid);
}
table_close(matviewRel, NoLock);
@@ -1535,6 +1545,13 @@ IVM_immediate_maintenance(PG_FUNCTION_ARGS)
int rte_index = lfirst_int(lc2);
TupleDesc tupdesc_old;
TupleDesc tupdesc_new;
+ bool use_count = false;
+ char *count_colname = NULL;
+
+ count_colname = pstrdup("__ivm_count__");
+
+ if (query->distinctClause)
+ use_count = true;
/* calculate delta tables */
calc_delta(table, rte_index, rewritten, dest_old, dest_new,
@@ -1547,7 +1564,8 @@ IVM_immediate_maintenance(PG_FUNCTION_ARGS)
{
/* apply the delta tables to the materialized view */
apply_delta(matviewOid, old_tuplestore, new_tuplestore,
- tupdesc_old, tupdesc_new, query);
+ tupdesc_old, tupdesc_new, query, use_count,
+ count_colname);
}
PG_CATCH();
{
@@ -2020,7 +2038,7 @@ rewrite_query_for_postupdate_state(Query *query, MV_TriggerTable *table, int rte
static void
apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *new_tuplestores,
TupleDesc tupdesc_old, TupleDesc tupdesc_new,
- Query *query)
+ Query *query, bool use_count, char *count_colname)
{
StringInfoData querybuf;
StringInfoData target_list_buf;
@@ -2096,7 +2114,12 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
if (rc != SPI_OK_REL_REGISTER)
elog(ERROR, "SPI_register failed");
- apply_old_delta(matviewname, OLD_DELTA_ENRNAME, keys);
+ if (use_count)
+ /* apply old delta and get rows to be recalculated */
+ apply_old_delta_with_count(matviewname, OLD_DELTA_ENRNAME,
+ keys, count_colname);
+ else
+ apply_old_delta(matviewname, OLD_DELTA_ENRNAME, keys);
}
/* For tuple insertion */
@@ -2118,7 +2141,11 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
elog(ERROR, "SPI_register failed");
/* apply new delta */
- apply_new_delta(matviewname, NEW_DELTA_ENRNAME, &target_list_buf);
+ if (use_count)
+ apply_new_delta_with_count(matviewname, NEW_DELTA_ENRNAME,
+ keys, &target_list_buf, count_colname);
+ else
+ apply_new_delta(matviewname, NEW_DELTA_ENRNAME, &target_list_buf);
}
/* We're done maintaining the materialized view. */
@@ -2131,6 +2158,51 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
elog(ERROR, "SPI_finish failed");
}
+/*
+ * apply_old_delta_with_count
+ *
+ * Execute a query for applying a delta table given by deltname_old
+ * which contains tuples to be deleted from to a materialized view given by
+ * matviewname. This is used when counting is required, that is, the view
+ * has aggregate or distinct.
+ */
+static void
+apply_old_delta_with_count(const char *matviewname, const char *deltaname_old,
+ List *keys, const char *count_colname)
+{
+ StringInfoData querybuf;
+ char *match_cond;
+
+ /* build WHERE condition for searching tuples to be deleted */
+ match_cond = get_matching_condition_string(keys);
+
+ /* Search for matching tuples from the view and update or delete if found. */
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH t AS (" /* collecting tid of target tuples in the view */
+ "SELECT diff.%s, " /* count column */
+ "(diff.%s OPERATOR(pg_catalog.=) mv.%s) AS for_dlt, "
+ "mv.ctid "
+ "FROM %s AS mv, %s AS diff "
+ "WHERE %s" /* tuple matching condition */
+ "), updt AS (" /* update a tuple if this is not to be deleted */
+ "UPDATE %s AS mv SET %s = mv.%s OPERATOR(pg_catalog.-) t.%s "
+ "FROM t WHERE mv.ctid OPERATOR(pg_catalog.=) t.ctid AND NOT for_dlt "
+ ")"
+ /* delete a tuple if this is to be deleted */
+ "DELETE FROM %s AS mv USING t "
+ "WHERE mv.ctid OPERATOR(pg_catalog.=) t.ctid AND for_dlt",
+ count_colname,
+ count_colname, count_colname,
+ matviewname, deltaname_old,
+ match_cond,
+ matviewname, count_colname, count_colname, count_colname,
+ matviewname);
+
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_DELETE)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+}
+
/*
* apply_old_delta
*
@@ -2180,6 +2252,66 @@ apply_old_delta(const char *matviewname, const char *deltaname_old,
elog(ERROR, "SPI_exec failed: %s", querybuf.data);
}
+/*
+ * apply_new_delta_with_count
+ *
+ * Execute a query for applying a delta table given by deltname_new
+ * which contains tuples to be inserted into a materialized view given by
+ * matviewname. This is used when counting is required, that is, the view
+ * has aggregate or distinct. Also, when a table in EXISTS sub queries
+ * is modified.
+ */
+static void
+apply_new_delta_with_count(const char *matviewname, const char* deltaname_new,
+ List *keys, StringInfo target_list, const char* count_colname)
+{
+ StringInfoData querybuf;
+ StringInfoData returning_keys;
+ ListCell *lc;
+ char *match_cond = "";
+
+ /* build WHERE condition for searching tuples to be updated */
+ match_cond = get_matching_condition_string(keys);
+
+ /* build string of keys list */
+ initStringInfo(&returning_keys);
+ if (keys)
+ {
+ foreach (lc, keys)
+ {
+ Form_pg_attribute attr = (Form_pg_attribute) lfirst(lc);
+ char *resname = NameStr(attr->attname);
+ appendStringInfo(&returning_keys, "%s", quote_qualified_identifier("mv", resname));
+ if (lnext(keys, lc))
+ appendStringInfo(&returning_keys, ", ");
+ }
+ }
+ else
+ appendStringInfo(&returning_keys, "NULL");
+
+ /* Search for matching tuples from the view and update if found or insert if not. */
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH updt AS (" /* update a tuple if this exists in the view */
+ "UPDATE %s AS mv SET %s = mv.%s OPERATOR(pg_catalog.+) diff.%s "
+ "FROM %s AS diff "
+ "WHERE %s " /* tuple matching condition */
+ "RETURNING %s" /* returning keys of updated tuples */
+ ") INSERT INTO %s (%s) " /* insert a new tuple if this doesn't exist */
+ "SELECT %s FROM %s AS diff "
+ "WHERE NOT EXISTS (SELECT 1 FROM updt AS mv WHERE %s);",
+ matviewname, count_colname, count_colname, count_colname,
+ deltaname_new,
+ match_cond,
+ returning_keys.data,
+ matviewname, target_list->data,
+ target_list->data, deltaname_new,
+ match_cond);
+
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+}
+
/*
* apply_new_delta
*
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index a28f405e27..12b5b5df64 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -56,6 +56,7 @@
#include "commands/cluster.h"
#include "commands/comment.h"
#include "commands/defrem.h"
+#include "commands/matview.h"
#include "commands/event_trigger.h"
#include "commands/sequence.h"
#include "commands/tablecmds.h"
@@ -3797,6 +3798,14 @@ renameatt_internal(Oid myrelid,
targetrelation = relation_open(myrelid, AccessExclusiveLock);
renameatt_check(myrelid, RelationGetForm(targetrelation), recursing);
+ /*
+ * Don't rename IVM columns.
+ */
+ if (RelationIsIVM(targetrelation) && isIvmName(oldattname))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("IVM column can not be renamed")));
+
/*
* if the 'recurse' flag is set then we are supposed to rename this
* attribute in all classes that inherit from 'relname' (as well as in
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 3337b77ae6..c191f70a6f 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -510,6 +510,7 @@ _outRangeTblEntry(StringInfo str, const RangeTblEntry *node)
WRITE_INT_FIELD(rellockmode);
WRITE_UINT_FIELD(perminfoindex);
WRITE_NODE_FIELD(tablesample);
+ WRITE_BOOL_FIELD(relisivm);
break;
case RTE_SUBQUERY:
WRITE_NODE_FIELD(subquery);
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index c4d01a441a..ffcab8cda2 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -361,6 +361,7 @@ _readRangeTblEntry(void)
READ_INT_FIELD(rellockmode);
READ_UINT_FIELD(perminfoindex);
READ_NODE_FIELD(tablesample);
+ READ_BOOL_FIELD(relisivm);
break;
case RTE_SUBQUERY:
READ_NODE_FIELD(subquery);
diff --git a/src/backend/parser/parse_relation.c b/src/backend/parser/parse_relation.c
index 427b7325db..65aecc96a7 100644
--- a/src/backend/parser/parse_relation.c
+++ b/src/backend/parser/parse_relation.c
@@ -36,6 +36,7 @@
#include "utils/rel.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
+#include "commands/matview.h"
/*
@@ -97,7 +98,7 @@ static void expandTupleDesc(TupleDesc tupdesc, Alias *eref,
int count, int offset,
int rtindex, int sublevels_up,
int location, bool include_dropped,
- List **colnames, List **colvars);
+ List **colnames, List **colvars, bool is_ivm);
static int specialAttNum(const char *attname);
static bool rte_visible_if_lateral(ParseState *pstate, RangeTblEntry *rte);
static bool rte_visible_if_qualified(ParseState *pstate, RangeTblEntry *rte);
@@ -1503,6 +1504,7 @@ addRangeTableEntry(ParseState *pstate,
rte->inh = inh;
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -1588,6 +1590,7 @@ addRangeTableEntryForRelation(ParseState *pstate,
rte->inh = inh;
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -2752,7 +2755,7 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
expandTupleDesc(tupdesc, rte->eref,
rtfunc->funccolcount, atts_done,
rtindex, sublevels_up, location,
- include_dropped, colnames, colvars);
+ include_dropped, colnames, colvars, false);
}
else if (functypclass == TYPEFUNC_SCALAR)
{
@@ -3020,7 +3023,7 @@ expandRelation(Oid relid, Alias *eref, int rtindex, int sublevels_up,
expandTupleDesc(rel->rd_att, eref, rel->rd_att->natts, 0,
rtindex, sublevels_up,
location, include_dropped,
- colnames, colvars);
+ colnames, colvars, RelationIsIVM(rel));
relation_close(rel, AccessShareLock);
}
@@ -3037,7 +3040,7 @@ static void
expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
int rtindex, int sublevels_up,
int location, bool include_dropped,
- List **colnames, List **colvars)
+ List **colnames, List **colvars, bool is_ivm)
{
ListCell *aliascell;
int varattno;
@@ -3050,6 +3053,9 @@ expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
{
Form_pg_attribute attr = TupleDescAttr(tupdesc, varattno);
+ if (is_ivm && isIvmName(NameStr(attr->attname)) && !MatViewIncrementalMaintenanceIsEnabled())
+ continue;
+
if (attr->attisdropped)
{
if (include_dropped)
@@ -3212,6 +3218,10 @@ expandNSItemAttrs(ParseState *pstate, ParseNamespaceItem *nsitem,
Var *varnode = (Var *) lfirst(var);
TargetEntry *te;
+ /* if transform * into columnlist with IMMV, remove IVM columns */
+ if (rte->relisivm && isIvmName(label) && !MatViewIncrementalMaintenanceIsEnabled())
+ continue;
+
te = makeTargetEntry((Expr *) varnode,
(AttrNumber) pstate->p_next_resno++,
label,
diff --git a/src/backend/rewrite/rewriteDefine.c b/src/backend/rewrite/rewriteDefine.c
index 6cc9a8d8bf..5d22dbcfcf 100644
--- a/src/backend/rewrite/rewriteDefine.c
+++ b/src/backend/rewrite/rewriteDefine.c
@@ -614,7 +614,8 @@ checkRuleResultList(List *targetList, TupleDesc resultDesc, bool isSelect,
attr->atttypmod))));
}
- if (i != resultDesc->natts)
+ /* No check for materialized views since this could have special columns for IVM */
+ if ((!isSelect || requireColumnNameMatch) && i != resultDesc->natts)
ereport(ERROR,
(errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
isSelect ?
diff --git a/src/include/commands/createas.h b/src/include/commands/createas.h
index 396ad1bb4c..6b47e66bfd 100644
--- a/src/include/commands/createas.h
+++ b/src/include/commands/createas.h
@@ -29,6 +29,8 @@ extern ObjectAddress ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *st
extern void CreateIvmTriggersOnBaseTables(Query *qry, Oid matviewOid);
extern void CreateIndexOnIMMV(Query *query, Relation matviewRel);
+extern Query *rewriteQueryForIMMV(Query *query, List *colNames);
+
extern int GetIntoRelEFlags(IntoClause *intoClause);
extern DestReceiver *CreateIntoRelDestReceiver(IntoClause *intoClause);
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index a690ebc6e5..2e96bce175 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -1090,6 +1090,8 @@ typedef struct RangeTblEntry
Index perminfoindex pg_node_attr(query_jumble_ignore);
/* sampling info, or NULL */
struct TableSampleClause *tablesample;
+ /* incrementally maintainable materialized view? */
+ bool relisivm;
/*
* Fields valid for a subquery RTE (else NULL):
--
2.25.1
--Multipart=_Sun__31_Mar_2024_22_59_31_+0900_msknEviJj08_wgqO
Content-Type: text/x-diff;
name="v32-0008-Add-aggregates-support-in-IVM.patch"
Content-Disposition: attachment;
filename="v32-0008-Add-aggregates-support-in-IVM.patch"
Content-Transfer-Encoding: quoted-printable
^ permalink raw reply [nested|flat] 24+ messages in thread
* [PATCH v37 07/11] Add DISTINCT support for IVM
@ 2023-05-31 10:08 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 24+ messages in thread
From: Yugo Nagata @ 2023-05-31 10:08 UTC (permalink / raw)
When IMMV is created with DISTINCT, multiplicity of tuples is
counted and stored in "__ivm_count__" column, which is a hidden
column of IMMV. The value in __ivm_count__ is updated when IMMV
is maintained incrementally. A tuple in IMMV can be removed if
and only if the count becomes zero.
---
src/backend/commands/createas.c | 141 +++++++++++++++++++++------
src/backend/commands/indexcmds.c | 40 ++++++++
src/backend/commands/matview.c | 145 ++++++++++++++++++++++++++--
src/backend/commands/tablecmds.c | 9 ++
src/backend/nodes/outfuncs.c | 1 +
src/backend/nodes/readfuncs.c | 1 +
src/backend/parser/parse_relation.c | 18 +++-
src/backend/rewrite/rewriteDefine.c | 3 +-
src/include/commands/createas.h | 2 +
src/include/nodes/parsenodes.h | 2 +
10 files changed, 319 insertions(+), 43 deletions(-)
diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c
index a499688b79d..cd8db0059f9 100644
--- a/src/backend/commands/createas.c
+++ b/src/backend/commands/createas.c
@@ -55,6 +55,7 @@
#include "parser/parser.h"
#include "parser/parsetree.h"
#include "parser/parse_clause.h"
+#include "parser/parse_func.h"
#include "rewrite/rewriteHandler.h"
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
@@ -307,6 +308,9 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt,
errhint("functions must be marked IMMUTABLE")));
check_ivm_restriction((Node *) query);
+
+ /* For IMMV, we need to rewrite matview query */
+ query = rewriteQueryForIMMV(query, into->colNames);
}
if (into->skipData)
@@ -416,6 +420,49 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt,
return address;
}
+/*
+ * rewriteQueryForIMMV -- rewrite view definition query for IMMV
+ *
+ * count(*) is added for counting distinct tuples in views.
+ */
+Query *
+rewriteQueryForIMMV(Query *query, List *colNames)
+{
+ Query *rewritten;
+
+ Node *node;
+ ParseState *pstate = make_parsestate(NULL);
+ FuncCall *fn;
+
+ rewritten = copyObject(query);
+ pstate->p_expr_kind = EXPR_KIND_SELECT_TARGET;
+
+ /*
+ * Convert DISTINCT to GROUP BY and add count(*) for counting distinct
+ * tuples in views.
+ */
+ if (rewritten->distinctClause)
+ {
+ TargetEntry *tle;
+
+ rewritten->groupClause = transformDistinctClause(NULL, &rewritten->targetList, rewritten->sortClause, false);
+
+ fn = makeFuncCall(SystemFuncName("count"), NIL, COERCE_EXPLICIT_CALL, -1);
+ fn->agg_star = true;
+
+ node = ParseFuncOrColumn(pstate, fn->funcname, NIL, NULL, fn, false, -1);
+
+ tle = makeTargetEntry((Expr *) node,
+ list_length(rewritten->targetList) + 1,
+ pstrdup("__ivm_count__"),
+ false);
+ rewritten->targetList = lappend(rewritten->targetList, tle);
+ rewritten->hasAggs = true;
+ }
+
+ return rewritten;
+}
+
/*
* GetIntoRelEFlags --- compute executor flags needed for CREATE TABLE AS
*
@@ -539,7 +586,8 @@ intorel_startup(DestReceiver *self, int operation, TupleDesc typeinfo)
ColumnDef *col;
char *colname;
- if (lc)
+ /* Don't override hidden columns added for IVM */
+ if (lc && !isIvmName(NameStr(attribute->attname)))
{
colname = strVal(lfirst(lc));
lc = lnext(into->colNames, lc);
@@ -943,10 +991,6 @@ check_ivm_restriction_walker(Node *node, void *context)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("LIMIT/OFFSET clause is not supported on incrementally maintainable materialized view")));
- if (qry->distinctClause)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("DISTINCT is not supported on incrementally maintainable materialized view")));
if (qry->hasDistinctOn)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -1108,12 +1152,18 @@ CreateIndexOnIMMV(Query *query, Relation matviewRel)
char idxname[NAMEDATALEN];
List *indexoidlist = RelationGetIndexList(matviewRel);
ListCell *indexoidscan;
- Bitmapset *key_attnos;
snprintf(idxname, sizeof(idxname), "%s_index", RelationGetRelationName(matviewRel));
index = makeNode(IndexStmt);
+ /*
+ * We consider null values not distinct to make sure that views with DISTINCT
+ * or GROUP BY don't contain multiple NULL rows when NULL is inserted to
+ * a base table concurrently.
+ */
+ index->nulls_not_distinct = true;
+
index->unique = true;
index->primary = false;
index->isconstraint = false;
@@ -1140,41 +1190,68 @@ CreateIndexOnIMMV(Query *query, Relation matviewRel)
index->concurrent = false;
index->if_not_exists = false;
- /* create index on the base tables' primary key columns */
- key_attnos = get_primary_key_attnos_from_query(query, &constraintList);
- if (key_attnos)
+ if (query->distinctClause)
{
+ /* create unique constraint on all columns */
foreach(lc, query->targetList)
{
TargetEntry *tle = (TargetEntry *) lfirst(lc);
Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, tle->resno - 1);
-
- if (bms_is_member(tle->resno - FirstLowInvalidHeapAttributeNumber, key_attnos))
- {
- IndexElem *iparam;
-
- iparam = makeNode(IndexElem);
- iparam->name = pstrdup(NameStr(attr->attname));
- iparam->expr = NULL;
- iparam->indexcolname = NULL;
- iparam->collation = NIL;
- iparam->opclass = NIL;
- iparam->opclassopts = NIL;
- iparam->ordering = SORTBY_DEFAULT;
- iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
- index->indexParams = lappend(index->indexParams, iparam);
- }
+ IndexElem *iparam;
+
+ iparam = makeNode(IndexElem);
+ iparam->name = pstrdup(NameStr(attr->attname));
+ iparam->expr = NULL;
+ iparam->indexcolname = NULL;
+ iparam->collation = NIL;
+ iparam->opclass = NIL;
+ iparam->opclassopts = NIL;
+ iparam->ordering = SORTBY_DEFAULT;
+ iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
+ index->indexParams = lappend(index->indexParams, iparam);
}
}
else
{
- /* create no index, just notice that an appropriate index is necessary for efficient IVM */
- ereport(NOTICE,
- (errmsg("could not create an index on materialized view \"%s\" automatically",
- RelationGetRelationName(matviewRel)),
- errdetail("This target list does not have all the primary key columns. "),
- errhint("Create an index on the materialized view for efficient incremental maintenance.")));
- return;
+ Bitmapset *key_attnos;
+
+ /* create index on the base tables' primary key columns */
+ key_attnos = get_primary_key_attnos_from_query(query, &constraintList);
+ if (key_attnos)
+ {
+ foreach(lc, query->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);
+ Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, tle->resno - 1);
+
+ if (bms_is_member(tle->resno - FirstLowInvalidHeapAttributeNumber, key_attnos))
+ {
+ IndexElem *iparam;
+
+ iparam = makeNode(IndexElem);
+ iparam->name = pstrdup(NameStr(attr->attname));
+ iparam->expr = NULL;
+ iparam->indexcolname = NULL;
+ iparam->collation = NIL;
+ iparam->opclass = NIL;
+ iparam->opclassopts = NIL;
+ iparam->ordering = SORTBY_DEFAULT;
+ iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
+ index->indexParams = lappend(index->indexParams, iparam);
+ }
+ }
+ }
+ else
+ {
+ /* create no index, just notice that an appropriate index is necessary for efficient IVM */
+ ereport(NOTICE,
+ (errmsg("could not create an index on materialized view \"%s\" automatically",
+ RelationGetRelationName(matviewRel)),
+ errdetail("This target list does not have all the primary key columns, "
+ "or this view does not contain DISTINCT clause."),
+ errhint("Create an index on the materialized view for efficient incremental maintenance.")));
+ return;
+ }
}
/* If we have a compatible index, we don't need to create another. */
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 9ab74c8df0a..1f95e6b229f 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -41,6 +41,7 @@
#include "commands/comment.h"
#include "commands/defrem.h"
#include "commands/event_trigger.h"
+#include "commands/matview.h"
#include "commands/progress.h"
#include "commands/tablecmds.h"
#include "commands/tablespace.h"
@@ -1179,6 +1180,45 @@ DefineIndex(ParseState *pstate,
safe_index = indexInfo->ii_Expressions == NIL &&
indexInfo->ii_Predicate == NIL;
+ /*
+ * We disallow unique indexes on IVM columns of IMMVs.
+ */
+ if (RelationIsIVM(rel) && stmt->unique)
+ {
+ for (int i = 0; i < indexInfo->ii_NumIndexKeyAttrs; i++)
+ {
+ AttrNumber attno = indexInfo->ii_IndexAttrNumbers[i];
+ if (attno > 0)
+ {
+ char *name = NameStr(TupleDescAttr(rel->rd_att, attno - 1)->attname);
+ if (name && isIvmName(name))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("unique index creation on IVM columns is not supported")));
+ }
+ }
+
+ if (indexInfo->ii_Expressions)
+ {
+ Bitmapset *indexattrs = NULL;
+ int varno = -1;
+
+ pull_varattnos((Node *) indexInfo->ii_Expressions, 1, &indexattrs);
+
+ while ((varno = bms_next_member(indexattrs, varno)) >= 0)
+ {
+ int attno = varno + FirstLowInvalidHeapAttributeNumber;
+ char *name = NameStr(TupleDescAttr(rel->rd_att, attno - 1)->attname);
+ if (name && isIvmName(name))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("unique index creation on IVM columns is not supported")));
+ }
+
+ }
+ }
+
+
/*
* Report index creation if appropriate (delay this till after most of the
* error checks)
diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c
index eaf39f80cd3..a2746ca9265 100644
--- a/src/backend/commands/matview.c
+++ b/src/backend/commands/matview.c
@@ -193,11 +193,15 @@ static Query *rewrite_query_for_postupdate_state(Query *query, MV_TriggerTable *
static void apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *new_tuplestores,
TupleDesc tupdesc_old, TupleDesc tupdesc_new,
- Query *query);
+ Query *query, bool use_count, char *count_colname);
static void apply_old_delta(const char *matviewname, const char *deltaname_old,
List *keys);
+static void apply_old_delta_with_count(const char *matviewname, const char *deltaname_old,
+ List *keys, const char *count_colname);
static void apply_new_delta(const char *matviewname, const char *deltaname_new,
StringInfo target_list);
+static void apply_new_delta_with_count(const char *matviewname, const char* deltaname_new,
+ List *keys, StringInfo target_list, const char* count_colname);
static char *get_matching_condition_string(List *keys);
static void generate_equal(StringInfo querybuf, Oid opttype,
const char *leftop, const char *rightop);
@@ -346,6 +350,7 @@ RefreshMatViewByOid(Oid matviewOid, bool is_create, bool skipData,
QueryCompletion *qc)
{
Relation matviewRel;
+ Query *viewQuery;
Query *dataQuery;
Oid tableSpace;
Oid relowner;
@@ -394,7 +399,13 @@ RefreshMatViewByOid(Oid matviewOid, bool is_create, bool skipData,
errmsg("%s options %s and %s cannot be used together",
"REFRESH", "CONCURRENTLY", "WITH NO DATA")));
- dataQuery = get_matview_query(matviewRel);
+ viewQuery = get_matview_query(matviewRel);
+
+ /* For IMMV, we need to rewrite matview query */
+ if (!skipData && RelationIsIVM(matviewRel))
+ dataQuery = rewriteQueryForIMMV(viewQuery,NIL);
+ else
+ dataQuery = viewQuery;
/*
* Check that there is a unique index with no WHERE clause on one or more
@@ -1688,6 +1699,13 @@ IVM_immediate_maintenance(PG_FUNCTION_ARGS)
int rte_index = lfirst_int(lc2);
TupleDesc tupdesc_old;
TupleDesc tupdesc_new;
+ bool use_count = false;
+ char *count_colname = NULL;
+
+ count_colname = pstrdup("__ivm_count__");
+
+ if (query->distinctClause)
+ use_count = true;
/* calculate delta tables */
calc_delta(table, rte_index, rewritten, dest_old, dest_new,
@@ -1700,7 +1718,8 @@ IVM_immediate_maintenance(PG_FUNCTION_ARGS)
{
/* apply the delta tables to the materialized view */
apply_delta(matviewOid, old_tuplestore, new_tuplestore,
- tupdesc_old, tupdesc_new, query);
+ tupdesc_old, tupdesc_new, query, use_count,
+ count_colname);
}
PG_CATCH();
{
@@ -2238,7 +2257,7 @@ rewrite_query_for_postupdate_state(Query *query, MV_TriggerTable *table, int rte
static void
apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *new_tuplestores,
TupleDesc tupdesc_old, TupleDesc tupdesc_new,
- Query *query)
+ Query *query, bool use_count, char *count_colname)
{
StringInfoData querybuf;
StringInfoData target_list_buf;
@@ -2314,7 +2333,12 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
if (rc != SPI_OK_REL_REGISTER)
elog(ERROR, "SPI_register failed");
- apply_old_delta(matviewname, OLD_DELTA_ENRNAME, keys);
+ if (use_count)
+ /* apply old delta and get rows to be recalculated */
+ apply_old_delta_with_count(matviewname, OLD_DELTA_ENRNAME,
+ keys, count_colname);
+ else
+ apply_old_delta(matviewname, OLD_DELTA_ENRNAME, keys);
}
/* For tuple insertion */
@@ -2336,7 +2360,11 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
elog(ERROR, "SPI_register failed");
/* apply new delta */
- apply_new_delta(matviewname, NEW_DELTA_ENRNAME, &target_list_buf);
+ if (use_count)
+ apply_new_delta_with_count(matviewname, NEW_DELTA_ENRNAME,
+ keys, &target_list_buf, count_colname);
+ else
+ apply_new_delta(matviewname, NEW_DELTA_ENRNAME, &target_list_buf);
}
/* We're done maintaining the materialized view. */
@@ -2349,6 +2377,51 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
elog(ERROR, "SPI_finish failed");
}
+/*
+ * apply_old_delta_with_count
+ *
+ * Execute a query for applying a delta table given by deltname_old
+ * which contains tuples to be deleted from to a materialized view given by
+ * matviewname. This is used when counting is required, that is, the view
+ * has aggregate or distinct.
+ */
+static void
+apply_old_delta_with_count(const char *matviewname, const char *deltaname_old,
+ List *keys, const char *count_colname)
+{
+ StringInfoData querybuf;
+ char *match_cond;
+
+ /* build WHERE condition for searching tuples to be deleted */
+ match_cond = get_matching_condition_string(keys);
+
+ /* Search for matching tuples from the view and update or delete if found. */
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH t AS (" /* collecting tid of target tuples in the view */
+ "SELECT diff.%s, " /* count column */
+ "(diff.%s OPERATOR(pg_catalog.=) mv.%s) AS for_dlt, "
+ "mv.ctid "
+ "FROM %s AS mv, %s AS diff "
+ "WHERE %s" /* tuple matching condition */
+ "), updt AS (" /* update a tuple if this is not to be deleted */
+ "UPDATE %s AS mv SET %s = mv.%s OPERATOR(pg_catalog.-) t.%s "
+ "FROM t WHERE mv.ctid OPERATOR(pg_catalog.=) t.ctid AND NOT for_dlt "
+ ")"
+ /* delete a tuple if this is to be deleted */
+ "DELETE FROM %s AS mv USING t "
+ "WHERE mv.ctid OPERATOR(pg_catalog.=) t.ctid AND for_dlt",
+ count_colname,
+ count_colname, count_colname,
+ matviewname, deltaname_old,
+ match_cond,
+ matviewname, count_colname, count_colname, count_colname,
+ matviewname);
+
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_DELETE)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+}
+
/*
* apply_old_delta
*
@@ -2398,6 +2471,66 @@ apply_old_delta(const char *matviewname, const char *deltaname_old,
elog(ERROR, "SPI_exec failed: %s", querybuf.data);
}
+/*
+ * apply_new_delta_with_count
+ *
+ * Execute a query for applying a delta table given by deltname_new
+ * which contains tuples to be inserted into a materialized view given by
+ * matviewname. This is used when counting is required, that is, the view
+ * has aggregate or distinct. Also, when a table in EXISTS sub queries
+ * is modified.
+ */
+static void
+apply_new_delta_with_count(const char *matviewname, const char* deltaname_new,
+ List *keys, StringInfo target_list, const char* count_colname)
+{
+ StringInfoData querybuf;
+ StringInfoData returning_keys;
+ ListCell *lc;
+ char *match_cond = "";
+
+ /* build WHERE condition for searching tuples to be updated */
+ match_cond = get_matching_condition_string(keys);
+
+ /* build string of keys list */
+ initStringInfo(&returning_keys);
+ if (keys)
+ {
+ foreach (lc, keys)
+ {
+ Form_pg_attribute attr = (Form_pg_attribute) lfirst(lc);
+ char *resname = NameStr(attr->attname);
+ appendStringInfo(&returning_keys, "%s", quote_qualified_identifier("mv", resname));
+ if (lnext(keys, lc))
+ appendStringInfo(&returning_keys, ", ");
+ }
+ }
+ else
+ appendStringInfo(&returning_keys, "NULL");
+
+ /* Search for matching tuples from the view and update if found or insert if not. */
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH updt AS (" /* update a tuple if this exists in the view */
+ "UPDATE %s AS mv SET %s = mv.%s OPERATOR(pg_catalog.+) diff.%s "
+ "FROM %s AS diff "
+ "WHERE %s " /* tuple matching condition */
+ "RETURNING %s" /* returning keys of updated tuples */
+ ") INSERT INTO %s (%s) " /* insert a new tuple if this doesn't exist */
+ "SELECT %s FROM %s AS diff "
+ "WHERE NOT EXISTS (SELECT 1 FROM updt AS mv WHERE %s);",
+ matviewname, count_colname, count_colname, count_colname,
+ deltaname_new,
+ match_cond,
+ returning_keys.data,
+ matviewname, target_list->data,
+ target_list->data, deltaname_new,
+ match_cond);
+
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+}
+
/*
* apply_new_delta
*
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 99bc709959a..ee65292deb0 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -60,6 +60,7 @@
#include "catalog/toasting.h"
#include "commands/comment.h"
#include "commands/defrem.h"
+#include "commands/matview.h"
#include "commands/event_trigger.h"
#include "commands/extension.h"
#include "commands/matview.h"
@@ -3923,6 +3924,14 @@ renameatt_internal(Oid myrelid,
targetrelation = relation_open(myrelid, AccessExclusiveLock);
renameatt_check(myrelid, RelationGetForm(targetrelation), recursing);
+ /*
+ * Don't rename IVM columns.
+ */
+ if (RelationIsIVM(targetrelation) && isIvmName(oldattname))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("IVM column can not be renamed")));
+
/*
* if the 'recurse' flag is set then we are supposed to rename this
* attribute in all classes that inherit from 'relname' (as well as in
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 953c5797c5d..0f228eca259 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -516,6 +516,7 @@ _outRangeTblEntry(StringInfo str, const RangeTblEntry *node)
WRITE_INT_FIELD(rellockmode);
WRITE_UINT_FIELD(perminfoindex);
WRITE_NODE_FIELD(tablesample);
+ WRITE_BOOL_FIELD(relisivm);
break;
case RTE_SUBQUERY:
WRITE_NODE_FIELD(subquery);
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index b6b2ce6c792..dba4d8ff2de 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -365,6 +365,7 @@ _readRangeTblEntry(void)
READ_INT_FIELD(rellockmode);
READ_UINT_FIELD(perminfoindex);
READ_NODE_FIELD(tablesample);
+ READ_BOOL_FIELD(relisivm);
break;
case RTE_SUBQUERY:
READ_NODE_FIELD(subquery);
diff --git a/src/backend/parser/parse_relation.c b/src/backend/parser/parse_relation.c
index 43460e4a5a5..dc048c21bed 100644
--- a/src/backend/parser/parse_relation.c
+++ b/src/backend/parser/parse_relation.c
@@ -33,6 +33,7 @@
#include "utils/lsyscache.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
+#include "commands/matview.h"
/*
@@ -96,7 +97,7 @@ static void expandTupleDesc(TupleDesc tupdesc, Alias *eref,
int rtindex, int sublevels_up,
VarReturningType returning_type,
int location, bool include_dropped,
- List **colnames, List **colvars);
+ List **colnames, List **colvars, bool is_ivm);
static int specialAttNum(const char *attname);
static bool rte_visible_if_lateral(ParseState *pstate, RangeTblEntry *rte);
static bool rte_visible_if_qualified(ParseState *pstate, RangeTblEntry *rte);
@@ -1512,6 +1513,7 @@ addRangeTableEntry(ParseState *pstate,
rte->inh = inh;
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -1593,6 +1595,7 @@ addRangeTableEntryForRelation(ParseState *pstate,
rte->inh = inh;
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -2935,7 +2938,7 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
rtfunc->funccolcount, atts_done,
rtindex, sublevels_up,
returning_type, location,
- include_dropped, colnames, colvars);
+ include_dropped, colnames, colvars, false);
}
else if (functypclass == TYPEFUNC_SCALAR)
{
@@ -3211,7 +3214,7 @@ expandRelation(Oid relid, Alias *eref, int rtindex, int sublevels_up,
expandTupleDesc(rel->rd_att, eref, rel->rd_att->natts, 0,
rtindex, sublevels_up, returning_type,
location, include_dropped,
- colnames, colvars);
+ colnames, colvars, RelationIsIVM(rel));
relation_close(rel, AccessShareLock);
}
@@ -3229,7 +3232,7 @@ expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
int rtindex, int sublevels_up,
VarReturningType returning_type,
int location, bool include_dropped,
- List **colnames, List **colvars)
+ List **colnames, List **colvars, bool is_ivm)
{
ListCell *aliascell;
int varattno;
@@ -3242,6 +3245,9 @@ expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
{
Form_pg_attribute attr = TupleDescAttr(tupdesc, varattno);
+ if (is_ivm && isIvmName(NameStr(attr->attname)) && !MatViewIncrementalMaintenanceIsEnabled())
+ continue;
+
if (attr->attisdropped)
{
if (include_dropped)
@@ -3406,6 +3412,10 @@ expandNSItemAttrs(ParseState *pstate, ParseNamespaceItem *nsitem,
Var *varnode = (Var *) lfirst(var);
TargetEntry *te;
+ /* if transform * into columnlist with IMMV, remove IVM columns */
+ if (rte->relisivm && isIvmName(label) && !MatViewIncrementalMaintenanceIsEnabled())
+ continue;
+
te = makeTargetEntry((Expr *) varnode,
(AttrNumber) pstate->p_next_resno++,
label,
diff --git a/src/backend/rewrite/rewriteDefine.c b/src/backend/rewrite/rewriteDefine.c
index 6a223fbeaa4..74ac631299c 100644
--- a/src/backend/rewrite/rewriteDefine.c
+++ b/src/backend/rewrite/rewriteDefine.c
@@ -614,7 +614,8 @@ checkRuleResultList(List *targetList, TupleDesc resultDesc, bool isSelect,
attr->atttypmod))));
}
- if (i != resultDesc->natts)
+ /* No check for materialized views since this could have special columns for IVM */
+ if ((!isSelect || requireColumnNameMatch) && i != resultDesc->natts)
ereport(ERROR,
(errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
isSelect ?
diff --git a/src/include/commands/createas.h b/src/include/commands/createas.h
index c286ebcd70e..bfd0249b10d 100644
--- a/src/include/commands/createas.h
+++ b/src/include/commands/createas.h
@@ -29,6 +29,8 @@ extern ObjectAddress ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *st
extern void CreateIvmTriggersOnBaseTables(Query *qry, Oid matviewOid);
extern void CreateIndexOnIMMV(Query *query, Relation matviewRel);
+extern Query *rewriteQueryForIMMV(Query *query, List *colNames);
+
extern int GetIntoRelEFlags(IntoClause *intoClause);
extern DestReceiver *CreateIntoRelDestReceiver(IntoClause *intoClause);
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 91377a6cde3..59e689a1fac 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -1207,6 +1207,8 @@ typedef struct RangeTblEntry
Index perminfoindex pg_node_attr(query_jumble_ignore);
/* sampling info, or NULL */
struct TableSampleClause *tablesample;
+ /* incrementally maintainable materialized view? */
+ bool relisivm;
/*
* Fields valid for a subquery RTE (else NULL):
--
2.43.0
--Multipart=_Fri__29_May_2026_23_14_17_+0900_Te0o73X2VqYK57Gd
Content-Type: text/x-diff;
name="v37-0006-Add-Incremental-View-Maintenance-support.patch"
Content-Disposition: attachment;
filename="v37-0006-Add-Incremental-View-Maintenance-support.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 24+ messages in thread
* [PATCH v38 07/11] Add DISTINCT support for IVM
@ 2023-05-31 10:08 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 24+ messages in thread
From: Yugo Nagata @ 2023-05-31 10:08 UTC (permalink / raw)
When IMMV is created with DISTINCT, multiplicity of tuples is
counted and stored in "__ivm_count__" column, which is a hidden
column of IMMV. The value in __ivm_count__ is updated when IMMV
is maintained incrementally. A tuple in IMMV can be removed if
and only if the count becomes zero.
---
src/backend/commands/createas.c | 141 +++++++++++++++++++++------
src/backend/commands/indexcmds.c | 40 ++++++++
src/backend/commands/matview.c | 145 ++++++++++++++++++++++++++--
src/backend/commands/tablecmds.c | 9 ++
src/backend/nodes/outfuncs.c | 1 +
src/backend/nodes/readfuncs.c | 1 +
src/backend/parser/parse_relation.c | 18 +++-
src/backend/rewrite/rewriteDefine.c | 3 +-
src/include/commands/createas.h | 2 +
src/include/nodes/parsenodes.h | 2 +
10 files changed, 319 insertions(+), 43 deletions(-)
diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c
index a499688b79d..cd8db0059f9 100644
--- a/src/backend/commands/createas.c
+++ b/src/backend/commands/createas.c
@@ -55,6 +55,7 @@
#include "parser/parser.h"
#include "parser/parsetree.h"
#include "parser/parse_clause.h"
+#include "parser/parse_func.h"
#include "rewrite/rewriteHandler.h"
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
@@ -307,6 +308,9 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt,
errhint("functions must be marked IMMUTABLE")));
check_ivm_restriction((Node *) query);
+
+ /* For IMMV, we need to rewrite matview query */
+ query = rewriteQueryForIMMV(query, into->colNames);
}
if (into->skipData)
@@ -416,6 +420,49 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt,
return address;
}
+/*
+ * rewriteQueryForIMMV -- rewrite view definition query for IMMV
+ *
+ * count(*) is added for counting distinct tuples in views.
+ */
+Query *
+rewriteQueryForIMMV(Query *query, List *colNames)
+{
+ Query *rewritten;
+
+ Node *node;
+ ParseState *pstate = make_parsestate(NULL);
+ FuncCall *fn;
+
+ rewritten = copyObject(query);
+ pstate->p_expr_kind = EXPR_KIND_SELECT_TARGET;
+
+ /*
+ * Convert DISTINCT to GROUP BY and add count(*) for counting distinct
+ * tuples in views.
+ */
+ if (rewritten->distinctClause)
+ {
+ TargetEntry *tle;
+
+ rewritten->groupClause = transformDistinctClause(NULL, &rewritten->targetList, rewritten->sortClause, false);
+
+ fn = makeFuncCall(SystemFuncName("count"), NIL, COERCE_EXPLICIT_CALL, -1);
+ fn->agg_star = true;
+
+ node = ParseFuncOrColumn(pstate, fn->funcname, NIL, NULL, fn, false, -1);
+
+ tle = makeTargetEntry((Expr *) node,
+ list_length(rewritten->targetList) + 1,
+ pstrdup("__ivm_count__"),
+ false);
+ rewritten->targetList = lappend(rewritten->targetList, tle);
+ rewritten->hasAggs = true;
+ }
+
+ return rewritten;
+}
+
/*
* GetIntoRelEFlags --- compute executor flags needed for CREATE TABLE AS
*
@@ -539,7 +586,8 @@ intorel_startup(DestReceiver *self, int operation, TupleDesc typeinfo)
ColumnDef *col;
char *colname;
- if (lc)
+ /* Don't override hidden columns added for IVM */
+ if (lc && !isIvmName(NameStr(attribute->attname)))
{
colname = strVal(lfirst(lc));
lc = lnext(into->colNames, lc);
@@ -943,10 +991,6 @@ check_ivm_restriction_walker(Node *node, void *context)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("LIMIT/OFFSET clause is not supported on incrementally maintainable materialized view")));
- if (qry->distinctClause)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("DISTINCT is not supported on incrementally maintainable materialized view")));
if (qry->hasDistinctOn)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -1108,12 +1152,18 @@ CreateIndexOnIMMV(Query *query, Relation matviewRel)
char idxname[NAMEDATALEN];
List *indexoidlist = RelationGetIndexList(matviewRel);
ListCell *indexoidscan;
- Bitmapset *key_attnos;
snprintf(idxname, sizeof(idxname), "%s_index", RelationGetRelationName(matviewRel));
index = makeNode(IndexStmt);
+ /*
+ * We consider null values not distinct to make sure that views with DISTINCT
+ * or GROUP BY don't contain multiple NULL rows when NULL is inserted to
+ * a base table concurrently.
+ */
+ index->nulls_not_distinct = true;
+
index->unique = true;
index->primary = false;
index->isconstraint = false;
@@ -1140,41 +1190,68 @@ CreateIndexOnIMMV(Query *query, Relation matviewRel)
index->concurrent = false;
index->if_not_exists = false;
- /* create index on the base tables' primary key columns */
- key_attnos = get_primary_key_attnos_from_query(query, &constraintList);
- if (key_attnos)
+ if (query->distinctClause)
{
+ /* create unique constraint on all columns */
foreach(lc, query->targetList)
{
TargetEntry *tle = (TargetEntry *) lfirst(lc);
Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, tle->resno - 1);
-
- if (bms_is_member(tle->resno - FirstLowInvalidHeapAttributeNumber, key_attnos))
- {
- IndexElem *iparam;
-
- iparam = makeNode(IndexElem);
- iparam->name = pstrdup(NameStr(attr->attname));
- iparam->expr = NULL;
- iparam->indexcolname = NULL;
- iparam->collation = NIL;
- iparam->opclass = NIL;
- iparam->opclassopts = NIL;
- iparam->ordering = SORTBY_DEFAULT;
- iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
- index->indexParams = lappend(index->indexParams, iparam);
- }
+ IndexElem *iparam;
+
+ iparam = makeNode(IndexElem);
+ iparam->name = pstrdup(NameStr(attr->attname));
+ iparam->expr = NULL;
+ iparam->indexcolname = NULL;
+ iparam->collation = NIL;
+ iparam->opclass = NIL;
+ iparam->opclassopts = NIL;
+ iparam->ordering = SORTBY_DEFAULT;
+ iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
+ index->indexParams = lappend(index->indexParams, iparam);
}
}
else
{
- /* create no index, just notice that an appropriate index is necessary for efficient IVM */
- ereport(NOTICE,
- (errmsg("could not create an index on materialized view \"%s\" automatically",
- RelationGetRelationName(matviewRel)),
- errdetail("This target list does not have all the primary key columns. "),
- errhint("Create an index on the materialized view for efficient incremental maintenance.")));
- return;
+ Bitmapset *key_attnos;
+
+ /* create index on the base tables' primary key columns */
+ key_attnos = get_primary_key_attnos_from_query(query, &constraintList);
+ if (key_attnos)
+ {
+ foreach(lc, query->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);
+ Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, tle->resno - 1);
+
+ if (bms_is_member(tle->resno - FirstLowInvalidHeapAttributeNumber, key_attnos))
+ {
+ IndexElem *iparam;
+
+ iparam = makeNode(IndexElem);
+ iparam->name = pstrdup(NameStr(attr->attname));
+ iparam->expr = NULL;
+ iparam->indexcolname = NULL;
+ iparam->collation = NIL;
+ iparam->opclass = NIL;
+ iparam->opclassopts = NIL;
+ iparam->ordering = SORTBY_DEFAULT;
+ iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
+ index->indexParams = lappend(index->indexParams, iparam);
+ }
+ }
+ }
+ else
+ {
+ /* create no index, just notice that an appropriate index is necessary for efficient IVM */
+ ereport(NOTICE,
+ (errmsg("could not create an index on materialized view \"%s\" automatically",
+ RelationGetRelationName(matviewRel)),
+ errdetail("This target list does not have all the primary key columns, "
+ "or this view does not contain DISTINCT clause."),
+ errhint("Create an index on the materialized view for efficient incremental maintenance.")));
+ return;
+ }
}
/* If we have a compatible index, we don't need to create another. */
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 9ab74c8df0a..1f95e6b229f 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -41,6 +41,7 @@
#include "commands/comment.h"
#include "commands/defrem.h"
#include "commands/event_trigger.h"
+#include "commands/matview.h"
#include "commands/progress.h"
#include "commands/tablecmds.h"
#include "commands/tablespace.h"
@@ -1179,6 +1180,45 @@ DefineIndex(ParseState *pstate,
safe_index = indexInfo->ii_Expressions == NIL &&
indexInfo->ii_Predicate == NIL;
+ /*
+ * We disallow unique indexes on IVM columns of IMMVs.
+ */
+ if (RelationIsIVM(rel) && stmt->unique)
+ {
+ for (int i = 0; i < indexInfo->ii_NumIndexKeyAttrs; i++)
+ {
+ AttrNumber attno = indexInfo->ii_IndexAttrNumbers[i];
+ if (attno > 0)
+ {
+ char *name = NameStr(TupleDescAttr(rel->rd_att, attno - 1)->attname);
+ if (name && isIvmName(name))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("unique index creation on IVM columns is not supported")));
+ }
+ }
+
+ if (indexInfo->ii_Expressions)
+ {
+ Bitmapset *indexattrs = NULL;
+ int varno = -1;
+
+ pull_varattnos((Node *) indexInfo->ii_Expressions, 1, &indexattrs);
+
+ while ((varno = bms_next_member(indexattrs, varno)) >= 0)
+ {
+ int attno = varno + FirstLowInvalidHeapAttributeNumber;
+ char *name = NameStr(TupleDescAttr(rel->rd_att, attno - 1)->attname);
+ if (name && isIvmName(name))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("unique index creation on IVM columns is not supported")));
+ }
+
+ }
+ }
+
+
/*
* Report index creation if appropriate (delay this till after most of the
* error checks)
diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c
index eaf39f80cd3..a2746ca9265 100644
--- a/src/backend/commands/matview.c
+++ b/src/backend/commands/matview.c
@@ -193,11 +193,15 @@ static Query *rewrite_query_for_postupdate_state(Query *query, MV_TriggerTable *
static void apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *new_tuplestores,
TupleDesc tupdesc_old, TupleDesc tupdesc_new,
- Query *query);
+ Query *query, bool use_count, char *count_colname);
static void apply_old_delta(const char *matviewname, const char *deltaname_old,
List *keys);
+static void apply_old_delta_with_count(const char *matviewname, const char *deltaname_old,
+ List *keys, const char *count_colname);
static void apply_new_delta(const char *matviewname, const char *deltaname_new,
StringInfo target_list);
+static void apply_new_delta_with_count(const char *matviewname, const char* deltaname_new,
+ List *keys, StringInfo target_list, const char* count_colname);
static char *get_matching_condition_string(List *keys);
static void generate_equal(StringInfo querybuf, Oid opttype,
const char *leftop, const char *rightop);
@@ -346,6 +350,7 @@ RefreshMatViewByOid(Oid matviewOid, bool is_create, bool skipData,
QueryCompletion *qc)
{
Relation matviewRel;
+ Query *viewQuery;
Query *dataQuery;
Oid tableSpace;
Oid relowner;
@@ -394,7 +399,13 @@ RefreshMatViewByOid(Oid matviewOid, bool is_create, bool skipData,
errmsg("%s options %s and %s cannot be used together",
"REFRESH", "CONCURRENTLY", "WITH NO DATA")));
- dataQuery = get_matview_query(matviewRel);
+ viewQuery = get_matview_query(matviewRel);
+
+ /* For IMMV, we need to rewrite matview query */
+ if (!skipData && RelationIsIVM(matviewRel))
+ dataQuery = rewriteQueryForIMMV(viewQuery,NIL);
+ else
+ dataQuery = viewQuery;
/*
* Check that there is a unique index with no WHERE clause on one or more
@@ -1688,6 +1699,13 @@ IVM_immediate_maintenance(PG_FUNCTION_ARGS)
int rte_index = lfirst_int(lc2);
TupleDesc tupdesc_old;
TupleDesc tupdesc_new;
+ bool use_count = false;
+ char *count_colname = NULL;
+
+ count_colname = pstrdup("__ivm_count__");
+
+ if (query->distinctClause)
+ use_count = true;
/* calculate delta tables */
calc_delta(table, rte_index, rewritten, dest_old, dest_new,
@@ -1700,7 +1718,8 @@ IVM_immediate_maintenance(PG_FUNCTION_ARGS)
{
/* apply the delta tables to the materialized view */
apply_delta(matviewOid, old_tuplestore, new_tuplestore,
- tupdesc_old, tupdesc_new, query);
+ tupdesc_old, tupdesc_new, query, use_count,
+ count_colname);
}
PG_CATCH();
{
@@ -2238,7 +2257,7 @@ rewrite_query_for_postupdate_state(Query *query, MV_TriggerTable *table, int rte
static void
apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *new_tuplestores,
TupleDesc tupdesc_old, TupleDesc tupdesc_new,
- Query *query)
+ Query *query, bool use_count, char *count_colname)
{
StringInfoData querybuf;
StringInfoData target_list_buf;
@@ -2314,7 +2333,12 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
if (rc != SPI_OK_REL_REGISTER)
elog(ERROR, "SPI_register failed");
- apply_old_delta(matviewname, OLD_DELTA_ENRNAME, keys);
+ if (use_count)
+ /* apply old delta and get rows to be recalculated */
+ apply_old_delta_with_count(matviewname, OLD_DELTA_ENRNAME,
+ keys, count_colname);
+ else
+ apply_old_delta(matviewname, OLD_DELTA_ENRNAME, keys);
}
/* For tuple insertion */
@@ -2336,7 +2360,11 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
elog(ERROR, "SPI_register failed");
/* apply new delta */
- apply_new_delta(matviewname, NEW_DELTA_ENRNAME, &target_list_buf);
+ if (use_count)
+ apply_new_delta_with_count(matviewname, NEW_DELTA_ENRNAME,
+ keys, &target_list_buf, count_colname);
+ else
+ apply_new_delta(matviewname, NEW_DELTA_ENRNAME, &target_list_buf);
}
/* We're done maintaining the materialized view. */
@@ -2349,6 +2377,51 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
elog(ERROR, "SPI_finish failed");
}
+/*
+ * apply_old_delta_with_count
+ *
+ * Execute a query for applying a delta table given by deltname_old
+ * which contains tuples to be deleted from to a materialized view given by
+ * matviewname. This is used when counting is required, that is, the view
+ * has aggregate or distinct.
+ */
+static void
+apply_old_delta_with_count(const char *matviewname, const char *deltaname_old,
+ List *keys, const char *count_colname)
+{
+ StringInfoData querybuf;
+ char *match_cond;
+
+ /* build WHERE condition for searching tuples to be deleted */
+ match_cond = get_matching_condition_string(keys);
+
+ /* Search for matching tuples from the view and update or delete if found. */
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH t AS (" /* collecting tid of target tuples in the view */
+ "SELECT diff.%s, " /* count column */
+ "(diff.%s OPERATOR(pg_catalog.=) mv.%s) AS for_dlt, "
+ "mv.ctid "
+ "FROM %s AS mv, %s AS diff "
+ "WHERE %s" /* tuple matching condition */
+ "), updt AS (" /* update a tuple if this is not to be deleted */
+ "UPDATE %s AS mv SET %s = mv.%s OPERATOR(pg_catalog.-) t.%s "
+ "FROM t WHERE mv.ctid OPERATOR(pg_catalog.=) t.ctid AND NOT for_dlt "
+ ")"
+ /* delete a tuple if this is to be deleted */
+ "DELETE FROM %s AS mv USING t "
+ "WHERE mv.ctid OPERATOR(pg_catalog.=) t.ctid AND for_dlt",
+ count_colname,
+ count_colname, count_colname,
+ matviewname, deltaname_old,
+ match_cond,
+ matviewname, count_colname, count_colname, count_colname,
+ matviewname);
+
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_DELETE)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+}
+
/*
* apply_old_delta
*
@@ -2398,6 +2471,66 @@ apply_old_delta(const char *matviewname, const char *deltaname_old,
elog(ERROR, "SPI_exec failed: %s", querybuf.data);
}
+/*
+ * apply_new_delta_with_count
+ *
+ * Execute a query for applying a delta table given by deltname_new
+ * which contains tuples to be inserted into a materialized view given by
+ * matviewname. This is used when counting is required, that is, the view
+ * has aggregate or distinct. Also, when a table in EXISTS sub queries
+ * is modified.
+ */
+static void
+apply_new_delta_with_count(const char *matviewname, const char* deltaname_new,
+ List *keys, StringInfo target_list, const char* count_colname)
+{
+ StringInfoData querybuf;
+ StringInfoData returning_keys;
+ ListCell *lc;
+ char *match_cond = "";
+
+ /* build WHERE condition for searching tuples to be updated */
+ match_cond = get_matching_condition_string(keys);
+
+ /* build string of keys list */
+ initStringInfo(&returning_keys);
+ if (keys)
+ {
+ foreach (lc, keys)
+ {
+ Form_pg_attribute attr = (Form_pg_attribute) lfirst(lc);
+ char *resname = NameStr(attr->attname);
+ appendStringInfo(&returning_keys, "%s", quote_qualified_identifier("mv", resname));
+ if (lnext(keys, lc))
+ appendStringInfo(&returning_keys, ", ");
+ }
+ }
+ else
+ appendStringInfo(&returning_keys, "NULL");
+
+ /* Search for matching tuples from the view and update if found or insert if not. */
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH updt AS (" /* update a tuple if this exists in the view */
+ "UPDATE %s AS mv SET %s = mv.%s OPERATOR(pg_catalog.+) diff.%s "
+ "FROM %s AS diff "
+ "WHERE %s " /* tuple matching condition */
+ "RETURNING %s" /* returning keys of updated tuples */
+ ") INSERT INTO %s (%s) " /* insert a new tuple if this doesn't exist */
+ "SELECT %s FROM %s AS diff "
+ "WHERE NOT EXISTS (SELECT 1 FROM updt AS mv WHERE %s);",
+ matviewname, count_colname, count_colname, count_colname,
+ deltaname_new,
+ match_cond,
+ returning_keys.data,
+ matviewname, target_list->data,
+ target_list->data, deltaname_new,
+ match_cond);
+
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+}
+
/*
* apply_new_delta
*
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 920b62d254d..f62765f0a89 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -60,6 +60,7 @@
#include "catalog/toasting.h"
#include "commands/comment.h"
#include "commands/defrem.h"
+#include "commands/matview.h"
#include "commands/event_trigger.h"
#include "commands/extension.h"
#include "commands/matview.h"
@@ -3930,6 +3931,14 @@ renameatt_internal(Oid myrelid,
targetrelation = relation_open(myrelid, AccessExclusiveLock);
renameatt_check(myrelid, RelationGetForm(targetrelation), recursing);
+ /*
+ * Don't rename IVM columns.
+ */
+ if (RelationIsIVM(targetrelation) && isIvmName(oldattname))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("IVM column can not be renamed")));
+
/*
* if the 'recurse' flag is set then we are supposed to rename this
* attribute in all classes that inherit from 'relname' (as well as in
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 953c5797c5d..0f228eca259 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -516,6 +516,7 @@ _outRangeTblEntry(StringInfo str, const RangeTblEntry *node)
WRITE_INT_FIELD(rellockmode);
WRITE_UINT_FIELD(perminfoindex);
WRITE_NODE_FIELD(tablesample);
+ WRITE_BOOL_FIELD(relisivm);
break;
case RTE_SUBQUERY:
WRITE_NODE_FIELD(subquery);
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index b6b2ce6c792..dba4d8ff2de 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -365,6 +365,7 @@ _readRangeTblEntry(void)
READ_INT_FIELD(rellockmode);
READ_UINT_FIELD(perminfoindex);
READ_NODE_FIELD(tablesample);
+ READ_BOOL_FIELD(relisivm);
break;
case RTE_SUBQUERY:
READ_NODE_FIELD(subquery);
diff --git a/src/backend/parser/parse_relation.c b/src/backend/parser/parse_relation.c
index ced210cd206..5392e9f15b2 100644
--- a/src/backend/parser/parse_relation.c
+++ b/src/backend/parser/parse_relation.c
@@ -33,6 +33,7 @@
#include "utils/lsyscache.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
+#include "commands/matview.h"
/*
@@ -96,7 +97,7 @@ static void expandTupleDesc(TupleDesc tupdesc, Alias *eref,
int rtindex, int sublevels_up,
VarReturningType returning_type,
int location, bool include_dropped,
- List **colnames, List **colvars);
+ List **colnames, List **colvars, bool is_ivm);
static int specialAttNum(const char *attname);
static bool rte_visible_if_lateral(ParseState *pstate, RangeTblEntry *rte);
static bool rte_visible_if_qualified(ParseState *pstate, RangeTblEntry *rte);
@@ -1544,6 +1545,7 @@ addRangeTableEntry(ParseState *pstate,
rte->inh = inh;
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -1625,6 +1627,7 @@ addRangeTableEntryForRelation(ParseState *pstate,
rte->inh = inh;
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -2967,7 +2970,7 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
rtfunc->funccolcount, atts_done,
rtindex, sublevels_up,
returning_type, location,
- include_dropped, colnames, colvars);
+ include_dropped, colnames, colvars, false);
}
else if (functypclass == TYPEFUNC_SCALAR)
{
@@ -3243,7 +3246,7 @@ expandRelation(Oid relid, Alias *eref, int rtindex, int sublevels_up,
expandTupleDesc(rel->rd_att, eref, rel->rd_att->natts, 0,
rtindex, sublevels_up, returning_type,
location, include_dropped,
- colnames, colvars);
+ colnames, colvars, RelationIsIVM(rel));
relation_close(rel, AccessShareLock);
}
@@ -3261,7 +3264,7 @@ expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
int rtindex, int sublevels_up,
VarReturningType returning_type,
int location, bool include_dropped,
- List **colnames, List **colvars)
+ List **colnames, List **colvars, bool is_ivm)
{
ListCell *aliascell;
int varattno;
@@ -3274,6 +3277,9 @@ expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
{
Form_pg_attribute attr = TupleDescAttr(tupdesc, varattno);
+ if (is_ivm && isIvmName(NameStr(attr->attname)) && !MatViewIncrementalMaintenanceIsEnabled())
+ continue;
+
if (attr->attisdropped)
{
if (include_dropped)
@@ -3438,6 +3444,10 @@ expandNSItemAttrs(ParseState *pstate, ParseNamespaceItem *nsitem,
Var *varnode = (Var *) lfirst(var);
TargetEntry *te;
+ /* if transform * into columnlist with IMMV, remove IVM columns */
+ if (rte->relisivm && isIvmName(label) && !MatViewIncrementalMaintenanceIsEnabled())
+ continue;
+
te = makeTargetEntry((Expr *) varnode,
(AttrNumber) pstate->p_next_resno++,
label,
diff --git a/src/backend/rewrite/rewriteDefine.c b/src/backend/rewrite/rewriteDefine.c
index 6a223fbeaa4..74ac631299c 100644
--- a/src/backend/rewrite/rewriteDefine.c
+++ b/src/backend/rewrite/rewriteDefine.c
@@ -614,7 +614,8 @@ checkRuleResultList(List *targetList, TupleDesc resultDesc, bool isSelect,
attr->atttypmod))));
}
- if (i != resultDesc->natts)
+ /* No check for materialized views since this could have special columns for IVM */
+ if ((!isSelect || requireColumnNameMatch) && i != resultDesc->natts)
ereport(ERROR,
(errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
isSelect ?
diff --git a/src/include/commands/createas.h b/src/include/commands/createas.h
index c286ebcd70e..bfd0249b10d 100644
--- a/src/include/commands/createas.h
+++ b/src/include/commands/createas.h
@@ -29,6 +29,8 @@ extern ObjectAddress ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *st
extern void CreateIvmTriggersOnBaseTables(Query *qry, Oid matviewOid);
extern void CreateIndexOnIMMV(Query *query, Relation matviewRel);
+extern Query *rewriteQueryForIMMV(Query *query, List *colNames);
+
extern int GetIntoRelEFlags(IntoClause *intoClause);
extern DestReceiver *CreateIntoRelDestReceiver(IntoClause *intoClause);
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 4133c404a6b..cbcef860061 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -1209,6 +1209,8 @@ typedef struct RangeTblEntry
Index perminfoindex pg_node_attr(query_jumble_ignore);
/* sampling info, or NULL */
struct TableSampleClause *tablesample;
+ /* incrementally maintainable materialized view? */
+ bool relisivm;
/*
* Fields valid for a subquery RTE (else NULL):
--
2.43.0
--Multipart=_Wed__1_Jul_2026_00_04_01_+0900_OVSy2WWK_9aByzDJ
Content-Type: text/x-diff;
name="v38-0006-Add-Incremental-View-Maintenance-support.patch"
Content-Disposition: attachment;
filename="v38-0006-Add-Incremental-View-Maintenance-support.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 24+ messages in thread
* [PATCH v28 07/11] Add DISTINCT support for IVM
@ 2023-05-31 10:08 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 24+ messages in thread
From: Yugo Nagata @ 2023-05-31 10:08 UTC (permalink / raw)
When IMMV is created with DISTINCT, multiplicity of tuples is
counted and stored in "__ivm_count__" column, which is a hidden
column of IMMV. The value in __ivm_count__ is updated when IMMV
is maintained incrementally. A tuple in IMMV can be removed if
and only if the count becomes zero.
---
src/backend/commands/createas.c | 141 ++++++++++++++++++++------
src/backend/commands/indexcmds.c | 40 ++++++++
src/backend/commands/matview.c | 148 ++++++++++++++++++++++++++--
src/backend/commands/tablecmds.c | 9 ++
src/backend/parser/parse_relation.c | 18 +++-
src/backend/rewrite/rewriteDefine.c | 3 +-
src/include/commands/createas.h | 2 +
src/include/nodes/parsenodes.h | 1 +
8 files changed, 317 insertions(+), 45 deletions(-)
diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c
index 415f110516..076f35ee6b 100644
--- a/src/backend/commands/createas.c
+++ b/src/backend/commands/createas.c
@@ -53,6 +53,7 @@
#include "parser/parser.h"
#include "parser/parsetree.h"
#include "parser/parse_clause.h"
+#include "parser/parse_func.h"
#include "rewrite/rewriteHandler.h"
#include "storage/smgr.h"
#include "tcop/tcopprot.h"
@@ -309,6 +310,9 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt,
errhint("functions must be marked IMMUTABLE")));
check_ivm_restriction((Node *) query);
+
+ /* For IMMV, we need to rewrite matview query */
+ query = rewriteQueryForIMMV(query, into->colNames);
}
if (into->skipData)
@@ -413,6 +417,49 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt,
return address;
}
+/*
+ * rewriteQueryForIMMV -- rewrite view definition query for IMMV
+ *
+ * count(*) is added for counting distinct tuples in views.
+ */
+Query *
+rewriteQueryForIMMV(Query *query, List *colNames)
+{
+ Query *rewritten;
+
+ Node *node;
+ ParseState *pstate = make_parsestate(NULL);
+ FuncCall *fn;
+
+ rewritten = copyObject(query);
+ pstate->p_expr_kind = EXPR_KIND_SELECT_TARGET;
+
+ /*
+ * Convert DISTINCT to GROUP BY and add count(*) for counting distinct
+ * tuples in views.
+ */
+ if (rewritten->distinctClause)
+ {
+ TargetEntry *tle;
+
+ rewritten->groupClause = transformDistinctClause(NULL, &rewritten->targetList, rewritten->sortClause, false);
+
+ fn = makeFuncCall(SystemFuncName("count"), NIL, COERCE_EXPLICIT_CALL, -1);
+ fn->agg_star = true;
+
+ node = ParseFuncOrColumn(pstate, fn->funcname, NIL, NULL, fn, false, -1);
+
+ tle = makeTargetEntry((Expr *) node,
+ list_length(rewritten->targetList) + 1,
+ pstrdup("__ivm_count__"),
+ false);
+ rewritten->targetList = lappend(rewritten->targetList, tle);
+ rewritten->hasAggs = true;
+ }
+
+ return rewritten;
+}
+
/*
* GetIntoRelEFlags --- compute executor flags needed for CREATE TABLE AS
*
@@ -536,7 +583,8 @@ intorel_startup(DestReceiver *self, int operation, TupleDesc typeinfo)
ColumnDef *col;
char *colname;
- if (lc)
+ /* Don't override hidden columns added for IVM */
+ if (lc && !isIvmName(NameStr(attribute->attname)))
{
colname = strVal(lfirst(lc));
lc = lnext(into->colNames, lc);
@@ -940,10 +988,6 @@ check_ivm_restriction_walker(Node *node, void *context)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("LIMIT/OFFSET clause is not supported on incrementally maintainable materialized view")));
- if (qry->distinctClause)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("DISTINCT is not supported on incrementally maintainable materialized view")));
if (qry->hasDistinctOn)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -1090,12 +1134,18 @@ CreateIndexOnIMMV(Query *query, Relation matviewRel)
char idxname[NAMEDATALEN];
List *indexoidlist = RelationGetIndexList(matviewRel);
ListCell *indexoidscan;
- Bitmapset *key_attnos;
snprintf(idxname, sizeof(idxname), "%s_index", RelationGetRelationName(matviewRel));
index = makeNode(IndexStmt);
+ /*
+ * We consider null values not distinct to make sure that views with DISTINCT
+ * or GROUP BY don't contain multiple NULL rows when NULL is inserted to
+ * a base table concurrently.
+ */
+ index->nulls_not_distinct = true;
+
index->unique = true;
index->primary = false;
index->isconstraint = false;
@@ -1122,41 +1172,68 @@ CreateIndexOnIMMV(Query *query, Relation matviewRel)
index->concurrent = false;
index->if_not_exists = false;
- /* create index on the base tables' primary key columns */
- key_attnos = get_primary_key_attnos_from_query(query, &constraintList);
- if (key_attnos)
+ if (query->distinctClause)
{
+ /* create unique constraint on all columns */
foreach(lc, query->targetList)
{
TargetEntry *tle = (TargetEntry *) lfirst(lc);
Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, tle->resno - 1);
-
- if (bms_is_member(tle->resno - FirstLowInvalidHeapAttributeNumber, key_attnos))
- {
- IndexElem *iparam;
-
- iparam = makeNode(IndexElem);
- iparam->name = pstrdup(NameStr(attr->attname));
- iparam->expr = NULL;
- iparam->indexcolname = NULL;
- iparam->collation = NIL;
- iparam->opclass = NIL;
- iparam->opclassopts = NIL;
- iparam->ordering = SORTBY_DEFAULT;
- iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
- index->indexParams = lappend(index->indexParams, iparam);
- }
+ IndexElem *iparam;
+
+ iparam = makeNode(IndexElem);
+ iparam->name = pstrdup(NameStr(attr->attname));
+ iparam->expr = NULL;
+ iparam->indexcolname = NULL;
+ iparam->collation = NIL;
+ iparam->opclass = NIL;
+ iparam->opclassopts = NIL;
+ iparam->ordering = SORTBY_DEFAULT;
+ iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
+ index->indexParams = lappend(index->indexParams, iparam);
}
}
else
{
- /* create no index, just notice that an appropriate index is necessary for efficient IVM */
- ereport(NOTICE,
- (errmsg("could not create an index on materialized view \"%s\" automatically",
- RelationGetRelationName(matviewRel)),
- errdetail("This target list does not have all the primary key columns. "),
- errhint("Create an index on the materialized view for efficient incremental maintenance.")));
- return;
+ Bitmapset *key_attnos;
+
+ /* create index on the base tables' primary key columns */
+ key_attnos = get_primary_key_attnos_from_query(query, &constraintList);
+ if (key_attnos)
+ {
+ foreach(lc, query->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);
+ Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, tle->resno - 1);
+
+ if (bms_is_member(tle->resno - FirstLowInvalidHeapAttributeNumber, key_attnos))
+ {
+ IndexElem *iparam;
+
+ iparam = makeNode(IndexElem);
+ iparam->name = pstrdup(NameStr(attr->attname));
+ iparam->expr = NULL;
+ iparam->indexcolname = NULL;
+ iparam->collation = NIL;
+ iparam->opclass = NIL;
+ iparam->opclassopts = NIL;
+ iparam->ordering = SORTBY_DEFAULT;
+ iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
+ index->indexParams = lappend(index->indexParams, iparam);
+ }
+ }
+ }
+ else
+ {
+ /* create no index, just notice that an appropriate index is necessary for efficient IVM */
+ ereport(NOTICE,
+ (errmsg("could not create an index on materialized view \"%s\" automatically",
+ RelationGetRelationName(matviewRel)),
+ errdetail("This target list does not have all the primary key columns, "
+ "or this view does not contain DISTINCT clause."),
+ errhint("Create an index on the materialized view for efficient incremental maintenance.")));
+ return;
+ }
}
/* If we have a compatible index, we don't need to create another. */
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index a5168c9f09..480c3a00ed 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -39,6 +39,7 @@
#include "commands/dbcommands.h"
#include "commands/defrem.h"
#include "commands/event_trigger.h"
+#include "commands/matview.h"
#include "commands/progress.h"
#include "commands/tablecmds.h"
#include "commands/tablespace.h"
@@ -1090,6 +1091,45 @@ DefineIndex(Oid relationId,
safe_index = indexInfo->ii_Expressions == NIL &&
indexInfo->ii_Predicate == NIL;
+ /*
+ * We disallow unique indexes on IVM columns of IMMVs.
+ */
+ if (RelationIsIVM(rel) && stmt->unique)
+ {
+ for (int i = 0; i < indexInfo->ii_NumIndexKeyAttrs; i++)
+ {
+ AttrNumber attno = indexInfo->ii_IndexAttrNumbers[i];
+ if (attno > 0)
+ {
+ char *name = NameStr(TupleDescAttr(rel->rd_att, attno - 1)->attname);
+ if (name && isIvmName(name))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("unique index creation on IVM columns is not supported")));
+ }
+ }
+
+ if (indexInfo->ii_Expressions)
+ {
+ Bitmapset *indexattrs = NULL;
+ int varno = -1;
+
+ pull_varattnos((Node *) indexInfo->ii_Expressions, 1, &indexattrs);
+
+ while ((varno = bms_next_member(indexattrs, varno)) >= 0)
+ {
+ int attno = varno + FirstLowInvalidHeapAttributeNumber;
+ char *name = NameStr(TupleDescAttr(rel->rd_att, attno - 1)->attname);
+ if (name && isIvmName(name))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("unique index creation on IVM columns is not supported")));
+ }
+
+ }
+ }
+
+
/*
* Report index creation if appropriate (delay this till after most of the
* error checks)
diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c
index fd9d0d99ae..6d8382180a 100644
--- a/src/backend/commands/matview.c
+++ b/src/backend/commands/matview.c
@@ -152,11 +152,15 @@ static Query *rewrite_query_for_postupdate_state(Query *query, MV_TriggerTable *
static void apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *new_tuplestores,
TupleDesc tupdesc_old, TupleDesc tupdesc_new,
- Query *query);
+ Query *query, bool use_count, char *count_colname);
static void apply_old_delta(const char *matviewname, const char *deltaname_old,
List *keys);
+static void apply_old_delta_with_count(const char *matviewname, const char *deltaname_old,
+ List *keys, const char *count_colname);
static void apply_new_delta(const char *matviewname, const char *deltaname_new,
StringInfo target_list);
+static void apply_new_delta_with_count(const char *matviewname, const char* deltaname_new,
+ List *keys, StringInfo target_list, const char* count_colname);
static char *get_matching_condition_string(List *keys);
static void generate_equal(StringInfo querybuf, Oid opttype,
const char *leftop, const char *rightop);
@@ -271,6 +275,7 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
Oid matviewOid;
Relation matviewRel;
Query *dataQuery;
+ Query *viewQuery;
Oid tableSpace;
Oid relowner;
Oid OIDNewHeap;
@@ -331,8 +336,13 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
"CONCURRENTLY", "WITH NO DATA")));
- dataQuery = get_matview_query(matviewRel);
+ viewQuery = get_matview_query(matviewRel);
+ /* For IMMV, we need to rewrite matview query */
+ if (!stmt->skipData && RelationIsIVM(matviewRel))
+ dataQuery = rewriteQueryForIMMV(viewQuery,NIL);
+ else
+ dataQuery = viewQuery;
/*
* Check that there is a unique index with no WHERE clause on one or more
@@ -512,8 +522,8 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
if (!stmt->skipData && RelationIsIVM(matviewRel) && !oldPopulated)
{
- CreateIndexOnIMMV(dataQuery, matviewRel);
- CreateIvmTriggersOnBaseTables(dataQuery, matviewOid);
+ CreateIndexOnIMMV(viewQuery, matviewRel);
+ CreateIvmTriggersOnBaseTables(viewQuery, matviewOid);
}
table_close(matviewRel, NoLock);
@@ -1513,6 +1523,13 @@ IVM_immediate_maintenance(PG_FUNCTION_ARGS)
int rte_index = lfirst_int(lc2);
TupleDesc tupdesc_old;
TupleDesc tupdesc_new;
+ bool use_count = false;
+ char *count_colname = NULL;
+
+ count_colname = pstrdup("__ivm_count__");
+
+ if (query->distinctClause)
+ use_count = true;
/* calculate delta tables */
calc_delta(table, rte_index, rewritten, dest_old, dest_new,
@@ -1525,7 +1542,8 @@ IVM_immediate_maintenance(PG_FUNCTION_ARGS)
{
/* apply the delta tables to the materialized view */
apply_delta(matviewOid, old_tuplestore, new_tuplestore,
- tupdesc_old, tupdesc_new, query);
+ tupdesc_old, tupdesc_new, query, use_count,
+ count_colname);
}
PG_CATCH();
{
@@ -1998,7 +2016,7 @@ rewrite_query_for_postupdate_state(Query *query, MV_TriggerTable *table, int rte
static void
apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *new_tuplestores,
TupleDesc tupdesc_old, TupleDesc tupdesc_new,
- Query *query)
+ Query *query, bool use_count, char *count_colname)
{
StringInfoData querybuf;
StringInfoData target_list_buf;
@@ -2074,7 +2092,12 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
if (rc != SPI_OK_REL_REGISTER)
elog(ERROR, "SPI_register failed");
- apply_old_delta(matviewname, OLD_DELTA_ENRNAME, keys);
+ if (use_count)
+ /* apply old delta and get rows to be recalculated */
+ apply_old_delta_with_count(matviewname, OLD_DELTA_ENRNAME,
+ keys, count_colname);
+ else
+ apply_old_delta(matviewname, OLD_DELTA_ENRNAME, keys);
}
/* For tuple insertion */
@@ -2096,7 +2119,11 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
elog(ERROR, "SPI_register failed");
/* apply new delta */
- apply_new_delta(matviewname, NEW_DELTA_ENRNAME, &target_list_buf);
+ if (use_count)
+ apply_new_delta_with_count(matviewname, NEW_DELTA_ENRNAME,
+ keys, &target_list_buf, count_colname);
+ else
+ apply_new_delta(matviewname, NEW_DELTA_ENRNAME, &target_list_buf);
}
/* We're done maintaining the materialized view. */
@@ -2109,6 +2136,51 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
elog(ERROR, "SPI_finish failed");
}
+/*
+ * apply_old_delta_with_count
+ *
+ * Execute a query for applying a delta table given by deltname_old
+ * which contains tuples to be deleted from to a materialized view given by
+ * matviewname. This is used when counting is required, that is, the view
+ * has aggregate or distinct.
+ */
+static void
+apply_old_delta_with_count(const char *matviewname, const char *deltaname_old,
+ List *keys, const char *count_colname)
+{
+ StringInfoData querybuf;
+ char *match_cond;
+
+ /* build WHERE condition for searching tuples to be deleted */
+ match_cond = get_matching_condition_string(keys);
+
+ /* Search for matching tuples from the view and update or delete if found. */
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH t AS (" /* collecting tid of target tuples in the view */
+ "SELECT diff.%s, " /* count column */
+ "(diff.%s OPERATOR(pg_catalog.=) mv.%s) AS for_dlt, "
+ "mv.ctid "
+ "FROM %s AS mv, %s AS diff "
+ "WHERE %s" /* tuple matching condition */
+ "), updt AS (" /* update a tuple if this is not to be deleted */
+ "UPDATE %s AS mv SET %s = mv.%s OPERATOR(pg_catalog.-) t.%s "
+ "FROM t WHERE mv.ctid OPERATOR(pg_catalog.=) t.ctid AND NOT for_dlt "
+ ")"
+ /* delete a tuple if this is to be deleted */
+ "DELETE FROM %s AS mv USING t "
+ "WHERE mv.ctid OPERATOR(pg_catalog.=) t.ctid AND for_dlt",
+ count_colname,
+ count_colname, count_colname,
+ matviewname, deltaname_old,
+ match_cond,
+ matviewname, count_colname, count_colname, count_colname,
+ matviewname);
+
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_DELETE)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+}
+
/*
* apply_old_delta
*
@@ -2158,6 +2230,66 @@ apply_old_delta(const char *matviewname, const char *deltaname_old,
elog(ERROR, "SPI_exec failed: %s", querybuf.data);
}
+/*
+ * apply_new_delta_with_count
+ *
+ * Execute a query for applying a delta table given by deltname_new
+ * which contains tuples to be inserted into a materialized view given by
+ * matviewname. This is used when counting is required, that is, the view
+ * has aggregate or distinct. Also, when a table in EXISTS sub queries
+ * is modified.
+ */
+static void
+apply_new_delta_with_count(const char *matviewname, const char* deltaname_new,
+ List *keys, StringInfo target_list, const char* count_colname)
+{
+ StringInfoData querybuf;
+ StringInfoData returning_keys;
+ ListCell *lc;
+ char *match_cond = "";
+
+ /* build WHERE condition for searching tuples to be updated */
+ match_cond = get_matching_condition_string(keys);
+
+ /* build string of keys list */
+ initStringInfo(&returning_keys);
+ if (keys)
+ {
+ foreach (lc, keys)
+ {
+ Form_pg_attribute attr = (Form_pg_attribute) lfirst(lc);
+ char *resname = NameStr(attr->attname);
+ appendStringInfo(&returning_keys, "%s", quote_qualified_identifier("mv", resname));
+ if (lnext(keys, lc))
+ appendStringInfo(&returning_keys, ", ");
+ }
+ }
+ else
+ appendStringInfo(&returning_keys, "NULL");
+
+ /* Search for matching tuples from the view and update if found or insert if not. */
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH updt AS (" /* update a tuple if this exists in the view */
+ "UPDATE %s AS mv SET %s = mv.%s OPERATOR(pg_catalog.+) diff.%s "
+ "FROM %s AS diff "
+ "WHERE %s " /* tuple matching condition */
+ "RETURNING %s" /* returning keys of updated tuples */
+ ") INSERT INTO %s (%s)" /* insert a new tuple if this doesn't existw */
+ "SELECT %s FROM %s AS diff "
+ "WHERE NOT EXISTS (SELECT 1 FROM updt AS mv WHERE %s);",
+ matviewname, count_colname, count_colname, count_colname,
+ deltaname_new,
+ match_cond,
+ returning_keys.data,
+ matviewname, target_list->data,
+ target_list->data, deltaname_new,
+ match_cond);
+
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+}
+
/*
* apply_new_delta
*
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 4d49d70c33..25ff2a23db 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -53,6 +53,7 @@
#include "commands/cluster.h"
#include "commands/comment.h"
#include "commands/defrem.h"
+#include "commands/matview.h"
#include "commands/event_trigger.h"
#include "commands/policy.h"
#include "commands/sequence.h"
@@ -3548,6 +3549,14 @@ renameatt_internal(Oid myrelid,
targetrelation = relation_open(myrelid, AccessExclusiveLock);
renameatt_check(myrelid, RelationGetForm(targetrelation), recursing);
+ /*
+ * Don't rename IVM columns.
+ */
+ if (RelationIsIVM(targetrelation) && isIvmName(oldattname))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("IVM column can not be renamed")));
+
/*
* if the 'recurse' flag is set then we are supposed to rename this
* attribute in all classes that inherit from 'relname' (as well as in
diff --git a/src/backend/parser/parse_relation.c b/src/backend/parser/parse_relation.c
index 41d60494b9..f516ac91e2 100644
--- a/src/backend/parser/parse_relation.c
+++ b/src/backend/parser/parse_relation.c
@@ -36,6 +36,7 @@
#include "utils/rel.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
+#include "commands/matview.h"
/*
@@ -97,7 +98,7 @@ static void expandTupleDesc(TupleDesc tupdesc, Alias *eref,
int count, int offset,
int rtindex, int sublevels_up,
int location, bool include_dropped,
- List **colnames, List **colvars);
+ List **colnames, List **colvars, bool is_ivm);
static int specialAttNum(const char *attname);
static bool rte_visible_if_lateral(ParseState *pstate, RangeTblEntry *rte);
static bool rte_visible_if_qualified(ParseState *pstate, RangeTblEntry *rte);
@@ -1501,6 +1502,7 @@ addRangeTableEntry(ParseState *pstate,
rte->relid = RelationGetRelid(rel);
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -1586,6 +1588,7 @@ addRangeTableEntryForRelation(ParseState *pstate,
rte->relid = RelationGetRelid(rel);
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -2757,7 +2760,7 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
expandTupleDesc(tupdesc, rte->eref,
rtfunc->funccolcount, atts_done,
rtindex, sublevels_up, location,
- include_dropped, colnames, colvars);
+ include_dropped, colnames, colvars, false);
}
else if (functypclass == TYPEFUNC_SCALAR)
{
@@ -3025,7 +3028,7 @@ expandRelation(Oid relid, Alias *eref, int rtindex, int sublevels_up,
expandTupleDesc(rel->rd_att, eref, rel->rd_att->natts, 0,
rtindex, sublevels_up,
location, include_dropped,
- colnames, colvars);
+ colnames, colvars, RelationIsIVM(rel));
relation_close(rel, AccessShareLock);
}
@@ -3042,7 +3045,7 @@ static void
expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
int rtindex, int sublevels_up,
int location, bool include_dropped,
- List **colnames, List **colvars)
+ List **colnames, List **colvars, bool is_ivm)
{
ListCell *aliascell;
int varattno;
@@ -3055,6 +3058,9 @@ expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
{
Form_pg_attribute attr = TupleDescAttr(tupdesc, varattno);
+ if (is_ivm && isIvmName(NameStr(attr->attname)) && !MatViewIncrementalMaintenanceIsEnabled())
+ continue;
+
if (attr->attisdropped)
{
if (include_dropped)
@@ -3217,6 +3223,10 @@ expandNSItemAttrs(ParseState *pstate, ParseNamespaceItem *nsitem,
Var *varnode = (Var *) lfirst(var);
TargetEntry *te;
+ /* if transform * into columnlist with IMMV, remove IVM columns */
+ if (rte->relisivm && isIvmName(label) && !MatViewIncrementalMaintenanceIsEnabled())
+ continue;
+
te = makeTargetEntry((Expr *) varnode,
(AttrNumber) pstate->p_next_resno++,
label,
diff --git a/src/backend/rewrite/rewriteDefine.c b/src/backend/rewrite/rewriteDefine.c
index e36fc72e1e..f6dc7ba202 100644
--- a/src/backend/rewrite/rewriteDefine.c
+++ b/src/backend/rewrite/rewriteDefine.c
@@ -621,7 +621,8 @@ checkRuleResultList(List *targetList, TupleDesc resultDesc, bool isSelect,
attr->atttypmod))));
}
- if (i != resultDesc->natts)
+ /* No check for materialized views since this could have special columns for IVM */
+ if ((!isSelect || requireColumnNameMatch) && i != resultDesc->natts)
ereport(ERROR,
(errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
isSelect ?
diff --git a/src/include/commands/createas.h b/src/include/commands/createas.h
index 09a64fa2e5..76a7873ebf 100644
--- a/src/include/commands/createas.h
+++ b/src/include/commands/createas.h
@@ -29,6 +29,8 @@ extern ObjectAddress ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *st
extern void CreateIvmTriggersOnBaseTables(Query *qry, Oid matviewOid);
extern void CreateIndexOnIMMV(Query *query, Relation matviewRel);
+extern Query *rewriteQueryForIMMV(Query *query, List *colNames);
+
extern int GetIntoRelEFlags(IntoClause *intoClause);
extern DestReceiver *CreateIntoRelDestReceiver(IntoClause *intoClause);
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 0ca298f5a1..43c4ed49f1 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -1074,6 +1074,7 @@ typedef struct RangeTblEntry
int rellockmode; /* lock level that query requires on the rel */
struct TableSampleClause *tablesample; /* sampling info, or NULL */
Index perminfoindex;
+ bool relisivm;
/*
* Fields valid for a subquery RTE (else NULL):
--
2.25.1
--Multipart=_Thu__1_Jun_2023_23_59_09_+0900_/G5+8nG46.f1T42K
Content-Type: text/x-diff;
name="v28-0008-Add-aggregates-support-in-IVM.patch"
Content-Disposition: attachment;
filename="v28-0008-Add-aggregates-support-in-IVM.patch"
Content-Transfer-Encoding: quoted-printable
^ permalink raw reply [nested|flat] 24+ messages in thread
* [PATCH v29 07/11] Add DISTINCT support for IVM
@ 2023-05-31 10:08 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 24+ messages in thread
From: Yugo Nagata @ 2023-05-31 10:08 UTC (permalink / raw)
When IMMV is created with DISTINCT, multiplicity of tuples is
counted and stored in "__ivm_count__" column, which is a hidden
column of IMMV. The value in __ivm_count__ is updated when IMMV
is maintained incrementally. A tuple in IMMV can be removed if
and only if the count becomes zero.
---
src/backend/commands/createas.c | 141 ++++++++++++++++++++------
src/backend/commands/indexcmds.c | 40 ++++++++
src/backend/commands/matview.c | 148 ++++++++++++++++++++++++++--
src/backend/commands/tablecmds.c | 9 ++
src/backend/parser/parse_relation.c | 18 +++-
src/backend/rewrite/rewriteDefine.c | 3 +-
src/include/commands/createas.h | 2 +
src/include/nodes/parsenodes.h | 1 +
8 files changed, 317 insertions(+), 45 deletions(-)
diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c
index 415f110516..076f35ee6b 100644
--- a/src/backend/commands/createas.c
+++ b/src/backend/commands/createas.c
@@ -53,6 +53,7 @@
#include "parser/parser.h"
#include "parser/parsetree.h"
#include "parser/parse_clause.h"
+#include "parser/parse_func.h"
#include "rewrite/rewriteHandler.h"
#include "storage/smgr.h"
#include "tcop/tcopprot.h"
@@ -309,6 +310,9 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt,
errhint("functions must be marked IMMUTABLE")));
check_ivm_restriction((Node *) query);
+
+ /* For IMMV, we need to rewrite matview query */
+ query = rewriteQueryForIMMV(query, into->colNames);
}
if (into->skipData)
@@ -413,6 +417,49 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt,
return address;
}
+/*
+ * rewriteQueryForIMMV -- rewrite view definition query for IMMV
+ *
+ * count(*) is added for counting distinct tuples in views.
+ */
+Query *
+rewriteQueryForIMMV(Query *query, List *colNames)
+{
+ Query *rewritten;
+
+ Node *node;
+ ParseState *pstate = make_parsestate(NULL);
+ FuncCall *fn;
+
+ rewritten = copyObject(query);
+ pstate->p_expr_kind = EXPR_KIND_SELECT_TARGET;
+
+ /*
+ * Convert DISTINCT to GROUP BY and add count(*) for counting distinct
+ * tuples in views.
+ */
+ if (rewritten->distinctClause)
+ {
+ TargetEntry *tle;
+
+ rewritten->groupClause = transformDistinctClause(NULL, &rewritten->targetList, rewritten->sortClause, false);
+
+ fn = makeFuncCall(SystemFuncName("count"), NIL, COERCE_EXPLICIT_CALL, -1);
+ fn->agg_star = true;
+
+ node = ParseFuncOrColumn(pstate, fn->funcname, NIL, NULL, fn, false, -1);
+
+ tle = makeTargetEntry((Expr *) node,
+ list_length(rewritten->targetList) + 1,
+ pstrdup("__ivm_count__"),
+ false);
+ rewritten->targetList = lappend(rewritten->targetList, tle);
+ rewritten->hasAggs = true;
+ }
+
+ return rewritten;
+}
+
/*
* GetIntoRelEFlags --- compute executor flags needed for CREATE TABLE AS
*
@@ -536,7 +583,8 @@ intorel_startup(DestReceiver *self, int operation, TupleDesc typeinfo)
ColumnDef *col;
char *colname;
- if (lc)
+ /* Don't override hidden columns added for IVM */
+ if (lc && !isIvmName(NameStr(attribute->attname)))
{
colname = strVal(lfirst(lc));
lc = lnext(into->colNames, lc);
@@ -940,10 +988,6 @@ check_ivm_restriction_walker(Node *node, void *context)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("LIMIT/OFFSET clause is not supported on incrementally maintainable materialized view")));
- if (qry->distinctClause)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("DISTINCT is not supported on incrementally maintainable materialized view")));
if (qry->hasDistinctOn)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -1090,12 +1134,18 @@ CreateIndexOnIMMV(Query *query, Relation matviewRel)
char idxname[NAMEDATALEN];
List *indexoidlist = RelationGetIndexList(matviewRel);
ListCell *indexoidscan;
- Bitmapset *key_attnos;
snprintf(idxname, sizeof(idxname), "%s_index", RelationGetRelationName(matviewRel));
index = makeNode(IndexStmt);
+ /*
+ * We consider null values not distinct to make sure that views with DISTINCT
+ * or GROUP BY don't contain multiple NULL rows when NULL is inserted to
+ * a base table concurrently.
+ */
+ index->nulls_not_distinct = true;
+
index->unique = true;
index->primary = false;
index->isconstraint = false;
@@ -1122,41 +1172,68 @@ CreateIndexOnIMMV(Query *query, Relation matviewRel)
index->concurrent = false;
index->if_not_exists = false;
- /* create index on the base tables' primary key columns */
- key_attnos = get_primary_key_attnos_from_query(query, &constraintList);
- if (key_attnos)
+ if (query->distinctClause)
{
+ /* create unique constraint on all columns */
foreach(lc, query->targetList)
{
TargetEntry *tle = (TargetEntry *) lfirst(lc);
Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, tle->resno - 1);
-
- if (bms_is_member(tle->resno - FirstLowInvalidHeapAttributeNumber, key_attnos))
- {
- IndexElem *iparam;
-
- iparam = makeNode(IndexElem);
- iparam->name = pstrdup(NameStr(attr->attname));
- iparam->expr = NULL;
- iparam->indexcolname = NULL;
- iparam->collation = NIL;
- iparam->opclass = NIL;
- iparam->opclassopts = NIL;
- iparam->ordering = SORTBY_DEFAULT;
- iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
- index->indexParams = lappend(index->indexParams, iparam);
- }
+ IndexElem *iparam;
+
+ iparam = makeNode(IndexElem);
+ iparam->name = pstrdup(NameStr(attr->attname));
+ iparam->expr = NULL;
+ iparam->indexcolname = NULL;
+ iparam->collation = NIL;
+ iparam->opclass = NIL;
+ iparam->opclassopts = NIL;
+ iparam->ordering = SORTBY_DEFAULT;
+ iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
+ index->indexParams = lappend(index->indexParams, iparam);
}
}
else
{
- /* create no index, just notice that an appropriate index is necessary for efficient IVM */
- ereport(NOTICE,
- (errmsg("could not create an index on materialized view \"%s\" automatically",
- RelationGetRelationName(matviewRel)),
- errdetail("This target list does not have all the primary key columns. "),
- errhint("Create an index on the materialized view for efficient incremental maintenance.")));
- return;
+ Bitmapset *key_attnos;
+
+ /* create index on the base tables' primary key columns */
+ key_attnos = get_primary_key_attnos_from_query(query, &constraintList);
+ if (key_attnos)
+ {
+ foreach(lc, query->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);
+ Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, tle->resno - 1);
+
+ if (bms_is_member(tle->resno - FirstLowInvalidHeapAttributeNumber, key_attnos))
+ {
+ IndexElem *iparam;
+
+ iparam = makeNode(IndexElem);
+ iparam->name = pstrdup(NameStr(attr->attname));
+ iparam->expr = NULL;
+ iparam->indexcolname = NULL;
+ iparam->collation = NIL;
+ iparam->opclass = NIL;
+ iparam->opclassopts = NIL;
+ iparam->ordering = SORTBY_DEFAULT;
+ iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
+ index->indexParams = lappend(index->indexParams, iparam);
+ }
+ }
+ }
+ else
+ {
+ /* create no index, just notice that an appropriate index is necessary for efficient IVM */
+ ereport(NOTICE,
+ (errmsg("could not create an index on materialized view \"%s\" automatically",
+ RelationGetRelationName(matviewRel)),
+ errdetail("This target list does not have all the primary key columns, "
+ "or this view does not contain DISTINCT clause."),
+ errhint("Create an index on the materialized view for efficient incremental maintenance.")));
+ return;
+ }
}
/* If we have a compatible index, we don't need to create another. */
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index ab8b81b302..4811a1c8df 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -38,6 +38,7 @@
#include "commands/dbcommands.h"
#include "commands/defrem.h"
#include "commands/event_trigger.h"
+#include "commands/matview.h"
#include "commands/progress.h"
#include "commands/tablecmds.h"
#include "commands/tablespace.h"
@@ -1104,6 +1105,45 @@ DefineIndex(Oid tableId,
safe_index = indexInfo->ii_Expressions == NIL &&
indexInfo->ii_Predicate == NIL;
+ /*
+ * We disallow unique indexes on IVM columns of IMMVs.
+ */
+ if (RelationIsIVM(rel) && stmt->unique)
+ {
+ for (int i = 0; i < indexInfo->ii_NumIndexKeyAttrs; i++)
+ {
+ AttrNumber attno = indexInfo->ii_IndexAttrNumbers[i];
+ if (attno > 0)
+ {
+ char *name = NameStr(TupleDescAttr(rel->rd_att, attno - 1)->attname);
+ if (name && isIvmName(name))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("unique index creation on IVM columns is not supported")));
+ }
+ }
+
+ if (indexInfo->ii_Expressions)
+ {
+ Bitmapset *indexattrs = NULL;
+ int varno = -1;
+
+ pull_varattnos((Node *) indexInfo->ii_Expressions, 1, &indexattrs);
+
+ while ((varno = bms_next_member(indexattrs, varno)) >= 0)
+ {
+ int attno = varno + FirstLowInvalidHeapAttributeNumber;
+ char *name = NameStr(TupleDescAttr(rel->rd_att, attno - 1)->attname);
+ if (name && isIvmName(name))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("unique index creation on IVM columns is not supported")));
+ }
+
+ }
+ }
+
+
/*
* Report index creation if appropriate (delay this till after most of the
* error checks)
diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c
index 39305f3c49..aa518f20ef 100644
--- a/src/backend/commands/matview.c
+++ b/src/backend/commands/matview.c
@@ -152,11 +152,15 @@ static Query *rewrite_query_for_postupdate_state(Query *query, MV_TriggerTable *
static void apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *new_tuplestores,
TupleDesc tupdesc_old, TupleDesc tupdesc_new,
- Query *query);
+ Query *query, bool use_count, char *count_colname);
static void apply_old_delta(const char *matviewname, const char *deltaname_old,
List *keys);
+static void apply_old_delta_with_count(const char *matviewname, const char *deltaname_old,
+ List *keys, const char *count_colname);
static void apply_new_delta(const char *matviewname, const char *deltaname_new,
StringInfo target_list);
+static void apply_new_delta_with_count(const char *matviewname, const char* deltaname_new,
+ List *keys, StringInfo target_list, const char* count_colname);
static char *get_matching_condition_string(List *keys);
static void generate_equal(StringInfo querybuf, Oid opttype,
const char *leftop, const char *rightop);
@@ -271,6 +275,7 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
Oid matviewOid;
Relation matviewRel;
Query *dataQuery;
+ Query *viewQuery;
Oid tableSpace;
Oid relowner;
Oid OIDNewHeap;
@@ -330,8 +335,13 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
"CONCURRENTLY", "WITH NO DATA")));
- dataQuery = get_matview_query(matviewRel);
+ viewQuery = get_matview_query(matviewRel);
+ /* For IMMV, we need to rewrite matview query */
+ if (!stmt->skipData && RelationIsIVM(matviewRel))
+ dataQuery = rewriteQueryForIMMV(viewQuery,NIL);
+ else
+ dataQuery = viewQuery;
/*
* Check that there is a unique index with no WHERE clause on one or more
@@ -511,8 +521,8 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
if (!stmt->skipData && RelationIsIVM(matviewRel) && !oldPopulated)
{
- CreateIndexOnIMMV(dataQuery, matviewRel);
- CreateIvmTriggersOnBaseTables(dataQuery, matviewOid);
+ CreateIndexOnIMMV(viewQuery, matviewRel);
+ CreateIvmTriggersOnBaseTables(viewQuery, matviewOid);
}
table_close(matviewRel, NoLock);
@@ -1512,6 +1522,13 @@ IVM_immediate_maintenance(PG_FUNCTION_ARGS)
int rte_index = lfirst_int(lc2);
TupleDesc tupdesc_old;
TupleDesc tupdesc_new;
+ bool use_count = false;
+ char *count_colname = NULL;
+
+ count_colname = pstrdup("__ivm_count__");
+
+ if (query->distinctClause)
+ use_count = true;
/* calculate delta tables */
calc_delta(table, rte_index, rewritten, dest_old, dest_new,
@@ -1524,7 +1541,8 @@ IVM_immediate_maintenance(PG_FUNCTION_ARGS)
{
/* apply the delta tables to the materialized view */
apply_delta(matviewOid, old_tuplestore, new_tuplestore,
- tupdesc_old, tupdesc_new, query);
+ tupdesc_old, tupdesc_new, query, use_count,
+ count_colname);
}
PG_CATCH();
{
@@ -1997,7 +2015,7 @@ rewrite_query_for_postupdate_state(Query *query, MV_TriggerTable *table, int rte
static void
apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *new_tuplestores,
TupleDesc tupdesc_old, TupleDesc tupdesc_new,
- Query *query)
+ Query *query, bool use_count, char *count_colname)
{
StringInfoData querybuf;
StringInfoData target_list_buf;
@@ -2073,7 +2091,12 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
if (rc != SPI_OK_REL_REGISTER)
elog(ERROR, "SPI_register failed");
- apply_old_delta(matviewname, OLD_DELTA_ENRNAME, keys);
+ if (use_count)
+ /* apply old delta and get rows to be recalculated */
+ apply_old_delta_with_count(matviewname, OLD_DELTA_ENRNAME,
+ keys, count_colname);
+ else
+ apply_old_delta(matviewname, OLD_DELTA_ENRNAME, keys);
}
/* For tuple insertion */
@@ -2095,7 +2118,11 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
elog(ERROR, "SPI_register failed");
/* apply new delta */
- apply_new_delta(matviewname, NEW_DELTA_ENRNAME, &target_list_buf);
+ if (use_count)
+ apply_new_delta_with_count(matviewname, NEW_DELTA_ENRNAME,
+ keys, &target_list_buf, count_colname);
+ else
+ apply_new_delta(matviewname, NEW_DELTA_ENRNAME, &target_list_buf);
}
/* We're done maintaining the materialized view. */
@@ -2108,6 +2135,51 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
elog(ERROR, "SPI_finish failed");
}
+/*
+ * apply_old_delta_with_count
+ *
+ * Execute a query for applying a delta table given by deltname_old
+ * which contains tuples to be deleted from to a materialized view given by
+ * matviewname. This is used when counting is required, that is, the view
+ * has aggregate or distinct.
+ */
+static void
+apply_old_delta_with_count(const char *matviewname, const char *deltaname_old,
+ List *keys, const char *count_colname)
+{
+ StringInfoData querybuf;
+ char *match_cond;
+
+ /* build WHERE condition for searching tuples to be deleted */
+ match_cond = get_matching_condition_string(keys);
+
+ /* Search for matching tuples from the view and update or delete if found. */
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH t AS (" /* collecting tid of target tuples in the view */
+ "SELECT diff.%s, " /* count column */
+ "(diff.%s OPERATOR(pg_catalog.=) mv.%s) AS for_dlt, "
+ "mv.ctid "
+ "FROM %s AS mv, %s AS diff "
+ "WHERE %s" /* tuple matching condition */
+ "), updt AS (" /* update a tuple if this is not to be deleted */
+ "UPDATE %s AS mv SET %s = mv.%s OPERATOR(pg_catalog.-) t.%s "
+ "FROM t WHERE mv.ctid OPERATOR(pg_catalog.=) t.ctid AND NOT for_dlt "
+ ")"
+ /* delete a tuple if this is to be deleted */
+ "DELETE FROM %s AS mv USING t "
+ "WHERE mv.ctid OPERATOR(pg_catalog.=) t.ctid AND for_dlt",
+ count_colname,
+ count_colname, count_colname,
+ matviewname, deltaname_old,
+ match_cond,
+ matviewname, count_colname, count_colname, count_colname,
+ matviewname);
+
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_DELETE)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+}
+
/*
* apply_old_delta
*
@@ -2157,6 +2229,66 @@ apply_old_delta(const char *matviewname, const char *deltaname_old,
elog(ERROR, "SPI_exec failed: %s", querybuf.data);
}
+/*
+ * apply_new_delta_with_count
+ *
+ * Execute a query for applying a delta table given by deltname_new
+ * which contains tuples to be inserted into a materialized view given by
+ * matviewname. This is used when counting is required, that is, the view
+ * has aggregate or distinct. Also, when a table in EXISTS sub queries
+ * is modified.
+ */
+static void
+apply_new_delta_with_count(const char *matviewname, const char* deltaname_new,
+ List *keys, StringInfo target_list, const char* count_colname)
+{
+ StringInfoData querybuf;
+ StringInfoData returning_keys;
+ ListCell *lc;
+ char *match_cond = "";
+
+ /* build WHERE condition for searching tuples to be updated */
+ match_cond = get_matching_condition_string(keys);
+
+ /* build string of keys list */
+ initStringInfo(&returning_keys);
+ if (keys)
+ {
+ foreach (lc, keys)
+ {
+ Form_pg_attribute attr = (Form_pg_attribute) lfirst(lc);
+ char *resname = NameStr(attr->attname);
+ appendStringInfo(&returning_keys, "%s", quote_qualified_identifier("mv", resname));
+ if (lnext(keys, lc))
+ appendStringInfo(&returning_keys, ", ");
+ }
+ }
+ else
+ appendStringInfo(&returning_keys, "NULL");
+
+ /* Search for matching tuples from the view and update if found or insert if not. */
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH updt AS (" /* update a tuple if this exists in the view */
+ "UPDATE %s AS mv SET %s = mv.%s OPERATOR(pg_catalog.+) diff.%s "
+ "FROM %s AS diff "
+ "WHERE %s " /* tuple matching condition */
+ "RETURNING %s" /* returning keys of updated tuples */
+ ") INSERT INTO %s (%s) " /* insert a new tuple if this doesn't exist */
+ "SELECT %s FROM %s AS diff "
+ "WHERE NOT EXISTS (SELECT 1 FROM updt AS mv WHERE %s);",
+ matviewname, count_colname, count_colname, count_colname,
+ deltaname_new,
+ match_cond,
+ returning_keys.data,
+ matviewname, target_list->data,
+ target_list->data, deltaname_new,
+ match_cond);
+
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+}
+
/*
* apply_new_delta
*
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 47c900445c..adbd768e0d 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -53,6 +53,7 @@
#include "commands/cluster.h"
#include "commands/comment.h"
#include "commands/defrem.h"
+#include "commands/matview.h"
#include "commands/event_trigger.h"
#include "commands/policy.h"
#include "commands/sequence.h"
@@ -3673,6 +3674,14 @@ renameatt_internal(Oid myrelid,
targetrelation = relation_open(myrelid, AccessExclusiveLock);
renameatt_check(myrelid, RelationGetForm(targetrelation), recursing);
+ /*
+ * Don't rename IVM columns.
+ */
+ if (RelationIsIVM(targetrelation) && isIvmName(oldattname))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("IVM column can not be renamed")));
+
/*
* if the 'recurse' flag is set then we are supposed to rename this
* attribute in all classes that inherit from 'relname' (as well as in
diff --git a/src/backend/parser/parse_relation.c b/src/backend/parser/parse_relation.c
index 864ea9b0d5..c257440414 100644
--- a/src/backend/parser/parse_relation.c
+++ b/src/backend/parser/parse_relation.c
@@ -36,6 +36,7 @@
#include "utils/rel.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
+#include "commands/matview.h"
/*
@@ -97,7 +98,7 @@ static void expandTupleDesc(TupleDesc tupdesc, Alias *eref,
int count, int offset,
int rtindex, int sublevels_up,
int location, bool include_dropped,
- List **colnames, List **colvars);
+ List **colnames, List **colvars, bool is_ivm);
static int specialAttNum(const char *attname);
static bool rte_visible_if_lateral(ParseState *pstate, RangeTblEntry *rte);
static bool rte_visible_if_qualified(ParseState *pstate, RangeTblEntry *rte);
@@ -1502,6 +1503,7 @@ addRangeTableEntry(ParseState *pstate,
rte->relid = RelationGetRelid(rel);
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -1587,6 +1589,7 @@ addRangeTableEntryForRelation(ParseState *pstate,
rte->relid = RelationGetRelid(rel);
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -2758,7 +2761,7 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
expandTupleDesc(tupdesc, rte->eref,
rtfunc->funccolcount, atts_done,
rtindex, sublevels_up, location,
- include_dropped, colnames, colvars);
+ include_dropped, colnames, colvars, false);
}
else if (functypclass == TYPEFUNC_SCALAR)
{
@@ -3026,7 +3029,7 @@ expandRelation(Oid relid, Alias *eref, int rtindex, int sublevels_up,
expandTupleDesc(rel->rd_att, eref, rel->rd_att->natts, 0,
rtindex, sublevels_up,
location, include_dropped,
- colnames, colvars);
+ colnames, colvars, RelationIsIVM(rel));
relation_close(rel, AccessShareLock);
}
@@ -3043,7 +3046,7 @@ static void
expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
int rtindex, int sublevels_up,
int location, bool include_dropped,
- List **colnames, List **colvars)
+ List **colnames, List **colvars, bool is_ivm)
{
ListCell *aliascell;
int varattno;
@@ -3056,6 +3059,9 @@ expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
{
Form_pg_attribute attr = TupleDescAttr(tupdesc, varattno);
+ if (is_ivm && isIvmName(NameStr(attr->attname)) && !MatViewIncrementalMaintenanceIsEnabled())
+ continue;
+
if (attr->attisdropped)
{
if (include_dropped)
@@ -3218,6 +3224,10 @@ expandNSItemAttrs(ParseState *pstate, ParseNamespaceItem *nsitem,
Var *varnode = (Var *) lfirst(var);
TargetEntry *te;
+ /* if transform * into columnlist with IMMV, remove IVM columns */
+ if (rte->relisivm && isIvmName(label) && !MatViewIncrementalMaintenanceIsEnabled())
+ continue;
+
te = makeTargetEntry((Expr *) varnode,
(AttrNumber) pstate->p_next_resno++,
label,
diff --git a/src/backend/rewrite/rewriteDefine.c b/src/backend/rewrite/rewriteDefine.c
index e36fc72e1e..f6dc7ba202 100644
--- a/src/backend/rewrite/rewriteDefine.c
+++ b/src/backend/rewrite/rewriteDefine.c
@@ -621,7 +621,8 @@ checkRuleResultList(List *targetList, TupleDesc resultDesc, bool isSelect,
attr->atttypmod))));
}
- if (i != resultDesc->natts)
+ /* No check for materialized views since this could have special columns for IVM */
+ if ((!isSelect || requireColumnNameMatch) && i != resultDesc->natts)
ereport(ERROR,
(errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
isSelect ?
diff --git a/src/include/commands/createas.h b/src/include/commands/createas.h
index 09a64fa2e5..76a7873ebf 100644
--- a/src/include/commands/createas.h
+++ b/src/include/commands/createas.h
@@ -29,6 +29,8 @@ extern ObjectAddress ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *st
extern void CreateIvmTriggersOnBaseTables(Query *qry, Oid matviewOid);
extern void CreateIndexOnIMMV(Query *query, Relation matviewRel);
+extern Query *rewriteQueryForIMMV(Query *query, List *colNames);
+
extern int GetIntoRelEFlags(IntoClause *intoClause);
extern DestReceiver *CreateIntoRelDestReceiver(IntoClause *intoClause);
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index fef4c714b8..1a2b8fa09e 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -1073,6 +1073,7 @@ typedef struct RangeTblEntry
int rellockmode; /* lock level that query requires on the rel */
struct TableSampleClause *tablesample; /* sampling info, or NULL */
Index perminfoindex;
+ bool relisivm;
/*
* Fields valid for a subquery RTE (else NULL):
--
2.25.1
--Multipart=_Mon__28_Aug_2023_11_52_52_+0900_hj6L5h176QaSGtg7
Content-Type: text/x-diff;
name="v29-0008-Add-aggregates-support-in-IVM.patch"
Content-Disposition: attachment;
filename="v29-0008-Add-aggregates-support-in-IVM.patch"
Content-Transfer-Encoding: quoted-printable
^ permalink raw reply [nested|flat] 24+ messages in thread
* [PATCH v30 07/11] Add DISTINCT support for IVM
@ 2023-05-31 10:08 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 24+ messages in thread
From: Yugo Nagata @ 2023-05-31 10:08 UTC (permalink / raw)
When IMMV is created with DISTINCT, multiplicity of tuples is
counted and stored in "__ivm_count__" column, which is a hidden
column of IMMV. The value in __ivm_count__ is updated when IMMV
is maintained incrementally. A tuple in IMMV can be removed if
and only if the count becomes zero.
---
src/backend/commands/createas.c | 141 ++++++++++++++++++++------
src/backend/commands/indexcmds.c | 40 ++++++++
src/backend/commands/matview.c | 148 ++++++++++++++++++++++++++--
src/backend/commands/tablecmds.c | 9 ++
src/backend/parser/parse_relation.c | 18 +++-
src/backend/rewrite/rewriteDefine.c | 3 +-
src/include/commands/createas.h | 2 +
src/include/nodes/parsenodes.h | 1 +
8 files changed, 317 insertions(+), 45 deletions(-)
diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c
index 08262100ea..cce44278fa 100644
--- a/src/backend/commands/createas.c
+++ b/src/backend/commands/createas.c
@@ -53,6 +53,7 @@
#include "parser/parser.h"
#include "parser/parsetree.h"
#include "parser/parse_clause.h"
+#include "parser/parse_func.h"
#include "rewrite/rewriteHandler.h"
#include "storage/smgr.h"
#include "tcop/tcopprot.h"
@@ -309,6 +310,9 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt,
errhint("functions must be marked IMMUTABLE")));
check_ivm_restriction((Node *) query);
+
+ /* For IMMV, we need to rewrite matview query */
+ query = rewriteQueryForIMMV(query, into->colNames);
}
if (into->skipData)
@@ -413,6 +417,49 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt,
return address;
}
+/*
+ * rewriteQueryForIMMV -- rewrite view definition query for IMMV
+ *
+ * count(*) is added for counting distinct tuples in views.
+ */
+Query *
+rewriteQueryForIMMV(Query *query, List *colNames)
+{
+ Query *rewritten;
+
+ Node *node;
+ ParseState *pstate = make_parsestate(NULL);
+ FuncCall *fn;
+
+ rewritten = copyObject(query);
+ pstate->p_expr_kind = EXPR_KIND_SELECT_TARGET;
+
+ /*
+ * Convert DISTINCT to GROUP BY and add count(*) for counting distinct
+ * tuples in views.
+ */
+ if (rewritten->distinctClause)
+ {
+ TargetEntry *tle;
+
+ rewritten->groupClause = transformDistinctClause(NULL, &rewritten->targetList, rewritten->sortClause, false);
+
+ fn = makeFuncCall(SystemFuncName("count"), NIL, COERCE_EXPLICIT_CALL, -1);
+ fn->agg_star = true;
+
+ node = ParseFuncOrColumn(pstate, fn->funcname, NIL, NULL, fn, false, -1);
+
+ tle = makeTargetEntry((Expr *) node,
+ list_length(rewritten->targetList) + 1,
+ pstrdup("__ivm_count__"),
+ false);
+ rewritten->targetList = lappend(rewritten->targetList, tle);
+ rewritten->hasAggs = true;
+ }
+
+ return rewritten;
+}
+
/*
* GetIntoRelEFlags --- compute executor flags needed for CREATE TABLE AS
*
@@ -536,7 +583,8 @@ intorel_startup(DestReceiver *self, int operation, TupleDesc typeinfo)
ColumnDef *col;
char *colname;
- if (lc)
+ /* Don't override hidden columns added for IVM */
+ if (lc && !isIvmName(NameStr(attribute->attname)))
{
colname = strVal(lfirst(lc));
lc = lnext(into->colNames, lc);
@@ -940,10 +988,6 @@ check_ivm_restriction_walker(Node *node, void *context)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("LIMIT/OFFSET clause is not supported on incrementally maintainable materialized view")));
- if (qry->distinctClause)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("DISTINCT is not supported on incrementally maintainable materialized view")));
if (qry->hasDistinctOn)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -1090,12 +1134,18 @@ CreateIndexOnIMMV(Query *query, Relation matviewRel)
char idxname[NAMEDATALEN];
List *indexoidlist = RelationGetIndexList(matviewRel);
ListCell *indexoidscan;
- Bitmapset *key_attnos;
snprintf(idxname, sizeof(idxname), "%s_index", RelationGetRelationName(matviewRel));
index = makeNode(IndexStmt);
+ /*
+ * We consider null values not distinct to make sure that views with DISTINCT
+ * or GROUP BY don't contain multiple NULL rows when NULL is inserted to
+ * a base table concurrently.
+ */
+ index->nulls_not_distinct = true;
+
index->unique = true;
index->primary = false;
index->isconstraint = false;
@@ -1122,41 +1172,68 @@ CreateIndexOnIMMV(Query *query, Relation matviewRel)
index->concurrent = false;
index->if_not_exists = false;
- /* create index on the base tables' primary key columns */
- key_attnos = get_primary_key_attnos_from_query(query, &constraintList);
- if (key_attnos)
+ if (query->distinctClause)
{
+ /* create unique constraint on all columns */
foreach(lc, query->targetList)
{
TargetEntry *tle = (TargetEntry *) lfirst(lc);
Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, tle->resno - 1);
-
- if (bms_is_member(tle->resno - FirstLowInvalidHeapAttributeNumber, key_attnos))
- {
- IndexElem *iparam;
-
- iparam = makeNode(IndexElem);
- iparam->name = pstrdup(NameStr(attr->attname));
- iparam->expr = NULL;
- iparam->indexcolname = NULL;
- iparam->collation = NIL;
- iparam->opclass = NIL;
- iparam->opclassopts = NIL;
- iparam->ordering = SORTBY_DEFAULT;
- iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
- index->indexParams = lappend(index->indexParams, iparam);
- }
+ IndexElem *iparam;
+
+ iparam = makeNode(IndexElem);
+ iparam->name = pstrdup(NameStr(attr->attname));
+ iparam->expr = NULL;
+ iparam->indexcolname = NULL;
+ iparam->collation = NIL;
+ iparam->opclass = NIL;
+ iparam->opclassopts = NIL;
+ iparam->ordering = SORTBY_DEFAULT;
+ iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
+ index->indexParams = lappend(index->indexParams, iparam);
}
}
else
{
- /* create no index, just notice that an appropriate index is necessary for efficient IVM */
- ereport(NOTICE,
- (errmsg("could not create an index on materialized view \"%s\" automatically",
- RelationGetRelationName(matviewRel)),
- errdetail("This target list does not have all the primary key columns. "),
- errhint("Create an index on the materialized view for efficient incremental maintenance.")));
- return;
+ Bitmapset *key_attnos;
+
+ /* create index on the base tables' primary key columns */
+ key_attnos = get_primary_key_attnos_from_query(query, &constraintList);
+ if (key_attnos)
+ {
+ foreach(lc, query->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);
+ Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, tle->resno - 1);
+
+ if (bms_is_member(tle->resno - FirstLowInvalidHeapAttributeNumber, key_attnos))
+ {
+ IndexElem *iparam;
+
+ iparam = makeNode(IndexElem);
+ iparam->name = pstrdup(NameStr(attr->attname));
+ iparam->expr = NULL;
+ iparam->indexcolname = NULL;
+ iparam->collation = NIL;
+ iparam->opclass = NIL;
+ iparam->opclassopts = NIL;
+ iparam->ordering = SORTBY_DEFAULT;
+ iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
+ index->indexParams = lappend(index->indexParams, iparam);
+ }
+ }
+ }
+ else
+ {
+ /* create no index, just notice that an appropriate index is necessary for efficient IVM */
+ ereport(NOTICE,
+ (errmsg("could not create an index on materialized view \"%s\" automatically",
+ RelationGetRelationName(matviewRel)),
+ errdetail("This target list does not have all the primary key columns, "
+ "or this view does not contain DISTINCT clause."),
+ errhint("Create an index on the materialized view for efficient incremental maintenance.")));
+ return;
+ }
}
/* If we have a compatible index, we don't need to create another. */
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 7a87626f5f..d661fbbb48 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -39,6 +39,7 @@
#include "commands/dbcommands.h"
#include "commands/defrem.h"
#include "commands/event_trigger.h"
+#include "commands/matview.h"
#include "commands/progress.h"
#include "commands/tablecmds.h"
#include "commands/tablespace.h"
@@ -1129,6 +1130,45 @@ DefineIndex(Oid tableId,
safe_index = indexInfo->ii_Expressions == NIL &&
indexInfo->ii_Predicate == NIL;
+ /*
+ * We disallow unique indexes on IVM columns of IMMVs.
+ */
+ if (RelationIsIVM(rel) && stmt->unique)
+ {
+ for (int i = 0; i < indexInfo->ii_NumIndexKeyAttrs; i++)
+ {
+ AttrNumber attno = indexInfo->ii_IndexAttrNumbers[i];
+ if (attno > 0)
+ {
+ char *name = NameStr(TupleDescAttr(rel->rd_att, attno - 1)->attname);
+ if (name && isIvmName(name))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("unique index creation on IVM columns is not supported")));
+ }
+ }
+
+ if (indexInfo->ii_Expressions)
+ {
+ Bitmapset *indexattrs = NULL;
+ int varno = -1;
+
+ pull_varattnos((Node *) indexInfo->ii_Expressions, 1, &indexattrs);
+
+ while ((varno = bms_next_member(indexattrs, varno)) >= 0)
+ {
+ int attno = varno + FirstLowInvalidHeapAttributeNumber;
+ char *name = NameStr(TupleDescAttr(rel->rd_att, attno - 1)->attname);
+ if (name && isIvmName(name))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("unique index creation on IVM columns is not supported")));
+ }
+
+ }
+ }
+
+
/*
* Report index creation if appropriate (delay this till after most of the
* error checks)
diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c
index a821992a37..dbcbc79fff 100644
--- a/src/backend/commands/matview.c
+++ b/src/backend/commands/matview.c
@@ -152,11 +152,15 @@ static Query *rewrite_query_for_postupdate_state(Query *query, MV_TriggerTable *
static void apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *new_tuplestores,
TupleDesc tupdesc_old, TupleDesc tupdesc_new,
- Query *query);
+ Query *query, bool use_count, char *count_colname);
static void apply_old_delta(const char *matviewname, const char *deltaname_old,
List *keys);
+static void apply_old_delta_with_count(const char *matviewname, const char *deltaname_old,
+ List *keys, const char *count_colname);
static void apply_new_delta(const char *matviewname, const char *deltaname_new,
StringInfo target_list);
+static void apply_new_delta_with_count(const char *matviewname, const char* deltaname_new,
+ List *keys, StringInfo target_list, const char* count_colname);
static char *get_matching_condition_string(List *keys);
static void generate_equal(StringInfo querybuf, Oid opttype,
const char *leftop, const char *rightop);
@@ -271,6 +275,7 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
Oid matviewOid;
Relation matviewRel;
Query *dataQuery;
+ Query *viewQuery;
Oid tableSpace;
Oid relowner;
Oid OIDNewHeap;
@@ -330,8 +335,13 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
"CONCURRENTLY", "WITH NO DATA")));
- dataQuery = get_matview_query(matviewRel);
+ viewQuery = get_matview_query(matviewRel);
+ /* For IMMV, we need to rewrite matview query */
+ if (!stmt->skipData && RelationIsIVM(matviewRel))
+ dataQuery = rewriteQueryForIMMV(viewQuery,NIL);
+ else
+ dataQuery = viewQuery;
/*
* Check that there is a unique index with no WHERE clause on one or more
@@ -511,8 +521,8 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
if (!stmt->skipData && RelationIsIVM(matviewRel) && !oldPopulated)
{
- CreateIndexOnIMMV(dataQuery, matviewRel);
- CreateIvmTriggersOnBaseTables(dataQuery, matviewOid);
+ CreateIndexOnIMMV(viewQuery, matviewRel);
+ CreateIvmTriggersOnBaseTables(viewQuery, matviewOid);
}
table_close(matviewRel, NoLock);
@@ -1534,6 +1544,13 @@ IVM_immediate_maintenance(PG_FUNCTION_ARGS)
int rte_index = lfirst_int(lc2);
TupleDesc tupdesc_old;
TupleDesc tupdesc_new;
+ bool use_count = false;
+ char *count_colname = NULL;
+
+ count_colname = pstrdup("__ivm_count__");
+
+ if (query->distinctClause)
+ use_count = true;
/* calculate delta tables */
calc_delta(table, rte_index, rewritten, dest_old, dest_new,
@@ -1546,7 +1563,8 @@ IVM_immediate_maintenance(PG_FUNCTION_ARGS)
{
/* apply the delta tables to the materialized view */
apply_delta(matviewOid, old_tuplestore, new_tuplestore,
- tupdesc_old, tupdesc_new, query);
+ tupdesc_old, tupdesc_new, query, use_count,
+ count_colname);
}
PG_CATCH();
{
@@ -2019,7 +2037,7 @@ rewrite_query_for_postupdate_state(Query *query, MV_TriggerTable *table, int rte
static void
apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *new_tuplestores,
TupleDesc tupdesc_old, TupleDesc tupdesc_new,
- Query *query)
+ Query *query, bool use_count, char *count_colname)
{
StringInfoData querybuf;
StringInfoData target_list_buf;
@@ -2095,7 +2113,12 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
if (rc != SPI_OK_REL_REGISTER)
elog(ERROR, "SPI_register failed");
- apply_old_delta(matviewname, OLD_DELTA_ENRNAME, keys);
+ if (use_count)
+ /* apply old delta and get rows to be recalculated */
+ apply_old_delta_with_count(matviewname, OLD_DELTA_ENRNAME,
+ keys, count_colname);
+ else
+ apply_old_delta(matviewname, OLD_DELTA_ENRNAME, keys);
}
/* For tuple insertion */
@@ -2117,7 +2140,11 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
elog(ERROR, "SPI_register failed");
/* apply new delta */
- apply_new_delta(matviewname, NEW_DELTA_ENRNAME, &target_list_buf);
+ if (use_count)
+ apply_new_delta_with_count(matviewname, NEW_DELTA_ENRNAME,
+ keys, &target_list_buf, count_colname);
+ else
+ apply_new_delta(matviewname, NEW_DELTA_ENRNAME, &target_list_buf);
}
/* We're done maintaining the materialized view. */
@@ -2130,6 +2157,51 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
elog(ERROR, "SPI_finish failed");
}
+/*
+ * apply_old_delta_with_count
+ *
+ * Execute a query for applying a delta table given by deltname_old
+ * which contains tuples to be deleted from to a materialized view given by
+ * matviewname. This is used when counting is required, that is, the view
+ * has aggregate or distinct.
+ */
+static void
+apply_old_delta_with_count(const char *matviewname, const char *deltaname_old,
+ List *keys, const char *count_colname)
+{
+ StringInfoData querybuf;
+ char *match_cond;
+
+ /* build WHERE condition for searching tuples to be deleted */
+ match_cond = get_matching_condition_string(keys);
+
+ /* Search for matching tuples from the view and update or delete if found. */
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH t AS (" /* collecting tid of target tuples in the view */
+ "SELECT diff.%s, " /* count column */
+ "(diff.%s OPERATOR(pg_catalog.=) mv.%s) AS for_dlt, "
+ "mv.ctid "
+ "FROM %s AS mv, %s AS diff "
+ "WHERE %s" /* tuple matching condition */
+ "), updt AS (" /* update a tuple if this is not to be deleted */
+ "UPDATE %s AS mv SET %s = mv.%s OPERATOR(pg_catalog.-) t.%s "
+ "FROM t WHERE mv.ctid OPERATOR(pg_catalog.=) t.ctid AND NOT for_dlt "
+ ")"
+ /* delete a tuple if this is to be deleted */
+ "DELETE FROM %s AS mv USING t "
+ "WHERE mv.ctid OPERATOR(pg_catalog.=) t.ctid AND for_dlt",
+ count_colname,
+ count_colname, count_colname,
+ matviewname, deltaname_old,
+ match_cond,
+ matviewname, count_colname, count_colname, count_colname,
+ matviewname);
+
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_DELETE)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+}
+
/*
* apply_old_delta
*
@@ -2179,6 +2251,66 @@ apply_old_delta(const char *matviewname, const char *deltaname_old,
elog(ERROR, "SPI_exec failed: %s", querybuf.data);
}
+/*
+ * apply_new_delta_with_count
+ *
+ * Execute a query for applying a delta table given by deltname_new
+ * which contains tuples to be inserted into a materialized view given by
+ * matviewname. This is used when counting is required, that is, the view
+ * has aggregate or distinct. Also, when a table in EXISTS sub queries
+ * is modified.
+ */
+static void
+apply_new_delta_with_count(const char *matviewname, const char* deltaname_new,
+ List *keys, StringInfo target_list, const char* count_colname)
+{
+ StringInfoData querybuf;
+ StringInfoData returning_keys;
+ ListCell *lc;
+ char *match_cond = "";
+
+ /* build WHERE condition for searching tuples to be updated */
+ match_cond = get_matching_condition_string(keys);
+
+ /* build string of keys list */
+ initStringInfo(&returning_keys);
+ if (keys)
+ {
+ foreach (lc, keys)
+ {
+ Form_pg_attribute attr = (Form_pg_attribute) lfirst(lc);
+ char *resname = NameStr(attr->attname);
+ appendStringInfo(&returning_keys, "%s", quote_qualified_identifier("mv", resname));
+ if (lnext(keys, lc))
+ appendStringInfo(&returning_keys, ", ");
+ }
+ }
+ else
+ appendStringInfo(&returning_keys, "NULL");
+
+ /* Search for matching tuples from the view and update if found or insert if not. */
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH updt AS (" /* update a tuple if this exists in the view */
+ "UPDATE %s AS mv SET %s = mv.%s OPERATOR(pg_catalog.+) diff.%s "
+ "FROM %s AS diff "
+ "WHERE %s " /* tuple matching condition */
+ "RETURNING %s" /* returning keys of updated tuples */
+ ") INSERT INTO %s (%s) " /* insert a new tuple if this doesn't exist */
+ "SELECT %s FROM %s AS diff "
+ "WHERE NOT EXISTS (SELECT 1 FROM updt AS mv WHERE %s);",
+ matviewname, count_colname, count_colname, count_colname,
+ deltaname_new,
+ match_cond,
+ returning_keys.data,
+ matviewname, target_list->data,
+ target_list->data, deltaname_new,
+ match_cond);
+
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+}
+
/*
* apply_new_delta
*
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index f798794556..02a688acb7 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -53,6 +53,7 @@
#include "commands/cluster.h"
#include "commands/comment.h"
#include "commands/defrem.h"
+#include "commands/matview.h"
#include "commands/event_trigger.h"
#include "commands/policy.h"
#include "commands/sequence.h"
@@ -3784,6 +3785,14 @@ renameatt_internal(Oid myrelid,
targetrelation = relation_open(myrelid, AccessExclusiveLock);
renameatt_check(myrelid, RelationGetForm(targetrelation), recursing);
+ /*
+ * Don't rename IVM columns.
+ */
+ if (RelationIsIVM(targetrelation) && isIvmName(oldattname))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("IVM column can not be renamed")));
+
/*
* if the 'recurse' flag is set then we are supposed to rename this
* attribute in all classes that inherit from 'relname' (as well as in
diff --git a/src/backend/parser/parse_relation.c b/src/backend/parser/parse_relation.c
index 34a0ec5901..d5b45c587d 100644
--- a/src/backend/parser/parse_relation.c
+++ b/src/backend/parser/parse_relation.c
@@ -36,6 +36,7 @@
#include "utils/rel.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
+#include "commands/matview.h"
/*
@@ -97,7 +98,7 @@ static void expandTupleDesc(TupleDesc tupdesc, Alias *eref,
int count, int offset,
int rtindex, int sublevels_up,
int location, bool include_dropped,
- List **colnames, List **colvars);
+ List **colnames, List **colvars, bool is_ivm);
static int specialAttNum(const char *attname);
static bool rte_visible_if_lateral(ParseState *pstate, RangeTblEntry *rte);
static bool rte_visible_if_qualified(ParseState *pstate, RangeTblEntry *rte);
@@ -1502,6 +1503,7 @@ addRangeTableEntry(ParseState *pstate,
rte->relid = RelationGetRelid(rel);
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -1587,6 +1589,7 @@ addRangeTableEntryForRelation(ParseState *pstate,
rte->relid = RelationGetRelid(rel);
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -2758,7 +2761,7 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
expandTupleDesc(tupdesc, rte->eref,
rtfunc->funccolcount, atts_done,
rtindex, sublevels_up, location,
- include_dropped, colnames, colvars);
+ include_dropped, colnames, colvars, false);
}
else if (functypclass == TYPEFUNC_SCALAR)
{
@@ -3026,7 +3029,7 @@ expandRelation(Oid relid, Alias *eref, int rtindex, int sublevels_up,
expandTupleDesc(rel->rd_att, eref, rel->rd_att->natts, 0,
rtindex, sublevels_up,
location, include_dropped,
- colnames, colvars);
+ colnames, colvars, RelationIsIVM(rel));
relation_close(rel, AccessShareLock);
}
@@ -3043,7 +3046,7 @@ static void
expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
int rtindex, int sublevels_up,
int location, bool include_dropped,
- List **colnames, List **colvars)
+ List **colnames, List **colvars, bool is_ivm)
{
ListCell *aliascell;
int varattno;
@@ -3056,6 +3059,9 @@ expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
{
Form_pg_attribute attr = TupleDescAttr(tupdesc, varattno);
+ if (is_ivm && isIvmName(NameStr(attr->attname)) && !MatViewIncrementalMaintenanceIsEnabled())
+ continue;
+
if (attr->attisdropped)
{
if (include_dropped)
@@ -3218,6 +3224,10 @@ expandNSItemAttrs(ParseState *pstate, ParseNamespaceItem *nsitem,
Var *varnode = (Var *) lfirst(var);
TargetEntry *te;
+ /* if transform * into columnlist with IMMV, remove IVM columns */
+ if (rte->relisivm && isIvmName(label) && !MatViewIncrementalMaintenanceIsEnabled())
+ continue;
+
te = makeTargetEntry((Expr *) varnode,
(AttrNumber) pstate->p_next_resno++,
label,
diff --git a/src/backend/rewrite/rewriteDefine.c b/src/backend/rewrite/rewriteDefine.c
index b449244a53..b9d161d6a3 100644
--- a/src/backend/rewrite/rewriteDefine.c
+++ b/src/backend/rewrite/rewriteDefine.c
@@ -621,7 +621,8 @@ checkRuleResultList(List *targetList, TupleDesc resultDesc, bool isSelect,
attr->atttypmod))));
}
- if (i != resultDesc->natts)
+ /* No check for materialized views since this could have special columns for IVM */
+ if ((!isSelect || requireColumnNameMatch) && i != resultDesc->natts)
ereport(ERROR,
(errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
isSelect ?
diff --git a/src/include/commands/createas.h b/src/include/commands/createas.h
index 396ad1bb4c..6b47e66bfd 100644
--- a/src/include/commands/createas.h
+++ b/src/include/commands/createas.h
@@ -29,6 +29,8 @@ extern ObjectAddress ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *st
extern void CreateIvmTriggersOnBaseTables(Query *qry, Oid matviewOid);
extern void CreateIndexOnIMMV(Query *query, Relation matviewRel);
+extern Query *rewriteQueryForIMMV(Query *query, List *colNames);
+
extern int GetIntoRelEFlags(IntoClause *intoClause);
extern DestReceiver *CreateIntoRelDestReceiver(IntoClause *intoClause);
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index d60e148ff2..fa3a95897c 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -1074,6 +1074,7 @@ typedef struct RangeTblEntry
int rellockmode; /* lock level that query requires on the rel */
struct TableSampleClause *tablesample; /* sampling info, or NULL */
Index perminfoindex;
+ bool relisivm;
/*
* Fields valid for a subquery RTE (else NULL):
--
2.25.1
--Multipart=_Mon__4_Mar_2024_11_58_46_+0900_UaponF/qQhQrVCFt
Content-Type: text/x-diff;
name="v30-0008-Add-aggregates-support-in-IVM.patch"
Content-Disposition: attachment;
filename="v30-0008-Add-aggregates-support-in-IVM.patch"
Content-Transfer-Encoding: quoted-printable
^ permalink raw reply [nested|flat] 24+ messages in thread
* [PATCH v37 07/11] Add DISTINCT support for IVM
@ 2023-05-31 10:08 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 24+ messages in thread
From: Yugo Nagata @ 2023-05-31 10:08 UTC (permalink / raw)
When IMMV is created with DISTINCT, multiplicity of tuples is
counted and stored in "__ivm_count__" column, which is a hidden
column of IMMV. The value in __ivm_count__ is updated when IMMV
is maintained incrementally. A tuple in IMMV can be removed if
and only if the count becomes zero.
---
src/backend/commands/createas.c | 141 +++++++++++++++++++++------
src/backend/commands/indexcmds.c | 40 ++++++++
src/backend/commands/matview.c | 145 ++++++++++++++++++++++++++--
src/backend/commands/tablecmds.c | 9 ++
src/backend/nodes/outfuncs.c | 1 +
src/backend/nodes/readfuncs.c | 1 +
src/backend/parser/parse_relation.c | 18 +++-
src/backend/rewrite/rewriteDefine.c | 3 +-
src/include/commands/createas.h | 2 +
src/include/nodes/parsenodes.h | 2 +
10 files changed, 319 insertions(+), 43 deletions(-)
diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c
index a499688b79d..cd8db0059f9 100644
--- a/src/backend/commands/createas.c
+++ b/src/backend/commands/createas.c
@@ -55,6 +55,7 @@
#include "parser/parser.h"
#include "parser/parsetree.h"
#include "parser/parse_clause.h"
+#include "parser/parse_func.h"
#include "rewrite/rewriteHandler.h"
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
@@ -307,6 +308,9 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt,
errhint("functions must be marked IMMUTABLE")));
check_ivm_restriction((Node *) query);
+
+ /* For IMMV, we need to rewrite matview query */
+ query = rewriteQueryForIMMV(query, into->colNames);
}
if (into->skipData)
@@ -416,6 +420,49 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt,
return address;
}
+/*
+ * rewriteQueryForIMMV -- rewrite view definition query for IMMV
+ *
+ * count(*) is added for counting distinct tuples in views.
+ */
+Query *
+rewriteQueryForIMMV(Query *query, List *colNames)
+{
+ Query *rewritten;
+
+ Node *node;
+ ParseState *pstate = make_parsestate(NULL);
+ FuncCall *fn;
+
+ rewritten = copyObject(query);
+ pstate->p_expr_kind = EXPR_KIND_SELECT_TARGET;
+
+ /*
+ * Convert DISTINCT to GROUP BY and add count(*) for counting distinct
+ * tuples in views.
+ */
+ if (rewritten->distinctClause)
+ {
+ TargetEntry *tle;
+
+ rewritten->groupClause = transformDistinctClause(NULL, &rewritten->targetList, rewritten->sortClause, false);
+
+ fn = makeFuncCall(SystemFuncName("count"), NIL, COERCE_EXPLICIT_CALL, -1);
+ fn->agg_star = true;
+
+ node = ParseFuncOrColumn(pstate, fn->funcname, NIL, NULL, fn, false, -1);
+
+ tle = makeTargetEntry((Expr *) node,
+ list_length(rewritten->targetList) + 1,
+ pstrdup("__ivm_count__"),
+ false);
+ rewritten->targetList = lappend(rewritten->targetList, tle);
+ rewritten->hasAggs = true;
+ }
+
+ return rewritten;
+}
+
/*
* GetIntoRelEFlags --- compute executor flags needed for CREATE TABLE AS
*
@@ -539,7 +586,8 @@ intorel_startup(DestReceiver *self, int operation, TupleDesc typeinfo)
ColumnDef *col;
char *colname;
- if (lc)
+ /* Don't override hidden columns added for IVM */
+ if (lc && !isIvmName(NameStr(attribute->attname)))
{
colname = strVal(lfirst(lc));
lc = lnext(into->colNames, lc);
@@ -943,10 +991,6 @@ check_ivm_restriction_walker(Node *node, void *context)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("LIMIT/OFFSET clause is not supported on incrementally maintainable materialized view")));
- if (qry->distinctClause)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("DISTINCT is not supported on incrementally maintainable materialized view")));
if (qry->hasDistinctOn)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -1108,12 +1152,18 @@ CreateIndexOnIMMV(Query *query, Relation matviewRel)
char idxname[NAMEDATALEN];
List *indexoidlist = RelationGetIndexList(matviewRel);
ListCell *indexoidscan;
- Bitmapset *key_attnos;
snprintf(idxname, sizeof(idxname), "%s_index", RelationGetRelationName(matviewRel));
index = makeNode(IndexStmt);
+ /*
+ * We consider null values not distinct to make sure that views with DISTINCT
+ * or GROUP BY don't contain multiple NULL rows when NULL is inserted to
+ * a base table concurrently.
+ */
+ index->nulls_not_distinct = true;
+
index->unique = true;
index->primary = false;
index->isconstraint = false;
@@ -1140,41 +1190,68 @@ CreateIndexOnIMMV(Query *query, Relation matviewRel)
index->concurrent = false;
index->if_not_exists = false;
- /* create index on the base tables' primary key columns */
- key_attnos = get_primary_key_attnos_from_query(query, &constraintList);
- if (key_attnos)
+ if (query->distinctClause)
{
+ /* create unique constraint on all columns */
foreach(lc, query->targetList)
{
TargetEntry *tle = (TargetEntry *) lfirst(lc);
Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, tle->resno - 1);
-
- if (bms_is_member(tle->resno - FirstLowInvalidHeapAttributeNumber, key_attnos))
- {
- IndexElem *iparam;
-
- iparam = makeNode(IndexElem);
- iparam->name = pstrdup(NameStr(attr->attname));
- iparam->expr = NULL;
- iparam->indexcolname = NULL;
- iparam->collation = NIL;
- iparam->opclass = NIL;
- iparam->opclassopts = NIL;
- iparam->ordering = SORTBY_DEFAULT;
- iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
- index->indexParams = lappend(index->indexParams, iparam);
- }
+ IndexElem *iparam;
+
+ iparam = makeNode(IndexElem);
+ iparam->name = pstrdup(NameStr(attr->attname));
+ iparam->expr = NULL;
+ iparam->indexcolname = NULL;
+ iparam->collation = NIL;
+ iparam->opclass = NIL;
+ iparam->opclassopts = NIL;
+ iparam->ordering = SORTBY_DEFAULT;
+ iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
+ index->indexParams = lappend(index->indexParams, iparam);
}
}
else
{
- /* create no index, just notice that an appropriate index is necessary for efficient IVM */
- ereport(NOTICE,
- (errmsg("could not create an index on materialized view \"%s\" automatically",
- RelationGetRelationName(matviewRel)),
- errdetail("This target list does not have all the primary key columns. "),
- errhint("Create an index on the materialized view for efficient incremental maintenance.")));
- return;
+ Bitmapset *key_attnos;
+
+ /* create index on the base tables' primary key columns */
+ key_attnos = get_primary_key_attnos_from_query(query, &constraintList);
+ if (key_attnos)
+ {
+ foreach(lc, query->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);
+ Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, tle->resno - 1);
+
+ if (bms_is_member(tle->resno - FirstLowInvalidHeapAttributeNumber, key_attnos))
+ {
+ IndexElem *iparam;
+
+ iparam = makeNode(IndexElem);
+ iparam->name = pstrdup(NameStr(attr->attname));
+ iparam->expr = NULL;
+ iparam->indexcolname = NULL;
+ iparam->collation = NIL;
+ iparam->opclass = NIL;
+ iparam->opclassopts = NIL;
+ iparam->ordering = SORTBY_DEFAULT;
+ iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
+ index->indexParams = lappend(index->indexParams, iparam);
+ }
+ }
+ }
+ else
+ {
+ /* create no index, just notice that an appropriate index is necessary for efficient IVM */
+ ereport(NOTICE,
+ (errmsg("could not create an index on materialized view \"%s\" automatically",
+ RelationGetRelationName(matviewRel)),
+ errdetail("This target list does not have all the primary key columns, "
+ "or this view does not contain DISTINCT clause."),
+ errhint("Create an index on the materialized view for efficient incremental maintenance.")));
+ return;
+ }
}
/* If we have a compatible index, we don't need to create another. */
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 9ab74c8df0a..1f95e6b229f 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -41,6 +41,7 @@
#include "commands/comment.h"
#include "commands/defrem.h"
#include "commands/event_trigger.h"
+#include "commands/matview.h"
#include "commands/progress.h"
#include "commands/tablecmds.h"
#include "commands/tablespace.h"
@@ -1179,6 +1180,45 @@ DefineIndex(ParseState *pstate,
safe_index = indexInfo->ii_Expressions == NIL &&
indexInfo->ii_Predicate == NIL;
+ /*
+ * We disallow unique indexes on IVM columns of IMMVs.
+ */
+ if (RelationIsIVM(rel) && stmt->unique)
+ {
+ for (int i = 0; i < indexInfo->ii_NumIndexKeyAttrs; i++)
+ {
+ AttrNumber attno = indexInfo->ii_IndexAttrNumbers[i];
+ if (attno > 0)
+ {
+ char *name = NameStr(TupleDescAttr(rel->rd_att, attno - 1)->attname);
+ if (name && isIvmName(name))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("unique index creation on IVM columns is not supported")));
+ }
+ }
+
+ if (indexInfo->ii_Expressions)
+ {
+ Bitmapset *indexattrs = NULL;
+ int varno = -1;
+
+ pull_varattnos((Node *) indexInfo->ii_Expressions, 1, &indexattrs);
+
+ while ((varno = bms_next_member(indexattrs, varno)) >= 0)
+ {
+ int attno = varno + FirstLowInvalidHeapAttributeNumber;
+ char *name = NameStr(TupleDescAttr(rel->rd_att, attno - 1)->attname);
+ if (name && isIvmName(name))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("unique index creation on IVM columns is not supported")));
+ }
+
+ }
+ }
+
+
/*
* Report index creation if appropriate (delay this till after most of the
* error checks)
diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c
index eaf39f80cd3..a2746ca9265 100644
--- a/src/backend/commands/matview.c
+++ b/src/backend/commands/matview.c
@@ -193,11 +193,15 @@ static Query *rewrite_query_for_postupdate_state(Query *query, MV_TriggerTable *
static void apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *new_tuplestores,
TupleDesc tupdesc_old, TupleDesc tupdesc_new,
- Query *query);
+ Query *query, bool use_count, char *count_colname);
static void apply_old_delta(const char *matviewname, const char *deltaname_old,
List *keys);
+static void apply_old_delta_with_count(const char *matviewname, const char *deltaname_old,
+ List *keys, const char *count_colname);
static void apply_new_delta(const char *matviewname, const char *deltaname_new,
StringInfo target_list);
+static void apply_new_delta_with_count(const char *matviewname, const char* deltaname_new,
+ List *keys, StringInfo target_list, const char* count_colname);
static char *get_matching_condition_string(List *keys);
static void generate_equal(StringInfo querybuf, Oid opttype,
const char *leftop, const char *rightop);
@@ -346,6 +350,7 @@ RefreshMatViewByOid(Oid matviewOid, bool is_create, bool skipData,
QueryCompletion *qc)
{
Relation matviewRel;
+ Query *viewQuery;
Query *dataQuery;
Oid tableSpace;
Oid relowner;
@@ -394,7 +399,13 @@ RefreshMatViewByOid(Oid matviewOid, bool is_create, bool skipData,
errmsg("%s options %s and %s cannot be used together",
"REFRESH", "CONCURRENTLY", "WITH NO DATA")));
- dataQuery = get_matview_query(matviewRel);
+ viewQuery = get_matview_query(matviewRel);
+
+ /* For IMMV, we need to rewrite matview query */
+ if (!skipData && RelationIsIVM(matviewRel))
+ dataQuery = rewriteQueryForIMMV(viewQuery,NIL);
+ else
+ dataQuery = viewQuery;
/*
* Check that there is a unique index with no WHERE clause on one or more
@@ -1688,6 +1699,13 @@ IVM_immediate_maintenance(PG_FUNCTION_ARGS)
int rte_index = lfirst_int(lc2);
TupleDesc tupdesc_old;
TupleDesc tupdesc_new;
+ bool use_count = false;
+ char *count_colname = NULL;
+
+ count_colname = pstrdup("__ivm_count__");
+
+ if (query->distinctClause)
+ use_count = true;
/* calculate delta tables */
calc_delta(table, rte_index, rewritten, dest_old, dest_new,
@@ -1700,7 +1718,8 @@ IVM_immediate_maintenance(PG_FUNCTION_ARGS)
{
/* apply the delta tables to the materialized view */
apply_delta(matviewOid, old_tuplestore, new_tuplestore,
- tupdesc_old, tupdesc_new, query);
+ tupdesc_old, tupdesc_new, query, use_count,
+ count_colname);
}
PG_CATCH();
{
@@ -2238,7 +2257,7 @@ rewrite_query_for_postupdate_state(Query *query, MV_TriggerTable *table, int rte
static void
apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *new_tuplestores,
TupleDesc tupdesc_old, TupleDesc tupdesc_new,
- Query *query)
+ Query *query, bool use_count, char *count_colname)
{
StringInfoData querybuf;
StringInfoData target_list_buf;
@@ -2314,7 +2333,12 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
if (rc != SPI_OK_REL_REGISTER)
elog(ERROR, "SPI_register failed");
- apply_old_delta(matviewname, OLD_DELTA_ENRNAME, keys);
+ if (use_count)
+ /* apply old delta and get rows to be recalculated */
+ apply_old_delta_with_count(matviewname, OLD_DELTA_ENRNAME,
+ keys, count_colname);
+ else
+ apply_old_delta(matviewname, OLD_DELTA_ENRNAME, keys);
}
/* For tuple insertion */
@@ -2336,7 +2360,11 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
elog(ERROR, "SPI_register failed");
/* apply new delta */
- apply_new_delta(matviewname, NEW_DELTA_ENRNAME, &target_list_buf);
+ if (use_count)
+ apply_new_delta_with_count(matviewname, NEW_DELTA_ENRNAME,
+ keys, &target_list_buf, count_colname);
+ else
+ apply_new_delta(matviewname, NEW_DELTA_ENRNAME, &target_list_buf);
}
/* We're done maintaining the materialized view. */
@@ -2349,6 +2377,51 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
elog(ERROR, "SPI_finish failed");
}
+/*
+ * apply_old_delta_with_count
+ *
+ * Execute a query for applying a delta table given by deltname_old
+ * which contains tuples to be deleted from to a materialized view given by
+ * matviewname. This is used when counting is required, that is, the view
+ * has aggregate or distinct.
+ */
+static void
+apply_old_delta_with_count(const char *matviewname, const char *deltaname_old,
+ List *keys, const char *count_colname)
+{
+ StringInfoData querybuf;
+ char *match_cond;
+
+ /* build WHERE condition for searching tuples to be deleted */
+ match_cond = get_matching_condition_string(keys);
+
+ /* Search for matching tuples from the view and update or delete if found. */
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH t AS (" /* collecting tid of target tuples in the view */
+ "SELECT diff.%s, " /* count column */
+ "(diff.%s OPERATOR(pg_catalog.=) mv.%s) AS for_dlt, "
+ "mv.ctid "
+ "FROM %s AS mv, %s AS diff "
+ "WHERE %s" /* tuple matching condition */
+ "), updt AS (" /* update a tuple if this is not to be deleted */
+ "UPDATE %s AS mv SET %s = mv.%s OPERATOR(pg_catalog.-) t.%s "
+ "FROM t WHERE mv.ctid OPERATOR(pg_catalog.=) t.ctid AND NOT for_dlt "
+ ")"
+ /* delete a tuple if this is to be deleted */
+ "DELETE FROM %s AS mv USING t "
+ "WHERE mv.ctid OPERATOR(pg_catalog.=) t.ctid AND for_dlt",
+ count_colname,
+ count_colname, count_colname,
+ matviewname, deltaname_old,
+ match_cond,
+ matviewname, count_colname, count_colname, count_colname,
+ matviewname);
+
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_DELETE)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+}
+
/*
* apply_old_delta
*
@@ -2398,6 +2471,66 @@ apply_old_delta(const char *matviewname, const char *deltaname_old,
elog(ERROR, "SPI_exec failed: %s", querybuf.data);
}
+/*
+ * apply_new_delta_with_count
+ *
+ * Execute a query for applying a delta table given by deltname_new
+ * which contains tuples to be inserted into a materialized view given by
+ * matviewname. This is used when counting is required, that is, the view
+ * has aggregate or distinct. Also, when a table in EXISTS sub queries
+ * is modified.
+ */
+static void
+apply_new_delta_with_count(const char *matviewname, const char* deltaname_new,
+ List *keys, StringInfo target_list, const char* count_colname)
+{
+ StringInfoData querybuf;
+ StringInfoData returning_keys;
+ ListCell *lc;
+ char *match_cond = "";
+
+ /* build WHERE condition for searching tuples to be updated */
+ match_cond = get_matching_condition_string(keys);
+
+ /* build string of keys list */
+ initStringInfo(&returning_keys);
+ if (keys)
+ {
+ foreach (lc, keys)
+ {
+ Form_pg_attribute attr = (Form_pg_attribute) lfirst(lc);
+ char *resname = NameStr(attr->attname);
+ appendStringInfo(&returning_keys, "%s", quote_qualified_identifier("mv", resname));
+ if (lnext(keys, lc))
+ appendStringInfo(&returning_keys, ", ");
+ }
+ }
+ else
+ appendStringInfo(&returning_keys, "NULL");
+
+ /* Search for matching tuples from the view and update if found or insert if not. */
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH updt AS (" /* update a tuple if this exists in the view */
+ "UPDATE %s AS mv SET %s = mv.%s OPERATOR(pg_catalog.+) diff.%s "
+ "FROM %s AS diff "
+ "WHERE %s " /* tuple matching condition */
+ "RETURNING %s" /* returning keys of updated tuples */
+ ") INSERT INTO %s (%s) " /* insert a new tuple if this doesn't exist */
+ "SELECT %s FROM %s AS diff "
+ "WHERE NOT EXISTS (SELECT 1 FROM updt AS mv WHERE %s);",
+ matviewname, count_colname, count_colname, count_colname,
+ deltaname_new,
+ match_cond,
+ returning_keys.data,
+ matviewname, target_list->data,
+ target_list->data, deltaname_new,
+ match_cond);
+
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+}
+
/*
* apply_new_delta
*
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 99bc709959a..ee65292deb0 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -60,6 +60,7 @@
#include "catalog/toasting.h"
#include "commands/comment.h"
#include "commands/defrem.h"
+#include "commands/matview.h"
#include "commands/event_trigger.h"
#include "commands/extension.h"
#include "commands/matview.h"
@@ -3923,6 +3924,14 @@ renameatt_internal(Oid myrelid,
targetrelation = relation_open(myrelid, AccessExclusiveLock);
renameatt_check(myrelid, RelationGetForm(targetrelation), recursing);
+ /*
+ * Don't rename IVM columns.
+ */
+ if (RelationIsIVM(targetrelation) && isIvmName(oldattname))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("IVM column can not be renamed")));
+
/*
* if the 'recurse' flag is set then we are supposed to rename this
* attribute in all classes that inherit from 'relname' (as well as in
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 953c5797c5d..0f228eca259 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -516,6 +516,7 @@ _outRangeTblEntry(StringInfo str, const RangeTblEntry *node)
WRITE_INT_FIELD(rellockmode);
WRITE_UINT_FIELD(perminfoindex);
WRITE_NODE_FIELD(tablesample);
+ WRITE_BOOL_FIELD(relisivm);
break;
case RTE_SUBQUERY:
WRITE_NODE_FIELD(subquery);
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index b6b2ce6c792..dba4d8ff2de 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -365,6 +365,7 @@ _readRangeTblEntry(void)
READ_INT_FIELD(rellockmode);
READ_UINT_FIELD(perminfoindex);
READ_NODE_FIELD(tablesample);
+ READ_BOOL_FIELD(relisivm);
break;
case RTE_SUBQUERY:
READ_NODE_FIELD(subquery);
diff --git a/src/backend/parser/parse_relation.c b/src/backend/parser/parse_relation.c
index 43460e4a5a5..dc048c21bed 100644
--- a/src/backend/parser/parse_relation.c
+++ b/src/backend/parser/parse_relation.c
@@ -33,6 +33,7 @@
#include "utils/lsyscache.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
+#include "commands/matview.h"
/*
@@ -96,7 +97,7 @@ static void expandTupleDesc(TupleDesc tupdesc, Alias *eref,
int rtindex, int sublevels_up,
VarReturningType returning_type,
int location, bool include_dropped,
- List **colnames, List **colvars);
+ List **colnames, List **colvars, bool is_ivm);
static int specialAttNum(const char *attname);
static bool rte_visible_if_lateral(ParseState *pstate, RangeTblEntry *rte);
static bool rte_visible_if_qualified(ParseState *pstate, RangeTblEntry *rte);
@@ -1512,6 +1513,7 @@ addRangeTableEntry(ParseState *pstate,
rte->inh = inh;
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -1593,6 +1595,7 @@ addRangeTableEntryForRelation(ParseState *pstate,
rte->inh = inh;
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -2935,7 +2938,7 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
rtfunc->funccolcount, atts_done,
rtindex, sublevels_up,
returning_type, location,
- include_dropped, colnames, colvars);
+ include_dropped, colnames, colvars, false);
}
else if (functypclass == TYPEFUNC_SCALAR)
{
@@ -3211,7 +3214,7 @@ expandRelation(Oid relid, Alias *eref, int rtindex, int sublevels_up,
expandTupleDesc(rel->rd_att, eref, rel->rd_att->natts, 0,
rtindex, sublevels_up, returning_type,
location, include_dropped,
- colnames, colvars);
+ colnames, colvars, RelationIsIVM(rel));
relation_close(rel, AccessShareLock);
}
@@ -3229,7 +3232,7 @@ expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
int rtindex, int sublevels_up,
VarReturningType returning_type,
int location, bool include_dropped,
- List **colnames, List **colvars)
+ List **colnames, List **colvars, bool is_ivm)
{
ListCell *aliascell;
int varattno;
@@ -3242,6 +3245,9 @@ expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
{
Form_pg_attribute attr = TupleDescAttr(tupdesc, varattno);
+ if (is_ivm && isIvmName(NameStr(attr->attname)) && !MatViewIncrementalMaintenanceIsEnabled())
+ continue;
+
if (attr->attisdropped)
{
if (include_dropped)
@@ -3406,6 +3412,10 @@ expandNSItemAttrs(ParseState *pstate, ParseNamespaceItem *nsitem,
Var *varnode = (Var *) lfirst(var);
TargetEntry *te;
+ /* if transform * into columnlist with IMMV, remove IVM columns */
+ if (rte->relisivm && isIvmName(label) && !MatViewIncrementalMaintenanceIsEnabled())
+ continue;
+
te = makeTargetEntry((Expr *) varnode,
(AttrNumber) pstate->p_next_resno++,
label,
diff --git a/src/backend/rewrite/rewriteDefine.c b/src/backend/rewrite/rewriteDefine.c
index 6a223fbeaa4..74ac631299c 100644
--- a/src/backend/rewrite/rewriteDefine.c
+++ b/src/backend/rewrite/rewriteDefine.c
@@ -614,7 +614,8 @@ checkRuleResultList(List *targetList, TupleDesc resultDesc, bool isSelect,
attr->atttypmod))));
}
- if (i != resultDesc->natts)
+ /* No check for materialized views since this could have special columns for IVM */
+ if ((!isSelect || requireColumnNameMatch) && i != resultDesc->natts)
ereport(ERROR,
(errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
isSelect ?
diff --git a/src/include/commands/createas.h b/src/include/commands/createas.h
index c286ebcd70e..bfd0249b10d 100644
--- a/src/include/commands/createas.h
+++ b/src/include/commands/createas.h
@@ -29,6 +29,8 @@ extern ObjectAddress ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *st
extern void CreateIvmTriggersOnBaseTables(Query *qry, Oid matviewOid);
extern void CreateIndexOnIMMV(Query *query, Relation matviewRel);
+extern Query *rewriteQueryForIMMV(Query *query, List *colNames);
+
extern int GetIntoRelEFlags(IntoClause *intoClause);
extern DestReceiver *CreateIntoRelDestReceiver(IntoClause *intoClause);
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 91377a6cde3..59e689a1fac 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -1207,6 +1207,8 @@ typedef struct RangeTblEntry
Index perminfoindex pg_node_attr(query_jumble_ignore);
/* sampling info, or NULL */
struct TableSampleClause *tablesample;
+ /* incrementally maintainable materialized view? */
+ bool relisivm;
/*
* Fields valid for a subquery RTE (else NULL):
--
2.43.0
--Multipart=_Fri__29_May_2026_23_14_17_+0900_Te0o73X2VqYK57Gd
Content-Type: text/x-diff;
name="v37-0006-Add-Incremental-View-Maintenance-support.patch"
Content-Disposition: attachment;
filename="v37-0006-Add-Incremental-View-Maintenance-support.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 24+ messages in thread
* [PATCH v38 07/11] Add DISTINCT support for IVM
@ 2023-05-31 10:08 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 24+ messages in thread
From: Yugo Nagata @ 2023-05-31 10:08 UTC (permalink / raw)
When IMMV is created with DISTINCT, multiplicity of tuples is
counted and stored in "__ivm_count__" column, which is a hidden
column of IMMV. The value in __ivm_count__ is updated when IMMV
is maintained incrementally. A tuple in IMMV can be removed if
and only if the count becomes zero.
---
src/backend/commands/createas.c | 141 +++++++++++++++++++++------
src/backend/commands/indexcmds.c | 40 ++++++++
src/backend/commands/matview.c | 145 ++++++++++++++++++++++++++--
src/backend/commands/tablecmds.c | 9 ++
src/backend/nodes/outfuncs.c | 1 +
src/backend/nodes/readfuncs.c | 1 +
src/backend/parser/parse_relation.c | 18 +++-
src/backend/rewrite/rewriteDefine.c | 3 +-
src/include/commands/createas.h | 2 +
src/include/nodes/parsenodes.h | 2 +
10 files changed, 319 insertions(+), 43 deletions(-)
diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c
index a499688b79d..cd8db0059f9 100644
--- a/src/backend/commands/createas.c
+++ b/src/backend/commands/createas.c
@@ -55,6 +55,7 @@
#include "parser/parser.h"
#include "parser/parsetree.h"
#include "parser/parse_clause.h"
+#include "parser/parse_func.h"
#include "rewrite/rewriteHandler.h"
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
@@ -307,6 +308,9 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt,
errhint("functions must be marked IMMUTABLE")));
check_ivm_restriction((Node *) query);
+
+ /* For IMMV, we need to rewrite matview query */
+ query = rewriteQueryForIMMV(query, into->colNames);
}
if (into->skipData)
@@ -416,6 +420,49 @@ ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt,
return address;
}
+/*
+ * rewriteQueryForIMMV -- rewrite view definition query for IMMV
+ *
+ * count(*) is added for counting distinct tuples in views.
+ */
+Query *
+rewriteQueryForIMMV(Query *query, List *colNames)
+{
+ Query *rewritten;
+
+ Node *node;
+ ParseState *pstate = make_parsestate(NULL);
+ FuncCall *fn;
+
+ rewritten = copyObject(query);
+ pstate->p_expr_kind = EXPR_KIND_SELECT_TARGET;
+
+ /*
+ * Convert DISTINCT to GROUP BY and add count(*) for counting distinct
+ * tuples in views.
+ */
+ if (rewritten->distinctClause)
+ {
+ TargetEntry *tle;
+
+ rewritten->groupClause = transformDistinctClause(NULL, &rewritten->targetList, rewritten->sortClause, false);
+
+ fn = makeFuncCall(SystemFuncName("count"), NIL, COERCE_EXPLICIT_CALL, -1);
+ fn->agg_star = true;
+
+ node = ParseFuncOrColumn(pstate, fn->funcname, NIL, NULL, fn, false, -1);
+
+ tle = makeTargetEntry((Expr *) node,
+ list_length(rewritten->targetList) + 1,
+ pstrdup("__ivm_count__"),
+ false);
+ rewritten->targetList = lappend(rewritten->targetList, tle);
+ rewritten->hasAggs = true;
+ }
+
+ return rewritten;
+}
+
/*
* GetIntoRelEFlags --- compute executor flags needed for CREATE TABLE AS
*
@@ -539,7 +586,8 @@ intorel_startup(DestReceiver *self, int operation, TupleDesc typeinfo)
ColumnDef *col;
char *colname;
- if (lc)
+ /* Don't override hidden columns added for IVM */
+ if (lc && !isIvmName(NameStr(attribute->attname)))
{
colname = strVal(lfirst(lc));
lc = lnext(into->colNames, lc);
@@ -943,10 +991,6 @@ check_ivm_restriction_walker(Node *node, void *context)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("LIMIT/OFFSET clause is not supported on incrementally maintainable materialized view")));
- if (qry->distinctClause)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("DISTINCT is not supported on incrementally maintainable materialized view")));
if (qry->hasDistinctOn)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -1108,12 +1152,18 @@ CreateIndexOnIMMV(Query *query, Relation matviewRel)
char idxname[NAMEDATALEN];
List *indexoidlist = RelationGetIndexList(matviewRel);
ListCell *indexoidscan;
- Bitmapset *key_attnos;
snprintf(idxname, sizeof(idxname), "%s_index", RelationGetRelationName(matviewRel));
index = makeNode(IndexStmt);
+ /*
+ * We consider null values not distinct to make sure that views with DISTINCT
+ * or GROUP BY don't contain multiple NULL rows when NULL is inserted to
+ * a base table concurrently.
+ */
+ index->nulls_not_distinct = true;
+
index->unique = true;
index->primary = false;
index->isconstraint = false;
@@ -1140,41 +1190,68 @@ CreateIndexOnIMMV(Query *query, Relation matviewRel)
index->concurrent = false;
index->if_not_exists = false;
- /* create index on the base tables' primary key columns */
- key_attnos = get_primary_key_attnos_from_query(query, &constraintList);
- if (key_attnos)
+ if (query->distinctClause)
{
+ /* create unique constraint on all columns */
foreach(lc, query->targetList)
{
TargetEntry *tle = (TargetEntry *) lfirst(lc);
Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, tle->resno - 1);
-
- if (bms_is_member(tle->resno - FirstLowInvalidHeapAttributeNumber, key_attnos))
- {
- IndexElem *iparam;
-
- iparam = makeNode(IndexElem);
- iparam->name = pstrdup(NameStr(attr->attname));
- iparam->expr = NULL;
- iparam->indexcolname = NULL;
- iparam->collation = NIL;
- iparam->opclass = NIL;
- iparam->opclassopts = NIL;
- iparam->ordering = SORTBY_DEFAULT;
- iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
- index->indexParams = lappend(index->indexParams, iparam);
- }
+ IndexElem *iparam;
+
+ iparam = makeNode(IndexElem);
+ iparam->name = pstrdup(NameStr(attr->attname));
+ iparam->expr = NULL;
+ iparam->indexcolname = NULL;
+ iparam->collation = NIL;
+ iparam->opclass = NIL;
+ iparam->opclassopts = NIL;
+ iparam->ordering = SORTBY_DEFAULT;
+ iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
+ index->indexParams = lappend(index->indexParams, iparam);
}
}
else
{
- /* create no index, just notice that an appropriate index is necessary for efficient IVM */
- ereport(NOTICE,
- (errmsg("could not create an index on materialized view \"%s\" automatically",
- RelationGetRelationName(matviewRel)),
- errdetail("This target list does not have all the primary key columns. "),
- errhint("Create an index on the materialized view for efficient incremental maintenance.")));
- return;
+ Bitmapset *key_attnos;
+
+ /* create index on the base tables' primary key columns */
+ key_attnos = get_primary_key_attnos_from_query(query, &constraintList);
+ if (key_attnos)
+ {
+ foreach(lc, query->targetList)
+ {
+ TargetEntry *tle = (TargetEntry *) lfirst(lc);
+ Form_pg_attribute attr = TupleDescAttr(matviewRel->rd_att, tle->resno - 1);
+
+ if (bms_is_member(tle->resno - FirstLowInvalidHeapAttributeNumber, key_attnos))
+ {
+ IndexElem *iparam;
+
+ iparam = makeNode(IndexElem);
+ iparam->name = pstrdup(NameStr(attr->attname));
+ iparam->expr = NULL;
+ iparam->indexcolname = NULL;
+ iparam->collation = NIL;
+ iparam->opclass = NIL;
+ iparam->opclassopts = NIL;
+ iparam->ordering = SORTBY_DEFAULT;
+ iparam->nulls_ordering = SORTBY_NULLS_DEFAULT;
+ index->indexParams = lappend(index->indexParams, iparam);
+ }
+ }
+ }
+ else
+ {
+ /* create no index, just notice that an appropriate index is necessary for efficient IVM */
+ ereport(NOTICE,
+ (errmsg("could not create an index on materialized view \"%s\" automatically",
+ RelationGetRelationName(matviewRel)),
+ errdetail("This target list does not have all the primary key columns, "
+ "or this view does not contain DISTINCT clause."),
+ errhint("Create an index on the materialized view for efficient incremental maintenance.")));
+ return;
+ }
}
/* If we have a compatible index, we don't need to create another. */
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 9ab74c8df0a..1f95e6b229f 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -41,6 +41,7 @@
#include "commands/comment.h"
#include "commands/defrem.h"
#include "commands/event_trigger.h"
+#include "commands/matview.h"
#include "commands/progress.h"
#include "commands/tablecmds.h"
#include "commands/tablespace.h"
@@ -1179,6 +1180,45 @@ DefineIndex(ParseState *pstate,
safe_index = indexInfo->ii_Expressions == NIL &&
indexInfo->ii_Predicate == NIL;
+ /*
+ * We disallow unique indexes on IVM columns of IMMVs.
+ */
+ if (RelationIsIVM(rel) && stmt->unique)
+ {
+ for (int i = 0; i < indexInfo->ii_NumIndexKeyAttrs; i++)
+ {
+ AttrNumber attno = indexInfo->ii_IndexAttrNumbers[i];
+ if (attno > 0)
+ {
+ char *name = NameStr(TupleDescAttr(rel->rd_att, attno - 1)->attname);
+ if (name && isIvmName(name))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("unique index creation on IVM columns is not supported")));
+ }
+ }
+
+ if (indexInfo->ii_Expressions)
+ {
+ Bitmapset *indexattrs = NULL;
+ int varno = -1;
+
+ pull_varattnos((Node *) indexInfo->ii_Expressions, 1, &indexattrs);
+
+ while ((varno = bms_next_member(indexattrs, varno)) >= 0)
+ {
+ int attno = varno + FirstLowInvalidHeapAttributeNumber;
+ char *name = NameStr(TupleDescAttr(rel->rd_att, attno - 1)->attname);
+ if (name && isIvmName(name))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("unique index creation on IVM columns is not supported")));
+ }
+
+ }
+ }
+
+
/*
* Report index creation if appropriate (delay this till after most of the
* error checks)
diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c
index eaf39f80cd3..a2746ca9265 100644
--- a/src/backend/commands/matview.c
+++ b/src/backend/commands/matview.c
@@ -193,11 +193,15 @@ static Query *rewrite_query_for_postupdate_state(Query *query, MV_TriggerTable *
static void apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *new_tuplestores,
TupleDesc tupdesc_old, TupleDesc tupdesc_new,
- Query *query);
+ Query *query, bool use_count, char *count_colname);
static void apply_old_delta(const char *matviewname, const char *deltaname_old,
List *keys);
+static void apply_old_delta_with_count(const char *matviewname, const char *deltaname_old,
+ List *keys, const char *count_colname);
static void apply_new_delta(const char *matviewname, const char *deltaname_new,
StringInfo target_list);
+static void apply_new_delta_with_count(const char *matviewname, const char* deltaname_new,
+ List *keys, StringInfo target_list, const char* count_colname);
static char *get_matching_condition_string(List *keys);
static void generate_equal(StringInfo querybuf, Oid opttype,
const char *leftop, const char *rightop);
@@ -346,6 +350,7 @@ RefreshMatViewByOid(Oid matviewOid, bool is_create, bool skipData,
QueryCompletion *qc)
{
Relation matviewRel;
+ Query *viewQuery;
Query *dataQuery;
Oid tableSpace;
Oid relowner;
@@ -394,7 +399,13 @@ RefreshMatViewByOid(Oid matviewOid, bool is_create, bool skipData,
errmsg("%s options %s and %s cannot be used together",
"REFRESH", "CONCURRENTLY", "WITH NO DATA")));
- dataQuery = get_matview_query(matviewRel);
+ viewQuery = get_matview_query(matviewRel);
+
+ /* For IMMV, we need to rewrite matview query */
+ if (!skipData && RelationIsIVM(matviewRel))
+ dataQuery = rewriteQueryForIMMV(viewQuery,NIL);
+ else
+ dataQuery = viewQuery;
/*
* Check that there is a unique index with no WHERE clause on one or more
@@ -1688,6 +1699,13 @@ IVM_immediate_maintenance(PG_FUNCTION_ARGS)
int rte_index = lfirst_int(lc2);
TupleDesc tupdesc_old;
TupleDesc tupdesc_new;
+ bool use_count = false;
+ char *count_colname = NULL;
+
+ count_colname = pstrdup("__ivm_count__");
+
+ if (query->distinctClause)
+ use_count = true;
/* calculate delta tables */
calc_delta(table, rte_index, rewritten, dest_old, dest_new,
@@ -1700,7 +1718,8 @@ IVM_immediate_maintenance(PG_FUNCTION_ARGS)
{
/* apply the delta tables to the materialized view */
apply_delta(matviewOid, old_tuplestore, new_tuplestore,
- tupdesc_old, tupdesc_new, query);
+ tupdesc_old, tupdesc_new, query, use_count,
+ count_colname);
}
PG_CATCH();
{
@@ -2238,7 +2257,7 @@ rewrite_query_for_postupdate_state(Query *query, MV_TriggerTable *table, int rte
static void
apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *new_tuplestores,
TupleDesc tupdesc_old, TupleDesc tupdesc_new,
- Query *query)
+ Query *query, bool use_count, char *count_colname)
{
StringInfoData querybuf;
StringInfoData target_list_buf;
@@ -2314,7 +2333,12 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
if (rc != SPI_OK_REL_REGISTER)
elog(ERROR, "SPI_register failed");
- apply_old_delta(matviewname, OLD_DELTA_ENRNAME, keys);
+ if (use_count)
+ /* apply old delta and get rows to be recalculated */
+ apply_old_delta_with_count(matviewname, OLD_DELTA_ENRNAME,
+ keys, count_colname);
+ else
+ apply_old_delta(matviewname, OLD_DELTA_ENRNAME, keys);
}
/* For tuple insertion */
@@ -2336,7 +2360,11 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
elog(ERROR, "SPI_register failed");
/* apply new delta */
- apply_new_delta(matviewname, NEW_DELTA_ENRNAME, &target_list_buf);
+ if (use_count)
+ apply_new_delta_with_count(matviewname, NEW_DELTA_ENRNAME,
+ keys, &target_list_buf, count_colname);
+ else
+ apply_new_delta(matviewname, NEW_DELTA_ENRNAME, &target_list_buf);
}
/* We're done maintaining the materialized view. */
@@ -2349,6 +2377,51 @@ apply_delta(Oid matviewOid, Tuplestorestate *old_tuplestores, Tuplestorestate *n
elog(ERROR, "SPI_finish failed");
}
+/*
+ * apply_old_delta_with_count
+ *
+ * Execute a query for applying a delta table given by deltname_old
+ * which contains tuples to be deleted from to a materialized view given by
+ * matviewname. This is used when counting is required, that is, the view
+ * has aggregate or distinct.
+ */
+static void
+apply_old_delta_with_count(const char *matviewname, const char *deltaname_old,
+ List *keys, const char *count_colname)
+{
+ StringInfoData querybuf;
+ char *match_cond;
+
+ /* build WHERE condition for searching tuples to be deleted */
+ match_cond = get_matching_condition_string(keys);
+
+ /* Search for matching tuples from the view and update or delete if found. */
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH t AS (" /* collecting tid of target tuples in the view */
+ "SELECT diff.%s, " /* count column */
+ "(diff.%s OPERATOR(pg_catalog.=) mv.%s) AS for_dlt, "
+ "mv.ctid "
+ "FROM %s AS mv, %s AS diff "
+ "WHERE %s" /* tuple matching condition */
+ "), updt AS (" /* update a tuple if this is not to be deleted */
+ "UPDATE %s AS mv SET %s = mv.%s OPERATOR(pg_catalog.-) t.%s "
+ "FROM t WHERE mv.ctid OPERATOR(pg_catalog.=) t.ctid AND NOT for_dlt "
+ ")"
+ /* delete a tuple if this is to be deleted */
+ "DELETE FROM %s AS mv USING t "
+ "WHERE mv.ctid OPERATOR(pg_catalog.=) t.ctid AND for_dlt",
+ count_colname,
+ count_colname, count_colname,
+ matviewname, deltaname_old,
+ match_cond,
+ matviewname, count_colname, count_colname, count_colname,
+ matviewname);
+
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_DELETE)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+}
+
/*
* apply_old_delta
*
@@ -2398,6 +2471,66 @@ apply_old_delta(const char *matviewname, const char *deltaname_old,
elog(ERROR, "SPI_exec failed: %s", querybuf.data);
}
+/*
+ * apply_new_delta_with_count
+ *
+ * Execute a query for applying a delta table given by deltname_new
+ * which contains tuples to be inserted into a materialized view given by
+ * matviewname. This is used when counting is required, that is, the view
+ * has aggregate or distinct. Also, when a table in EXISTS sub queries
+ * is modified.
+ */
+static void
+apply_new_delta_with_count(const char *matviewname, const char* deltaname_new,
+ List *keys, StringInfo target_list, const char* count_colname)
+{
+ StringInfoData querybuf;
+ StringInfoData returning_keys;
+ ListCell *lc;
+ char *match_cond = "";
+
+ /* build WHERE condition for searching tuples to be updated */
+ match_cond = get_matching_condition_string(keys);
+
+ /* build string of keys list */
+ initStringInfo(&returning_keys);
+ if (keys)
+ {
+ foreach (lc, keys)
+ {
+ Form_pg_attribute attr = (Form_pg_attribute) lfirst(lc);
+ char *resname = NameStr(attr->attname);
+ appendStringInfo(&returning_keys, "%s", quote_qualified_identifier("mv", resname));
+ if (lnext(keys, lc))
+ appendStringInfo(&returning_keys, ", ");
+ }
+ }
+ else
+ appendStringInfo(&returning_keys, "NULL");
+
+ /* Search for matching tuples from the view and update if found or insert if not. */
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "WITH updt AS (" /* update a tuple if this exists in the view */
+ "UPDATE %s AS mv SET %s = mv.%s OPERATOR(pg_catalog.+) diff.%s "
+ "FROM %s AS diff "
+ "WHERE %s " /* tuple matching condition */
+ "RETURNING %s" /* returning keys of updated tuples */
+ ") INSERT INTO %s (%s) " /* insert a new tuple if this doesn't exist */
+ "SELECT %s FROM %s AS diff "
+ "WHERE NOT EXISTS (SELECT 1 FROM updt AS mv WHERE %s);",
+ matviewname, count_colname, count_colname, count_colname,
+ deltaname_new,
+ match_cond,
+ returning_keys.data,
+ matviewname, target_list->data,
+ target_list->data, deltaname_new,
+ match_cond);
+
+ if (SPI_exec(querybuf.data, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+}
+
/*
* apply_new_delta
*
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 920b62d254d..f62765f0a89 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -60,6 +60,7 @@
#include "catalog/toasting.h"
#include "commands/comment.h"
#include "commands/defrem.h"
+#include "commands/matview.h"
#include "commands/event_trigger.h"
#include "commands/extension.h"
#include "commands/matview.h"
@@ -3930,6 +3931,14 @@ renameatt_internal(Oid myrelid,
targetrelation = relation_open(myrelid, AccessExclusiveLock);
renameatt_check(myrelid, RelationGetForm(targetrelation), recursing);
+ /*
+ * Don't rename IVM columns.
+ */
+ if (RelationIsIVM(targetrelation) && isIvmName(oldattname))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("IVM column can not be renamed")));
+
/*
* if the 'recurse' flag is set then we are supposed to rename this
* attribute in all classes that inherit from 'relname' (as well as in
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 953c5797c5d..0f228eca259 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -516,6 +516,7 @@ _outRangeTblEntry(StringInfo str, const RangeTblEntry *node)
WRITE_INT_FIELD(rellockmode);
WRITE_UINT_FIELD(perminfoindex);
WRITE_NODE_FIELD(tablesample);
+ WRITE_BOOL_FIELD(relisivm);
break;
case RTE_SUBQUERY:
WRITE_NODE_FIELD(subquery);
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index b6b2ce6c792..dba4d8ff2de 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -365,6 +365,7 @@ _readRangeTblEntry(void)
READ_INT_FIELD(rellockmode);
READ_UINT_FIELD(perminfoindex);
READ_NODE_FIELD(tablesample);
+ READ_BOOL_FIELD(relisivm);
break;
case RTE_SUBQUERY:
READ_NODE_FIELD(subquery);
diff --git a/src/backend/parser/parse_relation.c b/src/backend/parser/parse_relation.c
index ced210cd206..5392e9f15b2 100644
--- a/src/backend/parser/parse_relation.c
+++ b/src/backend/parser/parse_relation.c
@@ -33,6 +33,7 @@
#include "utils/lsyscache.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
+#include "commands/matview.h"
/*
@@ -96,7 +97,7 @@ static void expandTupleDesc(TupleDesc tupdesc, Alias *eref,
int rtindex, int sublevels_up,
VarReturningType returning_type,
int location, bool include_dropped,
- List **colnames, List **colvars);
+ List **colnames, List **colvars, bool is_ivm);
static int specialAttNum(const char *attname);
static bool rte_visible_if_lateral(ParseState *pstate, RangeTblEntry *rte);
static bool rte_visible_if_qualified(ParseState *pstate, RangeTblEntry *rte);
@@ -1544,6 +1545,7 @@ addRangeTableEntry(ParseState *pstate,
rte->inh = inh;
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -1625,6 +1627,7 @@ addRangeTableEntryForRelation(ParseState *pstate,
rte->inh = inh;
rte->relkind = rel->rd_rel->relkind;
rte->rellockmode = lockmode;
+ rte->relisivm = rel->rd_rel->relisivm;
/*
* Build the list of effective column names using user-supplied aliases
@@ -2967,7 +2970,7 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
rtfunc->funccolcount, atts_done,
rtindex, sublevels_up,
returning_type, location,
- include_dropped, colnames, colvars);
+ include_dropped, colnames, colvars, false);
}
else if (functypclass == TYPEFUNC_SCALAR)
{
@@ -3243,7 +3246,7 @@ expandRelation(Oid relid, Alias *eref, int rtindex, int sublevels_up,
expandTupleDesc(rel->rd_att, eref, rel->rd_att->natts, 0,
rtindex, sublevels_up, returning_type,
location, include_dropped,
- colnames, colvars);
+ colnames, colvars, RelationIsIVM(rel));
relation_close(rel, AccessShareLock);
}
@@ -3261,7 +3264,7 @@ expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
int rtindex, int sublevels_up,
VarReturningType returning_type,
int location, bool include_dropped,
- List **colnames, List **colvars)
+ List **colnames, List **colvars, bool is_ivm)
{
ListCell *aliascell;
int varattno;
@@ -3274,6 +3277,9 @@ expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
{
Form_pg_attribute attr = TupleDescAttr(tupdesc, varattno);
+ if (is_ivm && isIvmName(NameStr(attr->attname)) && !MatViewIncrementalMaintenanceIsEnabled())
+ continue;
+
if (attr->attisdropped)
{
if (include_dropped)
@@ -3438,6 +3444,10 @@ expandNSItemAttrs(ParseState *pstate, ParseNamespaceItem *nsitem,
Var *varnode = (Var *) lfirst(var);
TargetEntry *te;
+ /* if transform * into columnlist with IMMV, remove IVM columns */
+ if (rte->relisivm && isIvmName(label) && !MatViewIncrementalMaintenanceIsEnabled())
+ continue;
+
te = makeTargetEntry((Expr *) varnode,
(AttrNumber) pstate->p_next_resno++,
label,
diff --git a/src/backend/rewrite/rewriteDefine.c b/src/backend/rewrite/rewriteDefine.c
index 6a223fbeaa4..74ac631299c 100644
--- a/src/backend/rewrite/rewriteDefine.c
+++ b/src/backend/rewrite/rewriteDefine.c
@@ -614,7 +614,8 @@ checkRuleResultList(List *targetList, TupleDesc resultDesc, bool isSelect,
attr->atttypmod))));
}
- if (i != resultDesc->natts)
+ /* No check for materialized views since this could have special columns for IVM */
+ if ((!isSelect || requireColumnNameMatch) && i != resultDesc->natts)
ereport(ERROR,
(errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
isSelect ?
diff --git a/src/include/commands/createas.h b/src/include/commands/createas.h
index c286ebcd70e..bfd0249b10d 100644
--- a/src/include/commands/createas.h
+++ b/src/include/commands/createas.h
@@ -29,6 +29,8 @@ extern ObjectAddress ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *st
extern void CreateIvmTriggersOnBaseTables(Query *qry, Oid matviewOid);
extern void CreateIndexOnIMMV(Query *query, Relation matviewRel);
+extern Query *rewriteQueryForIMMV(Query *query, List *colNames);
+
extern int GetIntoRelEFlags(IntoClause *intoClause);
extern DestReceiver *CreateIntoRelDestReceiver(IntoClause *intoClause);
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 4133c404a6b..cbcef860061 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -1209,6 +1209,8 @@ typedef struct RangeTblEntry
Index perminfoindex pg_node_attr(query_jumble_ignore);
/* sampling info, or NULL */
struct TableSampleClause *tablesample;
+ /* incrementally maintainable materialized view? */
+ bool relisivm;
/*
* Fields valid for a subquery RTE (else NULL):
--
2.43.0
--Multipart=_Wed__1_Jul_2026_00_04_01_+0900_OVSy2WWK_9aByzDJ
Content-Type: text/x-diff;
name="v38-0006-Add-Incremental-View-Maintenance-support.patch"
Content-Disposition: attachment;
filename="v38-0006-Add-Incremental-View-Maintenance-support.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 24+ messages in thread
* Small optimization set tuple block/tableOid once
@ 2025-04-02 11:28 Ranier Vilela <[email protected]>
0 siblings, 1 reply; 24+ messages in thread
From: Ranier Vilela @ 2025-04-02 11:28 UTC (permalink / raw)
To: pgsql-hackers
Hi.
Inspired by [1]
There is an opportunity for optimization according to the commit:
2904324 <http://2904324a88f672b2ecc22735279c16d6e1ee178c;
" Due to splitting the block id into two 16 bit integers, BlockIdSet() is
more expensive than one might think. Doing it once per returned tuple shows
up as a small but reliably reproducible cost. It's simple enough to set the
block number just once per block in pagemode, so do so."
By moving the invariants out of the loop, it really is an improvement.
Then the following sources can be optimized:
contrib/pg_visibility/pg_visibility.c
contrib/pgstattuple/pgstatapprox.c
src/backend/access/heap/heapam.c
src/backend/access/heap/heapam_handler.c
src/backend/access/heap/pruneheap.c
src/backend/access/heap/vacuumlazy.c
src/backend/commands/dbcommands.c
Attached all patchs.
best regards,
Ranier Vilela
[1] Re: AIO writes vs hint bits vs checksums
<https://www.postgresql.org/message-id/lxzj26ga6ippdeunz6kuncectr5gfuugmm2ry22qu6hcx6oid6@lzx3sjsqhmt...;
Attachments:
[application/octet-stream] pg_visibility_set_tuple_block_once.patch (1.7K, ../../CAEudQAqc8TYoHXGgUcnTJm-nc2TF2WwORm8rh1=nLjMVj1UkbQ@mail.gmail.com/3-pg_visibility_set_tuple_block_once.patch)
download | inline diff:
diff --git a/contrib/pg_visibility/pg_visibility.c b/contrib/pg_visibility/pg_visibility.c
index d79ef35006..d0c73eaeb0 100644
--- a/contrib/pg_visibility/pg_visibility.c
+++ b/contrib/pg_visibility/pg_visibility.c
@@ -769,6 +769,7 @@ collect_corrupt_items(Oid relid, bool all_visible, bool all_frozen)
OffsetNumber offnum,
maxoff;
BlockNumber blkno;
+ HeapTupleData tuple;
/* Make sure we are interruptible. */
CHECK_FOR_INTERRUPTS();
@@ -793,12 +794,15 @@ collect_corrupt_items(Oid relid, bool all_visible, bool all_frozen)
continue;
}
+ /* block and tableOid is the same for all tuples, set it once outside the loop */
+ ItemPointerSetBlockNumber(&tuple.t_self, blkno);
+ tuple.t_tableOid = relid;
+
/* Iterate over each tuple on the page. */
for (offnum = FirstOffsetNumber;
offnum <= maxoff;
offnum = OffsetNumberNext(offnum))
{
- HeapTupleData tuple;
ItemId itemid;
itemid = PageGetItemId(page, offnum);
@@ -807,19 +811,18 @@ collect_corrupt_items(Oid relid, bool all_visible, bool all_frozen)
if (!ItemIdIsUsed(itemid) || ItemIdIsRedirected(itemid))
continue;
+ ItemPointerSetOffsetNumber(&tuple.t_self, offnum);
+
/* Dead line pointers are neither all-visible nor frozen. */
if (ItemIdIsDead(itemid))
{
- ItemPointerSet(&(tuple.t_self), blkno, offnum);
record_corrupt_item(items, &tuple.t_self);
continue;
}
/* Initialize a HeapTupleData structure for checks below. */
- ItemPointerSet(&(tuple.t_self), blkno, offnum);
tuple.t_data = (HeapTupleHeader) PageGetItem(page, itemid);
tuple.t_len = ItemIdGetLength(itemid);
- tuple.t_tableOid = relid;
/*
* If we're checking whether the page is all-visible, we expect
[application/octet-stream] pgstattuple_set_tuple_block_once.patch (1.3K, ../../CAEudQAqc8TYoHXGgUcnTJm-nc2TF2WwORm8rh1=nLjMVj1UkbQ@mail.gmail.com/4-pgstattuple_set_tuple_block_once.patch)
download | inline diff:
diff --git a/contrib/pgstattuple/pgstatapprox.c b/contrib/pgstattuple/pgstatapprox.c
index a59ff4e9d4..26afe55c01 100644
--- a/contrib/pgstattuple/pgstatapprox.c
+++ b/contrib/pgstattuple/pgstatapprox.c
@@ -78,6 +78,7 @@ statapprox_heap(Relation rel, output_type *stat)
OffsetNumber offnum,
maxoff;
Size freespace;
+ HeapTupleData tuple;
CHECK_FOR_INTERRUPTS();
@@ -118,12 +119,15 @@ statapprox_heap(Relation rel, output_type *stat)
*/
maxoff = PageGetMaxOffsetNumber(page);
+ /* block and tableOid is the same for all tuples, set it once outside the loop */
+ ItemPointerSetBlockNumber(&tuple.t_self, blkno);
+ tuple.t_tableOid = RelationGetRelid(rel);
+
for (offnum = FirstOffsetNumber;
offnum <= maxoff;
offnum = OffsetNumberNext(offnum))
{
ItemId itemid;
- HeapTupleData tuple;
itemid = PageGetItemId(page, offnum);
@@ -135,11 +139,9 @@ statapprox_heap(Relation rel, output_type *stat)
Assert(ItemIdIsNormal(itemid));
- ItemPointerSet(&(tuple.t_self), blkno, offnum);
-
tuple.t_data = (HeapTupleHeader) PageGetItem(page, itemid);
tuple.t_len = ItemIdGetLength(itemid);
- tuple.t_tableOid = RelationGetRelid(rel);
+ ItemPointerSetOffsetNumber(&tuple.t_self, offnum);
/*
* We follow VACUUM's lead in counting INSERT_IN_PROGRESS tuples
[application/octet-stream] heapam_set_tuple_block_once.patch (2.7K, ../../CAEudQAqc8TYoHXGgUcnTJm-nc2TF2WwORm8rh1=nLjMVj1UkbQ@mail.gmail.com/5-heapam_set_tuple_block_once.patch)
download | inline diff:
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index cedaa195cb..148a979699 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -508,13 +508,17 @@ page_collect_tuples(HeapScanDesc scan, Snapshot snapshot,
BlockNumber block, int lines,
bool all_visible, bool check_serializable)
{
+ HeapTupleData loctup;
int ntup = 0;
OffsetNumber lineoff;
+ /* block and tableOid is the same for all tuples, set it once outside the loop */
+ ItemPointerSetBlockNumber(&loctup.t_self, block);
+ loctup.t_tableOid = RelationGetRelid(scan->rs_base.rs_rd);
+
for (lineoff = FirstOffsetNumber; lineoff <= lines; lineoff++)
{
ItemId lpp = PageGetItemId(page, lineoff);
- HeapTupleData loctup;
bool valid;
if (!ItemIdIsNormal(lpp))
@@ -522,8 +526,7 @@ page_collect_tuples(HeapScanDesc scan, Snapshot snapshot,
loctup.t_data = (HeapTupleHeader) PageGetItem(page, lpp);
loctup.t_len = ItemIdGetLength(lpp);
- loctup.t_tableOid = RelationGetRelid(scan->rs_base.rs_rd);
- ItemPointerSet(&(loctup.t_self), block, lineoff);
+ ItemPointerSetOffsetNumber(&loctup.t_self, lineoff);
if (all_visible)
valid = true;
@@ -931,6 +934,10 @@ heapgettup(HeapScanDesc scan,
LockBuffer(scan->rs_cbuf, BUFFER_LOCK_SHARE);
page = heapgettup_start_page(scan, dir, &linesleft, &lineoff);
+
+ /* block is the same for all tuples, set it once outside the loop */
+ ItemPointerSetBlockNumber(&tuple->t_self, scan->rs_cblock);
+
continue_page:
/*
@@ -950,7 +957,7 @@ continue_page:
tuple->t_data = (HeapTupleHeader) PageGetItem(page, lpp);
tuple->t_len = ItemIdGetLength(lpp);
- ItemPointerSet(&(tuple->t_self), scan->rs_cblock, lineoff);
+ ItemPointerSetOffsetNumber(&tuple->t_self, lineoff);
visible = HeapTupleSatisfiesVisibility(tuple,
scan->rs_base.rs_snapshot,
@@ -1766,6 +1773,10 @@ heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
Assert(TransactionIdIsValid(RecentXmin));
Assert(BufferGetBlockNumber(buffer) == blkno);
+ /* block and tableOid is the same for all tuples, set it once outside the loop */
+ ItemPointerSetBlockNumber(&heapTuple->t_self, blkno);
+ heapTuple->t_tableOid = RelationGetRelid(relation);
+
/* Scan through possible multiple members of HOT-chain */
for (;;)
{
@@ -1800,8 +1811,7 @@ heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
*/
heapTuple->t_data = (HeapTupleHeader) PageGetItem(page, lp);
heapTuple->t_len = ItemIdGetLength(lp);
- heapTuple->t_tableOid = RelationGetRelid(relation);
- ItemPointerSet(&heapTuple->t_self, blkno, offnum);
+ ItemPointerSetOffsetNumber(&heapTuple->t_self, offnum);
/*
* Shouldn't see a HEAP_ONLY tuple at chain start.
[application/octet-stream] heapam_handler_set_tuple_block_once.patch (4.9K, ../../CAEudQAqc8TYoHXGgUcnTJm-nc2TF2WwORm8rh1=nLjMVj1UkbQ@mail.gmail.com/6-heapam_handler_set_tuple_block_once.patch)
download | inline diff:
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 24d3765aa2..5c42bce2de 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -1031,6 +1031,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin,
TupleTableSlot *slot)
{
HeapScanDesc hscan = (HeapScanDesc) scan;
+ HeapTuple targtuple;
Page targpage;
OffsetNumber maxoffset;
BufferHeapTupleTableSlot *hslot;
@@ -1038,14 +1039,18 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin,
Assert(TTS_IS_BUFFERTUPLE(slot));
hslot = (BufferHeapTupleTableSlot *) slot;
+ targtuple = &hslot->base.tupdata;
targpage = BufferGetPage(hscan->rs_cbuf);
maxoffset = PageGetMaxOffsetNumber(targpage);
+ /* block and tableOid is the same for all tuples, set it once outside the loop */
+ ItemPointerSetBlockNumber(&targtuple->t_self, hscan->rs_cblock);
+ targtuple->t_tableOid = RelationGetRelid(scan->rs_rd);
+
/* Inner loop over all tuples on the selected page */
for (; hscan->rs_cindex <= maxoffset; hscan->rs_cindex++)
{
ItemId itemid;
- HeapTuple targtuple = &hslot->base.tupdata;
bool sample_it = false;
itemid = PageGetItemId(targpage, hscan->rs_cindex);
@@ -1063,11 +1068,9 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin,
continue;
}
- ItemPointerSet(&targtuple->t_self, hscan->rs_cblock, hscan->rs_cindex);
-
- targtuple->t_tableOid = RelationGetRelid(scan->rs_rd);
targtuple->t_data = (HeapTupleHeader) PageGetItem(targpage, itemid);
targtuple->t_len = ItemIdGetLength(itemid);
+ ItemPointerSetOffsetNumber(&targtuple->t_self, hscan->rs_cindex);
switch (HeapTupleSatisfiesVacuum(targtuple, OldestXmin,
hscan->rs_cbuf))
@@ -2291,6 +2294,7 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate,
TupleTableSlot *slot)
{
HeapScanDesc hscan = (HeapScanDesc) scan;
+ HeapTuple tuple = &(hscan->rs_ctup);
TsmRoutine *tsm = scanstate->tsmroutine;
BlockNumber blockno = hscan->rs_cblock;
bool pagemode = (scan->rs_flags & SO_ALLOW_PAGEMODE) != 0;
@@ -2311,6 +2315,9 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate,
!scan->rs_snapshot->takenDuringRecovery;
maxoffset = PageGetMaxOffsetNumber(page);
+ /* block is the same for all tuples, set it once outside the loop */
+ ItemPointerSetBlockNumber(&tuple->t_self, blockno);
+
for (;;)
{
OffsetNumber tupoffset;
@@ -2326,7 +2333,6 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate,
{
ItemId itemid;
bool visible;
- HeapTuple tuple = &(hscan->rs_ctup);
/* Skip invalid tuple pointers. */
itemid = PageGetItemId(page, tupoffset);
@@ -2335,8 +2341,7 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate,
tuple->t_data = (HeapTupleHeader) PageGetItem(page, itemid);
tuple->t_len = ItemIdGetLength(itemid);
- ItemPointerSet(&(tuple->t_self), blockno, tupoffset);
-
+ ItemPointerSetOffsetNumber(&tuple->t_self, blockno);
if (all_visible)
visible = true;
@@ -2578,18 +2583,21 @@ BitmapHeapScanNextBlock(TableScanDesc scan,
* tbmres; but we have to follow any HOT chain starting at each such
* offset.
*/
+ ItemPointerData tid;
int curslot;
/* We must have extracted the tuple offsets by now */
Assert(noffsets > -1);
+ /* block is the same for all tuples, set it once outside the loop */
+ ItemPointerSetBlockNumber(&tid, block);
+
for (curslot = 0; curslot < noffsets; curslot++)
{
OffsetNumber offnum = offsets[curslot];
- ItemPointerData tid;
HeapTupleData heapTuple;
- ItemPointerSet(&tid, block, offnum);
+ ItemPointerSetOffsetNumber(&tid, offnum);
if (heap_hot_search_buffer(&tid, scan->rs_rd, buffer, snapshot,
&heapTuple, NULL, true))
hscan->rs_vistuples[ntup++] = ItemPointerGetOffsetNumber(&tid);
@@ -2604,11 +2612,15 @@ BitmapHeapScanNextBlock(TableScanDesc scan,
Page page = BufferGetPage(buffer);
OffsetNumber maxoff = PageGetMaxOffsetNumber(page);
OffsetNumber offnum;
+ HeapTupleData loctup;
+
+ /* block and tableOid is the same for all tuples, set it once outside the loop */
+ ItemPointerSetBlockNumber(&loctup.t_self, block);
+ loctup.t_tableOid = scan->rs_rd->rd_id;
for (offnum = FirstOffsetNumber; offnum <= maxoff; offnum = OffsetNumberNext(offnum))
{
ItemId lp;
- HeapTupleData loctup;
bool valid;
lp = PageGetItemId(page, offnum);
@@ -2616,8 +2628,8 @@ BitmapHeapScanNextBlock(TableScanDesc scan,
continue;
loctup.t_data = (HeapTupleHeader) PageGetItem(page, lp);
loctup.t_len = ItemIdGetLength(lp);
- loctup.t_tableOid = scan->rs_rd->rd_id;
- ItemPointerSet(&loctup.t_self, block, offnum);
+ ItemPointerSetOffsetNumber(&loctup.t_self, offnum);
+
valid = HeapTupleSatisfiesVisibility(&loctup, snapshot, buffer);
if (valid)
{
[application/octet-stream] pruneheap_set_tuple_block_once.patch (976B, ../../CAEudQAqc8TYoHXGgUcnTJm-nc2TF2WwORm8rh1=nLjMVj1UkbQ@mail.gmail.com/7-pruneheap_set_tuple_block_once.patch)
download | inline diff:
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index a8025889be..7bf73f4d93 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -467,6 +467,9 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer,
prstate.visibility_cutoff_xid = InvalidTransactionId;
maxoff = PageGetMaxOffsetNumber(page);
+
+ /* block and tableOid is the same for all tuples, set it once outside the loop */
+ ItemPointerSetBlockNumber(&tup.t_self, blockno);
tup.t_tableOid = RelationGetRelid(relation);
/*
@@ -540,7 +543,7 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer,
htup = (HeapTupleHeader) PageGetItem(page, itemid);
tup.t_data = htup;
tup.t_len = ItemIdGetLength(itemid);
- ItemPointerSet(&tup.t_self, blockno, offnum);
+ ItemPointerSetOffsetNumber(&tup.t_self, offnum);
prstate.htsv[offnum] = heap_prune_satisfies_vacuum(&prstate, &tup,
buffer);
[application/octet-stream] vacuumlazy_set_tuple_block_once.patch (2.9K, ../../CAEudQAqc8TYoHXGgUcnTJm-nc2TF2WwORm8rh1=nLjMVj1UkbQ@mail.gmail.com/8-vacuumlazy_set_tuple_block_once.patch)
download | inline diff:
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index f28326bad0..175f83cf64 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -2249,6 +2249,7 @@ lazy_scan_noprune(LVRelState *vacrel,
missed_dead_tuples;
bool hastup;
HeapTupleHeader tupleheader;
+ HeapTupleData tuple;
TransactionId NoFreezePageRelfrozenXid = vacrel->NewRelfrozenXid;
MultiXactId NoFreezePageRelminMxid = vacrel->NewRelminMxid;
OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
@@ -2262,13 +2263,16 @@ lazy_scan_noprune(LVRelState *vacrel,
recently_dead_tuples = 0;
missed_dead_tuples = 0;
+ /* block and tableOid is the same for all tuples, set it once outside the loop */
+ ItemPointerSetBlockNumber(&tuple.t_self, blkno);
+ tuple.t_tableOid = RelationGetRelid(vacrel->rel);
+
maxoff = PageGetMaxOffsetNumber(page);
for (offnum = FirstOffsetNumber;
offnum <= maxoff;
offnum = OffsetNumberNext(offnum))
{
ItemId itemid;
- HeapTupleData tuple;
vacrel->offnum = offnum;
itemid = PageGetItemId(page, offnum);
@@ -2326,10 +2330,9 @@ lazy_scan_noprune(LVRelState *vacrel,
*/
}
- ItemPointerSet(&(tuple.t_self), blkno, offnum);
tuple.t_data = (HeapTupleHeader) PageGetItem(page, itemid);
tuple.t_len = ItemIdGetLength(itemid);
- tuple.t_tableOid = RelationGetRelid(vacrel->rel);
+ ItemPointerSetOffsetNumber(&tuple.t_self, offnum);
switch (HeapTupleSatisfiesVacuum(&tuple, vacrel->cutoffs.OldestXmin,
buf))
@@ -3621,17 +3624,21 @@ heap_page_is_all_visible(LVRelState *vacrel, Buffer buf,
OffsetNumber offnum,
maxoff;
bool all_visible = true;
+ HeapTupleData tuple;
*visibility_cutoff_xid = InvalidTransactionId;
*all_frozen = true;
+ /* block and tableOid is the same for all tuples, set it once outside the loop */
+ ItemPointerSetBlockNumber(&tuple.t_self, blockno);
+ tuple.t_tableOid = RelationGetRelid(vacrel->rel);
+
maxoff = PageGetMaxOffsetNumber(page);
for (offnum = FirstOffsetNumber;
offnum <= maxoff && all_visible;
offnum = OffsetNumberNext(offnum))
{
ItemId itemid;
- HeapTupleData tuple;
/*
* Set the offset number so that we can display it along with any
@@ -3644,8 +3651,6 @@ heap_page_is_all_visible(LVRelState *vacrel, Buffer buf,
if (!ItemIdIsUsed(itemid) || ItemIdIsRedirected(itemid))
continue;
- ItemPointerSet(&(tuple.t_self), blockno, offnum);
-
/*
* Dead line pointers can have index pointers pointing to them. So
* they can't be treated as visible
@@ -3661,7 +3666,7 @@ heap_page_is_all_visible(LVRelState *vacrel, Buffer buf,
tuple.t_data = (HeapTupleHeader) PageGetItem(page, itemid);
tuple.t_len = ItemIdGetLength(itemid);
- tuple.t_tableOid = RelationGetRelid(vacrel->rel);
+ ItemPointerSetOffsetNumber(&tuple.t_self, offnum);
switch (HeapTupleSatisfiesVacuum(&tuple, vacrel->cutoffs.OldestXmin,
buf))
[application/octet-stream] dbcommands_set_tuple_block_once.patch (1.1K, ../../CAEudQAqc8TYoHXGgUcnTJm-nc2TF2WwORm8rh1=nLjMVj1UkbQ@mail.gmail.com/9-dbcommands_set_tuple_block_once.patch)
download | inline diff:
diff --git a/src/backend/commands/dbcommands.c b/src/backend/commands/dbcommands.c
index 5fbbcdaabb..9ec07e52e0 100644
--- a/src/backend/commands/dbcommands.c
+++ b/src/backend/commands/dbcommands.c
@@ -337,6 +337,10 @@ ScanSourceDatabasePgClassPage(Page page, Buffer buf, Oid tbid, Oid dbid,
maxoff = PageGetMaxOffsetNumber(page);
+ /* block and tableOid is the same for all tuples, set it once outside the loop */
+ ItemPointerSetBlockNumber(&tuple.t_self, blkno);
+ tuple.t_tableOid = RelationRelationId;
+
/* Loop over offsets. */
for (offnum = FirstOffsetNumber;
offnum <= maxoff;
@@ -352,12 +356,11 @@ ScanSourceDatabasePgClassPage(Page page, Buffer buf, Oid tbid, Oid dbid,
continue;
Assert(ItemIdIsNormal(itemid));
- ItemPointerSet(&(tuple.t_self), blkno, offnum);
/* Initialize a HeapTupleData structure. */
tuple.t_data = (HeapTupleHeader) PageGetItem(page, itemid);
tuple.t_len = ItemIdGetLength(itemid);
- tuple.t_tableOid = RelationRelationId;
+ ItemPointerSetOffsetNumber(&tuple.t_self, offnum);
/* Skip tuples that are not visible to this snapshot. */
if (HeapTupleSatisfiesVisibility(&tuple, snapshot, buf))
^ permalink raw reply [nested|flat] 24+ messages in thread
* Re: Small optimization set tuple block/tableOid once
@ 2025-04-03 10:09 Ranier Vilela <[email protected]>
parent: Ranier Vilela <[email protected]>
0 siblings, 1 reply; 24+ messages in thread
From: Ranier Vilela @ 2025-04-03 10:09 UTC (permalink / raw)
To: pgsql-hackers
rebased heapam.c and heapam_handler.c
best regards,
Ranier Vilela
Attachments:
[application/octet-stream] v1-heapam_set_tuple_block_once.patch (2.7K, ../../CAEudQAqQGT3X_qKirkFCO-ctsDX+0nKc-V5=EAz6WzDYbrcisg@mail.gmail.com/3-v1-heapam_set_tuple_block_once.patch)
download | inline diff:
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index cedaa195cb..148a979699 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -508,13 +508,17 @@ page_collect_tuples(HeapScanDesc scan, Snapshot snapshot,
BlockNumber block, int lines,
bool all_visible, bool check_serializable)
{
+ HeapTupleData loctup;
int ntup = 0;
OffsetNumber lineoff;
+ /* block and tableOid is the same for all tuples, set it once outside the loop */
+ ItemPointerSetBlockNumber(&loctup.t_self, block);
+ loctup.t_tableOid = RelationGetRelid(scan->rs_base.rs_rd);
+
for (lineoff = FirstOffsetNumber; lineoff <= lines; lineoff++)
{
ItemId lpp = PageGetItemId(page, lineoff);
- HeapTupleData loctup;
bool valid;
if (!ItemIdIsNormal(lpp))
@@ -522,8 +526,7 @@ page_collect_tuples(HeapScanDesc scan, Snapshot snapshot,
loctup.t_data = (HeapTupleHeader) PageGetItem(page, lpp);
loctup.t_len = ItemIdGetLength(lpp);
- loctup.t_tableOid = RelationGetRelid(scan->rs_base.rs_rd);
- ItemPointerSet(&(loctup.t_self), block, lineoff);
+ ItemPointerSetOffsetNumber(&loctup.t_self, lineoff);
if (all_visible)
valid = true;
@@ -931,6 +934,10 @@ heapgettup(HeapScanDesc scan,
LockBuffer(scan->rs_cbuf, BUFFER_LOCK_SHARE);
page = heapgettup_start_page(scan, dir, &linesleft, &lineoff);
+
+ /* block is the same for all tuples, set it once outside the loop */
+ ItemPointerSetBlockNumber(&tuple->t_self, scan->rs_cblock);
+
continue_page:
/*
@@ -950,7 +957,7 @@ continue_page:
tuple->t_data = (HeapTupleHeader) PageGetItem(page, lpp);
tuple->t_len = ItemIdGetLength(lpp);
- ItemPointerSet(&(tuple->t_self), scan->rs_cblock, lineoff);
+ ItemPointerSetOffsetNumber(&tuple->t_self, lineoff);
visible = HeapTupleSatisfiesVisibility(tuple,
scan->rs_base.rs_snapshot,
@@ -1766,6 +1773,10 @@ heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
Assert(TransactionIdIsValid(RecentXmin));
Assert(BufferGetBlockNumber(buffer) == blkno);
+ /* block and tableOid is the same for all tuples, set it once outside the loop */
+ ItemPointerSetBlockNumber(&heapTuple->t_self, blkno);
+ heapTuple->t_tableOid = RelationGetRelid(relation);
+
/* Scan through possible multiple members of HOT-chain */
for (;;)
{
@@ -1800,8 +1811,7 @@ heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
*/
heapTuple->t_data = (HeapTupleHeader) PageGetItem(page, lp);
heapTuple->t_len = ItemIdGetLength(lp);
- heapTuple->t_tableOid = RelationGetRelid(relation);
- ItemPointerSet(&heapTuple->t_self, blkno, offnum);
+ ItemPointerSetOffsetNumber(&heapTuple->t_self, offnum);
/*
* Shouldn't see a HEAP_ONLY tuple at chain start.
[application/octet-stream] v1-heapam_handler_set_tuple_block_once.patch (4.9K, ../../CAEudQAqQGT3X_qKirkFCO-ctsDX+0nKc-V5=EAz6WzDYbrcisg@mail.gmail.com/4-v1-heapam_handler_set_tuple_block_once.patch)
download | inline diff:
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 24d3765aa2..5c42bce2de 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -1031,6 +1031,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin,
TupleTableSlot *slot)
{
HeapScanDesc hscan = (HeapScanDesc) scan;
+ HeapTuple targtuple;
Page targpage;
OffsetNumber maxoffset;
BufferHeapTupleTableSlot *hslot;
@@ -1038,14 +1039,18 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin,
Assert(TTS_IS_BUFFERTUPLE(slot));
hslot = (BufferHeapTupleTableSlot *) slot;
+ targtuple = &hslot->base.tupdata;
targpage = BufferGetPage(hscan->rs_cbuf);
maxoffset = PageGetMaxOffsetNumber(targpage);
+ /* block and tableOid is the same for all tuples, set it once outside the loop */
+ ItemPointerSetBlockNumber(&targtuple->t_self, hscan->rs_cblock);
+ targtuple->t_tableOid = RelationGetRelid(scan->rs_rd);
+
/* Inner loop over all tuples on the selected page */
for (; hscan->rs_cindex <= maxoffset; hscan->rs_cindex++)
{
ItemId itemid;
- HeapTuple targtuple = &hslot->base.tupdata;
bool sample_it = false;
itemid = PageGetItemId(targpage, hscan->rs_cindex);
@@ -1063,11 +1068,9 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin,
continue;
}
- ItemPointerSet(&targtuple->t_self, hscan->rs_cblock, hscan->rs_cindex);
-
- targtuple->t_tableOid = RelationGetRelid(scan->rs_rd);
targtuple->t_data = (HeapTupleHeader) PageGetItem(targpage, itemid);
targtuple->t_len = ItemIdGetLength(itemid);
+ ItemPointerSetOffsetNumber(&targtuple->t_self, hscan->rs_cindex);
switch (HeapTupleSatisfiesVacuum(targtuple, OldestXmin,
hscan->rs_cbuf))
@@ -2291,6 +2294,7 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate,
TupleTableSlot *slot)
{
HeapScanDesc hscan = (HeapScanDesc) scan;
+ HeapTuple tuple = &(hscan->rs_ctup);
TsmRoutine *tsm = scanstate->tsmroutine;
BlockNumber blockno = hscan->rs_cblock;
bool pagemode = (scan->rs_flags & SO_ALLOW_PAGEMODE) != 0;
@@ -2311,6 +2315,9 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate,
!scan->rs_snapshot->takenDuringRecovery;
maxoffset = PageGetMaxOffsetNumber(page);
+ /* block is the same for all tuples, set it once outside the loop */
+ ItemPointerSetBlockNumber(&tuple->t_self, blockno);
+
for (;;)
{
OffsetNumber tupoffset;
@@ -2326,7 +2333,6 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate,
{
ItemId itemid;
bool visible;
- HeapTuple tuple = &(hscan->rs_ctup);
/* Skip invalid tuple pointers. */
itemid = PageGetItemId(page, tupoffset);
@@ -2335,8 +2341,7 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate,
tuple->t_data = (HeapTupleHeader) PageGetItem(page, itemid);
tuple->t_len = ItemIdGetLength(itemid);
- ItemPointerSet(&(tuple->t_self), blockno, tupoffset);
-
+ ItemPointerSetOffsetNumber(&tuple->t_self, blockno);
if (all_visible)
visible = true;
@@ -2578,18 +2583,21 @@ BitmapHeapScanNextBlock(TableScanDesc scan,
* tbmres; but we have to follow any HOT chain starting at each such
* offset.
*/
+ ItemPointerData tid;
int curslot;
/* We must have extracted the tuple offsets by now */
Assert(noffsets > -1);
+ /* block is the same for all tuples, set it once outside the loop */
+ ItemPointerSetBlockNumber(&tid, block);
+
for (curslot = 0; curslot < noffsets; curslot++)
{
OffsetNumber offnum = offsets[curslot];
- ItemPointerData tid;
HeapTupleData heapTuple;
- ItemPointerSet(&tid, block, offnum);
+ ItemPointerSetOffsetNumber(&tid, offnum);
if (heap_hot_search_buffer(&tid, scan->rs_rd, buffer, snapshot,
&heapTuple, NULL, true))
hscan->rs_vistuples[ntup++] = ItemPointerGetOffsetNumber(&tid);
@@ -2604,11 +2612,15 @@ BitmapHeapScanNextBlock(TableScanDesc scan,
Page page = BufferGetPage(buffer);
OffsetNumber maxoff = PageGetMaxOffsetNumber(page);
OffsetNumber offnum;
+ HeapTupleData loctup;
+
+ /* block and tableOid is the same for all tuples, set it once outside the loop */
+ ItemPointerSetBlockNumber(&loctup.t_self, block);
+ loctup.t_tableOid = scan->rs_rd->rd_id;
for (offnum = FirstOffsetNumber; offnum <= maxoff; offnum = OffsetNumberNext(offnum))
{
ItemId lp;
- HeapTupleData loctup;
bool valid;
lp = PageGetItemId(page, offnum);
@@ -2616,8 +2628,8 @@ BitmapHeapScanNextBlock(TableScanDesc scan,
continue;
loctup.t_data = (HeapTupleHeader) PageGetItem(page, lp);
loctup.t_len = ItemIdGetLength(lp);
- loctup.t_tableOid = scan->rs_rd->rd_id;
- ItemPointerSet(&loctup.t_self, block, offnum);
+ ItemPointerSetOffsetNumber(&loctup.t_self, offnum);
+
valid = HeapTupleSatisfiesVisibility(&loctup, snapshot, buffer);
if (valid)
{
^ permalink raw reply [nested|flat] 24+ messages in thread
* Re: Small optimization set tuple block/tableOid once
@ 2026-01-04 09:20 David Rowley <[email protected]>
parent: Ranier Vilela <[email protected]>
0 siblings, 0 replies; 24+ messages in thread
From: David Rowley @ 2026-01-04 09:20 UTC (permalink / raw)
To: Ranier Vilela <[email protected]>; +Cc: pgsql-hackers
On Thu, 3 Apr 2025 at 23:09, Ranier Vilela <[email protected]> wrote:
> rebased heapam.c and heapam_handler.c
This doesn't work. The tids are not being set in all cases.
patched:
select tableoid,ctid,oid from pg_class TABLESAMPLE BERNOULLI (100);
tableoid | ctid | oid
----------+---------+-------
1259 | (0,0) | 16409
1259 | (0,0) | 16403
1259 | (0,0) | 16412
1259 | (0,0) | 16407
1259 | (0,0) | 16415
1259 | (0,0) | 16418
1259 | (0,0) | 16434
...
master:
tableoid | ctid | oid
----------+---------+-------
1259 | (0,10) | 16409
1259 | (0,11) | 16403
1259 | (0,12) | 16412
1259 | (0,13) | 16407
1259 | (0,15) | 16415
1259 | (0,16) | 16418
1259 | (0,17) | 16434
...
David
^ permalink raw reply [nested|flat] 24+ messages in thread
end of thread, other threads:[~2026-01-04 09:20 UTC | newest]
Thread overview: 24+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2019-01-05 14:53 Re: [PATCH] Log PostgreSQL version number on startup Peter Eisentraut <[email protected]>
2019-01-16 19:58 ` Peter Eisentraut <[email protected]>
2019-01-29 15:46 ` Christoph Berg <[email protected]>
2019-01-30 22:29 ` Peter Eisentraut <[email protected]>
2019-02-01 11:16 [PATCH] ArchiveOpts structure Alvaro Herrera <[email protected]>
2023-05-31 10:08 [PATCH v37 07/11] Add DISTINCT support for IVM Yugo Nagata <[email protected]>
2023-05-31 10:08 [PATCH v28 07/11] Add DISTINCT support for IVM Yugo Nagata <[email protected]>
2023-05-31 10:08 [PATCH v30 07/11] Add DISTINCT support for IVM Yugo Nagata <[email protected]>
2023-05-31 10:08 [PATCH v31 07/11] Add DISTINCT support for IVM Yugo Nagata <[email protected]>
2023-05-31 10:08 [PATCH v29 07/11] Add DISTINCT support for IVM Yugo Nagata <[email protected]>
2023-05-31 10:08 [PATCH v32 07/11] Add DISTINCT support for IVM Yugo Nagata <[email protected]>
2023-05-31 10:08 [PATCH v37 07/11] Add DISTINCT support for IVM Yugo Nagata <[email protected]>
2023-05-31 10:08 [PATCH v38 07/11] Add DISTINCT support for IVM Yugo Nagata <[email protected]>
2023-05-31 10:08 [PATCH v38 07/11] Add DISTINCT support for IVM Yugo Nagata <[email protected]>
2023-05-31 10:08 [PATCH v37 07/11] Add DISTINCT support for IVM Yugo Nagata <[email protected]>
2023-05-31 10:08 [PATCH v29 07/11] Add DISTINCT support for IVM Yugo Nagata <[email protected]>
2023-05-31 10:08 [PATCH v38 07/11] Add DISTINCT support for IVM Yugo Nagata <[email protected]>
2023-05-31 10:08 [PATCH v30 07/11] Add DISTINCT support for IVM Yugo Nagata <[email protected]>
2023-05-31 10:08 [PATCH v30 07/11] Add DISTINCT support for IVM Yugo Nagata <[email protected]>
2023-05-31 10:08 [PATCH v29 07/11] Add DISTINCT support for IVM Yugo Nagata <[email protected]>
2023-05-31 10:08 [PATCH v30 07/11] Add DISTINCT support for IVM Yugo Nagata <[email protected]>
2025-04-02 11:28 Small optimization set tuple block/tableOid once Ranier Vilela <[email protected]>
2025-04-03 10:09 ` Re: Small optimization set tuple block/tableOid once Ranier Vilela <[email protected]>
2026-01-04 09:20 ` Re: Small optimization set tuple block/tableOid once David Rowley <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox