public inbox for [email protected]help / color / mirror / Atom feed
[PATCH v29 04/11] Add Incremental View Maintenance support to pg_dump 3+ messages / 3 participants [nested] [flat]
* [PATCH v29 04/11] Add Incremental View Maintenance support to pg_dump @ 2020-11-11 08:01 Yugo Nagata <[email protected]> 0 siblings, 0 replies; 3+ messages in thread From: Yugo Nagata @ 2020-11-11 08:01 UTC (permalink / raw) Support CREATE INCREMENTAL MATERIALIZED VIEW syntax. --- src/bin/pg_dump/pg_dump.c | 18 +++++++++++++++--- src/bin/pg_dump/pg_dump.h | 2 ++ src/bin/pg_dump/t/002_pg_dump.pl | 18 ++++++++++++++++++ 3 files changed, 35 insertions(+), 3 deletions(-) diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 65f64c282d..013ead7655 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -6354,6 +6354,7 @@ getTables(Archive *fout, int *numTables) int i_relacl; int i_acldefault; int i_ispartition; + int i_isivm; /* * Find all the tables and table-like objects. @@ -6456,10 +6457,17 @@ getTables(Archive *fout, int *numTables) if (fout->remoteVersion >= 100000) appendPQExpBufferStr(query, - "c.relispartition AS ispartition "); + "c.relispartition AS ispartition, "); else appendPQExpBufferStr(query, - "false AS ispartition "); + "false AS ispartition, "); + + if (fout->remoteVersion >= 170000) + appendPQExpBufferStr(query, + "c.relisivm AS isivm "); + else + appendPQExpBufferStr(query, + "false AS isivm "); /* * Left join to pg_depend to pick up dependency info linking sequences to @@ -6568,6 +6576,7 @@ getTables(Archive *fout, int *numTables) i_relacl = PQfnumber(res, "relacl"); i_acldefault = PQfnumber(res, "acldefault"); i_ispartition = PQfnumber(res, "ispartition"); + i_isivm = PQfnumber(res, "isivm"); if (dopt->lockWaitTimeout) { @@ -6647,6 +6656,7 @@ getTables(Archive *fout, int *numTables) tblinfo[i].amname = pg_strdup(PQgetvalue(res, i, i_amname)); tblinfo[i].is_identity_sequence = (strcmp(PQgetvalue(res, i, i_is_identity_sequence), "t") == 0); tblinfo[i].ispartition = (strcmp(PQgetvalue(res, i, i_ispartition), "t") == 0); + tblinfo[i].isivm = (strcmp(PQgetvalue(res, i, i_isivm), "t") == 0); /* other fields were zeroed above */ @@ -15737,9 +15747,11 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) binary_upgrade_set_pg_class_oids(fout, q, tbinfo->dobj.catId.oid, false); - appendPQExpBuffer(q, "CREATE %s%s %s", + appendPQExpBuffer(q, "CREATE %s%s%s %s", tbinfo->relpersistence == RELPERSISTENCE_UNLOGGED ? "UNLOGGED " : "", + tbinfo->relkind == RELKIND_MATVIEW && tbinfo->isivm ? + "INCREMENTAL " : "", reltypename, qualrelname); diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 9036b13f6a..71ea246abd 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -324,6 +324,8 @@ typedef struct _tableInfo int numParents; /* number of (immediate) parent tables */ struct _tableInfo **parents; /* TableInfos of immediate parents */ + bool isivm; /* is incrementally maintainable materialized view? */ + /* * These fields are computed only if we decide the table is interesting * (it's either a table to dump, or a direct parent of a dumpable table). diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 0758fe5ea0..f83c317268 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2799,6 +2799,24 @@ my %tests = ( }, }, + 'CREATE MATERIALIZED VIEW matview_ivm' => { + create_order => 21, + create_sql => 'CREATE INCREMENTAL MATERIALIZED VIEW dump_test.matview_ivm (col1) AS + SELECT col1 FROM dump_test.test_table;', + regexp => qr/^ + \QCREATE INCREMENTAL MATERIALIZED VIEW dump_test.matview_ivm AS\E + \n\s+\QSELECT col1\E + \n\s+\QFROM dump_test.test_table\E + \n\s+\QWITH NO DATA;\E + /xm, + like => + { %full_runs, %dump_test_schema_runs, section_pre_data => 1, }, + unlike => { + exclude_dump_test_schema => 1, + only_dump_measurement => 1, + }, + }, + 'CREATE POLICY p1 ON test_table' => { create_order => 22, create_sql => 'CREATE POLICY p1 ON dump_test.test_table -- 2.25.1 --Multipart=_Mon__28_Aug_2023_16_05_30_+0900_b1OvQD_3A3ZMTGvj Content-Type: text/x-diff; name="v29-0005-Add-Incremental-View-Maintenance-support-to-psql.patch" Content-Disposition: attachment; filename="v29-0005-Add-Incremental-View-Maintenance-support-to-psql.patch" Content-Transfer-Encoding: 7bit ^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: Logical replication timeout @ 2024-12-23 09:31 RECHTÉ Marc <[email protected]> 0 siblings, 1 reply; 3+ messages in thread From: RECHTÉ Marc @ 2024-12-23 09:31 UTC (permalink / raw) To: Hayato Kuroda (Fujitsu) <[email protected]>; +Cc: [email protected] > Can you enable the parameter "streaming" to on on your system [1]? It allows to > stream the in-progress transactions to the subscriber side. I feel this can avoid > the case that there are many .spill files on the publisher side. > Another approach is to tune the logical_decoding_work_mem parameter [2]. > This specifies the maximum amount of memory used by the logical decoding, and > some changes are spilled when it exceeds the limitation. Naively, this setting > can reduce the number of files. > [1]: https://www.postgresql.org/docs/14/sql-createsubscription.html > [2]: https://www.postgresql.org/docs/14/runtime-config-resource.html#GUC-LOGICAL-DECODING-WORK-MEM > Best regards, > Hayato Kuroda > FUJITSU LIMITED Dear Hayato, Thanks for your suggestions that were both already tested. In our (real) case (a single transaction with 12 millions sub-transactions): 1) setting the subscription as streaming, just delay a bit the spill file surge. It does not prevent the creation of spill files. 2) we set logical_decoding_work_mem to 20GB, which probably also delayed the problem, but did not solve it. The real problem is spill file deletions that can take days in this particular case ! Marc ^ permalink raw reply [nested|flat] 3+ messages in thread
* RE: Logical replication timeout @ 2024-12-23 11:17 Hayato Kuroda (Fujitsu) <[email protected]> parent: RECHTÉ Marc <[email protected]> 0 siblings, 0 replies; 3+ messages in thread From: Hayato Kuroda (Fujitsu) @ 2024-12-23 11:17 UTC (permalink / raw) To: 'RECHTÉ Marc' <[email protected]>; +Cc: [email protected] <[email protected]> Dear Marc, Thanks for the reply! > Thanks for your suggestions that were both already tested. In our (real) case (a > single transaction with 12 millions sub-transactions): > > 1) setting the subscription as streaming, just delay a bit the spill file surge. It does > not prevent the creation of spill files. It is bit surprised for me because I don't know the path which transactions can be serialized even in the streaming=on case. Let me think over it... > 2) we set logical_decoding_work_mem to 20GB, which probably also delayed the > problem, but did not solve it. Oh, I understood that you've already increased the parameter to the appropriate value on your env. Is it right? Best regards, Hayato Kuroda FUJITSU LIMITED ^ permalink raw reply [nested|flat] 3+ messages in thread
end of thread, other threads:[~2024-12-23 11:17 UTC | newest] Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2020-11-11 08:01 [PATCH v29 04/11] Add Incremental View Maintenance support to pg_dump Yugo Nagata <[email protected]> 2024-12-23 09:31 Re: Logical replication timeout RECHTÉ Marc <[email protected]> 2024-12-23 11:17 ` RE: Logical replication timeout Hayato Kuroda (Fujitsu) <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox