agora inbox for [email protected]help / color / mirror / Atom feed
Process startup infrastructure is a mess 93+ messages / 8 participants [nested] [flat]
* Process startup infrastructure is a mess @ 2017-09-14 21:44 Andres Freund <[email protected]> 0 siblings, 2 replies; 93+ messages in thread From: Andres Freund @ 2017-09-14 21:44 UTC (permalink / raw) To: pgsql-hackers Hi, The way we currently start and initialize individual postgres (sub-) processes is pretty complicated and duplicative. I've a couple complaints: 1) It's completely non-obvious that bootstrap.c:AuxiliaryProcessMain() can get invoked both via postmaster for subprocesses (startup, wal writer, bgwriter, checkpointer, wal receiver, "checker"), as well as directly via main.c for the bootstrap processes. Note the autovacuum launcher, archiver, logger are *not* started this way. 2) Most of the processes mentioned in 1) and some additional ones (autovac launcher and walsender / autovacuum workers to some degree) duplicate a lot of logic for startup, error handling and main loop. Especially the error handling code is order dependent, complex, and has been broken in various subprocesses in the past. 3) There exists yet *another* way of starting processes in the form of background workers. Initially that was "just" for extensions, but is now employed for builtin tasks too, like the logical rep launcher / workers. In the course of that special case handling had to be sprinkled around, because the bgworker logic isn't quite able to be good enough for something builtin. 4) The naming of initialization functions is, uh, not particularly clear. How many people can differentiate, without checking, BaseInit(), BackendInitialize(), InitPostgres(), InitProcess(). I think the complexity here just grew incrementally with the addition of more and more subprocesses, without sufficient refactoring. I did some in 31c453165b5a6, but that's not even remotely enough. I think we should seriously consider doing a larger refactoring of this soon. I've some ideas about what to do, but I'd welcome some thoughts on whether others consider this a serious problem or not, and what they think we should do about this, first. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers ^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: Process startup infrastructure is a mess @ 2017-09-15 00:06 Simon Riggs <[email protected]> parent: Andres Freund <[email protected]> 1 sibling, 1 reply; 93+ messages in thread From: Simon Riggs @ 2017-09-15 00:06 UTC (permalink / raw) To: Andres Freund <[email protected]>; +Cc: pgsql-hackers On 14 September 2017 at 22:44, Andres Freund <[email protected]> wrote: > The way we currently start and initialize individual postgres (sub-) > processes is pretty complicated and duplicative. I've a couple > complaints: ... > I think we should seriously consider doing a larger refactoring of this > soon. I've some ideas about what to do, but I'd welcome some thoughts > on whether others consider this a serious problem or not, and what they > think we should do about this, first. Refactoring without a purpose is a negative for me. It takes time, introduces bugs and means the greater code churn over time introduces more bugs because fewer people have seen the code. That is arguable, but when we compare the priority of that against things people want and need there is little contest in my mind. If we add something to an area then its a good time to refactor it since we were going to get bugs anyway. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers ^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: Process startup infrastructure is a mess @ 2017-09-15 00:17 Andres Freund <[email protected]> parent: Simon Riggs <[email protected]> 0 siblings, 1 reply; 93+ messages in thread From: Andres Freund @ 2017-09-15 00:17 UTC (permalink / raw) To: Simon Riggs <[email protected]>; +Cc: pgsql-hackers On 2017-09-15 01:06:54 +0100, Simon Riggs wrote: > On 14 September 2017 at 22:44, Andres Freund <[email protected]> wrote: > > > The way we currently start and initialize individual postgres (sub-) > > processes is pretty complicated and duplicative. I've a couple > > complaints: > ... > > I think we should seriously consider doing a larger refactoring of this > > soon. I've some ideas about what to do, but I'd welcome some thoughts > > on whether others consider this a serious problem or not, and what they > > think we should do about this, first. > > Refactoring without a purpose is a negative for me. It takes time, > introduces bugs and means the greater code churn over time introduces > more bugs because fewer people have seen the code. That is arguable, > but when we compare the priority of that against things people want > and need there is little contest in my mind. That's true for code that's not going to be touched anymore, largely bug free, and can be understood with reasonable effort. Neither is the case here. > If we add something to an area then its a good time to refactor it > since we were going to get bugs anyway. We've added something to the area on a regular basis. As in last in v10. The fundamental problem with your argument is that that makes it impossible for most contributors to get feature in that touch these parts of the code - many neither have the time nor the knowledge to do such a refactoring. By your argument we should have rejected logical replication for v10 because it complicated this further, without cleaning it up. Besides that, it also makes it harder to develop new features, not just to get them in. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers ^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: Process startup infrastructure is a mess @ 2017-09-15 00:43 Tsunakawa, Takayuki <[email protected]> parent: Andres Freund <[email protected]> 1 sibling, 0 replies; 93+ messages in thread From: Tsunakawa, Takayuki @ 2017-09-15 00:43 UTC (permalink / raw) To: 'Andres Freund' <[email protected]>; pgsql-hackers From: [email protected] > [mailto:[email protected]] On Behalf Of Andres Freund > I think we should seriously consider doing a larger refactoring of this > soon. I've some ideas about what to do, but I'd welcome some thoughts on > whether others consider this a serious problem or not, and what they think > we should do about this, first. Thank you for raising this. I think so, too. Some other things that quickly come to mind are: * The signal handlers are similar in many processes and redundant. It's not easy to see how the processes respond to a particular signal and what signal can be used for new things such as dumping the stack trace in the server log. Maybe we can have an array of process attributes that specify the signal handlers, startup/shutdown order, etc. Then the common process management code handles processes based on the information in the array. * postmaster.c is very large (more than 6,000 lines) and hard to read. * It may be easy to forget adding initialization code in the Windows-specific path (SubPostmasterMaain). * It's confusing to find out which process counts toward max_connections, MaxBackends, the count of auxiliary processes. As a user, I'm sometimes confused about the relationship between max_connections, autovacuum_max_workers, max_wal_senders, max_worker_processes. Regards Takayuki Tsunakawa -- Sent via pgsql-hackers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers ^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: Process startup infrastructure is a mess @ 2017-09-15 01:30 Stephen Frost <[email protected]> parent: Andres Freund <[email protected]> 0 siblings, 1 reply; 93+ messages in thread From: Stephen Frost @ 2017-09-15 01:30 UTC (permalink / raw) To: Andres Freund <[email protected]>; +Cc: Simon Riggs <[email protected]>; pgsql-hackers Andres, Simon, * Andres Freund ([email protected]) wrote: > On 2017-09-15 01:06:54 +0100, Simon Riggs wrote: > > If we add something to an area then its a good time to refactor it > > since we were going to get bugs anyway. > > We've added something to the area on a regular basis. As in last in > v10. The fundamental problem with your argument is that that makes it > impossible for most contributors to get feature in that touch these > parts of the code - many neither have the time nor the knowledge to do > such a refactoring. By your argument we should have rejected logical > replication for v10 because it complicated this further, without > cleaning it up. > > Besides that, it also makes it harder to develop new features, not just > to get them in. I'm definitely in agreement with Andres on this one. This isn't refactoring of little-to-never changed code, it's refactoring bits of the system which are changed with some regularity and looks likely to continue to need change as we add more features moving forward, and perhaps add greater controls over process startup. Thanks! Stephen Attachments: [application/pgp-signature] signature.asc (819B, ../../[email protected]/2-signature.asc) download ^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: Process startup infrastructure is a mess @ 2017-09-15 15:39 Robert Haas <[email protected]> parent: Stephen Frost <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Robert Haas @ 2017-09-15 15:39 UTC (permalink / raw) To: Stephen Frost <[email protected]>; +Cc: Andres Freund <[email protected]>; Simon Riggs <[email protected]>; pgsql-hackers On Thu, Sep 14, 2017 at 9:30 PM, Stephen Frost <[email protected]> wrote: > I'm definitely in agreement with Andres on this one. This isn't > refactoring of little-to-never changed code, it's refactoring bits of > the system which are changed with some regularity and looks likely to > continue to need change as we add more features moving forward, and > perhaps add greater controls over process startup. I agree to some extent. I think this is a mess and that cleaning it up would be better than not cleaning it up. I don't view it as a particularly urgent problem, though, and I'm not sure I see a need to attack it in a monolithic way. I think that unifying the error recovery paths in various processes would actually be more useful than unifying process startup, but that's not to say unifying process startup wouldn't be nice. Generally, I think the newer process startup methods are superior to the older ones. The new background worker mechanism provides a flexible way of adding new processes whose exact number doesn't need to be known in advance without having to tinker with so much code. In contrast, adding an auxiliary process is big nuisance. So if I were going to attack this problem, I'd try to make the background worker machinery sufficiently capable that we can do everything that way, and then gradually move more things over to that mechanism. However, I honestly probably wouldn't bother with this unless it were blocking something specific that I wanted to do. I wouldn't go as far as Simon did in his reply; I would not favor refusing a good patch from a highly qualified contributor that makes this all less ugly and fragile. On the other hand, I think he's right that we wouldn't necessarily get a lot of immediate benefit out of it apart from feeling good about having done it. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 24 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Tue_Jan_19_13_48_31_2021_529)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Wed_Jan_20_17_34_44_2021_328)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Thu_Jan_21_00_28_44_2021_003)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v5-0003-Poc-Keep-page-LSN-updated-while-WAL-skipping.patch" ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 9 ++++++++- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 31 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..1e2c11fdd3 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -552,9 +552,16 @@ typedef struct ViewOptions (relation)->rd_smgr->smgr_targblock = (targblock); \ } while (0) +/* + * RelationIsPermanent + * True if relation is WAL-logged. + */ +#define RelationIsWalLogged(relation) \ + ((relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT) + /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..35acccb29c 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + rel->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Mon_Jan_18_15_08_38_2021_069)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 24 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Tue_Jan_19_13_48_31_2021_529)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Wed_Jan_20_17_34_44_2021_328)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Thu_Jan_21_00_28_44_2021_003)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v5-0003-Poc-Keep-page-LSN-updated-while-WAL-skipping.patch" ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 9 ++++++++- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 31 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..1e2c11fdd3 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -552,9 +552,16 @@ typedef struct ViewOptions (relation)->rd_smgr->smgr_targblock = (targblock); \ } while (0) +/* + * RelationIsPermanent + * True if relation is WAL-logged. + */ +#define RelationIsWalLogged(relation) \ + ((relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT) + /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..35acccb29c 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + rel->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Mon_Jan_18_15_08_38_2021_069)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 24 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Tue_Jan_19_13_48_31_2021_529)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Wed_Jan_20_17_34_44_2021_328)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Thu_Jan_21_00_28_44_2021_003)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v5-0003-Poc-Keep-page-LSN-updated-while-WAL-skipping.patch" ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 9 ++++++++- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 31 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..1e2c11fdd3 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -552,9 +552,16 @@ typedef struct ViewOptions (relation)->rd_smgr->smgr_targblock = (targblock); \ } while (0) +/* + * RelationIsPermanent + * True if relation is WAL-logged. + */ +#define RelationIsWalLogged(relation) \ + ((relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT) + /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..35acccb29c 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + rel->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Mon_Jan_18_15_08_38_2021_069)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 24 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Tue_Jan_19_13_48_31_2021_529)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Wed_Jan_20_17_34_44_2021_328)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Thu_Jan_21_00_28_44_2021_003)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v5-0003-Poc-Keep-page-LSN-updated-while-WAL-skipping.patch" ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 9 ++++++++- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 31 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..1e2c11fdd3 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -552,9 +552,16 @@ typedef struct ViewOptions (relation)->rd_smgr->smgr_targblock = (targblock); \ } while (0) +/* + * RelationIsPermanent + * True if relation is WAL-logged. + */ +#define RelationIsWalLogged(relation) \ + ((relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT) + /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..35acccb29c 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + rel->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Mon_Jan_18_15_08_38_2021_069)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 24 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Tue_Jan_19_13_48_31_2021_529)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Wed_Jan_20_17_34_44_2021_328)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Thu_Jan_21_00_28_44_2021_003)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v5-0003-Poc-Keep-page-LSN-updated-while-WAL-skipping.patch" ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 9 ++++++++- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 31 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..1e2c11fdd3 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -552,9 +552,16 @@ typedef struct ViewOptions (relation)->rd_smgr->smgr_targblock = (targblock); \ } while (0) +/* + * RelationIsPermanent + * True if relation is WAL-logged. + */ +#define RelationIsWalLogged(relation) \ + ((relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT) + /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..35acccb29c 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + rel->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Mon_Jan_18_15_08_38_2021_069)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 24 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Tue_Jan_19_13_48_31_2021_529)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Wed_Jan_20_17_34_44_2021_328)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Thu_Jan_21_00_28_44_2021_003)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v5-0003-Poc-Keep-page-LSN-updated-while-WAL-skipping.patch" ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 9 ++++++++- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 31 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..1e2c11fdd3 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -552,9 +552,16 @@ typedef struct ViewOptions (relation)->rd_smgr->smgr_targblock = (targblock); \ } while (0) +/* + * RelationIsPermanent + * True if relation is WAL-logged. + */ +#define RelationIsWalLogged(relation) \ + ((relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT) + /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..35acccb29c 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + rel->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Mon_Jan_18_15_08_38_2021_069)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 24 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Tue_Jan_19_13_48_31_2021_529)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Wed_Jan_20_17_34_44_2021_328)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Thu_Jan_21_00_28_44_2021_003)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v5-0003-Poc-Keep-page-LSN-updated-while-WAL-skipping.patch" ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 9 ++++++++- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 31 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..1e2c11fdd3 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -552,9 +552,16 @@ typedef struct ViewOptions (relation)->rd_smgr->smgr_targblock = (targblock); \ } while (0) +/* + * RelationIsPermanent + * True if relation is WAL-logged. + */ +#define RelationIsWalLogged(relation) \ + ((relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT) + /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..35acccb29c 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + rel->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Mon_Jan_18_15_08_38_2021_069)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 24 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Tue_Jan_19_13_48_31_2021_529)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Wed_Jan_20_17_34_44_2021_328)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Thu_Jan_21_00_28_44_2021_003)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v5-0003-Poc-Keep-page-LSN-updated-while-WAL-skipping.patch" ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 9 ++++++++- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 31 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..1e2c11fdd3 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -552,9 +552,16 @@ typedef struct ViewOptions (relation)->rd_smgr->smgr_targblock = (targblock); \ } while (0) +/* + * RelationIsPermanent + * True if relation is WAL-logged. + */ +#define RelationIsWalLogged(relation) \ + ((relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT) + /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..35acccb29c 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + rel->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Mon_Jan_18_15_08_38_2021_069)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 24 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Tue_Jan_19_13_48_31_2021_529)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Wed_Jan_20_17_34_44_2021_328)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Thu_Jan_21_00_28_44_2021_003)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v5-0003-Poc-Keep-page-LSN-updated-while-WAL-skipping.patch" ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 9 ++++++++- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 31 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..1e2c11fdd3 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -552,9 +552,16 @@ typedef struct ViewOptions (relation)->rd_smgr->smgr_targblock = (targblock); \ } while (0) +/* + * RelationIsPermanent + * True if relation is WAL-logged. + */ +#define RelationIsWalLogged(relation) \ + ((relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT) + /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..35acccb29c 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + rel->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Mon_Jan_18_15_08_38_2021_069)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 24 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Tue_Jan_19_13_48_31_2021_529)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Wed_Jan_20_17_34_44_2021_328)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Thu_Jan_21_00_28_44_2021_003)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v5-0003-Poc-Keep-page-LSN-updated-while-WAL-skipping.patch" ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 9 ++++++++- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 31 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..1e2c11fdd3 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -552,9 +552,16 @@ typedef struct ViewOptions (relation)->rd_smgr->smgr_targblock = (targblock); \ } while (0) +/* + * RelationIsPermanent + * True if relation is WAL-logged. + */ +#define RelationIsWalLogged(relation) \ + ((relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT) + /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..35acccb29c 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + rel->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Mon_Jan_18_15_08_38_2021_069)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 24 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Tue_Jan_19_13_48_31_2021_529)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Wed_Jan_20_17_34_44_2021_328)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Thu_Jan_21_00_28_44_2021_003)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v5-0003-Poc-Keep-page-LSN-updated-while-WAL-skipping.patch" ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 9 ++++++++- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 31 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..1e2c11fdd3 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -552,9 +552,16 @@ typedef struct ViewOptions (relation)->rd_smgr->smgr_targblock = (targblock); \ } while (0) +/* + * RelationIsPermanent + * True if relation is WAL-logged. + */ +#define RelationIsWalLogged(relation) \ + ((relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT) + /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..35acccb29c 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + rel->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Mon_Jan_18_15_08_38_2021_069)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 24 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Tue_Jan_19_13_48_31_2021_529)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Wed_Jan_20_17_34_44_2021_328)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Thu_Jan_21_00_28_44_2021_003)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v5-0003-Poc-Keep-page-LSN-updated-while-WAL-skipping.patch" ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 9 ++++++++- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 31 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..1e2c11fdd3 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -552,9 +552,16 @@ typedef struct ViewOptions (relation)->rd_smgr->smgr_targblock = (targblock); \ } while (0) +/* + * RelationIsPermanent + * True if relation is WAL-logged. + */ +#define RelationIsWalLogged(relation) \ + ((relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT) + /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..35acccb29c 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + rel->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Mon_Jan_18_15_08_38_2021_069)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 24 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Tue_Jan_19_13_48_31_2021_529)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Wed_Jan_20_17_34_44_2021_328)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Thu_Jan_21_00_28_44_2021_003)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v5-0003-Poc-Keep-page-LSN-updated-while-WAL-skipping.patch" ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 9 ++++++++- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 31 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..1e2c11fdd3 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -552,9 +552,16 @@ typedef struct ViewOptions (relation)->rd_smgr->smgr_targblock = (targblock); \ } while (0) +/* + * RelationIsPermanent + * True if relation is WAL-logged. + */ +#define RelationIsWalLogged(relation) \ + ((relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT) + /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..35acccb29c 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + rel->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Mon_Jan_18_15_08_38_2021_069)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 24 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Tue_Jan_19_13_48_31_2021_529)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Wed_Jan_20_17_34_44_2021_328)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Thu_Jan_21_00_28_44_2021_003)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v5-0003-Poc-Keep-page-LSN-updated-while-WAL-skipping.patch" ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 9 ++++++++- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 31 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..1e2c11fdd3 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -552,9 +552,16 @@ typedef struct ViewOptions (relation)->rd_smgr->smgr_targblock = (targblock); \ } while (0) +/* + * RelationIsPermanent + * True if relation is WAL-logged. + */ +#define RelationIsWalLogged(relation) \ + ((relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT) + /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..35acccb29c 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + rel->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Mon_Jan_18_15_08_38_2021_069)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 24 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Tue_Jan_19_13_48_31_2021_529)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Wed_Jan_20_17_34_44_2021_328)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Thu_Jan_21_00_28_44_2021_003)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v5-0003-Poc-Keep-page-LSN-updated-while-WAL-skipping.patch" ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 9 ++++++++- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 31 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..1e2c11fdd3 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -552,9 +552,16 @@ typedef struct ViewOptions (relation)->rd_smgr->smgr_targblock = (targblock); \ } while (0) +/* + * RelationIsPermanent + * True if relation is WAL-logged. + */ +#define RelationIsWalLogged(relation) \ + ((relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT) + /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..35acccb29c 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + rel->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Mon_Jan_18_15_08_38_2021_069)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 24 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Tue_Jan_19_13_48_31_2021_529)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Wed_Jan_20_17_34_44_2021_328)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Thu_Jan_21_00_28_44_2021_003)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v5-0003-Poc-Keep-page-LSN-updated-while-WAL-skipping.patch" ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 9 ++++++++- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 31 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..1e2c11fdd3 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -552,9 +552,16 @@ typedef struct ViewOptions (relation)->rd_smgr->smgr_targblock = (targblock); \ } while (0) +/* + * RelationIsPermanent + * True if relation is WAL-logged. + */ +#define RelationIsWalLogged(relation) \ + ((relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT) + /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..35acccb29c 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + rel->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Mon_Jan_18_15_08_38_2021_069)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 24 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Tue_Jan_19_13_48_31_2021_529)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Wed_Jan_20_17_34_44_2021_328)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Thu_Jan_21_00_28_44_2021_003)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v5-0003-Poc-Keep-page-LSN-updated-while-WAL-skipping.patch" ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 9 ++++++++- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 31 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..1e2c11fdd3 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -552,9 +552,16 @@ typedef struct ViewOptions (relation)->rd_smgr->smgr_targblock = (targblock); \ } while (0) +/* + * RelationIsPermanent + * True if relation is WAL-logged. + */ +#define RelationIsWalLogged(relation) \ + ((relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT) + /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..35acccb29c 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + rel->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Mon_Jan_18_15_08_38_2021_069)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 24 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Tue_Jan_19_13_48_31_2021_529)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Wed_Jan_20_17_34_44_2021_328)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Thu_Jan_21_00_28_44_2021_003)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v5-0003-Poc-Keep-page-LSN-updated-while-WAL-skipping.patch" ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 9 ++++++++- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 31 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..1e2c11fdd3 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -552,9 +552,16 @@ typedef struct ViewOptions (relation)->rd_smgr->smgr_targblock = (targblock); \ } while (0) +/* + * RelationIsPermanent + * True if relation is WAL-logged. + */ +#define RelationIsWalLogged(relation) \ + ((relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT) + /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..35acccb29c 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + rel->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Mon_Jan_18_15_08_38_2021_069)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Wed_Jan_20_17_34_44_2021_328)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Thu_Jan_21_00_28_44_2021_003)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v5-0003-Poc-Keep-page-LSN-updated-while-WAL-skipping.patch" ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 9 ++++++++- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 31 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..1e2c11fdd3 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -552,9 +552,16 @@ typedef struct ViewOptions (relation)->rd_smgr->smgr_targblock = (targblock); \ } while (0) +/* + * RelationIsPermanent + * True if relation is WAL-logged. + */ +#define RelationIsWalLogged(relation) \ + ((relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT) + /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..35acccb29c 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + rel->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Mon_Jan_18_15_08_38_2021_069)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 24 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Tue_Jan_19_13_48_31_2021_529)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Wed_Jan_20_17_34_44_2021_328)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Thu_Jan_21_00_28_44_2021_003)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v5-0003-Poc-Keep-page-LSN-updated-while-WAL-skipping.patch" ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 24 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Tue_Jan_19_13_48_31_2021_529)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 9 ++++++++- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 31 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..1e2c11fdd3 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -552,9 +552,16 @@ typedef struct ViewOptions (relation)->rd_smgr->smgr_targblock = (targblock); \ } while (0) +/* + * RelationIsPermanent + * True if relation is WAL-logged. + */ +#define RelationIsWalLogged(relation) \ + ((relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT) + /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..35acccb29c 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + rel->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Mon_Jan_18_15_08_38_2021_069)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 24 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Tue_Jan_19_13_48_31_2021_529)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Wed_Jan_20_17_34_44_2021_328)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/storage/bufmgr.h | 8 +++++++- src/include/utils/rel.h | 2 +- src/include/utils/snapmgr.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index fb00fda6a7..e641174798 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -14,6 +14,7 @@ #ifndef BUFMGR_H #define BUFMGR_H +#include "access/xlog.h" #include "storage/block.h" #include "storage/buf.h" #include "storage/bufpage.h" @@ -278,12 +279,17 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page) { Assert(relation != NULL); + /* + * While wal_level=minimal, early pruning can happen without updating page + * LSN. Don't take the fast-return path while wal_level=minimal so that we + * don't miss early pruning. + */ if (old_snapshot_threshold >= 0 && (snapshot) != NULL && ((snapshot)->snapshot_type == SNAPSHOT_MVCC || (snapshot)->snapshot_type == SNAPSHOT_TOAST) && !XLogRecPtrIsInvalid((snapshot)->lsn) - && PageGetLSN(page) > (snapshot)->lsn) + && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn)) TestForOldSnapshot_impl(snapshot, relation); } diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..f58d65cf28 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -554,7 +554,7 @@ typedef struct ViewOptions /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..c21ee3c289 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) -- 2.27.0 ----Next_Part(Thu_Jan_21_00_28_44_2021_003)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v5-0003-Poc-Keep-page-LSN-updated-while-WAL-skipping.patch" ^ permalink raw reply [nested|flat] 93+ messages in thread
* [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence @ 2021-01-18 05:47 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Kyotaro Horiguchi @ 2021-01-18 05:47 UTC (permalink / raw) RelationNeedsWAL() may return false for a permanent relation when wal_level=minimal and the relation is created or truncated in the current transaction. Directly examine relpersistence instead of using the function to know relation persistence. --- src/backend/catalog/pg_publication.c | 2 +- src/backend/optimizer/util/plancat.c | 3 ++- src/include/utils/rel.h | 9 ++++++++- src/include/utils/snapmgr.h | 2 +- src/test/subscription/t/001_rep_changes.pl | 20 +++++++++++++++++++- 5 files changed, 31 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 5f8e1c64e1..84d2efcfd2 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel) errdetail("System tables cannot be added to publications."))); /* UNLOGGED and TEMP relations cannot be part of publication. */ - if (!RelationNeedsWAL(targetrel)) + if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("table \"%s\" cannot be replicated", diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..177e6e336a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -126,7 +126,8 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, relation = table_open(relationObjectId, NoLock); /* Temporary and unlogged relations are inaccessible during recovery. */ - if (!RelationNeedsWAL(relation) && RecoveryInProgress()) + if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT && + RecoveryInProgress()) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot access temporary or unlogged relations during recovery"))); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 10b63982c0..1e2c11fdd3 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -552,9 +552,16 @@ typedef struct ViewOptions (relation)->rd_smgr->smgr_targblock = (targblock); \ } while (0) +/* + * RelationIsPermanent + * True if relation is WAL-logged. + */ +#define RelationIsWalLogged(relation) \ + ((relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT) + /* * RelationNeedsWAL - * True if relation needs WAL. + * True if relation needs WAL at the time. * * Returns false if wal_level = minimal and this relation is created or * truncated in the current transaction. See "Skipping WAL for New diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..35acccb29c 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -37,7 +37,7 @@ */ #define RelationAllowsEarlyPruning(rel) \ ( \ - RelationNeedsWAL(rel) \ + rel->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \ && !IsCatalogRelation(rel) \ && !RelationIsAccessibleInLogicalDecoding(rel) \ ) diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl index 0680f44a1a..ed9b48e8bc 100644 --- a/src/test/subscription/t/001_rep_changes.pl +++ b/src/test/subscription/t/001_rep_changes.pl @@ -3,7 +3,7 @@ use strict; use warnings; use PostgresNode; use TestLib; -use Test::More tests => 23; +use Test::More tests => 24; # Initialize publisher node my $node_publisher = get_new_node('publisher'); @@ -358,3 +358,21 @@ is($result, qq(0), 'check replication origin was dropped on subscriber'); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); + +# +# CREATE PUBLICATION while wal_level=mimal should succeed, with a WARNING +$node_publisher->append_conf( + 'postgresql.conf', qq( +wal_level=minimal +max_wal_senders=0 +)); +$node_publisher->start; +($result, my $retout, my $reterr) = $node_publisher->psql( + 'postgres', qq{ +BEGIN; +CREATE TABLE skip_wal(); +CREATE PUBLICATION tap_pub2 FOR TABLE skip_wal; +ROLLBACK; +}); +ok($reterr =~ m/WARNING: wal_level is insufficient to publish logical changes/, + 'test CREATE PUBLICATION can be done while wal_leve=minimal'); -- 2.27.0 ----Next_Part(Mon_Jan_18_15_08_38_2021_069)---- ^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: [PATCH] allow src/tools/msvc/clean.bat script to be called from the root of the source tree @ 2022-01-18 16:49 Andrew Dunstan <[email protected]> 0 siblings, 1 reply; 93+ messages in thread From: Andrew Dunstan @ 2022-01-18 16:49 UTC (permalink / raw) To: Juan José Santamaría Flecha <[email protected]>; PostgreSQL Hackers <[email protected]> On 1/18/22 04:41, Juan José Santamaría Flecha wrote: > In [1] capacity for $SUBJECT was added to most of the batch scripts, > but clean.bat was not included. I propose to do so with the attached > patch. That looks a bit ugly. How about this (untested) instead? > > By the way, are pgbison.bat and pgflex.bat directly called anywhere? Not to my knowledge. One of the things that's annoying about them is that the processor names are hardcoded, so if you install winflexbison as I usually do you have to rename the executables (or rename the chocolatey shims) or the scripts won't work. cheers andrew -- Andrew Dunstan EDB: https://www.enterprisedb.com Attachments: [text/x-patch] clean-from-anywhere.patch (675B, ../../[email protected]/2-clean-from-anywhere.patch) download | inline diff: diff --git a/src/tools/msvc/clean.bat b/src/tools/msvc/clean.bat index 0cc91e7d6c..42b3d1b87f 100755 --- a/src/tools/msvc/clean.bat +++ b/src/tools/msvc/clean.bat @@ -4,8 +4,9 @@ REM src/tools/msvc/clean.bat set DIST=0 if "%1"=="dist" set DIST=1 -set D=%CD% -if exist ..\msvc if exist ..\..\..\src cd ..\..\.. +setlocal + +cd "%~dp0\..\..\.." if exist debug rd /s /q debug if exist release rd /s /q release @@ -133,7 +134,7 @@ REM Clean up datafiles built with contrib REM cd contrib REM for /r %%f in (*.sql) do if exist %%f.in del %%f -cd %D% +cd "%~dp0" REM Clean up ecpg regression test files msbuild ecpg_regression.proj /NoLogo /v:q %MSBFLAGS% /t:clean ^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: [PATCH] allow src/tools/msvc/clean.bat script to be called from the root of the source tree @ 2022-01-18 20:08 Juan José Santamaría Flecha <[email protected]> parent: Andrew Dunstan <[email protected]> 0 siblings, 1 reply; 93+ messages in thread From: Juan José Santamaría Flecha @ 2022-01-18 20:08 UTC (permalink / raw) To: Andrew Dunstan <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]> On Tue, Jan 18, 2022 at 5:49 PM Andrew Dunstan <[email protected]> wrote: > > On 1/18/22 04:41, Juan José Santamaría Flecha wrote: > > In [1] capacity for $SUBJECT was added to most of the batch scripts, > > but clean.bat was not included. I propose to do so with the attached > > patch. > > > That looks a bit ugly. How about this (untested) instead? > > It is WFM, so I am fine with it. > > > By the way, are pgbison.bat and pgflex.bat directly called anywhere? > > Not to my knowledge. One of the things that's annoying about them is > that the processor names are hardcoded, so if you install winflexbison > as I usually do you have to rename the executables (or rename the > chocolatey shims) or the scripts won't work. > > We could use those batches to get to the hardcoded name, but that would probably be as annoying as renaming. If those batches serve no purpose right now, it should be fine to remove them from the tree. I use the executables from a MinGW installation, and they keep their actual name. Regards, Juan José Santamaría Flecha ^ permalink raw reply [nested|flat] 93+ messages in thread
* Re: [PATCH] allow src/tools/msvc/clean.bat script to be called from the root of the source tree @ 2022-01-18 21:17 Andrew Dunstan <[email protected]> parent: Juan José Santamaría Flecha <[email protected]> 0 siblings, 0 replies; 93+ messages in thread From: Andrew Dunstan @ 2022-01-18 21:17 UTC (permalink / raw) To: Juan José Santamaría Flecha <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]> On 1/18/22 15:08, Juan José Santamaría Flecha wrote: > > > > > By the way, are pgbison.bat and pgflex.bat directly called anywhere? > > Not to my knowledge. One of the things that's annoying about them is > that the processor names are hardcoded, so if you install winflexbison > as I usually do you have to rename the executables (or rename the > chocolatey shims) or the scripts won't work. > > We could use those batches to get to the hardcoded name, but that > would probably be as annoying as renaming. If those batches serve no > purpose right now, it should be fine to remove them from the tree. I > use the executables from a MinGW installation, and they keep their > actual name. > OK, but I don't think we should require installation of MinGW for an MSVC build. cheers andrew -- Andrew Dunstan EDB: https://www.enterprisedb.com ^ permalink raw reply [nested|flat] 93+ messages in thread
end of thread, other threads:[~2022-01-18 21:17 UTC | newest] Thread overview: 93+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2017-09-14 21:44 Process startup infrastructure is a mess Andres Freund <[email protected]> 2017-09-15 00:06 ` Simon Riggs <[email protected]> 2017-09-15 00:17 ` Andres Freund <[email protected]> 2017-09-15 01:30 ` Stephen Frost <[email protected]> 2017-09-15 15:39 ` Robert Haas <[email protected]> 2017-09-15 00:43 ` Tsunakawa, Takayuki <[email protected]> 2021-01-18 05:47 [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v2] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v5 2/3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v3] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2021-01-18 05:47 [PATCH v4] Do not use RelationNeedsWAL to identify relation persistence Kyotaro Horiguchi <[email protected]> 2022-01-18 16:49 Re: [PATCH] allow src/tools/msvc/clean.bat script to be called from the root of the source tree Andrew Dunstan <[email protected]> 2022-01-18 20:08 ` Re: [PATCH] allow src/tools/msvc/clean.bat script to be called from the root of the source tree Juan José Santamaría Flecha <[email protected]> 2022-01-18 21:17 ` Re: [PATCH] allow src/tools/msvc/clean.bat script to be called from the root of the source tree Andrew Dunstan <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox