public inbox for [email protected]
help / color / mirror / Atom feedTrouble with amcheck
36+ messages / 12 participants
[nested] [flat]
* Trouble with amcheck
@ 2017-09-15 00:03 Douglas Doole <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Douglas Doole @ 2017-09-15 00:03 UTC (permalink / raw)
To: pgsql-hackers
I just cloned PostgreSQL to a new machine today (Ubuntu 17.04). "make
install" and "make check-world" run fine but "make installcheck-world" is
having trouble with amcheck:
In contrib/amcheck/results:
CREATE EXTENSION amcheck;
ERROR: could not open extension control file
"/home/doole/pgCommunity/install/share/postgresql/extension/amcheck.control":
No such file or directory
I expect I'm missing something in the machine set up, but I'm stumped as to
what.
Any suggestions?
Thanks
- Doug
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Trouble with amcheck
@ 2017-09-15 00:11 Peter Geoghegan <[email protected]>
parent: Douglas Doole <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Peter Geoghegan @ 2017-09-15 00:11 UTC (permalink / raw)
To: Douglas Doole <[email protected]>; +Cc: pgsql-hackers
On Thu, Sep 14, 2017 at 5:03 PM, Douglas Doole <[email protected]> wrote:
> I just cloned PostgreSQL to a new machine today (Ubuntu 17.04). "make
> install" and "make check-world" run fine but "make installcheck-world" is
> having trouble with amcheck:
>
> In contrib/amcheck/results:
>
> CREATE EXTENSION amcheck;
> ERROR: could not open extension control file
> "/home/doole/pgCommunity/install/share/postgresql/extension/amcheck.control":
> No such file or directory
>
> I expect I'm missing something in the machine set up, but I'm stumped as to
> what.
I think you need to build and install contrib, so that it is available
to the server that you're running an installcheck against. amcheck is
alphabetically first among contrib modules that have tests, IIRC.
--
Peter Geoghegan
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Trouble with amcheck
@ 2017-09-15 01:31 Stephen Frost <[email protected]>
parent: Peter Geoghegan <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Stephen Frost @ 2017-09-15 01:31 UTC (permalink / raw)
To: Peter Geoghegan <[email protected]>; +Cc: Douglas Doole <[email protected]>; pgsql-hackers
Peter, Douglas,
* Peter Geoghegan ([email protected]) wrote:
> On Thu, Sep 14, 2017 at 5:03 PM, Douglas Doole <[email protected]> wrote:
> > I just cloned PostgreSQL to a new machine today (Ubuntu 17.04). "make
> > install" and "make check-world" run fine but "make installcheck-world" is
> > having trouble with amcheck:
> >
> > In contrib/amcheck/results:
> >
> > CREATE EXTENSION amcheck;
> > ERROR: could not open extension control file
> > "/home/doole/pgCommunity/install/share/postgresql/extension/amcheck.control":
> > No such file or directory
> >
> > I expect I'm missing something in the machine set up, but I'm stumped as to
> > what.
>
> I think you need to build and install contrib, so that it is available
> to the server that you're running an installcheck against. amcheck is
> alphabetically first among contrib modules that have tests, IIRC.
Yes, I was working with someone earlier today who ran into exactly the
same issue. If you don't 'make world' or make the individual contrib
modules, then 'make installcheck-world' isn't going to work.
I do think it'd be nice if we could provide a better error message in
such a case..
Thanks!
Stephen
Attachments:
[application/pgp-signature] signature.asc (819B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Trouble with amcheck
@ 2017-09-15 01:35 Michael Paquier <[email protected]>
parent: Stephen Frost <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Michael Paquier @ 2017-09-15 01:35 UTC (permalink / raw)
To: Stephen Frost <[email protected]>; +Cc: Peter Geoghegan <[email protected]>; Douglas Doole <[email protected]>; pgsql-hackers
On Fri, Sep 15, 2017 at 10:31 AM, Stephen Frost <[email protected]> wrote:
> Yes, I was working with someone earlier today who ran into exactly the
> same issue. If you don't 'make world' or make the individual contrib
> modules, then 'make installcheck-world' isn't going to work.
Or should installcheck-world imply world? It is easy for newcomers or
even advanced hackers to fall into this trap from time to time.
--
Michael
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Trouble with amcheck
@ 2017-09-15 02:22 Douglas Doole <[email protected]>
parent: Michael Paquier <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Douglas Doole @ 2017-09-15 02:22 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; Stephen Frost <[email protected]>; +Cc: Peter Geoghegan <[email protected]>; pgsql-hackers
Thanks all. Making and installing the contribs got me rolling again. (I
tried "make world" but ran into trouble with the XML docs. But that's pain
and suffering for another day.)
I'd agree that "make installcheck-world" should imply that all prereqs are
met - that's certainsly the normal behaviour for make.
- Doug
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Trouble with amcheck
@ 2017-09-15 02:29 Andres Freund <[email protected]>
parent: Douglas Doole <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Andres Freund @ 2017-09-15 02:29 UTC (permalink / raw)
To: Douglas Doole <[email protected]>; +Cc: Michael Paquier <[email protected]>; Stephen Frost <[email protected]>; Peter Geoghegan <[email protected]>; pgsql-hackers
On 2017-09-15 02:22:49 +0000, Douglas Doole wrote:
> Thanks all. Making and installing the contribs got me rolling again. (I
> tried "make world" but ran into trouble with the XML docs. But that's pain
> and suffering for another day.)
>
> I'd agree that "make installcheck-world" should imply that all prereqs are
> met - that's certainsly the normal behaviour for make.
I'm very unconvinced by this, given that one use of installcheck is to
run against an existing server. For which one might not even have access
to the relevant directories to install extensions into.
Regards,
Andres Freund
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Trouble with amcheck
@ 2017-09-15 02:36 Stephen Frost <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 2 replies; 36+ messages in thread
From: Stephen Frost @ 2017-09-15 02:36 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Douglas Doole <[email protected]>; Michael Paquier <[email protected]>; Peter Geoghegan <[email protected]>; pgsql-hackers
Andres,
* Andres Freund ([email protected]) wrote:
> On 2017-09-15 02:22:49 +0000, Douglas Doole wrote:
> > Thanks all. Making and installing the contribs got me rolling again. (I
> > tried "make world" but ran into trouble with the XML docs. But that's pain
> > and suffering for another day.)
> >
> > I'd agree that "make installcheck-world" should imply that all prereqs are
> > met - that's certainsly the normal behaviour for make.
>
> I'm very unconvinced by this, given that one use of installcheck is to
> run against an existing server. For which one might not even have access
> to the relevant directories to install extensions into.
Sure, but if the extensions aren't in place and you're trying to run
make installcheck-world, it's not like it's somehow going to succeed.
Failing earlier on the install seems like a reasonable thing to do
rather than failing later halfway through the check process.
Now, that said, perhaps a bit more smarts would be in order here to,
instead, check that the extensions are available before trying to run
the checks for them. I'm thinking about something like this: check if
the extension is available and, if not, skip the check of that module,
with a warning or notification that it was skipped because it wasn't
available.
Thanks!
Stephen
Attachments:
[application/pgp-signature] signature.asc (819B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Trouble with amcheck
@ 2017-09-15 02:38 Andres Freund <[email protected]>
parent: Stephen Frost <[email protected]>
1 sibling, 0 replies; 36+ messages in thread
From: Andres Freund @ 2017-09-15 02:38 UTC (permalink / raw)
To: Stephen Frost <[email protected]>; +Cc: Douglas Doole <[email protected]>; Michael Paquier <[email protected]>; Peter Geoghegan <[email protected]>; pgsql-hackers
On 2017-09-14 22:36:38 -0400, Stephen Frost wrote:
> Andres,
>
> * Andres Freund ([email protected]) wrote:
> > On 2017-09-15 02:22:49 +0000, Douglas Doole wrote:
> > > Thanks all. Making and installing the contribs got me rolling again. (I
> > > tried "make world" but ran into trouble with the XML docs. But that's pain
> > > and suffering for another day.)
> > >
> > > I'd agree that "make installcheck-world" should imply that all prereqs are
> > > met - that's certainsly the normal behaviour for make.
> >
> > I'm very unconvinced by this, given that one use of installcheck is to
> > run against an existing server. For which one might not even have access
> > to the relevant directories to install extensions into.
>
> Sure, but if the extensions aren't in place and you're trying to run
> make installcheck-world, it's not like it's somehow going to succeed.
>
> Failing earlier on the install seems like a reasonable thing to do
> rather than failing later halfway through the check process.
But, uh, aren't you now provoking errors because of non-existing rights
to install stuff on the system? I'm all for improving error messages
and/or adding extra checks, but this doesn't seem to be a solution.
> Now, that said, perhaps a bit more smarts would be in order here to,
> instead, check that the extensions are available before trying to run
> the checks for them. I'm thinking about something like this: check if
> the extension is available and, if not, skip the check of that module,
> with a warning or notification that it was skipped because it wasn't
> available.
I think that'd just lead to people not noticing that they're not
executing all tests. I'm ok with adding a hard error with a better
message or such tho.
Greetings,
Andres Freund
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Trouble with amcheck
@ 2017-09-15 02:43 Tom Lane <[email protected]>
parent: Stephen Frost <[email protected]>
1 sibling, 0 replies; 36+ messages in thread
From: Tom Lane @ 2017-09-15 02:43 UTC (permalink / raw)
To: Stephen Frost <[email protected]>; +Cc: Andres Freund <[email protected]>; Douglas Doole <[email protected]>; Michael Paquier <[email protected]>; Peter Geoghegan <[email protected]>; pgsql-hackers
Stephen Frost <[email protected]> writes:
> * Andres Freund ([email protected]) wrote:
>> I'm very unconvinced by this, given that one use of installcheck is to
>> run against an existing server. For which one might not even have access
>> to the relevant directories to install extensions into.
> Sure, but if the extensions aren't in place and you're trying to run
> make installcheck-world, it's not like it's somehow going to succeed.
I'm more or less with Andres: this is just pilot error. If you didn't
do install-world you shouldn't expect installcheck-world to succeed.
> Now, that said, perhaps a bit more smarts would be in order here to,
> instead, check that the extensions are available before trying to run
> the checks for them. I'm thinking about something like this: check if
> the extension is available and, if not, skip the check of that module,
> with a warning or notification that it was skipped because it wasn't
> available.
Meh. I'm worried that that would have undesirable failure modes,
ie letting something pass when it should have failed.
Maybe we need some documentation improvements, though, to clarify
the relationships between these make targets.
regards, tom lane
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 36+ messages in thread
* security release
@ 2018-11-08 17:30 Pavel Stehule <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Pavel Stehule @ 2018-11-08 17:30 UTC (permalink / raw)
To: pgsql-hackers
Ahoj
dnes byly zverejneny opravne verze viz
https://www.postgresql.org/about/news/1905/
Pokud si hrajete s PostgreSQL 11, pripadne ji pouzivate produkcne, tak
neodkladejte upgrade.
Pavel
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: security release
@ 2018-11-08 17:39 Robert Haas <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Robert Haas @ 2018-11-08 17:39 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: pgsql-hackers
On Thu, Nov 8, 2018 at 12:31 PM Pavel Stehule <[email protected]> wrote:
> Ahoj
>
> dnes byly zverejneny opravne verze viz https://www.postgresql.org/about/news/1905/
>
> Pokud si hrajete s PostgreSQL 11, pripadne ji pouzivate produkcne, tak neodkladejte upgrade.
Wrong list?
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: security release
@ 2018-11-08 17:54 Pavel Stehule <[email protected]>
parent: Robert Haas <[email protected]>
0 siblings, 0 replies; 36+ messages in thread
From: Pavel Stehule @ 2018-11-08 17:54 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: pgsql-hackers
čt 8. 11. 2018 v 18:40 odesílatel Robert Haas <[email protected]>
napsal:
> On Thu, Nov 8, 2018 at 12:31 PM Pavel Stehule <[email protected]>
> wrote:
> > Ahoj
> >
> > dnes byly zverejneny opravne verze viz
> https://www.postgresql.org/about/news/1905/
> >
> > Pokud si hrajete s PostgreSQL 11, pripadne ji pouzivate produkcne, tak
> neodkladejte upgrade.
>
> Wrong list?
>
yes, I am sorry.
Regards
Pavel
>
> --
> Robert Haas
> EnterpriseDB: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>
^ permalink raw reply [nested|flat] 36+ messages in thread
* [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping.
@ 2021-01-20 14:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 36+ messages in thread
From: Kyotaro Horiguchi @ 2021-01-20 14:59 UTC (permalink / raw)
WAL-skipping optimization omits bumping page LSN when ALTER TABLE SET
TABLESPACE. Also heap_page_prune does the same. However,
old_snapshot_threshold feature needs page LSN to be kept updated on
these operations so that TestForOldSnapshot properly find identify
whether a snapshot is invalidated.
---
src/backend/access/heap/pruneheap.c | 14 ++++++++++++++
src/backend/catalog/storage.c | 8 ++++++++
src/include/storage/bufmgr.h | 3 ++-
src/include/utils/rel.h | 4 ++--
4 files changed, 26 insertions(+), 3 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index e3a716a2a2..33c0841b36 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -70,6 +70,8 @@ static void heap_prune_record_redirect(PruneState *prstate,
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* Optionally prune and repair fragmentation in the specified page.
@@ -331,6 +333,18 @@ heap_page_prune(Relation relation, Buffer buffer,
PageSetLSN(BufferGetPage(buffer), recptr);
}
+ else if (relation->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT)
+ {
+ XLogRecPtr currlsn = GetXLogInsertRecPtr();
+ Page page = BufferGetPage(buffer);
+
+ Assert(wal_level < WAL_LEVEL_REPLICA);
+
+ /* we need to update page LSN to notify reader of pruning */
+ if (PageGetLSN(page) == currlsn)
+ currlsn = gistXLogAssignLSN();
+ PageSetLSN(page, currlsn);
+ }
}
else
{
diff --git a/src/backend/catalog/storage.c b/src/backend/catalog/storage.c
index cba7a9ada0..488ac6f760 100644
--- a/src/backend/catalog/storage.c
+++ b/src/backend/catalog/storage.c
@@ -75,6 +75,8 @@ typedef struct PendingRelSync
static PendingRelDelete *pendingDeletes = NULL; /* head of linked list */
HTAB *pendingSyncHash = NULL;
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* AddPendingSync
@@ -414,6 +416,7 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
bool copying_initfork;
BlockNumber nblocks;
BlockNumber blkno;
+ XLogRecPtr fakepagelsn;
page = (Page) buf.data;
@@ -434,6 +437,9 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
use_wal = XLogIsNeeded() &&
(relpersistence == RELPERSISTENCE_PERMANENT || copying_initfork);
+ if (!use_wal)
+ fakepagelsn = gistXLogAssignLSN();
+
nblocks = smgrnblocks(src, forkNum);
for (blkno = 0; blkno < nblocks; blkno++)
@@ -460,6 +466,8 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
*/
if (use_wal)
log_newpage(&dst->smgr_rnode.node, forkNum, blkno, page, false);
+ else
+ PageSetLSN(page, fakepagelsn);
PageSetChecksumInplace(page, blkno);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index e641174798..9ab9e7394c 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -19,6 +19,7 @@
#include "storage/buf.h"
#include "storage/bufpage.h"
#include "storage/relfilenode.h"
+#include "utils/rel.h"
#include "utils/relcache.h"
#include "utils/snapmgr.h"
@@ -289,7 +290,7 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page)
&& ((snapshot)->snapshot_type == SNAPSHOT_MVCC
|| (snapshot)->snapshot_type == SNAPSHOT_TOAST)
&& !XLogRecPtrIsInvalid((snapshot)->lsn)
- && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn))
+ && PageGetLSN(page) > (snapshot)->lsn)
TestForOldSnapshot_impl(snapshot, relation);
}
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index f58d65cf28..35e53c8d74 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -618,8 +618,8 @@ typedef struct ViewOptions
* decoding snapshot.
*/
#define RelationIsAccessibleInLogicalDecoding(relation) \
- (XLogLogicalInfoActive() && \
- RelationNeedsWAL(relation) && \
+ (XLogLogicalInfoActive() && \
+ (relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT && \
(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation)))
/*
--
2.27.0
----Next_Part(Thu_Jan_21_00_28_44_2021_003)----
^ permalink raw reply [nested|flat] 36+ messages in thread
* [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping.
@ 2021-01-20 14:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 36+ messages in thread
From: Kyotaro Horiguchi @ 2021-01-20 14:59 UTC (permalink / raw)
WAL-skipping optimization omits bumping page LSN when ALTER TABLE SET
TABLESPACE. Also heap_page_prune does the same. However,
old_snapshot_threshold feature needs page LSN to be kept updated on
these operations so that TestForOldSnapshot properly find identify
whether a snapshot is invalidated.
---
src/backend/access/heap/pruneheap.c | 14 ++++++++++++++
src/backend/catalog/storage.c | 8 ++++++++
src/include/storage/bufmgr.h | 3 ++-
src/include/utils/rel.h | 4 ++--
4 files changed, 26 insertions(+), 3 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index e3a716a2a2..33c0841b36 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -70,6 +70,8 @@ static void heap_prune_record_redirect(PruneState *prstate,
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* Optionally prune and repair fragmentation in the specified page.
@@ -331,6 +333,18 @@ heap_page_prune(Relation relation, Buffer buffer,
PageSetLSN(BufferGetPage(buffer), recptr);
}
+ else if (relation->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT)
+ {
+ XLogRecPtr currlsn = GetXLogInsertRecPtr();
+ Page page = BufferGetPage(buffer);
+
+ Assert(wal_level < WAL_LEVEL_REPLICA);
+
+ /* we need to update page LSN to notify reader of pruning */
+ if (PageGetLSN(page) == currlsn)
+ currlsn = gistXLogAssignLSN();
+ PageSetLSN(page, currlsn);
+ }
}
else
{
diff --git a/src/backend/catalog/storage.c b/src/backend/catalog/storage.c
index cba7a9ada0..488ac6f760 100644
--- a/src/backend/catalog/storage.c
+++ b/src/backend/catalog/storage.c
@@ -75,6 +75,8 @@ typedef struct PendingRelSync
static PendingRelDelete *pendingDeletes = NULL; /* head of linked list */
HTAB *pendingSyncHash = NULL;
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* AddPendingSync
@@ -414,6 +416,7 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
bool copying_initfork;
BlockNumber nblocks;
BlockNumber blkno;
+ XLogRecPtr fakepagelsn;
page = (Page) buf.data;
@@ -434,6 +437,9 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
use_wal = XLogIsNeeded() &&
(relpersistence == RELPERSISTENCE_PERMANENT || copying_initfork);
+ if (!use_wal)
+ fakepagelsn = gistXLogAssignLSN();
+
nblocks = smgrnblocks(src, forkNum);
for (blkno = 0; blkno < nblocks; blkno++)
@@ -460,6 +466,8 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
*/
if (use_wal)
log_newpage(&dst->smgr_rnode.node, forkNum, blkno, page, false);
+ else
+ PageSetLSN(page, fakepagelsn);
PageSetChecksumInplace(page, blkno);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index e641174798..9ab9e7394c 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -19,6 +19,7 @@
#include "storage/buf.h"
#include "storage/bufpage.h"
#include "storage/relfilenode.h"
+#include "utils/rel.h"
#include "utils/relcache.h"
#include "utils/snapmgr.h"
@@ -289,7 +290,7 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page)
&& ((snapshot)->snapshot_type == SNAPSHOT_MVCC
|| (snapshot)->snapshot_type == SNAPSHOT_TOAST)
&& !XLogRecPtrIsInvalid((snapshot)->lsn)
- && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn))
+ && PageGetLSN(page) > (snapshot)->lsn)
TestForOldSnapshot_impl(snapshot, relation);
}
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index f58d65cf28..35e53c8d74 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -618,8 +618,8 @@ typedef struct ViewOptions
* decoding snapshot.
*/
#define RelationIsAccessibleInLogicalDecoding(relation) \
- (XLogLogicalInfoActive() && \
- RelationNeedsWAL(relation) && \
+ (XLogLogicalInfoActive() && \
+ (relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT && \
(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation)))
/*
--
2.27.0
----Next_Part(Thu_Jan_21_00_28_44_2021_003)----
^ permalink raw reply [nested|flat] 36+ messages in thread
* [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping.
@ 2021-01-20 14:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 36+ messages in thread
From: Kyotaro Horiguchi @ 2021-01-20 14:59 UTC (permalink / raw)
WAL-skipping optimization omits bumping page LSN when ALTER TABLE SET
TABLESPACE. Also heap_page_prune does the same. However,
old_snapshot_threshold feature needs page LSN to be kept updated on
these operations so that TestForOldSnapshot properly find identify
whether a snapshot is invalidated.
---
src/backend/access/heap/pruneheap.c | 14 ++++++++++++++
src/backend/catalog/storage.c | 8 ++++++++
src/include/storage/bufmgr.h | 3 ++-
src/include/utils/rel.h | 4 ++--
4 files changed, 26 insertions(+), 3 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index e3a716a2a2..33c0841b36 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -70,6 +70,8 @@ static void heap_prune_record_redirect(PruneState *prstate,
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* Optionally prune and repair fragmentation in the specified page.
@@ -331,6 +333,18 @@ heap_page_prune(Relation relation, Buffer buffer,
PageSetLSN(BufferGetPage(buffer), recptr);
}
+ else if (relation->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT)
+ {
+ XLogRecPtr currlsn = GetXLogInsertRecPtr();
+ Page page = BufferGetPage(buffer);
+
+ Assert(wal_level < WAL_LEVEL_REPLICA);
+
+ /* we need to update page LSN to notify reader of pruning */
+ if (PageGetLSN(page) == currlsn)
+ currlsn = gistXLogAssignLSN();
+ PageSetLSN(page, currlsn);
+ }
}
else
{
diff --git a/src/backend/catalog/storage.c b/src/backend/catalog/storage.c
index cba7a9ada0..488ac6f760 100644
--- a/src/backend/catalog/storage.c
+++ b/src/backend/catalog/storage.c
@@ -75,6 +75,8 @@ typedef struct PendingRelSync
static PendingRelDelete *pendingDeletes = NULL; /* head of linked list */
HTAB *pendingSyncHash = NULL;
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* AddPendingSync
@@ -414,6 +416,7 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
bool copying_initfork;
BlockNumber nblocks;
BlockNumber blkno;
+ XLogRecPtr fakepagelsn;
page = (Page) buf.data;
@@ -434,6 +437,9 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
use_wal = XLogIsNeeded() &&
(relpersistence == RELPERSISTENCE_PERMANENT || copying_initfork);
+ if (!use_wal)
+ fakepagelsn = gistXLogAssignLSN();
+
nblocks = smgrnblocks(src, forkNum);
for (blkno = 0; blkno < nblocks; blkno++)
@@ -460,6 +466,8 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
*/
if (use_wal)
log_newpage(&dst->smgr_rnode.node, forkNum, blkno, page, false);
+ else
+ PageSetLSN(page, fakepagelsn);
PageSetChecksumInplace(page, blkno);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index e641174798..9ab9e7394c 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -19,6 +19,7 @@
#include "storage/buf.h"
#include "storage/bufpage.h"
#include "storage/relfilenode.h"
+#include "utils/rel.h"
#include "utils/relcache.h"
#include "utils/snapmgr.h"
@@ -289,7 +290,7 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page)
&& ((snapshot)->snapshot_type == SNAPSHOT_MVCC
|| (snapshot)->snapshot_type == SNAPSHOT_TOAST)
&& !XLogRecPtrIsInvalid((snapshot)->lsn)
- && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn))
+ && PageGetLSN(page) > (snapshot)->lsn)
TestForOldSnapshot_impl(snapshot, relation);
}
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index f58d65cf28..35e53c8d74 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -618,8 +618,8 @@ typedef struct ViewOptions
* decoding snapshot.
*/
#define RelationIsAccessibleInLogicalDecoding(relation) \
- (XLogLogicalInfoActive() && \
- RelationNeedsWAL(relation) && \
+ (XLogLogicalInfoActive() && \
+ (relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT && \
(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation)))
/*
--
2.27.0
----Next_Part(Thu_Jan_21_00_28_44_2021_003)----
^ permalink raw reply [nested|flat] 36+ messages in thread
* [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping.
@ 2021-01-20 14:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 36+ messages in thread
From: Kyotaro Horiguchi @ 2021-01-20 14:59 UTC (permalink / raw)
WAL-skipping optimization omits bumping page LSN when ALTER TABLE SET
TABLESPACE. Also heap_page_prune does the same. However,
old_snapshot_threshold feature needs page LSN to be kept updated on
these operations so that TestForOldSnapshot properly find identify
whether a snapshot is invalidated.
---
src/backend/access/heap/pruneheap.c | 14 ++++++++++++++
src/backend/catalog/storage.c | 8 ++++++++
src/include/storage/bufmgr.h | 3 ++-
src/include/utils/rel.h | 4 ++--
4 files changed, 26 insertions(+), 3 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index e3a716a2a2..33c0841b36 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -70,6 +70,8 @@ static void heap_prune_record_redirect(PruneState *prstate,
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* Optionally prune and repair fragmentation in the specified page.
@@ -331,6 +333,18 @@ heap_page_prune(Relation relation, Buffer buffer,
PageSetLSN(BufferGetPage(buffer), recptr);
}
+ else if (relation->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT)
+ {
+ XLogRecPtr currlsn = GetXLogInsertRecPtr();
+ Page page = BufferGetPage(buffer);
+
+ Assert(wal_level < WAL_LEVEL_REPLICA);
+
+ /* we need to update page LSN to notify reader of pruning */
+ if (PageGetLSN(page) == currlsn)
+ currlsn = gistXLogAssignLSN();
+ PageSetLSN(page, currlsn);
+ }
}
else
{
diff --git a/src/backend/catalog/storage.c b/src/backend/catalog/storage.c
index cba7a9ada0..488ac6f760 100644
--- a/src/backend/catalog/storage.c
+++ b/src/backend/catalog/storage.c
@@ -75,6 +75,8 @@ typedef struct PendingRelSync
static PendingRelDelete *pendingDeletes = NULL; /* head of linked list */
HTAB *pendingSyncHash = NULL;
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* AddPendingSync
@@ -414,6 +416,7 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
bool copying_initfork;
BlockNumber nblocks;
BlockNumber blkno;
+ XLogRecPtr fakepagelsn;
page = (Page) buf.data;
@@ -434,6 +437,9 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
use_wal = XLogIsNeeded() &&
(relpersistence == RELPERSISTENCE_PERMANENT || copying_initfork);
+ if (!use_wal)
+ fakepagelsn = gistXLogAssignLSN();
+
nblocks = smgrnblocks(src, forkNum);
for (blkno = 0; blkno < nblocks; blkno++)
@@ -460,6 +466,8 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
*/
if (use_wal)
log_newpage(&dst->smgr_rnode.node, forkNum, blkno, page, false);
+ else
+ PageSetLSN(page, fakepagelsn);
PageSetChecksumInplace(page, blkno);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index e641174798..9ab9e7394c 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -19,6 +19,7 @@
#include "storage/buf.h"
#include "storage/bufpage.h"
#include "storage/relfilenode.h"
+#include "utils/rel.h"
#include "utils/relcache.h"
#include "utils/snapmgr.h"
@@ -289,7 +290,7 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page)
&& ((snapshot)->snapshot_type == SNAPSHOT_MVCC
|| (snapshot)->snapshot_type == SNAPSHOT_TOAST)
&& !XLogRecPtrIsInvalid((snapshot)->lsn)
- && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn))
+ && PageGetLSN(page) > (snapshot)->lsn)
TestForOldSnapshot_impl(snapshot, relation);
}
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index f58d65cf28..35e53c8d74 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -618,8 +618,8 @@ typedef struct ViewOptions
* decoding snapshot.
*/
#define RelationIsAccessibleInLogicalDecoding(relation) \
- (XLogLogicalInfoActive() && \
- RelationNeedsWAL(relation) && \
+ (XLogLogicalInfoActive() && \
+ (relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT && \
(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation)))
/*
--
2.27.0
----Next_Part(Thu_Jan_21_00_28_44_2021_003)----
^ permalink raw reply [nested|flat] 36+ messages in thread
* [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping.
@ 2021-01-20 14:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 36+ messages in thread
From: Kyotaro Horiguchi @ 2021-01-20 14:59 UTC (permalink / raw)
WAL-skipping optimization omits bumping page LSN when ALTER TABLE SET
TABLESPACE. Also heap_page_prune does the same. However,
old_snapshot_threshold feature needs page LSN to be kept updated on
these operations so that TestForOldSnapshot properly find identify
whether a snapshot is invalidated.
---
src/backend/access/heap/pruneheap.c | 14 ++++++++++++++
src/backend/catalog/storage.c | 8 ++++++++
src/include/storage/bufmgr.h | 3 ++-
src/include/utils/rel.h | 4 ++--
4 files changed, 26 insertions(+), 3 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index e3a716a2a2..33c0841b36 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -70,6 +70,8 @@ static void heap_prune_record_redirect(PruneState *prstate,
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* Optionally prune and repair fragmentation in the specified page.
@@ -331,6 +333,18 @@ heap_page_prune(Relation relation, Buffer buffer,
PageSetLSN(BufferGetPage(buffer), recptr);
}
+ else if (relation->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT)
+ {
+ XLogRecPtr currlsn = GetXLogInsertRecPtr();
+ Page page = BufferGetPage(buffer);
+
+ Assert(wal_level < WAL_LEVEL_REPLICA);
+
+ /* we need to update page LSN to notify reader of pruning */
+ if (PageGetLSN(page) == currlsn)
+ currlsn = gistXLogAssignLSN();
+ PageSetLSN(page, currlsn);
+ }
}
else
{
diff --git a/src/backend/catalog/storage.c b/src/backend/catalog/storage.c
index cba7a9ada0..488ac6f760 100644
--- a/src/backend/catalog/storage.c
+++ b/src/backend/catalog/storage.c
@@ -75,6 +75,8 @@ typedef struct PendingRelSync
static PendingRelDelete *pendingDeletes = NULL; /* head of linked list */
HTAB *pendingSyncHash = NULL;
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* AddPendingSync
@@ -414,6 +416,7 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
bool copying_initfork;
BlockNumber nblocks;
BlockNumber blkno;
+ XLogRecPtr fakepagelsn;
page = (Page) buf.data;
@@ -434,6 +437,9 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
use_wal = XLogIsNeeded() &&
(relpersistence == RELPERSISTENCE_PERMANENT || copying_initfork);
+ if (!use_wal)
+ fakepagelsn = gistXLogAssignLSN();
+
nblocks = smgrnblocks(src, forkNum);
for (blkno = 0; blkno < nblocks; blkno++)
@@ -460,6 +466,8 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
*/
if (use_wal)
log_newpage(&dst->smgr_rnode.node, forkNum, blkno, page, false);
+ else
+ PageSetLSN(page, fakepagelsn);
PageSetChecksumInplace(page, blkno);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index e641174798..9ab9e7394c 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -19,6 +19,7 @@
#include "storage/buf.h"
#include "storage/bufpage.h"
#include "storage/relfilenode.h"
+#include "utils/rel.h"
#include "utils/relcache.h"
#include "utils/snapmgr.h"
@@ -289,7 +290,7 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page)
&& ((snapshot)->snapshot_type == SNAPSHOT_MVCC
|| (snapshot)->snapshot_type == SNAPSHOT_TOAST)
&& !XLogRecPtrIsInvalid((snapshot)->lsn)
- && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn))
+ && PageGetLSN(page) > (snapshot)->lsn)
TestForOldSnapshot_impl(snapshot, relation);
}
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index f58d65cf28..35e53c8d74 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -618,8 +618,8 @@ typedef struct ViewOptions
* decoding snapshot.
*/
#define RelationIsAccessibleInLogicalDecoding(relation) \
- (XLogLogicalInfoActive() && \
- RelationNeedsWAL(relation) && \
+ (XLogLogicalInfoActive() && \
+ (relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT && \
(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation)))
/*
--
2.27.0
----Next_Part(Thu_Jan_21_00_28_44_2021_003)----
^ permalink raw reply [nested|flat] 36+ messages in thread
* [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping.
@ 2021-01-20 14:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 36+ messages in thread
From: Kyotaro Horiguchi @ 2021-01-20 14:59 UTC (permalink / raw)
WAL-skipping optimization omits bumping page LSN when ALTER TABLE SET
TABLESPACE. Also heap_page_prune does the same. However,
old_snapshot_threshold feature needs page LSN to be kept updated on
these operations so that TestForOldSnapshot properly find identify
whether a snapshot is invalidated.
---
src/backend/access/heap/pruneheap.c | 14 ++++++++++++++
src/backend/catalog/storage.c | 8 ++++++++
src/include/storage/bufmgr.h | 3 ++-
src/include/utils/rel.h | 4 ++--
4 files changed, 26 insertions(+), 3 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index e3a716a2a2..33c0841b36 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -70,6 +70,8 @@ static void heap_prune_record_redirect(PruneState *prstate,
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* Optionally prune and repair fragmentation in the specified page.
@@ -331,6 +333,18 @@ heap_page_prune(Relation relation, Buffer buffer,
PageSetLSN(BufferGetPage(buffer), recptr);
}
+ else if (relation->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT)
+ {
+ XLogRecPtr currlsn = GetXLogInsertRecPtr();
+ Page page = BufferGetPage(buffer);
+
+ Assert(wal_level < WAL_LEVEL_REPLICA);
+
+ /* we need to update page LSN to notify reader of pruning */
+ if (PageGetLSN(page) == currlsn)
+ currlsn = gistXLogAssignLSN();
+ PageSetLSN(page, currlsn);
+ }
}
else
{
diff --git a/src/backend/catalog/storage.c b/src/backend/catalog/storage.c
index cba7a9ada0..488ac6f760 100644
--- a/src/backend/catalog/storage.c
+++ b/src/backend/catalog/storage.c
@@ -75,6 +75,8 @@ typedef struct PendingRelSync
static PendingRelDelete *pendingDeletes = NULL; /* head of linked list */
HTAB *pendingSyncHash = NULL;
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* AddPendingSync
@@ -414,6 +416,7 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
bool copying_initfork;
BlockNumber nblocks;
BlockNumber blkno;
+ XLogRecPtr fakepagelsn;
page = (Page) buf.data;
@@ -434,6 +437,9 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
use_wal = XLogIsNeeded() &&
(relpersistence == RELPERSISTENCE_PERMANENT || copying_initfork);
+ if (!use_wal)
+ fakepagelsn = gistXLogAssignLSN();
+
nblocks = smgrnblocks(src, forkNum);
for (blkno = 0; blkno < nblocks; blkno++)
@@ -460,6 +466,8 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
*/
if (use_wal)
log_newpage(&dst->smgr_rnode.node, forkNum, blkno, page, false);
+ else
+ PageSetLSN(page, fakepagelsn);
PageSetChecksumInplace(page, blkno);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index e641174798..9ab9e7394c 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -19,6 +19,7 @@
#include "storage/buf.h"
#include "storage/bufpage.h"
#include "storage/relfilenode.h"
+#include "utils/rel.h"
#include "utils/relcache.h"
#include "utils/snapmgr.h"
@@ -289,7 +290,7 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page)
&& ((snapshot)->snapshot_type == SNAPSHOT_MVCC
|| (snapshot)->snapshot_type == SNAPSHOT_TOAST)
&& !XLogRecPtrIsInvalid((snapshot)->lsn)
- && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn))
+ && PageGetLSN(page) > (snapshot)->lsn)
TestForOldSnapshot_impl(snapshot, relation);
}
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index f58d65cf28..35e53c8d74 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -618,8 +618,8 @@ typedef struct ViewOptions
* decoding snapshot.
*/
#define RelationIsAccessibleInLogicalDecoding(relation) \
- (XLogLogicalInfoActive() && \
- RelationNeedsWAL(relation) && \
+ (XLogLogicalInfoActive() && \
+ (relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT && \
(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation)))
/*
--
2.27.0
----Next_Part(Thu_Jan_21_00_28_44_2021_003)----
^ permalink raw reply [nested|flat] 36+ messages in thread
* [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping.
@ 2021-01-20 14:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 36+ messages in thread
From: Kyotaro Horiguchi @ 2021-01-20 14:59 UTC (permalink / raw)
WAL-skipping optimization omits bumping page LSN when ALTER TABLE SET
TABLESPACE. Also heap_page_prune does the same. However,
old_snapshot_threshold feature needs page LSN to be kept updated on
these operations so that TestForOldSnapshot properly find identify
whether a snapshot is invalidated.
---
src/backend/access/heap/pruneheap.c | 14 ++++++++++++++
src/backend/catalog/storage.c | 8 ++++++++
src/include/storage/bufmgr.h | 3 ++-
src/include/utils/rel.h | 4 ++--
4 files changed, 26 insertions(+), 3 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index e3a716a2a2..33c0841b36 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -70,6 +70,8 @@ static void heap_prune_record_redirect(PruneState *prstate,
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* Optionally prune and repair fragmentation in the specified page.
@@ -331,6 +333,18 @@ heap_page_prune(Relation relation, Buffer buffer,
PageSetLSN(BufferGetPage(buffer), recptr);
}
+ else if (relation->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT)
+ {
+ XLogRecPtr currlsn = GetXLogInsertRecPtr();
+ Page page = BufferGetPage(buffer);
+
+ Assert(wal_level < WAL_LEVEL_REPLICA);
+
+ /* we need to update page LSN to notify reader of pruning */
+ if (PageGetLSN(page) == currlsn)
+ currlsn = gistXLogAssignLSN();
+ PageSetLSN(page, currlsn);
+ }
}
else
{
diff --git a/src/backend/catalog/storage.c b/src/backend/catalog/storage.c
index cba7a9ada0..488ac6f760 100644
--- a/src/backend/catalog/storage.c
+++ b/src/backend/catalog/storage.c
@@ -75,6 +75,8 @@ typedef struct PendingRelSync
static PendingRelDelete *pendingDeletes = NULL; /* head of linked list */
HTAB *pendingSyncHash = NULL;
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* AddPendingSync
@@ -414,6 +416,7 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
bool copying_initfork;
BlockNumber nblocks;
BlockNumber blkno;
+ XLogRecPtr fakepagelsn;
page = (Page) buf.data;
@@ -434,6 +437,9 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
use_wal = XLogIsNeeded() &&
(relpersistence == RELPERSISTENCE_PERMANENT || copying_initfork);
+ if (!use_wal)
+ fakepagelsn = gistXLogAssignLSN();
+
nblocks = smgrnblocks(src, forkNum);
for (blkno = 0; blkno < nblocks; blkno++)
@@ -460,6 +466,8 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
*/
if (use_wal)
log_newpage(&dst->smgr_rnode.node, forkNum, blkno, page, false);
+ else
+ PageSetLSN(page, fakepagelsn);
PageSetChecksumInplace(page, blkno);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index e641174798..9ab9e7394c 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -19,6 +19,7 @@
#include "storage/buf.h"
#include "storage/bufpage.h"
#include "storage/relfilenode.h"
+#include "utils/rel.h"
#include "utils/relcache.h"
#include "utils/snapmgr.h"
@@ -289,7 +290,7 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page)
&& ((snapshot)->snapshot_type == SNAPSHOT_MVCC
|| (snapshot)->snapshot_type == SNAPSHOT_TOAST)
&& !XLogRecPtrIsInvalid((snapshot)->lsn)
- && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn))
+ && PageGetLSN(page) > (snapshot)->lsn)
TestForOldSnapshot_impl(snapshot, relation);
}
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index f58d65cf28..35e53c8d74 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -618,8 +618,8 @@ typedef struct ViewOptions
* decoding snapshot.
*/
#define RelationIsAccessibleInLogicalDecoding(relation) \
- (XLogLogicalInfoActive() && \
- RelationNeedsWAL(relation) && \
+ (XLogLogicalInfoActive() && \
+ (relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT && \
(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation)))
/*
--
2.27.0
----Next_Part(Thu_Jan_21_00_28_44_2021_003)----
^ permalink raw reply [nested|flat] 36+ messages in thread
* [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping.
@ 2021-01-20 14:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 36+ messages in thread
From: Kyotaro Horiguchi @ 2021-01-20 14:59 UTC (permalink / raw)
WAL-skipping optimization omits bumping page LSN when ALTER TABLE SET
TABLESPACE. Also heap_page_prune does the same. However,
old_snapshot_threshold feature needs page LSN to be kept updated on
these operations so that TestForOldSnapshot properly find identify
whether a snapshot is invalidated.
---
src/backend/access/heap/pruneheap.c | 14 ++++++++++++++
src/backend/catalog/storage.c | 8 ++++++++
src/include/storage/bufmgr.h | 3 ++-
src/include/utils/rel.h | 4 ++--
4 files changed, 26 insertions(+), 3 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index e3a716a2a2..33c0841b36 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -70,6 +70,8 @@ static void heap_prune_record_redirect(PruneState *prstate,
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* Optionally prune and repair fragmentation in the specified page.
@@ -331,6 +333,18 @@ heap_page_prune(Relation relation, Buffer buffer,
PageSetLSN(BufferGetPage(buffer), recptr);
}
+ else if (relation->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT)
+ {
+ XLogRecPtr currlsn = GetXLogInsertRecPtr();
+ Page page = BufferGetPage(buffer);
+
+ Assert(wal_level < WAL_LEVEL_REPLICA);
+
+ /* we need to update page LSN to notify reader of pruning */
+ if (PageGetLSN(page) == currlsn)
+ currlsn = gistXLogAssignLSN();
+ PageSetLSN(page, currlsn);
+ }
}
else
{
diff --git a/src/backend/catalog/storage.c b/src/backend/catalog/storage.c
index cba7a9ada0..488ac6f760 100644
--- a/src/backend/catalog/storage.c
+++ b/src/backend/catalog/storage.c
@@ -75,6 +75,8 @@ typedef struct PendingRelSync
static PendingRelDelete *pendingDeletes = NULL; /* head of linked list */
HTAB *pendingSyncHash = NULL;
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* AddPendingSync
@@ -414,6 +416,7 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
bool copying_initfork;
BlockNumber nblocks;
BlockNumber blkno;
+ XLogRecPtr fakepagelsn;
page = (Page) buf.data;
@@ -434,6 +437,9 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
use_wal = XLogIsNeeded() &&
(relpersistence == RELPERSISTENCE_PERMANENT || copying_initfork);
+ if (!use_wal)
+ fakepagelsn = gistXLogAssignLSN();
+
nblocks = smgrnblocks(src, forkNum);
for (blkno = 0; blkno < nblocks; blkno++)
@@ -460,6 +466,8 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
*/
if (use_wal)
log_newpage(&dst->smgr_rnode.node, forkNum, blkno, page, false);
+ else
+ PageSetLSN(page, fakepagelsn);
PageSetChecksumInplace(page, blkno);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index e641174798..9ab9e7394c 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -19,6 +19,7 @@
#include "storage/buf.h"
#include "storage/bufpage.h"
#include "storage/relfilenode.h"
+#include "utils/rel.h"
#include "utils/relcache.h"
#include "utils/snapmgr.h"
@@ -289,7 +290,7 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page)
&& ((snapshot)->snapshot_type == SNAPSHOT_MVCC
|| (snapshot)->snapshot_type == SNAPSHOT_TOAST)
&& !XLogRecPtrIsInvalid((snapshot)->lsn)
- && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn))
+ && PageGetLSN(page) > (snapshot)->lsn)
TestForOldSnapshot_impl(snapshot, relation);
}
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index f58d65cf28..35e53c8d74 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -618,8 +618,8 @@ typedef struct ViewOptions
* decoding snapshot.
*/
#define RelationIsAccessibleInLogicalDecoding(relation) \
- (XLogLogicalInfoActive() && \
- RelationNeedsWAL(relation) && \
+ (XLogLogicalInfoActive() && \
+ (relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT && \
(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation)))
/*
--
2.27.0
----Next_Part(Thu_Jan_21_00_28_44_2021_003)----
^ permalink raw reply [nested|flat] 36+ messages in thread
* [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping.
@ 2021-01-20 14:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 36+ messages in thread
From: Kyotaro Horiguchi @ 2021-01-20 14:59 UTC (permalink / raw)
WAL-skipping optimization omits bumping page LSN when ALTER TABLE SET
TABLESPACE. Also heap_page_prune does the same. However,
old_snapshot_threshold feature needs page LSN to be kept updated on
these operations so that TestForOldSnapshot properly find identify
whether a snapshot is invalidated.
---
src/backend/access/heap/pruneheap.c | 14 ++++++++++++++
src/backend/catalog/storage.c | 8 ++++++++
src/include/storage/bufmgr.h | 3 ++-
src/include/utils/rel.h | 4 ++--
4 files changed, 26 insertions(+), 3 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index e3a716a2a2..33c0841b36 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -70,6 +70,8 @@ static void heap_prune_record_redirect(PruneState *prstate,
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* Optionally prune and repair fragmentation in the specified page.
@@ -331,6 +333,18 @@ heap_page_prune(Relation relation, Buffer buffer,
PageSetLSN(BufferGetPage(buffer), recptr);
}
+ else if (relation->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT)
+ {
+ XLogRecPtr currlsn = GetXLogInsertRecPtr();
+ Page page = BufferGetPage(buffer);
+
+ Assert(wal_level < WAL_LEVEL_REPLICA);
+
+ /* we need to update page LSN to notify reader of pruning */
+ if (PageGetLSN(page) == currlsn)
+ currlsn = gistXLogAssignLSN();
+ PageSetLSN(page, currlsn);
+ }
}
else
{
diff --git a/src/backend/catalog/storage.c b/src/backend/catalog/storage.c
index cba7a9ada0..488ac6f760 100644
--- a/src/backend/catalog/storage.c
+++ b/src/backend/catalog/storage.c
@@ -75,6 +75,8 @@ typedef struct PendingRelSync
static PendingRelDelete *pendingDeletes = NULL; /* head of linked list */
HTAB *pendingSyncHash = NULL;
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* AddPendingSync
@@ -414,6 +416,7 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
bool copying_initfork;
BlockNumber nblocks;
BlockNumber blkno;
+ XLogRecPtr fakepagelsn;
page = (Page) buf.data;
@@ -434,6 +437,9 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
use_wal = XLogIsNeeded() &&
(relpersistence == RELPERSISTENCE_PERMANENT || copying_initfork);
+ if (!use_wal)
+ fakepagelsn = gistXLogAssignLSN();
+
nblocks = smgrnblocks(src, forkNum);
for (blkno = 0; blkno < nblocks; blkno++)
@@ -460,6 +466,8 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
*/
if (use_wal)
log_newpage(&dst->smgr_rnode.node, forkNum, blkno, page, false);
+ else
+ PageSetLSN(page, fakepagelsn);
PageSetChecksumInplace(page, blkno);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index e641174798..9ab9e7394c 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -19,6 +19,7 @@
#include "storage/buf.h"
#include "storage/bufpage.h"
#include "storage/relfilenode.h"
+#include "utils/rel.h"
#include "utils/relcache.h"
#include "utils/snapmgr.h"
@@ -289,7 +290,7 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page)
&& ((snapshot)->snapshot_type == SNAPSHOT_MVCC
|| (snapshot)->snapshot_type == SNAPSHOT_TOAST)
&& !XLogRecPtrIsInvalid((snapshot)->lsn)
- && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn))
+ && PageGetLSN(page) > (snapshot)->lsn)
TestForOldSnapshot_impl(snapshot, relation);
}
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index f58d65cf28..35e53c8d74 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -618,8 +618,8 @@ typedef struct ViewOptions
* decoding snapshot.
*/
#define RelationIsAccessibleInLogicalDecoding(relation) \
- (XLogLogicalInfoActive() && \
- RelationNeedsWAL(relation) && \
+ (XLogLogicalInfoActive() && \
+ (relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT && \
(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation)))
/*
--
2.27.0
----Next_Part(Thu_Jan_21_00_28_44_2021_003)----
^ permalink raw reply [nested|flat] 36+ messages in thread
* [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping.
@ 2021-01-20 14:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 36+ messages in thread
From: Kyotaro Horiguchi @ 2021-01-20 14:59 UTC (permalink / raw)
WAL-skipping optimization omits bumping page LSN when ALTER TABLE SET
TABLESPACE. Also heap_page_prune does the same. However,
old_snapshot_threshold feature needs page LSN to be kept updated on
these operations so that TestForOldSnapshot properly find identify
whether a snapshot is invalidated.
---
src/backend/access/heap/pruneheap.c | 14 ++++++++++++++
src/backend/catalog/storage.c | 8 ++++++++
src/include/storage/bufmgr.h | 3 ++-
src/include/utils/rel.h | 4 ++--
4 files changed, 26 insertions(+), 3 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index e3a716a2a2..33c0841b36 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -70,6 +70,8 @@ static void heap_prune_record_redirect(PruneState *prstate,
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* Optionally prune and repair fragmentation in the specified page.
@@ -331,6 +333,18 @@ heap_page_prune(Relation relation, Buffer buffer,
PageSetLSN(BufferGetPage(buffer), recptr);
}
+ else if (relation->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT)
+ {
+ XLogRecPtr currlsn = GetXLogInsertRecPtr();
+ Page page = BufferGetPage(buffer);
+
+ Assert(wal_level < WAL_LEVEL_REPLICA);
+
+ /* we need to update page LSN to notify reader of pruning */
+ if (PageGetLSN(page) == currlsn)
+ currlsn = gistXLogAssignLSN();
+ PageSetLSN(page, currlsn);
+ }
}
else
{
diff --git a/src/backend/catalog/storage.c b/src/backend/catalog/storage.c
index cba7a9ada0..488ac6f760 100644
--- a/src/backend/catalog/storage.c
+++ b/src/backend/catalog/storage.c
@@ -75,6 +75,8 @@ typedef struct PendingRelSync
static PendingRelDelete *pendingDeletes = NULL; /* head of linked list */
HTAB *pendingSyncHash = NULL;
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* AddPendingSync
@@ -414,6 +416,7 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
bool copying_initfork;
BlockNumber nblocks;
BlockNumber blkno;
+ XLogRecPtr fakepagelsn;
page = (Page) buf.data;
@@ -434,6 +437,9 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
use_wal = XLogIsNeeded() &&
(relpersistence == RELPERSISTENCE_PERMANENT || copying_initfork);
+ if (!use_wal)
+ fakepagelsn = gistXLogAssignLSN();
+
nblocks = smgrnblocks(src, forkNum);
for (blkno = 0; blkno < nblocks; blkno++)
@@ -460,6 +466,8 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
*/
if (use_wal)
log_newpage(&dst->smgr_rnode.node, forkNum, blkno, page, false);
+ else
+ PageSetLSN(page, fakepagelsn);
PageSetChecksumInplace(page, blkno);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index e641174798..9ab9e7394c 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -19,6 +19,7 @@
#include "storage/buf.h"
#include "storage/bufpage.h"
#include "storage/relfilenode.h"
+#include "utils/rel.h"
#include "utils/relcache.h"
#include "utils/snapmgr.h"
@@ -289,7 +290,7 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page)
&& ((snapshot)->snapshot_type == SNAPSHOT_MVCC
|| (snapshot)->snapshot_type == SNAPSHOT_TOAST)
&& !XLogRecPtrIsInvalid((snapshot)->lsn)
- && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn))
+ && PageGetLSN(page) > (snapshot)->lsn)
TestForOldSnapshot_impl(snapshot, relation);
}
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index f58d65cf28..35e53c8d74 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -618,8 +618,8 @@ typedef struct ViewOptions
* decoding snapshot.
*/
#define RelationIsAccessibleInLogicalDecoding(relation) \
- (XLogLogicalInfoActive() && \
- RelationNeedsWAL(relation) && \
+ (XLogLogicalInfoActive() && \
+ (relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT && \
(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation)))
/*
--
2.27.0
----Next_Part(Thu_Jan_21_00_28_44_2021_003)----
^ permalink raw reply [nested|flat] 36+ messages in thread
* [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping.
@ 2021-01-20 14:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 36+ messages in thread
From: Kyotaro Horiguchi @ 2021-01-20 14:59 UTC (permalink / raw)
WAL-skipping optimization omits bumping page LSN when ALTER TABLE SET
TABLESPACE. Also heap_page_prune does the same. However,
old_snapshot_threshold feature needs page LSN to be kept updated on
these operations so that TestForOldSnapshot properly find identify
whether a snapshot is invalidated.
---
src/backend/access/heap/pruneheap.c | 14 ++++++++++++++
src/backend/catalog/storage.c | 8 ++++++++
src/include/storage/bufmgr.h | 3 ++-
src/include/utils/rel.h | 4 ++--
4 files changed, 26 insertions(+), 3 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index e3a716a2a2..33c0841b36 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -70,6 +70,8 @@ static void heap_prune_record_redirect(PruneState *prstate,
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* Optionally prune and repair fragmentation in the specified page.
@@ -331,6 +333,18 @@ heap_page_prune(Relation relation, Buffer buffer,
PageSetLSN(BufferGetPage(buffer), recptr);
}
+ else if (relation->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT)
+ {
+ XLogRecPtr currlsn = GetXLogInsertRecPtr();
+ Page page = BufferGetPage(buffer);
+
+ Assert(wal_level < WAL_LEVEL_REPLICA);
+
+ /* we need to update page LSN to notify reader of pruning */
+ if (PageGetLSN(page) == currlsn)
+ currlsn = gistXLogAssignLSN();
+ PageSetLSN(page, currlsn);
+ }
}
else
{
diff --git a/src/backend/catalog/storage.c b/src/backend/catalog/storage.c
index cba7a9ada0..488ac6f760 100644
--- a/src/backend/catalog/storage.c
+++ b/src/backend/catalog/storage.c
@@ -75,6 +75,8 @@ typedef struct PendingRelSync
static PendingRelDelete *pendingDeletes = NULL; /* head of linked list */
HTAB *pendingSyncHash = NULL;
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* AddPendingSync
@@ -414,6 +416,7 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
bool copying_initfork;
BlockNumber nblocks;
BlockNumber blkno;
+ XLogRecPtr fakepagelsn;
page = (Page) buf.data;
@@ -434,6 +437,9 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
use_wal = XLogIsNeeded() &&
(relpersistence == RELPERSISTENCE_PERMANENT || copying_initfork);
+ if (!use_wal)
+ fakepagelsn = gistXLogAssignLSN();
+
nblocks = smgrnblocks(src, forkNum);
for (blkno = 0; blkno < nblocks; blkno++)
@@ -460,6 +466,8 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
*/
if (use_wal)
log_newpage(&dst->smgr_rnode.node, forkNum, blkno, page, false);
+ else
+ PageSetLSN(page, fakepagelsn);
PageSetChecksumInplace(page, blkno);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index e641174798..9ab9e7394c 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -19,6 +19,7 @@
#include "storage/buf.h"
#include "storage/bufpage.h"
#include "storage/relfilenode.h"
+#include "utils/rel.h"
#include "utils/relcache.h"
#include "utils/snapmgr.h"
@@ -289,7 +290,7 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page)
&& ((snapshot)->snapshot_type == SNAPSHOT_MVCC
|| (snapshot)->snapshot_type == SNAPSHOT_TOAST)
&& !XLogRecPtrIsInvalid((snapshot)->lsn)
- && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn))
+ && PageGetLSN(page) > (snapshot)->lsn)
TestForOldSnapshot_impl(snapshot, relation);
}
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index f58d65cf28..35e53c8d74 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -618,8 +618,8 @@ typedef struct ViewOptions
* decoding snapshot.
*/
#define RelationIsAccessibleInLogicalDecoding(relation) \
- (XLogLogicalInfoActive() && \
- RelationNeedsWAL(relation) && \
+ (XLogLogicalInfoActive() && \
+ (relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT && \
(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation)))
/*
--
2.27.0
----Next_Part(Thu_Jan_21_00_28_44_2021_003)----
^ permalink raw reply [nested|flat] 36+ messages in thread
* [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping.
@ 2021-01-20 14:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 36+ messages in thread
From: Kyotaro Horiguchi @ 2021-01-20 14:59 UTC (permalink / raw)
WAL-skipping optimization omits bumping page LSN when ALTER TABLE SET
TABLESPACE. Also heap_page_prune does the same. However,
old_snapshot_threshold feature needs page LSN to be kept updated on
these operations so that TestForOldSnapshot properly find identify
whether a snapshot is invalidated.
---
src/backend/access/heap/pruneheap.c | 14 ++++++++++++++
src/backend/catalog/storage.c | 8 ++++++++
src/include/storage/bufmgr.h | 3 ++-
src/include/utils/rel.h | 4 ++--
4 files changed, 26 insertions(+), 3 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index e3a716a2a2..33c0841b36 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -70,6 +70,8 @@ static void heap_prune_record_redirect(PruneState *prstate,
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* Optionally prune and repair fragmentation in the specified page.
@@ -331,6 +333,18 @@ heap_page_prune(Relation relation, Buffer buffer,
PageSetLSN(BufferGetPage(buffer), recptr);
}
+ else if (relation->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT)
+ {
+ XLogRecPtr currlsn = GetXLogInsertRecPtr();
+ Page page = BufferGetPage(buffer);
+
+ Assert(wal_level < WAL_LEVEL_REPLICA);
+
+ /* we need to update page LSN to notify reader of pruning */
+ if (PageGetLSN(page) == currlsn)
+ currlsn = gistXLogAssignLSN();
+ PageSetLSN(page, currlsn);
+ }
}
else
{
diff --git a/src/backend/catalog/storage.c b/src/backend/catalog/storage.c
index cba7a9ada0..488ac6f760 100644
--- a/src/backend/catalog/storage.c
+++ b/src/backend/catalog/storage.c
@@ -75,6 +75,8 @@ typedef struct PendingRelSync
static PendingRelDelete *pendingDeletes = NULL; /* head of linked list */
HTAB *pendingSyncHash = NULL;
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* AddPendingSync
@@ -414,6 +416,7 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
bool copying_initfork;
BlockNumber nblocks;
BlockNumber blkno;
+ XLogRecPtr fakepagelsn;
page = (Page) buf.data;
@@ -434,6 +437,9 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
use_wal = XLogIsNeeded() &&
(relpersistence == RELPERSISTENCE_PERMANENT || copying_initfork);
+ if (!use_wal)
+ fakepagelsn = gistXLogAssignLSN();
+
nblocks = smgrnblocks(src, forkNum);
for (blkno = 0; blkno < nblocks; blkno++)
@@ -460,6 +466,8 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
*/
if (use_wal)
log_newpage(&dst->smgr_rnode.node, forkNum, blkno, page, false);
+ else
+ PageSetLSN(page, fakepagelsn);
PageSetChecksumInplace(page, blkno);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index e641174798..9ab9e7394c 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -19,6 +19,7 @@
#include "storage/buf.h"
#include "storage/bufpage.h"
#include "storage/relfilenode.h"
+#include "utils/rel.h"
#include "utils/relcache.h"
#include "utils/snapmgr.h"
@@ -289,7 +290,7 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page)
&& ((snapshot)->snapshot_type == SNAPSHOT_MVCC
|| (snapshot)->snapshot_type == SNAPSHOT_TOAST)
&& !XLogRecPtrIsInvalid((snapshot)->lsn)
- && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn))
+ && PageGetLSN(page) > (snapshot)->lsn)
TestForOldSnapshot_impl(snapshot, relation);
}
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index f58d65cf28..35e53c8d74 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -618,8 +618,8 @@ typedef struct ViewOptions
* decoding snapshot.
*/
#define RelationIsAccessibleInLogicalDecoding(relation) \
- (XLogLogicalInfoActive() && \
- RelationNeedsWAL(relation) && \
+ (XLogLogicalInfoActive() && \
+ (relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT && \
(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation)))
/*
--
2.27.0
----Next_Part(Thu_Jan_21_00_28_44_2021_003)----
^ permalink raw reply [nested|flat] 36+ messages in thread
* [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping.
@ 2021-01-20 14:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 36+ messages in thread
From: Kyotaro Horiguchi @ 2021-01-20 14:59 UTC (permalink / raw)
WAL-skipping optimization omits bumping page LSN when ALTER TABLE SET
TABLESPACE. Also heap_page_prune does the same. However,
old_snapshot_threshold feature needs page LSN to be kept updated on
these operations so that TestForOldSnapshot properly find identify
whether a snapshot is invalidated.
---
src/backend/access/heap/pruneheap.c | 14 ++++++++++++++
src/backend/catalog/storage.c | 8 ++++++++
src/include/storage/bufmgr.h | 3 ++-
src/include/utils/rel.h | 4 ++--
4 files changed, 26 insertions(+), 3 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index e3a716a2a2..33c0841b36 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -70,6 +70,8 @@ static void heap_prune_record_redirect(PruneState *prstate,
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* Optionally prune and repair fragmentation in the specified page.
@@ -331,6 +333,18 @@ heap_page_prune(Relation relation, Buffer buffer,
PageSetLSN(BufferGetPage(buffer), recptr);
}
+ else if (relation->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT)
+ {
+ XLogRecPtr currlsn = GetXLogInsertRecPtr();
+ Page page = BufferGetPage(buffer);
+
+ Assert(wal_level < WAL_LEVEL_REPLICA);
+
+ /* we need to update page LSN to notify reader of pruning */
+ if (PageGetLSN(page) == currlsn)
+ currlsn = gistXLogAssignLSN();
+ PageSetLSN(page, currlsn);
+ }
}
else
{
diff --git a/src/backend/catalog/storage.c b/src/backend/catalog/storage.c
index cba7a9ada0..488ac6f760 100644
--- a/src/backend/catalog/storage.c
+++ b/src/backend/catalog/storage.c
@@ -75,6 +75,8 @@ typedef struct PendingRelSync
static PendingRelDelete *pendingDeletes = NULL; /* head of linked list */
HTAB *pendingSyncHash = NULL;
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* AddPendingSync
@@ -414,6 +416,7 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
bool copying_initfork;
BlockNumber nblocks;
BlockNumber blkno;
+ XLogRecPtr fakepagelsn;
page = (Page) buf.data;
@@ -434,6 +437,9 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
use_wal = XLogIsNeeded() &&
(relpersistence == RELPERSISTENCE_PERMANENT || copying_initfork);
+ if (!use_wal)
+ fakepagelsn = gistXLogAssignLSN();
+
nblocks = smgrnblocks(src, forkNum);
for (blkno = 0; blkno < nblocks; blkno++)
@@ -460,6 +466,8 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
*/
if (use_wal)
log_newpage(&dst->smgr_rnode.node, forkNum, blkno, page, false);
+ else
+ PageSetLSN(page, fakepagelsn);
PageSetChecksumInplace(page, blkno);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index e641174798..9ab9e7394c 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -19,6 +19,7 @@
#include "storage/buf.h"
#include "storage/bufpage.h"
#include "storage/relfilenode.h"
+#include "utils/rel.h"
#include "utils/relcache.h"
#include "utils/snapmgr.h"
@@ -289,7 +290,7 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page)
&& ((snapshot)->snapshot_type == SNAPSHOT_MVCC
|| (snapshot)->snapshot_type == SNAPSHOT_TOAST)
&& !XLogRecPtrIsInvalid((snapshot)->lsn)
- && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn))
+ && PageGetLSN(page) > (snapshot)->lsn)
TestForOldSnapshot_impl(snapshot, relation);
}
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index f58d65cf28..35e53c8d74 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -618,8 +618,8 @@ typedef struct ViewOptions
* decoding snapshot.
*/
#define RelationIsAccessibleInLogicalDecoding(relation) \
- (XLogLogicalInfoActive() && \
- RelationNeedsWAL(relation) && \
+ (XLogLogicalInfoActive() && \
+ (relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT && \
(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation)))
/*
--
2.27.0
----Next_Part(Thu_Jan_21_00_28_44_2021_003)----
^ permalink raw reply [nested|flat] 36+ messages in thread
* [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping.
@ 2021-01-20 14:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 36+ messages in thread
From: Kyotaro Horiguchi @ 2021-01-20 14:59 UTC (permalink / raw)
WAL-skipping optimization omits bumping page LSN when ALTER TABLE SET
TABLESPACE. Also heap_page_prune does the same. However,
old_snapshot_threshold feature needs page LSN to be kept updated on
these operations so that TestForOldSnapshot properly find identify
whether a snapshot is invalidated.
---
src/backend/access/heap/pruneheap.c | 14 ++++++++++++++
src/backend/catalog/storage.c | 8 ++++++++
src/include/storage/bufmgr.h | 3 ++-
src/include/utils/rel.h | 4 ++--
4 files changed, 26 insertions(+), 3 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index e3a716a2a2..33c0841b36 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -70,6 +70,8 @@ static void heap_prune_record_redirect(PruneState *prstate,
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* Optionally prune and repair fragmentation in the specified page.
@@ -331,6 +333,18 @@ heap_page_prune(Relation relation, Buffer buffer,
PageSetLSN(BufferGetPage(buffer), recptr);
}
+ else if (relation->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT)
+ {
+ XLogRecPtr currlsn = GetXLogInsertRecPtr();
+ Page page = BufferGetPage(buffer);
+
+ Assert(wal_level < WAL_LEVEL_REPLICA);
+
+ /* we need to update page LSN to notify reader of pruning */
+ if (PageGetLSN(page) == currlsn)
+ currlsn = gistXLogAssignLSN();
+ PageSetLSN(page, currlsn);
+ }
}
else
{
diff --git a/src/backend/catalog/storage.c b/src/backend/catalog/storage.c
index cba7a9ada0..488ac6f760 100644
--- a/src/backend/catalog/storage.c
+++ b/src/backend/catalog/storage.c
@@ -75,6 +75,8 @@ typedef struct PendingRelSync
static PendingRelDelete *pendingDeletes = NULL; /* head of linked list */
HTAB *pendingSyncHash = NULL;
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* AddPendingSync
@@ -414,6 +416,7 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
bool copying_initfork;
BlockNumber nblocks;
BlockNumber blkno;
+ XLogRecPtr fakepagelsn;
page = (Page) buf.data;
@@ -434,6 +437,9 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
use_wal = XLogIsNeeded() &&
(relpersistence == RELPERSISTENCE_PERMANENT || copying_initfork);
+ if (!use_wal)
+ fakepagelsn = gistXLogAssignLSN();
+
nblocks = smgrnblocks(src, forkNum);
for (blkno = 0; blkno < nblocks; blkno++)
@@ -460,6 +466,8 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
*/
if (use_wal)
log_newpage(&dst->smgr_rnode.node, forkNum, blkno, page, false);
+ else
+ PageSetLSN(page, fakepagelsn);
PageSetChecksumInplace(page, blkno);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index e641174798..9ab9e7394c 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -19,6 +19,7 @@
#include "storage/buf.h"
#include "storage/bufpage.h"
#include "storage/relfilenode.h"
+#include "utils/rel.h"
#include "utils/relcache.h"
#include "utils/snapmgr.h"
@@ -289,7 +290,7 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page)
&& ((snapshot)->snapshot_type == SNAPSHOT_MVCC
|| (snapshot)->snapshot_type == SNAPSHOT_TOAST)
&& !XLogRecPtrIsInvalid((snapshot)->lsn)
- && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn))
+ && PageGetLSN(page) > (snapshot)->lsn)
TestForOldSnapshot_impl(snapshot, relation);
}
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index f58d65cf28..35e53c8d74 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -618,8 +618,8 @@ typedef struct ViewOptions
* decoding snapshot.
*/
#define RelationIsAccessibleInLogicalDecoding(relation) \
- (XLogLogicalInfoActive() && \
- RelationNeedsWAL(relation) && \
+ (XLogLogicalInfoActive() && \
+ (relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT && \
(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation)))
/*
--
2.27.0
----Next_Part(Thu_Jan_21_00_28_44_2021_003)----
^ permalink raw reply [nested|flat] 36+ messages in thread
* [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping.
@ 2021-01-20 14:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 36+ messages in thread
From: Kyotaro Horiguchi @ 2021-01-20 14:59 UTC (permalink / raw)
WAL-skipping optimization omits bumping page LSN when ALTER TABLE SET
TABLESPACE. Also heap_page_prune does the same. However,
old_snapshot_threshold feature needs page LSN to be kept updated on
these operations so that TestForOldSnapshot properly find identify
whether a snapshot is invalidated.
---
src/backend/access/heap/pruneheap.c | 14 ++++++++++++++
src/backend/catalog/storage.c | 8 ++++++++
src/include/storage/bufmgr.h | 3 ++-
src/include/utils/rel.h | 4 ++--
4 files changed, 26 insertions(+), 3 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index e3a716a2a2..33c0841b36 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -70,6 +70,8 @@ static void heap_prune_record_redirect(PruneState *prstate,
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* Optionally prune and repair fragmentation in the specified page.
@@ -331,6 +333,18 @@ heap_page_prune(Relation relation, Buffer buffer,
PageSetLSN(BufferGetPage(buffer), recptr);
}
+ else if (relation->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT)
+ {
+ XLogRecPtr currlsn = GetXLogInsertRecPtr();
+ Page page = BufferGetPage(buffer);
+
+ Assert(wal_level < WAL_LEVEL_REPLICA);
+
+ /* we need to update page LSN to notify reader of pruning */
+ if (PageGetLSN(page) == currlsn)
+ currlsn = gistXLogAssignLSN();
+ PageSetLSN(page, currlsn);
+ }
}
else
{
diff --git a/src/backend/catalog/storage.c b/src/backend/catalog/storage.c
index cba7a9ada0..488ac6f760 100644
--- a/src/backend/catalog/storage.c
+++ b/src/backend/catalog/storage.c
@@ -75,6 +75,8 @@ typedef struct PendingRelSync
static PendingRelDelete *pendingDeletes = NULL; /* head of linked list */
HTAB *pendingSyncHash = NULL;
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* AddPendingSync
@@ -414,6 +416,7 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
bool copying_initfork;
BlockNumber nblocks;
BlockNumber blkno;
+ XLogRecPtr fakepagelsn;
page = (Page) buf.data;
@@ -434,6 +437,9 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
use_wal = XLogIsNeeded() &&
(relpersistence == RELPERSISTENCE_PERMANENT || copying_initfork);
+ if (!use_wal)
+ fakepagelsn = gistXLogAssignLSN();
+
nblocks = smgrnblocks(src, forkNum);
for (blkno = 0; blkno < nblocks; blkno++)
@@ -460,6 +466,8 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
*/
if (use_wal)
log_newpage(&dst->smgr_rnode.node, forkNum, blkno, page, false);
+ else
+ PageSetLSN(page, fakepagelsn);
PageSetChecksumInplace(page, blkno);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index e641174798..9ab9e7394c 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -19,6 +19,7 @@
#include "storage/buf.h"
#include "storage/bufpage.h"
#include "storage/relfilenode.h"
+#include "utils/rel.h"
#include "utils/relcache.h"
#include "utils/snapmgr.h"
@@ -289,7 +290,7 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page)
&& ((snapshot)->snapshot_type == SNAPSHOT_MVCC
|| (snapshot)->snapshot_type == SNAPSHOT_TOAST)
&& !XLogRecPtrIsInvalid((snapshot)->lsn)
- && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn))
+ && PageGetLSN(page) > (snapshot)->lsn)
TestForOldSnapshot_impl(snapshot, relation);
}
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index f58d65cf28..35e53c8d74 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -618,8 +618,8 @@ typedef struct ViewOptions
* decoding snapshot.
*/
#define RelationIsAccessibleInLogicalDecoding(relation) \
- (XLogLogicalInfoActive() && \
- RelationNeedsWAL(relation) && \
+ (XLogLogicalInfoActive() && \
+ (relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT && \
(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation)))
/*
--
2.27.0
----Next_Part(Thu_Jan_21_00_28_44_2021_003)----
^ permalink raw reply [nested|flat] 36+ messages in thread
* [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping.
@ 2021-01-20 14:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 36+ messages in thread
From: Kyotaro Horiguchi @ 2021-01-20 14:59 UTC (permalink / raw)
WAL-skipping optimization omits bumping page LSN when ALTER TABLE SET
TABLESPACE. Also heap_page_prune does the same. However,
old_snapshot_threshold feature needs page LSN to be kept updated on
these operations so that TestForOldSnapshot properly find identify
whether a snapshot is invalidated.
---
src/backend/access/heap/pruneheap.c | 14 ++++++++++++++
src/backend/catalog/storage.c | 8 ++++++++
src/include/storage/bufmgr.h | 3 ++-
src/include/utils/rel.h | 4 ++--
4 files changed, 26 insertions(+), 3 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index e3a716a2a2..33c0841b36 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -70,6 +70,8 @@ static void heap_prune_record_redirect(PruneState *prstate,
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* Optionally prune and repair fragmentation in the specified page.
@@ -331,6 +333,18 @@ heap_page_prune(Relation relation, Buffer buffer,
PageSetLSN(BufferGetPage(buffer), recptr);
}
+ else if (relation->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT)
+ {
+ XLogRecPtr currlsn = GetXLogInsertRecPtr();
+ Page page = BufferGetPage(buffer);
+
+ Assert(wal_level < WAL_LEVEL_REPLICA);
+
+ /* we need to update page LSN to notify reader of pruning */
+ if (PageGetLSN(page) == currlsn)
+ currlsn = gistXLogAssignLSN();
+ PageSetLSN(page, currlsn);
+ }
}
else
{
diff --git a/src/backend/catalog/storage.c b/src/backend/catalog/storage.c
index cba7a9ada0..488ac6f760 100644
--- a/src/backend/catalog/storage.c
+++ b/src/backend/catalog/storage.c
@@ -75,6 +75,8 @@ typedef struct PendingRelSync
static PendingRelDelete *pendingDeletes = NULL; /* head of linked list */
HTAB *pendingSyncHash = NULL;
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* AddPendingSync
@@ -414,6 +416,7 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
bool copying_initfork;
BlockNumber nblocks;
BlockNumber blkno;
+ XLogRecPtr fakepagelsn;
page = (Page) buf.data;
@@ -434,6 +437,9 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
use_wal = XLogIsNeeded() &&
(relpersistence == RELPERSISTENCE_PERMANENT || copying_initfork);
+ if (!use_wal)
+ fakepagelsn = gistXLogAssignLSN();
+
nblocks = smgrnblocks(src, forkNum);
for (blkno = 0; blkno < nblocks; blkno++)
@@ -460,6 +466,8 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
*/
if (use_wal)
log_newpage(&dst->smgr_rnode.node, forkNum, blkno, page, false);
+ else
+ PageSetLSN(page, fakepagelsn);
PageSetChecksumInplace(page, blkno);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index e641174798..9ab9e7394c 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -19,6 +19,7 @@
#include "storage/buf.h"
#include "storage/bufpage.h"
#include "storage/relfilenode.h"
+#include "utils/rel.h"
#include "utils/relcache.h"
#include "utils/snapmgr.h"
@@ -289,7 +290,7 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page)
&& ((snapshot)->snapshot_type == SNAPSHOT_MVCC
|| (snapshot)->snapshot_type == SNAPSHOT_TOAST)
&& !XLogRecPtrIsInvalid((snapshot)->lsn)
- && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn))
+ && PageGetLSN(page) > (snapshot)->lsn)
TestForOldSnapshot_impl(snapshot, relation);
}
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index f58d65cf28..35e53c8d74 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -618,8 +618,8 @@ typedef struct ViewOptions
* decoding snapshot.
*/
#define RelationIsAccessibleInLogicalDecoding(relation) \
- (XLogLogicalInfoActive() && \
- RelationNeedsWAL(relation) && \
+ (XLogLogicalInfoActive() && \
+ (relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT && \
(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation)))
/*
--
2.27.0
----Next_Part(Thu_Jan_21_00_28_44_2021_003)----
^ permalink raw reply [nested|flat] 36+ messages in thread
* [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping.
@ 2021-01-20 14:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 36+ messages in thread
From: Kyotaro Horiguchi @ 2021-01-20 14:59 UTC (permalink / raw)
WAL-skipping optimization omits bumping page LSN when ALTER TABLE SET
TABLESPACE. Also heap_page_prune does the same. However,
old_snapshot_threshold feature needs page LSN to be kept updated on
these operations so that TestForOldSnapshot properly find identify
whether a snapshot is invalidated.
---
src/backend/access/heap/pruneheap.c | 14 ++++++++++++++
src/backend/catalog/storage.c | 8 ++++++++
src/include/storage/bufmgr.h | 3 ++-
src/include/utils/rel.h | 4 ++--
4 files changed, 26 insertions(+), 3 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index e3a716a2a2..33c0841b36 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -70,6 +70,8 @@ static void heap_prune_record_redirect(PruneState *prstate,
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* Optionally prune and repair fragmentation in the specified page.
@@ -331,6 +333,18 @@ heap_page_prune(Relation relation, Buffer buffer,
PageSetLSN(BufferGetPage(buffer), recptr);
}
+ else if (relation->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT)
+ {
+ XLogRecPtr currlsn = GetXLogInsertRecPtr();
+ Page page = BufferGetPage(buffer);
+
+ Assert(wal_level < WAL_LEVEL_REPLICA);
+
+ /* we need to update page LSN to notify reader of pruning */
+ if (PageGetLSN(page) == currlsn)
+ currlsn = gistXLogAssignLSN();
+ PageSetLSN(page, currlsn);
+ }
}
else
{
diff --git a/src/backend/catalog/storage.c b/src/backend/catalog/storage.c
index cba7a9ada0..488ac6f760 100644
--- a/src/backend/catalog/storage.c
+++ b/src/backend/catalog/storage.c
@@ -75,6 +75,8 @@ typedef struct PendingRelSync
static PendingRelDelete *pendingDeletes = NULL; /* head of linked list */
HTAB *pendingSyncHash = NULL;
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* AddPendingSync
@@ -414,6 +416,7 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
bool copying_initfork;
BlockNumber nblocks;
BlockNumber blkno;
+ XLogRecPtr fakepagelsn;
page = (Page) buf.data;
@@ -434,6 +437,9 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
use_wal = XLogIsNeeded() &&
(relpersistence == RELPERSISTENCE_PERMANENT || copying_initfork);
+ if (!use_wal)
+ fakepagelsn = gistXLogAssignLSN();
+
nblocks = smgrnblocks(src, forkNum);
for (blkno = 0; blkno < nblocks; blkno++)
@@ -460,6 +466,8 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
*/
if (use_wal)
log_newpage(&dst->smgr_rnode.node, forkNum, blkno, page, false);
+ else
+ PageSetLSN(page, fakepagelsn);
PageSetChecksumInplace(page, blkno);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index e641174798..9ab9e7394c 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -19,6 +19,7 @@
#include "storage/buf.h"
#include "storage/bufpage.h"
#include "storage/relfilenode.h"
+#include "utils/rel.h"
#include "utils/relcache.h"
#include "utils/snapmgr.h"
@@ -289,7 +290,7 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page)
&& ((snapshot)->snapshot_type == SNAPSHOT_MVCC
|| (snapshot)->snapshot_type == SNAPSHOT_TOAST)
&& !XLogRecPtrIsInvalid((snapshot)->lsn)
- && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn))
+ && PageGetLSN(page) > (snapshot)->lsn)
TestForOldSnapshot_impl(snapshot, relation);
}
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index f58d65cf28..35e53c8d74 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -618,8 +618,8 @@ typedef struct ViewOptions
* decoding snapshot.
*/
#define RelationIsAccessibleInLogicalDecoding(relation) \
- (XLogLogicalInfoActive() && \
- RelationNeedsWAL(relation) && \
+ (XLogLogicalInfoActive() && \
+ (relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT && \
(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation)))
/*
--
2.27.0
----Next_Part(Thu_Jan_21_00_28_44_2021_003)----
^ permalink raw reply [nested|flat] 36+ messages in thread
* [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping.
@ 2021-01-20 14:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 36+ messages in thread
From: Kyotaro Horiguchi @ 2021-01-20 14:59 UTC (permalink / raw)
WAL-skipping optimization omits bumping page LSN when ALTER TABLE SET
TABLESPACE. Also heap_page_prune does the same. However,
old_snapshot_threshold feature needs page LSN to be kept updated on
these operations so that TestForOldSnapshot properly find identify
whether a snapshot is invalidated.
---
src/backend/access/heap/pruneheap.c | 14 ++++++++++++++
src/backend/catalog/storage.c | 8 ++++++++
src/include/storage/bufmgr.h | 3 ++-
src/include/utils/rel.h | 4 ++--
4 files changed, 26 insertions(+), 3 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index e3a716a2a2..33c0841b36 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -70,6 +70,8 @@ static void heap_prune_record_redirect(PruneState *prstate,
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* Optionally prune and repair fragmentation in the specified page.
@@ -331,6 +333,18 @@ heap_page_prune(Relation relation, Buffer buffer,
PageSetLSN(BufferGetPage(buffer), recptr);
}
+ else if (relation->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT)
+ {
+ XLogRecPtr currlsn = GetXLogInsertRecPtr();
+ Page page = BufferGetPage(buffer);
+
+ Assert(wal_level < WAL_LEVEL_REPLICA);
+
+ /* we need to update page LSN to notify reader of pruning */
+ if (PageGetLSN(page) == currlsn)
+ currlsn = gistXLogAssignLSN();
+ PageSetLSN(page, currlsn);
+ }
}
else
{
diff --git a/src/backend/catalog/storage.c b/src/backend/catalog/storage.c
index cba7a9ada0..488ac6f760 100644
--- a/src/backend/catalog/storage.c
+++ b/src/backend/catalog/storage.c
@@ -75,6 +75,8 @@ typedef struct PendingRelSync
static PendingRelDelete *pendingDeletes = NULL; /* head of linked list */
HTAB *pendingSyncHash = NULL;
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* AddPendingSync
@@ -414,6 +416,7 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
bool copying_initfork;
BlockNumber nblocks;
BlockNumber blkno;
+ XLogRecPtr fakepagelsn;
page = (Page) buf.data;
@@ -434,6 +437,9 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
use_wal = XLogIsNeeded() &&
(relpersistence == RELPERSISTENCE_PERMANENT || copying_initfork);
+ if (!use_wal)
+ fakepagelsn = gistXLogAssignLSN();
+
nblocks = smgrnblocks(src, forkNum);
for (blkno = 0; blkno < nblocks; blkno++)
@@ -460,6 +466,8 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
*/
if (use_wal)
log_newpage(&dst->smgr_rnode.node, forkNum, blkno, page, false);
+ else
+ PageSetLSN(page, fakepagelsn);
PageSetChecksumInplace(page, blkno);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index e641174798..9ab9e7394c 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -19,6 +19,7 @@
#include "storage/buf.h"
#include "storage/bufpage.h"
#include "storage/relfilenode.h"
+#include "utils/rel.h"
#include "utils/relcache.h"
#include "utils/snapmgr.h"
@@ -289,7 +290,7 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page)
&& ((snapshot)->snapshot_type == SNAPSHOT_MVCC
|| (snapshot)->snapshot_type == SNAPSHOT_TOAST)
&& !XLogRecPtrIsInvalid((snapshot)->lsn)
- && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn))
+ && PageGetLSN(page) > (snapshot)->lsn)
TestForOldSnapshot_impl(snapshot, relation);
}
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index f58d65cf28..35e53c8d74 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -618,8 +618,8 @@ typedef struct ViewOptions
* decoding snapshot.
*/
#define RelationIsAccessibleInLogicalDecoding(relation) \
- (XLogLogicalInfoActive() && \
- RelationNeedsWAL(relation) && \
+ (XLogLogicalInfoActive() && \
+ (relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT && \
(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation)))
/*
--
2.27.0
----Next_Part(Thu_Jan_21_00_28_44_2021_003)----
^ permalink raw reply [nested|flat] 36+ messages in thread
* [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping.
@ 2021-01-20 14:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 36+ messages in thread
From: Kyotaro Horiguchi @ 2021-01-20 14:59 UTC (permalink / raw)
WAL-skipping optimization omits bumping page LSN when ALTER TABLE SET
TABLESPACE. Also heap_page_prune does the same. However,
old_snapshot_threshold feature needs page LSN to be kept updated on
these operations so that TestForOldSnapshot properly find identify
whether a snapshot is invalidated.
---
src/backend/access/heap/pruneheap.c | 14 ++++++++++++++
src/backend/catalog/storage.c | 8 ++++++++
src/include/storage/bufmgr.h | 3 ++-
src/include/utils/rel.h | 4 ++--
4 files changed, 26 insertions(+), 3 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index e3a716a2a2..33c0841b36 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -70,6 +70,8 @@ static void heap_prune_record_redirect(PruneState *prstate,
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* Optionally prune and repair fragmentation in the specified page.
@@ -331,6 +333,18 @@ heap_page_prune(Relation relation, Buffer buffer,
PageSetLSN(BufferGetPage(buffer), recptr);
}
+ else if (relation->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT)
+ {
+ XLogRecPtr currlsn = GetXLogInsertRecPtr();
+ Page page = BufferGetPage(buffer);
+
+ Assert(wal_level < WAL_LEVEL_REPLICA);
+
+ /* we need to update page LSN to notify reader of pruning */
+ if (PageGetLSN(page) == currlsn)
+ currlsn = gistXLogAssignLSN();
+ PageSetLSN(page, currlsn);
+ }
}
else
{
diff --git a/src/backend/catalog/storage.c b/src/backend/catalog/storage.c
index cba7a9ada0..488ac6f760 100644
--- a/src/backend/catalog/storage.c
+++ b/src/backend/catalog/storage.c
@@ -75,6 +75,8 @@ typedef struct PendingRelSync
static PendingRelDelete *pendingDeletes = NULL; /* head of linked list */
HTAB *pendingSyncHash = NULL;
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* AddPendingSync
@@ -414,6 +416,7 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
bool copying_initfork;
BlockNumber nblocks;
BlockNumber blkno;
+ XLogRecPtr fakepagelsn;
page = (Page) buf.data;
@@ -434,6 +437,9 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
use_wal = XLogIsNeeded() &&
(relpersistence == RELPERSISTENCE_PERMANENT || copying_initfork);
+ if (!use_wal)
+ fakepagelsn = gistXLogAssignLSN();
+
nblocks = smgrnblocks(src, forkNum);
for (blkno = 0; blkno < nblocks; blkno++)
@@ -460,6 +466,8 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
*/
if (use_wal)
log_newpage(&dst->smgr_rnode.node, forkNum, blkno, page, false);
+ else
+ PageSetLSN(page, fakepagelsn);
PageSetChecksumInplace(page, blkno);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index e641174798..9ab9e7394c 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -19,6 +19,7 @@
#include "storage/buf.h"
#include "storage/bufpage.h"
#include "storage/relfilenode.h"
+#include "utils/rel.h"
#include "utils/relcache.h"
#include "utils/snapmgr.h"
@@ -289,7 +290,7 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page)
&& ((snapshot)->snapshot_type == SNAPSHOT_MVCC
|| (snapshot)->snapshot_type == SNAPSHOT_TOAST)
&& !XLogRecPtrIsInvalid((snapshot)->lsn)
- && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn))
+ && PageGetLSN(page) > (snapshot)->lsn)
TestForOldSnapshot_impl(snapshot, relation);
}
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index f58d65cf28..35e53c8d74 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -618,8 +618,8 @@ typedef struct ViewOptions
* decoding snapshot.
*/
#define RelationIsAccessibleInLogicalDecoding(relation) \
- (XLogLogicalInfoActive() && \
- RelationNeedsWAL(relation) && \
+ (XLogLogicalInfoActive() && \
+ (relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT && \
(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation)))
/*
--
2.27.0
----Next_Part(Thu_Jan_21_00_28_44_2021_003)----
^ permalink raw reply [nested|flat] 36+ messages in thread
* [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping.
@ 2021-01-20 14:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 36+ messages in thread
From: Kyotaro Horiguchi @ 2021-01-20 14:59 UTC (permalink / raw)
WAL-skipping optimization omits bumping page LSN when ALTER TABLE SET
TABLESPACE. Also heap_page_prune does the same. However,
old_snapshot_threshold feature needs page LSN to be kept updated on
these operations so that TestForOldSnapshot properly find identify
whether a snapshot is invalidated.
---
src/backend/access/heap/pruneheap.c | 14 ++++++++++++++
src/backend/catalog/storage.c | 8 ++++++++
src/include/storage/bufmgr.h | 3 ++-
src/include/utils/rel.h | 4 ++--
4 files changed, 26 insertions(+), 3 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index e3a716a2a2..33c0841b36 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -70,6 +70,8 @@ static void heap_prune_record_redirect(PruneState *prstate,
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* Optionally prune and repair fragmentation in the specified page.
@@ -331,6 +333,18 @@ heap_page_prune(Relation relation, Buffer buffer,
PageSetLSN(BufferGetPage(buffer), recptr);
}
+ else if (relation->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT)
+ {
+ XLogRecPtr currlsn = GetXLogInsertRecPtr();
+ Page page = BufferGetPage(buffer);
+
+ Assert(wal_level < WAL_LEVEL_REPLICA);
+
+ /* we need to update page LSN to notify reader of pruning */
+ if (PageGetLSN(page) == currlsn)
+ currlsn = gistXLogAssignLSN();
+ PageSetLSN(page, currlsn);
+ }
}
else
{
diff --git a/src/backend/catalog/storage.c b/src/backend/catalog/storage.c
index cba7a9ada0..488ac6f760 100644
--- a/src/backend/catalog/storage.c
+++ b/src/backend/catalog/storage.c
@@ -75,6 +75,8 @@ typedef struct PendingRelSync
static PendingRelDelete *pendingDeletes = NULL; /* head of linked list */
HTAB *pendingSyncHash = NULL;
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* AddPendingSync
@@ -414,6 +416,7 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
bool copying_initfork;
BlockNumber nblocks;
BlockNumber blkno;
+ XLogRecPtr fakepagelsn;
page = (Page) buf.data;
@@ -434,6 +437,9 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
use_wal = XLogIsNeeded() &&
(relpersistence == RELPERSISTENCE_PERMANENT || copying_initfork);
+ if (!use_wal)
+ fakepagelsn = gistXLogAssignLSN();
+
nblocks = smgrnblocks(src, forkNum);
for (blkno = 0; blkno < nblocks; blkno++)
@@ -460,6 +466,8 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
*/
if (use_wal)
log_newpage(&dst->smgr_rnode.node, forkNum, blkno, page, false);
+ else
+ PageSetLSN(page, fakepagelsn);
PageSetChecksumInplace(page, blkno);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index e641174798..9ab9e7394c 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -19,6 +19,7 @@
#include "storage/buf.h"
#include "storage/bufpage.h"
#include "storage/relfilenode.h"
+#include "utils/rel.h"
#include "utils/relcache.h"
#include "utils/snapmgr.h"
@@ -289,7 +290,7 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page)
&& ((snapshot)->snapshot_type == SNAPSHOT_MVCC
|| (snapshot)->snapshot_type == SNAPSHOT_TOAST)
&& !XLogRecPtrIsInvalid((snapshot)->lsn)
- && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn))
+ && PageGetLSN(page) > (snapshot)->lsn)
TestForOldSnapshot_impl(snapshot, relation);
}
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index f58d65cf28..35e53c8d74 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -618,8 +618,8 @@ typedef struct ViewOptions
* decoding snapshot.
*/
#define RelationIsAccessibleInLogicalDecoding(relation) \
- (XLogLogicalInfoActive() && \
- RelationNeedsWAL(relation) && \
+ (XLogLogicalInfoActive() && \
+ (relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT && \
(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation)))
/*
--
2.27.0
----Next_Part(Thu_Jan_21_00_28_44_2021_003)----
^ permalink raw reply [nested|flat] 36+ messages in thread
* [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping.
@ 2021-01-20 14:59 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 36+ messages in thread
From: Kyotaro Horiguchi @ 2021-01-20 14:59 UTC (permalink / raw)
WAL-skipping optimization omits bumping page LSN when ALTER TABLE SET
TABLESPACE. Also heap_page_prune does the same. However,
old_snapshot_threshold feature needs page LSN to be kept updated on
these operations so that TestForOldSnapshot properly find identify
whether a snapshot is invalidated.
---
src/backend/access/heap/pruneheap.c | 14 ++++++++++++++
src/backend/catalog/storage.c | 8 ++++++++
src/include/storage/bufmgr.h | 3 ++-
src/include/utils/rel.h | 4 ++--
4 files changed, 26 insertions(+), 3 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index e3a716a2a2..33c0841b36 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -70,6 +70,8 @@ static void heap_prune_record_redirect(PruneState *prstate,
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* Optionally prune and repair fragmentation in the specified page.
@@ -331,6 +333,18 @@ heap_page_prune(Relation relation, Buffer buffer,
PageSetLSN(BufferGetPage(buffer), recptr);
}
+ else if (relation->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT)
+ {
+ XLogRecPtr currlsn = GetXLogInsertRecPtr();
+ Page page = BufferGetPage(buffer);
+
+ Assert(wal_level < WAL_LEVEL_REPLICA);
+
+ /* we need to update page LSN to notify reader of pruning */
+ if (PageGetLSN(page) == currlsn)
+ currlsn = gistXLogAssignLSN();
+ PageSetLSN(page, currlsn);
+ }
}
else
{
diff --git a/src/backend/catalog/storage.c b/src/backend/catalog/storage.c
index cba7a9ada0..488ac6f760 100644
--- a/src/backend/catalog/storage.c
+++ b/src/backend/catalog/storage.c
@@ -75,6 +75,8 @@ typedef struct PendingRelSync
static PendingRelDelete *pendingDeletes = NULL; /* head of linked list */
HTAB *pendingSyncHash = NULL;
+/* XXXXXXXX tmporary modification */
+extern XLogRecPtr gistXLogAssignLSN(void);
/*
* AddPendingSync
@@ -414,6 +416,7 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
bool copying_initfork;
BlockNumber nblocks;
BlockNumber blkno;
+ XLogRecPtr fakepagelsn;
page = (Page) buf.data;
@@ -434,6 +437,9 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
use_wal = XLogIsNeeded() &&
(relpersistence == RELPERSISTENCE_PERMANENT || copying_initfork);
+ if (!use_wal)
+ fakepagelsn = gistXLogAssignLSN();
+
nblocks = smgrnblocks(src, forkNum);
for (blkno = 0; blkno < nblocks; blkno++)
@@ -460,6 +466,8 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
*/
if (use_wal)
log_newpage(&dst->smgr_rnode.node, forkNum, blkno, page, false);
+ else
+ PageSetLSN(page, fakepagelsn);
PageSetChecksumInplace(page, blkno);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index e641174798..9ab9e7394c 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -19,6 +19,7 @@
#include "storage/buf.h"
#include "storage/bufpage.h"
#include "storage/relfilenode.h"
+#include "utils/rel.h"
#include "utils/relcache.h"
#include "utils/snapmgr.h"
@@ -289,7 +290,7 @@ TestForOldSnapshot(Snapshot snapshot, Relation relation, Page page)
&& ((snapshot)->snapshot_type == SNAPSHOT_MVCC
|| (snapshot)->snapshot_type == SNAPSHOT_TOAST)
&& !XLogRecPtrIsInvalid((snapshot)->lsn)
- && (!XLogIsNeeded() || PageGetLSN(page) > (snapshot)->lsn))
+ && PageGetLSN(page) > (snapshot)->lsn)
TestForOldSnapshot_impl(snapshot, relation);
}
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index f58d65cf28..35e53c8d74 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -618,8 +618,8 @@ typedef struct ViewOptions
* decoding snapshot.
*/
#define RelationIsAccessibleInLogicalDecoding(relation) \
- (XLogLogicalInfoActive() && \
- RelationNeedsWAL(relation) && \
+ (XLogLogicalInfoActive() && \
+ (relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT && \
(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation)))
/*
--
2.27.0
----Next_Part(Thu_Jan_21_00_28_44_2021_003)----
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: add PROCESS_MAIN to VACUUM
@ 2023-03-01 19:13 Nathan Bossart <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Nathan Bossart @ 2023-03-01 19:13 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Michael Paquier <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
On Wed, Mar 01, 2023 at 07:09:53PM +0100, Alvaro Herrera wrote:
> On 2023-Mar-01, Michael Paquier wrote:
>
>> +-- PROCESS_MAIN option
>> +VACUUM (PROCESS_MAIN FALSE) vactst;
>> +VACUUM (PROCESS_MAIN FALSE, PROCESS_TOAST FALSE) vactst;
>> +VACUUM (PROCESS_MAIN FALSE, FULL) vactst;
>>
>> Thinking a bit here. This set of tests does not make sure that the
>> main relation and/or the toast relation have been actually processed.
>> pg_stat_user_tables does not track what's happening on the toast
>> relations. So... What about adding some tests in 100_vacuumdb.pl
>> that rely on vacuumdb --verbose and check the logs produced? We
>> should make sure that the toast or the main relation are processed,
>> by tracking, for example, logs like vacuuming "schema.table". When
>> FULL is involved, we may want to track the changes on relfilenodes
>> depending on what's wanted.
>
> Maybe instead of reading the log, read values from pg_stat_all_tables.
Here is an attempt at that. Thanks for the idea.
--
Nathan Bossart
Amazon Web Services: https://aws.amazon.com
Attachments:
[text/x-diff] v4-0001-add-PROCESS_MAIN-to-VACUUM.patch (17.7K, ../../20230301191344.GA1543729@nathanxps13/2-v4-0001-add-PROCESS_MAIN-to-VACUUM.patch)
download | inline diff:
From 3b628a96fdebf1a052472de307f36b20a5a1414b Mon Sep 17 00:00:00 2001
From: Nathan Bossart <[email protected]>
Date: Thu, 29 Dec 2022 15:31:49 -0800
Subject: [PATCH v4 1/1] add PROCESS_MAIN to VACUUM
---
doc/src/sgml/ref/vacuum.sgml | 13 ++++++
doc/src/sgml/ref/vacuumdb.sgml | 15 +++++++
src/backend/commands/vacuum.c | 28 ++++++++++---
src/backend/postmaster/autovacuum.c | 4 +-
src/bin/psql/tab-complete.c | 4 +-
src/bin/scripts/t/100_vacuumdb.pl | 7 ++++
src/bin/scripts/vacuumdb.c | 24 +++++++++++
src/include/commands/vacuum.h | 9 +++--
src/test/regress/expected/vacuum.out | 60 ++++++++++++++++++++++++++++
src/test/regress/sql/vacuum.sql | 24 +++++++++++
10 files changed, 175 insertions(+), 13 deletions(-)
diff --git a/doc/src/sgml/ref/vacuum.sgml b/doc/src/sgml/ref/vacuum.sgml
index 545b23b54f..b6d30b5764 100644
--- a/doc/src/sgml/ref/vacuum.sgml
+++ b/doc/src/sgml/ref/vacuum.sgml
@@ -33,6 +33,7 @@ VACUUM [ FULL ] [ FREEZE ] [ VERBOSE ] [ ANALYZE ] [ <replaceable class="paramet
DISABLE_PAGE_SKIPPING [ <replaceable class="parameter">boolean</replaceable> ]
SKIP_LOCKED [ <replaceable class="parameter">boolean</replaceable> ]
INDEX_CLEANUP { AUTO | ON | OFF }
+ PROCESS_MAIN [ <replaceable class="parameter">boolean</replaceable> ]
PROCESS_TOAST [ <replaceable class="parameter">boolean</replaceable> ]
TRUNCATE [ <replaceable class="parameter">boolean</replaceable> ]
PARALLEL <replaceable class="parameter">integer</replaceable>
@@ -238,6 +239,18 @@ VACUUM [ FULL ] [ FREEZE ] [ VERBOSE ] [ ANALYZE ] [ <replaceable class="paramet
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>PROCESS_MAIN</literal></term>
+ <listitem>
+ <para>
+ Specifies that <command>VACUUM</command> should attempt to process the
+ main relation. This is usually the desired behavior and is the default.
+ Setting this option to false may be useful when it is only necessary to
+ vacuum a relation's corresponding <literal>TOAST</literal> table.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>PROCESS_TOAST</literal></term>
<listitem>
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index 841aced3bd..74bac2d4ba 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -317,6 +317,21 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--no-process-main</option></term>
+ <listitem>
+ <para>
+ Skip the main relation.
+ </para>
+ <note>
+ <para>
+ This option is only available for servers running
+ <productname>PostgreSQL</productname> 16 and later.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--no-process-toast</option></term>
<listitem>
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index aa79d9de4d..88ac1df2b5 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -115,6 +115,7 @@ ExecVacuum(ParseState *pstate, VacuumStmt *vacstmt, bool isTopLevel)
bool freeze = false;
bool full = false;
bool disable_page_skipping = false;
+ bool process_main = true;
bool process_toast = true;
bool skip_database_stats = false;
bool only_database_stats = false;
@@ -168,6 +169,8 @@ ExecVacuum(ParseState *pstate, VacuumStmt *vacstmt, bool isTopLevel)
params.index_cleanup = get_vacoptval_from_boolean(opt);
}
}
+ else if (strcmp(opt->defname, "process_main") == 0)
+ process_main = defGetBoolean(opt);
else if (strcmp(opt->defname, "process_toast") == 0)
process_toast = defGetBoolean(opt);
else if (strcmp(opt->defname, "truncate") == 0)
@@ -226,7 +229,8 @@ ExecVacuum(ParseState *pstate, VacuumStmt *vacstmt, bool isTopLevel)
(disable_page_skipping ? VACOPT_DISABLE_PAGE_SKIPPING : 0) |
(process_toast ? VACOPT_PROCESS_TOAST : 0) |
(skip_database_stats ? VACOPT_SKIP_DATABASE_STATS : 0) |
- (only_database_stats ? VACOPT_ONLY_DATABASE_STATS : 0);
+ (only_database_stats ? VACOPT_ONLY_DATABASE_STATS : 0) |
+ (process_main ? VACOPT_PROCESS_MAIN : 0);
/* sanity checks on options */
Assert(params.options & (VACOPT_VACUUM | VACOPT_ANALYZE));
@@ -367,9 +371,10 @@ vacuum(List *relations, VacuumParams *params,
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("ONLY_DATABASE_STATS cannot be specified with a list of tables")));
- /* don't require people to turn off PROCESS_TOAST explicitly */
+ /* don't require people to turn off PROCESS_TOAST/MAIN explicitly */
if (params->options & ~(VACOPT_VACUUM |
VACOPT_VERBOSE |
+ VACOPT_PROCESS_MAIN |
VACOPT_PROCESS_TOAST |
VACOPT_ONLY_DATABASE_STATS))
ereport(ERROR,
@@ -2031,10 +2036,12 @@ vacuum_rel(Oid relid, RangeVar *relation, VacuumParams *params, bool skip_privs)
/*
* Remember the relation's TOAST relation for later, if the caller asked
* us to process it. In VACUUM FULL, though, the toast table is
- * automatically rebuilt by cluster_rel so we shouldn't recurse to it.
+ * automatically rebuilt by cluster_rel so we shouldn't recurse to it
+ * unless PROCESS_MAIN is disabled.
*/
if ((params->options & VACOPT_PROCESS_TOAST) != 0 &&
- (params->options & VACOPT_FULL) == 0)
+ ((params->options & VACOPT_FULL) == 0 ||
+ (params->options & VACOPT_PROCESS_MAIN) == 0))
toast_relid = rel->rd_rel->reltoastrelid;
else
toast_relid = InvalidOid;
@@ -2053,7 +2060,8 @@ vacuum_rel(Oid relid, RangeVar *relation, VacuumParams *params, bool skip_privs)
/*
* Do the actual work --- either FULL or "lazy" vacuum
*/
- if (params->options & VACOPT_FULL)
+ if (params->options & VACOPT_FULL &&
+ params->options & VACOPT_PROCESS_MAIN)
{
ClusterParams cluster_params = {0};
@@ -2067,7 +2075,7 @@ vacuum_rel(Oid relid, RangeVar *relation, VacuumParams *params, bool skip_privs)
/* VACUUM FULL is now a variant of CLUSTER; see cluster.c */
cluster_rel(relid, InvalidOid, &cluster_params);
}
- else
+ else if (params->options & VACOPT_PROCESS_MAIN)
table_relation_vacuum(rel, params, vac_strategy);
/* Roll back any GUC changes executed by index functions */
@@ -2094,7 +2102,15 @@ vacuum_rel(Oid relid, RangeVar *relation, VacuumParams *params, bool skip_privs)
* totally unimportant for toast relations.
*/
if (toast_relid != InvalidOid)
+ {
+ /* we force VACOPT_PROCESS_MAIN so vacuum_rel() processes it */
+ bool force_opt = ((params->options & VACOPT_PROCESS_MAIN) == 0);
+
+ params->options |= VACOPT_PROCESS_MAIN;
vacuum_rel(toast_relid, NULL, params, true);
+ if (force_opt)
+ params->options &= ~VACOPT_PROCESS_MAIN;
+ }
/*
* Now release the session-level lock on the main table.
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index ff6149a179..c0e2e00a7e 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -2860,7 +2860,9 @@ table_recheck_autovac(Oid relid, HTAB *table_toast_map,
* skip vac_update_datfrozenxid(); we'll do that separately.
*/
tab->at_params.options =
- (dovacuum ? (VACOPT_VACUUM | VACOPT_SKIP_DATABASE_STATS) : 0) |
+ (dovacuum ? (VACOPT_VACUUM |
+ VACOPT_PROCESS_MAIN |
+ VACOPT_SKIP_DATABASE_STATS) : 0) |
(doanalyze ? VACOPT_ANALYZE : 0) |
(!wraparound ? VACOPT_SKIP_LOCKED : 0);
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 5e1882eaea..8f12af799b 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -4618,10 +4618,10 @@ psql_completion(const char *text, int start, int end)
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
COMPLETE_WITH("FULL", "FREEZE", "ANALYZE", "VERBOSE",
"DISABLE_PAGE_SKIPPING", "SKIP_LOCKED",
- "INDEX_CLEANUP", "PROCESS_TOAST",
+ "INDEX_CLEANUP", "PROCESS_MAIN", "PROCESS_TOAST",
"TRUNCATE", "PARALLEL", "SKIP_DATABASE_STATS",
"ONLY_DATABASE_STATS");
- else if (TailMatches("FULL|FREEZE|ANALYZE|VERBOSE|DISABLE_PAGE_SKIPPING|SKIP_LOCKED|PROCESS_TOAST|TRUNCATE|SKIP_DATABASE_STATS|ONLY_DATABASE_STATS"))
+ else if (TailMatches("FULL|FREEZE|ANALYZE|VERBOSE|DISABLE_PAGE_SKIPPING|SKIP_LOCKED|PROCESS_MAIN|PROCESS_TOAST|TRUNCATE|SKIP_DATABASE_STATS|ONLY_DATABASE_STATS"))
COMPLETE_WITH("ON", "OFF");
else if (TailMatches("INDEX_CLEANUP"))
COMPLETE_WITH("AUTO", "ON", "OFF");
diff --git a/src/bin/scripts/t/100_vacuumdb.pl b/src/bin/scripts/t/100_vacuumdb.pl
index 3cfbaaec0d..46101899ae 100644
--- a/src/bin/scripts/t/100_vacuumdb.pl
+++ b/src/bin/scripts/t/100_vacuumdb.pl
@@ -65,6 +65,13 @@ $node->issues_sql_like(
$node->command_fails(
[ 'vacuumdb', '--analyze-only', '--no-truncate', 'postgres' ],
'--analyze-only and --no-truncate specified together');
+$node->issues_sql_like(
+ [ 'vacuumdb', '--no-process-main', 'postgres' ],
+ qr/statement: VACUUM \(PROCESS_MAIN FALSE, SKIP_DATABASE_STATS\).*;/,
+ 'vacuumdb --no-process-main');
+$node->command_fails(
+ [ 'vacuumdb', '--analyze-only', '--no-process-main', 'postgres' ],
+ '--analyze-only and --no-process_main specified together');
$node->issues_sql_like(
[ 'vacuumdb', '--no-process-toast', 'postgres' ],
qr/statement: VACUUM \(PROCESS_TOAST FALSE, SKIP_DATABASE_STATS\).*;/,
diff --git a/src/bin/scripts/vacuumdb.c b/src/bin/scripts/vacuumdb.c
index 58b894216b..39be265b5b 100644
--- a/src/bin/scripts/vacuumdb.c
+++ b/src/bin/scripts/vacuumdb.c
@@ -43,6 +43,7 @@ typedef struct vacuumingOptions
bool no_index_cleanup;
bool force_index_cleanup;
bool do_truncate;
+ bool process_main;
bool process_toast;
bool skip_database_stats;
} vacuumingOptions;
@@ -121,6 +122,7 @@ main(int argc, char *argv[])
{"force-index-cleanup", no_argument, NULL, 9},
{"no-truncate", no_argument, NULL, 10},
{"no-process-toast", no_argument, NULL, 11},
+ {"no-process-main", no_argument, NULL, 12},
{NULL, 0, NULL, 0}
};
@@ -148,6 +150,7 @@ main(int argc, char *argv[])
vacopts.no_index_cleanup = false;
vacopts.force_index_cleanup = false;
vacopts.do_truncate = true;
+ vacopts.process_main = true;
vacopts.process_toast = true;
pg_logging_init(argv[0]);
@@ -260,6 +263,9 @@ main(int argc, char *argv[])
case 11:
vacopts.process_toast = false;
break;
+ case 12:
+ vacopts.process_main = false;
+ break;
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -312,6 +318,9 @@ main(int argc, char *argv[])
if (!vacopts.do_truncate)
pg_fatal("cannot use the \"%s\" option when performing only analyze",
"no-truncate");
+ if (!vacopts.process_main)
+ pg_fatal("cannot use the \"%s\" option when performing only analyze",
+ "no-process-main");
if (!vacopts.process_toast)
pg_fatal("cannot use the \"%s\" option when performing only analyze",
"no-process-toast");
@@ -508,6 +517,13 @@ vacuum_one_database(ConnParams *cparams,
"no-truncate", "12");
}
+ if (!vacopts->process_main && PQserverVersion(conn) < 160000)
+ {
+ PQfinish(conn);
+ pg_fatal("cannot use the \"%s\" option on server versions older than PostgreSQL %s",
+ "no-process-main", "16");
+ }
+
if (!vacopts->process_toast && PQserverVersion(conn) < 140000)
{
PQfinish(conn);
@@ -976,6 +992,13 @@ prepare_vacuum_command(PQExpBuffer sql, int serverVersion,
appendPQExpBuffer(sql, "%sTRUNCATE FALSE", sep);
sep = comma;
}
+ if (!vacopts->process_main)
+ {
+ /* PROCESS_MAIN is supported since v16 */
+ Assert(serverVersion >= 160000);
+ appendPQExpBuffer(sql, "%sPROCESS_MAIN FALSE", sep);
+ sep = comma;
+ }
if (!vacopts->process_toast)
{
/* PROCESS_TOAST is supported since v14 */
@@ -1090,6 +1113,7 @@ help(const char *progname)
printf(_(" --min-mxid-age=MXID_AGE minimum multixact ID age of tables to vacuum\n"));
printf(_(" --min-xid-age=XID_AGE minimum transaction ID age of tables to vacuum\n"));
printf(_(" --no-index-cleanup don't remove index entries that point to dead tuples\n"));
+ printf(_(" --no-process-main skip the main relation\n"));
printf(_(" --no-process-toast skip the TOAST table associated with the table to vacuum\n"));
printf(_(" --no-truncate don't truncate empty pages at the end of the table\n"));
printf(_(" -n, --schema=PATTERN vacuum tables in the specified schema(s) only\n"));
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 689dbb7702..bdfd96cfec 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -186,10 +186,11 @@ typedef struct VacAttrStats
#define VACOPT_FREEZE 0x08 /* FREEZE option */
#define VACOPT_FULL 0x10 /* FULL (non-concurrent) vacuum */
#define VACOPT_SKIP_LOCKED 0x20 /* skip if cannot get lock */
-#define VACOPT_PROCESS_TOAST 0x40 /* process the TOAST table, if any */
-#define VACOPT_DISABLE_PAGE_SKIPPING 0x80 /* don't skip any pages */
-#define VACOPT_SKIP_DATABASE_STATS 0x100 /* skip vac_update_datfrozenxid() */
-#define VACOPT_ONLY_DATABASE_STATS 0x200 /* only vac_update_datfrozenxid() */
+#define VACOPT_PROCESS_MAIN 0x40 /* process main relation */
+#define VACOPT_PROCESS_TOAST 0x80 /* process the TOAST table, if any */
+#define VACOPT_DISABLE_PAGE_SKIPPING 0x100 /* don't skip any pages */
+#define VACOPT_SKIP_DATABASE_STATS 0x200 /* skip vac_update_datfrozenxid() */
+#define VACOPT_ONLY_DATABASE_STATS 0x400 /* only vac_update_datfrozenxid() */
/*
* Values used by index_cleanup and truncate params.
diff --git a/src/test/regress/expected/vacuum.out b/src/test/regress/expected/vacuum.out
index 458adee7f8..17fba0fe06 100644
--- a/src/test/regress/expected/vacuum.out
+++ b/src/test/regress/expected/vacuum.out
@@ -276,10 +276,29 @@ RESET default_transaction_isolation;
BEGIN TRANSACTION ISOLATION LEVEL SERIALIZABLE;
ANALYZE vactst;
COMMIT;
+CREATE VIEW vactst_vacuum_counts AS
+ SELECT left(s.relname, 8), s.vacuum_count
+ FROM pg_stat_all_tables s
+ LEFT JOIN pg_class c ON s.relid = c.reltoastrelid
+ WHERE c.relname = 'vactst' OR s.relname = 'vactst'
+ ORDER BY s.relname;
-- PROCESS_TOAST option
ALTER TABLE vactst ADD COLUMN t TEXT;
ALTER TABLE vactst ALTER COLUMN t SET STORAGE EXTERNAL;
+SELECT pg_stat_reset();
+ pg_stat_reset
+---------------
+
+(1 row)
+
VACUUM (PROCESS_TOAST FALSE) vactst;
+SELECT * FROM vactst_vacuum_counts;
+ left | vacuum_count
+----------+--------------
+ pg_toast | 0
+ vactst | 1
+(2 rows)
+
VACUUM (PROCESS_TOAST FALSE, FULL) vactst;
ERROR: PROCESS_TOAST required with VACUUM FULL
-- SKIP_DATABASE_STATS option
@@ -288,6 +307,47 @@ VACUUM (SKIP_DATABASE_STATS) vactst;
VACUUM (ONLY_DATABASE_STATS);
VACUUM (ONLY_DATABASE_STATS) vactst; -- error
ERROR: ONLY_DATABASE_STATS cannot be specified with a list of tables
+-- PROCESS_MAIN option
+SELECT pg_stat_reset();
+ pg_stat_reset
+---------------
+
+(1 row)
+
+VACUUM (PROCESS_MAIN FALSE) vactst;
+SELECT * FROM vactst_vacuum_counts;
+ left | vacuum_count
+----------+--------------
+ pg_toast | 1
+ vactst | 0
+(2 rows)
+
+VACUUM (PROCESS_MAIN FALSE, PROCESS_TOAST FALSE) vactst;
+SELECT * FROM vactst_vacuum_counts;
+ left | vacuum_count
+----------+--------------
+ pg_toast | 1
+ vactst | 0
+(2 rows)
+
+SELECT relfilenode AS main_filenode FROM pg_class WHERE relname = 'vactst' \gset
+SELECT t.relfilenode AS toast_filenode FROM pg_class c, pg_class t
+ WHERE c.reltoastrelid = t.oid AND c.relname = 'vactst' \gset
+VACUUM (PROCESS_MAIN FALSE, FULL) vactst;
+SELECT relfilenode = :main_filenode FROM pg_class WHERE relname = 'vactst';
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT t.relfilenode = :toast_filenode FROM pg_class c, pg_class t
+ WHERE c.reltoastrelid = t.oid AND c.relname = 'vactst';
+ ?column?
+----------
+ f
+(1 row)
+
+DROP VIEW vactst_vacuum_counts;
DROP TABLE vaccluster;
DROP TABLE vactst;
DROP TABLE vacparted;
diff --git a/src/test/regress/sql/vacuum.sql b/src/test/regress/sql/vacuum.sql
index 9da8f3e830..2985c1a99b 100644
--- a/src/test/regress/sql/vacuum.sql
+++ b/src/test/regress/sql/vacuum.sql
@@ -231,10 +231,19 @@ BEGIN TRANSACTION ISOLATION LEVEL SERIALIZABLE;
ANALYZE vactst;
COMMIT;
+CREATE VIEW vactst_vacuum_counts AS
+ SELECT left(s.relname, 8), s.vacuum_count
+ FROM pg_stat_all_tables s
+ LEFT JOIN pg_class c ON s.relid = c.reltoastrelid
+ WHERE c.relname = 'vactst' OR s.relname = 'vactst'
+ ORDER BY s.relname;
+
-- PROCESS_TOAST option
ALTER TABLE vactst ADD COLUMN t TEXT;
ALTER TABLE vactst ALTER COLUMN t SET STORAGE EXTERNAL;
+SELECT pg_stat_reset();
VACUUM (PROCESS_TOAST FALSE) vactst;
+SELECT * FROM vactst_vacuum_counts;
VACUUM (PROCESS_TOAST FALSE, FULL) vactst;
-- SKIP_DATABASE_STATS option
@@ -244,6 +253,21 @@ VACUUM (SKIP_DATABASE_STATS) vactst;
VACUUM (ONLY_DATABASE_STATS);
VACUUM (ONLY_DATABASE_STATS) vactst; -- error
+-- PROCESS_MAIN option
+SELECT pg_stat_reset();
+VACUUM (PROCESS_MAIN FALSE) vactst;
+SELECT * FROM vactst_vacuum_counts;
+VACUUM (PROCESS_MAIN FALSE, PROCESS_TOAST FALSE) vactst;
+SELECT * FROM vactst_vacuum_counts;
+SELECT relfilenode AS main_filenode FROM pg_class WHERE relname = 'vactst' \gset
+SELECT t.relfilenode AS toast_filenode FROM pg_class c, pg_class t
+ WHERE c.reltoastrelid = t.oid AND c.relname = 'vactst' \gset
+VACUUM (PROCESS_MAIN FALSE, FULL) vactst;
+SELECT relfilenode = :main_filenode FROM pg_class WHERE relname = 'vactst';
+SELECT t.relfilenode = :toast_filenode FROM pg_class c, pg_class t
+ WHERE c.reltoastrelid = t.oid AND c.relname = 'vactst';
+
+DROP VIEW vactst_vacuum_counts;
DROP TABLE vaccluster;
DROP TABLE vactst;
DROP TABLE vacparted;
--
2.25.1
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: add PROCESS_MAIN to VACUUM
@ 2023-03-02 03:58 Masahiko Sawada <[email protected]>
parent: Nathan Bossart <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Masahiko Sawada @ 2023-03-02 03:58 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Michael Paquier <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
On Thu, Mar 2, 2023 at 4:13 AM Nathan Bossart <[email protected]> wrote:
>
> On Wed, Mar 01, 2023 at 07:09:53PM +0100, Alvaro Herrera wrote:
> > On 2023-Mar-01, Michael Paquier wrote:
> >
> >> +-- PROCESS_MAIN option
> >> +VACUUM (PROCESS_MAIN FALSE) vactst;
> >> +VACUUM (PROCESS_MAIN FALSE, PROCESS_TOAST FALSE) vactst;
> >> +VACUUM (PROCESS_MAIN FALSE, FULL) vactst;
> >>
> >> Thinking a bit here. This set of tests does not make sure that the
> >> main relation and/or the toast relation have been actually processed.
> >> pg_stat_user_tables does not track what's happening on the toast
> >> relations. So... What about adding some tests in 100_vacuumdb.pl
> >> that rely on vacuumdb --verbose and check the logs produced? We
> >> should make sure that the toast or the main relation are processed,
> >> by tracking, for example, logs like vacuuming "schema.table". When
> >> FULL is involved, we may want to track the changes on relfilenodes
> >> depending on what's wanted.
> >
> > Maybe instead of reading the log, read values from pg_stat_all_tables.
>
> Here is an attempt at that. Thanks for the idea.
I've reviewed the v4 patch. Here is a minor comment:
+SELECT * FROM vactst_vacuum_counts;
+ left | vacuum_count
+----------+--------------
+ pg_toast | 1
+ vactst | 0
+(2 rows)
+CREATE VIEW vactst_vacuum_counts AS
+ SELECT left(s.relname, 8), s.vacuum_count
+ FROM pg_stat_all_tables s
+ LEFT JOIN pg_class c ON s.relid = c.reltoastrelid
+ WHERE c.relname = 'vactst' OR s.relname = 'vactst'
+ ORDER BY s.relname;
Cutting the toast relation name to 'pg_toast' is a bit confusing to me
as we have the pg_toast schema. How about using the following query
instead to improve the readability?
SELECT
CASE WHEN c.relname IS NULL THEN
s.relname
ELSE
'toast for ' || c.relname
END as relname,
s.vacuum_count
FROM pg_stat_all_tables s
LEFT JOIN pg_class c ON s.relid = c.reltoastrelid
WHERE c.relname = 'vactst' OR s.relname = 'vactst'
We will get like:
SELECT * FROM vactst_vacuum_counts;
relname | vacuum_count
------------------+--------------
toast for vactst | 0
vactst | 1
(2 rows)
The rest looks good to me.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: add PROCESS_MAIN to VACUUM
@ 2023-03-02 05:21 Michael Paquier <[email protected]>
parent: Masahiko Sawada <[email protected]>
0 siblings, 0 replies; 36+ messages in thread
From: Michael Paquier @ 2023-03-02 05:21 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
On Thu, Mar 02, 2023 at 12:58:32PM +0900, Masahiko Sawada wrote:
> Cutting the toast relation name to 'pg_toast' is a bit confusing to me
> as we have the pg_toast schema. How about using the following query
> instead to improve the readability?
>
> SELECT
> CASE WHEN c.relname IS NULL THEN
> s.relname
> ELSE
> 'toast for ' || c.relname
> END as relname,
> s.vacuum_count
> FROM pg_stat_all_tables s
> LEFT JOIN pg_class c ON s.relid = c.reltoastrelid
> WHERE c.relname = 'vactst' OR s.relname = 'vactst'
Another tweak that I have learnt to like is to apply a filter with
regexp_replace(), see 090_reindexdb.pl:
regexp_replace(b.indname::text, '(pg_toast.pg_toast_)\\d+(_index)', '\\1<oid>\\2')
If you make that part of the view definition, the result is the same,
so that's up to which solution one prefers.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 36+ messages in thread
end of thread, other threads:[~2023-03-02 05:21 UTC | newest]
Thread overview: 36+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2017-09-15 00:03 Trouble with amcheck Douglas Doole <[email protected]>
2017-09-15 00:11 ` Peter Geoghegan <[email protected]>
2017-09-15 01:31 ` Stephen Frost <[email protected]>
2017-09-15 01:35 ` Michael Paquier <[email protected]>
2017-09-15 02:22 ` Douglas Doole <[email protected]>
2017-09-15 02:29 ` Andres Freund <[email protected]>
2017-09-15 02:36 ` Stephen Frost <[email protected]>
2017-09-15 02:38 ` Andres Freund <[email protected]>
2017-09-15 02:43 ` Tom Lane <[email protected]>
2018-11-08 17:30 security release Pavel Stehule <[email protected]>
2018-11-08 17:39 ` Re: security release Robert Haas <[email protected]>
2018-11-08 17:54 ` Re: security release Pavel Stehule <[email protected]>
2021-01-20 14:59 [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping. Kyotaro Horiguchi <[email protected]>
2021-01-20 14:59 [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping. Kyotaro Horiguchi <[email protected]>
2021-01-20 14:59 [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping. Kyotaro Horiguchi <[email protected]>
2021-01-20 14:59 [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping. Kyotaro Horiguchi <[email protected]>
2021-01-20 14:59 [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping. Kyotaro Horiguchi <[email protected]>
2021-01-20 14:59 [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping. Kyotaro Horiguchi <[email protected]>
2021-01-20 14:59 [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping. Kyotaro Horiguchi <[email protected]>
2021-01-20 14:59 [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping. Kyotaro Horiguchi <[email protected]>
2021-01-20 14:59 [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping. Kyotaro Horiguchi <[email protected]>
2021-01-20 14:59 [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping. Kyotaro Horiguchi <[email protected]>
2021-01-20 14:59 [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping. Kyotaro Horiguchi <[email protected]>
2021-01-20 14:59 [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping. Kyotaro Horiguchi <[email protected]>
2021-01-20 14:59 [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping. Kyotaro Horiguchi <[email protected]>
2021-01-20 14:59 [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping. Kyotaro Horiguchi <[email protected]>
2021-01-20 14:59 [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping. Kyotaro Horiguchi <[email protected]>
2021-01-20 14:59 [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping. Kyotaro Horiguchi <[email protected]>
2021-01-20 14:59 [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping. Kyotaro Horiguchi <[email protected]>
2021-01-20 14:59 [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping. Kyotaro Horiguchi <[email protected]>
2021-01-20 14:59 [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping. Kyotaro Horiguchi <[email protected]>
2021-01-20 14:59 [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping. Kyotaro Horiguchi <[email protected]>
2021-01-20 14:59 [PATCH v5 3/3] Poc: Keep page-LSN updated while WAL-skipping. Kyotaro Horiguchi <[email protected]>
2023-03-01 19:13 Re: add PROCESS_MAIN to VACUUM Nathan Bossart <[email protected]>
2023-03-02 03:58 ` Re: add PROCESS_MAIN to VACUUM Masahiko Sawada <[email protected]>
2023-03-02 05:21 ` Re: add PROCESS_MAIN to VACUUM Michael Paquier <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox