agora inbox for [email protected]
help / color / mirror / Atom feedLogical replication - TRAP: FailedAssertion in pgstat.c
53+ messages / 7 participants
[nested] [flat]
* Logical replication - TRAP: FailedAssertion in pgstat.c
@ 2017-04-15 02:47 Erik Rijkers <[email protected]>
0 siblings, 1 reply; 53+ messages in thread
From: Erik Rijkers @ 2017-04-15 02:47 UTC (permalink / raw)
To: pgsql-hackers
Testing logical replication, with the following patches on top of
yesterday's master:
0001-Reserve-global-xmin-for-create-slot-snasphot-export.patch +
0002-Don-t-use-on-disk-snapshots-for-snapshot-export-in-l.patch+
0003-Prevent-snapshot-builder-xmin-from-going-backwards.patch +
0004-Fix-xl_running_xacts-usage-in-snapshot-builder.patch +
0005-Skip-unnecessary-snapshot-builds.patch
Is applying that patch set is still correct?
It builds fine, but when I run the old pbench-over-logical-replication
test I get:
TRAP: FailedAssertion("!(entry->trans == ((void *)0))", File:
"pgstat.c", Line: 828)
reliably (often within a minute).
The test itself does not fail, at least not that I saw (but I only ran a
few).
thanks,
Erik Rijkers
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 53+ messages in thread
* Re: Logical replication - TRAP: FailedAssertion in pgstat.c
@ 2017-04-16 08:46 Erik Rijkers <[email protected]>
parent: Erik Rijkers <[email protected]>
0 siblings, 1 reply; 53+ messages in thread
From: Erik Rijkers @ 2017-04-16 08:46 UTC (permalink / raw)
To: pgsql-hackers; +Cc: [email protected]
On 2017-04-15 04:47, Erik Rijkers wrote:
>
> 0001-Reserve-global-xmin-for-create-slot-snasphot-export.patch +
> 0002-Don-t-use-on-disk-snapshots-for-snapshot-export-in-l.patch+
> 0003-Prevent-snapshot-builder-xmin-from-going-backwards.patch +
> 0004-Fix-xl_running_xacts-usage-in-snapshot-builder.patch +
> 0005-Skip-unnecessary-snapshot-builds.patch
I am now using these newer patches:
https://www.postgresql.org/message-id/30242bc6-eca4-b7bb-670e-8d0458753a8c%402ndquadrant.com
> It builds fine, but when I run the old pbench-over-logical-replication
> test I get:
>
> TRAP: FailedAssertion("!(entry->trans == ((void *)0))", File:
> "pgstat.c", Line: 828)
To get that error:
--------------
#!/bin/sh
port1=6972 port2=6973 scale=25 clients=16 duration=60
echo "drop table if exists pgbench_accounts;
drop table if exists pgbench_branches;
drop table if exists pgbench_tellers;
drop table if exists pgbench_history;" | psql -qXp $port1 \
&& echo "drop table if exists pgbench_accounts;
drop table if exists pgbench_branches;
drop table if exists pgbench_tellers;
drop table if exists pgbench_history;" | psql -qXp $port2 \
&& pgbench -p $port1 -qis ${scale//_/} && echo "
alter table pgbench_history add column hid serial primary key;
" | psql -q1Xp $port1 \
&& pg_dump -F c -p $port1 \
--exclude-table-data=pgbench_history \
--exclude-table-data=pgbench_accounts \
--exclude-table-data=pgbench_branches \
--exclude-table-data=pgbench_tellers \
-t pgbench_history \
-t pgbench_accounts \
-t pgbench_branches \
-t pgbench_tellers \
| pg_restore -1 -p $port2 -d testdb
appname=pgbench_derail
echo "create publication pub1 for all tables;" | psql -p $port1 -aqtAX
echo "create subscription sub1 connection 'port=${port1}
application_name=${appname}' publication pub1 with (disabled);
alter subscription sub1 enable;
" | psql -p $port2 -aqtAX
echo "-- pgbench -p $port1 -c $clients -T $duration -n -- scale $scale
"
pgbench -p $port1 -c $clients -T $duration -n
--------------
Erik Rijkers
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 53+ messages in thread
* Re: Logical replication - TRAP: FailedAssertion in pgstat.c
@ 2017-04-16 18:41 Andres Freund <[email protected]>
parent: Erik Rijkers <[email protected]>
0 siblings, 2 replies; 53+ messages in thread
From: Andres Freund @ 2017-04-16 18:41 UTC (permalink / raw)
To: Erik Rijkers <[email protected]>; Petr Jelinek <[email protected]>; Stas Kelvich <[email protected]>; Peter Eisentraut <[email protected]>; +Cc: pgsql-hackers; [email protected]
On 2017-04-16 10:46:21 +0200, Erik Rijkers wrote:
> On 2017-04-15 04:47, Erik Rijkers wrote:
> >
> > 0001-Reserve-global-xmin-for-create-slot-snasphot-export.patch +
> > 0002-Don-t-use-on-disk-snapshots-for-snapshot-export-in-l.patch+
> > 0003-Prevent-snapshot-builder-xmin-from-going-backwards.patch +
> > 0004-Fix-xl_running_xacts-usage-in-snapshot-builder.patch +
> > 0005-Skip-unnecessary-snapshot-builds.patch
>
> I am now using these newer patches:
> https://www.postgresql.org/message-id/30242bc6-eca4-b7bb-670e-8d0458753a8c%402ndquadrant.com
>
> > It builds fine, but when I run the old pbench-over-logical-replication
> > test I get:
> >
> > TRAP: FailedAssertion("!(entry->trans == ((void *)0))", File:
> > "pgstat.c", Line: 828)
>
>
> To get that error:
I presume this is the fault of
http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=139eb9673cb84c76f493af7e68301ae2041997...
if you git revert that individual commit, do things work again?
- Andres
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 53+ messages in thread
* Re: Logical replication - TRAP: FailedAssertion in pgstat.c
@ 2017-04-16 20:11 Petr Jelinek <[email protected]>
parent: Andres Freund <[email protected]>
1 sibling, 1 reply; 53+ messages in thread
From: Petr Jelinek @ 2017-04-16 20:11 UTC (permalink / raw)
To: Andres Freund <[email protected]>; Erik Rijkers <[email protected]>; Stas Kelvich <[email protected]>; Peter Eisentraut <[email protected]>; +Cc: pgsql-hackers; [email protected]
On 16/04/17 20:41, Andres Freund wrote:
> On 2017-04-16 10:46:21 +0200, Erik Rijkers wrote:
>> On 2017-04-15 04:47, Erik Rijkers wrote:
>>>
>>> 0001-Reserve-global-xmin-for-create-slot-snasphot-export.patch +
>>> 0002-Don-t-use-on-disk-snapshots-for-snapshot-export-in-l.patch+
>>> 0003-Prevent-snapshot-builder-xmin-from-going-backwards.patch +
>>> 0004-Fix-xl_running_xacts-usage-in-snapshot-builder.patch +
>>> 0005-Skip-unnecessary-snapshot-builds.patch
>>
>> I am now using these newer patches:
>> https://www.postgresql.org/message-id/30242bc6-eca4-b7bb-670e-8d0458753a8c%402ndquadrant.com
>>
>>> It builds fine, but when I run the old pbench-over-logical-replication
>>> test I get:
>>>
>>> TRAP: FailedAssertion("!(entry->trans == ((void *)0))", File:
>>> "pgstat.c", Line: 828)
>>
>>
>> To get that error:
>
> I presume this is the fault of
> http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=139eb9673cb84c76f493af7e68301ae2041997...
> if you git revert that individual commit, do things work again?
>
> - Andres
>
Yeah it is, it needs to be fenced to happen only after commit, which is
not guaranteed at the point of code, we probably need to put the
pgstat_report_stat() inside the if above after the
CommitTransactionCommand() (that will make it report stats for changes
apply did to pg_subscription_rel after next transaction though) or put
it into it's own if that checks if tx was indeed committed (which is bit
ugly).
--
Petr Jelinek http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 53+ messages in thread
* Re: Logical replication - TRAP: FailedAssertion in pgstat.c
@ 2017-04-17 07:30 Erik Rijkers <[email protected]>
parent: Andres Freund <[email protected]>
1 sibling, 1 reply; 53+ messages in thread
From: Erik Rijkers @ 2017-04-17 07:30 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Petr Jelinek <[email protected]>; Stas Kelvich <[email protected]>; Peter Eisentraut <[email protected]>; pgsql-hackers; [email protected]
On 2017-04-16 20:41, Andres Freund wrote:
> On 2017-04-16 10:46:21 +0200, Erik Rijkers wrote:
>> On 2017-04-15 04:47, Erik Rijkers wrote:
>> >
>> > 0001-Reserve-global-xmin-for-create-slot-snasphot-export.patch +
>> > 0002-Don-t-use-on-disk-snapshots-for-snapshot-export-in-l.patch+
>> > 0003-Prevent-snapshot-builder-xmin-from-going-backwards.patch +
>> > 0004-Fix-xl_running_xacts-usage-in-snapshot-builder.patch +
>> > 0005-Skip-unnecessary-snapshot-builds.patch
>>
>> I am now using these newer patches:
>> https://www.postgresql.org/message-id/30242bc6-eca4-b7bb-670e-8d0458753a8c%402ndquadrant.com
>>
>> > It builds fine, but when I run the old pbench-over-logical-replication
>> > test I get:
>> >
>> > TRAP: FailedAssertion("!(entry->trans == ((void *)0))", File:
>> > "pgstat.c", Line: 828)
>>
>>
>> To get that error:
>
> I presume this is the fault of
> http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=139eb9673cb84c76f493af7e68301ae2041997...
> if you git revert that individual commit, do things work again?
>
Yes, compiled from 67c2def11d4 with the above 4 patches, it runs
flawlessly again. (flawlessly= a few hours without any error)
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 53+ messages in thread
* Re: Logical replication - TRAP: FailedAssertion in pgstat.c
@ 2017-04-17 13:59 Stas Kelvich <[email protected]>
parent: Erik Rijkers <[email protected]>
0 siblings, 1 reply; 53+ messages in thread
From: Stas Kelvich @ 2017-04-17 13:59 UTC (permalink / raw)
To: Erik Rijkers <[email protected]>; +Cc: Andres Freund <[email protected]>; Petr Jelinek <[email protected]>; Peter Eisentraut <[email protected]>; pgsql-hackers; [email protected]
> On 17 Apr 2017, at 10:30, Erik Rijkers <[email protected]> wrote:
>
> On 2017-04-16 20:41, Andres Freund wrote:
>> On 2017-04-16 10:46:21 +0200, Erik Rijkers wrote:
>>> On 2017-04-15 04:47, Erik Rijkers wrote:
>>> >
>>> > 0001-Reserve-global-xmin-for-create-slot-snasphot-export.patch +
>>> > 0002-Don-t-use-on-disk-snapshots-for-snapshot-export-in-l.patch+
>>> > 0003-Prevent-snapshot-builder-xmin-from-going-backwards.patch +
>>> > 0004-Fix-xl_running_xacts-usage-in-snapshot-builder.patch +
>>> > 0005-Skip-unnecessary-snapshot-builds.patch
>>> I am now using these newer patches:
>>> https://www.postgresql.org/message-id/30242bc6-eca4-b7bb-670e-8d0458753a8c%402ndquadrant.com
>>> > It builds fine, but when I run the old pbench-over-logical-replication
>>> > test I get:
>>> >
>>> > TRAP: FailedAssertion("!(entry->trans == ((void *)0))", File:
>>> > "pgstat.c", Line: 828)
>>> To get that error:
>> I presume this is the fault of
>> http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=139eb9673cb84c76f493af7e68301ae2041997...
>> if you git revert that individual commit, do things work again?
>
> Yes, compiled from 67c2def11d4 with the above 4 patches, it runs flawlessly again. (flawlessly= a few hours without any error)
>
I’ve reproduced failure, this happens under tablesync worker and putting
pgstat_report_stat() under the previous condition block should help.
However for me it took about an hour of running this script to catch original assert.
Can you check with that patch applied?
Stas Kelvich
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[application/octet-stream] logical_worker.patch (704B, ../../[email protected]/2-logical_worker.patch)
download | inline diff:
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 656d399..7a02c33 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -453,6 +453,7 @@ apply_handle_commit(StringInfo s)
replorigin_session_origin_timestamp = commit_data.committime;
CommitTransactionCommand();
+ pgstat_report_stat(false);
store_flush_position(commit_data.end_lsn);
}
@@ -462,7 +463,6 @@ apply_handle_commit(StringInfo s)
/* Process any tables that are being synchronized in parallel. */
process_syncing_tables(commit_data.end_lsn);
- pgstat_report_stat(false);
pgstat_report_activity(STATE_IDLE, NULL);
}
^ permalink raw reply [nested|flat] 53+ messages in thread
* Re: Logical replication - TRAP: FailedAssertion in pgstat.c
@ 2017-04-17 19:16 Erik Rijkers <[email protected]>
parent: Stas Kelvich <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Erik Rijkers @ 2017-04-17 19:16 UTC (permalink / raw)
To: Stas Kelvich <[email protected]>; +Cc: Andres Freund <[email protected]>; Petr Jelinek <[email protected]>; Peter Eisentraut <[email protected]>; pgsql-hackers; [email protected]
On 2017-04-17 15:59, Stas Kelvich wrote:
>> On 17 Apr 2017, at 10:30, Erik Rijkers <[email protected]> wrote:
>>
>> On 2017-04-16 20:41, Andres Freund wrote:
>>> On 2017-04-16 10:46:21 +0200, Erik Rijkers wrote:
>>>> On 2017-04-15 04:47, Erik Rijkers wrote:
>>>> >
>>>> > 0001-Reserve-global-xmin-for-create-slot-snasphot-export.patch +
>>>> > 0002-Don-t-use-on-disk-snapshots-for-snapshot-export-in-l.patch+
>>>> > 0003-Prevent-snapshot-builder-xmin-from-going-backwards.patch +
>>>> > 0004-Fix-xl_running_xacts-usage-in-snapshot-builder.patch +
>>>> > 0005-Skip-unnecessary-snapshot-builds.patch
>>>> I am now using these newer patches:
>>>> https://www.postgresql.org/message-id/30242bc6-eca4-b7bb-670e-8d0458753a8c%402ndquadrant.com
>>>> > It builds fine, but when I run the old pbench-over-logical-replication
>>>> > test I get:
>>>> >
>>>> > TRAP: FailedAssertion("!(entry->trans == ((void *)0))", File:
>>>> > "pgstat.c", Line: 828)
>>>> To get that error:
>>> I presume this is the fault of
>>> http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=139eb9673cb84c76f493af7e68301ae2041997...
>>> if you git revert that individual commit, do things work again?
>>
>> Yes, compiled from 67c2def11d4 with the above 4 patches, it runs
>> flawlessly again. (flawlessly= a few hours without any error)
>>
>
> I’ve reproduced failure, this happens under tablesync worker and
> putting
> pgstat_report_stat() under the previous condition block should help.
>
> However for me it took about an hour of running this script to catch
> original assert.
>
> Can you check with that patch applied?
Your patch on top of the 5 patches above seem to solve the matter too:
no problems after running for 2 hours (previously it failed within half
a minute).
Erik Rijkers
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 53+ messages in thread
* Re: Logical replication - TRAP: FailedAssertion in pgstat.c
@ 2017-04-20 18:58 Peter Eisentraut <[email protected]>
parent: Petr Jelinek <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Peter Eisentraut @ 2017-04-20 18:58 UTC (permalink / raw)
To: Petr Jelinek <[email protected]>; Andres Freund <[email protected]>; Erik Rijkers <[email protected]>; Stas Kelvich <[email protected]>; +Cc: pgsql-hackers; [email protected]
On 4/16/17 16:11, Petr Jelinek wrote:
> Yeah it is, it needs to be fenced to happen only after commit, which is
> not guaranteed at the point of code, we probably need to put the
> pgstat_report_stat() inside the if above after the
> CommitTransactionCommand() (that will make it report stats for changes
> apply did to pg_subscription_rel after next transaction though)
I think to avoid the latter, we should add more pgstat_report_stat()
calls, such as in process_syncing_tables_for_apply(). Basically every
code path that calls CommitTransactionCommand() should have one, no?
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 03:25 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 03:25 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +++++-
src/backend/access/transam/xlog.c | 4 ++++
src/backend/access/transam/xloginsert.c | 26 ++++++++++++++++++++++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 18 ++++++++++++++++
src/include/access/xloginsert.h | 2 ++
src/include/catalog/pg_control.h | 1 +
7 files changed, 57 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..eebca0fc16 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,32 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ *
+ * Callable while holding just share lock on the buffer content.
+ *
+ * The LSN of the inserted wal record is returned if we had to write,
+ * InvalidXLogRecPtr otherwise.
+ *
+ * It is possible that multiple concurrent backends could attempt to write WAL
+ * records. In that case, multiple copies of the same block would be recorded
+ * in separate WAL records by different backends, though that is still OK from
+ * a correctness perspective.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..fd59f9fc1d 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3865,6 +3865,24 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
{
dirtied = true; /* Means "will be dirtied by this action" */
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page.
+ */
+ if (XLogRecPtrIsInvalid(lsn) /* XXX && file_encryption_method != 0 */)
+ lsn = XLogLSNForHint();
+
/*
* Set the page LSN if we wrote a backup block. We aren't supposed
* to set this when only holding a share lock but as long as we
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--ikeVEW9yuYc//A+q--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 03:25 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 03:25 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +++++-
src/backend/access/transam/xlog.c | 4 ++++
src/backend/access/transam/xloginsert.c | 26 ++++++++++++++++++++++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 18 ++++++++++++++++
src/include/access/xloginsert.h | 2 ++
src/include/catalog/pg_control.h | 1 +
7 files changed, 57 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..eebca0fc16 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,32 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ *
+ * Callable while holding just share lock on the buffer content.
+ *
+ * The LSN of the inserted wal record is returned if we had to write,
+ * InvalidXLogRecPtr otherwise.
+ *
+ * It is possible that multiple concurrent backends could attempt to write WAL
+ * records. In that case, multiple copies of the same block would be recorded
+ * in separate WAL records by different backends, though that is still OK from
+ * a correctness perspective.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..fd59f9fc1d 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3865,6 +3865,24 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
{
dirtied = true; /* Means "will be dirtied by this action" */
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page.
+ */
+ if (XLogRecPtrIsInvalid(lsn) /* XXX && file_encryption_method != 0 */)
+ lsn = XLogLSNForHint();
+
/*
* Set the page LSN if we wrote a backup block. We aren't supposed
* to set this when only holding a share lock but as long as we
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--ikeVEW9yuYc//A+q--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 03:25 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 03:25 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +++++-
src/backend/access/transam/xlog.c | 4 ++++
src/backend/access/transam/xloginsert.c | 26 ++++++++++++++++++++++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 18 ++++++++++++++++
src/include/access/xloginsert.h | 2 ++
src/include/catalog/pg_control.h | 1 +
7 files changed, 57 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..eebca0fc16 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,32 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ *
+ * Callable while holding just share lock on the buffer content.
+ *
+ * The LSN of the inserted wal record is returned if we had to write,
+ * InvalidXLogRecPtr otherwise.
+ *
+ * It is possible that multiple concurrent backends could attempt to write WAL
+ * records. In that case, multiple copies of the same block would be recorded
+ * in separate WAL records by different backends, though that is still OK from
+ * a correctness perspective.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..fd59f9fc1d 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3865,6 +3865,24 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
{
dirtied = true; /* Means "will be dirtied by this action" */
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page.
+ */
+ if (XLogRecPtrIsInvalid(lsn) /* XXX && file_encryption_method != 0 */)
+ lsn = XLogLSNForHint();
+
/*
* Set the page LSN if we wrote a backup block. We aren't supposed
* to set this when only holding a share lock but as long as we
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--ikeVEW9yuYc//A+q--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 03:25 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 03:25 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +++++-
src/backend/access/transam/xlog.c | 4 ++++
src/backend/access/transam/xloginsert.c | 26 ++++++++++++++++++++++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 18 ++++++++++++++++
src/include/access/xloginsert.h | 2 ++
src/include/catalog/pg_control.h | 1 +
7 files changed, 57 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..eebca0fc16 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,32 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ *
+ * Callable while holding just share lock on the buffer content.
+ *
+ * The LSN of the inserted wal record is returned if we had to write,
+ * InvalidXLogRecPtr otherwise.
+ *
+ * It is possible that multiple concurrent backends could attempt to write WAL
+ * records. In that case, multiple copies of the same block would be recorded
+ * in separate WAL records by different backends, though that is still OK from
+ * a correctness perspective.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..fd59f9fc1d 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3865,6 +3865,24 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
{
dirtied = true; /* Means "will be dirtied by this action" */
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page.
+ */
+ if (XLogRecPtrIsInvalid(lsn) /* XXX && file_encryption_method != 0 */)
+ lsn = XLogLSNForHint();
+
/*
* Set the page LSN if we wrote a backup block. We aren't supposed
* to set this when only holding a share lock but as long as we
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--ikeVEW9yuYc//A+q--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 03:25 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 03:25 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +++++-
src/backend/access/transam/xlog.c | 4 ++++
src/backend/access/transam/xloginsert.c | 26 ++++++++++++++++++++++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 18 ++++++++++++++++
src/include/access/xloginsert.h | 2 ++
src/include/catalog/pg_control.h | 1 +
7 files changed, 57 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..eebca0fc16 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,32 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ *
+ * Callable while holding just share lock on the buffer content.
+ *
+ * The LSN of the inserted wal record is returned if we had to write,
+ * InvalidXLogRecPtr otherwise.
+ *
+ * It is possible that multiple concurrent backends could attempt to write WAL
+ * records. In that case, multiple copies of the same block would be recorded
+ * in separate WAL records by different backends, though that is still OK from
+ * a correctness perspective.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..fd59f9fc1d 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3865,6 +3865,24 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
{
dirtied = true; /* Means "will be dirtied by this action" */
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page.
+ */
+ if (XLogRecPtrIsInvalid(lsn) /* XXX && file_encryption_method != 0 */)
+ lsn = XLogLSNForHint();
+
/*
* Set the page LSN if we wrote a backup block. We aren't supposed
* to set this when only holding a share lock but as long as we
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--ikeVEW9yuYc//A+q--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 03:25 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 03:25 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +++++-
src/backend/access/transam/xlog.c | 4 ++++
src/backend/access/transam/xloginsert.c | 26 ++++++++++++++++++++++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 18 ++++++++++++++++
src/include/access/xloginsert.h | 2 ++
src/include/catalog/pg_control.h | 1 +
7 files changed, 57 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..eebca0fc16 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,32 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ *
+ * Callable while holding just share lock on the buffer content.
+ *
+ * The LSN of the inserted wal record is returned if we had to write,
+ * InvalidXLogRecPtr otherwise.
+ *
+ * It is possible that multiple concurrent backends could attempt to write WAL
+ * records. In that case, multiple copies of the same block would be recorded
+ * in separate WAL records by different backends, though that is still OK from
+ * a correctness perspective.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..fd59f9fc1d 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3865,6 +3865,24 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
{
dirtied = true; /* Means "will be dirtied by this action" */
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page.
+ */
+ if (XLogRecPtrIsInvalid(lsn) /* XXX && file_encryption_method != 0 */)
+ lsn = XLogLSNForHint();
+
/*
* Set the page LSN if we wrote a backup block. We aren't supposed
* to set this when only holding a share lock but as long as we
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--ikeVEW9yuYc//A+q--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 03:25 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 03:25 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +++++-
src/backend/access/transam/xlog.c | 4 ++++
src/backend/access/transam/xloginsert.c | 26 ++++++++++++++++++++++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 18 ++++++++++++++++
src/include/access/xloginsert.h | 2 ++
src/include/catalog/pg_control.h | 1 +
7 files changed, 57 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..eebca0fc16 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,32 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ *
+ * Callable while holding just share lock on the buffer content.
+ *
+ * The LSN of the inserted wal record is returned if we had to write,
+ * InvalidXLogRecPtr otherwise.
+ *
+ * It is possible that multiple concurrent backends could attempt to write WAL
+ * records. In that case, multiple copies of the same block would be recorded
+ * in separate WAL records by different backends, though that is still OK from
+ * a correctness perspective.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..fd59f9fc1d 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3865,6 +3865,24 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
{
dirtied = true; /* Means "will be dirtied by this action" */
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page.
+ */
+ if (XLogRecPtrIsInvalid(lsn) /* XXX && file_encryption_method != 0 */)
+ lsn = XLogLSNForHint();
+
/*
* Set the page LSN if we wrote a backup block. We aren't supposed
* to set this when only holding a share lock but as long as we
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--ikeVEW9yuYc//A+q--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 03:25 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 03:25 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +++++-
src/backend/access/transam/xlog.c | 4 ++++
src/backend/access/transam/xloginsert.c | 26 ++++++++++++++++++++++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 18 ++++++++++++++++
src/include/access/xloginsert.h | 2 ++
src/include/catalog/pg_control.h | 1 +
7 files changed, 57 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..eebca0fc16 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,32 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ *
+ * Callable while holding just share lock on the buffer content.
+ *
+ * The LSN of the inserted wal record is returned if we had to write,
+ * InvalidXLogRecPtr otherwise.
+ *
+ * It is possible that multiple concurrent backends could attempt to write WAL
+ * records. In that case, multiple copies of the same block would be recorded
+ * in separate WAL records by different backends, though that is still OK from
+ * a correctness perspective.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..fd59f9fc1d 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3865,6 +3865,24 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
{
dirtied = true; /* Means "will be dirtied by this action" */
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page.
+ */
+ if (XLogRecPtrIsInvalid(lsn) /* XXX && file_encryption_method != 0 */)
+ lsn = XLogLSNForHint();
+
/*
* Set the page LSN if we wrote a backup block. We aren't supposed
* to set this when only holding a share lock but as long as we
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--ikeVEW9yuYc//A+q--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 03:25 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 03:25 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +++++-
src/backend/access/transam/xlog.c | 4 ++++
src/backend/access/transam/xloginsert.c | 26 ++++++++++++++++++++++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 18 ++++++++++++++++
src/include/access/xloginsert.h | 2 ++
src/include/catalog/pg_control.h | 1 +
7 files changed, 57 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..eebca0fc16 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,32 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ *
+ * Callable while holding just share lock on the buffer content.
+ *
+ * The LSN of the inserted wal record is returned if we had to write,
+ * InvalidXLogRecPtr otherwise.
+ *
+ * It is possible that multiple concurrent backends could attempt to write WAL
+ * records. In that case, multiple copies of the same block would be recorded
+ * in separate WAL records by different backends, though that is still OK from
+ * a correctness perspective.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..fd59f9fc1d 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3865,6 +3865,24 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
{
dirtied = true; /* Means "will be dirtied by this action" */
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page.
+ */
+ if (XLogRecPtrIsInvalid(lsn) /* XXX && file_encryption_method != 0 */)
+ lsn = XLogLSNForHint();
+
/*
* Set the page LSN if we wrote a backup block. We aren't supposed
* to set this when only holding a share lock but as long as we
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--ikeVEW9yuYc//A+q--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 03:25 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 03:25 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +++++-
src/backend/access/transam/xlog.c | 4 ++++
src/backend/access/transam/xloginsert.c | 26 ++++++++++++++++++++++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 18 ++++++++++++++++
src/include/access/xloginsert.h | 2 ++
src/include/catalog/pg_control.h | 1 +
7 files changed, 57 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..eebca0fc16 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,32 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ *
+ * Callable while holding just share lock on the buffer content.
+ *
+ * The LSN of the inserted wal record is returned if we had to write,
+ * InvalidXLogRecPtr otherwise.
+ *
+ * It is possible that multiple concurrent backends could attempt to write WAL
+ * records. In that case, multiple copies of the same block would be recorded
+ * in separate WAL records by different backends, though that is still OK from
+ * a correctness perspective.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..fd59f9fc1d 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3865,6 +3865,24 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
{
dirtied = true; /* Means "will be dirtied by this action" */
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page.
+ */
+ if (XLogRecPtrIsInvalid(lsn) /* XXX && file_encryption_method != 0 */)
+ lsn = XLogLSNForHint();
+
/*
* Set the page LSN if we wrote a backup block. We aren't supposed
* to set this when only holding a share lock but as long as we
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--ikeVEW9yuYc//A+q--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 03:25 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 03:25 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +++++-
src/backend/access/transam/xlog.c | 4 ++++
src/backend/access/transam/xloginsert.c | 26 ++++++++++++++++++++++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 18 ++++++++++++++++
src/include/access/xloginsert.h | 2 ++
src/include/catalog/pg_control.h | 1 +
7 files changed, 57 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..eebca0fc16 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,32 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ *
+ * Callable while holding just share lock on the buffer content.
+ *
+ * The LSN of the inserted wal record is returned if we had to write,
+ * InvalidXLogRecPtr otherwise.
+ *
+ * It is possible that multiple concurrent backends could attempt to write WAL
+ * records. In that case, multiple copies of the same block would be recorded
+ * in separate WAL records by different backends, though that is still OK from
+ * a correctness perspective.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..fd59f9fc1d 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3865,6 +3865,24 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
{
dirtied = true; /* Means "will be dirtied by this action" */
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page.
+ */
+ if (XLogRecPtrIsInvalid(lsn) /* XXX && file_encryption_method != 0 */)
+ lsn = XLogLSNForHint();
+
/*
* Set the page LSN if we wrote a backup block. We aren't supposed
* to set this when only holding a share lock but as long as we
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--ikeVEW9yuYc//A+q--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 03:25 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 03:25 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +++++-
src/backend/access/transam/xlog.c | 4 ++++
src/backend/access/transam/xloginsert.c | 26 ++++++++++++++++++++++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 18 ++++++++++++++++
src/include/access/xloginsert.h | 2 ++
src/include/catalog/pg_control.h | 1 +
7 files changed, 57 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..eebca0fc16 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,32 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ *
+ * Callable while holding just share lock on the buffer content.
+ *
+ * The LSN of the inserted wal record is returned if we had to write,
+ * InvalidXLogRecPtr otherwise.
+ *
+ * It is possible that multiple concurrent backends could attempt to write WAL
+ * records. In that case, multiple copies of the same block would be recorded
+ * in separate WAL records by different backends, though that is still OK from
+ * a correctness perspective.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..fd59f9fc1d 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3865,6 +3865,24 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
{
dirtied = true; /* Means "will be dirtied by this action" */
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page.
+ */
+ if (XLogRecPtrIsInvalid(lsn) /* XXX && file_encryption_method != 0 */)
+ lsn = XLogLSNForHint();
+
/*
* Set the page LSN if we wrote a backup block. We aren't supposed
* to set this when only holding a share lock but as long as we
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--ikeVEW9yuYc//A+q--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 03:25 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 03:25 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +++++-
src/backend/access/transam/xlog.c | 4 ++++
src/backend/access/transam/xloginsert.c | 26 ++++++++++++++++++++++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 18 ++++++++++++++++
src/include/access/xloginsert.h | 2 ++
src/include/catalog/pg_control.h | 1 +
7 files changed, 57 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..eebca0fc16 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,32 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ *
+ * Callable while holding just share lock on the buffer content.
+ *
+ * The LSN of the inserted wal record is returned if we had to write,
+ * InvalidXLogRecPtr otherwise.
+ *
+ * It is possible that multiple concurrent backends could attempt to write WAL
+ * records. In that case, multiple copies of the same block would be recorded
+ * in separate WAL records by different backends, though that is still OK from
+ * a correctness perspective.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..fd59f9fc1d 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3865,6 +3865,24 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
{
dirtied = true; /* Means "will be dirtied by this action" */
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page.
+ */
+ if (XLogRecPtrIsInvalid(lsn) /* XXX && file_encryption_method != 0 */)
+ lsn = XLogLSNForHint();
+
/*
* Set the page LSN if we wrote a backup block. We aren't supposed
* to set this when only holding a share lock but as long as we
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--ikeVEW9yuYc//A+q--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 03:25 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 03:25 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +++++-
src/backend/access/transam/xlog.c | 4 ++++
src/backend/access/transam/xloginsert.c | 26 ++++++++++++++++++++++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 18 ++++++++++++++++
src/include/access/xloginsert.h | 2 ++
src/include/catalog/pg_control.h | 1 +
7 files changed, 57 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..eebca0fc16 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,32 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ *
+ * Callable while holding just share lock on the buffer content.
+ *
+ * The LSN of the inserted wal record is returned if we had to write,
+ * InvalidXLogRecPtr otherwise.
+ *
+ * It is possible that multiple concurrent backends could attempt to write WAL
+ * records. In that case, multiple copies of the same block would be recorded
+ * in separate WAL records by different backends, though that is still OK from
+ * a correctness perspective.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..fd59f9fc1d 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3865,6 +3865,24 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
{
dirtied = true; /* Means "will be dirtied by this action" */
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page.
+ */
+ if (XLogRecPtrIsInvalid(lsn) /* XXX && file_encryption_method != 0 */)
+ lsn = XLogLSNForHint();
+
/*
* Set the page LSN if we wrote a backup block. We aren't supposed
* to set this when only holding a share lock but as long as we
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--ikeVEW9yuYc//A+q--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 03:25 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 03:25 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +++++-
src/backend/access/transam/xlog.c | 4 ++++
src/backend/access/transam/xloginsert.c | 26 ++++++++++++++++++++++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 18 ++++++++++++++++
src/include/access/xloginsert.h | 2 ++
src/include/catalog/pg_control.h | 1 +
7 files changed, 57 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..eebca0fc16 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,32 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ *
+ * Callable while holding just share lock on the buffer content.
+ *
+ * The LSN of the inserted wal record is returned if we had to write,
+ * InvalidXLogRecPtr otherwise.
+ *
+ * It is possible that multiple concurrent backends could attempt to write WAL
+ * records. In that case, multiple copies of the same block would be recorded
+ * in separate WAL records by different backends, though that is still OK from
+ * a correctness perspective.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..fd59f9fc1d 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3865,6 +3865,24 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
{
dirtied = true; /* Means "will be dirtied by this action" */
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page.
+ */
+ if (XLogRecPtrIsInvalid(lsn) /* XXX && file_encryption_method != 0 */)
+ lsn = XLogLSNForHint();
+
/*
* Set the page LSN if we wrote a backup block. We aren't supposed
* to set this when only holding a share lock but as long as we
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--ikeVEW9yuYc//A+q--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 03:25 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 03:25 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +++++-
src/backend/access/transam/xlog.c | 4 ++++
src/backend/access/transam/xloginsert.c | 26 ++++++++++++++++++++++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 18 ++++++++++++++++
src/include/access/xloginsert.h | 2 ++
src/include/catalog/pg_control.h | 1 +
7 files changed, 57 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..eebca0fc16 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,32 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ *
+ * Callable while holding just share lock on the buffer content.
+ *
+ * The LSN of the inserted wal record is returned if we had to write,
+ * InvalidXLogRecPtr otherwise.
+ *
+ * It is possible that multiple concurrent backends could attempt to write WAL
+ * records. In that case, multiple copies of the same block would be recorded
+ * in separate WAL records by different backends, though that is still OK from
+ * a correctness perspective.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..fd59f9fc1d 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3865,6 +3865,24 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
{
dirtied = true; /* Means "will be dirtied by this action" */
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page.
+ */
+ if (XLogRecPtrIsInvalid(lsn) /* XXX && file_encryption_method != 0 */)
+ lsn = XLogLSNForHint();
+
/*
* Set the page LSN if we wrote a backup block. We aren't supposed
* to set this when only holding a share lock but as long as we
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--ikeVEW9yuYc//A+q--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 03:25 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 03:25 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +++++-
src/backend/access/transam/xlog.c | 4 ++++
src/backend/access/transam/xloginsert.c | 26 ++++++++++++++++++++++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 18 ++++++++++++++++
src/include/access/xloginsert.h | 2 ++
src/include/catalog/pg_control.h | 1 +
7 files changed, 57 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..eebca0fc16 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,32 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ *
+ * Callable while holding just share lock on the buffer content.
+ *
+ * The LSN of the inserted wal record is returned if we had to write,
+ * InvalidXLogRecPtr otherwise.
+ *
+ * It is possible that multiple concurrent backends could attempt to write WAL
+ * records. In that case, multiple copies of the same block would be recorded
+ * in separate WAL records by different backends, though that is still OK from
+ * a correctness perspective.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..fd59f9fc1d 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3865,6 +3865,24 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
{
dirtied = true; /* Means "will be dirtied by this action" */
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page.
+ */
+ if (XLogRecPtrIsInvalid(lsn) /* XXX && file_encryption_method != 0 */)
+ lsn = XLogLSNForHint();
+
/*
* Set the page LSN if we wrote a backup block. We aren't supposed
* to set this when only holding a share lock but as long as we
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--ikeVEW9yuYc//A+q--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 03:25 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 03:25 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +++++-
src/backend/access/transam/xlog.c | 4 ++++
src/backend/access/transam/xloginsert.c | 26 ++++++++++++++++++++++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 18 ++++++++++++++++
src/include/access/xloginsert.h | 2 ++
src/include/catalog/pg_control.h | 1 +
7 files changed, 57 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..eebca0fc16 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,32 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ *
+ * Callable while holding just share lock on the buffer content.
+ *
+ * The LSN of the inserted wal record is returned if we had to write,
+ * InvalidXLogRecPtr otherwise.
+ *
+ * It is possible that multiple concurrent backends could attempt to write WAL
+ * records. In that case, multiple copies of the same block would be recorded
+ * in separate WAL records by different backends, though that is still OK from
+ * a correctness perspective.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..fd59f9fc1d 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3865,6 +3865,24 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
{
dirtied = true; /* Means "will be dirtied by this action" */
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page.
+ */
+ if (XLogRecPtrIsInvalid(lsn) /* XXX && file_encryption_method != 0 */)
+ lsn = XLogLSNForHint();
+
/*
* Set the page LSN if we wrote a backup block. We aren't supposed
* to set this when only holding a share lock but as long as we
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--ikeVEW9yuYc//A+q--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 03:25 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 03:25 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +++++-
src/backend/access/transam/xlog.c | 4 ++++
src/backend/access/transam/xloginsert.c | 26 ++++++++++++++++++++++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 18 ++++++++++++++++
src/include/access/xloginsert.h | 2 ++
src/include/catalog/pg_control.h | 1 +
7 files changed, 57 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..eebca0fc16 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,32 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ *
+ * Callable while holding just share lock on the buffer content.
+ *
+ * The LSN of the inserted wal record is returned if we had to write,
+ * InvalidXLogRecPtr otherwise.
+ *
+ * It is possible that multiple concurrent backends could attempt to write WAL
+ * records. In that case, multiple copies of the same block would be recorded
+ * in separate WAL records by different backends, though that is still OK from
+ * a correctness perspective.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..fd59f9fc1d 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3865,6 +3865,24 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
{
dirtied = true; /* Means "will be dirtied by this action" */
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page.
+ */
+ if (XLogRecPtrIsInvalid(lsn) /* XXX && file_encryption_method != 0 */)
+ lsn = XLogLSNForHint();
+
/*
* Set the page LSN if we wrote a backup block. We aren't supposed
* to set this when only holding a share lock but as long as we
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--ikeVEW9yuYc//A+q--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 03:25 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 03:25 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +++++-
src/backend/access/transam/xlog.c | 4 ++++
src/backend/access/transam/xloginsert.c | 26 ++++++++++++++++++++++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 18 ++++++++++++++++
src/include/access/xloginsert.h | 2 ++
src/include/catalog/pg_control.h | 1 +
7 files changed, 57 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..eebca0fc16 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,32 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ *
+ * Callable while holding just share lock on the buffer content.
+ *
+ * The LSN of the inserted wal record is returned if we had to write,
+ * InvalidXLogRecPtr otherwise.
+ *
+ * It is possible that multiple concurrent backends could attempt to write WAL
+ * records. In that case, multiple copies of the same block would be recorded
+ * in separate WAL records by different backends, though that is still OK from
+ * a correctness perspective.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..fd59f9fc1d 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3865,6 +3865,24 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
{
dirtied = true; /* Means "will be dirtied by this action" */
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page.
+ */
+ if (XLogRecPtrIsInvalid(lsn) /* XXX && file_encryption_method != 0 */)
+ lsn = XLogLSNForHint();
+
/*
* Set the page LSN if we wrote a backup block. We aren't supposed
* to set this when only holding a share lock but as long as we
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--ikeVEW9yuYc//A+q--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 03:25 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 03:25 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +++++-
src/backend/access/transam/xlog.c | 4 ++++
src/backend/access/transam/xloginsert.c | 26 ++++++++++++++++++++++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 18 ++++++++++++++++
src/include/access/xloginsert.h | 2 ++
src/include/catalog/pg_control.h | 1 +
7 files changed, 57 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..eebca0fc16 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,32 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ *
+ * Callable while holding just share lock on the buffer content.
+ *
+ * The LSN of the inserted wal record is returned if we had to write,
+ * InvalidXLogRecPtr otherwise.
+ *
+ * It is possible that multiple concurrent backends could attempt to write WAL
+ * records. In that case, multiple copies of the same block would be recorded
+ * in separate WAL records by different backends, though that is still OK from
+ * a correctness perspective.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..fd59f9fc1d 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3865,6 +3865,24 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
{
dirtied = true; /* Means "will be dirtied by this action" */
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page.
+ */
+ if (XLogRecPtrIsInvalid(lsn) /* XXX && file_encryption_method != 0 */)
+ lsn = XLogLSNForHint();
+
/*
* Set the page LSN if we wrote a backup block. We aren't supposed
* to set this when only holding a share lock but as long as we
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--ikeVEW9yuYc//A+q--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 03:25 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 03:25 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +++++-
src/backend/access/transam/xlog.c | 4 ++++
src/backend/access/transam/xloginsert.c | 26 ++++++++++++++++++++++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 18 ++++++++++++++++
src/include/access/xloginsert.h | 2 ++
src/include/catalog/pg_control.h | 1 +
7 files changed, 57 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..eebca0fc16 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,32 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ *
+ * Callable while holding just share lock on the buffer content.
+ *
+ * The LSN of the inserted wal record is returned if we had to write,
+ * InvalidXLogRecPtr otherwise.
+ *
+ * It is possible that multiple concurrent backends could attempt to write WAL
+ * records. In that case, multiple copies of the same block would be recorded
+ * in separate WAL records by different backends, though that is still OK from
+ * a correctness perspective.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..fd59f9fc1d 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3865,6 +3865,24 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
{
dirtied = true; /* Means "will be dirtied by this action" */
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page.
+ */
+ if (XLogRecPtrIsInvalid(lsn) /* XXX && file_encryption_method != 0 */)
+ lsn = XLogLSNForHint();
+
/*
* Set the page LSN if we wrote a backup block. We aren't supposed
* to set this when only holding a share lock but as long as we
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--ikeVEW9yuYc//A+q--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 18:43 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 18:43 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +-
src/backend/access/transam/xlog.c | 4 ++
src/backend/access/transam/xloginsert.c | 16 +++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 89 ++++++++++++++++--------
src/include/access/xloginsert.h | 2 +
src/include/catalog/pg_control.h | 1 +
7 files changed, 90 insertions(+), 29 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..7635a3d44d 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,22 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..b9f3f76798 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3810,13 +3810,14 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
* If we need to protect hint bit updates from torn writes, WAL-log a
* full page image of the page. This full page image is only necessary
* if the hint bit update is the first change to the page since the
- * last checkpoint.
+ * last checkpoint. If cluster file encryption is enabled, we also
+ * need to generate new page LSNs for non-first-page writes during
+ * a checkpoint.
*
* We don't check full_page_writes here because that logic is included
* when we call XLogInsert() since the value changes dynamically.
*/
- if (XLogHintBitIsNeeded() &&
- (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT))
+ if (XLogHintBitIsNeeded())
{
/*
* If we must not write WAL, due to a relfilenode-specific
@@ -3826,35 +3827,67 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
*
* See src/backend/storage/page/README for longer discussion.
*/
- if (RecoveryInProgress() ||
- RelFileNodeSkippingWAL(bufHdr->tag.rnode))
+ if (((pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ /* XXX || file_encryption_method != 0 */) &&
+ (RecoveryInProgress() ||
+ RelFileNodeSkippingWAL(bufHdr->tag.rnode)))
return;
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ {
+ /*
+ * If the block is already dirty because we either made a change
+ * or set a hint already, then we don't need to write a full page
+ * image. Note that aggressive cleaning of blocks dirtied by hint
+ * bit setting would increase the call rate. Bulk setting of hint
+ * bits would reduce the call rate...
+ *
+ * We must issue the WAL record before we mark the buffer dirty.
+ * Otherwise we might write the page before we write the WAL. That
+ * causes a race condition, since a checkpoint might occur between
+ * writing the WAL record and marking the buffer dirty. We solve
+ * that with a kluge, but one that is already in use during
+ * transaction commit to prevent race conditions. Basically, we
+ * simply prevent the checkpoint WAL record from being written
+ * until we have marked the buffer dirty. We don't start the
+ * checkpoint flush until we have marked dirty, so our checkpoint
+ * must flush the change to disk successfully or the checkpoint
+ * never gets written, so crash recovery will fix.
+ *
+ * It's possible we may enter here without an xid, so it is
+ * essential that CreateCheckpoint waits for virtual transactions
+ * rather than full transactionids.
+ */
+ MyProc->delayChkpt = delayChkpt = true;
+ lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ }
+
/*
- * If the block is already dirty because we either made a change
- * or set a hint already, then we don't need to write a full page
- * image. Note that aggressive cleaning of blocks dirtied by hint
- * bit setting would increase the call rate. Bulk setting of hint
- * bits would reduce the call rate...
- *
- * We must issue the WAL record before we mark the buffer dirty.
- * Otherwise we might write the page before we write the WAL. That
- * causes a race condition, since a checkpoint might occur between
- * writing the WAL record and marking the buffer dirty. We solve
- * that with a kluge, but one that is already in use during
- * transaction commit to prevent race conditions. Basically, we
- * simply prevent the checkpoint WAL record from being written
- * until we have marked the buffer dirty. We don't start the
- * checkpoint flush until we have marked dirty, so our checkpoint
- * must flush the change to disk successfully or the checkpoint
- * never gets written, so crash recovery will fix.
- *
- * It's possible we may enter here without an xid, so it is
- * essential that CreateCheckpoint waits for virtual transactions
- * rather than full transactionids.
+ * For cluster file encryption, we generate a new page LSN
+ * for hint-bit non-permanent and non-first page writes.
*/
- MyProc->delayChkpt = delayChkpt = true;
- lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ if (/* XXX file_encryption_method != 0 && */
+ XLogRecPtrIsInvalid(lsn) &&
+ /* XXX can we check BM_DIRTY without a page lock? */
+ !(pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY))
+ {
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page below.
+ */
+ lsn = XLogLSNForHint();
+ }
}
buf_state = LockBufHdr(bufHdr);
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--T4sUOijqQbZv57TR--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 18:43 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 18:43 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +-
src/backend/access/transam/xlog.c | 4 ++
src/backend/access/transam/xloginsert.c | 16 +++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 89 ++++++++++++++++--------
src/include/access/xloginsert.h | 2 +
src/include/catalog/pg_control.h | 1 +
7 files changed, 90 insertions(+), 29 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..7635a3d44d 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,22 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..b9f3f76798 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3810,13 +3810,14 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
* If we need to protect hint bit updates from torn writes, WAL-log a
* full page image of the page. This full page image is only necessary
* if the hint bit update is the first change to the page since the
- * last checkpoint.
+ * last checkpoint. If cluster file encryption is enabled, we also
+ * need to generate new page LSNs for non-first-page writes during
+ * a checkpoint.
*
* We don't check full_page_writes here because that logic is included
* when we call XLogInsert() since the value changes dynamically.
*/
- if (XLogHintBitIsNeeded() &&
- (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT))
+ if (XLogHintBitIsNeeded())
{
/*
* If we must not write WAL, due to a relfilenode-specific
@@ -3826,35 +3827,67 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
*
* See src/backend/storage/page/README for longer discussion.
*/
- if (RecoveryInProgress() ||
- RelFileNodeSkippingWAL(bufHdr->tag.rnode))
+ if (((pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ /* XXX || file_encryption_method != 0 */) &&
+ (RecoveryInProgress() ||
+ RelFileNodeSkippingWAL(bufHdr->tag.rnode)))
return;
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ {
+ /*
+ * If the block is already dirty because we either made a change
+ * or set a hint already, then we don't need to write a full page
+ * image. Note that aggressive cleaning of blocks dirtied by hint
+ * bit setting would increase the call rate. Bulk setting of hint
+ * bits would reduce the call rate...
+ *
+ * We must issue the WAL record before we mark the buffer dirty.
+ * Otherwise we might write the page before we write the WAL. That
+ * causes a race condition, since a checkpoint might occur between
+ * writing the WAL record and marking the buffer dirty. We solve
+ * that with a kluge, but one that is already in use during
+ * transaction commit to prevent race conditions. Basically, we
+ * simply prevent the checkpoint WAL record from being written
+ * until we have marked the buffer dirty. We don't start the
+ * checkpoint flush until we have marked dirty, so our checkpoint
+ * must flush the change to disk successfully or the checkpoint
+ * never gets written, so crash recovery will fix.
+ *
+ * It's possible we may enter here without an xid, so it is
+ * essential that CreateCheckpoint waits for virtual transactions
+ * rather than full transactionids.
+ */
+ MyProc->delayChkpt = delayChkpt = true;
+ lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ }
+
/*
- * If the block is already dirty because we either made a change
- * or set a hint already, then we don't need to write a full page
- * image. Note that aggressive cleaning of blocks dirtied by hint
- * bit setting would increase the call rate. Bulk setting of hint
- * bits would reduce the call rate...
- *
- * We must issue the WAL record before we mark the buffer dirty.
- * Otherwise we might write the page before we write the WAL. That
- * causes a race condition, since a checkpoint might occur between
- * writing the WAL record and marking the buffer dirty. We solve
- * that with a kluge, but one that is already in use during
- * transaction commit to prevent race conditions. Basically, we
- * simply prevent the checkpoint WAL record from being written
- * until we have marked the buffer dirty. We don't start the
- * checkpoint flush until we have marked dirty, so our checkpoint
- * must flush the change to disk successfully or the checkpoint
- * never gets written, so crash recovery will fix.
- *
- * It's possible we may enter here without an xid, so it is
- * essential that CreateCheckpoint waits for virtual transactions
- * rather than full transactionids.
+ * For cluster file encryption, we generate a new page LSN
+ * for hint-bit non-permanent and non-first page writes.
*/
- MyProc->delayChkpt = delayChkpt = true;
- lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ if (/* XXX file_encryption_method != 0 && */
+ XLogRecPtrIsInvalid(lsn) &&
+ /* XXX can we check BM_DIRTY without a page lock? */
+ !(pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY))
+ {
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page below.
+ */
+ lsn = XLogLSNForHint();
+ }
}
buf_state = LockBufHdr(bufHdr);
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--T4sUOijqQbZv57TR--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 18:43 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 18:43 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +-
src/backend/access/transam/xlog.c | 4 ++
src/backend/access/transam/xloginsert.c | 16 +++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 89 ++++++++++++++++--------
src/include/access/xloginsert.h | 2 +
src/include/catalog/pg_control.h | 1 +
7 files changed, 90 insertions(+), 29 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..7635a3d44d 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,22 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..b9f3f76798 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3810,13 +3810,14 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
* If we need to protect hint bit updates from torn writes, WAL-log a
* full page image of the page. This full page image is only necessary
* if the hint bit update is the first change to the page since the
- * last checkpoint.
+ * last checkpoint. If cluster file encryption is enabled, we also
+ * need to generate new page LSNs for non-first-page writes during
+ * a checkpoint.
*
* We don't check full_page_writes here because that logic is included
* when we call XLogInsert() since the value changes dynamically.
*/
- if (XLogHintBitIsNeeded() &&
- (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT))
+ if (XLogHintBitIsNeeded())
{
/*
* If we must not write WAL, due to a relfilenode-specific
@@ -3826,35 +3827,67 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
*
* See src/backend/storage/page/README for longer discussion.
*/
- if (RecoveryInProgress() ||
- RelFileNodeSkippingWAL(bufHdr->tag.rnode))
+ if (((pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ /* XXX || file_encryption_method != 0 */) &&
+ (RecoveryInProgress() ||
+ RelFileNodeSkippingWAL(bufHdr->tag.rnode)))
return;
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ {
+ /*
+ * If the block is already dirty because we either made a change
+ * or set a hint already, then we don't need to write a full page
+ * image. Note that aggressive cleaning of blocks dirtied by hint
+ * bit setting would increase the call rate. Bulk setting of hint
+ * bits would reduce the call rate...
+ *
+ * We must issue the WAL record before we mark the buffer dirty.
+ * Otherwise we might write the page before we write the WAL. That
+ * causes a race condition, since a checkpoint might occur between
+ * writing the WAL record and marking the buffer dirty. We solve
+ * that with a kluge, but one that is already in use during
+ * transaction commit to prevent race conditions. Basically, we
+ * simply prevent the checkpoint WAL record from being written
+ * until we have marked the buffer dirty. We don't start the
+ * checkpoint flush until we have marked dirty, so our checkpoint
+ * must flush the change to disk successfully or the checkpoint
+ * never gets written, so crash recovery will fix.
+ *
+ * It's possible we may enter here without an xid, so it is
+ * essential that CreateCheckpoint waits for virtual transactions
+ * rather than full transactionids.
+ */
+ MyProc->delayChkpt = delayChkpt = true;
+ lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ }
+
/*
- * If the block is already dirty because we either made a change
- * or set a hint already, then we don't need to write a full page
- * image. Note that aggressive cleaning of blocks dirtied by hint
- * bit setting would increase the call rate. Bulk setting of hint
- * bits would reduce the call rate...
- *
- * We must issue the WAL record before we mark the buffer dirty.
- * Otherwise we might write the page before we write the WAL. That
- * causes a race condition, since a checkpoint might occur between
- * writing the WAL record and marking the buffer dirty. We solve
- * that with a kluge, but one that is already in use during
- * transaction commit to prevent race conditions. Basically, we
- * simply prevent the checkpoint WAL record from being written
- * until we have marked the buffer dirty. We don't start the
- * checkpoint flush until we have marked dirty, so our checkpoint
- * must flush the change to disk successfully or the checkpoint
- * never gets written, so crash recovery will fix.
- *
- * It's possible we may enter here without an xid, so it is
- * essential that CreateCheckpoint waits for virtual transactions
- * rather than full transactionids.
+ * For cluster file encryption, we generate a new page LSN
+ * for hint-bit non-permanent and non-first page writes.
*/
- MyProc->delayChkpt = delayChkpt = true;
- lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ if (/* XXX file_encryption_method != 0 && */
+ XLogRecPtrIsInvalid(lsn) &&
+ /* XXX can we check BM_DIRTY without a page lock? */
+ !(pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY))
+ {
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page below.
+ */
+ lsn = XLogLSNForHint();
+ }
}
buf_state = LockBufHdr(bufHdr);
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--T4sUOijqQbZv57TR--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 18:43 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 18:43 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +-
src/backend/access/transam/xlog.c | 4 ++
src/backend/access/transam/xloginsert.c | 16 +++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 89 ++++++++++++++++--------
src/include/access/xloginsert.h | 2 +
src/include/catalog/pg_control.h | 1 +
7 files changed, 90 insertions(+), 29 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..7635a3d44d 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,22 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..b9f3f76798 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3810,13 +3810,14 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
* If we need to protect hint bit updates from torn writes, WAL-log a
* full page image of the page. This full page image is only necessary
* if the hint bit update is the first change to the page since the
- * last checkpoint.
+ * last checkpoint. If cluster file encryption is enabled, we also
+ * need to generate new page LSNs for non-first-page writes during
+ * a checkpoint.
*
* We don't check full_page_writes here because that logic is included
* when we call XLogInsert() since the value changes dynamically.
*/
- if (XLogHintBitIsNeeded() &&
- (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT))
+ if (XLogHintBitIsNeeded())
{
/*
* If we must not write WAL, due to a relfilenode-specific
@@ -3826,35 +3827,67 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
*
* See src/backend/storage/page/README for longer discussion.
*/
- if (RecoveryInProgress() ||
- RelFileNodeSkippingWAL(bufHdr->tag.rnode))
+ if (((pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ /* XXX || file_encryption_method != 0 */) &&
+ (RecoveryInProgress() ||
+ RelFileNodeSkippingWAL(bufHdr->tag.rnode)))
return;
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ {
+ /*
+ * If the block is already dirty because we either made a change
+ * or set a hint already, then we don't need to write a full page
+ * image. Note that aggressive cleaning of blocks dirtied by hint
+ * bit setting would increase the call rate. Bulk setting of hint
+ * bits would reduce the call rate...
+ *
+ * We must issue the WAL record before we mark the buffer dirty.
+ * Otherwise we might write the page before we write the WAL. That
+ * causes a race condition, since a checkpoint might occur between
+ * writing the WAL record and marking the buffer dirty. We solve
+ * that with a kluge, but one that is already in use during
+ * transaction commit to prevent race conditions. Basically, we
+ * simply prevent the checkpoint WAL record from being written
+ * until we have marked the buffer dirty. We don't start the
+ * checkpoint flush until we have marked dirty, so our checkpoint
+ * must flush the change to disk successfully or the checkpoint
+ * never gets written, so crash recovery will fix.
+ *
+ * It's possible we may enter here without an xid, so it is
+ * essential that CreateCheckpoint waits for virtual transactions
+ * rather than full transactionids.
+ */
+ MyProc->delayChkpt = delayChkpt = true;
+ lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ }
+
/*
- * If the block is already dirty because we either made a change
- * or set a hint already, then we don't need to write a full page
- * image. Note that aggressive cleaning of blocks dirtied by hint
- * bit setting would increase the call rate. Bulk setting of hint
- * bits would reduce the call rate...
- *
- * We must issue the WAL record before we mark the buffer dirty.
- * Otherwise we might write the page before we write the WAL. That
- * causes a race condition, since a checkpoint might occur between
- * writing the WAL record and marking the buffer dirty. We solve
- * that with a kluge, but one that is already in use during
- * transaction commit to prevent race conditions. Basically, we
- * simply prevent the checkpoint WAL record from being written
- * until we have marked the buffer dirty. We don't start the
- * checkpoint flush until we have marked dirty, so our checkpoint
- * must flush the change to disk successfully or the checkpoint
- * never gets written, so crash recovery will fix.
- *
- * It's possible we may enter here without an xid, so it is
- * essential that CreateCheckpoint waits for virtual transactions
- * rather than full transactionids.
+ * For cluster file encryption, we generate a new page LSN
+ * for hint-bit non-permanent and non-first page writes.
*/
- MyProc->delayChkpt = delayChkpt = true;
- lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ if (/* XXX file_encryption_method != 0 && */
+ XLogRecPtrIsInvalid(lsn) &&
+ /* XXX can we check BM_DIRTY without a page lock? */
+ !(pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY))
+ {
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page below.
+ */
+ lsn = XLogLSNForHint();
+ }
}
buf_state = LockBufHdr(bufHdr);
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--T4sUOijqQbZv57TR--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 18:43 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 18:43 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +-
src/backend/access/transam/xlog.c | 4 ++
src/backend/access/transam/xloginsert.c | 16 +++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 89 ++++++++++++++++--------
src/include/access/xloginsert.h | 2 +
src/include/catalog/pg_control.h | 1 +
7 files changed, 90 insertions(+), 29 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..7635a3d44d 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,22 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..b9f3f76798 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3810,13 +3810,14 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
* If we need to protect hint bit updates from torn writes, WAL-log a
* full page image of the page. This full page image is only necessary
* if the hint bit update is the first change to the page since the
- * last checkpoint.
+ * last checkpoint. If cluster file encryption is enabled, we also
+ * need to generate new page LSNs for non-first-page writes during
+ * a checkpoint.
*
* We don't check full_page_writes here because that logic is included
* when we call XLogInsert() since the value changes dynamically.
*/
- if (XLogHintBitIsNeeded() &&
- (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT))
+ if (XLogHintBitIsNeeded())
{
/*
* If we must not write WAL, due to a relfilenode-specific
@@ -3826,35 +3827,67 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
*
* See src/backend/storage/page/README for longer discussion.
*/
- if (RecoveryInProgress() ||
- RelFileNodeSkippingWAL(bufHdr->tag.rnode))
+ if (((pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ /* XXX || file_encryption_method != 0 */) &&
+ (RecoveryInProgress() ||
+ RelFileNodeSkippingWAL(bufHdr->tag.rnode)))
return;
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ {
+ /*
+ * If the block is already dirty because we either made a change
+ * or set a hint already, then we don't need to write a full page
+ * image. Note that aggressive cleaning of blocks dirtied by hint
+ * bit setting would increase the call rate. Bulk setting of hint
+ * bits would reduce the call rate...
+ *
+ * We must issue the WAL record before we mark the buffer dirty.
+ * Otherwise we might write the page before we write the WAL. That
+ * causes a race condition, since a checkpoint might occur between
+ * writing the WAL record and marking the buffer dirty. We solve
+ * that with a kluge, but one that is already in use during
+ * transaction commit to prevent race conditions. Basically, we
+ * simply prevent the checkpoint WAL record from being written
+ * until we have marked the buffer dirty. We don't start the
+ * checkpoint flush until we have marked dirty, so our checkpoint
+ * must flush the change to disk successfully or the checkpoint
+ * never gets written, so crash recovery will fix.
+ *
+ * It's possible we may enter here without an xid, so it is
+ * essential that CreateCheckpoint waits for virtual transactions
+ * rather than full transactionids.
+ */
+ MyProc->delayChkpt = delayChkpt = true;
+ lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ }
+
/*
- * If the block is already dirty because we either made a change
- * or set a hint already, then we don't need to write a full page
- * image. Note that aggressive cleaning of blocks dirtied by hint
- * bit setting would increase the call rate. Bulk setting of hint
- * bits would reduce the call rate...
- *
- * We must issue the WAL record before we mark the buffer dirty.
- * Otherwise we might write the page before we write the WAL. That
- * causes a race condition, since a checkpoint might occur between
- * writing the WAL record and marking the buffer dirty. We solve
- * that with a kluge, but one that is already in use during
- * transaction commit to prevent race conditions. Basically, we
- * simply prevent the checkpoint WAL record from being written
- * until we have marked the buffer dirty. We don't start the
- * checkpoint flush until we have marked dirty, so our checkpoint
- * must flush the change to disk successfully or the checkpoint
- * never gets written, so crash recovery will fix.
- *
- * It's possible we may enter here without an xid, so it is
- * essential that CreateCheckpoint waits for virtual transactions
- * rather than full transactionids.
+ * For cluster file encryption, we generate a new page LSN
+ * for hint-bit non-permanent and non-first page writes.
*/
- MyProc->delayChkpt = delayChkpt = true;
- lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ if (/* XXX file_encryption_method != 0 && */
+ XLogRecPtrIsInvalid(lsn) &&
+ /* XXX can we check BM_DIRTY without a page lock? */
+ !(pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY))
+ {
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page below.
+ */
+ lsn = XLogLSNForHint();
+ }
}
buf_state = LockBufHdr(bufHdr);
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--T4sUOijqQbZv57TR--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 18:43 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 18:43 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +-
src/backend/access/transam/xlog.c | 4 ++
src/backend/access/transam/xloginsert.c | 16 +++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 89 ++++++++++++++++--------
src/include/access/xloginsert.h | 2 +
src/include/catalog/pg_control.h | 1 +
7 files changed, 90 insertions(+), 29 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..7635a3d44d 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,22 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..b9f3f76798 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3810,13 +3810,14 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
* If we need to protect hint bit updates from torn writes, WAL-log a
* full page image of the page. This full page image is only necessary
* if the hint bit update is the first change to the page since the
- * last checkpoint.
+ * last checkpoint. If cluster file encryption is enabled, we also
+ * need to generate new page LSNs for non-first-page writes during
+ * a checkpoint.
*
* We don't check full_page_writes here because that logic is included
* when we call XLogInsert() since the value changes dynamically.
*/
- if (XLogHintBitIsNeeded() &&
- (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT))
+ if (XLogHintBitIsNeeded())
{
/*
* If we must not write WAL, due to a relfilenode-specific
@@ -3826,35 +3827,67 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
*
* See src/backend/storage/page/README for longer discussion.
*/
- if (RecoveryInProgress() ||
- RelFileNodeSkippingWAL(bufHdr->tag.rnode))
+ if (((pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ /* XXX || file_encryption_method != 0 */) &&
+ (RecoveryInProgress() ||
+ RelFileNodeSkippingWAL(bufHdr->tag.rnode)))
return;
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ {
+ /*
+ * If the block is already dirty because we either made a change
+ * or set a hint already, then we don't need to write a full page
+ * image. Note that aggressive cleaning of blocks dirtied by hint
+ * bit setting would increase the call rate. Bulk setting of hint
+ * bits would reduce the call rate...
+ *
+ * We must issue the WAL record before we mark the buffer dirty.
+ * Otherwise we might write the page before we write the WAL. That
+ * causes a race condition, since a checkpoint might occur between
+ * writing the WAL record and marking the buffer dirty. We solve
+ * that with a kluge, but one that is already in use during
+ * transaction commit to prevent race conditions. Basically, we
+ * simply prevent the checkpoint WAL record from being written
+ * until we have marked the buffer dirty. We don't start the
+ * checkpoint flush until we have marked dirty, so our checkpoint
+ * must flush the change to disk successfully or the checkpoint
+ * never gets written, so crash recovery will fix.
+ *
+ * It's possible we may enter here without an xid, so it is
+ * essential that CreateCheckpoint waits for virtual transactions
+ * rather than full transactionids.
+ */
+ MyProc->delayChkpt = delayChkpt = true;
+ lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ }
+
/*
- * If the block is already dirty because we either made a change
- * or set a hint already, then we don't need to write a full page
- * image. Note that aggressive cleaning of blocks dirtied by hint
- * bit setting would increase the call rate. Bulk setting of hint
- * bits would reduce the call rate...
- *
- * We must issue the WAL record before we mark the buffer dirty.
- * Otherwise we might write the page before we write the WAL. That
- * causes a race condition, since a checkpoint might occur between
- * writing the WAL record and marking the buffer dirty. We solve
- * that with a kluge, but one that is already in use during
- * transaction commit to prevent race conditions. Basically, we
- * simply prevent the checkpoint WAL record from being written
- * until we have marked the buffer dirty. We don't start the
- * checkpoint flush until we have marked dirty, so our checkpoint
- * must flush the change to disk successfully or the checkpoint
- * never gets written, so crash recovery will fix.
- *
- * It's possible we may enter here without an xid, so it is
- * essential that CreateCheckpoint waits for virtual transactions
- * rather than full transactionids.
+ * For cluster file encryption, we generate a new page LSN
+ * for hint-bit non-permanent and non-first page writes.
*/
- MyProc->delayChkpt = delayChkpt = true;
- lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ if (/* XXX file_encryption_method != 0 && */
+ XLogRecPtrIsInvalid(lsn) &&
+ /* XXX can we check BM_DIRTY without a page lock? */
+ !(pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY))
+ {
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page below.
+ */
+ lsn = XLogLSNForHint();
+ }
}
buf_state = LockBufHdr(bufHdr);
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--T4sUOijqQbZv57TR--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 18:43 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 18:43 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +-
src/backend/access/transam/xlog.c | 4 ++
src/backend/access/transam/xloginsert.c | 16 +++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 89 ++++++++++++++++--------
src/include/access/xloginsert.h | 2 +
src/include/catalog/pg_control.h | 1 +
7 files changed, 90 insertions(+), 29 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..7635a3d44d 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,22 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..b9f3f76798 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3810,13 +3810,14 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
* If we need to protect hint bit updates from torn writes, WAL-log a
* full page image of the page. This full page image is only necessary
* if the hint bit update is the first change to the page since the
- * last checkpoint.
+ * last checkpoint. If cluster file encryption is enabled, we also
+ * need to generate new page LSNs for non-first-page writes during
+ * a checkpoint.
*
* We don't check full_page_writes here because that logic is included
* when we call XLogInsert() since the value changes dynamically.
*/
- if (XLogHintBitIsNeeded() &&
- (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT))
+ if (XLogHintBitIsNeeded())
{
/*
* If we must not write WAL, due to a relfilenode-specific
@@ -3826,35 +3827,67 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
*
* See src/backend/storage/page/README for longer discussion.
*/
- if (RecoveryInProgress() ||
- RelFileNodeSkippingWAL(bufHdr->tag.rnode))
+ if (((pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ /* XXX || file_encryption_method != 0 */) &&
+ (RecoveryInProgress() ||
+ RelFileNodeSkippingWAL(bufHdr->tag.rnode)))
return;
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ {
+ /*
+ * If the block is already dirty because we either made a change
+ * or set a hint already, then we don't need to write a full page
+ * image. Note that aggressive cleaning of blocks dirtied by hint
+ * bit setting would increase the call rate. Bulk setting of hint
+ * bits would reduce the call rate...
+ *
+ * We must issue the WAL record before we mark the buffer dirty.
+ * Otherwise we might write the page before we write the WAL. That
+ * causes a race condition, since a checkpoint might occur between
+ * writing the WAL record and marking the buffer dirty. We solve
+ * that with a kluge, but one that is already in use during
+ * transaction commit to prevent race conditions. Basically, we
+ * simply prevent the checkpoint WAL record from being written
+ * until we have marked the buffer dirty. We don't start the
+ * checkpoint flush until we have marked dirty, so our checkpoint
+ * must flush the change to disk successfully or the checkpoint
+ * never gets written, so crash recovery will fix.
+ *
+ * It's possible we may enter here without an xid, so it is
+ * essential that CreateCheckpoint waits for virtual transactions
+ * rather than full transactionids.
+ */
+ MyProc->delayChkpt = delayChkpt = true;
+ lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ }
+
/*
- * If the block is already dirty because we either made a change
- * or set a hint already, then we don't need to write a full page
- * image. Note that aggressive cleaning of blocks dirtied by hint
- * bit setting would increase the call rate. Bulk setting of hint
- * bits would reduce the call rate...
- *
- * We must issue the WAL record before we mark the buffer dirty.
- * Otherwise we might write the page before we write the WAL. That
- * causes a race condition, since a checkpoint might occur between
- * writing the WAL record and marking the buffer dirty. We solve
- * that with a kluge, but one that is already in use during
- * transaction commit to prevent race conditions. Basically, we
- * simply prevent the checkpoint WAL record from being written
- * until we have marked the buffer dirty. We don't start the
- * checkpoint flush until we have marked dirty, so our checkpoint
- * must flush the change to disk successfully or the checkpoint
- * never gets written, so crash recovery will fix.
- *
- * It's possible we may enter here without an xid, so it is
- * essential that CreateCheckpoint waits for virtual transactions
- * rather than full transactionids.
+ * For cluster file encryption, we generate a new page LSN
+ * for hint-bit non-permanent and non-first page writes.
*/
- MyProc->delayChkpt = delayChkpt = true;
- lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ if (/* XXX file_encryption_method != 0 && */
+ XLogRecPtrIsInvalid(lsn) &&
+ /* XXX can we check BM_DIRTY without a page lock? */
+ !(pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY))
+ {
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page below.
+ */
+ lsn = XLogLSNForHint();
+ }
}
buf_state = LockBufHdr(bufHdr);
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--T4sUOijqQbZv57TR--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 18:43 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 18:43 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +-
src/backend/access/transam/xlog.c | 4 ++
src/backend/access/transam/xloginsert.c | 16 +++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 89 ++++++++++++++++--------
src/include/access/xloginsert.h | 2 +
src/include/catalog/pg_control.h | 1 +
7 files changed, 90 insertions(+), 29 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..7635a3d44d 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,22 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..b9f3f76798 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3810,13 +3810,14 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
* If we need to protect hint bit updates from torn writes, WAL-log a
* full page image of the page. This full page image is only necessary
* if the hint bit update is the first change to the page since the
- * last checkpoint.
+ * last checkpoint. If cluster file encryption is enabled, we also
+ * need to generate new page LSNs for non-first-page writes during
+ * a checkpoint.
*
* We don't check full_page_writes here because that logic is included
* when we call XLogInsert() since the value changes dynamically.
*/
- if (XLogHintBitIsNeeded() &&
- (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT))
+ if (XLogHintBitIsNeeded())
{
/*
* If we must not write WAL, due to a relfilenode-specific
@@ -3826,35 +3827,67 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
*
* See src/backend/storage/page/README for longer discussion.
*/
- if (RecoveryInProgress() ||
- RelFileNodeSkippingWAL(bufHdr->tag.rnode))
+ if (((pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ /* XXX || file_encryption_method != 0 */) &&
+ (RecoveryInProgress() ||
+ RelFileNodeSkippingWAL(bufHdr->tag.rnode)))
return;
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ {
+ /*
+ * If the block is already dirty because we either made a change
+ * or set a hint already, then we don't need to write a full page
+ * image. Note that aggressive cleaning of blocks dirtied by hint
+ * bit setting would increase the call rate. Bulk setting of hint
+ * bits would reduce the call rate...
+ *
+ * We must issue the WAL record before we mark the buffer dirty.
+ * Otherwise we might write the page before we write the WAL. That
+ * causes a race condition, since a checkpoint might occur between
+ * writing the WAL record and marking the buffer dirty. We solve
+ * that with a kluge, but one that is already in use during
+ * transaction commit to prevent race conditions. Basically, we
+ * simply prevent the checkpoint WAL record from being written
+ * until we have marked the buffer dirty. We don't start the
+ * checkpoint flush until we have marked dirty, so our checkpoint
+ * must flush the change to disk successfully or the checkpoint
+ * never gets written, so crash recovery will fix.
+ *
+ * It's possible we may enter here without an xid, so it is
+ * essential that CreateCheckpoint waits for virtual transactions
+ * rather than full transactionids.
+ */
+ MyProc->delayChkpt = delayChkpt = true;
+ lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ }
+
/*
- * If the block is already dirty because we either made a change
- * or set a hint already, then we don't need to write a full page
- * image. Note that aggressive cleaning of blocks dirtied by hint
- * bit setting would increase the call rate. Bulk setting of hint
- * bits would reduce the call rate...
- *
- * We must issue the WAL record before we mark the buffer dirty.
- * Otherwise we might write the page before we write the WAL. That
- * causes a race condition, since a checkpoint might occur between
- * writing the WAL record and marking the buffer dirty. We solve
- * that with a kluge, but one that is already in use during
- * transaction commit to prevent race conditions. Basically, we
- * simply prevent the checkpoint WAL record from being written
- * until we have marked the buffer dirty. We don't start the
- * checkpoint flush until we have marked dirty, so our checkpoint
- * must flush the change to disk successfully or the checkpoint
- * never gets written, so crash recovery will fix.
- *
- * It's possible we may enter here without an xid, so it is
- * essential that CreateCheckpoint waits for virtual transactions
- * rather than full transactionids.
+ * For cluster file encryption, we generate a new page LSN
+ * for hint-bit non-permanent and non-first page writes.
*/
- MyProc->delayChkpt = delayChkpt = true;
- lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ if (/* XXX file_encryption_method != 0 && */
+ XLogRecPtrIsInvalid(lsn) &&
+ /* XXX can we check BM_DIRTY without a page lock? */
+ !(pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY))
+ {
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page below.
+ */
+ lsn = XLogLSNForHint();
+ }
}
buf_state = LockBufHdr(bufHdr);
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--T4sUOijqQbZv57TR--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 18:43 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 18:43 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +-
src/backend/access/transam/xlog.c | 4 ++
src/backend/access/transam/xloginsert.c | 16 +++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 89 ++++++++++++++++--------
src/include/access/xloginsert.h | 2 +
src/include/catalog/pg_control.h | 1 +
7 files changed, 90 insertions(+), 29 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..7635a3d44d 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,22 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..b9f3f76798 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3810,13 +3810,14 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
* If we need to protect hint bit updates from torn writes, WAL-log a
* full page image of the page. This full page image is only necessary
* if the hint bit update is the first change to the page since the
- * last checkpoint.
+ * last checkpoint. If cluster file encryption is enabled, we also
+ * need to generate new page LSNs for non-first-page writes during
+ * a checkpoint.
*
* We don't check full_page_writes here because that logic is included
* when we call XLogInsert() since the value changes dynamically.
*/
- if (XLogHintBitIsNeeded() &&
- (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT))
+ if (XLogHintBitIsNeeded())
{
/*
* If we must not write WAL, due to a relfilenode-specific
@@ -3826,35 +3827,67 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
*
* See src/backend/storage/page/README for longer discussion.
*/
- if (RecoveryInProgress() ||
- RelFileNodeSkippingWAL(bufHdr->tag.rnode))
+ if (((pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ /* XXX || file_encryption_method != 0 */) &&
+ (RecoveryInProgress() ||
+ RelFileNodeSkippingWAL(bufHdr->tag.rnode)))
return;
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ {
+ /*
+ * If the block is already dirty because we either made a change
+ * or set a hint already, then we don't need to write a full page
+ * image. Note that aggressive cleaning of blocks dirtied by hint
+ * bit setting would increase the call rate. Bulk setting of hint
+ * bits would reduce the call rate...
+ *
+ * We must issue the WAL record before we mark the buffer dirty.
+ * Otherwise we might write the page before we write the WAL. That
+ * causes a race condition, since a checkpoint might occur between
+ * writing the WAL record and marking the buffer dirty. We solve
+ * that with a kluge, but one that is already in use during
+ * transaction commit to prevent race conditions. Basically, we
+ * simply prevent the checkpoint WAL record from being written
+ * until we have marked the buffer dirty. We don't start the
+ * checkpoint flush until we have marked dirty, so our checkpoint
+ * must flush the change to disk successfully or the checkpoint
+ * never gets written, so crash recovery will fix.
+ *
+ * It's possible we may enter here without an xid, so it is
+ * essential that CreateCheckpoint waits for virtual transactions
+ * rather than full transactionids.
+ */
+ MyProc->delayChkpt = delayChkpt = true;
+ lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ }
+
/*
- * If the block is already dirty because we either made a change
- * or set a hint already, then we don't need to write a full page
- * image. Note that aggressive cleaning of blocks dirtied by hint
- * bit setting would increase the call rate. Bulk setting of hint
- * bits would reduce the call rate...
- *
- * We must issue the WAL record before we mark the buffer dirty.
- * Otherwise we might write the page before we write the WAL. That
- * causes a race condition, since a checkpoint might occur between
- * writing the WAL record and marking the buffer dirty. We solve
- * that with a kluge, but one that is already in use during
- * transaction commit to prevent race conditions. Basically, we
- * simply prevent the checkpoint WAL record from being written
- * until we have marked the buffer dirty. We don't start the
- * checkpoint flush until we have marked dirty, so our checkpoint
- * must flush the change to disk successfully or the checkpoint
- * never gets written, so crash recovery will fix.
- *
- * It's possible we may enter here without an xid, so it is
- * essential that CreateCheckpoint waits for virtual transactions
- * rather than full transactionids.
+ * For cluster file encryption, we generate a new page LSN
+ * for hint-bit non-permanent and non-first page writes.
*/
- MyProc->delayChkpt = delayChkpt = true;
- lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ if (/* XXX file_encryption_method != 0 && */
+ XLogRecPtrIsInvalid(lsn) &&
+ /* XXX can we check BM_DIRTY without a page lock? */
+ !(pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY))
+ {
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page below.
+ */
+ lsn = XLogLSNForHint();
+ }
}
buf_state = LockBufHdr(bufHdr);
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--T4sUOijqQbZv57TR--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 18:43 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 18:43 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +-
src/backend/access/transam/xlog.c | 4 ++
src/backend/access/transam/xloginsert.c | 16 +++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 89 ++++++++++++++++--------
src/include/access/xloginsert.h | 2 +
src/include/catalog/pg_control.h | 1 +
7 files changed, 90 insertions(+), 29 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..7635a3d44d 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,22 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..b9f3f76798 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3810,13 +3810,14 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
* If we need to protect hint bit updates from torn writes, WAL-log a
* full page image of the page. This full page image is only necessary
* if the hint bit update is the first change to the page since the
- * last checkpoint.
+ * last checkpoint. If cluster file encryption is enabled, we also
+ * need to generate new page LSNs for non-first-page writes during
+ * a checkpoint.
*
* We don't check full_page_writes here because that logic is included
* when we call XLogInsert() since the value changes dynamically.
*/
- if (XLogHintBitIsNeeded() &&
- (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT))
+ if (XLogHintBitIsNeeded())
{
/*
* If we must not write WAL, due to a relfilenode-specific
@@ -3826,35 +3827,67 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
*
* See src/backend/storage/page/README for longer discussion.
*/
- if (RecoveryInProgress() ||
- RelFileNodeSkippingWAL(bufHdr->tag.rnode))
+ if (((pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ /* XXX || file_encryption_method != 0 */) &&
+ (RecoveryInProgress() ||
+ RelFileNodeSkippingWAL(bufHdr->tag.rnode)))
return;
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ {
+ /*
+ * If the block is already dirty because we either made a change
+ * or set a hint already, then we don't need to write a full page
+ * image. Note that aggressive cleaning of blocks dirtied by hint
+ * bit setting would increase the call rate. Bulk setting of hint
+ * bits would reduce the call rate...
+ *
+ * We must issue the WAL record before we mark the buffer dirty.
+ * Otherwise we might write the page before we write the WAL. That
+ * causes a race condition, since a checkpoint might occur between
+ * writing the WAL record and marking the buffer dirty. We solve
+ * that with a kluge, but one that is already in use during
+ * transaction commit to prevent race conditions. Basically, we
+ * simply prevent the checkpoint WAL record from being written
+ * until we have marked the buffer dirty. We don't start the
+ * checkpoint flush until we have marked dirty, so our checkpoint
+ * must flush the change to disk successfully or the checkpoint
+ * never gets written, so crash recovery will fix.
+ *
+ * It's possible we may enter here without an xid, so it is
+ * essential that CreateCheckpoint waits for virtual transactions
+ * rather than full transactionids.
+ */
+ MyProc->delayChkpt = delayChkpt = true;
+ lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ }
+
/*
- * If the block is already dirty because we either made a change
- * or set a hint already, then we don't need to write a full page
- * image. Note that aggressive cleaning of blocks dirtied by hint
- * bit setting would increase the call rate. Bulk setting of hint
- * bits would reduce the call rate...
- *
- * We must issue the WAL record before we mark the buffer dirty.
- * Otherwise we might write the page before we write the WAL. That
- * causes a race condition, since a checkpoint might occur between
- * writing the WAL record and marking the buffer dirty. We solve
- * that with a kluge, but one that is already in use during
- * transaction commit to prevent race conditions. Basically, we
- * simply prevent the checkpoint WAL record from being written
- * until we have marked the buffer dirty. We don't start the
- * checkpoint flush until we have marked dirty, so our checkpoint
- * must flush the change to disk successfully or the checkpoint
- * never gets written, so crash recovery will fix.
- *
- * It's possible we may enter here without an xid, so it is
- * essential that CreateCheckpoint waits for virtual transactions
- * rather than full transactionids.
+ * For cluster file encryption, we generate a new page LSN
+ * for hint-bit non-permanent and non-first page writes.
*/
- MyProc->delayChkpt = delayChkpt = true;
- lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ if (/* XXX file_encryption_method != 0 && */
+ XLogRecPtrIsInvalid(lsn) &&
+ /* XXX can we check BM_DIRTY without a page lock? */
+ !(pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY))
+ {
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page below.
+ */
+ lsn = XLogLSNForHint();
+ }
}
buf_state = LockBufHdr(bufHdr);
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--T4sUOijqQbZv57TR--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 18:43 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 18:43 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +-
src/backend/access/transam/xlog.c | 4 ++
src/backend/access/transam/xloginsert.c | 16 +++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 89 ++++++++++++++++--------
src/include/access/xloginsert.h | 2 +
src/include/catalog/pg_control.h | 1 +
7 files changed, 90 insertions(+), 29 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..7635a3d44d 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,22 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..b9f3f76798 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3810,13 +3810,14 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
* If we need to protect hint bit updates from torn writes, WAL-log a
* full page image of the page. This full page image is only necessary
* if the hint bit update is the first change to the page since the
- * last checkpoint.
+ * last checkpoint. If cluster file encryption is enabled, we also
+ * need to generate new page LSNs for non-first-page writes during
+ * a checkpoint.
*
* We don't check full_page_writes here because that logic is included
* when we call XLogInsert() since the value changes dynamically.
*/
- if (XLogHintBitIsNeeded() &&
- (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT))
+ if (XLogHintBitIsNeeded())
{
/*
* If we must not write WAL, due to a relfilenode-specific
@@ -3826,35 +3827,67 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
*
* See src/backend/storage/page/README for longer discussion.
*/
- if (RecoveryInProgress() ||
- RelFileNodeSkippingWAL(bufHdr->tag.rnode))
+ if (((pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ /* XXX || file_encryption_method != 0 */) &&
+ (RecoveryInProgress() ||
+ RelFileNodeSkippingWAL(bufHdr->tag.rnode)))
return;
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ {
+ /*
+ * If the block is already dirty because we either made a change
+ * or set a hint already, then we don't need to write a full page
+ * image. Note that aggressive cleaning of blocks dirtied by hint
+ * bit setting would increase the call rate. Bulk setting of hint
+ * bits would reduce the call rate...
+ *
+ * We must issue the WAL record before we mark the buffer dirty.
+ * Otherwise we might write the page before we write the WAL. That
+ * causes a race condition, since a checkpoint might occur between
+ * writing the WAL record and marking the buffer dirty. We solve
+ * that with a kluge, but one that is already in use during
+ * transaction commit to prevent race conditions. Basically, we
+ * simply prevent the checkpoint WAL record from being written
+ * until we have marked the buffer dirty. We don't start the
+ * checkpoint flush until we have marked dirty, so our checkpoint
+ * must flush the change to disk successfully or the checkpoint
+ * never gets written, so crash recovery will fix.
+ *
+ * It's possible we may enter here without an xid, so it is
+ * essential that CreateCheckpoint waits for virtual transactions
+ * rather than full transactionids.
+ */
+ MyProc->delayChkpt = delayChkpt = true;
+ lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ }
+
/*
- * If the block is already dirty because we either made a change
- * or set a hint already, then we don't need to write a full page
- * image. Note that aggressive cleaning of blocks dirtied by hint
- * bit setting would increase the call rate. Bulk setting of hint
- * bits would reduce the call rate...
- *
- * We must issue the WAL record before we mark the buffer dirty.
- * Otherwise we might write the page before we write the WAL. That
- * causes a race condition, since a checkpoint might occur between
- * writing the WAL record and marking the buffer dirty. We solve
- * that with a kluge, but one that is already in use during
- * transaction commit to prevent race conditions. Basically, we
- * simply prevent the checkpoint WAL record from being written
- * until we have marked the buffer dirty. We don't start the
- * checkpoint flush until we have marked dirty, so our checkpoint
- * must flush the change to disk successfully or the checkpoint
- * never gets written, so crash recovery will fix.
- *
- * It's possible we may enter here without an xid, so it is
- * essential that CreateCheckpoint waits for virtual transactions
- * rather than full transactionids.
+ * For cluster file encryption, we generate a new page LSN
+ * for hint-bit non-permanent and non-first page writes.
*/
- MyProc->delayChkpt = delayChkpt = true;
- lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ if (/* XXX file_encryption_method != 0 && */
+ XLogRecPtrIsInvalid(lsn) &&
+ /* XXX can we check BM_DIRTY without a page lock? */
+ !(pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY))
+ {
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page below.
+ */
+ lsn = XLogLSNForHint();
+ }
}
buf_state = LockBufHdr(bufHdr);
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--T4sUOijqQbZv57TR--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 18:43 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 18:43 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +-
src/backend/access/transam/xlog.c | 4 ++
src/backend/access/transam/xloginsert.c | 16 +++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 89 ++++++++++++++++--------
src/include/access/xloginsert.h | 2 +
src/include/catalog/pg_control.h | 1 +
7 files changed, 90 insertions(+), 29 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..7635a3d44d 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,22 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..b9f3f76798 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3810,13 +3810,14 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
* If we need to protect hint bit updates from torn writes, WAL-log a
* full page image of the page. This full page image is only necessary
* if the hint bit update is the first change to the page since the
- * last checkpoint.
+ * last checkpoint. If cluster file encryption is enabled, we also
+ * need to generate new page LSNs for non-first-page writes during
+ * a checkpoint.
*
* We don't check full_page_writes here because that logic is included
* when we call XLogInsert() since the value changes dynamically.
*/
- if (XLogHintBitIsNeeded() &&
- (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT))
+ if (XLogHintBitIsNeeded())
{
/*
* If we must not write WAL, due to a relfilenode-specific
@@ -3826,35 +3827,67 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
*
* See src/backend/storage/page/README for longer discussion.
*/
- if (RecoveryInProgress() ||
- RelFileNodeSkippingWAL(bufHdr->tag.rnode))
+ if (((pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ /* XXX || file_encryption_method != 0 */) &&
+ (RecoveryInProgress() ||
+ RelFileNodeSkippingWAL(bufHdr->tag.rnode)))
return;
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ {
+ /*
+ * If the block is already dirty because we either made a change
+ * or set a hint already, then we don't need to write a full page
+ * image. Note that aggressive cleaning of blocks dirtied by hint
+ * bit setting would increase the call rate. Bulk setting of hint
+ * bits would reduce the call rate...
+ *
+ * We must issue the WAL record before we mark the buffer dirty.
+ * Otherwise we might write the page before we write the WAL. That
+ * causes a race condition, since a checkpoint might occur between
+ * writing the WAL record and marking the buffer dirty. We solve
+ * that with a kluge, but one that is already in use during
+ * transaction commit to prevent race conditions. Basically, we
+ * simply prevent the checkpoint WAL record from being written
+ * until we have marked the buffer dirty. We don't start the
+ * checkpoint flush until we have marked dirty, so our checkpoint
+ * must flush the change to disk successfully or the checkpoint
+ * never gets written, so crash recovery will fix.
+ *
+ * It's possible we may enter here without an xid, so it is
+ * essential that CreateCheckpoint waits for virtual transactions
+ * rather than full transactionids.
+ */
+ MyProc->delayChkpt = delayChkpt = true;
+ lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ }
+
/*
- * If the block is already dirty because we either made a change
- * or set a hint already, then we don't need to write a full page
- * image. Note that aggressive cleaning of blocks dirtied by hint
- * bit setting would increase the call rate. Bulk setting of hint
- * bits would reduce the call rate...
- *
- * We must issue the WAL record before we mark the buffer dirty.
- * Otherwise we might write the page before we write the WAL. That
- * causes a race condition, since a checkpoint might occur between
- * writing the WAL record and marking the buffer dirty. We solve
- * that with a kluge, but one that is already in use during
- * transaction commit to prevent race conditions. Basically, we
- * simply prevent the checkpoint WAL record from being written
- * until we have marked the buffer dirty. We don't start the
- * checkpoint flush until we have marked dirty, so our checkpoint
- * must flush the change to disk successfully or the checkpoint
- * never gets written, so crash recovery will fix.
- *
- * It's possible we may enter here without an xid, so it is
- * essential that CreateCheckpoint waits for virtual transactions
- * rather than full transactionids.
+ * For cluster file encryption, we generate a new page LSN
+ * for hint-bit non-permanent and non-first page writes.
*/
- MyProc->delayChkpt = delayChkpt = true;
- lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ if (/* XXX file_encryption_method != 0 && */
+ XLogRecPtrIsInvalid(lsn) &&
+ /* XXX can we check BM_DIRTY without a page lock? */
+ !(pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY))
+ {
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page below.
+ */
+ lsn = XLogLSNForHint();
+ }
}
buf_state = LockBufHdr(bufHdr);
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--T4sUOijqQbZv57TR--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 18:43 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 18:43 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +-
src/backend/access/transam/xlog.c | 4 ++
src/backend/access/transam/xloginsert.c | 16 +++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 89 ++++++++++++++++--------
src/include/access/xloginsert.h | 2 +
src/include/catalog/pg_control.h | 1 +
7 files changed, 90 insertions(+), 29 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..7635a3d44d 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,22 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..b9f3f76798 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3810,13 +3810,14 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
* If we need to protect hint bit updates from torn writes, WAL-log a
* full page image of the page. This full page image is only necessary
* if the hint bit update is the first change to the page since the
- * last checkpoint.
+ * last checkpoint. If cluster file encryption is enabled, we also
+ * need to generate new page LSNs for non-first-page writes during
+ * a checkpoint.
*
* We don't check full_page_writes here because that logic is included
* when we call XLogInsert() since the value changes dynamically.
*/
- if (XLogHintBitIsNeeded() &&
- (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT))
+ if (XLogHintBitIsNeeded())
{
/*
* If we must not write WAL, due to a relfilenode-specific
@@ -3826,35 +3827,67 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
*
* See src/backend/storage/page/README for longer discussion.
*/
- if (RecoveryInProgress() ||
- RelFileNodeSkippingWAL(bufHdr->tag.rnode))
+ if (((pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ /* XXX || file_encryption_method != 0 */) &&
+ (RecoveryInProgress() ||
+ RelFileNodeSkippingWAL(bufHdr->tag.rnode)))
return;
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ {
+ /*
+ * If the block is already dirty because we either made a change
+ * or set a hint already, then we don't need to write a full page
+ * image. Note that aggressive cleaning of blocks dirtied by hint
+ * bit setting would increase the call rate. Bulk setting of hint
+ * bits would reduce the call rate...
+ *
+ * We must issue the WAL record before we mark the buffer dirty.
+ * Otherwise we might write the page before we write the WAL. That
+ * causes a race condition, since a checkpoint might occur between
+ * writing the WAL record and marking the buffer dirty. We solve
+ * that with a kluge, but one that is already in use during
+ * transaction commit to prevent race conditions. Basically, we
+ * simply prevent the checkpoint WAL record from being written
+ * until we have marked the buffer dirty. We don't start the
+ * checkpoint flush until we have marked dirty, so our checkpoint
+ * must flush the change to disk successfully or the checkpoint
+ * never gets written, so crash recovery will fix.
+ *
+ * It's possible we may enter here without an xid, so it is
+ * essential that CreateCheckpoint waits for virtual transactions
+ * rather than full transactionids.
+ */
+ MyProc->delayChkpt = delayChkpt = true;
+ lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ }
+
/*
- * If the block is already dirty because we either made a change
- * or set a hint already, then we don't need to write a full page
- * image. Note that aggressive cleaning of blocks dirtied by hint
- * bit setting would increase the call rate. Bulk setting of hint
- * bits would reduce the call rate...
- *
- * We must issue the WAL record before we mark the buffer dirty.
- * Otherwise we might write the page before we write the WAL. That
- * causes a race condition, since a checkpoint might occur between
- * writing the WAL record and marking the buffer dirty. We solve
- * that with a kluge, but one that is already in use during
- * transaction commit to prevent race conditions. Basically, we
- * simply prevent the checkpoint WAL record from being written
- * until we have marked the buffer dirty. We don't start the
- * checkpoint flush until we have marked dirty, so our checkpoint
- * must flush the change to disk successfully or the checkpoint
- * never gets written, so crash recovery will fix.
- *
- * It's possible we may enter here without an xid, so it is
- * essential that CreateCheckpoint waits for virtual transactions
- * rather than full transactionids.
+ * For cluster file encryption, we generate a new page LSN
+ * for hint-bit non-permanent and non-first page writes.
*/
- MyProc->delayChkpt = delayChkpt = true;
- lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ if (/* XXX file_encryption_method != 0 && */
+ XLogRecPtrIsInvalid(lsn) &&
+ /* XXX can we check BM_DIRTY without a page lock? */
+ !(pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY))
+ {
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page below.
+ */
+ lsn = XLogLSNForHint();
+ }
}
buf_state = LockBufHdr(bufHdr);
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--T4sUOijqQbZv57TR--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 18:43 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 18:43 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +-
src/backend/access/transam/xlog.c | 4 ++
src/backend/access/transam/xloginsert.c | 16 +++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 89 ++++++++++++++++--------
src/include/access/xloginsert.h | 2 +
src/include/catalog/pg_control.h | 1 +
7 files changed, 90 insertions(+), 29 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..7635a3d44d 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,22 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..b9f3f76798 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3810,13 +3810,14 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
* If we need to protect hint bit updates from torn writes, WAL-log a
* full page image of the page. This full page image is only necessary
* if the hint bit update is the first change to the page since the
- * last checkpoint.
+ * last checkpoint. If cluster file encryption is enabled, we also
+ * need to generate new page LSNs for non-first-page writes during
+ * a checkpoint.
*
* We don't check full_page_writes here because that logic is included
* when we call XLogInsert() since the value changes dynamically.
*/
- if (XLogHintBitIsNeeded() &&
- (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT))
+ if (XLogHintBitIsNeeded())
{
/*
* If we must not write WAL, due to a relfilenode-specific
@@ -3826,35 +3827,67 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
*
* See src/backend/storage/page/README for longer discussion.
*/
- if (RecoveryInProgress() ||
- RelFileNodeSkippingWAL(bufHdr->tag.rnode))
+ if (((pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ /* XXX || file_encryption_method != 0 */) &&
+ (RecoveryInProgress() ||
+ RelFileNodeSkippingWAL(bufHdr->tag.rnode)))
return;
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ {
+ /*
+ * If the block is already dirty because we either made a change
+ * or set a hint already, then we don't need to write a full page
+ * image. Note that aggressive cleaning of blocks dirtied by hint
+ * bit setting would increase the call rate. Bulk setting of hint
+ * bits would reduce the call rate...
+ *
+ * We must issue the WAL record before we mark the buffer dirty.
+ * Otherwise we might write the page before we write the WAL. That
+ * causes a race condition, since a checkpoint might occur between
+ * writing the WAL record and marking the buffer dirty. We solve
+ * that with a kluge, but one that is already in use during
+ * transaction commit to prevent race conditions. Basically, we
+ * simply prevent the checkpoint WAL record from being written
+ * until we have marked the buffer dirty. We don't start the
+ * checkpoint flush until we have marked dirty, so our checkpoint
+ * must flush the change to disk successfully or the checkpoint
+ * never gets written, so crash recovery will fix.
+ *
+ * It's possible we may enter here without an xid, so it is
+ * essential that CreateCheckpoint waits for virtual transactions
+ * rather than full transactionids.
+ */
+ MyProc->delayChkpt = delayChkpt = true;
+ lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ }
+
/*
- * If the block is already dirty because we either made a change
- * or set a hint already, then we don't need to write a full page
- * image. Note that aggressive cleaning of blocks dirtied by hint
- * bit setting would increase the call rate. Bulk setting of hint
- * bits would reduce the call rate...
- *
- * We must issue the WAL record before we mark the buffer dirty.
- * Otherwise we might write the page before we write the WAL. That
- * causes a race condition, since a checkpoint might occur between
- * writing the WAL record and marking the buffer dirty. We solve
- * that with a kluge, but one that is already in use during
- * transaction commit to prevent race conditions. Basically, we
- * simply prevent the checkpoint WAL record from being written
- * until we have marked the buffer dirty. We don't start the
- * checkpoint flush until we have marked dirty, so our checkpoint
- * must flush the change to disk successfully or the checkpoint
- * never gets written, so crash recovery will fix.
- *
- * It's possible we may enter here without an xid, so it is
- * essential that CreateCheckpoint waits for virtual transactions
- * rather than full transactionids.
+ * For cluster file encryption, we generate a new page LSN
+ * for hint-bit non-permanent and non-first page writes.
*/
- MyProc->delayChkpt = delayChkpt = true;
- lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ if (/* XXX file_encryption_method != 0 && */
+ XLogRecPtrIsInvalid(lsn) &&
+ /* XXX can we check BM_DIRTY without a page lock? */
+ !(pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY))
+ {
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page below.
+ */
+ lsn = XLogLSNForHint();
+ }
}
buf_state = LockBufHdr(bufHdr);
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--T4sUOijqQbZv57TR--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 18:43 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 18:43 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +-
src/backend/access/transam/xlog.c | 4 ++
src/backend/access/transam/xloginsert.c | 16 +++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 89 ++++++++++++++++--------
src/include/access/xloginsert.h | 2 +
src/include/catalog/pg_control.h | 1 +
7 files changed, 90 insertions(+), 29 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..7635a3d44d 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,22 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..b9f3f76798 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3810,13 +3810,14 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
* If we need to protect hint bit updates from torn writes, WAL-log a
* full page image of the page. This full page image is only necessary
* if the hint bit update is the first change to the page since the
- * last checkpoint.
+ * last checkpoint. If cluster file encryption is enabled, we also
+ * need to generate new page LSNs for non-first-page writes during
+ * a checkpoint.
*
* We don't check full_page_writes here because that logic is included
* when we call XLogInsert() since the value changes dynamically.
*/
- if (XLogHintBitIsNeeded() &&
- (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT))
+ if (XLogHintBitIsNeeded())
{
/*
* If we must not write WAL, due to a relfilenode-specific
@@ -3826,35 +3827,67 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
*
* See src/backend/storage/page/README for longer discussion.
*/
- if (RecoveryInProgress() ||
- RelFileNodeSkippingWAL(bufHdr->tag.rnode))
+ if (((pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ /* XXX || file_encryption_method != 0 */) &&
+ (RecoveryInProgress() ||
+ RelFileNodeSkippingWAL(bufHdr->tag.rnode)))
return;
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ {
+ /*
+ * If the block is already dirty because we either made a change
+ * or set a hint already, then we don't need to write a full page
+ * image. Note that aggressive cleaning of blocks dirtied by hint
+ * bit setting would increase the call rate. Bulk setting of hint
+ * bits would reduce the call rate...
+ *
+ * We must issue the WAL record before we mark the buffer dirty.
+ * Otherwise we might write the page before we write the WAL. That
+ * causes a race condition, since a checkpoint might occur between
+ * writing the WAL record and marking the buffer dirty. We solve
+ * that with a kluge, but one that is already in use during
+ * transaction commit to prevent race conditions. Basically, we
+ * simply prevent the checkpoint WAL record from being written
+ * until we have marked the buffer dirty. We don't start the
+ * checkpoint flush until we have marked dirty, so our checkpoint
+ * must flush the change to disk successfully or the checkpoint
+ * never gets written, so crash recovery will fix.
+ *
+ * It's possible we may enter here without an xid, so it is
+ * essential that CreateCheckpoint waits for virtual transactions
+ * rather than full transactionids.
+ */
+ MyProc->delayChkpt = delayChkpt = true;
+ lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ }
+
/*
- * If the block is already dirty because we either made a change
- * or set a hint already, then we don't need to write a full page
- * image. Note that aggressive cleaning of blocks dirtied by hint
- * bit setting would increase the call rate. Bulk setting of hint
- * bits would reduce the call rate...
- *
- * We must issue the WAL record before we mark the buffer dirty.
- * Otherwise we might write the page before we write the WAL. That
- * causes a race condition, since a checkpoint might occur between
- * writing the WAL record and marking the buffer dirty. We solve
- * that with a kluge, but one that is already in use during
- * transaction commit to prevent race conditions. Basically, we
- * simply prevent the checkpoint WAL record from being written
- * until we have marked the buffer dirty. We don't start the
- * checkpoint flush until we have marked dirty, so our checkpoint
- * must flush the change to disk successfully or the checkpoint
- * never gets written, so crash recovery will fix.
- *
- * It's possible we may enter here without an xid, so it is
- * essential that CreateCheckpoint waits for virtual transactions
- * rather than full transactionids.
+ * For cluster file encryption, we generate a new page LSN
+ * for hint-bit non-permanent and non-first page writes.
*/
- MyProc->delayChkpt = delayChkpt = true;
- lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ if (/* XXX file_encryption_method != 0 && */
+ XLogRecPtrIsInvalid(lsn) &&
+ /* XXX can we check BM_DIRTY without a page lock? */
+ !(pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY))
+ {
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page below.
+ */
+ lsn = XLogLSNForHint();
+ }
}
buf_state = LockBufHdr(bufHdr);
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--T4sUOijqQbZv57TR--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 18:43 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 18:43 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +-
src/backend/access/transam/xlog.c | 4 ++
src/backend/access/transam/xloginsert.c | 16 +++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 89 ++++++++++++++++--------
src/include/access/xloginsert.h | 2 +
src/include/catalog/pg_control.h | 1 +
7 files changed, 90 insertions(+), 29 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..7635a3d44d 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,22 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..b9f3f76798 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3810,13 +3810,14 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
* If we need to protect hint bit updates from torn writes, WAL-log a
* full page image of the page. This full page image is only necessary
* if the hint bit update is the first change to the page since the
- * last checkpoint.
+ * last checkpoint. If cluster file encryption is enabled, we also
+ * need to generate new page LSNs for non-first-page writes during
+ * a checkpoint.
*
* We don't check full_page_writes here because that logic is included
* when we call XLogInsert() since the value changes dynamically.
*/
- if (XLogHintBitIsNeeded() &&
- (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT))
+ if (XLogHintBitIsNeeded())
{
/*
* If we must not write WAL, due to a relfilenode-specific
@@ -3826,35 +3827,67 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
*
* See src/backend/storage/page/README for longer discussion.
*/
- if (RecoveryInProgress() ||
- RelFileNodeSkippingWAL(bufHdr->tag.rnode))
+ if (((pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ /* XXX || file_encryption_method != 0 */) &&
+ (RecoveryInProgress() ||
+ RelFileNodeSkippingWAL(bufHdr->tag.rnode)))
return;
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ {
+ /*
+ * If the block is already dirty because we either made a change
+ * or set a hint already, then we don't need to write a full page
+ * image. Note that aggressive cleaning of blocks dirtied by hint
+ * bit setting would increase the call rate. Bulk setting of hint
+ * bits would reduce the call rate...
+ *
+ * We must issue the WAL record before we mark the buffer dirty.
+ * Otherwise we might write the page before we write the WAL. That
+ * causes a race condition, since a checkpoint might occur between
+ * writing the WAL record and marking the buffer dirty. We solve
+ * that with a kluge, but one that is already in use during
+ * transaction commit to prevent race conditions. Basically, we
+ * simply prevent the checkpoint WAL record from being written
+ * until we have marked the buffer dirty. We don't start the
+ * checkpoint flush until we have marked dirty, so our checkpoint
+ * must flush the change to disk successfully or the checkpoint
+ * never gets written, so crash recovery will fix.
+ *
+ * It's possible we may enter here without an xid, so it is
+ * essential that CreateCheckpoint waits for virtual transactions
+ * rather than full transactionids.
+ */
+ MyProc->delayChkpt = delayChkpt = true;
+ lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ }
+
/*
- * If the block is already dirty because we either made a change
- * or set a hint already, then we don't need to write a full page
- * image. Note that aggressive cleaning of blocks dirtied by hint
- * bit setting would increase the call rate. Bulk setting of hint
- * bits would reduce the call rate...
- *
- * We must issue the WAL record before we mark the buffer dirty.
- * Otherwise we might write the page before we write the WAL. That
- * causes a race condition, since a checkpoint might occur between
- * writing the WAL record and marking the buffer dirty. We solve
- * that with a kluge, but one that is already in use during
- * transaction commit to prevent race conditions. Basically, we
- * simply prevent the checkpoint WAL record from being written
- * until we have marked the buffer dirty. We don't start the
- * checkpoint flush until we have marked dirty, so our checkpoint
- * must flush the change to disk successfully or the checkpoint
- * never gets written, so crash recovery will fix.
- *
- * It's possible we may enter here without an xid, so it is
- * essential that CreateCheckpoint waits for virtual transactions
- * rather than full transactionids.
+ * For cluster file encryption, we generate a new page LSN
+ * for hint-bit non-permanent and non-first page writes.
*/
- MyProc->delayChkpt = delayChkpt = true;
- lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ if (/* XXX file_encryption_method != 0 && */
+ XLogRecPtrIsInvalid(lsn) &&
+ /* XXX can we check BM_DIRTY without a page lock? */
+ !(pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY))
+ {
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page below.
+ */
+ lsn = XLogLSNForHint();
+ }
}
buf_state = LockBufHdr(bufHdr);
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--T4sUOijqQbZv57TR--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 18:43 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 18:43 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +-
src/backend/access/transam/xlog.c | 4 ++
src/backend/access/transam/xloginsert.c | 16 +++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 89 ++++++++++++++++--------
src/include/access/xloginsert.h | 2 +
src/include/catalog/pg_control.h | 1 +
7 files changed, 90 insertions(+), 29 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..7635a3d44d 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,22 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..b9f3f76798 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3810,13 +3810,14 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
* If we need to protect hint bit updates from torn writes, WAL-log a
* full page image of the page. This full page image is only necessary
* if the hint bit update is the first change to the page since the
- * last checkpoint.
+ * last checkpoint. If cluster file encryption is enabled, we also
+ * need to generate new page LSNs for non-first-page writes during
+ * a checkpoint.
*
* We don't check full_page_writes here because that logic is included
* when we call XLogInsert() since the value changes dynamically.
*/
- if (XLogHintBitIsNeeded() &&
- (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT))
+ if (XLogHintBitIsNeeded())
{
/*
* If we must not write WAL, due to a relfilenode-specific
@@ -3826,35 +3827,67 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
*
* See src/backend/storage/page/README for longer discussion.
*/
- if (RecoveryInProgress() ||
- RelFileNodeSkippingWAL(bufHdr->tag.rnode))
+ if (((pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ /* XXX || file_encryption_method != 0 */) &&
+ (RecoveryInProgress() ||
+ RelFileNodeSkippingWAL(bufHdr->tag.rnode)))
return;
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ {
+ /*
+ * If the block is already dirty because we either made a change
+ * or set a hint already, then we don't need to write a full page
+ * image. Note that aggressive cleaning of blocks dirtied by hint
+ * bit setting would increase the call rate. Bulk setting of hint
+ * bits would reduce the call rate...
+ *
+ * We must issue the WAL record before we mark the buffer dirty.
+ * Otherwise we might write the page before we write the WAL. That
+ * causes a race condition, since a checkpoint might occur between
+ * writing the WAL record and marking the buffer dirty. We solve
+ * that with a kluge, but one that is already in use during
+ * transaction commit to prevent race conditions. Basically, we
+ * simply prevent the checkpoint WAL record from being written
+ * until we have marked the buffer dirty. We don't start the
+ * checkpoint flush until we have marked dirty, so our checkpoint
+ * must flush the change to disk successfully or the checkpoint
+ * never gets written, so crash recovery will fix.
+ *
+ * It's possible we may enter here without an xid, so it is
+ * essential that CreateCheckpoint waits for virtual transactions
+ * rather than full transactionids.
+ */
+ MyProc->delayChkpt = delayChkpt = true;
+ lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ }
+
/*
- * If the block is already dirty because we either made a change
- * or set a hint already, then we don't need to write a full page
- * image. Note that aggressive cleaning of blocks dirtied by hint
- * bit setting would increase the call rate. Bulk setting of hint
- * bits would reduce the call rate...
- *
- * We must issue the WAL record before we mark the buffer dirty.
- * Otherwise we might write the page before we write the WAL. That
- * causes a race condition, since a checkpoint might occur between
- * writing the WAL record and marking the buffer dirty. We solve
- * that with a kluge, but one that is already in use during
- * transaction commit to prevent race conditions. Basically, we
- * simply prevent the checkpoint WAL record from being written
- * until we have marked the buffer dirty. We don't start the
- * checkpoint flush until we have marked dirty, so our checkpoint
- * must flush the change to disk successfully or the checkpoint
- * never gets written, so crash recovery will fix.
- *
- * It's possible we may enter here without an xid, so it is
- * essential that CreateCheckpoint waits for virtual transactions
- * rather than full transactionids.
+ * For cluster file encryption, we generate a new page LSN
+ * for hint-bit non-permanent and non-first page writes.
*/
- MyProc->delayChkpt = delayChkpt = true;
- lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ if (/* XXX file_encryption_method != 0 && */
+ XLogRecPtrIsInvalid(lsn) &&
+ /* XXX can we check BM_DIRTY without a page lock? */
+ !(pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY))
+ {
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page below.
+ */
+ lsn = XLogLSNForHint();
+ }
}
buf_state = LockBufHdr(bufHdr);
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--T4sUOijqQbZv57TR--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 18:43 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 18:43 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +-
src/backend/access/transam/xlog.c | 4 ++
src/backend/access/transam/xloginsert.c | 16 +++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 89 ++++++++++++++++--------
src/include/access/xloginsert.h | 2 +
src/include/catalog/pg_control.h | 1 +
7 files changed, 90 insertions(+), 29 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..7635a3d44d 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,22 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..b9f3f76798 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3810,13 +3810,14 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
* If we need to protect hint bit updates from torn writes, WAL-log a
* full page image of the page. This full page image is only necessary
* if the hint bit update is the first change to the page since the
- * last checkpoint.
+ * last checkpoint. If cluster file encryption is enabled, we also
+ * need to generate new page LSNs for non-first-page writes during
+ * a checkpoint.
*
* We don't check full_page_writes here because that logic is included
* when we call XLogInsert() since the value changes dynamically.
*/
- if (XLogHintBitIsNeeded() &&
- (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT))
+ if (XLogHintBitIsNeeded())
{
/*
* If we must not write WAL, due to a relfilenode-specific
@@ -3826,35 +3827,67 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
*
* See src/backend/storage/page/README for longer discussion.
*/
- if (RecoveryInProgress() ||
- RelFileNodeSkippingWAL(bufHdr->tag.rnode))
+ if (((pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ /* XXX || file_encryption_method != 0 */) &&
+ (RecoveryInProgress() ||
+ RelFileNodeSkippingWAL(bufHdr->tag.rnode)))
return;
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ {
+ /*
+ * If the block is already dirty because we either made a change
+ * or set a hint already, then we don't need to write a full page
+ * image. Note that aggressive cleaning of blocks dirtied by hint
+ * bit setting would increase the call rate. Bulk setting of hint
+ * bits would reduce the call rate...
+ *
+ * We must issue the WAL record before we mark the buffer dirty.
+ * Otherwise we might write the page before we write the WAL. That
+ * causes a race condition, since a checkpoint might occur between
+ * writing the WAL record and marking the buffer dirty. We solve
+ * that with a kluge, but one that is already in use during
+ * transaction commit to prevent race conditions. Basically, we
+ * simply prevent the checkpoint WAL record from being written
+ * until we have marked the buffer dirty. We don't start the
+ * checkpoint flush until we have marked dirty, so our checkpoint
+ * must flush the change to disk successfully or the checkpoint
+ * never gets written, so crash recovery will fix.
+ *
+ * It's possible we may enter here without an xid, so it is
+ * essential that CreateCheckpoint waits for virtual transactions
+ * rather than full transactionids.
+ */
+ MyProc->delayChkpt = delayChkpt = true;
+ lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ }
+
/*
- * If the block is already dirty because we either made a change
- * or set a hint already, then we don't need to write a full page
- * image. Note that aggressive cleaning of blocks dirtied by hint
- * bit setting would increase the call rate. Bulk setting of hint
- * bits would reduce the call rate...
- *
- * We must issue the WAL record before we mark the buffer dirty.
- * Otherwise we might write the page before we write the WAL. That
- * causes a race condition, since a checkpoint might occur between
- * writing the WAL record and marking the buffer dirty. We solve
- * that with a kluge, but one that is already in use during
- * transaction commit to prevent race conditions. Basically, we
- * simply prevent the checkpoint WAL record from being written
- * until we have marked the buffer dirty. We don't start the
- * checkpoint flush until we have marked dirty, so our checkpoint
- * must flush the change to disk successfully or the checkpoint
- * never gets written, so crash recovery will fix.
- *
- * It's possible we may enter here without an xid, so it is
- * essential that CreateCheckpoint waits for virtual transactions
- * rather than full transactionids.
+ * For cluster file encryption, we generate a new page LSN
+ * for hint-bit non-permanent and non-first page writes.
*/
- MyProc->delayChkpt = delayChkpt = true;
- lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ if (/* XXX file_encryption_method != 0 && */
+ XLogRecPtrIsInvalid(lsn) &&
+ /* XXX can we check BM_DIRTY without a page lock? */
+ !(pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY))
+ {
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page below.
+ */
+ lsn = XLogLSNForHint();
+ }
}
buf_state = LockBufHdr(bufHdr);
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--T4sUOijqQbZv57TR--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 18:43 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 18:43 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +-
src/backend/access/transam/xlog.c | 4 ++
src/backend/access/transam/xloginsert.c | 16 +++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 89 ++++++++++++++++--------
src/include/access/xloginsert.h | 2 +
src/include/catalog/pg_control.h | 1 +
7 files changed, 90 insertions(+), 29 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..7635a3d44d 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,22 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..b9f3f76798 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3810,13 +3810,14 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
* If we need to protect hint bit updates from torn writes, WAL-log a
* full page image of the page. This full page image is only necessary
* if the hint bit update is the first change to the page since the
- * last checkpoint.
+ * last checkpoint. If cluster file encryption is enabled, we also
+ * need to generate new page LSNs for non-first-page writes during
+ * a checkpoint.
*
* We don't check full_page_writes here because that logic is included
* when we call XLogInsert() since the value changes dynamically.
*/
- if (XLogHintBitIsNeeded() &&
- (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT))
+ if (XLogHintBitIsNeeded())
{
/*
* If we must not write WAL, due to a relfilenode-specific
@@ -3826,35 +3827,67 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
*
* See src/backend/storage/page/README for longer discussion.
*/
- if (RecoveryInProgress() ||
- RelFileNodeSkippingWAL(bufHdr->tag.rnode))
+ if (((pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ /* XXX || file_encryption_method != 0 */) &&
+ (RecoveryInProgress() ||
+ RelFileNodeSkippingWAL(bufHdr->tag.rnode)))
return;
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ {
+ /*
+ * If the block is already dirty because we either made a change
+ * or set a hint already, then we don't need to write a full page
+ * image. Note that aggressive cleaning of blocks dirtied by hint
+ * bit setting would increase the call rate. Bulk setting of hint
+ * bits would reduce the call rate...
+ *
+ * We must issue the WAL record before we mark the buffer dirty.
+ * Otherwise we might write the page before we write the WAL. That
+ * causes a race condition, since a checkpoint might occur between
+ * writing the WAL record and marking the buffer dirty. We solve
+ * that with a kluge, but one that is already in use during
+ * transaction commit to prevent race conditions. Basically, we
+ * simply prevent the checkpoint WAL record from being written
+ * until we have marked the buffer dirty. We don't start the
+ * checkpoint flush until we have marked dirty, so our checkpoint
+ * must flush the change to disk successfully or the checkpoint
+ * never gets written, so crash recovery will fix.
+ *
+ * It's possible we may enter here without an xid, so it is
+ * essential that CreateCheckpoint waits for virtual transactions
+ * rather than full transactionids.
+ */
+ MyProc->delayChkpt = delayChkpt = true;
+ lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ }
+
/*
- * If the block is already dirty because we either made a change
- * or set a hint already, then we don't need to write a full page
- * image. Note that aggressive cleaning of blocks dirtied by hint
- * bit setting would increase the call rate. Bulk setting of hint
- * bits would reduce the call rate...
- *
- * We must issue the WAL record before we mark the buffer dirty.
- * Otherwise we might write the page before we write the WAL. That
- * causes a race condition, since a checkpoint might occur between
- * writing the WAL record and marking the buffer dirty. We solve
- * that with a kluge, but one that is already in use during
- * transaction commit to prevent race conditions. Basically, we
- * simply prevent the checkpoint WAL record from being written
- * until we have marked the buffer dirty. We don't start the
- * checkpoint flush until we have marked dirty, so our checkpoint
- * must flush the change to disk successfully or the checkpoint
- * never gets written, so crash recovery will fix.
- *
- * It's possible we may enter here without an xid, so it is
- * essential that CreateCheckpoint waits for virtual transactions
- * rather than full transactionids.
+ * For cluster file encryption, we generate a new page LSN
+ * for hint-bit non-permanent and non-first page writes.
*/
- MyProc->delayChkpt = delayChkpt = true;
- lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ if (/* XXX file_encryption_method != 0 && */
+ XLogRecPtrIsInvalid(lsn) &&
+ /* XXX can we check BM_DIRTY without a page lock? */
+ !(pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY))
+ {
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page below.
+ */
+ lsn = XLogLSNForHint();
+ }
}
buf_state = LockBufHdr(bufHdr);
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--T4sUOijqQbZv57TR--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 18:43 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 18:43 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +-
src/backend/access/transam/xlog.c | 4 ++
src/backend/access/transam/xloginsert.c | 16 +++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 89 ++++++++++++++++--------
src/include/access/xloginsert.h | 2 +
src/include/catalog/pg_control.h | 1 +
7 files changed, 90 insertions(+), 29 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..7635a3d44d 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,22 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..b9f3f76798 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3810,13 +3810,14 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
* If we need to protect hint bit updates from torn writes, WAL-log a
* full page image of the page. This full page image is only necessary
* if the hint bit update is the first change to the page since the
- * last checkpoint.
+ * last checkpoint. If cluster file encryption is enabled, we also
+ * need to generate new page LSNs for non-first-page writes during
+ * a checkpoint.
*
* We don't check full_page_writes here because that logic is included
* when we call XLogInsert() since the value changes dynamically.
*/
- if (XLogHintBitIsNeeded() &&
- (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT))
+ if (XLogHintBitIsNeeded())
{
/*
* If we must not write WAL, due to a relfilenode-specific
@@ -3826,35 +3827,67 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
*
* See src/backend/storage/page/README for longer discussion.
*/
- if (RecoveryInProgress() ||
- RelFileNodeSkippingWAL(bufHdr->tag.rnode))
+ if (((pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ /* XXX || file_encryption_method != 0 */) &&
+ (RecoveryInProgress() ||
+ RelFileNodeSkippingWAL(bufHdr->tag.rnode)))
return;
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ {
+ /*
+ * If the block is already dirty because we either made a change
+ * or set a hint already, then we don't need to write a full page
+ * image. Note that aggressive cleaning of blocks dirtied by hint
+ * bit setting would increase the call rate. Bulk setting of hint
+ * bits would reduce the call rate...
+ *
+ * We must issue the WAL record before we mark the buffer dirty.
+ * Otherwise we might write the page before we write the WAL. That
+ * causes a race condition, since a checkpoint might occur between
+ * writing the WAL record and marking the buffer dirty. We solve
+ * that with a kluge, but one that is already in use during
+ * transaction commit to prevent race conditions. Basically, we
+ * simply prevent the checkpoint WAL record from being written
+ * until we have marked the buffer dirty. We don't start the
+ * checkpoint flush until we have marked dirty, so our checkpoint
+ * must flush the change to disk successfully or the checkpoint
+ * never gets written, so crash recovery will fix.
+ *
+ * It's possible we may enter here without an xid, so it is
+ * essential that CreateCheckpoint waits for virtual transactions
+ * rather than full transactionids.
+ */
+ MyProc->delayChkpt = delayChkpt = true;
+ lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ }
+
/*
- * If the block is already dirty because we either made a change
- * or set a hint already, then we don't need to write a full page
- * image. Note that aggressive cleaning of blocks dirtied by hint
- * bit setting would increase the call rate. Bulk setting of hint
- * bits would reduce the call rate...
- *
- * We must issue the WAL record before we mark the buffer dirty.
- * Otherwise we might write the page before we write the WAL. That
- * causes a race condition, since a checkpoint might occur between
- * writing the WAL record and marking the buffer dirty. We solve
- * that with a kluge, but one that is already in use during
- * transaction commit to prevent race conditions. Basically, we
- * simply prevent the checkpoint WAL record from being written
- * until we have marked the buffer dirty. We don't start the
- * checkpoint flush until we have marked dirty, so our checkpoint
- * must flush the change to disk successfully or the checkpoint
- * never gets written, so crash recovery will fix.
- *
- * It's possible we may enter here without an xid, so it is
- * essential that CreateCheckpoint waits for virtual transactions
- * rather than full transactionids.
+ * For cluster file encryption, we generate a new page LSN
+ * for hint-bit non-permanent and non-first page writes.
*/
- MyProc->delayChkpt = delayChkpt = true;
- lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ if (/* XXX file_encryption_method != 0 && */
+ XLogRecPtrIsInvalid(lsn) &&
+ /* XXX can we check BM_DIRTY without a page lock? */
+ !(pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY))
+ {
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page below.
+ */
+ lsn = XLogLSNForHint();
+ }
}
buf_state = LockBufHdr(bufHdr);
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--T4sUOijqQbZv57TR--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 18:43 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 18:43 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +-
src/backend/access/transam/xlog.c | 4 ++
src/backend/access/transam/xloginsert.c | 16 +++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 89 ++++++++++++++++--------
src/include/access/xloginsert.h | 2 +
src/include/catalog/pg_control.h | 1 +
7 files changed, 90 insertions(+), 29 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..7635a3d44d 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,22 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..b9f3f76798 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3810,13 +3810,14 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
* If we need to protect hint bit updates from torn writes, WAL-log a
* full page image of the page. This full page image is only necessary
* if the hint bit update is the first change to the page since the
- * last checkpoint.
+ * last checkpoint. If cluster file encryption is enabled, we also
+ * need to generate new page LSNs for non-first-page writes during
+ * a checkpoint.
*
* We don't check full_page_writes here because that logic is included
* when we call XLogInsert() since the value changes dynamically.
*/
- if (XLogHintBitIsNeeded() &&
- (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT))
+ if (XLogHintBitIsNeeded())
{
/*
* If we must not write WAL, due to a relfilenode-specific
@@ -3826,35 +3827,67 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
*
* See src/backend/storage/page/README for longer discussion.
*/
- if (RecoveryInProgress() ||
- RelFileNodeSkippingWAL(bufHdr->tag.rnode))
+ if (((pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ /* XXX || file_encryption_method != 0 */) &&
+ (RecoveryInProgress() ||
+ RelFileNodeSkippingWAL(bufHdr->tag.rnode)))
return;
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ {
+ /*
+ * If the block is already dirty because we either made a change
+ * or set a hint already, then we don't need to write a full page
+ * image. Note that aggressive cleaning of blocks dirtied by hint
+ * bit setting would increase the call rate. Bulk setting of hint
+ * bits would reduce the call rate...
+ *
+ * We must issue the WAL record before we mark the buffer dirty.
+ * Otherwise we might write the page before we write the WAL. That
+ * causes a race condition, since a checkpoint might occur between
+ * writing the WAL record and marking the buffer dirty. We solve
+ * that with a kluge, but one that is already in use during
+ * transaction commit to prevent race conditions. Basically, we
+ * simply prevent the checkpoint WAL record from being written
+ * until we have marked the buffer dirty. We don't start the
+ * checkpoint flush until we have marked dirty, so our checkpoint
+ * must flush the change to disk successfully or the checkpoint
+ * never gets written, so crash recovery will fix.
+ *
+ * It's possible we may enter here without an xid, so it is
+ * essential that CreateCheckpoint waits for virtual transactions
+ * rather than full transactionids.
+ */
+ MyProc->delayChkpt = delayChkpt = true;
+ lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ }
+
/*
- * If the block is already dirty because we either made a change
- * or set a hint already, then we don't need to write a full page
- * image. Note that aggressive cleaning of blocks dirtied by hint
- * bit setting would increase the call rate. Bulk setting of hint
- * bits would reduce the call rate...
- *
- * We must issue the WAL record before we mark the buffer dirty.
- * Otherwise we might write the page before we write the WAL. That
- * causes a race condition, since a checkpoint might occur between
- * writing the WAL record and marking the buffer dirty. We solve
- * that with a kluge, but one that is already in use during
- * transaction commit to prevent race conditions. Basically, we
- * simply prevent the checkpoint WAL record from being written
- * until we have marked the buffer dirty. We don't start the
- * checkpoint flush until we have marked dirty, so our checkpoint
- * must flush the change to disk successfully or the checkpoint
- * never gets written, so crash recovery will fix.
- *
- * It's possible we may enter here without an xid, so it is
- * essential that CreateCheckpoint waits for virtual transactions
- * rather than full transactionids.
+ * For cluster file encryption, we generate a new page LSN
+ * for hint-bit non-permanent and non-first page writes.
*/
- MyProc->delayChkpt = delayChkpt = true;
- lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ if (/* XXX file_encryption_method != 0 && */
+ XLogRecPtrIsInvalid(lsn) &&
+ /* XXX can we check BM_DIRTY without a page lock? */
+ !(pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY))
+ {
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page below.
+ */
+ lsn = XLogLSNForHint();
+ }
}
buf_state = LockBufHdr(bufHdr);
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--T4sUOijqQbZv57TR--
^ permalink raw reply [nested|flat] 53+ messages in thread
* [PATCH] hint squash commit
@ 2021-02-04 18:43 Bruce Momjian <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Bruce Momjian @ 2021-02-04 18:43 UTC (permalink / raw)
---
src/backend/access/rmgrdesc/xlogdesc.c | 6 +-
src/backend/access/transam/xlog.c | 4 ++
src/backend/access/transam/xloginsert.c | 16 +++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 89 ++++++++++++++++--------
src/include/access/xloginsert.h | 2 +
src/include/catalog/pg_control.h | 1 +
7 files changed, 90 insertions(+), 29 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index 92cc7ea073..16a967bb71 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_HINT_LSN)
{
/* no further information to print */
}
@@ -185,6 +186,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_HINT_LSN:
+ id = "HINT_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f03bd473e2..f67316ee07 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_HINT_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..7635a3d44d 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,22 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * On the first hint bit change during a checkpoint, XLogSaveBufferForHint()
+ * writes a full page image to the WAL and returns a new LSN which can be
+ * assigned to the page. Cluster file encryption needs a new LSN for
+ * every hint bit page write because the LSN is used in the nonce, and
+ * the nonce must be unique. Therefore, this routine just advances the LSN
+ * so the page can be assigned a new LSN.
+ */
+XLogRecPtr
+XLogLSNForHint(void)
+{
+ XLogBeginInsert();
+
+ return XLogInsert(RM_XLOG_ID, XLOG_HINT_LSN);
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index afa1df00d0..2ada89c6b1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -223,6 +223,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_HINT_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..b9f3f76798 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3810,13 +3810,14 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
* If we need to protect hint bit updates from torn writes, WAL-log a
* full page image of the page. This full page image is only necessary
* if the hint bit update is the first change to the page since the
- * last checkpoint.
+ * last checkpoint. If cluster file encryption is enabled, we also
+ * need to generate new page LSNs for non-first-page writes during
+ * a checkpoint.
*
* We don't check full_page_writes here because that logic is included
* when we call XLogInsert() since the value changes dynamically.
*/
- if (XLogHintBitIsNeeded() &&
- (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT))
+ if (XLogHintBitIsNeeded())
{
/*
* If we must not write WAL, due to a relfilenode-specific
@@ -3826,35 +3827,67 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
*
* See src/backend/storage/page/README for longer discussion.
*/
- if (RecoveryInProgress() ||
- RelFileNodeSkippingWAL(bufHdr->tag.rnode))
+ if (((pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ /* XXX || file_encryption_method != 0 */) &&
+ (RecoveryInProgress() ||
+ RelFileNodeSkippingWAL(bufHdr->tag.rnode)))
return;
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)
+ {
+ /*
+ * If the block is already dirty because we either made a change
+ * or set a hint already, then we don't need to write a full page
+ * image. Note that aggressive cleaning of blocks dirtied by hint
+ * bit setting would increase the call rate. Bulk setting of hint
+ * bits would reduce the call rate...
+ *
+ * We must issue the WAL record before we mark the buffer dirty.
+ * Otherwise we might write the page before we write the WAL. That
+ * causes a race condition, since a checkpoint might occur between
+ * writing the WAL record and marking the buffer dirty. We solve
+ * that with a kluge, but one that is already in use during
+ * transaction commit to prevent race conditions. Basically, we
+ * simply prevent the checkpoint WAL record from being written
+ * until we have marked the buffer dirty. We don't start the
+ * checkpoint flush until we have marked dirty, so our checkpoint
+ * must flush the change to disk successfully or the checkpoint
+ * never gets written, so crash recovery will fix.
+ *
+ * It's possible we may enter here without an xid, so it is
+ * essential that CreateCheckpoint waits for virtual transactions
+ * rather than full transactionids.
+ */
+ MyProc->delayChkpt = delayChkpt = true;
+ lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ }
+
/*
- * If the block is already dirty because we either made a change
- * or set a hint already, then we don't need to write a full page
- * image. Note that aggressive cleaning of blocks dirtied by hint
- * bit setting would increase the call rate. Bulk setting of hint
- * bits would reduce the call rate...
- *
- * We must issue the WAL record before we mark the buffer dirty.
- * Otherwise we might write the page before we write the WAL. That
- * causes a race condition, since a checkpoint might occur between
- * writing the WAL record and marking the buffer dirty. We solve
- * that with a kluge, but one that is already in use during
- * transaction commit to prevent race conditions. Basically, we
- * simply prevent the checkpoint WAL record from being written
- * until we have marked the buffer dirty. We don't start the
- * checkpoint flush until we have marked dirty, so our checkpoint
- * must flush the change to disk successfully or the checkpoint
- * never gets written, so crash recovery will fix.
- *
- * It's possible we may enter here without an xid, so it is
- * essential that CreateCheckpoint waits for virtual transactions
- * rather than full transactionids.
+ * For cluster file encryption, we generate a new page LSN
+ * for hint-bit non-permanent and non-first page writes.
*/
- MyProc->delayChkpt = delayChkpt = true;
- lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ if (/* XXX file_encryption_method != 0 && */
+ XLogRecPtrIsInvalid(lsn) &&
+ /* XXX can we check BM_DIRTY without a page lock? */
+ !(pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY))
+ {
+ /*
+ * If the buffer is clean, and lsn is valid, it must be the
+ * first hint bit change of the checkpoint (the old page lsn
+ * is earlier than the RedoRecPtr). If the page is clean and
+ * the lsn is invalid, the page must have been already written
+ * during this checkpoint, and this is a new hint bit change.
+ * Such page changes usually don't create new page lsns, but we
+ * need one for cluster file encryption because the lsn is used as
+ * part of the nonce, and we don't want to reencrypt the page
+ * with the hint bit changes using the same nonce as the previous
+ * writes during this checkpoint. (It would expose the hint bit
+ * change locations.) Therefore we create a simple WAL record
+ * to advance the lsn, which can can then be assigned to the
+ * page below.
+ */
+ lsn = XLogLSNForHint();
+ }
}
buf_state = LockBufHdr(bufHdr);
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..a066f65229 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr XLogLSNForHint(void);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..36ac7dfbb8 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_HINT_LSN 0xC0
/*
--
2.20.1
--T4sUOijqQbZv57TR--
^ permalink raw reply [nested|flat] 53+ messages in thread
* Re: Estimating HugePages Requirements?
@ 2022-03-15 02:41 Michael Paquier <[email protected]>
0 siblings, 0 replies; 53+ messages in thread
From: Michael Paquier @ 2022-03-15 02:41 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Magnus Hagander <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; Justin Pryzby <[email protected]>; Andres Freund <[email protected]>; Mark Dilger <[email protected]>; Don Seiler <[email protected]>; pgsql-hackers
On Mon, Mar 14, 2022 at 10:34:17AM -0700, Nathan Bossart wrote:
> On Mon, Mar 14, 2022 at 04:26:43PM +0100, Magnus Hagander wrote:
>> And in fact, the command documented on
>> https://www.postgresql.org/docs/devel/kernel-resources.html doesn't
>> actually produce the output that the docs show, it also shows the log
>> line, in the default config? If we can't fix the extra logging we
>> should at least have our docs represent reality -- maybe by adding a
>> "2>/dev/null" entry? But it'd be better to have it not output that log
>> in the first place...
>
> I attached a patch to adjust the documentation for now. This apparently
> crossed my mind earlier [0], but I didn't follow through with it for some
> reason.
Another thing that we can add is -c log_min_messages=fatal, but my
method is more complicated than a simple redirection, of course :)
>> (Of course what I'd really want is to be able to run it on a cluster
>> that's running, but that was discussed downthread so I'm not bringing
>> that one up for changes now)
>
> I think it is worth revisiting the extra logging and the ability to view
> runtime-computed GUCs on a running server. Should this be an open item for
> v15, or do you think it's alright to leave it for the v16 development
> cycle?
Well, this is a completely new problem as it opens the door of
potential concurrent access issues with the data directory lock file
while reading values from the control file. And that's not mandatory
to be able to get those estimations without having to allocate a large
chunk of memory, which was the primary goal discussed upthread as far
as I recall. So I would leave that as an item to potentially tackle
in future versions.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../Yi%[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 53+ messages in thread
end of thread, other threads:[~2022-03-15 02:41 UTC | newest]
Thread overview: 53+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2017-04-15 02:47 Logical replication - TRAP: FailedAssertion in pgstat.c Erik Rijkers <[email protected]>
2017-04-16 08:46 ` Erik Rijkers <[email protected]>
2017-04-16 18:41 ` Andres Freund <[email protected]>
2017-04-16 20:11 ` Petr Jelinek <[email protected]>
2017-04-20 18:58 ` Peter Eisentraut <[email protected]>
2017-04-17 07:30 ` Erik Rijkers <[email protected]>
2017-04-17 13:59 ` Stas Kelvich <[email protected]>
2017-04-17 19:16 ` Erik Rijkers <[email protected]>
2021-02-04 03:25 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 03:25 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 03:25 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 03:25 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 03:25 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 03:25 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 03:25 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 03:25 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 03:25 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 03:25 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 03:25 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 03:25 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 03:25 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 03:25 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 03:25 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 03:25 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 03:25 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 03:25 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 03:25 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 03:25 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 03:25 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 03:25 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 18:43 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 18:43 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 18:43 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 18:43 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 18:43 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 18:43 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 18:43 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 18:43 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 18:43 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 18:43 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 18:43 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 18:43 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 18:43 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 18:43 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 18:43 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 18:43 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 18:43 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 18:43 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 18:43 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 18:43 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 18:43 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2021-02-04 18:43 [PATCH] hint squash commit Bruce Momjian <[email protected]>
2022-03-15 02:41 Re: Estimating HugePages Requirements? Michael Paquier <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox