public inbox for [email protected]
help / color / mirror / Atom feedFrom: Bharath Rupireddy <[email protected]>
To: Kyotaro Horiguchi <[email protected]>
Cc: Andres Freund <[email protected]>
Cc: Ashutosh Sharma <[email protected]>
Cc: Stephen Frost <[email protected]>
Cc: Jeff Davis <[email protected]>
Cc: Robert Haas <[email protected]>
Cc: Andrew Dunstan <[email protected]>
Cc: Greg Stark <[email protected]>
Cc: Jeremy Schneider <[email protected]>
Cc: Bruce Momjian <[email protected]>
Cc: PostgreSQL Hackers <[email protected]>
Cc: SATYANARAYANA NARLAPURAM <[email protected]>
Cc: [email protected]
Cc: [email protected]
Subject: Re: pg_walinspect - a new extension to get raw WAL data and WAL stats
Date: Thu, 24 Mar 2022 15:02:29 +0530
Message-ID: <CALj2ACWHxU2brc1vRJKxmtgr6hAB6YnL9fjVdN_qEvv1HiWLFQ@mail.gmail.com> (raw)
In-Reply-To: <[email protected]>
References: <CALj2ACUvU2fGLw7keEpxZhGAoMQ9vrCPX-13hexQPoR+QRbuOw@mail.gmail.com>
<[email protected]>
<CALj2ACUEbx7B_FJbyJxF_NxqmgPYXkNo3czZAv1iVBRKzAP3XA@mail.gmail.com>
<[email protected]>
On Thu, Mar 24, 2022 at 10:22 AM Kyotaro Horiguchi
<[email protected]> wrote:
> +void
> +XLogRecGetBlockRefInfo(XLogReaderState *record, char *delimiter,
> + uint32 *fpi_len, bool detailed_format,
> + StringInfo buf)
> ...
> + if (detailed_format && delimiter)
> + appendStringInfoChar(buf, '\n');
>
> It is odd that the variable "delimiter" is used as a bool in the
> function, though it is a "char *", which I meant that it is used as
> delimiter string (assuming that you might want to insert ", " between
> two blkref descriptions).
I'm passing NULL if the delimiter isn't required (for pg_walinspect)
and I wanted to check if it's passed, so I was using the delimiter in
the condition. However, I now changed it to delimiter != NULL.
> +get_forkname(ForkNumber num)
>
> forkNames[] is public and used in reinit.c. I think we don't need
> this function.
Yes. I removed it.
> +#define MAX_XLINFO_TYPES 16
> ...
> + XLogRecStats rmgr_stats[RM_NEXT_ID];
> + XLogRecStats record_stats[RM_NEXT_ID][MAX_XLINFO_TYPES];
> +} XLogStats;
> +
>
> This doesn't seem to be a part of xlogreader. Couldn't we add a new
> module "xlogstats"? XLogRecGetBlockRefInfo also doesn't seem to me as
> a part of xlogreader, the xlogstats looks like a better place.
I'm not sure if it's worth adding new files xlogstats.h/.c just for 2
structures, 1 macro, and 2 functions with no plan to add new stats
structures or functions. Since xlogreader is the one that reads the
WAL, and is being included by both backend and other modules (tools
and extensions) IMO it's the right place. However, I can specify in
xlogreader that if at all the new stats related structures or
functions are going to be added, it's good to move them into a new
header and .c file.
Thoughts?
> +#define XLOG_GET_STATS_PERCENTAGE(n_pct, rec_len_pct, fpi_len_pct, \
> + tot_len_pct, total_count, \
>
> It doesn't need to be a macro. However in the first place I don't
> think it is useful to have. Rather it may be harmful since it doesn't
> reduce complexity much but instead just hides details. If we want to
> avoid tedious repetitions of the same statements, a macro like the
> following may work.
>
> #define CALC_PCT (num, denom) ((denom) == 0 ? 0.0 ? 100.0 * (num) / (denom))
> ...
> > n_pct = CALC_PCT(n, total_count);
> > rec_len_pct = CALC_PCT(rec_len, total_rec_len);
> > fpi_len_pct = CALC_PCT(fpi_len, total_fpi_len);
> > tot_len_pct = CALC_PCT(tot_len, total_len);
>
> But it is not seem that different if we directly write out the detail.
>
> > n_pct = (total_count == 0 ? 0 : 100.0 * n / total_count);
> > rec_len_pct = (total_rec_len == 0 ? 0 : 100.0 * rec_len / total_rec_len);
> > fpi_len_pct = (total_fpi_len == 0 ? 0 : 100.0 * fpi_len / total_fpi_len);
> > tot_len_pct = (total_len == 0 ? 0 : 100.0 * tot_len / total_len);
I removed the XLOG_GET_STATS_PERCENTAGE macro.
Attaching v14 patch-set here with. It has bunch of other changes along
with the above:
1) Used STRICT for all the functions and introduced _till_end_of_wal
versions for pg_get_wal_records_info and pg_get_wal_stats.
2) Most of the code is reused between pg_walinspect and pg_waldump and
also within pg_walinspect.
3) Added read_local_xlog_page_no_wait without duplicating the code so
that the pg_walinspect functions don't wait even while finding the
first valid WAL record.
4) No function waits for future WAL lsn even to find the first valid WAL record.
5) Addressed the review comments raised upthread by Andres.
I hope this version makes the patch cleaner, please review it further.
Regards,
Bharath Rupireddy.
Attachments:
[application/octet-stream] v14-0001-Refactor-pg_waldump-code.patch (15.3K, ../CALj2ACWHxU2brc1vRJKxmtgr6hAB6YnL9fjVdN_qEvv1HiWLFQ@mail.gmail.com/2-v14-0001-Refactor-pg_waldump-code.patch)
download | inline diff:
From f38e8ee17cf2b6e413ae6c2ffd8400c26457180d Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <[email protected]>
Date: Thu, 24 Mar 2022 09:23:15 +0000
Subject: [PATCH v14] Refactor pg_waldump code
This patch puts some generic chunks of pg_waldump's code
into separate reusable functions in xlogreader.h/.c.
---
src/backend/access/transam/xlogreader.c | 202 ++++++++++++++++++++++++
src/bin/pg_waldump/pg_waldump.c | 197 ++---------------------
src/include/access/xlogreader.h | 28 ++++
3 files changed, 240 insertions(+), 187 deletions(-)
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index e437c42992..5909ec180e 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -1950,6 +1950,208 @@ XLogRecGetBlockTag(XLogReaderState *record, uint8 block_id,
return true;
}
+/*
+ * Calculate the size of a record, split into !FPI and FPI parts.
+ */
+void
+XLogRecGetLen(XLogReaderState *record, uint32 *rec_len,
+ uint32 *fpi_len)
+{
+ int block_id;
+
+ /*
+ * Calculate the amount of FPI data in the record.
+ *
+ * XXX: We peek into xlogreader's private decoded backup blocks for the
+ * bimg_len indicating the length of FPI data.
+ */
+ *fpi_len = 0;
+ for (block_id = 0; block_id <= XLogRecMaxBlockId(record); block_id++)
+ {
+ if (XLogRecHasBlockImage(record, block_id))
+ *fpi_len += XLogRecGetBlock(record, block_id)->bimg_len;
+ }
+
+ /*
+ * Calculate the length of the record as the total length - the length of
+ * all the block images.
+ */
+ *rec_len = XLogRecGetTotalLen(record) - *fpi_len;
+}
+
+/*
+ * Returns a string giving information about all the blocks in an
+ * XLogRecord.
+ */
+void
+XLogRecGetBlockRefInfo(XLogReaderState *record, char *delimiter,
+ uint32 *fpi_len, bool detailed_format,
+ StringInfo buf)
+{
+ int block_id;
+
+ Assert(record != NULL);
+
+ if (detailed_format && delimiter != NULL)
+ appendStringInfoChar(buf, '\n');
+
+ for (block_id = 0; block_id <= XLogRecMaxBlockId(record); block_id++)
+ {
+ RelFileNode rnode = {InvalidOid, InvalidOid, InvalidOid};
+ ForkNumber forknum = InvalidForkNumber;
+ BlockNumber blk = InvalidBlockNumber;
+
+ if (!XLogRecHasBlockRef(record, block_id))
+ continue;
+
+ XLogRecGetBlockTag(record, block_id, &rnode, &forknum, &blk);
+
+ if (detailed_format)
+ {
+ /* Get block references in detailed format. */
+
+ appendStringInfo(buf,
+ "\tblkref #%d: rel %u/%u/%u fork %s blk %u",
+ block_id,
+ rnode.spcNode, rnode.dbNode, rnode.relNode,
+ forkNames[forknum],
+ blk);
+
+ if (XLogRecHasBlockImage(record, block_id))
+ {
+ uint8 bimg_info = XLogRecGetBlock(record, block_id)->bimg_info;
+
+ /* Calculate the amount of FPI data in the record. */
+ if (fpi_len)
+ *fpi_len += XLogRecGetBlock(record, block_id)->bimg_len;
+
+ if (BKPIMAGE_COMPRESSED(bimg_info))
+ {
+ const char *method;
+
+ if ((bimg_info & BKPIMAGE_COMPRESS_PGLZ) != 0)
+ method = "pglz";
+ else if ((bimg_info & BKPIMAGE_COMPRESS_LZ4) != 0)
+ method = "lz4";
+ else if ((bimg_info & BKPIMAGE_COMPRESS_ZSTD) != 0)
+ method = "zstd";
+ else
+ method = "unknown";
+
+ appendStringInfo(buf,
+ " (FPW%s); hole: offset: %u, length: %u, "
+ "compression saved: %u, method: %s",
+ XLogRecBlockImageApply(record, block_id) ?
+ "" : " for WAL verification",
+ XLogRecGetBlock(record, block_id)->hole_offset,
+ XLogRecGetBlock(record, block_id)->hole_length,
+ BLCKSZ -
+ XLogRecGetBlock(record, block_id)->hole_length -
+ XLogRecGetBlock(record, block_id)->bimg_len,
+ method);
+ }
+ else
+ {
+ appendStringInfo(buf,
+ " (FPW%s); hole: offset: %u, length: %u",
+ XLogRecBlockImageApply(record, block_id) ?
+ "" : " for WAL verification",
+ XLogRecGetBlock(record, block_id)->hole_offset,
+ XLogRecGetBlock(record, block_id)->hole_length);
+ }
+ }
+ }
+ else
+ {
+ /* Get block references in short format. */
+
+ if (forknum != MAIN_FORKNUM)
+ {
+ appendStringInfo(buf,
+ ", blkref #%d: rel %u/%u/%u fork %s blk %u",
+ block_id,
+ rnode.spcNode, rnode.dbNode, rnode.relNode,
+ forkNames[forknum],
+ blk);
+ }
+ else
+ {
+ appendStringInfo(buf,
+ ", blkref #%d: rel %u/%u/%u blk %u",
+ block_id,
+ rnode.spcNode, rnode.dbNode, rnode.relNode,
+ blk);
+ }
+
+ if (XLogRecHasBlockImage(record, block_id))
+ {
+ /* Calculate the amount of FPI data in the record. */
+ if (fpi_len)
+ *fpi_len += XLogRecGetBlock(record, block_id)->bimg_len;
+
+ if (XLogRecBlockImageApply(record, block_id))
+ appendStringInfo(buf, " FPW");
+ else
+ appendStringInfo(buf, " FPW for WAL verification");
+ }
+ }
+
+ if (detailed_format && delimiter != NULL)
+ appendStringInfoChar(buf, '\n');
+ }
+
+ if (!detailed_format && delimiter != NULL)
+ appendStringInfoChar(buf, '\n');
+}
+
+/*
+ * Store per-rmgr and per-record statistics for a given record.
+ */
+void
+XLogRecStoreStats(XLogStats *stats, XLogReaderState *record)
+{
+ RmgrId rmid;
+ uint8 recid;
+ uint32 rec_len;
+ uint32 fpi_len;
+
+ Assert(stats != NULL && record != NULL);
+
+ stats->count++;
+
+ rmid = XLogRecGetRmid(record);
+
+ XLogRecGetLen(record, &rec_len, &fpi_len);
+
+ /* Update per-rmgr statistics */
+
+ stats->rmgr_stats[rmid].count++;
+ stats->rmgr_stats[rmid].rec_len += rec_len;
+ stats->rmgr_stats[rmid].fpi_len += fpi_len;
+
+ /*
+ * Update per-record statistics, where the record is identified by a
+ * combination of the RmgrId and the four bits of the xl_info field that
+ * are the rmgr's domain (resulting in sixteen possible entries per
+ * RmgrId).
+ */
+
+ recid = XLogRecGetInfo(record) >> 4;
+
+ /*
+ * XACT records need to be handled differently. Those records use the
+ * first bit of those four bits for an optional flag variable and the
+ * following three bits for the opcode. We filter opcode out of xl_info
+ * and use it as the identifier of the record.
+ */
+ if (rmid == RM_XACT_ID)
+ recid &= 0x07;
+
+ stats->record_stats[rmid][recid].count++;
+ stats->record_stats[rmid][recid].rec_len += rec_len;
+ stats->record_stats[rmid][recid].fpi_len += fpi_len;
+}
+
/*
* Returns the data associated with a block reference, or NULL if there is
* no data (e.g. because a full-page image was taken instead). The returned
diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c
index 92238f30c9..179cd617e4 100644
--- a/src/bin/pg_waldump/pg_waldump.c
+++ b/src/bin/pg_waldump/pg_waldump.c
@@ -66,23 +66,6 @@ typedef struct XLogDumpConfig
bool filter_by_fpw;
} XLogDumpConfig;
-typedef struct Stats
-{
- uint64 count;
- uint64 rec_len;
- uint64 fpi_len;
-} Stats;
-
-#define MAX_XLINFO_TYPES 16
-
-typedef struct XLogDumpStats
-{
- uint64 count;
- XLogRecPtr startptr;
- XLogRecPtr endptr;
- Stats rmgr_stats[RM_NEXT_ID];
- Stats record_stats[RM_NEXT_ID][MAX_XLINFO_TYPES];
-} XLogDumpStats;
#define fatal_error(...) do { pg_log_fatal(__VA_ARGS__); exit(EXIT_FAILURE); } while(0)
@@ -453,81 +436,6 @@ XLogRecordHasFPW(XLogReaderState *record)
return false;
}
-/*
- * Calculate the size of a record, split into !FPI and FPI parts.
- */
-static void
-XLogDumpRecordLen(XLogReaderState *record, uint32 *rec_len, uint32 *fpi_len)
-{
- int block_id;
-
- /*
- * Calculate the amount of FPI data in the record.
- *
- * XXX: We peek into xlogreader's private decoded backup blocks for the
- * bimg_len indicating the length of FPI data.
- */
- *fpi_len = 0;
- for (block_id = 0; block_id <= XLogRecMaxBlockId(record); block_id++)
- {
- if (XLogRecHasBlockImage(record, block_id))
- *fpi_len += XLogRecGetBlock(record, block_id)->bimg_len;
- }
-
- /*
- * Calculate the length of the record as the total length - the length of
- * all the block images.
- */
- *rec_len = XLogRecGetTotalLen(record) - *fpi_len;
-}
-
-/*
- * Store per-rmgr and per-record statistics for a given record.
- */
-static void
-XLogDumpCountRecord(XLogDumpConfig *config, XLogDumpStats *stats,
- XLogReaderState *record)
-{
- RmgrId rmid;
- uint8 recid;
- uint32 rec_len;
- uint32 fpi_len;
-
- stats->count++;
-
- rmid = XLogRecGetRmid(record);
-
- XLogDumpRecordLen(record, &rec_len, &fpi_len);
-
- /* Update per-rmgr statistics */
-
- stats->rmgr_stats[rmid].count++;
- stats->rmgr_stats[rmid].rec_len += rec_len;
- stats->rmgr_stats[rmid].fpi_len += fpi_len;
-
- /*
- * Update per-record statistics, where the record is identified by a
- * combination of the RmgrId and the four bits of the xl_info field that
- * are the rmgr's domain (resulting in sixteen possible entries per
- * RmgrId).
- */
-
- recid = XLogRecGetInfo(record) >> 4;
-
- /*
- * XACT records need to be handled differently. Those records use the
- * first bit of those four bits for an optional flag variable and the
- * following three bits for the opcode. We filter opcode out of xl_info
- * and use it as the identifier of the record.
- */
- if (rmid == RM_XACT_ID)
- recid &= 0x07;
-
- stats->record_stats[rmid][recid].count++;
- stats->record_stats[rmid][recid].rec_len += rec_len;
- stats->record_stats[rmid][recid].fpi_len += fpi_len;
-}
-
/*
* Print a record to stdout
*/
@@ -538,15 +446,12 @@ XLogDumpDisplayRecord(XLogDumpConfig *config, XLogReaderState *record)
const RmgrDescData *desc = &RmgrDescTable[XLogRecGetRmid(record)];
uint32 rec_len;
uint32 fpi_len;
- RelFileNode rnode;
- ForkNumber forknum;
- BlockNumber blk;
- int block_id;
uint8 info = XLogRecGetInfo(record);
XLogRecPtr xl_prev = XLogRecGetPrev(record);
StringInfoData s;
+ char delim = {'\n'};
- XLogDumpRecordLen(record, &rec_len, &fpi_len);
+ XLogRecGetLen(record, &rec_len, &fpi_len);
printf("rmgr: %-11s len (rec/tot): %6u/%6u, tx: %10u, lsn: %X/%08X, prev %X/%08X, ",
desc->rm_name,
@@ -564,93 +469,11 @@ XLogDumpDisplayRecord(XLogDumpConfig *config, XLogReaderState *record)
initStringInfo(&s);
desc->rm_desc(&s, record);
printf("%s", s.data);
- pfree(s.data);
-
- if (!config->bkp_details)
- {
- /* print block references (short format) */
- for (block_id = 0; block_id <= XLogRecMaxBlockId(record); block_id++)
- {
- if (!XLogRecHasBlockRef(record, block_id))
- continue;
-
- XLogRecGetBlockTag(record, block_id, &rnode, &forknum, &blk);
- if (forknum != MAIN_FORKNUM)
- printf(", blkref #%d: rel %u/%u/%u fork %s blk %u",
- block_id,
- rnode.spcNode, rnode.dbNode, rnode.relNode,
- forkNames[forknum],
- blk);
- else
- printf(", blkref #%d: rel %u/%u/%u blk %u",
- block_id,
- rnode.spcNode, rnode.dbNode, rnode.relNode,
- blk);
- if (XLogRecHasBlockImage(record, block_id))
- {
- if (XLogRecBlockImageApply(record, block_id))
- printf(" FPW");
- else
- printf(" FPW for WAL verification");
- }
- }
- putchar('\n');
- }
- else
- {
- /* print block references (detailed format) */
- putchar('\n');
- for (block_id = 0; block_id <= XLogRecMaxBlockId(record); block_id++)
- {
- if (!XLogRecHasBlockRef(record, block_id))
- continue;
-
- XLogRecGetBlockTag(record, block_id, &rnode, &forknum, &blk);
- printf("\tblkref #%d: rel %u/%u/%u fork %s blk %u",
- block_id,
- rnode.spcNode, rnode.dbNode, rnode.relNode,
- forkNames[forknum],
- blk);
- if (XLogRecHasBlockImage(record, block_id))
- {
- uint8 bimg_info = XLogRecGetBlock(record, block_id)->bimg_info;
- if (BKPIMAGE_COMPRESSED(bimg_info))
- {
- const char *method;
-
- if ((bimg_info & BKPIMAGE_COMPRESS_PGLZ) != 0)
- method = "pglz";
- else if ((bimg_info & BKPIMAGE_COMPRESS_LZ4) != 0)
- method = "lz4";
- else if ((bimg_info & BKPIMAGE_COMPRESS_ZSTD) != 0)
- method = "zstd";
- else
- method = "unknown";
-
- printf(" (FPW%s); hole: offset: %u, length: %u, "
- "compression saved: %u, method: %s",
- XLogRecBlockImageApply(record, block_id) ?
- "" : " for WAL verification",
- XLogRecGetBlock(record, block_id)->hole_offset,
- XLogRecGetBlock(record, block_id)->hole_length,
- BLCKSZ -
- XLogRecGetBlock(record, block_id)->hole_length -
- XLogRecGetBlock(record, block_id)->bimg_len,
- method);
- }
- else
- {
- printf(" (FPW%s); hole: offset: %u, length: %u",
- XLogRecBlockImageApply(record, block_id) ?
- "" : " for WAL verification",
- XLogRecGetBlock(record, block_id)->hole_offset,
- XLogRecGetBlock(record, block_id)->hole_length);
- }
- }
- putchar('\n');
- }
- }
+ resetStringInfo(&s);
+ XLogRecGetBlockRefInfo(record, &delim, NULL, config->bkp_details, &s);
+ printf("%s", s.data);
+ pfree(s.data);
}
/*
@@ -698,7 +521,7 @@ XLogDumpStatsRow(const char *name,
* Display summary statistics about the records seen so far.
*/
static void
-XLogDumpDisplayStats(XLogDumpConfig *config, XLogDumpStats *stats)
+XLogDumpDisplayStats(XLogDumpConfig *config, XLogStats *stats)
{
int ri,
rj;
@@ -859,7 +682,7 @@ main(int argc, char **argv)
XLogReaderState *xlogreader_state;
XLogDumpPrivate private;
XLogDumpConfig config;
- XLogDumpStats stats;
+ XLogStats stats;
XLogRecord *record;
XLogRecPtr first_record;
char *waldir = NULL;
@@ -913,7 +736,7 @@ main(int argc, char **argv)
memset(&private, 0, sizeof(XLogDumpPrivate));
memset(&config, 0, sizeof(XLogDumpConfig));
- memset(&stats, 0, sizeof(XLogDumpStats));
+ memset(&stats, 0, sizeof(XLogStats));
private.timeline = 1;
private.startptr = InvalidXLogRecPtr;
@@ -1295,7 +1118,7 @@ main(int argc, char **argv)
{
if (config.stats == true)
{
- XLogDumpCountRecord(&config, &stats, xlogreader_state);
+ XLogRecStoreStats(&stats, xlogreader_state);
stats.endptr = xlogreader_state->EndRecPtr;
}
else
diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h
index f4388cc9be..4021ae05df 100644
--- a/src/include/access/xlogreader.h
+++ b/src/include/access/xlogreader.h
@@ -39,6 +39,7 @@
#endif
#include "access/xlogrecord.h"
+#include "lib/stringinfo.h"
/* WALOpenSegment represents a WAL segment being read. */
typedef struct WALOpenSegment
@@ -314,6 +315,26 @@ struct XLogReaderState
bool nonblocking;
};
+#define MAX_XLINFO_TYPES 16
+
+typedef struct XLogRecStats
+{
+ uint64 count;
+ uint64 rec_len;
+ uint64 fpi_len;
+} XLogRecStats;
+
+typedef struct XLogStats
+{
+ uint64 count;
+#ifdef FRONTEND
+ XLogRecPtr startptr;
+ XLogRecPtr endptr;
+#endif
+ XLogRecStats rmgr_stats[RM_NEXT_ID];
+ XLogRecStats record_stats[RM_NEXT_ID][MAX_XLINFO_TYPES];
+} XLogStats;
+
/*
* Check if XLogNextRecord() has any more queued records or an error to return.
*/
@@ -430,5 +451,12 @@ extern char *XLogRecGetBlockData(XLogReaderState *record, uint8 block_id, Size *
extern bool XLogRecGetBlockTag(XLogReaderState *record, uint8 block_id,
RelFileNode *rnode, ForkNumber *forknum,
BlockNumber *blknum);
+extern void XLogRecGetLen(XLogReaderState *record, uint32 *rec_len,
+ uint32 *fpi_len);
+extern void XLogRecGetBlockRefInfo(XLogReaderState *record,
+ char *delimiter, uint32 *fpi_len,
+ bool detailed_format,
+ StringInfo blk_ref);
+extern void XLogRecStoreStats(XLogStats *stats, XLogReaderState *record);
#endif /* XLOGREADER_H */
--
2.25.1
[application/octet-stream] v14-0002-pg_walinspect.patch (30.1K, ../CALj2ACWHxU2brc1vRJKxmtgr6hAB6YnL9fjVdN_qEvv1HiWLFQ@mail.gmail.com/3-v14-0002-pg_walinspect.patch)
download | inline diff:
From 302ccd68143ea08e9df83c76ae95a7d4d388ee8d Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <[email protected]>
Date: Thu, 24 Mar 2022 09:24:57 +0000
Subject: [PATCH v14] pg_walinspect
---
contrib/Makefile | 1 +
contrib/pg_walinspect/.gitignore | 4 +
contrib/pg_walinspect/Makefile | 26 +
contrib/pg_walinspect/pg_walinspect--1.0.sql | 132 ++++
contrib/pg_walinspect/pg_walinspect.c | 652 +++++++++++++++++++
contrib/pg_walinspect/pg_walinspect.control | 5 +
src/backend/access/transam/xlogreader.c | 13 +-
src/backend/access/transam/xlogutils.c | 33 +
src/bin/pg_waldump/pg_waldump.c | 5 +
src/include/access/xlog.h | 2 +-
src/include/access/xlog_internal.h | 2 +-
src/include/access/xlogreader.h | 2 -
src/include/access/xlogutils.h | 4 +
13 files changed, 870 insertions(+), 11 deletions(-)
create mode 100644 contrib/pg_walinspect/.gitignore
create mode 100644 contrib/pg_walinspect/Makefile
create mode 100644 contrib/pg_walinspect/pg_walinspect--1.0.sql
create mode 100644 contrib/pg_walinspect/pg_walinspect.c
create mode 100644 contrib/pg_walinspect/pg_walinspect.control
diff --git a/contrib/Makefile b/contrib/Makefile
index 332b486ecc..bbf220407b 100644
--- a/contrib/Makefile
+++ b/contrib/Makefile
@@ -41,6 +41,7 @@ SUBDIRS = \
pgrowlocks \
pgstattuple \
pg_visibility \
+ pg_walinspect \
postgres_fdw \
seg \
spi \
diff --git a/contrib/pg_walinspect/.gitignore b/contrib/pg_walinspect/.gitignore
new file mode 100644
index 0000000000..5dcb3ff972
--- /dev/null
+++ b/contrib/pg_walinspect/.gitignore
@@ -0,0 +1,4 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
diff --git a/contrib/pg_walinspect/Makefile b/contrib/pg_walinspect/Makefile
new file mode 100644
index 0000000000..c92a97447f
--- /dev/null
+++ b/contrib/pg_walinspect/Makefile
@@ -0,0 +1,26 @@
+# contrib/pg_walinspect/Makefile
+
+MODULE_big = pg_walinspect
+OBJS = \
+ $(WIN32RES) \
+ pg_walinspect.o
+PGFILEDESC = "pg_walinspect - functions to inspect contents of PostgreSQL Write-Ahead Log"
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+SHLIB_LINK_INTERNAL = $(libpq)
+
+EXTENSION = pg_walinspect
+DATA = pg_walinspect--1.0.sql
+
+REGRESS = pg_walinspect
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = contrib/pg_walinspect
+top_builddir = ../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/contrib/pg_walinspect/pg_walinspect--1.0.sql b/contrib/pg_walinspect/pg_walinspect--1.0.sql
new file mode 100644
index 0000000000..9b192dd2c1
--- /dev/null
+++ b/contrib/pg_walinspect/pg_walinspect--1.0.sql
@@ -0,0 +1,132 @@
+/* contrib/pg_walinspect/pg_walinspect--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION pg_walinspect" to load this file. \quit
+
+--
+-- pg_get_wal_record()
+--
+CREATE FUNCTION pg_get_wal_record(IN in_lsn pg_lsn,
+ OUT start_lsn pg_lsn,
+ OUT end_lsn pg_lsn,
+ OUT prev_lsn pg_lsn,
+ OUT record_length int4,
+ OUT record bytea
+)
+AS 'MODULE_PATHNAME', 'pg_get_wal_record'
+LANGUAGE C STRICT PARALLEL SAFE;
+
+REVOKE EXECUTE ON FUNCTION pg_get_wal_record(pg_lsn) FROM PUBLIC;
+GRANT EXECUTE ON FUNCTION pg_get_wal_record(pg_lsn) TO pg_read_server_files;
+
+--
+-- pg_get_wal_record_info()
+--
+CREATE FUNCTION pg_get_wal_record_info(IN in_lsn pg_lsn,
+ OUT start_lsn pg_lsn,
+ OUT end_lsn pg_lsn,
+ OUT prev_lsn pg_lsn,
+ OUT xid xid,
+ OUT resource_manager text,
+ OUT record_length int4,
+ OUT fpi_length int4,
+ OUT description text,
+ OUT block_ref text,
+ OUT data_length int4,
+ OUT data bytea
+)
+AS 'MODULE_PATHNAME', 'pg_get_wal_record_info'
+LANGUAGE C STRICT PARALLEL SAFE;
+
+REVOKE EXECUTE ON FUNCTION pg_get_wal_record_info(pg_lsn) FROM PUBLIC;
+GRANT EXECUTE ON FUNCTION pg_get_wal_record_info(pg_lsn) TO pg_read_server_files;
+
+--
+-- pg_get_wal_records_info()
+--
+CREATE FUNCTION pg_get_wal_records_info(IN start_lsn pg_lsn,
+ IN end_lsn pg_lsn,
+ OUT start_lsn pg_lsn,
+ OUT end_lsn pg_lsn,
+ OUT prev_lsn pg_lsn,
+ OUT xid xid,
+ OUT resource_manager text,
+ OUT record_length int4,
+ OUT fpi_length int4,
+ OUT description text,
+ OUT block_ref text,
+ OUT data_length int4,
+ OUT data bytea
+)
+RETURNS SETOF record
+AS 'MODULE_PATHNAME', 'pg_get_wal_records_info'
+LANGUAGE C STRICT PARALLEL SAFE;
+
+REVOKE EXECUTE ON FUNCTION pg_get_wal_records_info(pg_lsn, pg_lsn) FROM PUBLIC;
+GRANT EXECUTE ON FUNCTION pg_get_wal_records_info(pg_lsn, pg_lsn) TO pg_read_server_files;
+
+--
+-- pg_get_wal_records_info_till_end_of_wal()
+--
+CREATE FUNCTION pg_get_wal_records_info_till_end_of_wal(IN start_lsn pg_lsn,
+ OUT start_lsn pg_lsn,
+ OUT end_lsn pg_lsn,
+ OUT prev_lsn pg_lsn,
+ OUT xid xid,
+ OUT resource_manager text,
+ OUT record_length int4,
+ OUT fpi_length int4,
+ OUT description text,
+ OUT block_ref text,
+ OUT data_length int4,
+ OUT data bytea
+)
+RETURNS SETOF record
+AS 'MODULE_PATHNAME', 'pg_get_wal_records_info_till_end_of_wal'
+LANGUAGE C STRICT PARALLEL SAFE;
+
+REVOKE EXECUTE ON FUNCTION pg_get_wal_records_info_till_end_of_wal(pg_lsn) FROM PUBLIC;
+GRANT EXECUTE ON FUNCTION pg_get_wal_records_info_till_end_of_wal(pg_lsn) TO pg_read_server_files;
+
+--
+-- pg_get_wal_stats()
+--
+CREATE FUNCTION pg_get_wal_stats(IN start_lsn pg_lsn,
+ IN end_lsn pg_lsn,
+ OUT resource_manager text,
+ OUT count int8,
+ OUT count_percentage float4,
+ OUT record_size int8,
+ OUT record_size_percentage float4,
+ OUT fpi_size int8,
+ OUT fpi_size_percentage float4,
+ OUT combined_size int8,
+ OUT combined_size_percentage float4
+)
+RETURNS SETOF record
+AS 'MODULE_PATHNAME', 'pg_get_wal_stats'
+LANGUAGE C STRICT PARALLEL SAFE;
+
+REVOKE EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn) FROM PUBLIC;
+GRANT EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn) TO pg_read_server_files;
+
+--
+-- pg_get_wal_stats_till_end_of_wal()
+--
+CREATE FUNCTION pg_get_wal_stats_till_end_of_wal(IN start_lsn pg_lsn,
+ OUT resource_manager text,
+ OUT count int8,
+ OUT count_percentage float4,
+ OUT record_size int8,
+ OUT record_size_percentage float4,
+ OUT fpi_size int8,
+ OUT fpi_size_percentage float4,
+ OUT combined_size int8,
+ OUT combined_size_percentage float4
+)
+RETURNS SETOF record
+AS 'MODULE_PATHNAME', 'pg_get_wal_stats_till_end_of_wal'
+LANGUAGE C STRICT PARALLEL SAFE;
+
+REVOKE EXECUTE ON FUNCTION pg_get_wal_stats_till_end_of_wal(pg_lsn) FROM PUBLIC;
+GRANT EXECUTE ON FUNCTION pg_get_wal_stats_till_end_of_wal(pg_lsn) TO pg_read_server_files;
diff --git a/contrib/pg_walinspect/pg_walinspect.c b/contrib/pg_walinspect/pg_walinspect.c
new file mode 100644
index 0000000000..20e43b403e
--- /dev/null
+++ b/contrib/pg_walinspect/pg_walinspect.c
@@ -0,0 +1,652 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_walinspect.c
+ * Functions to inspect contents of PostgreSQL Write-Ahead Log
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * contrib/pg_walinspect/pg_walinspect.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/xlog.h"
+#include "access/xlog_internal.h"
+#include "access/xlogreader.h"
+#include "access/xlogrecovery.h"
+#include "access/xlogutils.h"
+#include "funcapi.h"
+#include "miscadmin.h"
+#include "utils/builtins.h"
+#include "utils/pg_lsn.h"
+
+/*
+ * NOTE: For any code change or issue fix here, it is highly recommended to
+ * give a thought about doing the same in pg_waldump tool as well.
+ */
+
+PG_MODULE_MAGIC;
+
+PG_FUNCTION_INFO_V1(pg_get_wal_record);
+PG_FUNCTION_INFO_V1(pg_get_wal_record_info);
+PG_FUNCTION_INFO_V1(pg_get_wal_records_info);
+PG_FUNCTION_INFO_V1(pg_get_wal_records_info_till_end_of_wal);
+PG_FUNCTION_INFO_V1(pg_get_wal_stats);
+PG_FUNCTION_INFO_V1(pg_get_wal_stats_till_end_of_wal);
+
+typedef void (*GetWALDetailsCB) (FunctionCallInfo fcinfo,
+ XLogRecPtr start_lsn,
+ XLogRecPtr end_lsn);
+
+static bool IsFutureLSN(XLogRecPtr lsn, XLogRecPtr *curr_lsn);
+static XLogReaderState *InitXLogReaderState(XLogRecPtr lsn,
+ XLogRecPtr *first_record);
+static XLogRecord *ReadNextXLogRecord(XLogReaderState *xlogreader,
+ XLogRecPtr first_record);
+static Datum GetWALRecordInternal(FunctionCallInfo fcinfo, Datum *values,
+ bool *nulls, uint32 ncols, bool get_info);
+static void GetXLogRecordInfo(XLogReaderState *record, XLogRecPtr lsn,
+ Datum *values, bool *nulls, uint32 ncols);
+static void GetWALDetailsGuts(FunctionCallInfo fcinfo, bool till_end_of_wal,
+ GetWALDetailsCB wal_details_cb);
+static void GetWALRecordsInfo(FunctionCallInfo fcinfo, XLogRecPtr start_lsn,
+ XLogRecPtr end_lsn);
+static void GetXLogSummaryStats(XLogStats * stats, ReturnSetInfo *rsinfo,
+ Datum *values, bool *nulls, uint32 ncols);
+static void FillXLogStatsRow(const char *name, uint64 n, uint64 total_count,
+ uint64 rec_len, uint64 total_rec_len,
+ uint64 fpi_len, uint64 total_fpi_len,
+ uint64 tot_len, uint64 total_len,
+ Datum *values, bool *nulls, uint32 ncols);
+static void GetWalStats(FunctionCallInfo fcinfo, XLogRecPtr start_lsn,
+ XLogRecPtr end_lsn);
+
+/*
+ * Determinte if the given LSN is in future and return the LSN up to which the
+ * server has WAL.
+ */
+static bool
+IsFutureLSN(XLogRecPtr lsn, XLogRecPtr *curr_lsn)
+{
+ /*
+ * We determine the current LSN of the server similar to how page_read
+ * callback read_local_xlog_page_no_wait does.
+ */
+ if (!RecoveryInProgress())
+ *curr_lsn = GetFlushRecPtr(NULL);
+ else
+ *curr_lsn = GetXLogReplayRecPtr(NULL);
+
+ Assert(!XLogRecPtrIsInvalid(*curr_lsn));
+
+ if (lsn >= *curr_lsn)
+ return true;
+
+ return false;
+}
+
+/*
+ * Intialize WAL reader and identify first valid LSN.
+ */
+static XLogReaderState *
+InitXLogReaderState(XLogRecPtr lsn, XLogRecPtr *first_record)
+{
+ XLogReaderState *xlogreader;
+
+ /*
+ * Reading WAL below the first page of the first sgements isn't allowed.
+ * This is a bootstrap WAL page and the page_read callback fails to read
+ * it.
+ */
+ if (lsn < XLOG_BLCKSZ)
+ ereport(ERROR,
+ (errmsg("could not read WAL at LSN %X/%X",
+ LSN_FORMAT_ARGS(lsn))));
+
+ xlogreader = XLogReaderAllocate(wal_segment_size, NULL,
+ XL_ROUTINE(.page_read = &read_local_xlog_page_no_wait,
+ .segment_open = &wal_segment_open,
+ .segment_close = &wal_segment_close),
+ NULL);
+
+ if (xlogreader == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_OUT_OF_MEMORY),
+ errmsg("out of memory"),
+ errdetail("Failed while allocating a WAL reading processor.")));
+
+ /* First find a valid recptr to start from. */
+ *first_record = XLogFindNextRecord(xlogreader, lsn);
+
+ if (XLogRecPtrIsInvalid(*first_record))
+ ereport(ERROR,
+ (errmsg("could not find a valid record after %X/%X",
+ LSN_FORMAT_ARGS(lsn))));
+
+ return xlogreader;
+}
+
+/*
+ * Read next WAL record.
+ */
+static XLogRecord *
+ReadNextXLogRecord(XLogReaderState *xlogreader, XLogRecPtr first_record)
+{
+ XLogRecord *record;
+ char *errormsg;
+
+ record = XLogReadRecord(xlogreader, &errormsg);
+
+ if (record == NULL)
+ {
+ if (errormsg)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read WAL at %X/%X: %s",
+ LSN_FORMAT_ARGS(first_record), errormsg)));
+ else
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read WAL at %X/%X",
+ LSN_FORMAT_ARGS(first_record))));
+ }
+
+ return record;
+}
+
+/*
+ * Get WAL record info.
+ */
+static void
+GetXLogRecordInfo(XLogReaderState *record, XLogRecPtr lsn,
+ Datum *values, bool *nulls, uint32 ncols)
+{
+ const char *id;
+ const RmgrData *desc;
+ uint32 fpi_len = 0;
+ StringInfoData rec_desc;
+ StringInfoData rec_blk_ref;
+ bytea *data;
+ char *main_data;
+ uint32 main_data_len;
+ int i = 0;
+
+ desc = &RmgrTable[XLogRecGetRmid(record)];
+ initStringInfo(&rec_desc);
+ id = desc->rm_identify(XLogRecGetInfo(record));
+
+ if (id == NULL)
+ appendStringInfo(&rec_desc, "UNKNOWN (%x) ", XLogRecGetInfo(record) & ~XLR_INFO_MASK);
+ else
+ appendStringInfo(&rec_desc, "%s ", id);
+
+ desc->rm_desc(&rec_desc, record);
+
+ /* Block references. */
+ initStringInfo(&rec_blk_ref);
+ XLogRecGetBlockRefInfo(record, NULL, &fpi_len, true, &rec_blk_ref);
+
+ main_data_len = XLogRecGetDataLen(record);
+ data = (bytea *) palloc(main_data_len + VARHDRSZ);
+ SET_VARSIZE(data, main_data_len + VARHDRSZ);
+ main_data = VARDATA(data);
+ memcpy(main_data, XLogRecGetData(record), main_data_len);
+
+ values[i++] = LSNGetDatum(lsn);
+ values[i++] = LSNGetDatum(record->EndRecPtr - 1);
+ values[i++] = LSNGetDatum(XLogRecGetPrev(record));
+ values[i++] = TransactionIdGetDatum(XLogRecGetXid(record));
+ values[i++] = CStringGetTextDatum(desc->rm_name);
+ values[i++] = UInt32GetDatum(XLogRecGetTotalLen(record));
+ values[i++] = UInt32GetDatum(fpi_len);
+ values[i++] = CStringGetTextDatum(rec_desc.data);
+ values[i++] = CStringGetTextDatum(rec_blk_ref.data);
+ values[i++] = UInt32GetDatum(main_data_len);
+ values[i++] = PointerGetDatum(data);
+
+ Assert(i == ncols);
+}
+
+/*
+ * Get WAL record data or info.
+ */
+Datum
+GetWALRecordInternal(FunctionCallInfo fcinfo, Datum *values, bool *nulls,
+ uint32 ncols, bool get_info)
+{
+ XLogRecPtr lsn;
+ XLogRecPtr curr_lsn;
+ XLogRecord *record;
+ XLogRecPtr first_record;
+ XLogReaderState *xlogreader;
+ TupleDesc tupdesc;
+ HeapTuple tuple;
+ Datum result;
+
+ lsn = PG_GETARG_LSN(0);
+
+ if (IsFutureLSN(lsn, &curr_lsn))
+ {
+ /*
+ * GetFlushRecPtr or GetXLogReplayRecPtr gives "end+1" LSN of the last
+ * record flushed or replayed respectively. But let's use the LSN up
+ * to "end" in user facing message.
+ */
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot accept future input LSN"),
+ errdetail("Last WAL record on the database system ends at LSN %X/%X.",
+ LSN_FORMAT_ARGS(curr_lsn - 1))));
+ }
+
+ /* Build a tuple descriptor for our result type. */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ xlogreader = InitXLogReaderState(lsn, &first_record);
+
+ Assert(xlogreader);
+
+ record = ReadNextXLogRecord(xlogreader, first_record);
+
+ MemSet(values, 0, ncols);
+ MemSet(nulls, 0, ncols);
+
+ if (get_info)
+ {
+ GetXLogRecordInfo(xlogreader, first_record, values, nulls,
+ ncols);
+ }
+ else
+ {
+ bytea *rec;
+ uint32 rec_len;
+ char *rec_data;
+ int i = 0;
+
+ rec_len = XLogRecGetTotalLen(xlogreader);
+
+ Assert(rec_len > 0);
+
+ rec = (bytea *) palloc(rec_len + VARHDRSZ);
+ SET_VARSIZE(rec, rec_len + VARHDRSZ);
+ rec_data = VARDATA(rec);
+
+ memcpy(rec_data, record, rec_len);
+
+ values[i++] = LSNGetDatum(first_record);
+ values[i++] = LSNGetDatum(xlogreader->EndRecPtr - 1);
+ values[i++] = LSNGetDatum(XLogRecGetPrev(xlogreader));
+ values[i++] = UInt32GetDatum(rec_len);
+ values[i++] = PointerGetDatum(rec);
+
+ Assert(i == ncols);
+ }
+
+ XLogReaderFree(xlogreader);
+
+ tuple = heap_form_tuple(tupdesc, values, nulls);
+ result = HeapTupleGetDatum(tuple);
+
+ return result;
+}
+
+/*
+ * Get WAL record.
+ *
+ * This function emits an error if a future WAL LSN i.e. WAL LSN the database
+ * system doesn't know about is specified.
+ */
+Datum
+pg_get_wal_record(PG_FUNCTION_ARGS)
+{
+#define PG_GET_WAL_RECORD_COLS 5
+ Datum result;
+ Datum values[PG_GET_WAL_RECORD_COLS];
+ bool nulls[PG_GET_WAL_RECORD_COLS];
+
+ result = GetWALRecordInternal(fcinfo, values, nulls,
+ PG_GET_WAL_RECORD_COLS,
+ false);
+
+ PG_RETURN_DATUM(result);
+#undef PG_GET_WAL_RECORD_COLS
+}
+
+/*
+ * Get WAL record info and data.
+ *
+ * This function emits an error if a future WAL LSN i.e. WAL LSN the database
+ * system doesn't know about is specified.
+ */
+Datum
+pg_get_wal_record_info(PG_FUNCTION_ARGS)
+{
+#define PG_GET_WAL_RECORD_INFO_COLS 11
+ Datum result;
+ Datum values[PG_GET_WAL_RECORD_INFO_COLS];
+ bool nulls[PG_GET_WAL_RECORD_INFO_COLS];
+
+ result = GetWALRecordInternal(fcinfo, values, nulls,
+ PG_GET_WAL_RECORD_INFO_COLS,
+ true);
+
+ PG_RETURN_DATUM(result);
+#undef PG_GET_WAL_RECORD_INFO_COLS
+}
+
+/*
+ * Get WAL details such as record info, stats using the passed in callback.
+ */
+static void
+GetWALDetailsGuts(FunctionCallInfo fcinfo, bool till_end_of_wal,
+ GetWALDetailsCB wal_details_cb)
+{
+ XLogRecPtr start_lsn;
+ XLogRecPtr end_lsn;
+ XLogRecPtr curr_lsn;
+
+ start_lsn = PG_GETARG_LSN(0);
+
+ /* If not till end of wal, end_lsn would have been specified. */
+ if (!till_end_of_wal)
+ end_lsn = PG_GETARG_LSN(1);
+
+ if (IsFutureLSN(start_lsn, &curr_lsn))
+ {
+ /*
+ * GetFlushRecPtr or GetXLogReplayRecPtr gives "end+1" LSN of the last
+ * record flushed or replayed respectively. But let's use the LSN up
+ * to "end" in user facing message.
+ */
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot accept future start LSN"),
+ errdetail("Last WAL record on the database system ends at LSN %X/%X.",
+ LSN_FORMAT_ARGS(curr_lsn - 1))));
+ }
+
+ if (!till_end_of_wal && end_lsn >= curr_lsn)
+ {
+ /*
+ * GetFlushRecPtr or GetXLogReplayRecPtr gives "end+1" LSN of the last
+ * record flushed or replayed respectively. But let's use the LSN up
+ * to "end" in user facing message.
+ */
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot accept future end LSN"),
+ errdetail("Last WAL record on the database system ends at LSN %X/%X.",
+ LSN_FORMAT_ARGS(curr_lsn - 1))));
+ }
+
+ /*
+ * GetFlushRecPtr or GetXLogReplayRecPtr gives "end+1" LSN of the last
+ * record flushed or replayed respectively. But let's use the LSN up to
+ * "end".
+ */
+ if (till_end_of_wal)
+ end_lsn = curr_lsn - 1;
+
+ if (start_lsn >= end_lsn)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("WAL start LSN must be less than end LSN")));
+
+ wal_details_cb(fcinfo, start_lsn, end_lsn);
+}
+
+/*
+ * Get info and data of all WAL records between start LSN and end LSN.
+ */
+static void
+GetWALRecordsInfo(FunctionCallInfo fcinfo, XLogRecPtr start_lsn,
+ XLogRecPtr end_lsn)
+{
+#define PG_GET_WAL_RECORDS_INFO_COLS 11
+ XLogRecPtr first_record;
+ XLogReaderState *xlogreader;
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ Datum values[PG_GET_WAL_RECORDS_INFO_COLS];
+ bool nulls[PG_GET_WAL_RECORDS_INFO_COLS];
+
+ SetSingleFuncCall(fcinfo, 0);
+
+ xlogreader = InitXLogReaderState(start_lsn, &first_record);
+
+ Assert(xlogreader);
+
+ MemSet(values, 0, sizeof(values));
+ MemSet(nulls, 0, sizeof(nulls));
+
+ for (;;)
+ {
+ (void) ReadNextXLogRecord(xlogreader, first_record);
+
+ /*
+ * Let's not show the record info if it is spanning more than the
+ * end_lsn. EndRecPtr is "end+1" of the last read record, hence
+ * use "end" here.
+ */
+ if ((xlogreader->EndRecPtr - 1) <= end_lsn)
+ {
+ GetXLogRecordInfo(xlogreader, xlogreader->currRecPtr, values, nulls,
+ PG_GET_WAL_RECORDS_INFO_COLS);
+
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc,
+ values, nulls);
+ }
+
+ /* Exit loop if read up to end_lsn. */
+ if (xlogreader->EndRecPtr >= end_lsn)
+ break;
+
+ CHECK_FOR_INTERRUPTS();
+ }
+
+ XLogReaderFree(xlogreader);
+
+#undef PG_GET_WAL_RECORDS_INFO_COLS
+}
+
+/*
+ * Get info and data of all WAL records between start LSN and end LSN.
+ *
+ * This function emits an error if a future start or end WAL LSN i.e. WAL LSN
+ * the database system doesn't know about is specified.
+ */
+Datum
+pg_get_wal_records_info(PG_FUNCTION_ARGS)
+{
+ GetWALDetailsGuts(fcinfo, false, GetWALRecordsInfo);
+
+ PG_RETURN_VOID();
+}
+
+/*
+ * Get info and data of all WAL records from start LSN till end of WAL.
+ *
+ * This function emits an error if a future start i.e. WAL LSN the database
+ * system doesn't know about is specified.
+ */
+Datum
+pg_get_wal_records_info_till_end_of_wal(PG_FUNCTION_ARGS)
+{
+ GetWALDetailsGuts(fcinfo, true, GetWALRecordsInfo);
+
+ PG_RETURN_VOID();
+}
+
+/*
+ * Fill single row of record counts and sizes for an rmgr or record.
+ */
+static void
+FillXLogStatsRow(const char *name,
+ uint64 n, uint64 total_count,
+ uint64 rec_len, uint64 total_rec_len,
+ uint64 fpi_len, uint64 total_fpi_len,
+ uint64 tot_len, uint64 total_len,
+ Datum *values, bool *nulls, uint32 ncols)
+{
+ double n_pct,
+ rec_len_pct,
+ fpi_len_pct,
+ tot_len_pct;
+ int i = 0;
+
+ n_pct = 0;
+ if (total_count != 0)
+ n_pct = 100 * (double) n / total_count;
+
+ rec_len_pct = 0;
+ if (total_rec_len != 0)
+ rec_len_pct = 100 * (double) rec_len / total_rec_len;
+
+ fpi_len_pct = 0;
+ if (total_fpi_len != 0)
+ fpi_len_pct = 100 * (double) fpi_len / total_fpi_len;
+
+ tot_len_pct = 0;
+ if (total_len != 0)
+ tot_len_pct = 100 * (double) tot_len / total_len;
+
+ values[i++] = CStringGetTextDatum(name);
+ values[i++] = Int64GetDatum(n);
+ values[i++] = Float4GetDatum(n_pct);
+ values[i++] = Int64GetDatum(rec_len);
+ values[i++] = Float4GetDatum(rec_len_pct);
+ values[i++] = Int64GetDatum(fpi_len);
+ values[i++] = Float4GetDatum(fpi_len_pct);
+ values[i++] = Int64GetDatum(tot_len);
+ values[i++] = Float4GetDatum(tot_len_pct);
+
+ Assert(i == ncols);
+}
+
+/*
+ * Get summary statistics about the records seen so far.
+ */
+static void
+GetXLogSummaryStats(XLogStats *stats, ReturnSetInfo *rsinfo,
+ Datum *values, bool *nulls, uint32 ncols)
+{
+ uint64 total_count = 0;
+ uint64 total_rec_len = 0;
+ uint64 total_fpi_len = 0;
+ uint64 total_len = 0;
+ int ri;
+
+ /*
+ * Each row shows its percentages of the total, so make a first pass to
+ * calculate column totals.
+ */
+ for (ri = 0; ri < RM_NEXT_ID; ri++)
+ {
+ total_count += stats->rmgr_stats[ri].count;
+ total_rec_len += stats->rmgr_stats[ri].rec_len;
+ total_fpi_len += stats->rmgr_stats[ri].fpi_len;
+ }
+ total_len = total_rec_len + total_fpi_len;
+
+ for (ri = 0; ri < RM_NEXT_ID; ri++)
+ {
+ uint64 count;
+ uint64 rec_len;
+ uint64 fpi_len;
+ uint64 tot_len;
+ const RmgrData *desc = &RmgrTable[ri];
+
+ count = stats->rmgr_stats[ri].count;
+ rec_len = stats->rmgr_stats[ri].rec_len;
+ fpi_len = stats->rmgr_stats[ri].fpi_len;
+ tot_len = rec_len + fpi_len;
+
+ FillXLogStatsRow(desc->rm_name, count, total_count, rec_len,
+ total_rec_len, fpi_len, total_fpi_len, tot_len,
+ total_len, values, nulls, ncols);
+
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc,
+ values, nulls);
+ }
+}
+
+/*
+ * Get WAL stats between start LSN and end LSN.
+ */
+static void
+GetWalStats(FunctionCallInfo fcinfo, XLogRecPtr start_lsn,
+ XLogRecPtr end_lsn)
+{
+#define PG_GET_WAL_STATS_COLS 9
+ XLogRecPtr first_record;
+ XLogReaderState *xlogreader;
+ XLogStats stats;
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ Datum values[PG_GET_WAL_STATS_COLS];
+ bool nulls[PG_GET_WAL_STATS_COLS];
+
+ SetSingleFuncCall(fcinfo, 0);
+
+ xlogreader = InitXLogReaderState(start_lsn, &first_record);
+
+ MemSet(&stats, 0, sizeof(stats));
+
+ for (;;)
+ {
+ (void) ReadNextXLogRecord(xlogreader, first_record);
+
+ /*
+ * Let's not show the record info if it is spanning more than the
+ * end_lsn. EndRecPtr is "end+1" of the last read record, hence
+ * use "end" here.
+ */
+ if ((xlogreader->EndRecPtr - 1) <= end_lsn)
+ XLogRecStoreStats(&stats, xlogreader);
+
+ /* Exit loop if read up to end_lsn. */
+ if (xlogreader->EndRecPtr >= end_lsn)
+ break;
+
+ CHECK_FOR_INTERRUPTS();
+ }
+
+ XLogReaderFree(xlogreader);
+
+ MemSet(values, 0, sizeof(values));
+ MemSet(nulls, 0, sizeof(nulls));
+
+ GetXLogSummaryStats(&stats, rsinfo, values, nulls,
+ PG_GET_WAL_STATS_COLS);
+
+#undef PG_GET_WAL_STATS_COLS
+}
+
+/*
+ * Get stats of all WAL records between start LSN and end LSN.
+ *
+ * This function emits an error if a future start or end WAL LSN i.e. WAL LSN
+ * the database system doesn't know about is specified.
+ */
+Datum
+pg_get_wal_stats(PG_FUNCTION_ARGS)
+{
+ GetWALDetailsGuts(fcinfo, false, GetWalStats);
+
+ PG_RETURN_VOID();
+}
+
+/*
+ * Get stats of all WAL records from start LSN till end of WAL.
+ *
+ * This function emits an error if a future start i.e. WAL LSN the database
+ * system doesn't know about is specified.
+ */
+Datum
+pg_get_wal_stats_till_end_of_wal(PG_FUNCTION_ARGS)
+{
+ GetWALDetailsGuts(fcinfo, true, GetWalStats);
+
+ PG_RETURN_VOID();
+}
diff --git a/contrib/pg_walinspect/pg_walinspect.control b/contrib/pg_walinspect/pg_walinspect.control
new file mode 100644
index 0000000000..017e56a2bb
--- /dev/null
+++ b/contrib/pg_walinspect/pg_walinspect.control
@@ -0,0 +1,5 @@
+# pg_walinspect extension
+comment = 'functions to inspect contents of PostgreSQL Write-Ahead Log'
+default_version = '1.0'
+module_pathname = '$libdir/pg_walinspect'
+relocatable = true
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 5909ec180e..d9d6889f67 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -1320,13 +1320,6 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr,
return true;
}
-#ifdef FRONTEND
-/*
- * Functions that are currently not needed in the backend, but are better
- * implemented inside xlogreader.c because of the internal facilities available
- * here.
- */
-
/*
* Find the first record with an lsn >= RecPtr.
*
@@ -1447,6 +1440,12 @@ err:
return InvalidXLogRecPtr;
}
+#ifdef FRONTEND
+/*
+ * Functions that are currently not needed in the backend, but are better
+ * implemented inside xlogreader.c because of the internal facilities available
+ * here.
+ */
#endif /* FRONTEND */
/*
diff --git a/src/backend/access/transam/xlogutils.c b/src/backend/access/transam/xlogutils.c
index 511f2f186f..7ade716b40 100644
--- a/src/backend/access/transam/xlogutils.c
+++ b/src/backend/access/transam/xlogutils.c
@@ -79,6 +79,10 @@ typedef struct xl_invalid_page
static HTAB *invalid_page_tab = NULL;
+static int
+read_local_xlog_page_guts(XLogReaderState *state, XLogRecPtr targetPagePtr,
+ int reqLen, XLogRecPtr targetRecPtr,
+ char *cur_page, bool wait_for_wal);
/* Report a reference to an invalid page */
static void
@@ -851,6 +855,31 @@ wal_segment_close(XLogReaderState *state)
int
read_local_xlog_page(XLogReaderState *state, XLogRecPtr targetPagePtr,
int reqLen, XLogRecPtr targetRecPtr, char *cur_page)
+{
+ return read_local_xlog_page_guts(state, targetPagePtr, reqLen,
+ targetRecPtr, cur_page, true);
+}
+
+/*
+ * Same as read_local_xlog_page except that it doesn't wait for future WAL
+ * to be available.
+ */
+int
+read_local_xlog_page_no_wait(XLogReaderState *state, XLogRecPtr targetPagePtr,
+ int reqLen, XLogRecPtr targetRecPtr,
+ char *cur_page)
+{
+ return read_local_xlog_page_guts(state, targetPagePtr, reqLen,
+ targetRecPtr, cur_page, false);
+}
+
+/*
+ * Implementation of read_local_xlog_page and its no wait version.
+ */
+static int
+read_local_xlog_page_guts(XLogReaderState *state, XLogRecPtr targetPagePtr,
+ int reqLen, XLogRecPtr targetRecPtr,
+ char *cur_page, bool wait_for_wal)
{
XLogRecPtr read_upto,
loc;
@@ -906,6 +935,10 @@ read_local_xlog_page(XLogReaderState *state, XLogRecPtr targetPagePtr,
if (loc <= read_upto)
break;
+ /* If asked, let's not wait for future WAL. */
+ if (!wait_for_wal)
+ break;
+
CHECK_FOR_INTERRUPTS();
pg_usleep(1000L);
}
diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c
index 179cd617e4..2072fdeed0 100644
--- a/src/bin/pg_waldump/pg_waldump.c
+++ b/src/bin/pg_waldump/pg_waldump.c
@@ -26,6 +26,11 @@
#include "getopt_long.h"
#include "rmgrdesc.h"
+/*
+ * NOTE: For any code change or issue fix here, it is highly recommended to
+ * give a thought about doing the same in pg_walinspect contrib module as well.
+ */
+
static const char *progname;
static int WalSegSz;
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 09f6464331..3e644372f9 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -31,7 +31,7 @@ extern XLogRecPtr XactLastRecEnd;
extern PGDLLIMPORT XLogRecPtr XactLastCommitEnd;
/* these variables are GUC parameters related to XLOG */
-extern int wal_segment_size;
+extern PGDLLIMPORT int wal_segment_size;
extern int min_wal_size_mb;
extern int max_wal_size_mb;
extern int wal_keep_size_mb;
diff --git a/src/include/access/xlog_internal.h b/src/include/access/xlog_internal.h
index 0e94833129..8468732d76 100644
--- a/src/include/access/xlog_internal.h
+++ b/src/include/access/xlog_internal.h
@@ -319,7 +319,7 @@ typedef struct RmgrData
struct XLogRecordBuffer *buf);
} RmgrData;
-extern const RmgrData RmgrTable[];
+extern PGDLLIMPORT const RmgrData RmgrTable[];
/*
* Exported to support xlog switching from checkpointer
diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h
index 4021ae05df..fbd251aed2 100644
--- a/src/include/access/xlogreader.h
+++ b/src/include/access/xlogreader.h
@@ -361,9 +361,7 @@ extern void XLogReaderSetDecodeBuffer(XLogReaderState *state,
/* Position the XLogReader to given record */
extern void XLogBeginRead(XLogReaderState *state, XLogRecPtr RecPtr);
-#ifdef FRONTEND
extern XLogRecPtr XLogFindNextRecord(XLogReaderState *state, XLogRecPtr RecPtr);
-#endif /* FRONTEND */
/* Return values from XLogPageReadCB. */
typedef enum XLogPageReadResult
diff --git a/src/include/access/xlogutils.h b/src/include/access/xlogutils.h
index 64708949db..22c2299d68 100644
--- a/src/include/access/xlogutils.h
+++ b/src/include/access/xlogutils.h
@@ -92,6 +92,10 @@ extern void FreeFakeRelcacheEntry(Relation fakerel);
extern int read_local_xlog_page(XLogReaderState *state,
XLogRecPtr targetPagePtr, int reqLen,
XLogRecPtr targetRecPtr, char *cur_page);
+extern int read_local_xlog_page_no_wait(XLogReaderState *state,
+ XLogRecPtr targetPagePtr, int reqLen,
+ XLogRecPtr targetRecPtr,
+ char *cur_page);
extern void wal_segment_open(XLogReaderState *state,
XLogSegNo nextSegNo,
TimeLineID *tli_p);
--
2.25.1
[application/octet-stream] v14-0003-pg_walinspect-tests.patch (13.0K, ../CALj2ACWHxU2brc1vRJKxmtgr6hAB6YnL9fjVdN_qEvv1HiWLFQ@mail.gmail.com/4-v14-0003-pg_walinspect-tests.patch)
download | inline diff:
From e5c03874351e6e9960f070a6f6feb42157cdc2bf Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <[email protected]>
Date: Thu, 24 Mar 2022 09:25:50 +0000
Subject: [PATCH v14] pg_walinspect tests
---
.../pg_walinspect/expected/pg_walinspect.out | 216 ++++++++++++++++++
contrib/pg_walinspect/sql/pg_walinspect.sql | 146 ++++++++++++
2 files changed, 362 insertions(+)
create mode 100644 contrib/pg_walinspect/expected/pg_walinspect.out
create mode 100644 contrib/pg_walinspect/sql/pg_walinspect.sql
diff --git a/contrib/pg_walinspect/expected/pg_walinspect.out b/contrib/pg_walinspect/expected/pg_walinspect.out
new file mode 100644
index 0000000000..7445c7dfde
--- /dev/null
+++ b/contrib/pg_walinspect/expected/pg_walinspect.out
@@ -0,0 +1,216 @@
+CREATE EXTENSION pg_walinspect;
+CREATE TABLE sample_tbl(col1 int, col2 int);
+SELECT pg_current_wal_lsn() AS wal_lsn1 \gset
+INSERT INTO sample_tbl SELECT * FROM generate_series(1, 2);
+SELECT pg_current_wal_lsn() AS wal_lsn2 \gset
+INSERT INTO sample_tbl SELECT * FROM generate_series(1, 2);
+-- ===================================================================
+-- Tests for input validation
+-- ===================================================================
+SELECT COUNT(*) >= 0 AS ok FROM pg_get_wal_records_info(:'wal_lsn2', :'wal_lsn1'); -- ERROR
+ERROR: WAL start LSN must be less than end LSN
+SELECT COUNT(*) >= 0 AS ok FROM pg_get_wal_stats(:'wal_lsn2', :'wal_lsn1'); -- ERROR
+ERROR: WAL start LSN must be less than end LSN
+-- ===================================================================
+-- Tests for all function executions
+-- ===================================================================
+SELECT COUNT(*) >= 0 AS ok FROM pg_get_wal_record(:'wal_lsn1');
+ ok
+----
+ t
+(1 row)
+
+SELECT COUNT(*) >= 0 AS ok FROM pg_get_wal_records_info(:'wal_lsn1', :'wal_lsn2');
+ ok
+----
+ t
+(1 row)
+
+SELECT COUNT(*) >= 0 AS ok FROM pg_get_wal_records_info_till_end_of_wal(:'wal_lsn1');
+ ok
+----
+ t
+(1 row)
+
+SELECT COUNT(*) >= 0 AS ok FROM pg_get_wal_stats(:'wal_lsn1', :'wal_lsn2');
+ ok
+----
+ t
+(1 row)
+
+SELECT COUNT(*) >= 0 AS ok FROM pg_get_wal_stats_till_end_of_wal(:'wal_lsn1');
+ ok
+----
+ t
+(1 row)
+
+-- ===================================================================
+-- Tests for filtering out WAL records of a particular table
+-- ===================================================================
+SELECT oid AS sample_tbl_oid FROM pg_class WHERE relname = 'sample_tbl' \gset
+SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_records_info(:'wal_lsn1', :'wal_lsn2')
+ WHERE block_ref LIKE concat('%', :'sample_tbl_oid', '%') AND resource_manager = 'Heap';
+ ok
+----
+ t
+(1 row)
+
+-- ===================================================================
+-- Tests for permissions
+-- ===================================================================
+CREATE ROLE regress_pg_walinspect;
+SELECT has_function_privilege('regress_pg_walinspect',
+ 'pg_get_wal_record(pg_lsn)', 'EXECUTE'); -- no
+ has_function_privilege
+------------------------
+ f
+(1 row)
+
+SELECT has_function_privilege('regress_pg_walinspect',
+ 'pg_get_wal_record_info(pg_lsn)', 'EXECUTE'); -- no
+ has_function_privilege
+------------------------
+ f
+(1 row)
+
+SELECT has_function_privilege('regress_pg_walinspect',
+ 'pg_get_wal_records_info(pg_lsn, pg_lsn) ', 'EXECUTE'); -- no
+ has_function_privilege
+------------------------
+ f
+(1 row)
+
+SELECT has_function_privilege('regress_pg_walinspect',
+ 'pg_get_wal_records_info_till_end_of_wal(pg_lsn) ', 'EXECUTE'); -- no
+ has_function_privilege
+------------------------
+ f
+(1 row)
+
+SELECT has_function_privilege('regress_pg_walinspect',
+ 'pg_get_wal_stats(pg_lsn, pg_lsn) ', 'EXECUTE'); -- no
+ has_function_privilege
+------------------------
+ f
+(1 row)
+
+SELECT has_function_privilege('regress_pg_walinspect',
+ 'pg_get_wal_stats_till_end_of_wal(pg_lsn) ', 'EXECUTE'); -- no
+ has_function_privilege
+------------------------
+ f
+(1 row)
+
+-- Functions accessible by users with role pg_read_server_files
+GRANT pg_read_server_files TO regress_pg_walinspect;
+SELECT has_function_privilege('regress_pg_walinspect',
+ 'pg_get_wal_record(pg_lsn)', 'EXECUTE'); -- yes
+ has_function_privilege
+------------------------
+ t
+(1 row)
+
+SELECT has_function_privilege('regress_pg_walinspect',
+ 'pg_get_wal_record_info(pg_lsn)', 'EXECUTE'); -- yes
+ has_function_privilege
+------------------------
+ t
+(1 row)
+
+SELECT has_function_privilege('regress_pg_walinspect',
+ 'pg_get_wal_records_info(pg_lsn, pg_lsn) ', 'EXECUTE'); -- yes
+ has_function_privilege
+------------------------
+ t
+(1 row)
+
+SELECT has_function_privilege('regress_pg_walinspect',
+ 'pg_get_wal_records_info_till_end_of_wal(pg_lsn) ', 'EXECUTE'); -- yes
+ has_function_privilege
+------------------------
+ t
+(1 row)
+
+SELECT has_function_privilege('regress_pg_walinspect',
+ 'pg_get_wal_stats(pg_lsn, pg_lsn) ', 'EXECUTE'); -- yes
+ has_function_privilege
+------------------------
+ t
+(1 row)
+
+SELECT has_function_privilege('regress_pg_walinspect',
+ 'pg_get_wal_stats_till_end_of_wal(pg_lsn) ', 'EXECUTE'); -- yes
+ has_function_privilege
+------------------------
+ t
+(1 row)
+
+REVOKE pg_read_server_files FROM regress_pg_walinspect;
+-- Superuser can grant execute to other users
+GRANT EXECUTE ON FUNCTION pg_get_wal_record(pg_lsn)
+ TO regress_pg_walinspect;
+GRANT EXECUTE ON FUNCTION pg_get_wal_record_info(pg_lsn)
+ TO regress_pg_walinspect;
+GRANT EXECUTE ON FUNCTION pg_get_wal_records_info(pg_lsn, pg_lsn)
+ TO regress_pg_walinspect;
+GRANT EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn)
+ TO regress_pg_walinspect;
+SELECT has_function_privilege('regress_pg_walinspect',
+ 'pg_get_wal_record(pg_lsn)', 'EXECUTE'); -- yes
+ has_function_privilege
+------------------------
+ t
+(1 row)
+
+SELECT has_function_privilege('regress_pg_walinspect',
+ 'pg_get_wal_record_info(pg_lsn)', 'EXECUTE'); -- yes
+ has_function_privilege
+------------------------
+ t
+(1 row)
+
+SELECT has_function_privilege('regress_pg_walinspect',
+ 'pg_get_wal_records_info(pg_lsn, pg_lsn) ', 'EXECUTE'); -- yes
+ has_function_privilege
+------------------------
+ t
+(1 row)
+
+SELECT has_function_privilege('regress_pg_walinspect',
+ 'pg_get_wal_records_info_till_end_of_wal(pg_lsn) ', 'EXECUTE'); -- yes
+ has_function_privilege
+------------------------
+ f
+(1 row)
+
+SELECT has_function_privilege('regress_pg_walinspect',
+ 'pg_get_wal_stats(pg_lsn, pg_lsn) ', 'EXECUTE'); -- yes
+ has_function_privilege
+------------------------
+ t
+(1 row)
+
+SELECT has_function_privilege('regress_pg_walinspect',
+ 'pg_get_wal_stats_till_end_of_wal(pg_lsn) ', 'EXECUTE'); -- yes
+ has_function_privilege
+------------------------
+ f
+(1 row)
+
+REVOKE EXECUTE ON FUNCTION pg_get_wal_record(pg_lsn)
+ FROM regress_pg_walinspect;
+REVOKE EXECUTE ON FUNCTION pg_get_wal_record_info(pg_lsn)
+ FROM regress_pg_walinspect;
+REVOKE EXECUTE ON FUNCTION pg_get_wal_records_info(pg_lsn, pg_lsn)
+ FROM regress_pg_walinspect;
+REVOKE EXECUTE ON FUNCTION pg_get_wal_records_info_till_end_of_wal(pg_lsn)
+ FROM regress_pg_walinspect;
+REVOKE EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn)
+ FROM regress_pg_walinspect;
+REVOKE EXECUTE ON FUNCTION pg_get_wal_stats_till_end_of_wal(pg_lsn)
+ FROM regress_pg_walinspect;
+-- ===================================================================
+-- Clean up
+-- ===================================================================
+DROP ROLE regress_pg_walinspect;
+DROP TABLE sample_tbl;
diff --git a/contrib/pg_walinspect/sql/pg_walinspect.sql b/contrib/pg_walinspect/sql/pg_walinspect.sql
new file mode 100644
index 0000000000..3a97ef7153
--- /dev/null
+++ b/contrib/pg_walinspect/sql/pg_walinspect.sql
@@ -0,0 +1,146 @@
+CREATE EXTENSION pg_walinspect;
+
+CREATE TABLE sample_tbl(col1 int, col2 int);
+
+SELECT pg_current_wal_lsn() AS wal_lsn1 \gset
+
+INSERT INTO sample_tbl SELECT * FROM generate_series(1, 2);
+
+SELECT pg_current_wal_lsn() AS wal_lsn2 \gset
+
+INSERT INTO sample_tbl SELECT * FROM generate_series(1, 2);
+
+-- ===================================================================
+-- Tests for input validation
+-- ===================================================================
+
+SELECT COUNT(*) >= 0 AS ok FROM pg_get_wal_records_info(:'wal_lsn2', :'wal_lsn1'); -- ERROR
+
+SELECT COUNT(*) >= 0 AS ok FROM pg_get_wal_stats(:'wal_lsn2', :'wal_lsn1'); -- ERROR
+
+-- ===================================================================
+-- Tests for all function executions
+-- ===================================================================
+
+SELECT COUNT(*) >= 0 AS ok FROM pg_get_wal_record(:'wal_lsn1');
+
+SELECT COUNT(*) >= 0 AS ok FROM pg_get_wal_records_info(:'wal_lsn1', :'wal_lsn2');
+
+SELECT COUNT(*) >= 0 AS ok FROM pg_get_wal_records_info_till_end_of_wal(:'wal_lsn1');
+
+SELECT COUNT(*) >= 0 AS ok FROM pg_get_wal_stats(:'wal_lsn1', :'wal_lsn2');
+
+SELECT COUNT(*) >= 0 AS ok FROM pg_get_wal_stats_till_end_of_wal(:'wal_lsn1');
+
+-- ===================================================================
+-- Tests for filtering out WAL records of a particular table
+-- ===================================================================
+
+SELECT oid AS sample_tbl_oid FROM pg_class WHERE relname = 'sample_tbl' \gset
+
+SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_records_info(:'wal_lsn1', :'wal_lsn2')
+ WHERE block_ref LIKE concat('%', :'sample_tbl_oid', '%') AND resource_manager = 'Heap';
+
+-- ===================================================================
+-- Tests for permissions
+-- ===================================================================
+CREATE ROLE regress_pg_walinspect;
+
+SELECT has_function_privilege('regress_pg_walinspect',
+ 'pg_get_wal_record(pg_lsn)', 'EXECUTE'); -- no
+
+SELECT has_function_privilege('regress_pg_walinspect',
+ 'pg_get_wal_record_info(pg_lsn)', 'EXECUTE'); -- no
+
+SELECT has_function_privilege('regress_pg_walinspect',
+ 'pg_get_wal_records_info(pg_lsn, pg_lsn) ', 'EXECUTE'); -- no
+
+SELECT has_function_privilege('regress_pg_walinspect',
+ 'pg_get_wal_records_info_till_end_of_wal(pg_lsn) ', 'EXECUTE'); -- no
+
+SELECT has_function_privilege('regress_pg_walinspect',
+ 'pg_get_wal_stats(pg_lsn, pg_lsn) ', 'EXECUTE'); -- no
+
+SELECT has_function_privilege('regress_pg_walinspect',
+ 'pg_get_wal_stats_till_end_of_wal(pg_lsn) ', 'EXECUTE'); -- no
+
+-- Functions accessible by users with role pg_read_server_files
+
+GRANT pg_read_server_files TO regress_pg_walinspect;
+
+SELECT has_function_privilege('regress_pg_walinspect',
+ 'pg_get_wal_record(pg_lsn)', 'EXECUTE'); -- yes
+
+SELECT has_function_privilege('regress_pg_walinspect',
+ 'pg_get_wal_record_info(pg_lsn)', 'EXECUTE'); -- yes
+
+SELECT has_function_privilege('regress_pg_walinspect',
+ 'pg_get_wal_records_info(pg_lsn, pg_lsn) ', 'EXECUTE'); -- yes
+
+SELECT has_function_privilege('regress_pg_walinspect',
+ 'pg_get_wal_records_info_till_end_of_wal(pg_lsn) ', 'EXECUTE'); -- yes
+
+SELECT has_function_privilege('regress_pg_walinspect',
+ 'pg_get_wal_stats(pg_lsn, pg_lsn) ', 'EXECUTE'); -- yes
+
+SELECT has_function_privilege('regress_pg_walinspect',
+ 'pg_get_wal_stats_till_end_of_wal(pg_lsn) ', 'EXECUTE'); -- yes
+
+REVOKE pg_read_server_files FROM regress_pg_walinspect;
+
+-- Superuser can grant execute to other users
+GRANT EXECUTE ON FUNCTION pg_get_wal_record(pg_lsn)
+ TO regress_pg_walinspect;
+
+GRANT EXECUTE ON FUNCTION pg_get_wal_record_info(pg_lsn)
+ TO regress_pg_walinspect;
+
+GRANT EXECUTE ON FUNCTION pg_get_wal_records_info(pg_lsn, pg_lsn)
+ TO regress_pg_walinspect;
+
+GRANT EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn)
+ TO regress_pg_walinspect;
+
+SELECT has_function_privilege('regress_pg_walinspect',
+ 'pg_get_wal_record(pg_lsn)', 'EXECUTE'); -- yes
+
+SELECT has_function_privilege('regress_pg_walinspect',
+ 'pg_get_wal_record_info(pg_lsn)', 'EXECUTE'); -- yes
+
+SELECT has_function_privilege('regress_pg_walinspect',
+ 'pg_get_wal_records_info(pg_lsn, pg_lsn) ', 'EXECUTE'); -- yes
+
+SELECT has_function_privilege('regress_pg_walinspect',
+ 'pg_get_wal_records_info_till_end_of_wal(pg_lsn) ', 'EXECUTE'); -- yes
+
+SELECT has_function_privilege('regress_pg_walinspect',
+ 'pg_get_wal_stats(pg_lsn, pg_lsn) ', 'EXECUTE'); -- yes
+
+SELECT has_function_privilege('regress_pg_walinspect',
+ 'pg_get_wal_stats_till_end_of_wal(pg_lsn) ', 'EXECUTE'); -- yes
+
+REVOKE EXECUTE ON FUNCTION pg_get_wal_record(pg_lsn)
+ FROM regress_pg_walinspect;
+
+REVOKE EXECUTE ON FUNCTION pg_get_wal_record_info(pg_lsn)
+ FROM regress_pg_walinspect;
+
+REVOKE EXECUTE ON FUNCTION pg_get_wal_records_info(pg_lsn, pg_lsn)
+ FROM regress_pg_walinspect;
+
+REVOKE EXECUTE ON FUNCTION pg_get_wal_records_info_till_end_of_wal(pg_lsn)
+ FROM regress_pg_walinspect;
+
+REVOKE EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn)
+ FROM regress_pg_walinspect;
+
+REVOKE EXECUTE ON FUNCTION pg_get_wal_stats_till_end_of_wal(pg_lsn)
+ FROM regress_pg_walinspect;
+
+-- ===================================================================
+-- Clean up
+-- ===================================================================
+
+DROP ROLE regress_pg_walinspect;
+
+DROP TABLE sample_tbl;
--
2.25.1
[application/octet-stream] v14-0004-pg_walinspect-docs.patch (12.2K, ../CALj2ACWHxU2brc1vRJKxmtgr6hAB6YnL9fjVdN_qEvv1HiWLFQ@mail.gmail.com/5-v14-0004-pg_walinspect-docs.patch)
download | inline diff:
From f9a0b5c9884e6239eca29b3ab134578af86acb32 Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <[email protected]>
Date: Thu, 24 Mar 2022 09:26:42 +0000
Subject: [PATCH v14] pg_walinspect docs
---
doc/src/sgml/contrib.sgml | 1 +
doc/src/sgml/filelist.sgml | 1 +
doc/src/sgml/pgwalinspect.sgml | 234 +++++++++++++++++++++++++++++++++
3 files changed, 236 insertions(+)
create mode 100644 doc/src/sgml/pgwalinspect.sgml
diff --git a/doc/src/sgml/contrib.sgml b/doc/src/sgml/contrib.sgml
index 1e42ce1a7f..4e7b87a42f 100644
--- a/doc/src/sgml/contrib.sgml
+++ b/doc/src/sgml/contrib.sgml
@@ -131,6 +131,7 @@ CREATE EXTENSION <replaceable>module_name</replaceable>;
&pgsurgery;
&pgtrgm;
&pgvisibility;
+ &pgwalinspect;
&postgres-fdw;
&seg;
&sepgsql;
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index fd853af01f..34c19c80f1 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -147,6 +147,7 @@
<!ENTITY pgsurgery SYSTEM "pgsurgery.sgml">
<!ENTITY pgtrgm SYSTEM "pgtrgm.sgml">
<!ENTITY pgvisibility SYSTEM "pgvisibility.sgml">
+<!ENTITY pgwalinspect SYSTEM "pgwalinspect.sgml">
<!ENTITY postgres-fdw SYSTEM "postgres-fdw.sgml">
<!ENTITY seg SYSTEM "seg.sgml">
<!ENTITY contrib-spi SYSTEM "contrib-spi.sgml">
diff --git a/doc/src/sgml/pgwalinspect.sgml b/doc/src/sgml/pgwalinspect.sgml
new file mode 100644
index 0000000000..8ab13c38c4
--- /dev/null
+++ b/doc/src/sgml/pgwalinspect.sgml
@@ -0,0 +1,234 @@
+<!-- doc/src/sgml/pgwalinspect.sgml -->
+
+<sect1 id="pgwalinspect" xreflabel="pg_walinspect">
+ <title>pg_walinspect</title>
+
+ <indexterm zone="pgwalinspect">
+ <primary>pg_walinspect</primary>
+ </indexterm>
+
+ <para>
+ The <filename>pg_walinspect</filename> module provides functions that allow
+ you to inspect the contents of write-ahead log of <productname>PostgreSQL</productname>
+ database cluster at a low level, which is useful for debugging or analytical
+ or reporting or educational purposes.
+ </para>
+
+ <para>
+ All the functions of this module will provide the WAL information using the
+ current server's timeline ID.
+ </para>
+
+ <para>
+ By default, use of these functions is restricted to superusers and members of
+ the <literal>pg_read_server_files</literal> role. Access may be granted by
+ superusers to others using <command>GRANT</command>.
+ </para>
+
+ <sect2>
+ <title>General Functions</title>
+
+ <variablelist>
+ <varlistentry>
+ <term>
+ <function>
+ pg_get_wal_record(in_lsn pg_lsn,
+ start_lsn OUT pg_lsn,
+ end_lsn OUT pg_lsn,
+ record_length OUT int4,
+ record OUT bytea)
+ </function>
+ </term>
+
+ <listitem>
+ <para>
+ Gets raw WAL record data of a given LSN. This function emits an error if
+ a future (the LSN database system doesn't know about)
+ <replaceable>in_lsn</replaceable> is specified.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <function>
+ pg_get_wal_record_info(in_lsn pg_lsn,
+ start_lsn OUT pg_lsn,
+ end_lsn OUT pg_lsn,
+ prev_lsn OUT pg_lsn,
+ xid OUT xid,
+ resource_manager OUT text,
+ record_length OUT int4,
+ fpi_length OUT int4,
+ description OUT text,
+ block_ref OUT text,
+ data_length OUT int4,
+ data OUT bytea)
+ </function>
+ </term>
+
+ <listitem>
+ <para>
+ Gets WAL record information of a given LSN. This function emits an error
+ if a future (the LSN database system doesn't know about)
+ <replaceable>in_lsn</replaceable> is specified.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <function>
+ pg_get_wal_records_info(start_lsn pg_lsn,
+ end_lsn pg_lsn,
+ start_lsn OUT pg_lsn,
+ end_lsn OUT pg_lsn,
+ prev_lsn OUT pg_lsn,
+ xid OUT xid,
+ resource_manager OUT text,
+ record_length OUT int4,
+ fpi_length OUT int4,
+ description OUT text,
+ block_ref OUT text,
+ data_length OUT int4,
+ data OUT bytea)
+ returns setof record
+ </function>
+ </term>
+
+ <listitem>
+ <para>
+ Gets information of all the valid WAL records between
+ <replaceable>start_lsn</replaceable> and <replaceable>end_lsn</replaceable>.
+ Returns one row per each valid WAL record. This function emits an error
+ if a future (the LSN database system doesn't know about)
+ <replaceable>start_lsn</replaceable> or <replaceable>end_lsn</replaceable>
+ is specified. For example, usage of the function is as follows:
+<screen>
+postgres=# select start_lsn, end_lsn, prev_lsn, xid, resource_manager, record_length, fpi_length, block_ref, data_length from pg_get_wal_records_info('0/1401790', '0/1401D88');
+ start_lsn | end_lsn | prev_lsn | xid | resource_manager | record_length | fpi_length | block_ref | data_length
+-----------+-----------+-----------+-----+------------------+---------------+------------+----------------------------------------------------+-------------
+ 0/14018B0 | 0/14018EF | 0/1401788 | 673 | Btree | 64 | 0 | blkref #0: rel 1663/1/2696 fork main blk 1 | 2
+ 0/14018F0 | 0/1401A17 | 0/14018B0 | 673 | Heap | 290 | 0 | blkref #0: rel 1663/1/2619 fork main blk 1 | 3
+ 0/1401A18 | 0/1401A57 | 0/14018F0 | 673 | Btree | 64 | 0 | blkref #0: rel 1663/1/2696 fork main blk 1 | 2
+ 0/1401A58 | 0/1401B47 | 0/1401A18 | 673 | Heap | 234 | 0 | blkref #0: rel 1663/1/2619 fork main blk 1 | 3
+ 0/1401B48 | 0/1401B87 | 0/1401A58 | 673 | Btree | 64 | 0 | blkref #0: rel 1663/1/2696 fork main blk 1 | 2
+ 0/1401B88 | 0/1401C67 | 0/1401B48 | 673 | Heap | 222 | 0 | blkref #0: rel 1663/1/2619 fork main blk 1 | 3
+ 0/1401C68 | 0/1401CA7 | 0/1401B88 | 673 | Btree | 64 | 0 | blkref #0: rel 1663/1/2696 fork main blk 1 | 2
+ 0/1401CA8 | 0/1401D87 | 0/1401C68 | 673 | Heap | 217 | 0 | blkref #0: rel 1663/1/2619 fork main blk 1 | 3
+(8 rows)
+</screen>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <function>
+ pg_get_wal_records_info_till_end_of_wal(start_lsn pg_lsn,
+ start_lsn OUT pg_lsn,
+ end_lsn OUT pg_lsn,
+ prev_lsn OUT pg_lsn,
+ xid OUT xid,
+ resource_manager OUT text,
+ record_length OUT int4,
+ fpi_length OUT int4,
+ description OUT text,
+ block_ref OUT text,
+ data_length OUT int4,
+ data OUT bytea)
+ returns setof record
+ </function>
+ </term>
+
+ <listitem>
+ <para>
+ This function is same as <function>pg_get_wal_records_info()</function>
+ except that it gets information of all the valid WAL records from
+ <replaceable>start_lsn</replaceable> till end of WAL.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <function>
+ pg_get_wal_stats(start_lsn pg_lsn,
+ end_lsn pg_lsn,
+ resource_manager OUT text,
+ count OUT int8,
+ count_percentage OUT float4,
+ record_length OUT int8,
+ record_length_percentage OUT float4,
+ fpi_length OUT int8,
+ fpi_length_percentage OUT float4,
+ combined_size OUT int8,
+ combined_size_percentage OUT float4)
+ returns setof record
+ </function>
+ </term>
+
+ <listitem>
+ <para>
+ Gets statistics of all the valid WAL records between
+ <replaceable>start_lsn</replaceable> and <replaceable>end_lsn</replaceable>.
+ Returns one row per each valid WAL record. This function emits an error
+ if a future (the LSN database system doesn't know about)
+ <replaceable>start_lsn</replaceable> or <replaceable>end_lsn</replaceable>
+ is specified. For example, usage of the function is as follows:
+<screen>
+postgres=# select * from pg_get_wal_stats('0/13E80C0', '0/1600000') where count > 0;
+ resource_manager | count | count_percentage | record_size | record_size_percentage | fpi_size | fpi_size_percentage | combined_size | combined_size_percentage
+------------------+-------+------------------+-------------+------------------------+----------+---------------------+---------------+--------------------------
+ XLOG | 6 | 0.032156065 | 600 | 1.3965058e-05 | 0 | 0 | 600 | 1.3963012e-05
+ Transaction | 61 | 0.32691997 | 16399 | 0.00038168833 | 0 | 0 | 16399 | 0.0003816324
+ Storage | 1 | 0.005359344 | 42 | 9.775541e-07 | 0 | 0 | 42 | 9.774109e-07
+ Database | 2 | 0.010718688 | 84 | 1.9551082e-06 | 0 | 0 | 84 | 1.9548218e-06
+ Standby | 147 | 0.78782356 | 12726 | 0.0002961989 | 0 | 0 | 12726 | 0.0002961555
+ Heap2 | 804 | 4.3089128 | 190283 | 0.0044288556 | 374096 | 59.41106 | 564379 | 0.013134051
+ Heap | 16790 | 89.98338 | 1191835 | 0.027740076 | 157528 | 25.01739 | 1349363 | 0.031401955
+ Btree | 848 | 4.5447235 | 4295025395 | 99.96713 | 98050 | 15.571549 | 4295123445 | 99.95477
+(8 rows)
+</screen>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <function>
+ pg_get_wal_stats_till_end_of_wal(start_lsn pg_lsn,
+ resource_manager OUT text,
+ count OUT int8,
+ count_percentage OUT float4,
+ record_length OUT int8,
+ record_length_percentage OUT float4,
+ fpi_length OUT int8,
+ fpi_length_percentage OUT float4,
+ combined_size OUT int8,
+ combined_size_percentage OUT float4)
+ returns setof record
+ </function>
+ </term>
+
+ <listitem>
+ <para>
+ This function is same as <function>pg_get_wal_stats()</function> except
+ that it gets stats of all the valid WAL records from
+ <replaceable>start_lsn</replaceable> till end of WAL.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </sect2>
+
+ <sect2>
+ <title>Author</title>
+
+ <para>
+ Bharath Rupireddy <email>[email protected]</email>
+ </para>
+ </sect2>
+
+</sect1>
--
2.25.1
view thread (67+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: pg_walinspect - a new extension to get raw WAL data and WAL stats
In-Reply-To: <CALj2ACWHxU2brc1vRJKxmtgr6hAB6YnL9fjVdN_qEvv1HiWLFQ@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox