Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1osXLU-0008F9-I1 for pgsql-hackers@arkaria.postgresql.org; Tue, 08 Nov 2022 22:45:28 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1osXLT-0002k9-Bn for pgsql-hackers@arkaria.postgresql.org; Tue, 08 Nov 2022 22:45:27 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1osXLS-0002jy-S2 for pgsql-hackers@lists.postgresql.org; Tue, 08 Nov 2022 22:45:27 +0000 Received: from mail-qv1-xf2d.google.com ([2607:f8b0:4864:20::f2d]) by magus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1osXLP-0005VR-TJ for pgsql-hackers@postgresql.org; Tue, 08 Nov 2022 22:45:26 +0000 Received: by mail-qv1-xf2d.google.com with SMTP id w10so11234649qvr.3 for ; Tue, 08 Nov 2022 14:45:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telsasoft-com.20210112.gappssmtp.com; s=20210112; h=user-agent:in-reply-to:content-disposition:mime-version:references :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=Rn0bYQ8wbosvRiUm3gxwcAOMiFEeQ1q6o0aGgGSe14Y=; b=srnKQcteVKEKoxFKc7v17x4CwY/xfvlpzbdjYFjQFAePZ0v7QI55ZZNb/AQK+6/x9U sSbbYnjLTaEzEwyO5t/LmqNvv9NzO4mgeeGdKAn4pSzsx3y+tQbkvYz/mA/ii16D56EB 2MnWLtnazvs+4Vh77Ia5GwRg8AGOTqT9NbdwGLt45u42BOQcBwBHhwiGbVGM8WPxC3HO GVN8hpCPkjLbi6hHEpAyATYIDRWUve+KQh6l7saZs7E9SqaSY5HPdaStey8nIEuVAg4R 4sZjS47B9+cZdRs6zNWKWdUArhr7bIWfxee4OvI++XlTWzX0MFXBT4mq3p+6s00zooqH o/vg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=user-agent:in-reply-to:content-disposition:mime-version:references :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Rn0bYQ8wbosvRiUm3gxwcAOMiFEeQ1q6o0aGgGSe14Y=; b=pFmqRjiwVJJlXAVXYf30cX3o8cS8qMU1AoLUi5uwJdnFnhNabzqWaRpN6Khza4SaqY rga3uQKVfaaepnGogomy4YjDJd4SrWk3WWGcY8s8pIL30ciEZ9oT1CZz84cZ7kwKPWP5 vdYTNQ9rIXuMO8sZtbgnOYPnllAV2W0IpwVIq6qYmO//6xmqsKmkEhtk7cZ9E+M5E979 tBnD867+snXUa7cXyBm8J6ahov2mwDyhrdN7KR5hORKiBcSk36aeFOReY5gDvnt1eT6G S7jxYakb1GPCU+5132/qJ0f86HkQcMJLMrLf8e2qkpCeUD6GqSYqkKjFYgW94wtOQwDY q4BQ== X-Gm-Message-State: ACrzQf0PCs5rl0phnm2ElxL4dIKi6g/HvEToYbkGtVMxCiBbKECgO5cy DBP8wksyyexe8VuI+b/eb45hjQ== X-Google-Smtp-Source: AMsMyM7mccFDHP8pqzDQbvW34A5z5D0x5gA/g2M1lvXb08DNF3fR18zRxgXtgLy8IZFaRR8YDz38iQ== X-Received: by 2002:ad4:5ca7:0:b0:4bb:97ea:27b5 with SMTP id q7-20020ad45ca7000000b004bb97ea27b5mr50977161qvh.104.1667947521996; Tue, 08 Nov 2022 14:45:21 -0800 (PST) Received: from pryzbyj.telsasoft (charmander.telsasoft.com. [50.244.222.1]) by smtp.gmail.com with ESMTPSA id bs30-20020a05620a471e00b006bbc3724affsm10069521qkb.45.2022.11.08.14.45.20 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 08 Nov 2022 14:45:21 -0800 (PST) Received: by pryzbyj.telsasoft (Postfix, from userid 1000) id 04C98801307; Tue, 8 Nov 2022 16:45:19 -0600 (CST) Date: Tue, 8 Nov 2022 16:45:19 -0600 From: Justin Pryzby To: David Christensen Cc: Ian Lawrence Barwick , Michael Paquier , Matthias van de Meent , pgsql-hackers@postgresql.org Subject: Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL Message-ID: <20221108224519.GH3775@telsasoft.com> References: <20221104140238.GA16921@telsasoft.com> <4E379ADE-1FC3-4D53-BE46-562BDC77C5FE@crunchydata.com> <20221104183838.GD16921@telsasoft.com> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="phCU5ROyZO6kBE05" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.9.4 (2018-02-28) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --phCU5ROyZO6kBE05 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Mon, Nov 07, 2022 at 05:01:01PM -0600, David Christensen wrote: > Hi Justin et al, > > Enclosed is v5 of this patch which now passes the CirrusCI checks for > all supported OSes. I went ahead and reworked the test a bit so it's a > little more amenable to the OS-agnostic approach for testing. Great, thanks. This includes the changes that I'd started a few months ago. Plus adding the test which was missing for meson. + format: LSN.TSOID.DBOID.RELNODE.BLKNO I'd prefer if the abbreviations were "reltablespace" and "datoid" Also, should the test case call pg_relation_filenode() rather than using relfilenode directly ? Is it a problem that the test code assumes pagesize=8192 ? -- Justin --phCU5ROyZO6kBE05 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0001-Teach-pg_waldump-to-extract-FPIs-from-the-WAL-stream.patch" From d4cb23bc2f0edcc65b0bef75fd2e8d6e5aeda21f Mon Sep 17 00:00:00 2001 From: David Christensen Date: Wed, 20 Apr 2022 19:59:35 -0500 Subject: [PATCH 1/2] Teach pg_waldump to extract FPIs from the WAL stream Extracts full-page images from the WAL stream into a target directory, which must be empty or not exist. These images are subject to the same filtering rules as normal display in pg_waldump, which means that you can isolate the full page writes to a target relation, among other things. Files are saved with the filename: .... with formatting to make things somewhat sortable; for instance: 00000000-010000C0.1663.1.6117.0 00000000-01000150.1664.0.6115.0 00000000-010001E0.1664.0.6114.0 00000000-01000270.1663.1.6116.0 00000000-01000300.1663.1.6113.0 00000000-01000390.1663.1.6112.0 00000000-01000420.1663.1.8903.0 00000000-010004B0.1663.1.8902.0 00000000-01000540.1663.1.6111.0 00000000-010005D0.1663.1.6110.0 If the FPI comes from a fork other than the main fork, the fork name will be appended on the output file name; e.g.: 00000000-014A4758.1663.1.12864.0_vm It's noteworthy that the raw block images do not have the current LSN stored with them in the WAL stream (as would be true for on-heap versions of the blocks), nor would the checksum be updated in them (though WAL itself has checksums, so there is some protection there). As such there are two versions of this functionality, one which returns the raw page as it appears in the WAL (--save-fpi) and one which applies the updated pd_lsn and pd_checksum (--fixup-fpi). These images could be loaded/inspected via `pg_read_binary_file()` and used in the `pageinspect` suite of tools to perform detailed analysis on the pages in question, based on historical information, and may come in handy for forensics work. --- doc/src/sgml/ref/pg_waldump.sgml | 79 +++++++++++ src/bin/pg_waldump/pg_waldump.c | 151 +++++++++++++++++++++- src/bin/pg_waldump/t/002_save_fullpage.pl | 137 ++++++++++++++++++++ 3 files changed, 366 insertions(+), 1 deletion(-) create mode 100644 src/bin/pg_waldump/t/002_save_fullpage.pl diff --git a/doc/src/sgml/ref/pg_waldump.sgml b/doc/src/sgml/ref/pg_waldump.sgml index d559f091e5f..cc3ce918905 100644 --- a/doc/src/sgml/ref/pg_waldump.sgml +++ b/doc/src/sgml/ref/pg_waldump.sgml @@ -240,6 +240,85 @@ PostgreSQL documentation + + + + + + + + Save full page images seen in the WAL stream to the + save_path directory, which will be created + if it does not exist. The images saved will be subject to the same + filtering and limiting criteria as display records, but in this + mode pg_waldump will not output any other + information. + + + If invoked using + the -X/--fixup-fpi + option, this page image will include the pd_lsn of + the replayed record rather than the raw page image; as well, + the pd_checksum field will be updated if it had + previously existed. + + + The page images will be saved with the file + format: LSN.TSOID.DBOID.RELNODE.BLKNOFORK + + The dot-separated components are (in order): + + + + + + Component + Description + + + + + + LSN + The LSN of the record with this block, formatted + as two 8-character hexadecimal numbers %08X-%08X + + + + TSOID + tablespace OID for the block + + + + DBOID + database OID for the block + + + + RELNODE + relnode id for the block + + + + BLKNO + the block number of this block + + + + FORK + + if coming from the main fork, will be empty, otherwise will be + one of _fsm, _vm, + or _init. + + + + + + + + + diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c index 9993378ca58..2b6f772dc71 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@ -24,8 +24,12 @@ #include "access/xlogstats.h" #include "common/fe_memutils.h" #include "common/logging.h" +#include "common/relpath.h" #include "getopt_long.h" #include "rmgrdesc.h" +#include "storage/bufpage.h" +#include "storage/checksum.h" +#include "storage/checksum_impl.h" /* * NOTE: For any code change or issue fix here, it is highly recommended to @@ -70,6 +74,11 @@ typedef struct XLogDumpConfig bool filter_by_relation_block_enabled; ForkNumber filter_by_relation_forknum; bool filter_by_fpw; + + /* save options */ + bool save_fpw; + bool fixup_fpw; + char *save_fpw_path; } XLogDumpConfig; @@ -439,6 +448,109 @@ XLogRecordHasFPW(XLogReaderState *record) return false; } +/* + * Function to externally save all FPWs stored in the given WAL record + */ +static void +XLogRecordSaveFPWs(XLogReaderState *record, const char *savepath, bool fixup) +{ + int block_id; + + for (block_id = 0; block_id <= XLogRecMaxBlockId(record); block_id++) + { + if (!XLogRecHasBlockRef(record, block_id)) + continue; + + if (XLogRecHasBlockImage(record, block_id)) + { + char page[BLCKSZ]; + + memset(page, 0, BLCKSZ); + + if (RestoreBlockImage(record, block_id, page)) + { + /* we have our extracted FPI, let's save it now */ + char filename[MAXPGPATH]; + char forkname[FORKNAMECHARS + 2]; /* _ + \0 */ + FILE *OPF; + BlockNumber blk; + RelFileLocator rnode; + ForkNumber fork; + + XLogRecGetBlockTagExtended(record, block_id, + &rnode, &fork, &blk, NULL); + + /* + * The raw page as stored in the WAL record includes the LSN + * of the block as it appeared when it was originally written, + * however this differs than the effects of replaying this + * same FPI in recovery, as recovery calls RestoreBlockImage() + * and then sets the LSN as part of one action. What this + * means is that a page as recovered from WAL and the version + * of the page saved here will differ by the LSN and the + * checksum (if enabled). + * + * There are potentially use-cases for both versions (with and + * without mentioned fixups), so allow this to be + * user-selected, unless the restored page was empty, in which + * case we leave it alone. + */ + + if (fixup && !PageIsNew(page)) + { + PageSetLSN(page, record->ReadRecPtr); + + /* + * If checksum field is non-zero then we have checksums + * enabled, so recalculate the checksum with new LSN + * (whether this is considered a hack or heuristics is an + * exercise for the reader). + * + * We make this choice to allow pages saved by this + * function to work as expected with the checksum-related + * functions in pageinspect without having to worry about + * zero_damaged_pages or other considerations. + * + * FPIs in WAL do not have the checksum field updated in + * the page image; in a checksums-enabled cluster, this + * task is handled by FlushBuffer() when a dirty buffer is + * written out to disk. Since we are running outside of + * Postmaster that won't work in this case, so we handle + * ourselves. + */ + + if (((PageHeader) page)->pd_checksum) + ((PageHeader) page)->pd_checksum = pg_checksum_page((char *) page, blk); + } + + if (fork >= 0 && fork <= MAX_FORKNUM) + { + if (fork) + sprintf(forkname, "_%s", forkNames[fork]); + else + forkname[0] = 0; + } + else + pg_fatal("found invalid fork number: %u", fork); + + snprintf(filename, MAXPGPATH, "%s/%08X-%08X.%u.%u.%u.%u%s", savepath, + LSN_FORMAT_ARGS(record->ReadRecPtr), + rnode.spcOid, rnode.dbOid, rnode.relNumber, blk, forkname); + + OPF = fopen(filename, PG_BINARY_W); + if (!OPF) + pg_fatal("couldn't open file for output: %s", filename); + + if (pg_pwrite(fileno(OPF), page, BLCKSZ, 0) != BLCKSZ) + pg_fatal("couldn't write out complete fullpage image to file: %s", filename); + + fsync(fileno(OPF)); + fclose(OPF); + } + } + } +} + /* * Print a record to stdout */ @@ -679,6 +791,9 @@ usage(void) " (default: 1 or the value used in STARTSEG)\n")); printf(_(" -V, --version output version information, then exit\n")); printf(_(" -w, --fullpage only show records with a full page write\n")); + printf(_(" -W, --save-fpi=path save found full page images to given path\n" + " as LSN.T.D.R.B_F\n")); + printf(_(" -X, --fixup-fpi=path like --save-fpi but apply LSN fixups to \n")); printf(_(" -x, --xid=XID only show records with transaction ID XID\n")); printf(_(" -z, --stats[=record] show statistics instead of records\n" " (optionally, show per-record statistics)\n")); @@ -712,6 +827,8 @@ main(int argc, char **argv) {"limit", required_argument, NULL, 'n'}, {"path", required_argument, NULL, 'p'}, {"quiet", no_argument, NULL, 'q'}, + {"save-fpi", required_argument, NULL, 'W'}, + {"fixup-fpi", required_argument, NULL, 'X'}, {"relation", required_argument, NULL, 'R'}, {"rmgr", required_argument, NULL, 'r'}, {"start", required_argument, NULL, 's'}, @@ -772,6 +889,9 @@ main(int argc, char **argv) config.filter_by_fpw = false; config.stats = false; config.stats_per_record = false; + config.save_fpw = false; + config.fixup_fpw = false; + config.save_fpw_path = NULL; stats.startptr = InvalidXLogRecPtr; stats.endptr = InvalidXLogRecPtr; @@ -782,7 +902,7 @@ main(int argc, char **argv) goto bad_argument; } - while ((option = getopt_long(argc, argv, "bB:e:fF:n:p:qr:R:s:t:wx:z", + while ((option = getopt_long(argc, argv, "bB:e:fF:n:p:qr:R:s:t:wW:x:X:z", long_options, &optindex)) != -1) { switch (option) @@ -918,6 +1038,12 @@ main(int argc, char **argv) case 'w': config.filter_by_fpw = true; break; + case 'W': + case 'X': + config.save_fpw = true; + config.fixup_fpw = (option == 'X'); + config.save_fpw_path = pg_strdup(optarg); + break; case 'x': if (sscanf(optarg, "%u", &config.filter_by_xid) != 1) { @@ -972,6 +1098,24 @@ main(int argc, char **argv) } } + if (config.save_fpw_path != NULL) + { + int dir_status = pg_check_dir(config.save_fpw_path); + + if (dir_status < 0) + { + pg_log_error("could not access output directory: %s", config.save_fpw_path); + goto bad_argument; + } + + if (!dir_status && mkdir(config.save_fpw_path, 0700) < 0) + { + pg_log_error("could not create output directory \"%s\": %m", + config.save_fpw_path); + goto bad_argument; + } + } + /* parse files as start/end boundaries, extract path if not specified */ if (optind < argc) { @@ -1150,6 +1294,11 @@ main(int argc, char **argv) XLogRecStoreStats(&stats, xlogreader_state); stats.endptr = xlogreader_state->EndRecPtr; } + else if (config.save_fpw) + { + if (XLogRecordHasFPW(xlogreader_state)) + XLogRecordSaveFPWs(xlogreader_state, config.save_fpw_path, config.fixup_fpw); + } else XLogDumpDisplayRecord(&config, xlogreader_state); } diff --git a/src/bin/pg_waldump/t/002_save_fullpage.pl b/src/bin/pg_waldump/t/002_save_fullpage.pl new file mode 100644 index 00000000000..f605b4f6adb --- /dev/null +++ b/src/bin/pg_waldump/t/002_save_fullpage.pl @@ -0,0 +1,137 @@ + +# Copyright (c) 2022, PostgreSQL Global Development Group + +use strict; +use warnings; +use File::Basename; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::RecursiveCopy; +use PostgreSQL::Test::Utils; +use Test::More; + +# routine to extract the LSN and checksum from the given block structure +sub get_block_info +{ + my $path = shift; + my $block; + + open my $fh, '<', $path or die "couldn't open file: $path\n"; + die "couldn't read full block\n" if 8192 != read $fh, $block, 8192; + my ($lsn_hi, $lsn_lo, $checksum) = unpack('VVv', $block); + + $lsn_hi = sprintf('%08X', $lsn_hi); + $lsn_lo = sprintf('%08X', $lsn_lo); + + return ($lsn_hi, $lsn_lo, $checksum); +} + +# Set umask so test directories and files are created with default permissions +umask(0077); + +my $primary = PostgreSQL::Test::Cluster->new('primary'); +$primary->init('-k'); +$primary->append_conf('postgresql.conf', "max_wal_size='100MB'"); +$primary->append_conf('postgresql.conf', "wal_level='replica'"); +$primary->append_conf('postgresql.conf', 'archive_mode=on'); +$primary->append_conf('postgresql.conf', "archive_command='/bin/false'"); +$primary->start; + +# Sanity checks for command line options. +$primary->command_fails( + [ 'pg_waldump', '--save-fpi' ], + '--save-fpi fails without path'); +$primary->command_fails( + [ 'pg_waldump', '--fixup-fpi' ], + '--fixup-fpi fails without path'); + +# generate data/wal to examine +$primary->safe_psql('postgres', q(CREATE DATABASE db1)); +$primary->safe_psql('db1', <safe_psql('db1', + q{SELECT CASE WHEN reltablespace = 0 THEN dattablespace ELSE reltablespace END || '/' || pg_database.oid || '/' || relfilenode FROM pg_class, pg_database WHERE relname = 'test_table' AND datname = current_database()} +); + +diag $relation; + +$primary->stop; +my $waldir = $primary->basedir . '/pgdata/pg_wal'; +my $walfile = [glob("$waldir/*")]->[2]; # we want the 00000002 file +my $tmp_folder = PostgreSQL::Test::Utils::tempdir; +diag "using walfile: $walfile"; + +# # # extract files +# PostgreSQL::Test::RecursiveCopy::copypath( +# $primary->data_dir . "/pg_wal", +# $primary->tmp_folder); + +$primary->command_ok(['pg_waldump', '--save-fpi', "$tmp_folder/raw", '--relation', $relation, $walfile]); +$primary->command_ok(['pg_waldump', '--fixup-fpi', "$tmp_folder/fixup", '--relation', $relation, $walfile]); + +my $file_re = + qr/^([0-9A-F]{8})-([0-9A-F]{8})[.][0-9]+[.][0-9]+[.][0-9]+[.][0-9]+(?:_vm|_init|_fsm)?$/; + +my %checksums; +my %files; + +# verify filename formats matches w/--save-fpi +for my $fullpath (glob "$tmp_folder/raw/*") +{ + my $file = File::Basename::basename($fullpath); + + like($file, $file_re, "verify filename format for file $file"); + + # save filename for later verification + $files{$file}++; + + my ($hi_lsn_fn, $lo_lsn_fn) = ($file =~ $file_re); + my ($hi_lsn_bk, $lo_lsn_bk, $checksum) = get_block_info($fullpath); + + # since no fixup, verify the lsn in the block comes before the file's lsn + ok( $hi_lsn_fn . $lo_lsn_fn gt $hi_lsn_bk . $lo_lsn_bk, + 'verify file-based LSN precedes block-based one'); + + # stash checksum for later comparisons + $checksums{$file} = $checksum; +} + +# verify filename formats matches w/--fixup-fpi +for my $fullpath (glob "$tmp_folder/fixup/*") +{ + my $file = File::Basename::basename($fullpath); + + like($file, $file_re, "verify filename format for file $file"); + + # save filename for later verification + $files{$file}++; + + my ($hi_lsn_fn, $lo_lsn_fn) = ($file =~ $file_re); + my ($hi_lsn_bk, $lo_lsn_bk, $checksum) = get_block_info($fullpath); + + # since fixup, verify the lsn in the block equals file lsn + ok( $hi_lsn_fn . $lo_lsn_fn eq $hi_lsn_bk . $lo_lsn_bk, + 'verify file-based LSN is the same as block-based one'); + + # verify checksum change; XXX: there could be valid clashes here, + # just validate that the page matches the expected checksum instead? + ok( $checksum == 0 || $checksums{$file} != $checksum, + 'check for checksum change or no checksum'); +} + +# validate that we ended up with some files output and they were the same +ok(keys %files > 0, 'verify we processed some files'); +ok((grep { $_ != 2 } values %files) == 0, + 'ensure raw and fixup had same number of files'); + +done_testing(); -- 2.25.1 --phCU5ROyZO6kBE05 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-justin.patch" From 3c2c369c495760d8f96d3cdeab623c4f0372792a Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Thu, 15 Sep 2022 13:39:53 -0500 Subject: [PATCH 2/2] f!justin ci-os-only: freebsd --- src/bin/pg_waldump/meson.build | 1 + src/bin/pg_waldump/pg_waldump.c | 172 +++++++++++----------- src/bin/pg_waldump/t/002_save_fullpage.pl | 2 +- 3 files changed, 86 insertions(+), 89 deletions(-) diff --git a/src/bin/pg_waldump/meson.build b/src/bin/pg_waldump/meson.build index 9605976870d..34e37bffc36 100644 --- a/src/bin/pg_waldump/meson.build +++ b/src/bin/pg_waldump/meson.build @@ -29,6 +29,7 @@ tests += { 'tap': { 'tests': [ 't/001_basic.pl', + 't/002_save_fullpage.pl', ], }, } diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c index 2b6f772dc71..e35d15132f4 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@ -76,7 +76,6 @@ typedef struct XLogDumpConfig bool filter_by_fpw; /* save options */ - bool save_fpw; bool fixup_fpw; char *save_fpw_path; } XLogDumpConfig; @@ -458,96 +457,94 @@ XLogRecordSaveFPWs(XLogReaderState *record, const char *savepath, bool fixup) for (block_id = 0; block_id <= XLogRecMaxBlockId(record); block_id++) { + char page[BLCKSZ] = {0}; + char filename[MAXPGPATH]; + char forkname[FORKNAMECHARS + 2]; /* _ + \0 */ + FILE *OPF; + BlockNumber blk; + RelFileLocator rnode; + ForkNumber fork; + if (!XLogRecHasBlockRef(record, block_id)) continue; - if (XLogRecHasBlockImage(record, block_id)) - { - char page[BLCKSZ]; - - memset(page, 0, BLCKSZ); - - if (RestoreBlockImage(record, block_id, page)) - { - /* we have our extracted FPI, let's save it now */ - char filename[MAXPGPATH]; - char forkname[FORKNAMECHARS + 2]; /* _ + \0 */ - FILE *OPF; - BlockNumber blk; - RelFileLocator rnode; - ForkNumber fork; - - XLogRecGetBlockTagExtended(record, block_id, - &rnode, &fork, &blk, NULL); - - /* - * The raw page as stored in the WAL record includes the LSN - * of the block as it appeared when it was originally written, - * however this differs than the effects of replaying this - * same FPI in recovery, as recovery calls RestoreBlockImage() - * and then sets the LSN as part of one action. What this - * means is that a page as recovered from WAL and the version - * of the page saved here will differ by the LSN and the - * checksum (if enabled). - * - * There are potentially use-cases for both versions (with and - * without mentioned fixups), so allow this to be - * user-selected, unless the restored page was empty, in which - * case we leave it alone. - */ - - if (fixup && !PageIsNew(page)) - { - PageSetLSN(page, record->ReadRecPtr); + if (!XLogRecHasBlockImage(record, block_id)) + continue; - /* - * If checksum field is non-zero then we have checksums - * enabled, so recalculate the checksum with new LSN - * (whether this is considered a hack or heuristics is an - * exercise for the reader). - * - * We make this choice to allow pages saved by this - * function to work as expected with the checksum-related - * functions in pageinspect without having to worry about - * zero_damaged_pages or other considerations. - * - * FPIs in WAL do not have the checksum field updated in - * the page image; in a checksums-enabled cluster, this - * task is handled by FlushBuffer() when a dirty buffer is - * written out to disk. Since we are running outside of - * Postmaster that won't work in this case, so we handle - * ourselves. - */ + if (!RestoreBlockImage(record, block_id, page)) + continue; - if (((PageHeader) page)->pd_checksum) - ((PageHeader) page)->pd_checksum = pg_checksum_page((char *) page, blk); - } + /* we have our extracted FPI, let's save it now */ + + XLogRecGetBlockTagExtended(record, block_id, + &rnode, &fork, &blk, NULL); + + /* + * The raw page as stored in the WAL record includes the LSN + * of the block as it appeared when it was originally written, + * however this differs than the effects of replaying this + * same FPI in recovery, as recovery calls RestoreBlockImage() + * and then sets the LSN as part of one action. What this + * means is that a page as recovered from WAL and the version + * of the page saved here will differ by the LSN and the + * checksum (if enabled). + * + * There are potentially use-cases for both versions (with and + * without mentioned fixups), so allow this to be + * user-selected, unless the restored page was empty, in which + * case we leave it alone. + */ + + if (fixup && !PageIsNew(page)) + { + PageSetLSN(page, record->ReadRecPtr); + + /* + * If checksum field is non-zero then we have checksums + * enabled, so recalculate the checksum with new LSN + * (whether this is considered a hack or heuristics is an + * exercise for the reader). + * + * We make this choice to allow pages saved by this + * function to work as expected with the checksum-related + * functions in pageinspect without having to worry about + * zero_damaged_pages or other considerations. + * + * FPIs in WAL do not have the checksum field updated in + * the page image; in a checksums-enabled cluster, this + * task is handled by FlushBuffer() when a dirty buffer is + * written out to disk. Since we are running outside of + * Postmaster that won't work in this case, so we handle + * ourselves. + */ + + if (((PageHeader) page)->pd_checksum) + ((PageHeader) page)->pd_checksum = pg_checksum_page((char *) page, blk); + } - if (fork >= 0 && fork <= MAX_FORKNUM) - { - if (fork) - sprintf(forkname, "_%s", forkNames[fork]); - else - forkname[0] = 0; - } - else - pg_fatal("found invalid fork number: %u", fork); + if (fork >= 0 && fork <= MAX_FORKNUM) + { + if (fork) + sprintf(forkname, "_%s", forkNames[fork]); + else + forkname[0] = 0; + } + else + pg_fatal("found invalid fork number: %u", fork); - snprintf(filename, MAXPGPATH, "%s/%08X-%08X.%u.%u.%u.%u%s", savepath, - LSN_FORMAT_ARGS(record->ReadRecPtr), - rnode.spcOid, rnode.dbOid, rnode.relNumber, blk, forkname); + snprintf(filename, MAXPGPATH, "%s/%08X-%08X.%u.%u.%u.%u%s", savepath, + LSN_FORMAT_ARGS(record->ReadRecPtr), + rnode.spcOid, rnode.dbOid, rnode.relNumber, blk, forkname); - OPF = fopen(filename, PG_BINARY_W); - if (!OPF) - pg_fatal("couldn't open file for output: %s", filename); + OPF = fopen(filename, PG_BINARY_W); + if (!OPF) + pg_fatal("couldn't open file for output: %s", filename); - if (pg_pwrite(fileno(OPF), page, BLCKSZ, 0) != BLCKSZ) - pg_fatal("couldn't write out complete fullpage image to file: %s", filename); + if (pg_pwrite(fileno(OPF), page, BLCKSZ, 0) != BLCKSZ) + pg_fatal("couldn't write out complete fullpage image to file: %s", filename); - fsync(fileno(OPF)); - fclose(OPF); - } - } + fsync(fileno(OPF)); + fclose(OPF); } } @@ -791,9 +788,9 @@ usage(void) " (default: 1 or the value used in STARTSEG)\n")); printf(_(" -V, --version output version information, then exit\n")); printf(_(" -w, --fullpage only show records with a full page write\n")); - printf(_(" -W, --save-fpi=path save found full page images to given path\n" - " as LSN.T.D.R.B_F\n")); - printf(_(" -X, --fixup-fpi=path like --save-fpi but apply LSN fixups to \n")); + printf(_(" -W, --save-fpi=path save full page images to given path as\n" + " LSN.T.D.R.B_F\n")); + printf(_(" -X, --fixup-fpi=path like --save-fpi but apply LSN fixups to saved page\n")); printf(_(" -x, --xid=XID only show records with transaction ID XID\n")); printf(_(" -z, --stats[=record] show statistics instead of records\n" " (optionally, show per-record statistics)\n")); @@ -889,7 +886,6 @@ main(int argc, char **argv) config.filter_by_fpw = false; config.stats = false; config.stats_per_record = false; - config.save_fpw = false; config.fixup_fpw = false; config.save_fpw_path = NULL; @@ -1040,7 +1036,6 @@ main(int argc, char **argv) break; case 'W': case 'X': - config.save_fpw = true; config.fixup_fpw = (option == 'X'); config.save_fpw_path = pg_strdup(optarg); break; @@ -1108,7 +1103,8 @@ main(int argc, char **argv) goto bad_argument; } - if (!dir_status && mkdir(config.save_fpw_path, 0700) < 0) + /* Create the dir if it doesn't exist */ + if (dir_status == 0 && mkdir(config.save_fpw_path, 0700) < 0) { pg_log_error("could not create output directory \"%s\": %m", config.save_fpw_path); @@ -1294,7 +1290,7 @@ main(int argc, char **argv) XLogRecStoreStats(&stats, xlogreader_state); stats.endptr = xlogreader_state->EndRecPtr; } - else if (config.save_fpw) + else if (config.save_fpw_path) { if (XLogRecordHasFPW(xlogreader_state)) XLogRecordSaveFPWs(xlogreader_state, config.save_fpw_path, config.fixup_fpw); diff --git a/src/bin/pg_waldump/t/002_save_fullpage.pl b/src/bin/pg_waldump/t/002_save_fullpage.pl index f605b4f6adb..6df670c39ba 100644 --- a/src/bin/pg_waldump/t/002_save_fullpage.pl +++ b/src/bin/pg_waldump/t/002_save_fullpage.pl @@ -60,7 +60,7 @@ EOF # get the relation node, etc for the new table my $relation = $primary->safe_psql('db1', - q{SELECT CASE WHEN reltablespace = 0 THEN dattablespace ELSE reltablespace END || '/' || pg_database.oid || '/' || relfilenode FROM pg_class, pg_database WHERE relname = 'test_table' AND datname = current_database()} + q{SELECT format('%s/%s/%s', CASE WHEN reltablespace = 0 THEN dattablespace ELSE reltablespace END, pg_database.oid, relfilenode) FROM pg_class, pg_database WHERE relname = 'test_table' AND datname = current_database()} ); diag $relation; -- 2.25.1 --phCU5ROyZO6kBE05--