MIME-Version: 1.0
References: <20221104140238.GA16921@telsasoft.com>
 <4E379ADE-1FC3-4D53-BE46-562BDC77C5FE@crunchydata.com>
 <20221104183838.GD16921@telsasoft.com>
 <CAOxo6X+2-k6gqwv2okbgLR_B_MkBCVHOW3CqFomFta2wNkHYdw@mail.gmail.com>
 <CAOxo6XJt=F3eKS8-jvFYoW2MVP+DCpseHPHYiLVYGF43iET30Q@mail.gmail.com>
 <20221108224519.GH3775@telsasoft.com>
 <CAOxo6XJ_pf+cKDXZMvxprvA1x-Pi+-22fq0Tw50o9HgkXg7cTw@mail.gmail.com>
 <CAOxo6XLCwGGjLEK9--VK9SzOb4EFKiJuSrnOePSPfEyr-pLKPw@mail.gmail.com>
 <CALj2ACWU7ckJLu=9Ag0AahnWT=1E22FsPu9wmcyAEJCv04BCvA@mail.gmail.com>
 <CAOxo6X+SO5Z02m3J6aR3n9NwMqPJcsaCDtyjNhELoqG218vwkg@mail.gmail.com>
 <CALj2ACVnA8tQBsFWPfk4062YVe=HhaypVsD5j6PCfJYDML9Q+Q@mail.gmail.com>
 <CAOxo6XJR8OkQ7K-N9Ekq4ao+VJBPApYmP+yV4Cp+mbweHb_iLg@mail.gmail.com>
 <CALj2ACWTQCYGHf=2pHQeX=8qfkMi-4JsXN0t_6W8+A-cW8wJpw@mail.gmail.com>
 <CAOxo6X+wXEu1QfTFTf-WiDs3jr8ERvgEOddHfUhAH_5eje-G2w@mail.gmail.com>
 <CAOxo6XJ03w=PWDnsnzDpGK+4Er7XZwFkhqiMbVWmt7VLHzidHw@mail.gmail.com>
In-Reply-To: 
 <CAOxo6XJ03w=PWDnsnzDpGK+4Er7XZwFkhqiMbVWmt7VLHzidHw@mail.gmail.com>
From: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Date: Tue, 15 Nov 2022 16:11:05 +0530
Message-ID: 
 <CALj2ACUukW74XXb2n3FFAdRiisF06jSNDR8DegOpMevq_0yVoA@mail.gmail.com>
Subject: Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL
To: David Christensen <david.christensen@crunchydata.com>
Cc: Justin Pryzby <pryzby@telsasoft.com>,
 Ian Lawrence Barwick <barwick@gmail.com>,
	Michael Paquier <michael@paquier.xyz>,
 Matthias van de Meent <boekewurm+postgres@gmail.com>,
	pgsql-hackers@postgresql.org
Content-Type: text/plain; charset="UTF-8"
Archived-At: 
 <https://www.postgresql.org/message-id/CALj2ACUukW74XXb2n3FFAdRiisF06jSNDR8DegOpMevq_0yVoA%40mail.gmail.com>
Precedence: bulk

On Tue, Nov 15, 2022 at 1:29 AM David Christensen
<david.christensen@crunchydata.com> wrote:
>
> Enclosed is v8, which uses the replication slot method to retain WAL
> as well as fsync'ing the output directory when everything is done.

Thanks. It mostly is in good shape. However, few more comments:

1.
+        if it does not exist.  The images saved will be subject to the same
+        filtering and limiting criteria as display records, but in this
+        mode <application>pg_waldump</application> will not output any other
+        information.
May I know what's the intention of the statement 'The images saved
....'? If it's not necessary and convey anything useful to the user,
can we remove it?

2.
+#include "storage/checksum.h"
+#include "storage/checksum_impl.h"
I think we don't need the above includes as we got rid of verifying
page checksums. The patch compiles without them for me.

3.
+    char       *save_fpw_path;
Can we rename the above variable to save_fpi_path, just to be in sync
with what we expose to the user, the option name 'save-fpi'?

4.
+    if (config.save_fpw_path != NULL)
+    {
+        /* Fsync our output directory */
+        fsync_fname(config.save_fpw_path, true);
+    }
I guess adding a comment there as to why we aren't fsyncing for every
file that gets created, but once per the directory at the end. That'd
help clarify doubts that other members might get while looking at the
code.

5.
+    if (config.save_fpw_path != NULL)
+    {
+        /* Fsync our output directory */
+        fsync_fname(config.save_fpw_path, true);
+    }
So, are we sure that we don't want to fsync for time_to_stop exit(0)
cases, say when CTRL+C'ed. Looks like we handle time_to_stop safely
meaning exiting with return code 0, shouldn't we fsync the directory?

6.
+            else if (config.save_fpw_path)
Let's use the same convention to check non-NULLness,
config.save_fpw_path != NULL.

7.
+CHECKPOINT;
+SELECT pg_switch_wal();
+UPDATE test_table SET a = a + 1;
+SELECT pg_switch_wal();
I don't think switching WAL after checkpoint is necessary here,
because the checkpoint ensures all the WAL gets flushed to disk.
Please remove it.

PS: I've seen the following code:
+my $walfile = [sort { $a <=> $b } glob("$waldir/00*")]->[1]; # we
want the second WAL file, which will be a complete WAL file with
full-page writes for our specific relation.

8.
+$node->safe_psql('postgres', <<EOF);
+EOF
Why EOF is used here? Can't we do something like below to execute
multiple statements?
$node->safe_psql(
    'postgres', qq[
    SELECT data FROM pg_logical_slot_get_changes('regression_slot1', NULL,
    NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
    SELECT data FROM pg_logical_slot_get_changes('regression_slot2', NULL,
    NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
    SELECT data FROM pg_logical_slot_get_changes('regression_slot3', NULL,
    NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
    SELECT data FROM pg_logical_slot_get_changes('regression_slot4', NULL,
    NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
]);

Same here:
+$node->safe_psql('postgres', <<EOQ);
+SELECT pg_drop_replication_slot('regress_pg_waldump_slot');
+EOQ

9.
+my $walfile = [sort { $a <=> $b } glob("$waldir/00*")]->[1]; # we
want the second WAL file, which will be a complete WAL file with
full-page writes for our specific relation.
Is it guaranteed that just looking at the second WAL file in the
pg_wal directory assures WAL file with FPIs? I think we have to save
the WAL file that contains FPIs, that is the file after, CHECKPOINT,
UPDATE and pg_switch_wal. I think you can store output LSN of
pg_switch_wal

10.
+$node->safe_psql('postgres', <<EOQ);
+SELECT pg_drop_replication_slot('regress_pg_waldump_slot');
+EOQ
+done_testing();

Do we need to explicitly drop the slot here? I think we don't
specifically drop the replication slot in all the places, my guess is
after done_testing(), the node would get destroyed and also the slot.
I think it's not required.

11.
+# verify filename formats matches w/--save-fpi
+for my $fullpath (glob "$tmp_folder/raw/*")
Do we need to look for the exact match of the file that gets created
in the save-fpi path? While checking for this is great, it makes the
test code non-portable (may not work on Windows or other platforms,
no?) and complex? This way, you can get rid of get_block_info() as
well? And +for my $fullpath (glob "$tmp_folder/raw/*")
will also  get simplified.

I think you can further simplify the tests by:
create the node
generate an FPI
call pg_waldump with save-fpi option
check the target directory for a file that contains the relid,
something like '%relid%'.

The above would still serve the purpose, tests the code without much complexity.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com