public inbox for [email protected]  

help / color / mirror / Atom feed

From: Tomas Vondra <[email protected]>
To: Daniel Gustafsson <[email protected]>
To: Bernd Helmle <[email protected]>
Cc: Michael Paquier <[email protected]>
Cc: Michael Banck <[email protected]>
Cc: PostgreSQL Hackers <[email protected]>
Subject: Re: Changing the state of data checksums in a running cluster
Date: Wed, 20 Aug 2025 16:37:33 +0200
Message-ID: <[email protected]> (raw)
In-Reply-To: <[email protected]>
References: <[email protected]>
	<[email protected]>
	<[email protected]>
	<[email protected]>
	<[email protected]>
	<[email protected]>
	<[email protected]>
	<[email protected]>
	<[email protected]>
	<[email protected]>
	<[email protected]>
	<[email protected]>
	<[email protected]>
	<[email protected]>
	<[email protected]>
	<[email protected]>
	<[email protected]>
	<[email protected]>
	<[email protected]>
	<[email protected]>

On 8/16/25 21:34, Daniel Gustafsson wrote:
> Attached is a rebase on top of the func.sgml changes which caused this to no
> longer apply.
> 
> This version is also substantially updated with a new injection point based
> test suite, fixed a few bugs (found by said test suite), added checkpoint to
> disabling checksums, code cleanup, more granular wait events, comment rewrites
> and additions and more smaller cleanups.
> 

Thanks for the updated patch.

The injection points seem like a huge improvement, allowing testing of
different code paths in a more deterministic way.

I started running the stress test, using pretty much exactly the version
posted in March [1]. And so far I noticed only one issue, when the
standby reports mismatched checksums on a fsm:

LOG:  page verification failed, calculated checksum 24786 but expected 24760
CONTEXT:  WAL redo at 0/0344A290 for Heap2/MULTI_INSERT+INIT: ntuples:
185, flags: 0x28; blkref #0: rel 1663/16384/16403, blk 0
LOG:  invalid page in block 2 of relation base/16384/16403_fsm; zeroing
out page
CONTEXT:  WAL redo at 0/0344A290 for Heap2/MULTI_INSERT+INIT: ntuples:
185, flags: 0x28; blkref #0: rel 1663/16384/16403, blk 0
WARNING:  invalid page in block 2 of relation base/16384/16403_fsm;
zeroing out page
CONTEXT:  WAL redo at 0/0344A290 for Heap2/MULTI_INSERT+INIT: ntuples:
185, flags: 0x28; blkref #0: rel 1663/16384/16403, blk 0
LOG:  page verification failed, calculated checksum 37048 but expected 0
CONTEXT:  WAL redo at 0/0344D7E0 for Heap2/MULTI_INSERT+INIT: ntuples:
61, flags: 0x28; blkref #0: rel 1663/16384/16400, blk 0
LOG:  invalid page in block 2 of relation base/16384/16400_fsm; zeroing
out page

This happens quite regularly, it's not hard to hit. But I've only seen
it to happen on a FSM, and only right after immediate shutdown. I don't
think that's quite expected.

I believe the built-in TAP tests (with injection points) can't catch
this, because there's no concurrent activity while flipping checksums
on/off. It'd be good to do something like that, by running pgbench in
the background, or something like that.

I also don't see any restarts of the primary/standby. That might be good
to do too.

I plan to randomize the stress test a bit more, once this FSM issue gets
fixed. Maybe that'll find some additional issues.


[1]
https://www.postgresql.org/message-id/[email protected]

-- 
Tomas Vondra

view thread (58+ messages)  latest in thread

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
  Subject: Re: Changing the state of data checksums in a running cluster
  In-Reply-To: <[email protected]>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox