Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1vnJjf-0052o7-0V for pgsql-hackers@arkaria.postgresql.org; Tue, 03 Feb 2026 16:58:44 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.96) (envelope-from ) id 1vnJje-006Ne2-0U for pgsql-hackers@arkaria.postgresql.org; Tue, 03 Feb 2026 16:58:42 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1vnJjd-006Ndt-18 for pgsql-hackers@lists.postgresql.org; Tue, 03 Feb 2026 16:58:41 +0000 Received: from smtp.outgoing.loopia.se ([93.188.3.37]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1vnJjZ-00000000rvE-3HwV for pgsql-hackers@lists.postgresql.org; Tue, 03 Feb 2026 16:58:40 +0000 Received: from s807.loopia.se (localhost [127.0.0.1]) by s807.loopia.se (Postfix) with ESMTP id 3494A561BE1 for ; Tue, 03 Feb 2026 17:58:36 +0100 (CET) Received: from s979.loopia.se (unknown [172.22.191.6]) by s807.loopia.se (Postfix) with ESMTP id 1F221561AC7; Tue, 03 Feb 2026 17:58:36 +0100 (CET) Received: from localhost (unknown [172.22.191.6]) by s979.loopia.se (Postfix) with ESMTP id 1781D10BC4BA; Tue, 03 Feb 2026 17:58:36 +0100 (CET) X-Virus-Scanned: amavis at amavis.loopia.se X-Spam-Flag: NO X-Spam-Score: -1.2 X-Spam-Level: X-Spam-Status: No, score=-1.2 tagged_above=-999 required=6.2 tests=[ALL_TRUSTED=-1, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1] autolearn=disabled Authentication-Results: s470.loopia.se (amavis); dkim=pass (2048-bit key) header.d=yesql.se Received: from s980.loopia.se ([172.22.191.5]) by localhost (s470.loopia.se [172.22.190.34]) (amavis, port 10024) with LMTP id 5rP8pOgk5MoV; Tue, 3 Feb 2026 17:58:34 +0100 (CET) X-Loopia-Auth: user X-Loopia-User: daniel@yesql.se X-Loopia-Originating-IP: 95.193.180.186 Received: from smtpclient.apple (host-95-193-180-186.mobileonline.telia.com [95.193.180.186]) (Authenticated sender: daniel@yesql.se) by s980.loopia.se (Postfix) with ESMTPSA id A2D7A2201611; Tue, 03 Feb 2026 17:58:32 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yesql.se; s=loopiadkim1707475645; t=1770137913; bh=SxmZPIF/x4fObiQD3qEIgqRJOTybvVJxcd3sZMtXRd8=; h=From:Subject:Date:In-Reply-To:Cc:To:References; b=XBtbIXd9wlgr+m6OtfuPMsYOanZth7dPGjfIT46KlLbvv4gKXwComP40T+Lw7PBoI 74PttD3nPw1Q9yiSsar04n5IqWlklvAHbDpmGoZXeiKhlmpuhZgv03ue8s51IlaMhR QG/kAYZ1HcXKeDvnbIf5YIifxEr74sQ3v+YXrBIx+xSR94aJu8snPDMkY+KopNCbD8 8OJ6T8EHp+KgzQQ2YJrDpzRFwdCMszUn+5ufHyKJY9wsy6GYVMaD+juHSlWIg2uQ1x L1Undcebb6NRZMhXW1r5cxmOB5cAxmKSQFFdRBFw4M1FFt/G9y0+uajepuxGt5AAjc szSekGNOUSMrA== From: Daniel Gustafsson Message-Id: <2535E72D-5979-4F58-A2D3-2AA4AA335C20@yesql.se> Content-Type: multipart/mixed; boundary="Apple-Mail=_B8DC27FC-A3EC-4551-9E38-E86153E9E5F4" Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3776.700.51.11.2\)) Subject: Re: Changing the state of data checksums in a running cluster Date: Tue, 3 Feb 2026 17:58:22 +0100 In-Reply-To: <5mv5vtbqj2xm2wmevsi22smyn32jtznva3ya3daih3ixh3onex@s5fyab63xt3c> Cc: Tomas Vondra , Bernd Helmle , Michael Paquier , Michael Banck , PostgreSQL Hackers To: Andres Freund References: <7357A519-3391-4F07-BC69-5C9C2B120792@yesql.se> <25c734d1-18bb-4167-bb31-9afa736ea886@vondra.me> <3e67160c-3676-4419-b635-1fdb80dc128e@vondra.me> <67726BDA-2F21-4D93-B01F-FA68212ED6F0@yesql.se> <9CA2495B-2D6C-4308-97EF-A12CCCAA753E@yesql.se> <477897AE-1314-4724-9694-0BABC4F4ABDA@yesql.se> <5mv5vtbqj2xm2wmevsi22smyn32jtznva3ya3daih3ixh3onex@s5fyab63xt3c> X-Mailer: Apple Mail (2.3776.700.51.11.2) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --Apple-Mail=_B8DC27FC-A3EC-4551-9E38-E86153E9E5F4 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii > On 15 Dec 2025, at 23:21, Andres Freund wrote: > My suggestion for how to do this instead is to put the checksum state = into the > XLOG_CHECKPOINT_* records. When starting recovery from an online = checkpoint, > I think we should use the ChecksumType from the XLOG_CHECKPOINT_REDO = record, > that way the standby/recovery environment's assumption about whether = checksums > were enabled is the same as it was at the time the WAL was generated. = For > shutdown checkpoints, we could either start to emit a = XLOG_CHECKPOINT_REDO, or > we can use the information from the checkpoint record itself. >=20 > I think if we only ever use the checksum state from the point where we = start > recovery, we might not need to force *any* checkpoints. I've implemented this in the attached patch and it seems very promising. = There are no checksums at all left and it (mostly) just works. There are = still bugs left, which cause page verification failures in the 006/007 tests, but I = hope that yours or Tomas' eagle eyes can help me spot them since I failed to = (and this is far from my area of expertise so I appreciate pointers for = learning). This might be due to my implementation being incomplete, but I felt it = was time to show at least an intermediate step along the way. > Daniel and I chatted about that proposal, and couldn't immediately = come up > with scenarios where that would be wrong. For a while I thought there = would > be problems when doing PITR from a base backup that had checksums = initially > enabled, but where checksums were disabled before the base backup was > completed. My worry was that a later (once checksums were disabled) = hint bit > write (which would not necessarily be WAL logged) would corrupt the = checksum, > but I don't think that's a problem, because the startup process will = only read > data pages in the process of processing WAL records, and if there's a = WAL > record, there would also have to be an FPW, which would "cure" the > unchecksummed page. >=20 > More comments below, inline - I wrote these first, so it's possible = that I > missed updating some of them in light of what I now wrote above. Some were completely invalidated by the new approach, and I've removed = them since the code they referred to was removed as well. >> +static const ChecksumBarrierCondition checksum_barriers[] =3D >> +{ >> + {PG_DATA_CHECKSUM_OFF, {PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION, = PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION}, 2, {PG_DATA_CHECKSUM_VERSION}, = 1}, >> + {PG_DATA_CHECKSUM_VERSION, = {PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION}, 1, {0}, 0}, >> + {PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION, = {PG_DATA_CHECKSUM_ANY_VERSION}, 1, {PG_DATA_CHECKSUM_VERSION}, 1}, >> + {PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION, = {PG_DATA_CHECKSUM_VERSION}, 1, {0}, 0}, >> + {-1} >> +}; >=20 > The explanation for this doesn't really explain what the purpose of = this thing > is... Perhaps worth referencing datachecksumsworker.c or such? Done. Might need more justification, but at least there is an attempt. = I've tried to not spread the documentation into too many places to keep it = from drifting apart, so I added a reference to datachecksumsworker.c. > For a local, constantly sized, array, you shouldn't need a -1 = terminator, as > you can instead use lengthof() or such to detect invalid accesses. Fixed. >> +/* >> + * Flag to remember if the procsignalbarrier being absorbed for = checksums is >> + * the first one. The first procsignalbarrier can in rare cases be = for the >> + * state we've initialized, i.e. a duplicate. This may happen for = any >> + * data_checksum_version value when the process is spawned between = the update >> + * of XLogCtl->data_checksum_version and the barrier being emitted. = This can >> + * only happen on the very first barrier so mark that with this = flag. >> + */ >> +static bool InitialDataChecksumTransition =3D true; >=20 > This is pretty hard to understand right now, at the very least it = needs an > updated comment. But perhaps we can just get rid of this and accept = barriers > that are redundant. I think you are right, removed. >> - doPageWrites =3D (Insert->fullPageWrites || Insert->runningBackups = > 0); >> + doPageWrites =3D (Insert->fullPageWrites || >> + Insert->runningBackups > 0 || >> + DataChecksumsNeedWrite()); >>=20 >> if (doPageWrites && >> (!prevDoPageWrites || >=20 > Why do we need to separately check for DataChecksumsNeedWrite() if = turning on > checksums also forces Insert->fullPageWrites to on? Fair point, removed. >> + MyProc->delayChkptFlags |=3D DELAY_CHKPT_START; >> + START_CRIT_SECTION(); >=20 > ISTM that delayChkptFlags ought to only be set once within the = critical > section. Obviously we can't fail inbetween as the code stands, but = that's not > guaranteed to stay this way. Agreed, that was incorrect, fixed. >> + XLogChecksums(PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION); >> + >> + SpinLockAcquire(&XLogCtl->info_lck); >> + XLogCtl->data_checksum_version =3D = PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION; >> + SpinLockRelease(&XLogCtl->info_lck); >=20 > Maybe worth adding an assertion checking that we are currently in an = expected > state (off or inprogress, I think?). The inprogress state would be harmless, but I think we should make sure = to only ever be in the off state if we get as far as emitting a = procsignalbarrier. > Think it might be worth mentioning that we rely on the memory ordering = implied > by XLogChecksums() and WaitForProcSignalBarrier() for the changes to > delayChkptFlags. Unless we have a different logic around that? No other bespoke logic. Where do you propose calling it out to keep = from getting buried? >> + /* >> + * The only allowed state transition to "on" is from "inprogress-on" = since >> + * that state ensures that all pages will have data checksums = written. >> + */ >> + if (XLogCtl->data_checksum_version !=3D = PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION) >> + { >> + SpinLockRelease(&XLogCtl->info_lck); >> + elog(ERROR, "checksums not in \"inprogress-on\" mode"); >=20 > Seems like a PANIC condition to me... Fair point. Fixed. >> + } >> + >> + SpinLockRelease(&XLogCtl->info_lck); >> + >> + MyProc->delayChkptFlags |=3D DELAY_CHKPT_START; >> + INJECTION_POINT("datachecksums-enable-checksums-delay", NULL); >> + START_CRIT_SECTION(); >=20 > I think it's a really really bad idea to do something fallible, like > INJECTION_POINT, after setting delayChkptFlags, but before entering = the crit > section. Any error in the injection point will lead to a corrupted > delayChkptFlags, no? Agreed, reordered. >> + XLogChecksums(0); >=20 > Why no symbolic name here? For a long time this patch didn't add a name for the OFF case since the = current code only has a version for ON and not OFF. When it was I added I = missed updating all places where it should go. Fixed. >> + while (condition->target !=3D target_state && condition->target !=3D = -1) >> + condition++; >> + if (unlikely(condition->target =3D=3D -1)) >> + ereport(ERROR, >> + errcode(ERRCODE_INVALID_PARAMETER_VALUE), >> + errmsg("invalid target state %i for data checksum barrier", >> + target_state)); >=20 > FWIW, you don't need unlikely() when the branch does an = ereport(ERROR), as > ereports >=3DERROR are marked "cold" automatically. Nice, I didn't know that. unlikely() removed. Thinking more about ut, = this whole conditional should probably be an assertion instead. Also switched to looping around lengthof(), removing the -1 sentinel, as mentioned in another review comment. >> + default: >> + Assert(state.new_checksumtype =3D=3D 0); >> + barrier =3D EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_OFF); >> + WaitForProcSignalBarrier(barrier); >> + break; >=20 > I'd not add a default clause, but add a case each value of the enum. = That way > we'll get warnings if the set of states changes. This wasn't using the enum which was why it had a default, but I totally = agree and have switched that over now. > These WaitForProcSignalBarrier() are one of the scariest bits of the > patchset. If the startup process were to hold any lock that backends = need, and > the backends waited for that lock without processing interrupts, we'd = have an > undetected deadlock. This is much more likely to be a problem for the = startup > process, as it does the work of many backends on the primary. >=20 > We do process interrupts while waiting for heavyweight locks, so that = at least > is not an issue. Seems worth to call out rather explicitly. I've added a note about it, not entirely sure what else to do really. >> - if (!noverify_checksums && DataChecksumsEnabled() && >> + if (!noverify_checksums && >> + DataChecksumsNeedWrite() && >> RelFileNumberIsValid(relfilenumber)) >> verify_checksum =3D true; >=20 > Why is DataChecksumsNeedWrite() being tested here? That seems incorrect, it should check NeedVerify() to test if checksums = are enabled. I've also refactored this to do finer grained checking than = just a single check in sendFile as is sufficient when the state cannot change. >> + * have checksums set, but no reads will fail due to incorrect = checksum. >=20 > Maybe "... due to not yet set checksums."? Incorrect checksums sounds = like > it's about checksums that are actively wrong, rather than just not = set. Except > for the corner case of a torn page in the process of an hint bit = write, after > having disabled checksums, there shouldn't be incorrect ones, right? Correct, I meant invalid checksums and not incorrect checksums. Fixed, = and also added a sentence discussing why invalid checksums can be present in = the first place. >> The >> + * DataChecksumsWorker will compile a list of databases which exist = at the >> + * start of checksumming, and all of these which haven't been = dropped during >> + * the processing MUST have been processed successfully in order for = checksums >> + * to be enabled. Any new relation created during processing will = see the >> + * in-progress state and will automatically be checksummed. >=20 > What about new databases created while checksums are being enabled? = They could > be copied before the worker has processed them. At least for the = file_copy > strategy, the copy will be verbatim and thus will not necessarily have > checksums set. This is a pretty terrible sentence, mixing two concepts together and = making it unreadable. Enabling checksums is naive to avoid the very problem you refer to. A = list of databases is generated and processed. When done, it starts over and = generates a new list and process each new entry, skipping the ones already seen, = until no new entries appear in the list. Avoiding those created after = processing, where we can be certain there are checksums set, is a TODO optimization. I have reworded that sentence and also added it to the Optimizations = section of the header comment. >> + * There are two levels of synchronization required for changing = data_checksums >=20 > Maybe s/levels/steps/? Yeah, much better. Maybe 'stages' is even better than that though? >> + * Synchronizing the state change is done with procsignal barriers, = where the >> + * WAL logging backend updating the global state in the controlfile = will wait >=20 > It's not entirely obvious what "the WAL logging backend" means. Even I am not entirely sure what I meant, reworded. >> + * interrogating state and resumed when the IO operation has been = performed. >> + * >> + * When Enabling Data Checksums >> + * ---------------------------- >=20 > Odd change in indentation here. It was intended to make this a subsection under "Synchronization and Correctness" which is the main section. Doing that with actual section numbering will be a lot clearer though. Done that way. I also tried to = make the sections a bit less dense by adding some vertical whitespace. >> + * Bd: Backends in "off" state >> + * Bi: Backends in "inprogress-on" state >> + * >> + * If processing is started in an online cluster then all backends = are in Bd. >> + * If processing was halted by the cluster shutting down, the = controlfile >> + * state "inprogress-on" will be observed on system startup and = all backends >> + * will be placed in Bd. >=20 > Why not in Bi? Just for simplicities sake? ISTM we already need to be = sure > that new backends start in Bi, as they might never observe the = barrier... When we start up after a crash and find that the processing didn't = finish the state is reverted to "off", we don't know if the user want to restart processing at that point. What this comment fails to mention is that = the controlfile is also reverted to "off". >> + * all pages and enables data checksums cluster-wide via another >=20 > s/enables/enabled/? Fixed in the rewrite. >> + * * Invent a lightweight WAL record that doesn't contain the = full-page >> + * image but just the block number: On replay, the redo routine = would read >> + * the page from disk. >=20 > The last sentence might be truncated? This TODO item is a verbatim quote from Heikki which was brought up as a = review comment in b832a496-f0aa-c7e4-cca4-6bd0af4d4a90@iki.fi. Re-reading this = I am not entirely sure what it would imply, so I opted for removing it = instead. >> + /* >> + * As of now we only update the block counter for main forks in = order to >> + * not cause too frequent calls. TODO: investigate whether we should = do it >> + * more frequent? >> + */ >> + if (forkNum =3D=3D MAIN_FORKNUM) >> + pgstat_progress_update_param(PROGRESS_DATACHECKSUMS_BLOCKS_TOTAL, >> + numblocks); >=20 > That doesn't make much sense to me. Presumably the reason to skip it = for the > other forks is that they're small-ish. But if so, there's no point in = skipping > the reporting either, as presumably there wouldn't be a lot of = reporting? Fair point, removed. >> +static void >> +launcher_exit(int code, Datum arg) >> +{ >> + if (launcher_running) >> + { >> + LWLockAcquire(DataChecksumsWorkerLock, LW_EXCLUSIVE); >> + launcher_running =3D false; >> + DataChecksumsWorkerShmem->launcher_running =3D false; >> + LWLockRelease(DataChecksumsWorkerLock); >> + } >> +} >=20 > Could we end up exiting this with the worker still running? Not during regular operation AFAICT, but it might still be possible. = For now I've added a logging of it and a TODO comment. >> +/* >> + * WaitForAllTransactionsToFinish >=20 > I think either here, or at its callsites, the patch needs to explain = *why* we > are waiting for all transactions to finish. Presumably this is to = ensure that > other sessions haven't created relations that we can't see yet? Correct. I've updated the comments. > It actually doesn't seem to wait for all transactions, ust for ones = with an > xid? Yes, is that not enough for this purpose or am I missing something? >> + /* Retry every 3 seconds */ >> + ResetLatch(MyLatch); >> + rc =3D WaitLatch(MyLatch, >> + WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH, >> + 3000, >> + WAIT_EVENT_CHECKSUM_ENABLE_STARTCONDITION); >> + >> + /* >> + * If the postmaster died we won't be able to enable checksums >> + * cluster-wide so abort and hope to continue when restarted. >> + */ >> + if (rc & WL_POSTMASTER_DEATH) >> + ereport(FATAL, >> + errcode(ERRCODE_ADMIN_SHUTDOWN), >> + errmsg("postmaster exited during data checksum processing"), >> + errhint("Restart the database and restart data checksum processing = by calling pg_enable_data_checksums().")); >> + >> + LWLockAcquire(DataChecksumsWorkerLock, LW_SHARED); >> + if (DataChecksumsWorkerShmem->launch_operation !=3D operation) >> + abort_requested =3D true; >> + LWLockRelease(DataChecksumsWorkerLock); >> + if (abort_requested) >> + break; >=20 > I don't like this much - loops with a timeout are generally a really = bad idea > and we shouldn't add more instances. Presumably this also makes the = tests > slower... >=20 > How about collecting the to-be-waited-for virtualxids and then wait = for those? This seemed the safe option, how does one get the set of virtualxids to = wait for? >> + RESUME_INTERRUPTS(); >=20 > I don't understand what this interrupt stuff achieves here? Removed. >> -#define NUM_AUXILIARY_PROCS (6 + MAX_IO_WORKERS) >> - >> +#define NUM_AUXILIARY_PROCS (8 + MAX_IO_WORKERS) >=20 > Aren't they bgworkers now? Correct, this is a very old leftover from when the ChecksumHelper was = based off of autovacuum and wasn't a bgworker. Hunk reverted. >> + PROCSIGNAL_BARRIER_CHECKSUM_OFF, >> + PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_ON, >> + PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_OFF, >> + PROCSIGNAL_BARRIER_CHECKSUM_ON, >> } ProcSignalBarrierType; >=20 > I wonder if these really should be different barriers. What if we just = made it > one, and instead drove the transition on the current shmem content? I sort of like when the barrier types to encode state, but I don't have = super strong feelings on that. If you think it's something that we should do = then I can hack that up. > Other stuff: > - what protects against multiple backends enabling checksums at the = same time? >=20 > Afaict there isn't anything, and we just ignore the second request. = Which > seems ok-ish if it's the same request as before, but not great if = it's a > different one. If a different request comes in it will be recorded as a new = launch_operation, and the processing will after each block double check that the current operation matches the launch_operation. If datachecksumsworker is = enabling checksums and the launch_operation is set to disable, checksums will be disabled and processing end. I added a short to the documentation about = this. > Should also have tests for that. I have added some rudimentary tests covering this, but it's surprisingly = hard to make stable tests without putting all hope on synchronization around = delays. If this gets committed I doubt they should be part of that, but for now = they illustrate a point at least. > - I think this adds a bit too much of the logic to xlog.c, already an = unwieldy > file. A fair bit of all of this doesn't seem like it needs to be in = there. No disagreement, but thinking about it I struggled to figure out a good = place to put it. Do you have any suggestions? > - the code seems somewhat split brained about bgworkers and = auxprocesses Is it, or was it mainly the docs? I've tried to rectify what I saw but = I might be misunderstanding your point here. The attached .txt contains the fixes done above before they were = squashed into the patch, to assist reviewing in case anyone wants to see the diff. -- Daniel Gustafsson --Apple-Mail=_B8DC27FC-A3EC-4551-9E38-E86153E9E5F4 Content-Disposition: attachment; filename=v20250203-2-0001-Online-enabling-and-disabling-of-data-ch.patch Content-Type: application/octet-stream; x-unix-mode=0644; name="v20250203-2-0001-Online-enabling-and-disabling-of-data-ch.patch" Content-Transfer-Encoding: quoted-printable =46rom=2057e46fa1ee0f5b63f2b47e1f2ab37ad650d3f686=20Mon=20Sep=2017=20= 00:00:00=202001=0AFrom:=20Daniel=20Gustafsson=20= =0ADate:=20Fri,=2015=20Aug=202025=2014:48:02=20= +0200=0ASubject:=20[PATCH=20v20250203-2]=20Online=20enabling=20and=20= disabling=20of=20data=20checksums=0A=0AThis=20allows=20data=20checksums=20= to=20be=20enabled,=20or=20disabled,=20in=20a=20running=0Acluster=20= without=20restricting=20access=20to=20the=20cluster=20during=20= processing.=0A=0AData=20checksums=20could=20prior=20to=20this=20only=20= be=20enabled=20during=20initdb=20or=0Awhen=20the=20cluster=20is=20= offline=20using=20the=20pg_checksums=20app.=20This=20commit=0Aintroduce=20= functionality=20to=20enable,=20or=20disable,=20data=20checksums=20while=0A= the=20cluster=20is=20running=20regardless=20of=20how=20it=20was=20= initialized.=0A=0AA=20background=20worker=20launcher=20process=20is=20= responsible=20for=20launching=20a=0Adynamic=20per-database=20background=20= worker=20which=20will=20mark=20all=20buffers=0Adirty=20for=20all=20= relation=20with=20storage=20in=20order=20for=20them=20to=20have=20data=0A= checksums=20calcuated=20on=20write.=20=20Once=20all=20relations=20in=20= all=20databases=0Ahave=20been=20processed,=20the=20data_checksums=20= state=20will=20be=20set=20to=20on=20and=0Athe=20cluster=20will=20at=20= that=20point=20be=20identical=20to=20one=20which=20had=20data=0A= checksums=20enabled=20during=20initialization=20or=20via=20offline=20= processing.=0A=0AWhen=20data=20checksums=20are=20being=20enabled,=20= concurrent=20I/O=20operations=0Afrom=20backends=20other=20than=20the=20= data=20checksums=20worker=20will=20write=20the=0Achecksums=20but=20not=20= verify=20them=20on=20reading.=20=20Only=20when=20all=20backends=0Ahave=20= absorbed=20the=20procsignalbarrier=20for=20setting=20data_checksums=20to=0A= on=20will=20they=20also=20start=20verifying=20checksums=20on=20reading.=20= =20The=20same=0Aprocess=20is=20repeated=20during=20disabling;=20all=20= backends=20write=20checksums=0Abut=20do=20not=20verify=20them=20until=20= the=20barrier=20for=20setting=20the=20state=20to=0Aoff=20has=20been=20= absorbed=20by=20all.=20=20This=20in-progress=20state=20is=20used=20to=0A= ensure=20there=20are=20no=20false=20negatives=20(or=20positives)=20due=20= to=20reading=0Aa=20checksum=20which=20is=20not=20in=20sync=20with=20the=20= page.=0A=0AA=20new=20testmodule,=20test_checksums,=20is=20introduced=20= with=20an=20extensive=0Aset=20of=20tests=20covering=20both=20online=20= and=20offline=20data=20checksum=20mode=0Achanges.=20=20The=20tests=20for=20= online=20processing=20are=20gated=20begind=20the=0APG_TEST_EXTRA=20flag=20= to=20some=20degree=20due=20to=20being=20very=20time=20consuming=0Ato=20= run.=0A=0AThis=20work=20is=20based=20on=20an=20earlier=20version=20of=20= this=20patch=20which=20was=0Areviewed=20by=20among=20others=20Heikki=20= Linnakangas,=20Robert=20Haas,=20Andres=0AFreund,=20Tomas=20Vondra,=20= Michael=20Banck=20and=20Andrey=20Borodin.=20=20During=0Athe=20work=20on=20= this=20new=20version,=20Tomas=20Vondra=20has=20given=20invaluable=0A= assistance=20with=20not=20only=20coding=20and=20reviewing=20but=20very=20= in-depth=0Atesting.=0A=0AAuthor:=20Daniel=20Gustafsson=20= =0AAuthor:=20Magnus=20Hagander=20=0A= Co-authored-by:=20Tomas=20Vondra=20=0AReviewed-by:=20= Tomas=20Vondra=20=0AReviewed-by:=20Andres=20Freund=20= =0ADiscussion:=20= https://postgr.es/m/CABUevExz9hUUOLnJVr2kpw9Cx=3Do4MCr1SVKwbupzuxP7ckNutA@= mail.gmail.com=0ADiscussion:=20= https://postgr.es/m/20181030051643.elbxjww5jjgnjaxg@alap3.anarazel.de=0A= Discussion:=20= https://postgr.es/m/CABUevEwE3urLtwxxqdgd5O2oQz9J717ZzMbh+ziCSa5YLLU_BA@ma= il.gmail.com=0A---=0A=20doc/src/sgml/func/func-admin.sgml=20=20=20=20=20=20= =20=20=20=20=20=20=20|=20=20=2077=20+=0A=20doc/src/sgml/glossary.sgml=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20|=20=20=2024=20+=0A= =20doc/src/sgml/images/Makefile=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20|=20=20=20=201=20+=0A=20doc/src/sgml/images/datachecksums.gv=20= =20=20=20=20=20=20=20=20=20|=20=20=2014=20+=0A=20= doc/src/sgml/images/datachecksums.svg=20=20=20=20=20=20=20=20=20|=20=20=20= 81=20+=0A=20doc/src/sgml/monitoring.sgml=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20|=20=20235=20++-=0A=20= doc/src/sgml/ref/pg_checksums.sgml=20=20=20=20=20=20=20=20=20=20=20=20|=20= =20=20=206=20+=0A=20doc/src/sgml/regress.sgml=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20|=20=20=2012=20+=0A=20= doc/src/sgml/wal.sgml=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20|=20=20127=20+-=0A=20= src/backend/access/rmgrdesc/xlogdesc.c=20=20=20=20=20=20=20=20|=20=20=20= 42=20+-=0A=20src/backend/access/transam/xlog.c=20=20=20=20=20=20=20=20=20= =20=20=20=20|=20=20685=20+++++++-=0A=20= src/backend/access/transam/xlogfuncs.c=20=20=20=20=20=20=20=20|=20=20=20= 48=20+=0A=20src/backend/access/transam/xlogrecovery.c=20=20=20=20=20|=20=20= =2013=20+=0A=20src/backend/backup/basebackup.c=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20|=20=20=2027=20+-=0A=20= src/backend/catalog/system_functions.sql=20=20=20=20=20=20|=20=20=2012=20= +=0A=20src/backend/catalog/system_views.sql=20=20=20=20=20=20=20=20=20=20= |=20=20=2019=20+=0A=20src/backend/postmaster/Makefile=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20|=20=20=20=201=20+=0A=20= src/backend/postmaster/auxprocess.c=20=20=20=20=20=20=20=20=20=20=20|=20=20= =2019=20+=0A=20src/backend/postmaster/bgworker.c=20=20=20=20=20=20=20=20=20= =20=20=20=20|=20=20=20=209=20+=0A=20= src/backend/postmaster/datachecksumsworker.c=20=20|=201497=20= +++++++++++++++++=0A=20src/backend/postmaster/meson.build=20=20=20=20=20=20= =20=20=20=20=20=20|=20=20=20=201=20+=0A=20= src/backend/postmaster/postmaster.c=20=20=20=20=20=20=20=20=20=20=20|=20=20= =20=205=20+=0A=20src/backend/postmaster/walsummarizer.c=20=20=20=20=20=20= =20=20|=20=20=20=205=20+-=0A=20src/backend/replication/logical/decode.c=20= =20=20=20=20=20|=20=20=20=201=20+=0A=20src/backend/storage/ipc/ipci.c=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20|=20=20=20=203=20+=0A=20= src/backend/storage/ipc/procsignal.c=20=20=20=20=20=20=20=20=20=20|=20=20= =2014=20+=0A=20src/backend/storage/page/README=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20|=20=20=20=204=20+-=0A=20= src/backend/storage/page/bufpage.c=20=20=20=20=20=20=20=20=20=20=20=20|=20= =20=2022=20+-=0A=20src/backend/utils/activity/pgstat_backend.c=20=20=20|=20= =20=20=202=20+=0A=20src/backend/utils/activity/pgstat_io.c=20=20=20=20=20= =20=20=20|=20=20=20=202=20+=0A=20.../utils/activity/wait_event_names.txt=20= =20=20=20=20=20=20|=20=20=20=204=20+=0A=20= src/backend/utils/adt/pgstatfuncs.c=20=20=20=20=20=20=20=20=20=20=20|=20=20= =20=208=20+-=0A=20src/backend/utils/init/miscinit.c=20=20=20=20=20=20=20=20= =20=20=20=20=20|=20=20=20=203=20+-=0A=20= src/backend/utils/init/postinit.c=20=20=20=20=20=20=20=20=20=20=20=20=20= |=20=20=2020=20+-=0A=20src/backend/utils/misc/guc_parameters.dat=20=20=20= =20=20|=20=20=20=205=20+-=0A=20src/backend/utils/misc/guc_tables.c=20=20=20= =20=20=20=20=20=20=20=20|=20=20=20=209=20+-=0A=20= src/bin/pg_checksums/pg_checksums.c=20=20=20=20=20=20=20=20=20=20=20|=20=20= =20=204=20+-=0A=20src/bin/pg_controldata/pg_controldata.c=20=20=20=20=20=20= =20|=20=20=20=202=20+=0A=20src/bin/pg_upgrade/controldata.c=20=20=20=20=20= =20=20=20=20=20=20=20=20=20|=20=20=20=209=20+=0A=20= src/include/access/xlog.h=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20|=20=20=2014=20+-=0A=20src/include/access/xlog_internal.h=20= =20=20=20=20=20=20=20=20=20=20=20|=20=20=2013=20+=0A=20= src/include/catalog/pg_control.h=20=20=20=20=20=20=20=20=20=20=20=20=20=20= |=20=20=20=206=20+-=0A=20src/include/catalog/pg_proc.dat=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20|=20=20=2012=20+=0A=20= src/include/commands/progress.h=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20|=20=20=2016=20+=0A=20src/include/miscadmin.h=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20|=20=20=20=206=20+=0A=20= src/include/postmaster/datachecksumsworker.h=20=20|=20=20=2050=20+=0A=20= src/include/postmaster/proctypelist.h=20=20=20=20=20=20=20=20=20|=20=20=20= =202=20+=0A=20src/include/storage/bufpage.h=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20|=20=20=20=202=20+-=0A=20= src/include/storage/checksum.h=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20|=20=20=2014=20+=0A=20src/include/storage/lwlocklist.h=20=20=20=20=20=20= =20=20=20=20=20=20=20=20|=20=20=20=201=20+=0A=20= src/include/storage/proc.h=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20|=20=20=20=203=20+-=0A=20src/include/storage/procsignal.h=20=20= =20=20=20=20=20=20=20=20=20=20=20=20|=20=20=20=204=20+=0A=20= src/include/utils/backend_progress.h=20=20=20=20=20=20=20=20=20=20|=20=20= =20=201=20+=0A=20src/test/modules/Makefile=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20|=20=20=20=201=20+=0A=20= src/test/modules/meson.build=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20|=20=20=20=201=20+=0A=20= src/test/modules/test_checksums/.gitignore=20=20=20=20|=20=20=20=202=20+=0A= =20src/test/modules/test_checksums/Makefile=20=20=20=20=20=20|=20=20=20= 40=20+=0A=20src/test/modules/test_checksums/README=20=20=20=20=20=20=20=20= |=20=20=2022=20+=0A=20src/test/modules/test_checksums/meson.build=20=20=20= |=20=20=2036=20+=0A=20.../modules/test_checksums/t/001_basic.pl=20=20=20=20= =20|=20=20=2063=20+=0A=20.../modules/test_checksums/t/002_restarts.pl=20=20= |=20=20110=20++=0A=20.../test_checksums/t/003_standby_restarts.pl=20=20|=20= =20114=20++=0A=20.../modules/test_checksums/t/004_offline.pl=20=20=20|=20= =20=2082=20+=0A=20.../modules/test_checksums/t/005_injection.pl=20|=20=20= 109=20++=0A=20.../test_checksums/t/006_pgbench_single.pl=20=20=20=20|=20=20= 252=20+++=0A=20.../test_checksums/t/007_pgbench_standby.pl=20=20=20|=20=20= 376=20+++++=0A=20.../test_checksums/t/DataChecksums/Utils.pm=20=20=20|=20= =20261=20+++=0A=20.../test_checksums/test_checksums--1.0.sql=20=20=20=20= |=20=20=2032=20+=0A=20.../modules/test_checksums/test_checksums.c=20=20=20= |=20=20246=20+++=0A=20.../test_checksums/test_checksums.control=20=20=20=20= =20|=20=20=20=204=20+=0A=20src/test/perl/PostgreSQL/Test/Cluster.pm=20=20= =20=20=20=20|=20=20=2036=20+=0A=20src/test/regress/expected/rules.out=20=20= =20=20=20=20=20=20=20=20=20|=20=20=2035=20+=0A=20= src/test/regress/expected/stats.out=20=20=20=20=20=20=20=20=20=20=20|=20=20= =2018=20+-=0A=20src/tools/pgindent/typedefs.list=20=20=20=20=20=20=20=20=20= =20=20=20=20=20|=20=20=20=209=20+=0A=2074=20files=20changed,=205007=20= insertions(+),=2058=20deletions(-)=0A=20create=20mode=20100644=20= doc/src/sgml/images/datachecksums.gv=0A=20create=20mode=20100644=20= doc/src/sgml/images/datachecksums.svg=0A=20create=20mode=20100644=20= src/backend/postmaster/datachecksumsworker.c=0A=20create=20mode=20100644=20= src/include/postmaster/datachecksumsworker.h=0A=20create=20mode=20100644=20= src/test/modules/test_checksums/.gitignore=0A=20create=20mode=20100644=20= src/test/modules/test_checksums/Makefile=0A=20create=20mode=20100644=20= src/test/modules/test_checksums/README=0A=20create=20mode=20100644=20= src/test/modules/test_checksums/meson.build=0A=20create=20mode=20100644=20= src/test/modules/test_checksums/t/001_basic.pl=0A=20create=20mode=20= 100644=20src/test/modules/test_checksums/t/002_restarts.pl=0A=20create=20= mode=20100644=20= src/test/modules/test_checksums/t/003_standby_restarts.pl=0A=20create=20= mode=20100644=20src/test/modules/test_checksums/t/004_offline.pl=0A=20= create=20mode=20100644=20= src/test/modules/test_checksums/t/005_injection.pl=0A=20create=20mode=20= 100644=20src/test/modules/test_checksums/t/006_pgbench_single.pl=0A=20= create=20mode=20100644=20= src/test/modules/test_checksums/t/007_pgbench_standby.pl=0A=20create=20= mode=20100644=20src/test/modules/test_checksums/t/DataChecksums/Utils.pm=0A= =20create=20mode=20100644=20= src/test/modules/test_checksums/test_checksums--1.0.sql=0A=20create=20= mode=20100644=20src/test/modules/test_checksums/test_checksums.c=0A=20= create=20mode=20100644=20= src/test/modules/test_checksums/test_checksums.control=0A=0Adiff=20--git=20= a/doc/src/sgml/func/func-admin.sgml=20= b/doc/src/sgml/func/func-admin.sgml=0Aindex=203ac81905d1f..7762b774bf2=20= 100644=0A---=20a/doc/src/sgml/func/func-admin.sgml=0A+++=20= b/doc/src/sgml/func/func-admin.sgml=0A@@=20-3087,4=20+3087,81=20@@=20= SELECT=20convert_from(pg_read_binary_file('file_in_utf8.txt'),=20= 'UTF8');=0A=20=0A=20=20=20=0A=20=0A+=20=20=0A+=20=20=20Data=20Checksum=20= Functions=0A+=0A+=20=20=20=0A+=20=20=20=20The=20functions=20= shown=20in=20=20can=0A= +=20=20=20=20be=20used=20to=20enable=20or=20disable=20data=20checksums=20= in=20a=20running=20cluster.=0A+=20=20=20=0A+=20=20=20=0A+=20= =20=20=20Changing=20data=20checksums=20can=20be=20done=20in=20a=20= cluster=20with=20concurrent=20activity=0A+=20=20=20=20without=20blocking=20= queries,=20but=20overall=20system=20performance=20will=20be=20affected.=0A= +=20=20=20=20See=20=20for=20further=20= details=20on=20how=20changing=20the=0A+=20=20=20=20data=20checksums=20= mode=20can=20affect=20a=20system=20and=20possible=20mitigations=20for=20= how=0A+=20=20=20=20to=20reduce=20the=20impact.=0A+=20=20=20=0A+=0A= +=20=20=20=0A+=20=20=20=20= Data=20Checksum=20Functions=0A+=20=20=20=20=0A+=20=20=20=20=20=0A+=20=20=20=20=20=20=0A+=20=20= =20=20=20=20=20=0A+=20=20=20=20=20=20=20=20Function=0A+=20=20=20= =20=20=20=20=0A+=20=20=20=20=20=20=20=0A+=20=20=20=20=20=20=20= =20Description=0A+=20=20=20=20=20=20=20=0A+=20=20=20=20=20= =20=0A+=20=20=20=20=20=0A+=0A+=20=20=20=20=20=0A+=20= =20=20=20=20=20=0A+=20=20=20=20=20=20=20=0A+=20=20=20=20= =20=20=20=20=0A+=20=20=20=20=20=20=20=20=20= pg_enable_data_checksums=0A+=20=20=20=20=20=20=20=20= =0A+=20=20=20=20=20=20=20=20= pg_enable_data_checksums=20(=20= cost_delay=20int,=20= cost_limit=20int=20)=0A+=20= =20=20=20=20=20=20=20void=0A+=20=20=20=20=20=20= =20=0A+=20=20=20=20=20=20=20=0A+=20=20=20=20=20=20=20=20= Initiates=20the=20process=20of=20enabling=20data=20checksums=20for=20the=20= cluster.=20This=0A+=20=20=20=20=20=20=20=20will=20set=20the=20data=20= checksums=20mode=20to=20inprogress-on=0A+=20=20=20=20=20= =20=20=20as=20well=20as=20start=20a=20background=20worker=20that=20will=20= process=20all=20pages=20in=20all=0A+=20=20=20=20=20=20=20=20databases=20= and=20enable=20data=20checksums=20on=20them.=20=20When=20all=20data=20= checksums=0A+=20=20=20=20=20=20=20=20have=20been=20calculated,=20and=20= written,=20for=20all=20pages=20the=20cluster=20will=0A+=20=20=20=20=20=20= =20=20automatically=20set=20data=20checksums=20mode=20to=20= on.=0A+=20=20=20=20=20=20=20=0A+=20=20=20=20=20= =20=20=0A+=20=20=20=20=20=20=20=20If=20= cost_delay=20and=20= cost_limit=20are=0A+=20=20=20=20=20=20=20=20= specified,=20the=20process=20is=20throttled=20using=20the=20same=20= principles=20as=0A+=20=20=20=20=20=20=20=20Cost-based=20Vacuum=20= Delay.=0A+=20=20=20=20=20=20=20=0A+=20=20=20=20=20=20=20= =0A+=20=20=20=20=20=20=0A+=0A+=20=20=20=20=20=20=0A+=20= =20=20=20=20=20=20=0A+=20=20=20=20=20=20=20=20=0A+=20=20= =20=20=20=20=20=20=20pg_disable_data_checksums=0A+=20=20= =20=20=20=20=20=20=0A+=20=20=20=20=20=20=20=20= pg_disable_data_checksums=20()=0A+=20=20=20=20=20=20= =20=20void=0A+=20=20=20=20=20=20=20=0A= +=20=20=20=20=20=20=20=0A+=20=20=20=20=20=20=20=20Disables=20data=20= checksum=20calculation=20and=20validation=20for=20the=20cluster.=20This=0A= +=20=20=20=20=20=20=20=20will=20set=20the=20data=20checksum=20mode=20to=20= inprogress-off=0A+=20=20=20=20=20=20=20=20while=20= data=20checksums=20are=20being=20disabled.=20=20When=20all=20active=20= backends=20have=0A+=20=20=20=20=20=20=20=20stopped=20validating=20data=20= checksums,=20the=20data=20checksum=20mode=20will=20be=0A+=20=20=20=20=20=20= =20=20set=20to=20off.=0A+=20=20=20=20=20=20=20=0A= +=20=20=20=20=20=20=20=0A+=20=20=20=20=20=20=0A+=20=20=20=20= =20=0A+=20=20=20=20=0A+=20=20=20=0A+=0A+=20=20= =0A+=0A=20=20=20=0Adiff=20--git=20= a/doc/src/sgml/glossary.sgml=20b/doc/src/sgml/glossary.sgml=0Aindex=20= e2db5bcc78c..4f3120b7826=20100644=0A---=20a/doc/src/sgml/glossary.sgml=0A= +++=20b/doc/src/sgml/glossary.sgml=0A@@=20-199,6=20+199,8=20@@=0A=20=20=20= =20=20=20(but=20not=20the=20autovacuum=20workers),=0A=20=20=20=20=20=20= the=20background=20= writer,=0A=20=20=20=20=20=20the=20checkpointer,=0A+=20=20=20=20= =20the=20data=20= checksums=20worker,=0A+=20=20=20=20=20the=20data=20checksums=20= worker=20launcher,=0A=20=20=20=20=20=20the=20logger,=0A=20=20=20=20=20=20the=20= startup=20= process,=0A=20=20=20=20=20=20the=20WAL=20archiver,=0A@@=20= -574,6=20+576,28=20@@=0A=20=20=20=20=0A=20=20=20=0A=20= =0A+=20=20=0A+=20=20=20= Data=20Checksums=20Worker=0A+=20=20=20=0A= +=20=20=20=20=0A+=20=20=20=20=20A=20background=20worker=0A= +=20=20=20=20=20which=20enables=20data=20checksums=20in=20a=20specific=20= database.=0A+=20=20=20=20=0A+=20=20=20=0A+=20=20= =0A+=0A+=20=20=0A+=20=20=20= Data=20Checksums=20Worker=20Launcher=0A+=20=20=20= =0A+=20=20=20=20=0A+=20=20=20=20=20A=20background=20worker=0A= +=20=20=20=20=20which=20starts=20=20processes=0A+=20= =20=20=20=20for=20enabling=20data=20checksums=20in=20each=20database,=20= or=20disables=20data=20checksums=0A+=20=20=20=20=20cluster-wide.=0A+=20=20= =20=20=0A+=20=20=20=0A+=20=20=0A+=0A=20=20= =20=0A=20=20=20=20= Database=0A=20=20=20=20=0Adiff=20--git=20= a/doc/src/sgml/images/Makefile=20b/doc/src/sgml/images/Makefile=0Aindex=20= fd55b9ad23f..e805487344d=20100644=0A---=20a/doc/src/sgml/images/Makefile=0A= +++=20b/doc/src/sgml/images/Makefile=0A@@=20-3,6=20+3,7=20@@=0A=20#=20= see=20README=20in=20this=20directory=20about=20image=20handling=0A=20=0A=20= ALL_IMAGES=20=3D=20\=0A+=09datachecksums.svg=20\=0A=20=09= genetic-algorithm.svg=20\=0A=20=09gin.svg=20\=0A=20=09pagelayout.svg=20\=0A= diff=20--git=20a/doc/src/sgml/images/datachecksums.gv=20= b/doc/src/sgml/images/datachecksums.gv=0Anew=20file=20mode=20100644=0A= index=2000000000000..dff3ff7340a=0A---=20/dev/null=0A+++=20= b/doc/src/sgml/images/datachecksums.gv=0A@@=20-0,0=20+1,14=20@@=0A= +digraph=20G=20{=0A+=20=20=20=20A=20->=20B=20[label=3D"SELECT=20= pg_enable_data_checksums()"];=0A+=20=20=20=20B=20->=20C;=0A+=20=20=20=20= D=20->=20A;=0A+=20=20=20=20C=20->=20D=20[label=3D"SELECT=20= pg_disable_data_checksums()"];=0A+=20=20=20=20E=20->=20A=20[label=3D"=20= --no-data-checksums"];=0A+=20=20=20=20E=20->=20C=20[label=3D"=20= --data-checksums"];=0A+=0A+=20=20=20=20A=20[label=3D"off"];=0A+=20=20=20=20= B=20[label=3D"inprogress-on"];=0A+=20=20=20=20C=20[label=3D"on"];=0A+=20=20= =20=20D=20[label=3D"inprogress-off"];=0A+=20=20=20=20E=20= [label=3D"initdb"];=0A+}=0Adiff=20--git=20= a/doc/src/sgml/images/datachecksums.svg=20= b/doc/src/sgml/images/datachecksums.svg=0Anew=20file=20mode=20100644=0A= index=2000000000000..8c58f42922e=0A---=20/dev/null=0A+++=20= b/doc/src/sgml/images/datachecksums.svg=0A@@=20-0,0=20+1,81=20@@=0A= +=0A= +=0A+=0A+=0A+=0A+G=0A+=0A+=0A+=0A= +A=0A+=0A+off=0A+=0A+=0A+=0A+B=0A+=0A+inprogress-on= =0A+=0A+=0A+=0A+A->B=0A+=0A= +=0A+SELECT=20= pg_enable_data_checksums()=0A+=0A+=0A+=0A+C=0A+=0A+on=0A+=0A= +=0A+=0A= +B->C=0A+=0A= +=0A+=0A+=0A+=0A+D=0A= +=0A+inprogress-off=0A+=0A+=0A+=0A= +C->D=0A+=0A= +=0A+SELECT=20= pg_disable_data_checksums()=0A+=0A+=0A= +=0A+D->A=0A= +=0A+=0A+=0A+=0A= +=0A+E=0A+=0A+initdb=0A+=0A= +=0A+=0A= +E->A=0A+=0A= +=0A+=20= --no-data-checksums=0A+=0A+=0A+=0A+E->C=0A= +=0A+=0A+=20= --data-checksums=0A+=0A+=0A+=0Adiff=20= --git=20a/doc/src/sgml/monitoring.sgml=20b/doc/src/sgml/monitoring.sgml=0A= index=20b77d189a500..cc905378386=20100644=0A---=20= a/doc/src/sgml/monitoring.sgml=0A+++=20b/doc/src/sgml/monitoring.sgml=0A= @@=20-3622,9=20+3622,14=20@@=20description=20|=20Waiting=20for=20a=20= newly=20initialized=20WAL=20file=20to=20reach=20durable=20storage=0A=20=20= =20=20=20=20=20=0A=20=20=20=20=20=20=20=0A=20=20=20=20=20=20= =20=20Number=20of=20data=20page=20checksum=20failures=20detected=20in=20= this=0A-=20=20=20=20=20=20=20database=20(or=20on=20a=20shared=20object),=20= or=20NULL=20if=20data=20checksums=20are=0A-=20=20=20=20=20=20=20= disabled.=0A-=20=20=20=20=20=20=0A+=20=20=20=20=20=20=20= database=20(or=20on=20a=20shared=20object).=20=20Detected=20failures=20= are=20not=20reset=20if=0A+=20=20=20=20=20=20=20the=20=20setting=20changes.=20=20Clusters=0A+=20= =20=20=20=20=20=20which=20are=20initialized=20without=20data=20checksums=20= will=20show=20this=20as=0A+=20=20=20=20=20=20=200.=20= In=20PostgreSQL=20version=0A+=20=20=20=20=20=20= =2018=20and=20earlier,=20this=20was=20set=20to=20NULL=20= for=20clusters=0A+=20=20=20=20=20=20=20with=20data=20checksums=20= disabled.=0A+=20=20=20=20=20=20=0A+=20=20=20=20=20=0A=20=20= =20=20=20=20=0A=20=0A=20=20=20=20=20=20=0A@@=20-3633,8=20= +3638,8=20@@=20description=20|=20Waiting=20for=20a=20newly=20initialized=20= WAL=20file=20to=20reach=20durable=20storage=0A=20=20=20=20=20=20=20= =0A=20=20=20=20=20=20=20=0A=20=20=20=20=20=20=20=20Time=20= at=20which=20the=20last=20data=20page=20checksum=20failure=20was=20= detected=20in=0A-=20=20=20=20=20=20=20this=20database=20(or=20on=20a=20= shared=20object),=20or=20NULL=20if=20data=20checksums=20are=0A-=20=20=20=20= =20=20=20disabled.=0A+=20=20=20=20=20=20=20this=20database=20(or=20on=20= a=20shared=20object).=20Last=20failure=20is=20reported=0A+=20=20=20=20=20= =20=20regardless=20of=20the=20=20= setting.=0A=20=20=20=20=20=20=20=0A=20=20=20=20=20=20= =0A=20=0A@@=20-7117,6=20+7122,226=20@@=20FROM=20= pg_stat_get_backend_idset()=20AS=20backendid;=0A=20=0A=20=20=0A=20= =0A+=20=0A+=20=20= Data=20Checksum=20Progress=20Reporting=0A+=0A+=20=20= =0A+=20=20=20= pg_stat_progress_data_checksums=0A+=20=20=0A= +=0A+=20=20=0A+=20=20=20When=20data=20checksums=20are=20being=20= enabled=20on=20a=20running=20cluster,=20the=0A+=20=20=20= pg_stat_progress_data_checksums=20view=20will=20= contain=0A+=20=20=20a=20row=20for=20the=20launcher=20process,=20and=20= one=20row=20for=20each=20worker=20process=20which=0A+=20=20=20is=20= currently=20calculating=20and=20writing=20checksums=20for=20the=20data=20= pages=20in=20a=20database.=0A+=20=20=20The=20launcher=20provides=20= overview=20of=20the=20overall=20progress=20(how=20many=20database=0A+=20=20= =20have=20been=20processed,=20how=20many=20remain),=20while=20the=20= workers=20track=20progress=20for=0A+=20=20=20currently=20processed=20= databases.=0A+=20=20=0A+=0A+=20=20=0A+=20=20=20= <structname>pg_stat_progress_data_checksums</structname>=20= View=0A+=20=20=20=0A+=20=20=20=20=0A= +=20=20=20=20=20=0A+=20=20=20=20=20=20=0A+=20=20=20=20=20=20=20=0A+=20=20=20=20=20=20=20=20Column=20Type=0A+=20= =20=20=20=20=20=20=0A+=20=20=20=20=20=20=20=0A+=20=20=20=20=20= =20=20=20Description>=0A+=20=20=20=20=20=20=20=0A+=20=20=20=20=20=20= =0A+=20=20=20=20=20=0A+=20=20=20=20=0A+=0A+=20=20=20= =20=0A+=20=20=20=20=20=0A+=20=20=20=20=20=20=0A+=20=20=20=20=20=20=20=0A+=20=20=20=20=20=20=20=20= pid=20integer=0A+=20=20=20=20=20=20= =20=0A+=20=20=20=20=20=20=20=0A+=20=20=20=20=20=20=20=20= Process=20ID=20of=20the=20data=20checksum=20process,=20launcher=20or=20= worker.=0A+=20=20=20=20=20=20=20=0A+=20=20=20=20=20=20=0A= +=20=20=20=20=20=0A+=0A+=20=20=20=20=20=0A+=20=20=20=20=20=20= =0A+=20=20=20=20=20=20=20=0A+=20=20=20=20=20=20=20=20= datid=20oid=0A+=20=20=20=20=20=20= =20=0A+=20=20=20=20=20=20=20=0A+=20=20=20=20=20=20=20=20OID=20= of=20this=20database,=20or=200=20for=20the=20launcher=20process=20= relation=0A+=20=20=20=20=20=20=20=0A+=20=20=20=20=20=20=0A= +=20=20=20=20=20=0A+=0A+=20=20=20=20=20=0A+=20=20=20=20=20=20= =0A+=20=20=20=20=20=20=20=0A+=20=20=20=20=20=20=20=20= datname=20name=0A+=20=20=20=20=20= =20=20=0A+=20=20=20=20=20=20=20=0A+=20=20=20=20=20=20=20=20= Name=20of=20this=20database,=20or=20NULL=20for=20the=0A= +=20=20=20=20=20=20=20=20launcher=20process.=0A+=20=20=20=20=20=20=20= =0A+=20=20=20=20=20=20=0A+=20=20=20=20=20=0A+=0A+=20= =20=20=20=20=0A+=20=20=20=20=20=20=0A+=20=20=20=20=20=20=20=0A+=20=20=20=20=20=20=20=20= phase=20text=0A+=20=20=20=20=20=20= =20=0A+=20=20=20=20=20=20=20=0A+=20=20=20=20=20=20=20=20= Current=20processing=20phase,=20see=20=0A+=20=20=20=20=20=20=20=20for=20= description=20of=20the=20phases.=0A+=20=20=20=20=20=20=20=0A+=20=20= =20=20=20=20=0A+=20=20=20=20=20=0A+=0A+=20=20=20=20=20= =0A+=20=20=20=20=20=20=0A+=20=20= =20=20=20=20=20=0A+=20=20=20=20=20=20=20= =20databases_total=20integer=0A+=20= =20=20=20=20=20=20=0A+=20=20=20=20=20=20=20=0A+=20=20=20=20=20= =20=20=20The=20total=20number=20of=20databases=20which=20will=20be=20= processed.=20Only=20the=0A+=20=20=20=20=20=20=20=20launcher=20process=20= has=20this=20value=20set,=20the=20worker=20processes=20have=20this=0A+=20= =20=20=20=20=20=20=20set=20to=20NULL.=0A+=20=20=20=20=20= =20=20=0A+=20=20=20=20=20=20=0A+=20=20=20=20=20=0A+=0A= +=20=20=20=20=20=0A+=20=20=20=20=20=20=0A+=20=20=20=20=20=20=20=0A+=20=20=20=20=20=20=20=20= databases_done=20integer=0A+=20=20= =20=20=20=20=20=0A+=20=20=20=20=20=20=20=0A+=20=20=20=20=20=20= =20=20The=20number=20of=20databases=20which=20have=20been=20processed.=20= Only=20the=20launcher=0A+=20=20=20=20=20=20=20=20process=20has=20this=20= value=20set,=20the=20worker=20processes=20have=20this=20set=20to=0A+=20=20= =20=20=20=20=20=20NULL.=0A+=20=20=20=20=20=20=20= =0A+=20=20=20=20=20=20=0A+=20=20=20=20=20=0A+=0A+=20= =20=20=20=20=0A+=20=20=20=20=20=20=0A+=20=20=20=20=20=20=20=0A+=20=20=20=20=20=20=20=20= relations_total=20integer=0A+=20=20= =20=20=20=20=20=0A+=20=20=20=20=20=20=20=0A+=20=20=20=20=20=20= =20=20The=20total=20number=20of=20relations=20which=20will=20be=20= processed,=20or=0A+=20=20=20=20=20=20=20=20NULL=20if=20= the=20worker=20process=20hasn't=0A+=20=20=20=20=20=20=20=20calculated=20= the=20number=20of=20relations=20yet.=20The=20launcher=20process=20has=0A= +=20=20=20=20=20=20=20=20this=20set=20to=20NULL=20= since=20it=20isn't=20responsible=20for=0A+=20=20=20=20=20=20=20=20= processing=20relations,=20only=20launching=20worker=20processes.=0A+=20=20= =20=20=20=20=20=0A+=20=20=20=20=20=20=0A+=20=20=20=20=20= =0A+=0A+=20=20=20=20=20=0A+=20=20=20=20=20=20=0A+=20=20=20=20=20=20=20=0A+=20=20=20=20=20=20=20=20= relations_done=20integer=0A+=20=20= =20=20=20=20=20=0A+=20=20=20=20=20=20=20=0A+=20=20=20=20=20=20= =20=20The=20number=20of=20relations=20which=20have=20been=20processed.=20= The=20launcher=0A+=20=20=20=20=20=20=20=20process=20has=20this=20set=20= to=20NULL.=0A+=20=20=20=20=20=20=20=0A+=20=20=20= =20=20=20=0A+=20=20=20=20=20=0A+=0A+=20=20=20=20=20=0A= +=20=20=20=20=20=20=0A+=20=20=20=20= =20=20=20=0A+=20=20=20=20=20=20=20=20= blocks_total=20integer=0A+=20=20=20= =20=20=20=20=0A+=20=20=20=20=20=20=20=0A+=20=20=20=20=20=20=20= =20The=20number=20of=20blocks=20in=20the=20current=20relation=20which=20= will=20be=20processed,=0A+=20=20=20=20=20=20=20=20or=20= NULL=20if=20the=20worker=20process=20hasn't=0A+=20=20=20= =20=20=20=20=20calculated=20the=20number=20of=20blocks=20yet.=20The=20= launcher=20process=20has=0A+=20=20=20=20=20=20=20=20this=20set=20to=20= NULL.=0A+=20=20=20=20=20=20=20=0A+=20=20=20=20=20= =20=0A+=20=20=20=20=20=0A+=0A+=20=20=20=20=20=0A+=20=20= =20=20=20=20=0A+=20=20=20=20=20=20=20= =0A+=20=20=20=20=20=20=20=20= blocks_done=20integer=0A+=20=20=20= =20=20=20=20=0A+=20=20=20=20=20=20=20=0A+=20=20=20=20=20=20=20= =20The=20number=20of=20blocks=20in=20the=20current=20relation=20which=20= have=20been=20processed.=0A+=20=20=20=20=20=20=20=20The=20launcher=20= process=20has=20this=20set=20to=20NULL.=0A+=20=20=20=20= =20=20=20=0A+=20=20=20=20=20=20=0A+=20=20=20=20=20=0A= +=0A+=20=20=20=20=0A+=20=20=20=0A+=20=20=0A+=0A= +=20=20=0A+=20=20=20Data=20= Checksum=20Phases=0A+=20=20=20=0A+=20=20=20=20= =0A+=20=20=20=20=0A+=20=20=20=20=0A+=20=20=20=20= =20=0A+=20=20=20=20=20=20Phase=0A+=20=20=20=20=20=20= Description=0A+=20=20=20=20=20=0A+=20=20=20=20= =0A+=20=20=20=20=0A+=20=20=20=20=20=0A+=20=20=20=20=20= =20enabling=0A+=20=20=20=20=20=20= =0A+=20=20=20=20=20=20=20The=20command=20is=20currently=20= enabling=20data=20checksums=20on=20the=20cluster.=0A+=20=20=20=20=20=20= =0A+=20=20=20=20=20=0A+=20=20=20=20=20=0A+=20=20=20=20= =20=20disabling=0A+=20=20=20=20=20=20= =0A+=20=20=20=20=20=20=20The=20command=20is=20currently=20= disabling=20data=20checksums=20on=20the=20cluster.=0A+=20=20=20=20=20=20= =0A+=20=20=20=20=20=0A+=20=20=20=20=20=0A+=20=20=20=20= =20=20done=0A+=20=20=20=20=20=20= =0A+=20=20=20=20=20=20=20The=20command=20is=20done=20and=20the=20= data=20checksum=20state=20in=20the=20cluster=20has=0A+=20=20=20=20=20=20=20= changed.=0A+=20=20=20=20=20=20=0A+=20=20=20=20=20=0A+=20=20= =20=20=20=0A+=20=20=20=20=20=20waiting=20on=20= barrier=0A+=20=20=20=20=20=20=0A+=20=20=20=20=20= =20=20The=20command=20is=20currently=20waiting=20for=20the=20current=20= active=20backends=20to=0A+=20=20=20=20=20=20=20acknowledge=20the=20= change=20in=20data=20checksum=20state.=0A+=20=20=20=20=20=20=0A+=20= =20=20=20=20=0A+=20=20=20=20=20=0A+=20=20=20=20=20=20= waiting=20on=20checkpoint=0A+=20=20=20=20= =20=20=0A+=20=20=20=20=20=20=20The=20command=20is=20currently=20= waiting=20for=20a=20checkpoint=20to=20update=20the=20checksum=0A+=20=20=20= =20=20=20=20state=20before=20finishing.=0A+=20=20=20=20=20=20=0A= +=20=20=20=20=20=0A+=20=20=20=20=20=0A+=20=20=20=20=20=20= waiting=20on=20temporary=20tables=0A+=20= =20=20=20=20=20=0A+=20=20=20=20=20=20=20The=20command=20is=20= currently=20waiting=20for=20all=20temporary=20tables=20which=20existed=0A= +=20=20=20=20=20=20=20at=20the=20time=20the=20command=20was=20started=20= to=20be=20removed.=0A+=20=20=20=20=20=20=0A+=20=20=20=20=20= =0A+=20=20=20=20=0A+=20=20=20=0A+=20=20=0A= +=20=0A+=0A=20=20=0A=20=0A=20=20=0Adiff=20--git=20= a/doc/src/sgml/ref/pg_checksums.sgml=20= b/doc/src/sgml/ref/pg_checksums.sgml=0Aindex=20e9e393495df..e764b8be04d=20= 100644=0A---=20a/doc/src/sgml/ref/pg_checksums.sgml=0A+++=20= b/doc/src/sgml/ref/pg_checksums.sgml=0A@@=20-45,6=20+45,12=20@@=20= PostgreSQL=20documentation=0A=20=20=20=20exit=20status=20is=20nonzero=20= if=20the=20operation=20failed.=0A=20=20=20=0A=20=0A+=20=20=0A= +=20=20=20When=20enabling=20checksums,=20if=20checksums=20were=20in=20= the=20process=20of=20being=20enabled=0A+=20=20=20when=20the=20cluster=20= was=20shut=20down,=20pg_checksums=0A+=20=20=20= will=20still=20process=20all=20relations=20regardless=20of=20the=20= online=20processing.=0A+=20=20=0A+=0A=20=20=20=0A=20=20=20=20= When=20verifying=20checksums,=20every=20file=20in=20the=20cluster=20is=20= scanned.=20When=0A=20=20=20=20enabling=20checksums,=20each=20relation=20= file=20block=20with=20a=20changed=20checksum=20is=0Adiff=20--git=20= a/doc/src/sgml/regress.sgml=20b/doc/src/sgml/regress.sgml=0Aindex=20= d80dd46c5fd..ef4eab91bde=20100644=0A---=20a/doc/src/sgml/regress.sgml=0A= +++=20b/doc/src/sgml/regress.sgml=0A@@=20-275,6=20+275,18=20@@=20make=20= check-world=20PG_TEST_EXTRA=3D'kerberos=20ldap=20ssl=20load_balance=20= libpq_encryption'=0A=20=0A=20=20=20=20The=20following=20= values=20are=20currently=20supported:=0A=20=20=20=20=0A+=20= =20=20=20=0A+=20=20=20=20=20= checksum_extended=0A+=20=20=20=20=20= =0A+=20=20=20=20=20=20=0A+=20=20=20=20=20=20=20Runs=20= additional=20tests=20for=20enabling=20data=20checksums=20which=20inject=20= delays=0A+=20=20=20=20=20=20=20and=20re-tries=20in=20the=20processing,=20= as=20well=20as=20tests=20that=20run=20pgbench=0A+=20=20=20=20=20=20=20= concurrently=20and=20randomly=20restarts=20the=20cluster.=20=20Some=20of=20= these=20test=0A+=20=20=20=20=20=20=20suites=20requires=20injection=20= points=20enabled=20in=20the=20installation.=0A+=20=20=20=20=20=20=0A= +=20=20=20=20=20=0A+=20=20=20=20=0A+=0A=20=20=20= =20=20=0A=20=20=20=20=20=20= kerberos=0A=20=20=20=20=20=20=0A= diff=20--git=20a/doc/src/sgml/wal.sgml=20b/doc/src/sgml/wal.sgml=0Aindex=20= f3b86b26be9..df41fb54cb6=20100644=0A---=20a/doc/src/sgml/wal.sgml=0A+++=20= b/doc/src/sgml/wal.sgml=0A@@=20-246,9=20+246,10=20@@=0A=20=20=20=0A= =20=20=20=20Checksums=20can=20be=20disabled=20when=20the=20cluster=20is=20= initialized=20using=20initdb.=0A-=20=20=20They=20can=20also=20be=20enabled=20or=20disabled=20at=20= a=20later=20time=20as=20an=20offline=0A-=20=20=20operation.=20Data=20= checksums=20are=20enabled=20or=20disabled=20at=20the=20full=20cluster=0A= -=20=20=20level,=20and=20cannot=20be=20specified=20individually=20for=20= databases=20or=20tables.=0A+=20=20=20They=20can=20also=20be=20enabled=20= or=20disabled=20at=20a=20later=20time=20either=20as=20an=20offline=0A+=20= =20=20operation=20or=20online=20in=20a=20running=20cluster=20allowing=20= concurrent=20access.=20Data=0A+=20=20=20checksums=20are=20enabled=20or=20= disabled=20at=20the=20full=20cluster=20level,=20and=20cannot=20be=0A+=20=20= =20specified=20individually=20for=20databases=20or=20tables.=0A=20=20=20= =0A=20=0A=20=20=20=0A@@=20-265,7=20+266,7=20@@=0A=20=20=20= =0A=20=0A=20=20=20= =0A-=20=20=20Off-line=20Enabling=20of=20Checksums=0A+=20=20= =20Offline=20Enabling=20of=20Checksums=0A=20=0A=20=20=20=20= =0A=20=20=20=20=20The=20pg_checksums= =0A@@=20-274,6=20+275,124=20@@=0A=20=20=20=20=0A=20=0A=20=20=20= =0A+=0A+=20=20=0A= +=20=20=20Online=20Enabling=20of=20Checksums=0A+=0A+=20=20= =20=0A+=20=20=20=20Checksums=20can=20be=20enabled=20or=20disabled=20= online,=20by=20calling=20the=20appropriate=0A+=20=20=20=20functions.=0A+=20=20=20= =0A+=0A+=20=20=20=0A+=20=20=20=20Both=20enabling=20and=20= disabling=20data=20checksums=20happens=20in=20two=20phases,=20separated=0A= +=20=20=20=20by=20a=20checkpoint=20to=20ensure=20durability.=20=20The=20= different=20states,=20and=20their=0A+=20=20=20=20transitions,=20are=20= illustrated=20in=20=0A= +=20=20=20=20and=20discussed=20in=20further=20detail=20in=20this=20= section.=0A+=20=20=20=0A+=0A+=20=20=20=0A+=20=20=20=20= =0A+=20=20=20=20=20= data=20checksums=20states=0A+=20=20=20=20=20=0A= +=20=20=20=20=20=20=0A+=20=20=20=20=20=20=20=0A= +=20=20=20=20=20=20=0A+=20=20=20=20=20=0A+=20= =20=20=20=0A+=20=20=20=0A+=0A+=20=20=20=0A+=20=20=20= =20Enabling=20checksums=20will=20put=20the=20cluster=20checksum=20mode=20= in=0A+=20=20=20=20inprogress-on=20mode.=20=20During=20= this=20time,=20checksums=20will=20be=0A+=20=20=20=20written=20but=20not=20= verified.=20In=20addition=20to=20this,=20a=20background=20worker=20= process=0A+=20=20=20=20is=20started=20that=20enables=20checksums=20on=20= all=20existing=20data=20in=20the=20cluster.=20Once=0A+=20=20=20=20this=20= worker=20has=20completed=20processing=20all=20databases=20in=20the=20= cluster,=20the=0A+=20=20=20=20checksum=20mode=20will=20automatically=20= switch=20to=20on.=20The=0A+=20=20=20=20processing=20= will=20consume=20two=20background=20worker=20processes,=20make=20sure=20= that=0A+=20=20=20=20max_worker_processes=20allows=20= for=20at=20least=20two=20more=0A+=20=20=20=20additional=20processes.=0A+=20= =20=20=0A+=0A+=20=20=20=0A+=20=20=20=20The=20process=20will=20= initially=20wait=20for=20all=20open=20transactions=20to=20finish=20= before=0A+=20=20=20=20it=20starts,=20so=20that=20it=20can=20be=20certain=20= that=20there=20are=20no=20tables=20that=20have=20been=0A+=20=20=20=20= created=20inside=20a=20transaction=20that=20has=20not=20committed=20yet=20= and=20thus=20would=20not=0A+=20=20=20=20be=20visible=20to=20the=20= process=20enabling=20checksums.=20It=20will=20also,=20for=20each=20= database,=0A+=20=20=20=20wait=20for=20all=20pre-existing=20temporary=20= tables=20to=20get=20removed=20before=20it=20finishes.=0A+=20=20=20=20If=20= long-lived=20temporary=20tables=20are=20used=20in=20the=20application=20= it=20may=20be=20necessary=0A+=20=20=20=20to=20terminate=20these=20= application=20connections=20to=20allow=20the=20process=20to=20complete.=0A= +=20=20=20=0A+=0A+=20=20=20=0A+=20=20=20=20If=20the=20= cluster=20is=20stopped=20while=20in=20inprogress-on=20= mode,=0A+=20=20=20=20for=20any=20reason,=20then=20the=20checksum=20= enable=20process=20must=20be=20restarted=20manually.=0A+=20=20=20=20To=20= do=20this,=20re-execute=20the=20function=20= pg_enable_data_checksums()=0A+=20=20=20=20once=20= the=20cluster=20has=20been=20restarted.=20The=20process=20will=20start=20= over,=20there=20is=0A+=20=20=20=20no=20support=20for=20resuming=20work=20= from=20where=20it=20was=20interrupted.=20=20If=20the=20cluster=0A+=20=20=20= =20is=20stopped=20while=20in=20inprogress-off,=20then=20= the=20checksum=0A+=20=20=20=20state=20will=20be=20set=20to=20= off=20when=20the=20cluster=20is=0A+=20=20=20=20= restarted.=0A+=20=20=20=0A+=0A+=20=20=20=0A+=20=20=20=20= Disabling=20data=20checksums=20will=20set=20the=20data=20checksum=20mode=20= to=0A+=20=20=20=20inprogress-off.=20=20During=20this=20= time,=20checksums=20will=20be=0A+=20=20=20=20written=20but=20not=20= verified.=20=20After=20all=20processes=20acknowledge=20the=20change,=0A+=20= =20=20=20the=20mode=20will=20automatically=20be=20set=20to=20= off.=0A+=20=20=20=0A+=0A+=20=20=20=0A+=20= =20=20=20Disabling=20data=20checksums=20while=20data=20checksums=20are=20= actively=20being=20enabled=0A+=20=20=20=20will=20abort=20the=20current=20= processing.=20=20There=20is=20no=20way=20to=20restart=20from=20where=0A+=20= =20=20=20processing=20was=20interrupted.=0A+=20=20=20=0A+=0A+=20=20= =20=0A+=20=20=20=20= Impact=20on=20system=20of=20online=20operations=0A+=20=20=20= =20=0A+=20=20=20=20=20Enabling=20data=20checksums=20can=20cause=20= significant=20I/O=20to=20the=20system,=20as=20all=20of=20the=0A+=20=20=20= =20=20database=20pages=20will=20need=20to=20be=20rewritten,=20and=20will=20= be=20written=20both=20to=20the=0A+=20=20=20=20=20data=20files=20and=20= the=20WAL.=20The=20impact=20may=20be=20limited=20by=20throttling=20using=20= the=0A+=20=20=20=20=20cost_delay=20and=20= cost_limit=0A+=20=20=20=20=20parameters=20of=20= the=20pg_enable_data_checksums=20function.=0A+=20=20= =20=20=0A+=0A+=20=20=20=20=0A+=20=20=20=20=20=0A= +=20=20=20=20=20=20=0A+=20=20=20=20=20=20=20I/O:=20all=20= pages=20need=20to=20have=20data=20checksums=20calculated=20and=20written=20= which=0A+=20=20=20=20=20=20=20will=20generate=20a=20lot=20of=20dirty=20= pages=20that=20will=20need=20to=20be=20flushed=20to=20disk,=0A+=20=20=20=20= =20=20=20as=20well=20as=20WAL=20logged.=0A+=20=20=20=20=20=20= =0A+=20=20=20=20=20=20=0A+=20=20=20=20=20= =20=20Replication:=20When=20the=20standby=20receives=20the=20data=20= checksum=20state=20change=0A+=20=20=20=20=20=20=20in=20the=20WAL=20= stream=20it=20will=20issue=20a=20=0A+=20=20=20=20=20=20=20= restartpoint=20in=20order=20to=20flush=20the=20current=20= state=20into=20the=0A+=20=20=20=20=20=20=20= pg_control=20file.=20=20The=20restartpoint=20will=20= flush=20the=0A+=20=20=20=20=20=20=20current=20state=20to=20disk=20and=20= will=20block=20redo=20until=20finished.=20=20This=20in=20turn=0A+=20=20=20= =20=20=20=20will=20induce=20replication=20lag,=20which=20on=20= synchronous=20standbys=20also=20blocks=0A+=20=20=20=20=20=20=20the=20= primary.=20=20Reducing=20=20before=20= the=0A+=20=20=20=20=20=20=20process=20is=20started=20can=20help=20with=20= reducing=20the=20time=20it=20takes=20for=20the=0A+=20=20=20=20=20=20=20= restartpoint=20to=20finish.=0A+=20=20=20=20=20=20=0A+=20= =20=20=20=20=20=0A+=20=20=20=20=20=20=20= Shutdown/Restart:=20If=20the=20server=20is=20shut=20down=20or=20= restarted=20when=20data=0A+=20=20=20=20=20=20=20checksums=20are=20being=20= enabled,=20the=20process=20will=20not=20resume=20and=20all=20pages=0A+=20= =20=20=20=20=20=20need=20to=20be=20recalculated=20and=20rewritten.=20=20= Enabling=20data=20checksums=20should=0A+=20=20=20=20=20=20=20be=20done=20= when=20there=20is=20no=20need=20for=20regular=20maintenance=20or=20= during=20a=0A+=20=20=20=20=20=20=20service=20window.=0A+=20=20=20=20=20=20= =0A+=20=20=20=20=20=0A+=20=20=20=20= =0A+=0A+=20=20=20=20=0A+=20=20=20=20=20Rewriting=20all=20= pages=20is=20not=20needed=20when=20disabling=20data=20checksums,=20but=0A= +=20=20=20=20=20checkpoints=20are=20still=20required.=0A+=20=20=20=20= =0A+=20=20=20=0A+=0A+=20=20=0A=20=20=0A=20= =0A=20=20=20=0Adiff=20--git=20= a/src/backend/access/rmgrdesc/xlogdesc.c=20= b/src/backend/access/rmgrdesc/xlogdesc.c=0Aindex=20= ff078f22264..8a854312ea1=20100644=0A---=20= a/src/backend/access/rmgrdesc/xlogdesc.c=0A+++=20= b/src/backend/access/rmgrdesc/xlogdesc.c=0A@@=20-18,6=20+18,7=20@@=0A=20= #include=20"access/xlog.h"=0A=20#include=20"access/xlog_internal.h"=0A=20= #include=20"catalog/pg_control.h"=0A+#include=20"storage/bufpage.h"=0A=20= #include=20"utils/guc.h"=0A=20#include=20"utils/timestamp.h"=0A=20=0A@@=20= -54,6=20+55,25=20@@=20get_wal_level_string(int=20wal_level)=0A=20=09= return=20wal_level_str;=0A=20}=0A=20=0A+static=20const=20char=20*=0A= +get_checksum_version_string(ChecksumType=20checksum)=0A+{=0A+=09switch=20= (checksum)=0A+=09{=0A+=09=09case=20PG_DATA_CHECKSUM_VERSION:=0A+=09=09=09= return=20"on";=0A+=09=09case=20PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION:=0A= +=09=09=09return=20"inprogress-off";=0A+=09=09case=20= PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION:=0A+=09=09=09return=20= "inprogress-on";=0A+=09=09case=20PG_DATA_CHECKSUM_OFF:=0A+=09=09=09= return=20"off";=0A+=09}=0A+=0A+=09Assert(false);=0A+=09return=20"?";=0A= +}=0A+=0A=20void=0A=20xlog_desc(StringInfo=20buf,=20XLogReaderState=20= *record)=0A=20{=0A@@=20-69,7=20+89,8=20@@=20xlog_desc(StringInfo=20buf,=20= XLogReaderState=20*record)=0A=20=09=09=09=09=09=09=20"tli=20%u;=20prev=20= tli=20%u;=20fpw=20%s;=20wal_level=20%s;=20logical=20decoding=20%s;=20xid=20= %u:%u;=20oid=20%u;=20multi=20%u;=20offset=20%"=20PRIu64=20";=20"=0A=20=09= =09=09=09=09=09=20"oldest=20xid=20%u=20in=20DB=20%u;=20oldest=20multi=20= %u=20in=20DB=20%u;=20"=0A=20=09=09=09=09=09=09=20"oldest/newest=20commit=20= timestamp=20xid:=20%u/%u;=20"=0A-=09=09=09=09=09=09=20"oldest=20running=20= xid=20%u;=20%s",=0A+=09=09=09=09=09=09=20"oldest=20running=20xid=20%u;=20= "=0A+=09=09=09=09=09=09=20"checksums=20%s;=20%s",=0A=20=09=09=09=09=09=09= =20LSN_FORMAT_ARGS(checkpoint->redo),=0A=20=09=09=09=09=09=09=20= checkpoint->ThisTimeLineID,=0A=20=09=09=09=09=09=09=20= checkpoint->PrevTimeLineID,=0A@@=20-88,6=20+109,7=20@@=20= xlog_desc(StringInfo=20buf,=20XLogReaderState=20*record)=0A=20=09=09=09=09= =09=09=20checkpoint->oldestCommitTsXid,=0A=20=09=09=09=09=09=09=20= checkpoint->newestCommitTsXid,=0A=20=09=09=09=09=09=09=20= checkpoint->oldestActiveXid,=0A+=09=09=09=09=09=09=20= get_checksum_version_string(checkpoint->dataChecksumVersion),=0A=20=09=09= =09=09=09=09=20(info=20=3D=3D=20XLOG_CHECKPOINT_SHUTDOWN)=20?=20= "shutdown"=20:=20"online");=0A=20=09}=0A=20=09else=20if=20(info=20=3D=3D=20= XLOG_NEXTOID)=0A@@=20-163,10=20+185,12=20@@=20xlog_desc(StringInfo=20= buf,=20XLogReaderState=20*record)=0A=20=09}=0A=20=09else=20if=20(info=20= =3D=3D=20XLOG_CHECKPOINT_REDO)=0A=20=09{=0A-=09=09int=09=09=09wal_level;=0A= +=09=09xl_checkpoint_redo=20xlrec;=0A=20=0A-=09=09memcpy(&wal_level,=20= rec,=20sizeof(int));=0A-=09=09appendStringInfo(buf,=20"wal_level=20%s",=20= get_wal_level_string(wal_level));=0A+=09=09memcpy(&xlrec,=20rec,=20= sizeof(xl_checkpoint_redo));=0A+=09=09appendStringInfo(buf,=20"wal_level=20= %s;=20checksums=20%s",=0A+=09=09=09=09=09=09=20= get_wal_level_string(xlrec.wal_level),=0A+=09=09=09=09=09=09=20= get_checksum_version_string(xlrec.data_checksum_version));=0A=20=09}=0A=20= =09else=20if=20(info=20=3D=3D=20XLOG_LOGICAL_DECODING_STATUS_CHANGE)=0A=20= =09{=0A@@=20-175,6=20+199,13=20@@=20xlog_desc(StringInfo=20buf,=20= XLogReaderState=20*record)=0A=20=09=09memcpy(&enabled,=20rec,=20= sizeof(bool));=0A=20=09=09appendStringInfoString(buf,=20enabled=20?=20= "true"=20:=20"false");=0A=20=09}=0A+=09else=20if=20(info=20=3D=3D=20= XLOG_CHECKSUMS)=0A+=09{=0A+=09=09xl_checksum_state=20xlrec;=0A+=0A+=09=09= memcpy(&xlrec,=20rec,=20sizeof(xl_checksum_state));=0A+=09=09= appendStringInfoString(buf,=20= get_checksum_version_string(xlrec.new_checksumtype));=0A+=09}=0A=20}=0A=20= =0A=20const=20char=20*=0A@@=20-229,6=20+260,9=20@@=20xlog_identify(uint8=20= info)=0A=20=09=09case=20XLOG_LOGICAL_DECODING_STATUS_CHANGE:=0A=20=09=09=09= id=20=3D=20"LOGICAL_DECODING_STATUS_CHANGE";=0A=20=09=09=09break;=0A+=09=09= case=20XLOG_CHECKSUMS:=0A+=09=09=09id=20=3D=20"CHECKSUMS";=0A+=09=09=09= break;=0A=20=09}=0A=20=0A=20=09return=20id;=0Adiff=20--git=20= a/src/backend/access/transam/xlog.c=20= b/src/backend/access/transam/xlog.c=0Aindex=2013ec6225b85..28928e2195d=20= 100644=0A---=20a/src/backend/access/transam/xlog.c=0A+++=20= b/src/backend/access/transam/xlog.c=0A@@=20-552,6=20+552,9=20@@=20= typedef=20struct=20XLogCtlData=0A=20=09=20*/=0A=20=09XLogRecPtr=09= lastFpwDisableRecPtr;=0A=20=0A+=09/*=20last=20data_checksum_version=20= we've=20seen=20*/=0A+=09uint32=09=09data_checksum_version;=0A+=0A=20=09= slock_t=09=09info_lck;=09=09/*=20locks=20shared=20variables=20shown=20= above=20*/=0A=20}=20XLogCtlData;=0A=20=0A@@=20-575,6=20+578,46=20@@=20= static=20WALInsertLockPadded=20*WALInsertLocks=20=3D=20NULL;=0A=20=20*/=0A= =20static=20ControlFileData=20*ControlFile=20=3D=20NULL;=0A=20=0A+/*=0A+=20= *=20This=20must=20match=20largest=20number=20of=20sets=20in=20barrier_eq=20= and=20barrier_ne=20in=20the=0A+=20*=20below=20checksum_barriers=20= definition.=0A+=20*/=0A+#define=20MAX_BARRIER_CONDITIONS=202=0A+=0A+/*=0A= +=20*=20Configuration=20of=20conditions=20which=20must=20match=20when=20= absorbing=20a=20procsignal=0A+=20*=20barrier=20during=20data=20checksum=20= enable/disable=20operations.=20=20A=20single=20function=0A+=20*=20is=20= used=20for=20absorbing=20all=20barriers,=20and=20the=20set=20of=20= conditions=20to=20use=20is=0A+=20*=20looked=20up=20in=20the=20= checksum_barriers=20struct.=20=20The=20struct=20member=20for=20the=20= target=0A+=20*=20state=20defines=20which=20state=20the=20backend=20must=20= currently=20be=20in,=20and=20which=20it=0A+=20*=20must=20not=20be=20in.=0A= +=20*=0A+=20*=20The=20reason=20for=20this=20explicit=20checking=20is=20= to=20ensure=20that=20processing=20cannot=0A+=20*=20be=20started=20such=20= that=20it=20breaks=20the=20assumptions=20of=20the=20state=20machine.=20=20= See=0A+=20*=20datachecksumsworker.c=20for=20a=20lengthy=20discussion=20= on=20these=20states.=0A+=20*/=0A+typedef=20struct=20= ChecksumBarrierCondition=0A+{=0A+=09/*=20The=20target=20state=20of=20the=20= barrier=20*/=0A+=09int=09=09=09target;=0A+=09/*=20A=20set=20of=20states=20= in=20which=20at=20least=20one=20MUST=20match=20the=20current=20state=20= */=0A+=09int=09=09=09barrier_eq[MAX_BARRIER_CONDITIONS];=0A+=09/*=20The=20= number=20of=20elements=20in=20the=20barrier_eq=20set=20*/=0A+=09int=09=09= =09barrier_eq_sz;=0A+=09/*=20A=20set=20of=20states=20which=20all=20MUST=20= NOT=20match=20the=20current=20state=20*/=0A+=09int=09=09=09= barrier_ne[MAX_BARRIER_CONDITIONS];=0A+=09/*=20The=20number=20of=20= elements=20in=20the=20barrier_ne=20set=20*/=0A+=09int=09=09=09= barrier_ne_sz;=0A+}=20ChecksumBarrierCondition;=0A+=0A+static=20const=20= ChecksumBarrierCondition=20checksum_barriers[4]=20=3D=0A+{=0A+=09= {PG_DATA_CHECKSUM_OFF,=20{PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION,=20= PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION},=202,=20= {PG_DATA_CHECKSUM_VERSION},=201},=0A+=09{PG_DATA_CHECKSUM_VERSION,=20= {PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION},=201,=20{0},=200},=0A+=09= {PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION,=20{0},=200,=20= {PG_DATA_CHECKSUM_VERSION},=201},=0A+=09= {PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION,=20{PG_DATA_CHECKSUM_VERSION},=20= 1,=20{0},=200},=0A+};=0A+=0A=20/*=0A=20=20*=20Calculate=20the=20amount=20= of=20space=20left=20on=20the=20page=20after=20'endptr'.=20Beware=0A=20=20= *=20multiple=20evaluation!=0A@@=20-649,6=20+692,23=20@@=20static=20= XLogRecPtr=20LocalMinRecoveryPoint;=0A=20static=20TimeLineID=20= LocalMinRecoveryPointTLI;=0A=20static=20bool=20updateMinRecoveryPoint=20= =3D=20true;=0A=20=0A+/*=0A+=20*=20Local=20state=20for=20Controlfile=20= data_checksum_version.=20=20After=20initialization=0A+=20*=20this=20is=20= only=20updated=20when=20absorbing=20a=20procsignal=20barrier=20during=20= interrupt=0A+=20*=20processing.=20=20The=20reason=20for=20keeping=20a=20= copy=20in=20backend-private=20memory=20is=20to=0A+=20*=20avoid=20locking=20= for=20interrogating=20the=20data=20checksum=20state.=20=20Possible=20= values=0A+=20*=20are=20the=20data=20checksum=20versions=20defined=20in=20= storage/checksum.h.=0A+=20*/=0A+static=20ChecksumType=20= LocalDataChecksumVersion=20=3D=200;=0A+=0A+/*=0A+=20*=20Variable=20= backing=20the=20GUC,=20keep=20it=20in=20sync=20with=20= LocalDataChecksumVersion.=0A+=20*=20See=20SetLocalDataChecksumVersion().=0A= +=20*/=0A+int=09=09=09data_checksums=20=3D=200;=0A+=0A+static=20void=20= SetLocalDataChecksumVersion(uint32=20data_checksum_version);=0A+=0A=20/*=20= For=20WALInsertLockAcquire/Release=20functions=20*/=0A=20static=20int=09= MyLockNo=20=3D=200;=0A=20static=20bool=20holdingAllLocks=20=3D=20false;=0A= @@=20-716,6=20+776,8=20@@=20static=20void=20= WALInsertLockAcquireExclusive(void);=0A=20static=20void=20= WALInsertLockRelease(void);=0A=20static=20void=20= WALInsertLockUpdateInsertingAt(XLogRecPtr=20insertingAt);=0A=20=0A= +static=20void=20XLogChecksums(uint32=20new_type);=0A+=0A=20/*=0A=20=20*=20= Insert=20an=20XLOG=20record=20represented=20by=20an=20= already-constructed=20chain=20of=20data=0A=20=20*=20chunks.=20=20This=20= is=20a=20low-level=20routine;=20to=20construct=20the=20WAL=20record=20= header=0A@@=20-830,9=20+892,10=20@@=20XLogInsertRecord(XLogRecData=20= *rdata,=0A=20=09=09=20*=20only=20happen=20just=20after=20a=20checkpoint,=20= so=20it's=20better=20to=20be=20slow=20in=0A=20=09=09=20*=20this=20case=20= and=20fast=20otherwise.=0A=20=09=09=20*=0A-=09=09=20*=20Also=20check=20= to=20see=20if=20fullPageWrites=20was=20just=20turned=20on=20or=20there's=20= a=0A-=09=09=20*=20running=20backup=20(which=20forces=20full-page=20= writes);=20if=20we=20weren't=0A-=09=09=20*=20already=20doing=20full-page=20= writes=20then=20go=20back=20and=20recompute.=0A+=09=09=20*=20Also=20= check=20to=20see=20if=20fullPageWrites=20was=20just=20turned=20on,=20= there's=20a=0A+=09=09=20*=20running=20backup=20or=20if=20checksums=20are=20= enabled=20(all=20of=20which=20forces=0A+=09=09=20*=20full-page=20= writes);=20if=20we=20weren't=20already=20doing=20full-page=20writes=0A+=09= =09=20*=20then=20go=20back=20and=20recompute.=0A=20=09=09=20*=0A=20=09=09= =20*=20If=20we=20aren't=20doing=20full-page=20writes=20then=20RedoRecPtr=20= doesn't=0A=20=09=09=20*=20actually=20affect=20the=20contents=20of=20the=20= XLOG=20record,=20so=20we'll=20update=0A@@=20-4270,6=20+4333,12=20@@=20= InitControlFile(uint64=20sysidentifier,=20uint32=20= data_checksum_version)=0A=20=09ControlFile->wal_log_hints=20=3D=20= wal_log_hints;=0A=20=09ControlFile->track_commit_timestamp=20=3D=20= track_commit_timestamp;=0A=20=09ControlFile->data_checksum_version=20=3D=20= data_checksum_version;=0A+=0A+=09/*=0A+=09=20*=20Set=20the=20= data_checksum_version=20value=20into=20XLogCtl,=20which=20is=20where=20= all=0A+=09=20*=20processes=20get=20the=20current=20value=20from.=0A+=09=20= */=0A+=09XLogCtl->data_checksum_version=20=3D=20data_checksum_version;=0A= =20}=0A=20=0A=20static=20void=0A@@=20-4605,9=20+4674,9=20@@=20= ReadControlFile(void)=0A=20=0A=20=09CalculateCheckpointSegments();=0A=20=0A= -=09/*=20Make=20the=20initdb=20settings=20visible=20as=20GUC=20= variables,=20too=20*/=0A-=09SetConfigOption("data_checksums",=20= DataChecksumsEnabled()=20?=20"yes"=20:=20"no",=0A-=09=09=09=09=09= PGC_INTERNAL,=20PGC_S_DYNAMIC_DEFAULT);=0A+=09elog(LOG,=20= "ReadControlFile=20checkpoint=20%X/%08X=20redo=20%X/%08X",=0A+=09=09=20= LSN_FORMAT_ARGS(ControlFile->checkPoint),=0A+=09=09=20= LSN_FORMAT_ARGS(ControlFile->checkPointCopy.redo));=0A=20}=0A=20=0A=20/*=0A= @@=20-4641,13=20+4710,410=20@@=20GetMockAuthenticationNonce(void)=0A=20}=0A= =20=0A=20/*=0A-=20*=20Are=20checksums=20enabled=20for=20data=20pages?=0A= +=20*=20DataChecksumsNeedWrite=0A+=20*=09=09Returns=20whether=20data=20= checksums=20must=20be=20written=20or=20not=0A+=20*=0A+=20*=20Returns=20= true=20iff=20data=20checksums=20are=20enabled=20or=20are=20in=20the=20= process=20of=20being=0A+=20*=20enabled.=20During=20"inprogress-on"=20and=20= "inprogress-off"=20states=20checksums=20must=0A+=20*=20be=20written=20= even=20though=20they=20are=20not=20verified=20(see=20= datachecksumsworker.c=20for=0A+=20*=20a=20longer=20discussion).=0A+=20*=0A= +=20*=20This=20function=20is=20intended=20for=20callsites=20which=20are=20= about=20to=20write=20a=20data=20page=0A+=20*=20to=20storage,=20and=20= need=20to=20know=20whether=20to=20re-calculate=20the=20checksum=20for=20= the=0A+=20*=20page=20header.=20Calling=20this=20function=20must=20be=20= performed=20as=20close=20to=20the=20write=0A+=20*=20operation=20as=20= possible=20to=20keep=20the=20critical=20section=20short.=0A+=20*/=0A= +bool=0A+DataChecksumsNeedWrite(void)=0A+{=0A+=09return=20= (LocalDataChecksumVersion=20=3D=3D=20PG_DATA_CHECKSUM_VERSION=20||=0A+=09= =09=09LocalDataChecksumVersion=20=3D=3D=20= PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION=20||=0A+=09=09=09= LocalDataChecksumVersion=20=3D=3D=20= PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION);=0A+}=0A+=0A+/*=0A+=20*=20= DataChecksumsNeedVerify=0A+=20*=09=09Returns=20whether=20data=20= checksums=20must=20be=20verified=20or=20not=0A+=20*=0A+=20*=20Data=20= checksums=20are=20only=20verified=20if=20they=20are=20fully=20enabled=20= in=20the=20cluster.=0A+=20*=20During=20the=20"inprogress-on"=20and=20= "inprogress-off"=20states=20they=20are=20only=0A+=20*=20updated,=20not=20= verified=20(see=20datachecksumsworker.c=20for=20a=20longer=20= discussion).=0A+=20*=0A+=20*=20This=20function=20is=20intended=20for=20= callsites=20which=20have=20read=20data=20and=20are=20about=0A+=20*=20to=20= perform=20checksum=20validation=20based=20on=20the=20result=20of=20this.=20= =20Calling=20this=0A+=20*=20function=20must=20be=20performed=20as=20= close=20to=20the=20validation=20call=20as=20possible=20to=0A+=20*=20keep=20= the=20critical=20section=20short.=20This=20is=20in=20order=20to=20= protect=20against=20time=20of=0A+=20*=20check/time=20of=20use=20= situations=20around=20data=20checksum=20validation.=0A+=20*/=0A+bool=0A= +DataChecksumsNeedVerify(void)=0A+{=0A+=09return=20= (LocalDataChecksumVersion=20=3D=3D=20PG_DATA_CHECKSUM_VERSION);=0A+}=0A+=0A= +/*=0A+=20*=20DataChecksumsOnInProgress=0A+=20*=09=09Returns=20whether=20= data=20checksums=20are=20being=20enabled=0A+=20*=0A+=20*=20Most=20= operations=20don't=20need=20to=20worry=20about=20the=20"inprogress"=20= states,=20and=0A+=20*=20should=20use=20DataChecksumsNeedVerify()=20or=20= DataChecksumsNeedWrite().=20The=0A+=20*=20"inprogress-on"=20state=20for=20= enabling=20checksums=20is=20used=20when=20the=20checksum=0A+=20*=20= worker=20is=20setting=20checksums=20on=20all=20pages,=20it=20can=20thus=20= be=20used=20to=20check=20for=0A+=20*=20aborted=20checksum=20processing=20= which=20need=20to=20be=20restarted.=0A+=20*/=0A+inline=20bool=0A= +DataChecksumsOnInProgress(void)=0A+{=0A+=09return=20= (LocalDataChecksumVersion=20=3D=3D=20= PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION);=0A+}=0A+=0A+/*=0A+=20*=20= DataChecksumsOffInProgress=0A+=20*=09=09Returns=20whether=20data=20= checksums=20are=20being=20disabled=0A+=20*=0A+=20*=20The=20= "inprogress-off"=20state=20for=20disabling=20checksums=20is=20used=20for=20= when=20the=0A+=20*=20worker=20resets=20the=20catalog=20state.=20=20= DataChecksumsNeedVerify()=20or=0A+=20*=20DataChecksumsNeedWrite()=20= should=20be=20used=20for=20deciding=20whether=20to=20read/write=0A+=20*=20= checksums.=0A=20=20*/=0A=20bool=0A-DataChecksumsEnabled(void)=0A= +DataChecksumsOffInProgress(void)=0A+{=0A+=09return=20= (LocalDataChecksumVersion=20=3D=3D=20= PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION);=0A+}=0A+=0A+/*=0A+=20*=20= SetDataChecksumsOnInProgress=0A+=20*=09=09Sets=20the=20data=20checksum=20= state=20to=20"inprogress-on"=20to=20enable=20checksums=0A+=20*=0A+=20*=20= To=20start=20the=20process=20of=20enabling=20data=20checksums=20in=20a=20= running=20cluster=20the=0A+=20*=20data_checksum_version=20state=20must=20= be=20changed=20to=20"inprogress-on".=20See=0A+=20*=20SetDataChecksumsOn=20= below=20for=20a=20description=20on=20how=20this=20state=20change=20= works.=0A+=20*=20This=20function=20blocks=20until=20all=20backends=20in=20= the=20cluster=20have=20acknowledged=20the=0A+=20*=20state=20transition.=0A= +=20*/=0A+void=0A+SetDataChecksumsOnInProgress(void)=0A=20{=0A+=09uint64=09= =09barrier;=0A+=0A=20=09Assert(ControlFile=20!=3D=20NULL);=0A-=09return=20= (ControlFile->data_checksum_version=20>=200);=0A+=0A+=09/*=0A+=09=20*=20= The=20state=20transition=20is=20performed=20in=20a=20critical=20section=20= with=0A+=09=20*=20checkpoints=20held=20off=20to=20provide=20crash=20= safety.=0A+=09=20*/=0A+=09START_CRIT_SECTION();=0A+=09= MyProc->delayChkptFlags=20|=3D=20DELAY_CHKPT_START;=0A+=0A+=09= XLogChecksums(PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION);=0A+=0A+=09= SpinLockAcquire(&XLogCtl->info_lck);=0A+=09= Assert(XLogCtl->data_checksum_version=20=3D=3D=20PG_DATA_CHECKSUM_OFF);=0A= +=09XLogCtl->data_checksum_version=20=3D=20= PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION;=0A+=09= SpinLockRelease(&XLogCtl->info_lck);=0A+=0A+=09barrier=20=3D=20= EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_ON);=0A+=0A= +=09MyProc->delayChkptFlags=20&=3D=20~DELAY_CHKPT_START;=0A+=09= END_CRIT_SECTION();=0A+=0A+=09/*=0A+=09=20*=20Await=20state=20change=20= in=20all=20backends=20to=20ensure=20that=20all=20backends=20are=20in=0A+=09= =20*=20"inprogress-on".=20Once=20done=20we=20know=20that=20all=20= backends=20are=20writing=20data=0A+=09=20*=20checksums.=0A+=09=20*/=0A+=09= WaitForProcSignalBarrier(barrier);=0A+}=0A+=0A+/*=0A+=20*=20= SetDataChecksumsOn=0A+=20*=09=09Enables=20data=20checksums=20= cluster-wide=0A+=20*=0A+=20*=20Enabling=20data=20checksums=20is=20= performed=20using=20two=20barriers,=20the=20first=20one=20to=0A+=20*=20= set=20the=20state=20to=20"inprogress-on"=20(done=20by=20= SetDataChecksumsOnInProgress())=0A+=20*=20and=20the=20second=20one=20to=20= set=20the=20state=20to=20"on"=20(done=20here).=20Below=20is=20a=20short=0A= +=20*=20description=20of=20the=20processing,=20a=20more=20detailed=20= write-up=20can=20be=20found=20in=0A+=20*=20datachecksumsworker.c.=0A+=20= *=0A+=20*=20To=20start=20the=20process=20of=20enabling=20data=20= checksums=20in=20a=20running=20cluster=20the=0A+=20*=20= data_checksum_version=20state=20must=20be=20changed=20to=20= "inprogress-on".=20=20This=20state=0A+=20*=20requires=20data=20checksums=20= to=20be=20written=20but=20not=20verified.=20This=20ensures=20that=0A+=20= *=20all=20data=20pages=20can=20be=20checksummed=20without=20the=20risk=20= of=20false=20negatives=20in=0A+=20*=20validation=20during=20the=20= process.=20=20When=20all=20existing=20pages=20are=20guaranteed=20to=0A+=20= *=20have=20checksums,=20and=20all=20new=20pages=20will=20be=20initiated=20= with=20checksums,=20the=0A+=20*=20state=20can=20be=20changed=20to=20= "on".=20Once=20the=20state=20is=20"on"=20checksums=20will=20be=20both=0A= +=20*=20written=20and=20verified.=20See=20datachecksumsworker.c=20for=20= a=20longer=20discussion=20on=0A+=20*=20how=20data=20checksums=20can=20be=20= enabled=20in=20a=20running=20cluster.=0A+=20*=0A+=20*=20This=20function=20= blocks=20until=20all=20backends=20in=20the=20cluster=20have=20= acknowledged=20the=0A+=20*=20state=20transition.=0A+=20*/=0A+void=0A= +SetDataChecksumsOn(void)=0A+{=0A+=09uint64=09=09barrier;=0A+=0A+=09= Assert(ControlFile=20!=3D=20NULL);=0A+=0A+=09= SpinLockAcquire(&XLogCtl->info_lck);=0A+=0A+=09/*=0A+=09=20*=20The=20= only=20allowed=20state=20transition=20to=20"on"=20is=20from=20= "inprogress-on"=20since=0A+=09=20*=20that=20state=20ensures=20that=20all=20= pages=20will=20have=20data=20checksums=20written.=0A+=09=20*/=0A+=09if=20= (XLogCtl->data_checksum_version=20!=3D=20= PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION)=0A+=09{=0A+=09=09= SpinLockRelease(&XLogCtl->info_lck);=0A+=09=09elog(PANIC,=20"checksums=20= not=20in=20\"inprogress-on\"=20mode");=0A+=09}=0A+=0A+=09= SpinLockRelease(&XLogCtl->info_lck);=0A+=0A+=09= INJECTION_POINT("datachecksums-enable-checksums-delay",=20NULL);=0A+=09= START_CRIT_SECTION();=0A+=09MyProc->delayChkptFlags=20|=3D=20= DELAY_CHKPT_START;=0A+=0A+=09XLogChecksums(PG_DATA_CHECKSUM_VERSION);=0A= +=0A+=09SpinLockAcquire(&XLogCtl->info_lck);=0A+=09= XLogCtl->data_checksum_version=20=3D=20PG_DATA_CHECKSUM_VERSION;=0A+=09= SpinLockRelease(&XLogCtl->info_lck);=0A+=0A+=09barrier=20=3D=20= EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_ON);=0A+=0A+=09= MyProc->delayChkptFlags=20&=3D=20~DELAY_CHKPT_START;=0A+=09= END_CRIT_SECTION();=0A+=0A+=09/*=0A+=09=20*=20Await=20state=20transition=20= of=20"on"=20in=20all=20backends.=20When=20done=20we=20know=20that=0A+=09=20= *=20data=20checksums=20are=20enabled=20in=20all=20backends=20and=20data=20= checksums=20are=20both=0A+=09=20*=20written=20and=20verified.=0A+=09=20= */=0A+=09WaitForProcSignalBarrier(barrier);=0A+=0A+=09= INJECTION_POINT("datachecksums-enable-checksums-done",=20NULL);=0A+}=0A+=0A= +/*=0A+=20*=20SetDataChecksumsOff=0A+=20*=09=09Disables=20data=20= checksums=20cluster-wide=0A+=20*=0A+=20*=20Disabling=20data=20checksums=20= must=20be=20performed=20with=20two=20sets=20of=20barriers,=20each=0A+=20= *=20carrying=20a=20different=20state.=20The=20state=20is=20first=20set=20= to=20"inprogress-off"=0A+=20*=20during=20which=20checksums=20are=20still=20= written=20but=20not=20verified.=20This=20ensures=20that=0A+=20*=20= backends=20which=20have=20yet=20to=20observe=20the=20state=20change=20= from=20"on"=20won't=20get=0A+=20*=20validation=20errors=20on=20= concurrently=20modified=20pages.=20Once=20all=20backends=20have=0A+=20*=20= changed=20to=20"inprogress-off",=20the=20barrier=20for=20moving=20to=20= "off"=20can=20be=20emitted.=0A+=20*=20This=20function=20blocks=20until=20= all=20backends=20in=20the=20cluster=20have=20acknowledged=20the=0A+=20*=20= state=20transition.=0A+=20*/=0A+void=0A+SetDataChecksumsOff(void)=0A+{=0A= +=09uint64=09=09barrier;=0A+=0A+=09Assert(ControlFile=20!=3D=20NULL);=0A= +=0A+=09SpinLockAcquire(&XLogCtl->info_lck);=0A+=0A+=09/*=20If=20data=20= checksums=20are=20already=20disabled=20there=20is=20nothing=20to=20do=20= */=0A+=09if=20(XLogCtl->data_checksum_version=20=3D=3D=200)=0A+=09{=0A+=09= =09SpinLockRelease(&XLogCtl->info_lck);=0A+=09=09return;=0A+=09}=0A+=0A+=09= /*=0A+=09=20*=20If=20data=20checksums=20are=20currently=20enabled=20we=20= first=20transition=20to=20the=0A+=09=20*=20"inprogress-off"=20state=20= during=20which=20backends=20continue=20to=20write=0A+=09=20*=20checksums=20= without=20verifying=20them.=20When=20all=20backends=20are=20in=0A+=09=20= *=20"inprogress-off"=20the=20next=20transition=20to=20"off"=20can=20be=20= performed,=20after=0A+=09=20*=20which=20all=20data=20checksum=20= processing=20is=20disabled.=0A+=09=20*/=0A+=09if=20= (XLogCtl->data_checksum_version=20=3D=3D=20PG_DATA_CHECKSUM_VERSION)=0A+=09= {=0A+=09=09SpinLockRelease(&XLogCtl->info_lck);=0A+=0A+=09=09= START_CRIT_SECTION();=0A+=09=09MyProc->delayChkptFlags=20|=3D=20= DELAY_CHKPT_START;=0A+=0A+=09=09= XLogChecksums(PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION);=0A+=0A+=09=09= SpinLockAcquire(&XLogCtl->info_lck);=0A+=09=09= XLogCtl->data_checksum_version=20=3D=20= PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION;=0A+=09=09= SpinLockRelease(&XLogCtl->info_lck);=0A+=0A+=09=09barrier=20=3D=20= EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_OFF);=0A+=0A= +=09=09MyProc->delayChkptFlags=20&=3D=20~DELAY_CHKPT_START;=0A+=09=09= END_CRIT_SECTION();=0A+=0A+=09=09/*=0A+=09=09=20*=20Update=20local=20= state=20in=20all=20backends=20to=20ensure=20that=20any=20backend=20in=0A= +=09=09=20*=20"on"=20state=20is=20changed=20to=20"inprogress-off".=0A+=09= =09=20*/=0A+=09=09WaitForProcSignalBarrier(barrier);=0A+=0A+=09=09/*=0A+=09= =09=20*=20At=20this=20point=20we=20know=20that=20no=20backends=20are=20= verifying=20data=20checksums=0A+=09=09=20*=20during=20reading.=20Next,=20= we=20can=20safely=20move=20to=20state=20"off"=20to=20also=0A+=09=09=20*=20= stop=20writing=20checksums.=0A+=09=09=20*/=0A+=09}=0A+=09else=0A+=09{=0A= +=09=09/*=0A+=09=09=20*=20Ending=20up=20here=20implies=20that=20the=20= checksums=20state=20is=20"inprogress-on"=0A+=09=09=20*=20or=20= "inprogress-off"=20and=20we=20can=20transition=20directly=20to=20"off"=20= from=0A+=09=09=20*=20there.=0A+=09=09=20*/=0A+=09=09= SpinLockRelease(&XLogCtl->info_lck);=0A+=09}=0A+=0A+=09= START_CRIT_SECTION();=0A+=09/*=20Ensure=20that=20we=20don't=20incur=20a=20= checkpoint=20during=20disabling=20checksums=20*/=0A+=09= MyProc->delayChkptFlags=20|=3D=20DELAY_CHKPT_START;=0A+=0A+=09= XLogChecksums(PG_DATA_CHECKSUM_OFF);=0A+=0A+=09= SpinLockAcquire(&XLogCtl->info_lck);=0A+=09= XLogCtl->data_checksum_version=20=3D=200;=0A+=09= SpinLockRelease(&XLogCtl->info_lck);=0A+=0A+=09barrier=20=3D=20= EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_OFF);=0A+=0A+=09= MyProc->delayChkptFlags=20&=3D=20~DELAY_CHKPT_START;=0A+=09= END_CRIT_SECTION();=0A+=0A+=09WaitForProcSignalBarrier(barrier);=0A+}=0A= +=0A+/*=0A+=20*=20AbsorbDataChecksumsBarrier=0A+=20*=09=09Generic=20= function=20for=20absorbing=20data=20checksum=20state=20changes=0A+=20*=0A= +=20*=20All=20procsignalbarriers=20regarding=20data=20checksum=20state=20= changes=20are=20absorbed=0A+=20*=20with=20this=20function.=20=20The=20= set=20of=20conditions=20required=20for=20the=20state=20change=20to=0A+=20= *=20be=20accepted=20are=20listed=20in=20the=20checksum_barriers=20= struct,=20target_state=20is=0A+=20*=20used=20to=20look=20up=20the=20= relevant=20entry.=0A+=20*/=0A+bool=0A+AbsorbDataChecksumsBarrier(int=20= target_state)=0A+{=0A+=09const=20ChecksumBarrierCondition=20*condition=20= =3D=20NULL;=0A+=09int=09=09=09current=20=3D=20LocalDataChecksumVersion;=0A= +=09bool=09=09found=20=3D=20false;=0A+=0A+=09/*=0A+=09=20*=20If=20the=20= target=20state=20matches=20the=20current=20state=20then=20the=20barrier=20= has=20been=0A+=09=20*=20repeated.=0A+=09=20*/=0A+=09if=20(current=20=3D=3D= =20target_state)=0A+=09=09return=20true;=0A+=0A+=09/*=0A+=09=20*=20If=20= the=20cluster=20is=20in=20recovery=20we=20skip=20the=20validation=20of=20= current=20state=0A+=09=20*=20since=20the=20replay=20is=20trusted.=0A+=09=20= */=0A+=09if=20(InRecovery)=0A+=09{=0A+=09=09= SetLocalDataChecksumVersion(target_state);=0A+=09=09return=20true;=0A+=09= }=0A+=0A+=09/*=0A+=09=20*=20Find=20the=20barrier=20condition=20= definition=20for=20the=20target=20state.=20Not=20finding=0A+=09=20*=20a=20= condition=20would=20be=20a=20grave=20programmer=20error=20as=20the=20= states=20are=20a=0A+=09=20*=20discrete=20set.=0A+=09=20*/=0A+=09for=20= (int=20i=20=3D=200;=20i=20<=20lengthof(checksum_barriers);=20i++)=0A+=09= {=0A+=09=09if=20(checksum_barriers[i].target=20=3D=3D=20target_state)=0A= +=09=09=09condition=20=3D=20&checksum_barriers[i];=0A+=09}=0A+=09= Assert(condition);=0A+=0A+=09/*=0A+=09=20*=20The=20current=20state=20= MUST=20be=20equal=20to=20one=20of=20the=20EQ=20states=20defined=20in=20= this=0A+=09=20*=20barrier=20condition.=20If=20the=20EQ=20states=20array=20= is=20zero=20then=20that=20imples=20that=0A+=09=20*=20the=20current=20= state=20can=20match=20any=20state,=20so=20fastpath=20check=20for=20that=0A= +=09=20*=20first.=0A+=09=20*/=0A+=09if=20(condition->barrier_eq_sz=20=3D=3D= =200)=0A+=09=09found=20=3D=20true;=0A+=09else=0A+=09{=0A+=09=09for=20= (int=20i=20=3D=200;=20i=20<=20condition->barrier_eq_sz;=20i++)=0A+=09=09= {=0A+=09=09=09if=20(current=20=3D=3D=20condition->barrier_eq[i])=0A+=09=09= =09=09found=20=3D=20true;=0A+=09=09}=0A+=09}=0A+=0A+=09/*=0A+=09=20*=20= The=20current=20state=20MUST=20NOT=20be=20equal=20to=20any=20of=20the=20= NE=20states=20defined=20in=0A+=09=20*=20this=20barrier=20condition.=0A+=09= =20*/=0A+=09for=20(int=20i=20=3D=200;=20i=20<=20= condition->barrier_ne_sz;=20i++)=0A+=09{=0A+=09=09if=20(current=20=3D=3D=20= condition->barrier_ne[i])=0A+=09=09=09found=20=3D=20false;=0A+=09}=0A+=0A= +=09/*=0A+=09=20*=20If=20the=20relevant=20state=20criteria=20aren't=20= satisfied,=20throw=20an=20error=20which=0A+=09=20*=20will=20be=20caught=20= by=20the=20procsignal=20machinery=20for=20a=20later=20retry.=0A+=09=20*/=0A= +=09if=20(!found)=0A+=09=09ereport(ERROR,=0A+=09=09=09=09= errcode(ERRCODE_INVALID_PARAMETER_VALUE),=0A+=09=09=09=09= errmsg("incorrect=20data=20checksum=20state=20%i=20for=20target=20state=20= %i",=0A+=09=09=09=09=09=20=20=20current,=20target_state));=0A+=0A+=09= SetLocalDataChecksumVersion(target_state);=0A+=09return=20true;=0A+}=0A+=0A= +/*=0A+=20*=20InitLocalControlData=0A+=20*=0A+=20*=20Set=20up=20backend=20= local=20caches=20of=20controldata=20variables=20which=20may=20change=20= at=0A+=20*=20any=20point=20during=20runtime=20and=20thus=20require=20= special=20cased=20locking.=20So=20far=0A+=20*=20this=20only=20applies=20= to=20data_checksum_version,=20but=20it's=20intended=20to=20be=20general=0A= +=20*=20purpose=20enough=20to=20handle=20future=20cases.=0A+=20*/=0A= +void=0A+InitLocalDataChecksumVersion(void)=0A+{=0A+=09= SpinLockAcquire(&XLogCtl->info_lck);=0A+=09= SetLocalDataChecksumVersion(XLogCtl->data_checksum_version);=0A+=09= SpinLockRelease(&XLogCtl->info_lck);=0A+}=0A+=0A+void=0A= +SetLocalDataChecksumVersion(uint32=20data_checksum_version)=0A+{=0A+=09= LocalDataChecksumVersion=20=3D=20data_checksum_version;=0A+=0A+=09= data_checksums=20=3D=20data_checksum_version;=0A+}=0A+=0A+/*=20guc=20= hook=20*/=0A+const=20char=20*=0A+show_data_checksums(void)=0A+{=0A+=09= switch=20(LocalDataChecksumVersion)=0A+=09{=0A+=09=09case=20= PG_DATA_CHECKSUM_VERSION:=0A+=09=09=09return=20"on";=0A+=09=09case=20= PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION:=0A+=09=09=09return=20= "inprogress-on";=0A+=09=09case=20= PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION:=0A+=09=09=09return=20= "inprogress-off";=0A+=09=09case=20PG_DATA_CHECKSUM_OFF:=0A+=09=09=09= return=20"off";=0A+=09}=0A=20}=0A=20=0A=20/*=0A@@=20-4941,6=20+5407,7=20= @@=20LocalProcessControlFile(bool=20reset)=0A=20=09Assert(reset=20||=20= ControlFile=20=3D=3D=20NULL);=0A=20=09ControlFile=20=3D=20= palloc_object(ControlFileData);=0A=20=09ReadControlFile();=0A+=09= SetLocalDataChecksumVersion(ControlFile->data_checksum_version);=0A=20}=0A= =20=0A=20/*=0A@@=20-5110,6=20+5577,11=20@@=20XLOGShmemInit(void)=0A=20=09= XLogCtl->InstallXLogFileSegmentActive=20=3D=20false;=0A=20=09= XLogCtl->WalWriterSleeping=20=3D=20false;=0A=20=0A+=09/*=20Use=20the=20= checksum=20info=20from=20control=20file=20*/=0A+=09= XLogCtl->data_checksum_version=20=3D=20= ControlFile->data_checksum_version;=0A+=0A+=09= SetLocalDataChecksumVersion(XLogCtl->data_checksum_version);=0A+=0A=20=09= SpinLockInit(&XLogCtl->Insert.insertpos_lck);=0A=20=09= SpinLockInit(&XLogCtl->info_lck);=0A=20=09= pg_atomic_init_u64(&XLogCtl->logInsertResult,=20InvalidXLogRecPtr);=0A@@=20= -5798,6=20+6270,11=20@@=20StartupXLOG(void)=0A=20=09=09=09= XLogCtl->SharedRecoveryState=20=3D=20RECOVERY_STATE_ARCHIVE;=0A=20=09=09= else=0A=20=09=09=09XLogCtl->SharedRecoveryState=20=3D=20= RECOVERY_STATE_CRASH;=0A+=0A+=09=09XLogCtl->data_checksum_version=20=3D=20= checkPoint.dataChecksumVersion;=0A+=09=09= SetLocalDataChecksumVersion(checkPoint.dataChecksumVersion);=0A+=09=09= ControlFile->data_checksum_version=20=3D=20= checkPoint.dataChecksumVersion;=0A+=0A=20=09=09= SpinLockRelease(&XLogCtl->info_lck);=0A=20=0A=20=09=09/*=0A@@=20-6260,6=20= +6737,47=20@@=20StartupXLOG(void)=0A=20=09= pfree(endOfRecoveryInfo->recoveryStopReason);=0A=20=09= pfree(endOfRecoveryInfo);=0A=20=0A+=09/*=0A+=09=20*=20If=20we=20reach=20= this=20point=20with=20checksums=20in=20the=20state=20inprogress-on,=20it=0A= +=09=20*=20means=20that=20data=20checksums=20were=20in=20the=20process=20= of=20being=20enabled=20when=20the=0A+=09=20*=20cluster=20shut=20down.=20= Since=20processing=20didn't=20finish,=20the=20operation=20will=0A+=09=20= *=20have=20to=20be=20restarted=20from=20scratch=20since=20there=20is=20= no=20capability=20to=0A+=09=20*=20continue=20where=20it=20was=20when=20= the=20cluster=20shut=20down.=20Thus,=20revert=20the=0A+=09=20*=20state=20= back=20to=20off,=20and=20inform=20the=20user=20with=20a=20warning=20= message.=20Being=0A+=09=20*=20able=20to=20restart=20processing=20is=20a=20= TODO,=20but=20it=20wouldn't=20be=20possible=20to=0A+=09=20*=20restart=20= here=20since=20we=20cannot=20launch=20a=20dynamic=20background=20worker=0A= +=09=20*=20directly=20from=20here=20(it=20has=20to=20be=20from=20a=20= regular=20backend).=0A+=09=20*/=0A+=09if=20= (XLogCtl->data_checksum_version=20=3D=3D=20= PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION)=0A+=09{=0A+=09=09= XLogChecksums(PG_DATA_CHECKSUM_OFF);=0A+=0A+=09=09= SpinLockAcquire(&XLogCtl->info_lck);=0A+=09=09= XLogCtl->data_checksum_version=20=3D=200;=0A+=09=09= SetLocalDataChecksumVersion(XLogCtl->data_checksum_version);=0A+=09=09= SpinLockRelease(&XLogCtl->info_lck);=0A+=0A+=09=09ereport(WARNING,=0A+=09= =09=09=09errmsg("data=20checksums=20state=20has=20been=20set=20to=20= off"),=0A+=09=09=09=09errhint("If=20checksums=20were=20being=20enabled=20= during=20shutdown=20then=20processing=20must=20be=20manually=20= restarted."));=0A+=09}=0A+=0A+=09/*=0A+=09=20*=20If=20data=20checksums=20= were=20being=20disabled=20when=20the=20cluster=20was=20shut=20down,=0A+=09= =20*=20we=20know=20that=20we=20have=20a=20state=20where=20all=20backends=20= have=20stopped=20validating=0A+=09=20*=20checksums=20and=20we=20can=20= move=20to=20off=20instead=20of=20prompting=20the=20user=20to=0A+=09=20*=20= perform=20any=20action.=0A+=09=20*/=0A+=09if=20= (XLogCtl->data_checksum_version=20=3D=3D=20= PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION)=0A+=09{=0A+=09=09= XLogChecksums(PG_DATA_CHECKSUM_OFF);=0A+=0A+=09=09= SpinLockAcquire(&XLogCtl->info_lck);=0A+=09=09= XLogCtl->data_checksum_version=20=3D=200;=0A+=09=09= SetLocalDataChecksumVersion(XLogCtl->data_checksum_version);=0A+=09=09= SpinLockRelease(&XLogCtl->info_lck);=0A+=09}=0A+=0A=20=09/*=0A=20=09=20*=20= All=20done=20with=20end-of-recovery=20actions.=0A=20=09=20*=0A@@=20= -6565,7=20+7083,7=20@@=20GetRedoRecPtr(void)=0A=20=09XLogRecPtr=09ptr;=0A= =20=0A=20=09/*=0A-=09=20*=20The=20possibly=20not=20up-to-date=20copy=20= in=20XlogCtl=20is=20enough.=20Even=20if=20we=0A+=09=20*=20The=20possibly=20= not=20up-to-date=20copy=20in=20XLogCtl=20is=20enough.=20Even=20if=20we=0A= =20=09=20*=20grabbed=20a=20WAL=20insertion=20lock=20to=20read=20the=20= authoritative=20value=20in=0A=20=09=20*=20Insert->RedoRecPtr,=20someone=20= might=20update=20it=20just=20after=20we've=20released=0A=20=09=20*=20the=20= lock.=0A@@=20-7133,6=20+7651,12=20@@=20CreateCheckPoint(int=20flags)=0A=20= =09checkPoint.fullPageWrites=20=3D=20Insert->fullPageWrites;=0A=20=09= checkPoint.wal_level=20=3D=20wal_level;=0A=20=0A+=09/*=0A+=09=20*=20Get=20= the=20current=20data_checksum_version=20value=20from=20xlogctl,=20valid=20= at=20the=0A+=09=20*=20time=20of=20the=20checkpoint.=0A+=09=20*/=0A+=09= checkPoint.dataChecksumVersion=20=3D=20XLogCtl->data_checksum_version;=0A= +=0A=20=09if=20(shutdown)=0A=20=09{=0A=20=09=09XLogRecPtr=09curInsert=20= =3D=20XLogBytePosToRecPtr(Insert->CurrBytePos);=0A@@=20-7185,9=20= +7709,18=20@@=20CreateCheckPoint(int=20flags)=0A=20=09=20*/=0A=20=09if=20= (!shutdown)=0A=20=09{=0A+=09=09xl_checkpoint_redo=20redo_rec;=0A+=0A+=09=09= WALInsertLockAcquire();=0A+=09=09redo_rec.wal_level=20=3D=20wal_level;=0A= +=09=09SpinLockAcquire(&XLogCtl->info_lck);=0A+=09=09= redo_rec.data_checksum_version=20=3D=20XLogCtl->data_checksum_version;=0A= +=09=09SpinLockRelease(&XLogCtl->info_lck);=0A+=09=09= WALInsertLockRelease();=0A+=0A=20=09=09/*=20Include=20WAL=20level=20in=20= record=20for=20WAL=20summarizer's=20benefit.=20*/=0A=20=09=09= XLogBeginInsert();=0A-=09=09XLogRegisterData(&wal_level,=20= sizeof(wal_level));=0A+=09=09XLogRegisterData(&redo_rec,=20= sizeof(xl_checkpoint_redo));=0A=20=09=09(void)=20XLogInsert(RM_XLOG_ID,=20= XLOG_CHECKPOINT_REDO);=0A=20=0A=20=09=09/*=0A@@=20-7243,6=20+7776,10=20= @@=20CreateCheckPoint(int=20flags)=0A=20=09=09checkPoint.nextOid=20+=3D=20= TransamVariables->oidCount;=0A=20=09LWLockRelease(OidGenLock);=0A=20=0A+=09= SpinLockAcquire(&XLogCtl->info_lck);=0A+=09= checkPoint.dataChecksumVersion=20=3D=20XLogCtl->data_checksum_version;=0A= +=09SpinLockRelease(&XLogCtl->info_lck);=0A+=0A=20=09= checkPoint.logicalDecodingEnabled=20=3D=20IsLogicalDecodingEnabled();=0A=20= =0A=20=09MultiXactGetCheckptMulti(shutdown,=0A@@=20-7392,6=20+7929,9=20= @@=20CreateCheckPoint(int=20flags)=0A=20=09ControlFile->minRecoveryPoint=20= =3D=20InvalidXLogRecPtr;=0A=20=09ControlFile->minRecoveryPointTLI=20=3D=20= 0;=0A=20=0A+=09/*=20make=20sure=20we=20start=20with=20the=20checksum=20= version=20as=20of=20the=20checkpoint=20*/=0A+=09= ControlFile->data_checksum_version=20=3D=20= checkPoint.dataChecksumVersion;=0A+=0A=20=09/*=0A=20=09=20*=20Persist=20= unloggedLSN=20value.=20It's=20reset=20on=20crash=20recovery,=20so=20this=20= goes=0A=20=09=20*=20unused=20on=20non-shutdown=20checkpoints,=20but=20= seems=20useful=20to=20store=20it=20always=0A@@=20-7535,6=20+8075,12=20@@=20= CreateEndOfRecoveryRecord(void)=0A=20=09LWLockAcquire(ControlFileLock,=20= LW_EXCLUSIVE);=0A=20=09ControlFile->minRecoveryPoint=20=3D=20recptr;=0A=20= =09ControlFile->minRecoveryPointTLI=20=3D=20xlrec.ThisTimeLineID;=0A+=0A= +=09/*=20start=20with=20the=20latest=20checksum=20version=20(as=20of=20= the=20end=20of=20recovery)=20*/=0A+=09= SpinLockAcquire(&XLogCtl->info_lck);=0A+=09= ControlFile->data_checksum_version=20=3D=20= XLogCtl->data_checksum_version;=0A+=09= SpinLockRelease(&XLogCtl->info_lck);=0A+=0A=20=09UpdateControlFile();=0A=20= =09LWLockRelease(ControlFileLock);=0A=20=0A@@=20-7876,6=20+8422,10=20@@=20= CreateRestartPoint(int=20flags)=0A=20=09=09=09if=20(flags=20&=20= CHECKPOINT_IS_SHUTDOWN)=0A=20=09=09=09=09ControlFile->state=20=3D=20= DB_SHUTDOWNED_IN_RECOVERY;=0A=20=09=09}=0A+=0A+=09=09/*=20we=20shall=20= start=20with=20the=20latest=20checksum=20version=20*/=0A+=09=09= ControlFile->data_checksum_version=20=3D=20= lastCheckPoint.dataChecksumVersion;=0A+=0A=20=09=09UpdateControlFile();=0A= =20=09}=0A=20=09LWLockRelease(ControlFileLock);=0A@@=20-8290,6=20= +8840,24=20@@=20XLogReportParameters(void)=0A=20=09}=0A=20}=0A=20=0A+/*=0A= +=20*=20Log=20the=20new=20state=20of=20checksums=0A+=20*/=0A+static=20= void=0A+XLogChecksums(uint32=20new_type)=0A+{=0A+=09xl_checksum_state=20= xlrec;=0A+=09XLogRecPtr=09recptr;=0A+=0A+=09xlrec.new_checksumtype=20=3D=20= new_type;=0A+=0A+=09XLogBeginInsert();=0A+=09XLogRegisterData((char=20*)=20= &xlrec,=20sizeof(xl_checksum_state));=0A+=0A+=09recptr=20=3D=20= XLogInsert(RM_XLOG_ID,=20XLOG_CHECKSUMS);=0A+=09XLogFlush(recptr);=0A+}=0A= +=0A=20/*=0A=20=20*=20Update=20full_page_writes=20in=20shared=20memory,=20= and=20write=20an=0A=20=20*=20XLOG_FPW_CHANGE=20record=20if=20necessary.=0A= @@=20-8416,6=20+8984,11=20@@=20xlog_redo(XLogReaderState=20*record)=0A=20= =09=09MultiXactAdvanceOldest(checkPoint.oldestMulti,=0A=20=09=09=09=09=09= =09=09=20=20=20checkPoint.oldestMultiDB);=0A=20=0A+=09=09= SpinLockAcquire(&XLogCtl->info_lck);=0A+=09=09= XLogCtl->data_checksum_version=20=3D=20checkPoint.dataChecksumVersion;=0A= +=09=09SetLocalDataChecksumVersion(checkPoint.dataChecksumVersion);=0A+=09= =09SpinLockRelease(&XLogCtl->info_lck);=0A+=0A=20=09=09/*=0A=20=09=09=20= *=20No=20need=20to=20set=20oldestClogXid=20here=20as=20well;=20it'll=20= be=20set=20when=20we=0A=20=09=09=20*=20redo=20an=20xl_clog_truncate=20if=20= it=20changed=20since=20initialization.=0A@@=20-8475,6=20+9048,7=20@@=20= xlog_redo(XLogReaderState=20*record)=0A=20=09=09/*=20= ControlFile->checkPointCopy=20always=20tracks=20the=20latest=20ckpt=20= XID=20*/=0A=20=09=09LWLockAcquire(ControlFileLock,=20LW_EXCLUSIVE);=0A=20= =09=09ControlFile->checkPointCopy.nextXid=20=3D=20checkPoint.nextXid;=0A= +=09=09ControlFile->data_checksum_version=20=3D=20= checkPoint.dataChecksumVersion;=0A=20=09=09= LWLockRelease(ControlFileLock);=0A=20=0A=20=09=09/*=0A@@=20-8539,8=20= +9113,14=20@@=20xlog_redo(XLogReaderState=20*record)=0A=20=09=09/*=20= ControlFile->checkPointCopy=20always=20tracks=20the=20latest=20ckpt=20= XID=20*/=0A=20=09=09LWLockAcquire(ControlFileLock,=20LW_EXCLUSIVE);=0A=20= =09=09ControlFile->checkPointCopy.nextXid=20=3D=20checkPoint.nextXid;=0A= +=09=09ControlFile->data_checksum_version=20=3D=20= checkPoint.dataChecksumVersion;=0A=20=09=09= LWLockRelease(ControlFileLock);=0A=20=0A+=09=09= SpinLockAcquire(&XLogCtl->info_lck);=0A+=09=09= XLogCtl->data_checksum_version=20=3D=20checkPoint.dataChecksumVersion;=0A= +=09=09SetLocalDataChecksumVersion(checkPoint.dataChecksumVersion);=0A+=09= =09SpinLockRelease(&XLogCtl->info_lck);=0A+=0A=20=09=09/*=20TLI=20should=20= not=20change=20in=20an=20on-line=20checkpoint=20*/=0A=20=09=09(void)=20= GetCurrentReplayRecPtr(&replayTLI);=0A=20=09=09if=20= (checkPoint.ThisTimeLineID=20!=3D=20replayTLI)=0A@@=20-8707,7=20+9287,43=20= @@=20xlog_redo(XLogReaderState=20*record)=0A=20=09}=0A=20=09else=20if=20= (info=20=3D=3D=20XLOG_CHECKPOINT_REDO)=0A=20=09{=0A-=09=09/*=20nothing=20= to=20do=20here,=20just=20for=20informational=20purposes=20*/=0A+=09=09= xl_checkpoint_redo=20redo_rec;=0A+=09=09bool=09=09new_state=20=3D=20= false;=0A+=09=09uint64=09=09barrier;=0A+=0A+=09=09memcpy(&redo_rec,=20= XLogRecGetData(record),=20sizeof(xl_checkpoint_redo));=0A+=0A+=09=09= SpinLockAcquire(&XLogCtl->info_lck);=0A+=09=09= XLogCtl->data_checksum_version=20=3D=20redo_rec.data_checksum_version;=0A= +=09=09if=20(redo_rec.data_checksum_version=20!=3D=20= ControlFile->data_checksum_version)=0A+=09=09=09new_state=20=3D=20true;=0A= +=09=09SpinLockRelease(&XLogCtl->info_lck);=0A+=0A+=09=09if=20= (new_state)=0A+=09=09{=0A+=09=09=09switch=20= (redo_rec.data_checksum_version)=0A+=09=09=09{=0A+=09=09=09=09case=20= PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION:=0A+=09=09=09=09=09barrier=20=3D=20= EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_ON);=0A+=09=09= =09=09=09WaitForProcSignalBarrier(barrier);=0A+=09=09=09=09=09break;=0A+=0A= +=09=09=09=09case=20PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION:=0A+=09=09=09= =09=09barrier=20=3D=20= EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_OFF);=0A+=09= =09=09=09=09WaitForProcSignalBarrier(barrier);=0A+=09=09=09=09=09break;=0A= +=0A+=09=09=09=09case=20PG_DATA_CHECKSUM_VERSION:=0A+=09=09=09=09=09= barrier=20=3D=20EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_ON);=0A= +=09=09=09=09=09WaitForProcSignalBarrier(barrier);=0A+=09=09=09=09=09= break;=0A+=0A+=09=09=09=09case=20PG_DATA_CHECKSUM_OFF:=0A+=09=09=09=09=09= barrier=20=3D=20EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_OFF);=0A= +=09=09=09=09=09WaitForProcSignalBarrier(barrier);=0A+=09=09=09=09=09= break;=0A+=09=09=09}=0A+=09=09}=0A=20=09}=0A=20=09else=20if=20(info=20=3D=3D= =20XLOG_LOGICAL_DECODING_STATUS_CHANGE)=0A=20=09{=0A@@=20-8758,6=20= +9374,51=20@@=20xlog_redo(XLogReaderState=20*record)=0A=20=09=09=09}=0A=20= =09=09}=0A=20=09}=0A+=09else=20if=20(info=20=3D=3D=20XLOG_CHECKSUMS)=0A+=09= {=0A+=09=09xl_checksum_state=20state;=0A+=09=09uint64=09=09barrier;=0A+=0A= +=09=09memcpy(&state,=20XLogRecGetData(record),=20= sizeof(xl_checksum_state));=0A+=0A+=09=09/*=0A+=09=09=20*=20XXX=20Could=20= this=20end=20up=20written=20to=20the=20control=20file=20prematurely?=20= IIRC=0A+=09=09=20*=20that=20happens=20during=20checkpoint,=20so=20what=20= if=20that=20gets=20triggered=20e.g.=0A+=09=09=20*=20because=20someone=20= runs=20CHECKPOINT?=20If=20we=20then=20crash=20(or=20something=0A+=09=09=20= *=20like=20that),=20could=20that=20confuse=20the=20instance?=0A+=09=09=20= */=0A+=09=09SpinLockAcquire(&XLogCtl->info_lck);=0A+=09=09= XLogCtl->data_checksum_version=20=3D=20state.new_checksumtype;=0A+=09=09= SpinLockRelease(&XLogCtl->info_lck);=0A+=0A+=09=09/*=0A+=09=09=20*=20= Block=20on=20a=20procsignalbarrier=20to=20await=20all=20processes=20= having=20seen=20the=0A+=09=09=20*=20change=20to=20checksum=20status.=20= Once=20the=20barrier=20has=20been=20passed=20we=20can=0A+=09=09=20*=20= initiate=20the=20corresponding=20processing.=0A+=09=09=20*/=0A+=09=09= switch=20(state.new_checksumtype)=0A+=09=09{=0A+=09=09=09case=20= PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION:=0A+=09=09=09=09barrier=20=3D=20= EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_ON);=0A+=09=09= =09=09WaitForProcSignalBarrier(barrier);=0A+=09=09=09=09break;=0A+=0A+=09= =09=09case=20PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION:=0A+=09=09=09=09= barrier=20=3D=20= EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_OFF);=0A+=09= =09=09=09WaitForProcSignalBarrier(barrier);=0A+=09=09=09=09break;=0A+=0A= +=09=09=09case=20PG_DATA_CHECKSUM_VERSION:=0A+=09=09=09=09barrier=20=3D=20= EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_ON);=0A+=09=09=09=09= WaitForProcSignalBarrier(barrier);=0A+=09=09=09=09break;=0A+=0A+=09=09=09= case=20PG_DATA_CHECKSUM_OFF:=0A+=09=09=09=09barrier=20=3D=20= EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_OFF);=0A+=09=09=09=09= WaitForProcSignalBarrier(barrier);=0A+=09=09=09=09break;=0A+=09=09}=0A+=09= }=0A=20}=0A=20=0A=20/*=0Adiff=20--git=20= a/src/backend/access/transam/xlogfuncs.c=20= b/src/backend/access/transam/xlogfuncs.c=0Aindex=20= 2efe4105efb..742ddcf124a=20100644=0A---=20= a/src/backend/access/transam/xlogfuncs.c=0A+++=20= b/src/backend/access/transam/xlogfuncs.c=0A@@=20-26,6=20+26,7=20@@=0A=20= #include=20"funcapi.h"=0A=20#include=20"miscadmin.h"=0A=20#include=20= "pgstat.h"=0A+#include=20"postmaster/datachecksumsworker.h"=0A=20= #include=20"replication/walreceiver.h"=0A=20#include=20"storage/fd.h"=0A=20= #include=20"storage/latch.h"=0A@@=20-748,3=20+749,50=20@@=20= pg_promote(PG_FUNCTION_ARGS)=0A=20=09=09=09=09=09=09=20=20=20= wait_seconds)));=0A=20=09PG_RETURN_BOOL(false);=0A=20}=0A+=0A+/*=0A+=20*=20= Disables=20data=20checksums=20for=20the=20cluster,=20if=20applicable.=20= Starts=20a=20background=0A+=20*=20worker=20which=20turns=20off=20the=20= data=20checksums.=0A+=20*/=0A+Datum=0A= +disable_data_checksums(PG_FUNCTION_ARGS)=0A+{=0A+=09if=20(!superuser())=0A= +=09=09ereport(ERROR,=0A+=09=09=09=09= errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),=0A+=09=09=09=09errmsg("must=20= be=20superuser=20to=20change=20data=20checksum=20state"));=0A+=0A+=09= StartDataChecksumsWorkerLauncher(DISABLE_DATACHECKSUMS,=200,=200);=0A+=09= PG_RETURN_VOID();=0A+}=0A+=0A+/*=0A+=20*=20Enables=20data=20checksums=20= for=20the=20cluster,=20if=20applicable.=20=20Supports=20vacuum-=0A+=20*=20= like=20cost=20based=20throttling=20to=20limit=20system=20load.=20Starts=20= a=20background=20worker=0A+=20*=20which=20updates=20data=20checksums=20= on=20existing=20data.=0A+=20*/=0A+Datum=0A= +enable_data_checksums(PG_FUNCTION_ARGS)=0A+{=0A+=09int=09=09=09= cost_delay=20=3D=20PG_GETARG_INT32(0);=0A+=09int=09=09=09cost_limit=20=3D=20= PG_GETARG_INT32(1);=0A+=0A+=09if=20(!superuser())=0A+=09=09= ereport(ERROR,=0A+=09=09=09=09errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),=0A= +=09=09=09=09errmsg("must=20be=20superuser=20to=20change=20data=20= checksum=20state"));=0A+=0A+=09if=20(cost_delay=20<=200)=0A+=09=09= ereport(ERROR,=0A+=09=09=09=09errcode(ERRCODE_INVALID_PARAMETER_VALUE),=0A= +=09=09=09=09errmsg("cost=20delay=20cannot=20be=20a=20negative=20= value"));=0A+=0A+=09if=20(cost_limit=20<=3D=200)=0A+=09=09ereport(ERROR,=0A= +=09=09=09=09errcode(ERRCODE_INVALID_PARAMETER_VALUE),=0A+=09=09=09=09= errmsg("cost=20limit=20must=20be=20greater=20than=20zero"));=0A+=0A+=09= StartDataChecksumsWorkerLauncher(ENABLE_DATACHECKSUMS,=20cost_delay,=20= cost_limit);=0A+=0A+=09PG_RETURN_VOID();=0A+}=0Adiff=20--git=20= a/src/backend/access/transam/xlogrecovery.c=20= b/src/backend/access/transam/xlogrecovery.c=0Aindex=20= 4fc37a031d9..fe0663b641a=20100644=0A---=20= a/src/backend/access/transam/xlogrecovery.c=0A+++=20= b/src/backend/access/transam/xlogrecovery.c=0A@@=20-782,6=20+782,10=20@@=20= InitWalRecovery(ControlFileData=20*ControlFile,=20bool=20= *wasShutdown_ptr,=0A=20=09=09CheckPointTLI=20=3D=20= ControlFile->checkPointCopy.ThisTimeLineID;=0A=20=09=09RedoStartLSN=20=3D=20= ControlFile->checkPointCopy.redo;=0A=20=09=09RedoStartTLI=20=3D=20= ControlFile->checkPointCopy.ThisTimeLineID;=0A+=0A+=09=09elog(LOG,=20= "InitWalRecovery=20checkpoint=20%X/%08X=20redo=20%X/%08X",=0A+=09=09=09=20= LSN_FORMAT_ARGS(CheckPointLoc),=20LSN_FORMAT_ARGS(RedoStartLSN));=0A+=0A=20= =09=09record=20=3D=20ReadCheckpointRecord(xlogprefetcher,=20= CheckPointLoc,=0A=20=09=09=09=09=09=09=09=09=09=20=20CheckPointTLI);=0A=20= =09=09if=20(record=20!=3D=20NULL)=0A@@=20-1675,6=20+1679,9=20@@=20= PerformWalRecovery(void)=0A=20=09bool=09=09reachedRecoveryTarget=20=3D=20= false;=0A=20=09TimeLineID=09replayTLI;=0A=20=0A+=09elog(LOG,=20= "PerformWalRecovery=20checkpoint=20%X/%08X=20redo=20%X/%08X",=0A+=09=09=20= LSN_FORMAT_ARGS(CheckPointLoc),=20LSN_FORMAT_ARGS(RedoStartLSN));=0A+=0A=20= =09/*=0A=20=09=20*=20Initialize=20shared=20variables=20for=20tracking=20= progress=20of=20WAL=20replay,=20as=20if=0A=20=09=20*=20we=20had=20just=20= replayed=20the=20record=20before=20the=20REDO=20location=20(or=20the=0A= @@=20-1683,12=20+1690,14=20@@=20PerformWalRecovery(void)=0A=20=09= SpinLockAcquire(&XLogRecoveryCtl->info_lck);=0A=20=09if=20(RedoStartLSN=20= <=20CheckPointLoc)=0A=20=09{=0A+=09=09elog(LOG,=20"(RedoStartLSN=20<=20= CheckPointLoc)");=0A=20=09=09XLogRecoveryCtl->lastReplayedReadRecPtr=20=3D= =20InvalidXLogRecPtr;=0A=20=09=09XLogRecoveryCtl->lastReplayedEndRecPtr=20= =3D=20RedoStartLSN;=0A=20=09=09XLogRecoveryCtl->lastReplayedTLI=20=3D=20= RedoStartTLI;=0A=20=09}=0A=20=09else=0A=20=09{=0A+=09=09elog(LOG,=20= "(RedoStartLSN=20>=3D=20CheckPointLoc)");=0A=20=09=09= XLogRecoveryCtl->lastReplayedReadRecPtr=20=3D=20xlogreader->ReadRecPtr;=0A= =20=09=09XLogRecoveryCtl->lastReplayedEndRecPtr=20=3D=20= xlogreader->EndRecPtr;=0A=20=09=09XLogRecoveryCtl->lastReplayedTLI=20=3D=20= CheckPointTLI;=0A@@=20-1700,6=20+1709,10=20@@=20PerformWalRecovery(void)=0A= =20=09XLogRecoveryCtl->recoveryPauseState=20=3D=20RECOVERY_NOT_PAUSED;=0A= =20=09SpinLockRelease(&XLogRecoveryCtl->info_lck);=0A=20=0A+=09elog(LOG,=20= "PerformWalRecovery=20lastReplayedReadRecPtr=20%X/%08X=20= lastReplayedEndRecPtr=20%X/%08X",=0A+=09=09=20= LSN_FORMAT_ARGS(XLogRecoveryCtl->lastReplayedReadRecPtr),=0A+=09=09=20= LSN_FORMAT_ARGS(XLogRecoveryCtl->lastReplayedEndRecPtr));=0A+=0A=20=09/*=20= Also=20ensure=20XLogReceiptTime=20has=20a=20sane=20value=20*/=0A=20=09= XLogReceiptTime=20=3D=20GetCurrentTimestamp();=0A=20=0Adiff=20--git=20= a/src/backend/backup/basebackup.c=20b/src/backend/backup/basebackup.c=0A= index=20463c0756b5e..2eb0a3af453=20100644=0A---=20= a/src/backend/backup/basebackup.c=0A+++=20= b/src/backend/backup/basebackup.c=0A@@=20-1613,8=20+1613,7=20@@=20= sendFile(bbsink=20*sink,=20const=20char=20*readfilename,=20const=20char=20= *tarfilename,=0A=20=09=20*=20enabled=20for=20this=20cluster,=20and=20if=20= this=20is=20a=20relation=20file,=20then=20verify=0A=20=09=20*=20the=20= checksum.=0A=20=09=20*/=0A-=09if=20(!noverify_checksums=20&&=20= DataChecksumsEnabled()=20&&=0A-=09=09= RelFileNumberIsValid(relfilenumber))=0A+=09if=20(!noverify_checksums=20= &&=20RelFileNumberIsValid(relfilenumber))=0A=20=09=09verify_checksum=20=3D= =20true;=0A=20=0A=20=09/*=0A@@=20-1747,7=20+1746,7=20@@=20= sendFile(bbsink=20*sink,=20const=20char=20*readfilename,=20const=20char=20= *tarfilename,=0A=20=09=09=20*=20If=20the=20amount=20of=20data=20we=20= were=20able=20to=20read=20was=20not=20a=20multiple=20of=0A=20=09=09=20*=20= BLCKSZ,=20we=20cannot=20verify=20checksums,=20which=20are=20block-level.=0A= =20=09=09=20*/=0A-=09=09if=20(verify_checksum=20&&=20(cnt=20%=20BLCKSZ=20= !=3D=200))=0A+=09=09if=20(verify_checksum=20&&=20= DataChecksumsNeedVerify()=20&&=20(cnt=20%=20BLCKSZ=20!=3D=200))=0A=20=09=09= {=0A=20=09=09=09ereport(WARNING,=0A=20=09=09=09=09=09(errmsg("could=20= not=20verify=20checksum=20in=20file=20\"%s\",=20block=20"=0A@@=20-1842,9=20= +1841,10=20@@=20sendFile(bbsink=20*sink,=20const=20char=20*readfilename,=20= const=20char=20*tarfilename,=0A=20=20*=20'blkno'=20is=20the=20block=20= number=20of=20the=20first=20page=20in=20the=20bbsink's=20buffer=0A=20=20= *=20relative=20to=20the=20start=20of=20the=20relation.=0A=20=20*=0A-=20*=20= 'verify_checksum'=20indicates=20whether=20we=20should=20try=20to=20= verify=20checksums=0A-=20*=20for=20the=20blocks=20we=20read.=20If=20we=20= do=20this,=20we'll=20update=20*checksum_failures=0A-=20*=20and=20issue=20= warnings=20as=20appropriate.=0A+=20*=20'verify_checksum'=20determins=20= if=20the=20user=20has=20asked=20to=20verify=20checksums,=20but=0A+=20*=20= since=20data=20checksums=20can=20be=20disabled,=20or=20become=20= disabled,=20we=20need=20to=20check=0A+=20*=20state=20before=20verifying=20= individual=20pages.=20=20If=20we=20do=20this,=20we'll=20update=0A+=20*=20= *checksum_failures=20and=20issue=20warnings=20as=20appropriate.=0A=20=20= */=0A=20static=20off_t=0A=20read_file_data_into_buffer(bbsink=20*sink,=20= const=20char=20*readfilename,=20int=20fd,=0A@@=20-1870,6=20+1870,13=20@@=20= read_file_data_into_buffer(bbsink=20*sink,=20const=20char=20= *readfilename,=20int=20fd,=0A=20=09=09int=09=09=09reread_cnt;=0A=20=09=09= uint16=09=09expected_checksum;=0A=20=0A+=09=09/*=0A+=09=09=20*=20The=20= data=20checksum=20state=20can=20change=20at=20any=20point,=20so=20we=20= need=20to=0A+=09=09=20*=20re-check=20before=20each=20page.=0A+=09=09=20= */=0A+=09=09if=20(!DataChecksumsNeedVerify())=0A+=09=09=09return=20cnt;=0A= +=0A=20=09=09page=20=3D=20sink->bbs_buffer=20+=20BLCKSZ=20*=20i;=0A=20=0A= =20=09=09/*=20If=20the=20page=20is=20OK,=20go=20on=20to=20the=20next=20= one.=20*/=0A@@=20-1892,7=20+1899,12=20@@=20= read_file_data_into_buffer(bbsink=20*sink,=20const=20char=20= *readfilename,=20int=20fd,=0A=20=09=09=20*=20allows=20us=20to=20wait=20= until=20we=20can=20be=20certain=20that=20no=20write=20to=20the=0A=20=09=09= =20*=20block=20is=20in=20progress.=20Since=20we=20don't=20have=20any=20= such=20thing=20right=20now,=0A=20=09=09=20*=20we=20just=20do=20this=20= and=20hope=20for=20the=20best.=0A+=09=09=20*=0A+=09=09=20*=20The=20data=20= checksum=20state=20may=20also=20have=20changed=20concurrently=20so=20= check=0A+=09=09=20*=20again.=0A=20=09=09=20*/=0A+=09=09if=20= (!DataChecksumsNeedVerify())=0A+=09=09=09return=20cnt;=0A=20=09=09= reread_cnt=20=3D=0A=20=09=09=09basebackup_read_file(fd,=20= sink->bbs_buffer=20+=20BLCKSZ=20*=20i,=0A=20=09=09=09=09=09=09=09=09=20= BLCKSZ,=20offset=20+=20BLCKSZ=20*=20i,=0A@@=20-2007,6=20+2019,9=20@@=20= verify_page_checksum(Page=20page,=20XLogRecPtr=20start_lsn,=20= BlockNumber=20blkno,=0A=20=09if=20(PageIsNew(page)=20||=20= PageGetLSN(page)=20>=3D=20start_lsn)=0A=20=09=09return=20true;=0A=20=0A+=09= if=20(!DataChecksumsNeedVerify())=0A+=09=09return=20true;=0A+=0A=20=09/*=20= Perform=20the=20actual=20checksum=20calculation.=20*/=0A=20=09checksum=20= =3D=20pg_checksum_page(page,=20blkno);=0A=20=0Adiff=20--git=20= a/src/backend/catalog/system_functions.sql=20= b/src/backend/catalog/system_functions.sql=0Aindex=20= eb9e31ae1bf..b4c58652af1=20100644=0A---=20= a/src/backend/catalog/system_functions.sql=0A+++=20= b/src/backend/catalog/system_functions.sql=0A@@=20-657,6=20+657,14=20@@=20= LANGUAGE=20INTERNAL=0A=20STRICT=20VOLATILE=20PARALLEL=20UNSAFE=0A=20AS=20= 'pg_replication_origin_session_setup';=0A=20=0A+CREATE=20OR=20REPLACE=20= FUNCTION=0A+=20=20pg_enable_data_checksums(cost_delay=20integer=20= DEFAULT=200,=0A+=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20cost_limit=20integer=20DEFAULT=20100)=0A+RETURNS=20= void=0A+STRICT=20VOLATILE=20LANGUAGE=20internal=0A+PARALLEL=20RESTRICTED=0A= +AS=20'enable_data_checksums';=0A+=0A=20--=0A=20--=20The=20default=20= permissions=20for=20functions=20mean=20that=20anyone=20can=20execute=20= them.=0A=20--=20A=20number=20of=20functions=20shouldn't=20be=20= executable=20by=20just=20anyone,=20but=20rather=0A@@=20-782,6=20+790,10=20= @@=20REVOKE=20EXECUTE=20ON=20FUNCTION=20pg_ls_logicalmapdir()=20FROM=20= PUBLIC;=0A=20=0A=20REVOKE=20EXECUTE=20ON=20FUNCTION=20= pg_ls_replslotdir(text)=20FROM=20PUBLIC;=0A=20=0A+REVOKE=20EXECUTE=20ON=20= FUNCTION=20pg_enable_data_checksums(integer,=20integer)=20FROM=20public;=0A= +=0A+REVOKE=20EXECUTE=20ON=20FUNCTION=20pg_disable_data_checksums()=20= FROM=20public;=0A+=0A=20--=0A=20--=20We=20also=20set=20up=20some=20= things=20as=20accessible=20to=20standard=20roles.=0A=20--=0Adiff=20--git=20= a/src/backend/catalog/system_views.sql=20= b/src/backend/catalog/system_views.sql=0Aindex=20= 7553f31fef0..47c661fe678=20100644=0A---=20= a/src/backend/catalog/system_views.sql=0A+++=20= b/src/backend/catalog/system_views.sql=0A@@=20-1385,6=20+1385,25=20@@=20= CREATE=20VIEW=20pg_stat_progress_copy=20AS=0A=20=20=20=20=20FROM=20= pg_stat_get_progress_info('COPY')=20AS=20S=0A=20=20=20=20=20=20=20=20=20= LEFT=20JOIN=20pg_database=20D=20ON=20S.datid=20=3D=20D.oid;=0A=20=0A= +CREATE=20VIEW=20pg_stat_progress_data_checksums=20AS=0A+=20=20=20=20= SELECT=0A+=20=20=20=20=20=20=20=20S.pid=20AS=20pid,=20S.datid,=20= D.datname=20AS=20datname,=0A+=20=20=20=20=20=20=20=20CASE=20S.param1=20= WHEN=200=20THEN=20'enabling'=0A+=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20WHEN=201=20THEN=20'disabling'=0A+=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20WHEN=202=20THEN=20= 'waiting=20on=20temporary=20tables'=0A+=09=09=09=09=09=20=20WHEN=203=20= THEN=20'waiting=20on=20barrier'=0A+=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20WHEN=204=20THEN=20'done'=0A+=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20END=20AS=20phase,=0A+=20=20= =20=20=20=20=20=20CASE=20S.param2=20WHEN=20-1=20THEN=20NULL=20ELSE=20= S.param2=20END=20AS=20databases_total,=0A+=20=20=20=20=20=20=20=20= S.param3=20AS=20databases_done,=0A+=20=20=20=20=20=20=20=20CASE=20= S.param4=20WHEN=20-1=20THEN=20NULL=20ELSE=20S.param4=20END=20AS=20= relations_total,=0A+=20=20=20=20=20=20=20=20CASE=20S.param5=20WHEN=20-1=20= THEN=20NULL=20ELSE=20S.param5=20END=20AS=20relations_done,=0A+=20=20=20=20= =20=20=20=20CASE=20S.param6=20WHEN=20-1=20THEN=20NULL=20ELSE=20S.param6=20= END=20AS=20blocks_total,=0A+=20=20=20=20=20=20=20=20CASE=20S.param7=20= WHEN=20-1=20THEN=20NULL=20ELSE=20S.param7=20END=20AS=20blocks_done=0A+=20= =20=20=20FROM=20pg_stat_get_progress_info('DATACHECKSUMS')=20AS=20S=0A+=20= =20=20=20=20=20=20=20LEFT=20JOIN=20pg_database=20D=20ON=20S.datid=20=3D=20= D.oid=0A+=20=20=20=20ORDER=20BY=20S.datid;=20--=20return=20the=20= launcher=20process=20first=0A+=0A=20CREATE=20VIEW=20pg_user_mappings=20= AS=0A=20=20=20=20=20SELECT=0A=20=20=20=20=20=20=20=20=20U.oid=20=20=20=20= =20=20=20AS=20umid,=0Adiff=20--git=20a/src/backend/postmaster/Makefile=20= b/src/backend/postmaster/Makefile=0Aindex=200f4435d2d97..0c36765acfe=20= 100644=0A---=20a/src/backend/postmaster/Makefile=0A+++=20= b/src/backend/postmaster/Makefile=0A@@=20-18,6=20+18,7=20@@=20OBJS=20=3D=20= \=0A=20=09bgworker.o=20\=0A=20=09bgwriter.o=20\=0A=20=09checkpointer.o=20= \=0A+=09datachecksumsworker.o=20\=0A=20=09fork_process.o=20\=0A=20=09= interrupt.o=20\=0A=20=09launch_backend.o=20\=0Adiff=20--git=20= a/src/backend/postmaster/auxprocess.c=20= b/src/backend/postmaster/auxprocess.c=0Aindex=208ea800c0bbd..9dc71dadd8a=20= 100644=0A---=20a/src/backend/postmaster/auxprocess.c=0A+++=20= b/src/backend/postmaster/auxprocess.c=0A@@=20-15,6=20+15,7=20@@=0A=20= #include=20=0A=20#include=20=0A=20=0A+#include=20= "access/xlog.h"=0A=20#include=20"miscadmin.h"=0A=20#include=20"pgstat.h"=0A= =20#include=20"postmaster/auxprocess.h"=0A@@=20-68,6=20+69,24=20@@=20= AuxiliaryProcessMainCommon(void)=0A=20=0A=20=09ProcSignalInit(NULL,=20= 0);=0A=20=0A+=09/*=0A+=09=20*=20Initialize=20a=20local=20cache=20of=20= the=20data_checksum_version,=20to=20be=20updated=20by=0A+=09=20*=20the=20= procsignal-based=20barriers.=0A+=09=20*=0A+=09=20*=20This=20= intentionally=20happens=20after=20initializing=20the=20procsignal,=20= otherwise=0A+=09=20*=20we=20might=20miss=20a=20state=20change.=20This=20= means=20we=20can=20get=20a=20barrier=20for=20the=0A+=09=20*=20state=20= we've=20just=20initialized=20-=20but=20it=20can=20happen=20only=20once.=0A= +=09=20*=0A+=09=20*=20The=20postmaster=20(which=20is=20what=20gets=20= forked=20into=20the=20new=20child=20process)=0A+=09=20*=20does=20not=20= handle=20barriers,=20therefore=20it=20may=20not=20have=20the=20current=20= value=0A+=09=20*=20of=20LocalDataChecksumVersion=20value=20(it'll=20have=20= the=20value=20read=20from=20the=0A+=09=20*=20control=20file,=20which=20= may=20be=20arbitrarily=20old).=0A+=09=20*=0A+=09=20*=20NB:=20Even=20if=20= the=20postmaster=20handled=20barriers,=20the=20value=20might=20still=20= be=0A+=09=20*=20stale,=20as=20it=20might=20have=20changed=20after=20this=20= process=20forked.=0A+=09=20*/=0A+=09InitLocalDataChecksumVersion();=0A+=0A= =20=09/*=0A=20=09=20*=20Auxiliary=20processes=20don't=20run=20= transactions,=20but=20they=20may=20need=20a=0A=20=09=20*=20resource=20= owner=20anyway=20to=20manage=20buffer=20pins=20acquired=20outside=0Adiff=20= --git=20a/src/backend/postmaster/bgworker.c=20= b/src/backend/postmaster/bgworker.c=0Aindex=2051874481751..25ecb5d691a=20= 100644=0A---=20a/src/backend/postmaster/bgworker.c=0A+++=20= b/src/backend/postmaster/bgworker.c=0A@@=20-18,6=20+18,7=20@@=0A=20= #include=20"pgstat.h"=0A=20#include=20"port/atomics.h"=0A=20#include=20= "postmaster/bgworker_internals.h"=0A+#include=20= "postmaster/datachecksumsworker.h"=0A=20#include=20= "postmaster/postmaster.h"=0A=20#include=20= "replication/logicallauncher.h"=0A=20#include=20= "replication/logicalworker.h"=0A@@=20-142,6=20+143,14=20@@=20static=20= const=20struct=0A=20=09{=0A=20=09=09.fn_name=20=3D=20= "SequenceSyncWorkerMain",=0A=20=09=09.fn_addr=20=3D=20= SequenceSyncWorkerMain=0A+=09},=0A+=09{=0A+=09=09.fn_name=20=3D=20= "DataChecksumsWorkerLauncherMain",=0A+=09=09.fn_addr=20=3D=20= DataChecksumsWorkerLauncherMain=0A+=09},=0A+=09{=0A+=09=09.fn_name=20=3D=20= "DataChecksumsWorkerMain",=0A+=09=09.fn_addr=20=3D=20= DataChecksumsWorkerMain=0A=20=09}=0A=20};=0A=20=0Adiff=20--git=20= a/src/backend/postmaster/datachecksumsworker.c=20= b/src/backend/postmaster/datachecksumsworker.c=0Anew=20file=20mode=20= 100644=0Aindex=2000000000000..24e6af63f6a=0A---=20/dev/null=0A+++=20= b/src/backend/postmaster/datachecksumsworker.c=0A@@=20-0,0=20+1,1497=20= @@=0A= +/*-----------------------------------------------------------------------= --=0A+=20*=0A+=20*=20datachecksumsworker.c=0A+=20*=09=20=20Background=20= worker=20for=20enabling=20or=20disabling=20data=20checksums=20online=0A+=20= *=0A+=20*=20When=20enabling=20data=20checksums=20on=20a=20database=20at=20= initdb=20time=20or=20when=20shut=20down=0A+=20*=20with=20pg_checksums,=20= no=20extra=20process=20is=20required=20as=20each=20page=20is=20= checksummed,=0A+=20*=20and=20verified,=20when=20accessed.=20=20When=20= enabling=20checksums=20on=20an=20already=20running=0A+=20*=20cluster,=20= this=20worker=20will=20ensure=20that=20all=20pages=20are=20checksummed=20= before=0A+=20*=20verification=20of=20the=20checksums=20is=20turned=20on.=20= In=20the=20case=20of=20disabling=0A+=20*=20checksums,=20the=20state=20= transition=20is=20performed=20only=20in=20the=20control=20file,=20no=0A+=20= *=20changes=20are=20performed=20on=20the=20data=20pages.=0A+=20*=0A+=20*=20= Checksums=20can=20be=20either=20enabled=20or=20disabled=20cluster-wide,=20= with=20on/off=20being=0A+=20*=20the=20end=20state=20for=20= data_checksums.=0A+=20*=0A+=20*=201.=20Enabling=20checksums=0A+=20*=20= ---------------------=0A+=20*=20When=20enabling=20checksums=20in=20an=20= online=20cluster,=20data_checksums=20will=20be=20set=20to=0A+=20*=20= "inprogress-on"=20which=20signals=20that=20write=20operations=20MUST=20= compute=20and=20write=0A+=20*=20the=20checksum=20on=20the=20data=20page,=20= but=20during=20reading=20the=20checksum=20SHALL=20NOT=20be=0A+=20*=20= verified.=20This=20ensures=20that=20all=20objects=20created=20during=20= when=20checksums=20are=0A+=20*=20being=20enabled=20will=20have=20= checksums=20set,=20but=20reads=20wont=20fail=20due=20to=20missing=20or=0A= +=20*=20invalid=20checksums.=20Invalid=20checksums=20can=20be=20present=20= in=20case=20the=20cluster=20had=0A+=20*=20checksums=20enabled,=20then=20= disabled=20them=20and=20updated=20the=20page=20while=20they=20were=0A+=20= *=20disabled.=0A+=20*=0A+=20*=20The=20DataChecksumsWorker=20will=20= compile=20a=20list=20of=20databases=20which=20exist=20at=20the=0A+=20*=20= start=20of=20checksumming,=20and=20once=20all=20are=20processed=20will=20= regenerate=20the=20list=0A+=20*=20and=20start=20over=20processing=20any=20= new=20entries.=20Once=20there=20are=20no=20new=20entries=20on=0A+=20*=20= the=20list,=20processing=20will=20end.=20=20All=20databases=20MUST=20BE=20= successfully=20processed=0A+=20*=20in=20order=20for=20data=20checksums=20= to=20be=20enabled,=20the=20only=20exception=20are=20databases=0A+=20*=20= which=20are=20dropped=20before=20having=20been=20processed.=0A+=0A+=20*=20= Any=20new=20relation=20in=20a=20processed=20database,=20created=20during=20= processing,=20will=0A+=20*=20see=20the=20in-progress=20state=20and=20= will=20automatically=20be=20checksummed.=0A+=20*=0A+=20*=20For=20each=20= database,=20all=20relations=20which=20have=20storage=20are=20read=20and=20= every=20data=0A+=20*=20page=20is=20marked=20dirty=20to=20force=20a=20= write=20with=20the=20checksum.=20This=20will=20generate=0A+=20*=20a=20= lot=20of=20WAL=20as=20the=20entire=20database=20is=20read=20and=20= written.=0A+=20*=0A+=20*=20If=20the=20processing=20is=20interrupted=20by=20= a=20cluster=20crash=20or=20restart,=20it=20needs=20to=0A+=20*=20be=20= restarted=20from=20the=20beginning=20again=20as=20state=20isn't=20= persisted.=0A+=20*=0A+=20*=202.=20Disabling=20checksums=0A+=20*=20= ----------------------=0A+=20*=20When=20disabling=20checksums,=20= data_checksums=20will=20be=20set=20to=20"inprogress-off"=0A+=20*=20which=20= signals=20that=20checksums=20are=20written=20but=20no=20longer=20= verified.=20This=20ensure=0A+=20*=20that=20backends=20which=20have=20yet=20= to=20move=20from=20the=20"on"=20state=20will=20still=20be=20able=0A+=20*=20= to=20process=20data=20checksum=20validation.=0A+=20*=0A+=20*=203.=20= Synchronization=20and=20Correctness=0A+=20*=20= ----------------------------------=0A+=20*=20The=20processes=20involved=20= in=20enabling,=20or=20disabling,=20data=20checksums=20in=20an=0A+=20*=20= online=20cluster=20must=20be=20properly=20synchronized=20with=20the=20= normal=20backends=0A+=20*=20serving=20concurrent=20queries=20to=20ensure=20= correctness.=20Correctness=20is=20defined=0A+=20*=20as=20the=20= following:=0A+=20*=0A+=20*=20=20=20=20-=20Backends=20SHALL=20NOT=20= violate=20the=20data_checksums=20state=20they=20have=20agreed=20to=0A+=20= *=20=20=20=20=20=20by=20acknowledging=20the=20procsignalbarrier:=20=20= This=20means=20that=20all=20backends=0A+=20*=20=20=20=20=20=20MUST=20= calculate=20and=20write=20data=20checksums=20during=20all=20states=20= except=20off;=0A+=20*=20=20=20=20=20=20MUST=20validate=20checksums=20= only=20in=20the=20'on'=20state.=0A+=20*=20=20=20=20-=20Data=20checksums=20= SHALL=20NOT=20be=20considered=20enabled=20cluster-wide=20until=20all=0A+=20= *=20=20=20=20=20=20currently=20connected=20backends=20have=20state=20= "on":=20This=20means=20that=20all=0A+=20*=20=20=20=20=20=20backends=20= must=20wait=20on=20the=20procsignalbarrier=20to=20be=20acknowledged=20by=20= all=0A+=20*=20=20=20=20=20=20before=20proceeding=20to=20validate=20data=20= checksums.=0A+=20*=0A+=20*=20There=20are=20two=20steps=20of=20= synchronization=20required=20for=20changing=20data_checksums=0A+=20*=20= in=20an=20online=20cluster:=20(i)=20changing=20state=20in=20the=20active=20= backends=20("on",=0A+=20*=20"off",=20"inprogress-on"=20and=20= "inprogress-off"),=20and=20(ii)=20ensuring=20no=0A+=20*=20incompatible=20= objects=20and=20processes=20are=20left=20in=20a=20database=20when=20= workers=20end.=0A+=20*=20The=20former=20deals=20with=20cluster-wide=20= agreement=20on=20data=20checksum=20state=20and=20the=0A+=20*=20latter=20= with=20ensuring=20that=20any=20concurrent=20activity=20cannot=20break=20= the=20data=0A+=20*=20checksum=20contract=20during=20processing.=0A+=20*=0A= +=20*=20Synchronizing=20the=20state=20change=20is=20done=20with=20= procsignal=20barriers,=20before=0A+=20*=20updating=20the=20controlfile=20= with=20the=20state=20all=20other=20backends=20must=20absorb=20the=0A+=20= *=20barrier.=20=20Barrier=20absorption=20will=20happen=20during=20= interrupt=20processing,=20which=0A+=20*=20means=20that=20connected=20= backends=20will=20change=20state=20at=20different=20times.=20=20If=0A+=20= *=20waiting=20for=20a=20barrier=20is=20done=20during=20startup,=20for=20= example=20during=20replay,=20it=0A+=20*=20is=20important=20to=20realize=20= that=20any=20locks=20held=20by=20the=20startup=20process=20might=0A+=20*=20= cause=20deadblocks=20if=20backends=20end=20up=20waiting=20for=20those=20= locks=20while=20startup=0A+=20*=20is=20waiting=20for=20a=20= procsignalbarrier.=0A+=20*=0A+=20*=203.1=20When=20Enabling=20Data=20= Checksums=0A+=20*=20--------------------------------=0A+=20*=20A=20= process=20which=20fails=20to=20observe=20data=20checksums=20being=20= enabled=20can=20induce=20two=0A+=20*=20types=20of=20errors:=20failing=20= to=20write=20the=20checksum=20when=20modifying=20the=20page=20and=0A+=20= *=20failing=20to=20validate=20the=20data=20checksum=20on=20the=20page=20= when=20reading=20it.=0A+=20*=0A+=20*=20When=20processing=20starts=20all=20= backends=20belong=20to=20one=20of=20the=20below=20sets,=20with=0A+=20*=20= one=20set=20being=20empty:=0A+=20*=0A+=20*=20Bd:=20Backends=20in=20"off"=20= state=0A+=20*=20Bi:=20Backends=20in=20"inprogress-on"=20state=0A+=20*=0A= +=20*=20If=20processing=20is=20started=20in=20an=20online=20cluster=20= then=20all=20backends=20are=20in=20Bd.=0A+=20*=20If=20processing=20was=20= halted=20by=20the=20cluster=20shutting=20down=20(due=20to=20a=20crash=20= or=0A+=20*=20intentional=20restart),=20the=20controlfile=20state=20= "inprogress-on"=20will=20be=20observed=0A+=20*=20on=20system=20startup=20= and=20all=20backends=20will=20be=20placed=20in=20Bd.=20The=20controlfile=0A= +=20*=20state=20will=20also=20be=20set=20of=20"off".=0A+=20*=0A+=20*=20= Backends=20transition=20Bd=20->=20Bi=20via=20a=20procsignalbarrier=20= which=20is=20emitted=20by=20the=0A+=20*=20DataChecksumsLauncher.=20=20= When=20all=20backends=20have=20acknowledged=20the=20barrier=20then=0A+=20= *=20Bd=20will=20be=20empty=20and=20the=20next=20phase=20can=20begin:=20= calculating=20and=20writing=20data=0A+=20*=20checksums=20with=20= DataChecksumsWorkers.=20=20When=20the=20DataChecksumsWorker=20processes=0A= +=20*=20have=20finished=20writing=20checksums=20on=20all=20pages,=20data=20= checksums=20are=20enabled=0A+=20*=20cluster-wide=20via=20another=20= procsignalbarrier.=20There=20are=20four=20sets=20of=20backends=0A+=20*=20= where=20Bd=20shall=20be=20an=20empty=20set:=0A+=20*=0A+=20*=20Bg:=20= Backend=20updating=20the=20global=20state=20and=20emitting=20the=20= procsignalbarrier=0A+=20*=20Bd:=20Backends=20in=20"off"=20state=0A+=20*=20= Be:=20Backends=20in=20"on"=20state=0A+=20*=20Bi:=20Backends=20in=20= "inprogress-on"=20state=0A+=20*=0A+=20*=20Backends=20in=20Bi=20and=20Be=20= will=20write=20checksums=20when=20modifying=20a=20page,=20but=20only=0A+=20= *=20backends=20in=20Be=20will=20verify=20the=20checksum=20during=20= reading.=20The=20Bg=20backend=20is=0A+=20*=20blocked=20waiting=20for=20= all=20backends=20in=20Bi=20to=20process=20interrupts=20and=20move=20to=0A= +=20*=20Be.=20Any=20backend=20starting=20while=20Bg=20is=20waiting=20on=20= the=20procsignalbarrier=20will=0A+=20*=20observe=20the=20global=20state=20= being=20"on"=20and=20will=20thus=20automatically=20belong=20to=0A+=20*=20= Be.=20=20Checksums=20are=20enabled=20cluster-wide=20when=20Bi=20is=20an=20= empty=20set.=20Bi=20and=20Be=0A+=20*=20are=20compatible=20sets=20while=20= still=20operating=20based=20on=20their=20local=20state=20as=0A+=20*=20= both=20write=20data=20checksums.=0A+=20*=0A+=20*=203.2=20When=20= Disabling=20Data=20Checksums=0A+=20*=20---------------------------------=0A= +=20*=20A=20process=20which=20fails=20to=20observe=20that=20data=20= checksums=20have=20been=20disabled=0A+=20*=20can=20induce=20two=20types=20= of=20errors:=20writing=20the=20checksum=20when=20modifying=20the=0A+=20*=20= page=20and=20validating=20a=20data=20checksum=20which=20is=20no=20longer=20= correct=20due=20to=0A+=20*=20modifications=20to=20the=20page.=20The=20= former=20is=20not=20an=20error=20per=20se=20as=20data=0A+=20*=20= integrity=20is=20maintained,=20but=20it=20is=20wasteful.=20=20The=20= latter=20will=20cause=20errors=0A+=20*=20in=20user=20operations.=20=20= Assuming=20the=20following=20sets=20of=20backends:=0A+=20*=0A+=20*=20Bg:=20= Backend=20updating=20the=20global=20state=20and=20emitting=20the=20= procsignalbarrier=0A+=20*=20Bd:=20Backends=20in=20"off"=20state=0A+=20*=20= Be:=20Backends=20in=20"on"=20state=0A+=20*=20Bo:=20Backends=20in=20= "inprogress-off"=20state=0A+=20*=20Bi:=20Backends=20in=20"inprogress-on"=20= state=0A+=20*=0A+=20*=20Backends=20transition=20from=20the=20Be=20state=20= to=20Bd=20like=20so:=20Be=20->=20Bo=20->=20Bd.=20=20From=0A+=20*=20all=20= other=20states,=20the=20transition=20can=20be=20straight=20to=20Bd.=0A+=20= *=0A+=20*=20The=20goal=20is=20to=20transition=20all=20backends=20to=20Bd=20= making=20the=20others=20empty=20sets.=0A+=20*=20Backends=20in=20Bo=20= write=20data=20checksums,=20but=20don't=20validate=20them,=20such=20that=0A= +=20*=20backends=20still=20in=20Be=20can=20continue=20to=20validate=20= pages=20until=20the=20barrier=20has=0A+=20*=20been=20absorbed=20such=20= that=20they=20are=20in=20Bo.=20Once=20all=20backends=20are=20in=20Bo,=20= the=0A+=20*=20barrier=20to=20transition=20to=20"off"=20can=20be=20raised=20= and=20all=20backends=20can=20safely=0A+=20*=20stop=20writing=20data=20= checksums=20as=20no=20backend=20is=20enforcing=20data=20checksum=0A+=20*=20= validation=20any=20longer.=0A+=20*=0A+=20*=204.=20Future=20opportunities=20= for=20optimizations=0A+=20*=20-----------------------------------------=0A= +=20*=20Below=20are=20some=20potential=20optimizations=20and=20= improvements=20which=20were=20brought=0A+=20*=20up=20during=20reviews=20= of=20this=20feature,=20but=20which=20weren't=20implemented=20in=20the=0A= +=20*=20initial=20version.=20These=20are=20ideas=20listed=20without=20= any=20validation=20on=20their=0A+=20*=20feasibility=20or=20potential=20= payoff.=20More=20discussion=20on=20(most=20of)=20these=20can=20be=0A+=20= *=20found=20on=20the=20-hackers=20threads=20linked=20to=20in=20the=20= commit=20message=20of=20this=0A+=20*=20feature.=0A+=20*=0A+=20*=20=20=20= *=20Launching=20datachecksumsworker=20for=20resuming=20operation=20from=20= the=20startup=0A+=20*=20=20=20=20=20process:=20Currently=20users=20have=20= to=20restart=20processing=20manually=20after=20a=0A+=20*=20=20=20=20=20= restart=20since=20dynamic=20background=20worker=20cannot=20be=20started=20= from=20the=0A+=20*=20=20=20=20=20postmaster.=20Changing=20the=20startup=20= process=20could=20make=20restarting=20the=0A+=20*=20=20=20=20=20= processing=20automatic=20on=20cluster=20restart.=0A+=20*=20=20=20*=20= Avoid=20dirtying=20the=20page=20when=20checksums=20already=20match:=20= Iff=20the=20checksum=0A+=20*=20=20=20=20=20on=20the=20page=20happens=20= to=20already=20match=20we=20still=20dirty=20the=20page.=20It=20should=0A= +=20*=20=20=20=20=20be=20enough=20to=20only=20do=20the=20= log_newpage_buffer()=20call=20in=20that=20case.=0A+=20*=20=20=20*=20= Teach=20pg_checksums=20to=20avoid=20checksummed=20pages=20when=20= pg_checksums=20is=20used=0A+=20*=20=20=20=20=20to=20enable=20checksums=20= on=20a=20cluster=20which=20is=20in=20inprogress-on=20state=20and=0A+=20*=20= =20=20=20=20may=20have=20checksummed=20pages=20(make=20pg_checksums=20be=20= able=20to=20resume=20an=0A+=20*=20=20=20=20=20online=20operation).=20= This=20should=20only=20be=20attempted=20for=20wal_level=20minimal.=0A+=20= *=20=20=20*=20Restartability=20(not=20necessarily=20with=20page=20= granularity).=0A+=20*=20=20=20*=20Avoid=20processing=20databases=20which=20= were=20created=20during=20inprogress-on.=0A+=20*=20=20=20=20=20Right=20= now=20all=20databases=20are=20processed=20regardless=20to=20be=20safe.=0A= +=20*=0A+=20*=0A+=20*=20Portions=20Copyright=20(c)=201996-2026,=20= PostgreSQL=20Global=20Development=20Group=0A+=20*=20Portions=20Copyright=20= (c)=201994,=20Regents=20of=20the=20University=20of=20California=0A+=20*=0A= +=20*=0A+=20*=20IDENTIFICATION=0A+=20*=09=20=20= src/backend/postmaster/datachecksumsworker.c=0A+=20*=0A+=20= *-------------------------------------------------------------------------= =0A+=20*/=0A+#include=20"postgres.h"=0A+=0A+#include=20"access/genam.h"=0A= +#include=20"access/heapam.h"=0A+#include=20"access/htup_details.h"=0A= +#include=20"access/xact.h"=0A+#include=20"access/xloginsert.h"=0A= +#include=20"catalog/indexing.h"=0A+#include=20"catalog/pg_class.h"=0A= +#include=20"catalog/pg_database.h"=0A+#include=20"commands/progress.h"=0A= +#include=20"commands/vacuum.h"=0A+#include=20"common/relpath.h"=0A= +#include=20"miscadmin.h"=0A+#include=20"pgstat.h"=0A+#include=20= "postmaster/bgworker.h"=0A+#include=20"postmaster/bgwriter.h"=0A= +#include=20"postmaster/datachecksumsworker.h"=0A+#include=20= "storage/bufmgr.h"=0A+#include=20"storage/checksum.h"=0A+#include=20= "storage/ipc.h"=0A+#include=20"storage/lmgr.h"=0A+#include=20= "storage/lwlock.h"=0A+#include=20"storage/procarray.h"=0A+#include=20= "storage/smgr.h"=0A+#include=20"tcop/tcopprot.h"=0A+#include=20= "utils/fmgroids.h"=0A+#include=20"utils/injection_point.h"=0A+#include=20= "utils/lsyscache.h"=0A+#include=20"utils/ps_status.h"=0A+#include=20= "utils/syscache.h"=0A+=0A+/*=0A+=20*=20Number=20of=20times=20we=20retry=20= to=20open=20a=20database=20before=20giving=20up=20and=20consider=0A+=20*=20= it=20to=20have=20failed=20processing.=0A+=20*/=0A+#define=20= DATACHECKSUMSWORKER_MAX_DB_RETRIES=205=0A+=0A+/*=0A+=20*=20Signaling=20= between=20backends=20calling=20pg_enable/disable_data_checksums,=20the=0A= +=20*=20checksums=20launcher=20process,=20and=20the=20checksums=20worker=20= process.=0A+=20*=0A+=20*=20This=20struct=20is=20protected=20by=20= DataChecksumsWorkerLock=0A+=20*/=0A+typedef=20struct=20= DataChecksumsWorkerShmemStruct=0A+{=0A+=09/*=0A+=09=20*=20These=20are=20= set=20by=20pg_{enable|disable|verify}_data_checksums,=20to=20tell=20the=0A= +=09=20*=20launcher=20what=20the=20target=20state=20is.=0A+=09=20*/=0A+=09= DataChecksumsWorkerOperation=20launch_operation;=0A+=09int=09=09=09= launch_cost_delay;=0A+=09int=09=09=09launch_cost_limit;=0A+=0A+=09/*=0A+=09= =20*=20Is=20a=20launcher=20process=20is=20currently=20running?=20=20This=20= is=20set=20by=20the=20main=0A+=09=20*=20launcher=20process,=20after=20it=20= has=20read=20the=20above=20launch_*=20parameters.=0A+=09=20*/=0A+=09bool=09= =09launcher_running;=0A+=0A+=09/*=0A+=09=20*=20If=20a=20worker=20process=20= currently=20running?=20=20This=20is=20set=20by=20the=20worker=0A+=09=20*=20= launcher=20when=20it=20starts=20waiting=20for=20a=20worker=20process=20= to=20finish.=0A+=09=20*/=0A+=09bool=09=09worker_running;=0A+=0A+=09/*=0A= +=09=20*=20These=20fields=20indicate=20the=20target=20state=20that=20the=20= launcher=20is=20currently=0A+=09=20*=20working=20towards.=20They=20can=20= be=20different=20from=20the=20corresponding=20launch_*=0A+=09=20*=20= fields,=20if=20a=20new=20pg_enable/disable_data_checksums()=20call=20was=20= made=20while=0A+=09=20*=20the=20launcher/worker=20was=20already=20= running.=0A+=09=20*=0A+=09=20*=20The=20below=20members=20are=20set=20= when=20the=20launcher=20starts,=20and=20are=20only=0A+=09=20*=20accessed=20= read-only=20by=20the=20single=20worker.=20Thus,=20we=20can=20access=20= these=0A+=09=20*=20without=20a=20lock.=20If=20multiple=20workers,=20or=20= dynamic=20cost=20parameters,=20are=0A+=09=20*=20supported=20at=20some=20= point=20then=20this=20would=20need=20to=20be=20revisited.=0A+=09=20*/=0A= +=09DataChecksumsWorkerOperation=20operation;=0A+=09int=09=09=09= cost_delay;=0A+=09int=09=09=09cost_limit;=0A+=0A+=09/*=0A+=09=20*=20= Signaling=20between=20the=20launcher=20and=20the=20worker=20process.=0A+=09= =20*=0A+=09=20*=20As=20there=20is=20only=20a=20single=20worker,=20and=20= the=20launcher=20won't=20read=20these=0A+=09=20*=20until=20the=20worker=20= exits,=20they=20can=20be=20accessed=20without=20the=20need=20for=20a=0A+=09= =20*=20lock.=20If=20multiple=20workers=20are=20supported=20then=20this=20= will=20have=20to=20be=0A+=09=20*=20revisited.=0A+=09=20*/=0A+=0A+=09/*=20= result,=20set=20by=20worker=20before=20exiting=20*/=0A+=09= DataChecksumsWorkerResult=20success;=0A+=0A+=09/*=0A+=09=20*=20tells=20= the=20worker=20process=20whether=20it=20should=20also=20process=20the=20= shared=0A+=09=20*=20catalogs=0A+=09=20*/=0A+=09bool=09=09= process_shared_catalogs;=0A+}=20DataChecksumsWorkerShmemStruct;=0A+=0A= +/*=20Shared=20memory=20segment=20for=20datachecksumsworker=20*/=0A= +static=20DataChecksumsWorkerShmemStruct=20*DataChecksumsWorkerShmem;=0A= +=0A+typedef=20struct=20DataChecksumsWorkerDatabase=0A+{=0A+=09Oid=09=09=09= dboid;=0A+=09char=09=20=20=20*dbname;=0A+}=20= DataChecksumsWorkerDatabase;=0A+=0A+typedef=20struct=20= DataChecksumsWorkerResultEntry=0A+{=0A+=09Oid=09=09=09dboid;=0A+=09= DataChecksumsWorkerResult=20result;=0A+=09int=09=09=09retries;=0A+}=20= DataChecksumsWorkerResultEntry;=0A+=0A+=0A+/*=0A+=20*=20Flag=20set=20by=20= the=20interrupt=20handler=0A+=20*/=0A+static=20volatile=20sig_atomic_t=20= abort_requested=20=3D=20false;=0A+=0A+/*=0A+=20*=20Have=20we=20set=20the=20= DataChecksumsWorkerShmemStruct->launcher_running=20flag?=0A+=20*=20If=20= we=20have,=20we=20need=20to=20clear=20it=20before=20exiting!=0A+=20*/=0A= +static=20volatile=20sig_atomic_t=20launcher_running=20=3D=20false;=0A+=0A= +/*=0A+=20*=20Are=20we=20enabling=20data=20checksums,=20or=20disabling=20= them?=0A+=20*/=0A+static=20DataChecksumsWorkerOperation=20operation;=0A+=0A= +/*=20Prototypes=20*/=0A+static=20List=20*BuildDatabaseList(void);=0A= +static=20List=20*BuildRelationList(bool=20temp_relations,=20bool=20= include_shared);=0A+static=20void=20FreeDatabaseList(List=20*dblist);=0A= +static=20DataChecksumsWorkerResult=20= ProcessDatabase(DataChecksumsWorkerDatabase=20*db);=0A+static=20bool=20= ProcessAllDatabases(void);=0A+static=20bool=20= ProcessSingleRelationFork(Relation=20reln,=20ForkNumber=20forkNum,=20= BufferAccessStrategy=20strategy);=0A+static=20void=20= launcher_cancel_handler(SIGNAL_ARGS);=0A+static=20void=20= WaitForAllTransactionsToFinish(void);=0A+=0A+/*=0A+=20*=20= StartDataChecksumsWorkerLauncher=0A+=20*=09=09Main=20entry=20point=20for=20= datachecksumsworker=20launcher=20process=0A+=20*=0A+=20*=20The=20main=20= entrypoint=20for=20starting=20data=20checksums=20processing=20for=20= enabling=20as=0A+=20*=20well=20as=20disabling.=0A+=20*/=0A+void=0A= +StartDataChecksumsWorkerLauncher(DataChecksumsWorkerOperation=20op,=0A+=09= =09=09=09=09=09=09=09=20int=20cost_delay,=0A+=09=09=09=09=09=09=09=09=20= int=20cost_limit)=0A+{=0A+=09BackgroundWorker=20bgw;=0A+=09= BackgroundWorkerHandle=20*bgw_handle;=0A+=09bool=09=09launcher_running;=0A= +=0A+#ifdef=20USE_ASSERT_CHECKING=0A+=09/*=20The=20cost=20delay=20= settings=20have=20no=20effect=20when=20disabling=20*/=0A+=09if=20(op=20= =3D=3D=20DISABLE_DATACHECKSUMS)=0A+=09=09Assert(cost_delay=20=3D=3D=200=20= &&=20cost_limit=20=3D=3D=200);=0A+#endif=0A+=0A+=09/*=20Store=20the=20= desired=20state=20in=20shared=20memory=20*/=0A+=09= LWLockAcquire(DataChecksumsWorkerLock,=20LW_EXCLUSIVE);=0A+=0A+=09= DataChecksumsWorkerShmem->launch_operation=20=3D=20op;=0A+=09= DataChecksumsWorkerShmem->launch_cost_delay=20=3D=20cost_delay;=0A+=09= DataChecksumsWorkerShmem->launch_cost_limit=20=3D=20cost_limit;=0A+=0A+=09= INJECTION_POINT("datachecksumsworker-startup-delay",=20NULL);=0A+=0A+=09= /*=20is=20the=20launcher=20already=20running?=20*/=0A+=09= launcher_running=20=3D=20DataChecksumsWorkerShmem->launcher_running;=0A+=0A= +=09LWLockRelease(DataChecksumsWorkerLock);=0A+=0A+=09/*=0A+=09=20*=20= Launch=20a=20new=20launcher=20process,=20if=20it's=20not=20running=20= already.=0A+=09=20*=0A+=09=20*=20If=20the=20launcher=20is=20currently=20= busy=20enabling=20the=20checksums,=20and=20we=20want=0A+=09=20*=20them=20= disabled=20(or=20vice=20versa),=20the=20launcher=20will=20notice=20that=20= at=20latest=0A+=09=20*=20when=20it's=20about=20to=20exit,=20and=20will=20= loop=20back=20process=20the=20new=20request.=20So=0A+=09=20*=20if=20the=20= launcher=20is=20already=20running,=20we=20don't=20need=20to=20do=20= anything=20more=0A+=09=20*=20here=20to=20abort=20it.=0A+=09=20*=0A+=09=20= *=20If=20you=20call=20pg_enable/disable_data_checksums()=20twice=20in=20= a=20row,=20before=0A+=09=20*=20the=20launcher=20has=20had=20a=20chance=20= to=20start=20up,=20we=20still=20end=20up=20launching=20it=0A+=09=20*=20= twice.=20=20That's=20OK,=20the=20second=20invocation=20will=20see=20that=20= a=20launcher=20is=0A+=09=20*=20already=20running=20and=20exit=20quickly.=0A= +=09=20*=0A+=09=20*=20TODO:=20We=20could=20optimize=20here=20and=20skip=20= launching=20the=20launcher,=20if=20we=20are=0A+=09=20*=20already=20in=20= the=20desired=20state,=20i.e.=20if=20the=20checksums=20are=20already=20= enabled=0A+=09=20*=20and=20you=20call=20pg_enable_data_checksums().=0A+=09= =20*/=0A+=09if=20(!launcher_running)=0A+=09{=0A+=09=09/*=0A+=09=09=20*=20= Prepare=20the=20BackgroundWorker=20and=20launch=20it.=0A+=09=09=20*/=0A+=09= =09memset(&bgw,=200,=20sizeof(bgw));=0A+=09=09bgw.bgw_flags=20=3D=20= BGWORKER_SHMEM_ACCESS=20|=20BGWORKER_BACKEND_DATABASE_CONNECTION;=0A+=09=09= bgw.bgw_start_time=20=3D=20BgWorkerStart_RecoveryFinished;=0A+=09=09= snprintf(bgw.bgw_library_name,=20BGW_MAXLEN,=20"postgres");=0A+=09=09= snprintf(bgw.bgw_function_name,=20BGW_MAXLEN,=20= "DataChecksumsWorkerLauncherMain");=0A+=09=09snprintf(bgw.bgw_name,=20= BGW_MAXLEN,=20"datachecksum=20launcher");=0A+=09=09= snprintf(bgw.bgw_type,=20BGW_MAXLEN,=20"datachecksum=20launcher");=0A+=09= =09bgw.bgw_restart_time=20=3D=20BGW_NEVER_RESTART;=0A+=09=09= bgw.bgw_notify_pid=20=3D=20MyProcPid;=0A+=09=09bgw.bgw_main_arg=20=3D=20= (Datum)=200;=0A+=0A+=09=09if=20(!RegisterDynamicBackgroundWorker(&bgw,=20= &bgw_handle))=0A+=09=09=09ereport(ERROR,=0A+=09=09=09=09=09= errcode(ERRCODE_INSUFFICIENT_RESOURCES),=0A+=09=09=09=09=09= errmsg("failed=20to=20start=20background=20worker=20to=20process=20data=20= checksums"));=0A+=09}=0A+=09else=0A+=09=09ereport(ERROR,=0A+=09=09=09=09= errmsg("data=20checksum=20processing=20already=20running"));=0A+}=0A+=0A= +/*=0A+=20*=20ProcessSingleRelationFork=0A+=20*=09=09Enable=20data=20= checksums=20in=20a=20single=20relation/fork.=0A+=20*=0A+=20*=20Returns=20= true=20if=20successful,=20and=20false=20if=20*aborted*.=20On=20error,=20= an=20actual=0A+=20*=20error=20is=20raised=20in=20the=20lower=20levels.=0A= +=20*/=0A+static=20bool=0A+ProcessSingleRelationFork(Relation=20reln,=20= ForkNumber=20forkNum,=20BufferAccessStrategy=20strategy)=0A+{=0A+=09= BlockNumber=20numblocks=20=3D=20RelationGetNumberOfBlocksInFork(reln,=20= forkNum);=0A+=09char=09=09activity[NAMEDATALEN=20*=202=20+=20128];=0A+=09= char=09=20=20=20*relns;=0A+=0A+=09relns=20=3D=20= get_namespace_name(RelationGetNamespace(reln));=0A+=0A+=09if=20(!relns)=0A= +=09=09return=20false;=0A+=0A+=09/*=20Report=20the=20current=20relation=20= to=20pgstat_activity=20*/=0A+=09snprintf(activity,=20sizeof(activity)=20= -=201,=20"processing:=20%s.%s=20(%s,=20%dblocks)",=0A+=09=09=09=20relns,=20= RelationGetRelationName(reln),=20forkNames[forkNum],=20numblocks);=0A+=09= pgstat_report_activity(STATE_RUNNING,=20activity);=0A+=09= pgstat_progress_update_param(PROGRESS_DATACHECKSUMS_BLOCKS_TOTAL,=20= numblocks);=0A+=0A+=09/*=0A+=09=20*=20We=20are=20looping=20over=20the=20= blocks=20which=20existed=20at=20the=20time=20of=20process=0A+=09=20*=20= start,=20which=20is=20safe=20since=20new=20blocks=20are=20created=20with=20= checksums=20set=0A+=09=20*=20already=20due=20to=20the=20state=20being=20= "inprogress-on".=0A+=09=20*/=0A+=09for=20(BlockNumber=20blknum=20=3D=20= 0;=20blknum=20<=20numblocks;=20blknum++)=0A+=09{=0A+=09=09Buffer=09=09= buf=20=3D=20ReadBufferExtended(reln,=20forkNum,=20blknum,=20RBM_NORMAL,=20= strategy);=0A+=0A+=09=09/*=20Need=20to=20get=20an=20exclusive=20lock=20= before=20we=20can=20flag=20as=20dirty=20*/=0A+=09=09LockBuffer(buf,=20= BUFFER_LOCK_EXCLUSIVE);=0A+=0A+=09=09/*=0A+=09=09=20*=20Mark=20the=20= buffer=20as=20dirty=20and=20force=20a=20full=20page=20write.=20=20We=20= have=20to=0A+=09=09=20*=20re-write=20the=20page=20to=20WAL=20even=20if=20= the=20checksum=20hasn't=20changed,=0A+=09=09=20*=20because=20if=20there=20= is=20a=20replica=20it=20might=20have=20a=20slightly=20different=0A+=09=09= =20*=20version=20of=20the=20page=20with=20an=20invalid=20checksum,=20= caused=20by=20unlogged=0A+=09=09=20*=20changes=20(e.g.=20hintbits)=20on=20= the=20master=20happening=20while=20checksums=0A+=09=09=20*=20were=20off.=20= This=20can=20happen=20if=20there=20was=20a=20valid=20checksum=20on=20the=20= page=0A+=09=09=20*=20at=20one=20point=20in=20the=20past,=20so=20only=20= when=20checksums=20are=20first=20on,=20then=0A+=09=09=20*=20off,=20and=20= then=20turned=20on=20again.=20=20TODO:=20investigate=20if=20this=20could=20= be=0A+=09=09=20*=20avoided=20if=20the=20checksum=20is=20calculated=20to=20= be=20correct=20and=20wal_level=0A+=09=09=20*=20is=20set=20to=20= "minimal",=0A+=09=09=20*/=0A+=09=09START_CRIT_SECTION();=0A+=09=09= MarkBufferDirty(buf);=0A+=09=09log_newpage_buffer(buf,=20false);=0A+=09=09= END_CRIT_SECTION();=0A+=0A+=09=09UnlockReleaseBuffer(buf);=0A+=0A+=09=09= /*=0A+=09=09=20*=20This=20is=20the=20only=20place=20where=20we=20check=20= if=20we=20are=20asked=20to=20abort,=20the=0A+=09=09=20*=20abortion=20= will=20bubble=20up=20from=20here.=20It's=20safe=20to=20check=20this=20= without=0A+=09=09=20*=20a=20lock,=20because=20if=20we=20miss=20it=20= being=20set,=20we=20will=20try=20again=20soon.=0A+=09=09=20*/=0A+=09=09= Assert(operation=20=3D=3D=20ENABLE_DATACHECKSUMS);=0A+=09=09= LWLockAcquire(DataChecksumsWorkerLock,=20LW_SHARED);=0A+=09=09if=20= (DataChecksumsWorkerShmem->launch_operation=20=3D=3D=20= DISABLE_DATACHECKSUMS)=0A+=09=09=09abort_requested=20=3D=20true;=0A+=09=09= LWLockRelease(DataChecksumsWorkerLock);=0A+=0A+=09=09if=20= (abort_requested)=0A+=09=09=09return=20false;=0A+=0A+=09=09/*=0A+=09=09=20= *=20As=20of=20now=20we=20only=20update=20the=20block=20counter=20for=20= main=20forks=20in=20order=0A+=09=09=20*=20to=20not=20cause=20too=20= frequent=20calls.=20TODO:=20investigate=20whether=20we=0A+=09=09=20*=20= should=20do=20it=20more=20frequent?=0A+=09=09=20*/=0A+=09=09if=20= (forkNum=20=3D=3D=20MAIN_FORKNUM)=0A+=09=09=09= pgstat_progress_update_param(PROGRESS_DATACHECKSUMS_BLOCKS_DONE,=0A+=09=09= =09=09=09=09=09=09=09=09=20(blknum=20+=201));=0A+=0A+=09=09/*=0A+=09=09=20= *=20Processing=20is=20re-using=20the=20vacuum=20cost=20delay=20for=20= process=0A+=09=09=20*=20throttling,=20hence=20why=20we=20call=20vacuum=20= APIs=20here.=0A+=09=09=20*/=0A+=09=09vacuum_delay_point(false);=0A+=09}=0A= +=0A+=09pfree(relns);=0A+=09return=20true;=0A+}=0A+=0A+/*=0A+=20*=20= ProcessSingleRelationByOid=0A+=20*=09=09Process=20a=20single=20relation=20= based=20on=20oid.=0A+=20*=0A+=20*=20Returns=20true=20if=20successful,=20= and=20false=20if=20*aborted*.=20On=20error,=20an=20actual=0A+=20*=20= error=20is=20raised=20in=20the=20lower=20levels.=0A+=20*/=0A+static=20= bool=0A+ProcessSingleRelationByOid(Oid=20relationId,=20= BufferAccessStrategy=20strategy)=0A+{=0A+=09Relation=09rel;=0A+=09bool=09= =09aborted=20=3D=20false;=0A+=0A+=09StartTransactionCommand();=0A+=0A+=09= rel=20=3D=20try_relation_open(relationId,=20AccessShareLock);=0A+=09if=20= (rel=20=3D=3D=20NULL)=0A+=09{=0A+=09=09/*=0A+=09=09=20*=20Relation=20no=20= longer=20exists.=20We=20don't=20consider=20this=20an=20error=20since=0A+=09= =09=20*=20there=20are=20no=20pages=20in=20it=20that=20need=20data=20= checksums,=20and=20thus=20return=0A+=09=09=20*=20true.=20The=20worker=20= operates=20off=20a=20list=20of=20relations=20generated=20at=20the=0A+=09=09= =20*=20start=20of=20processing,=20so=20relations=20being=20dropped=20in=20= the=20meantime=20is=0A+=09=09=20*=20to=20be=20expected.=0A+=09=09=20*/=0A= +=09=09CommitTransactionCommand();=0A+=09=09= pgstat_report_activity(STATE_IDLE,=20NULL);=0A+=09=09return=20true;=0A+=09= }=0A+=09RelationGetSmgr(rel);=0A+=0A+=09for=20(ForkNumber=20fnum=20=3D=20= 0;=20fnum=20<=3D=20MAX_FORKNUM;=20fnum++)=0A+=09{=0A+=09=09if=20= (smgrexists(rel->rd_smgr,=20fnum))=0A+=09=09{=0A+=09=09=09if=20= (!ProcessSingleRelationFork(rel,=20fnum,=20strategy))=0A+=09=09=09{=0A+=09= =09=09=09aborted=20=3D=20true;=0A+=09=09=09=09break;=0A+=09=09=09}=0A+=09= =09}=0A+=09}=0A+=09relation_close(rel,=20AccessShareLock);=0A+=09= elog(DEBUG2,=0A+=09=09=20"data=20checksum=20processing=20done=20for=20= relation=20with=20OID=20%u:=20%s",=0A+=09=09=20relationId,=20(aborted=20= ?=20"aborted"=20:=20"finished"));=0A+=0A+=09CommitTransactionCommand();=0A= +=0A+=09pgstat_report_activity(STATE_IDLE,=20NULL);=0A+=0A+=09return=20= !aborted;=0A+}=0A+=0A+/*=0A+=20*=20ProcessDatabase=0A+=20*=09=09Enable=20= data=20checksums=20in=20a=20single=20database.=0A+=20*=0A+=20*=20We=20do=20= this=20by=20launching=20a=20dynamic=20background=20worker=20into=20this=20= database,=20and=0A+=20*=20waiting=20for=20it=20to=20finish.=20=20We=20= have=20to=20do=20this=20in=20a=20separate=20worker,=20since=0A+=20*=20= each=20process=20can=20only=20be=20connected=20to=20one=20database=20= during=20its=20lifetime.=0A+=20*/=0A+static=20DataChecksumsWorkerResult=0A= +ProcessDatabase(DataChecksumsWorkerDatabase=20*db)=0A+{=0A+=09= BackgroundWorker=20bgw;=0A+=09BackgroundWorkerHandle=20*bgw_handle;=0A+=09= BgwHandleStatus=20status;=0A+=09pid_t=09=09pid;=0A+=09char=09=09= activity[NAMEDATALEN=20+=2064];=0A+=0A+=09= DataChecksumsWorkerShmem->success=20=3D=20DATACHECKSUMSWORKER_FAILED;=0A= +=0A+=09memset(&bgw,=200,=20sizeof(bgw));=0A+=09bgw.bgw_flags=20=3D=20= BGWORKER_SHMEM_ACCESS=20|=20BGWORKER_BACKEND_DATABASE_CONNECTION;=0A+=09= bgw.bgw_start_time=20=3D=20BgWorkerStart_RecoveryFinished;=0A+=09= snprintf(bgw.bgw_library_name,=20BGW_MAXLEN,=20"postgres");=0A+=09= snprintf(bgw.bgw_function_name,=20BGW_MAXLEN,=20"%s",=20= "DataChecksumsWorkerMain");=0A+=09snprintf(bgw.bgw_name,=20BGW_MAXLEN,=20= "datachecksum=20worker");=0A+=09snprintf(bgw.bgw_type,=20BGW_MAXLEN,=20= "datachecksum=20worker");=0A+=09bgw.bgw_restart_time=20=3D=20= BGW_NEVER_RESTART;=0A+=09bgw.bgw_notify_pid=20=3D=20MyProcPid;=0A+=09= bgw.bgw_main_arg=20=3D=20ObjectIdGetDatum(db->dboid);=0A+=0A+=09/*=0A+=09= =20*=20If=20there=20are=20no=20worker=20slots=20available,=20make=20sure=20= we=20retry=20processing=0A+=09=20*=20this=20database.=20This=20will=20= make=20the=20datachecksumsworker=20move=20on=20to=20the=0A+=09=20*=20= next=20database=20and=20quite=20likely=20fail=20with=20the=20same=20= problem.=20TODO:=20Maybe=0A+=09=20*=20we=20need=20a=20backoff=20to=20= avoid=20running=20through=20all=20the=20databases=20here=20in=0A+=09=20*=20= short=20order.=0A+=09=20*/=0A+=09if=20= (!RegisterDynamicBackgroundWorker(&bgw,=20&bgw_handle))=0A+=09{=0A+=09=09= ereport(WARNING,=0A+=09=09=09=09errmsg("failed=20to=20start=20worker=20= for=20enabling=20data=20checksums=20in=20database=20\"%s\",=20retrying",=0A= +=09=09=09=09=09=20=20=20db->dbname),=0A+=09=09=09=09errhint("The=20= max_worker_processes=20setting=20might=20be=20too=20low."));=0A+=09=09= return=20DATACHECKSUMSWORKER_RETRYDB;=0A+=09}=0A+=0A+=09status=20=3D=20= WaitForBackgroundWorkerStartup(bgw_handle,=20&pid);=0A+=09if=20(status=20= =3D=3D=20BGWH_STOPPED)=0A+=09{=0A+=09=09ereport(WARNING,=0A+=09=09=09=09= errmsg("could=20not=20start=20background=20worker=20for=20enabling=20= data=20checksums=20in=20database=20\"%s\"",=0A+=09=09=09=09=09=20=20=20= db->dbname),=0A+=09=09=09=09errhint("More=20details=20on=20the=20error=20= might=20be=20found=20in=20the=20server=20log."));=0A+=09=09return=20= DATACHECKSUMSWORKER_FAILED;=0A+=09}=0A+=0A+=09/*=0A+=09=20*=20If=20the=20= postmaster=20crashed=20we=20cannot=20end=20up=20with=20a=20processed=20= database=20so=0A+=09=20*=20we=20have=20no=20alternative=20other=20than=20= exiting.=20When=20enabling=20checksums=20we=0A+=09=20*=20won't=20at=20= this=20time=20have=20changed=20the=20pg_control=20version=20to=20enabled=20= so=0A+=09=20*=20when=20the=20cluster=20comes=20back=20up=20processing=20= will=20have=20to=20be=20restarted.=0A+=09=20*=20When=20disabling,=20the=20= pg_control=20version=20will=20be=20set=20to=20off=20before=20this=0A+=09=20= *=20so=20when=20the=20cluster=20comes=20up=20checksums=20will=20be=20off=20= as=20expected.=0A+=09=20*/=0A+=09if=20(status=20=3D=3D=20= BGWH_POSTMASTER_DIED)=0A+=09=09ereport(FATAL,=0A+=09=09=09=09= errcode(ERRCODE_ADMIN_SHUTDOWN),=0A+=09=09=09=09errmsg("cannot=20enable=20= data=20checksums=20without=20the=20postmaster=20process"),=0A+=09=09=09=09= errhint("Restart=20the=20database=20and=20restart=20data=20checksum=20= processing=20by=20calling=20pg_enable_data_checksums()."));=0A+=0A+=09= Assert(status=20=3D=3D=20BGWH_STARTED);=0A+=09ereport(DEBUG1,=0A+=09=09=09= errmsg("initiating=20data=20checksum=20processing=20in=20database=20= \"%s\"",=0A+=09=09=09=09=20=20=20db->dbname));=0A+=0A+=09= DataChecksumsWorkerShmem->worker_running=20=3D=20true;=0A+=0A+=09= snprintf(activity,=20sizeof(activity)=20-=201,=0A+=09=09=09=20"Waiting=20= for=20worker=20in=20database=20%s=20(pid=20%ld)",=20db->dbname,=20(long)=20= pid);=0A+=09pgstat_report_activity(STATE_RUNNING,=20activity);=0A+=0A+=09= status=20=3D=20WaitForBackgroundWorkerShutdown(bgw_handle);=0A+=09if=20= (status=20=3D=3D=20BGWH_POSTMASTER_DIED)=0A+=09=09ereport(FATAL,=0A+=09=09= =09=09errcode(ERRCODE_ADMIN_SHUTDOWN),=0A+=09=09=09=09errmsg("postmaster=20= exited=20during=20data=20checksum=20processing=20in=20\"%s\"",=0A+=09=09=09= =09=09=20=20=20db->dbname),=0A+=09=09=09=09errhint("Restart=20the=20= database=20and=20restart=20data=20checksum=20processing=20by=20calling=20= pg_enable_data_checksums()."));=0A+=0A+=09if=20= (DataChecksumsWorkerShmem->success=20=3D=3D=20= DATACHECKSUMSWORKER_ABORTED)=0A+=09=09ereport(LOG,=0A+=09=09=09=09= errmsg("data=20checksums=20processing=20was=20aborted=20in=20database=20= \"%s\"",=0A+=09=09=09=09=09=20=20=20db->dbname));=0A+=0A+=09= pgstat_report_activity(STATE_IDLE,=20NULL);=0A+=09= DataChecksumsWorkerShmem->worker_running=20=3D=20false;=0A+=0A+=09return=20= DataChecksumsWorkerShmem->success;=0A+}=0A+=0A+/*=0A+=20*=20= launcher_exit=0A+=20*=0A+=20*=20Internal=20routine=20for=20cleaning=20up=20= state=20when=20the=20launcher=20process=20exits.=20We=0A+=20*=20need=20= to=20clean=20up=20the=20abort=20flag=20to=20ensure=20that=20processing=20= started=20again=20if=0A+=20*=20it=20was=20previously=20aborted=20(note:=20= started=20again,=20*not*=20restarted=20from=20where=0A+=20*=20it=20left=20= off).=0A+=20*/=0A+static=20void=0A+launcher_exit(int=20code,=20Datum=20= arg)=0A+{=0A+=09if=20(launcher_running)=0A+=09{=0A+=09=09= LWLockAcquire(DataChecksumsWorkerLock,=20LW_EXCLUSIVE);=0A+=09=09= launcher_running=20=3D=20false;=0A+=09=09= DataChecksumsWorkerShmem->launcher_running=20=3D=20false;=0A+=0A+=09=09= /*=0A+=09=09=20*=20TODO:=20how=20to=20really=20handle=20the=20worker=20= still=20running=20when=20the=0A+=09=09=20*=20launcher=20exits?=0A+=09=09=20= */=0A+=09=09if=20(DataChecksumsWorkerShmem->worker_running)=0A+=09=09=09= ereport(LOG,=0A+=09=09=09=09=09errmsg("data=20checksums=20launcher=20= exiting=20while=20worker=20is=20still=20running"));=0A+=09=09= LWLockRelease(DataChecksumsWorkerLock);=0A+=09}=0A+}=0A+=0A+/*=0A+=20*=20= launcher_cancel_handler=0A+=20*=0A+=20*=20Internal=20routine=20for=20= reacting=20to=20SIGINT=20and=20flagging=20the=20worker=20to=20abort.=0A+=20= *=20The=20worker=20won't=20be=20interrupted=20immediately=20but=20will=20= check=20for=20abort=20flag=0A+=20*=20between=20each=20block=20in=20a=20= relation.=0A+=20*/=0A+static=20void=0A= +launcher_cancel_handler(SIGNAL_ARGS)=0A+{=0A+=09int=09=09=09save_errno=20= =3D=20errno;=0A+=0A+=09abort_requested=20=3D=20true;=0A+=0A+=09/*=0A+=09=20= *=20There=20is=20no=20sleeping=20in=20the=20main=20loop,=20the=20flag=20= will=20be=20checked=0A+=09=20*=20periodically=20in=20= ProcessSingleRelationFork.=20The=20worker=20does=20however=0A+=09=20*=20= sleep=20when=20waiting=20for=20concurrent=20transactions=20to=20end=20so=20= we=20still=20need=0A+=09=20*=20to=20set=20the=20latch.=0A+=09=20*/=0A+=09= SetLatch(MyLatch);=0A+=0A+=09errno=20=3D=20save_errno;=0A+}=0A+=0A+/*=0A= +=20*=20WaitForAllTransactionsToFinish=0A+=20*=09=09Blocks=20awaiting=20= all=20current=20transactions=20to=20finish=0A+=20*=0A+=20*=20Returns=20= when=20all=20transactions=20which=20are=20active=20at=20the=20call=20of=20= the=20function=0A+=20*=20have=20ended,=20or=20if=20the=20postmaster=20= dies=20while=20waiting.=20If=20the=20postmaster=20dies=0A+=20*=20the=20= abort=20flag=20will=20be=20set=20to=20indicate=20that=20the=20caller=20= of=20this=20shouldn't=0A+=20*=20proceed.=0A+=20*=0A+=20*=20NB:=20this=20= will=20return=20early,=20if=20aborted=20by=20SIGINT=20or=20if=20the=20= target=20state=0A+=20*=20is=20changed=20while=20we're=20running.=0A+=20= */=0A+static=20void=0A+WaitForAllTransactionsToFinish(void)=0A+{=0A+=09= TransactionId=20waitforxid;=0A+=0A+=09LWLockAcquire(XidGenLock,=20= LW_SHARED);=0A+=09waitforxid=20=3D=20= XidFromFullTransactionId(TransamVariables->nextXid);=0A+=09= LWLockRelease(XidGenLock);=0A+=0A+=09while=20= (TransactionIdPrecedes(GetOldestActiveTransactionId(false,=20true),=20= waitforxid))=0A+=09{=0A+=09=09char=09=09activity[64];=0A+=09=09int=09=09=09= rc;=0A+=0A+=09=09/*=20Oldest=20running=20xid=20is=20older=20than=20us,=20= so=20wait=20*/=0A+=09=09snprintf(activity,=0A+=09=09=09=09=20= sizeof(activity),=0A+=09=09=09=09=20"Waiting=20for=20current=20= transactions=20to=20finish=20(waiting=20for=20%u)",=0A+=09=09=09=09=20= waitforxid);=0A+=09=09pgstat_report_activity(STATE_RUNNING,=20activity);=0A= +=0A+=09=09/*=20Retry=20every=203=20seconds=20*/=0A+=09=09= ResetLatch(MyLatch);=0A+=09=09rc=20=3D=20WaitLatch(MyLatch,=0A+=09=09=09=09= =09=20=20=20WL_LATCH_SET=20|=20WL_TIMEOUT=20|=20WL_POSTMASTER_DEATH,=0A+=09= =09=09=09=09=20=20=203000,=0A+=09=09=09=09=09=20=20=20= WAIT_EVENT_CHECKSUM_ENABLE_STARTCONDITION);=0A+=0A+=09=09/*=0A+=09=09=20= *=20If=20the=20postmaster=20died=20we=20won't=20be=20able=20to=20enable=20= checksums=0A+=09=09=20*=20cluster-wide=20so=20abort=20and=20hope=20to=20= continue=20when=20restarted.=0A+=09=09=20*/=0A+=09=09if=20(rc=20&=20= WL_POSTMASTER_DEATH)=0A+=09=09=09ereport(FATAL,=0A+=09=09=09=09=09= errcode(ERRCODE_ADMIN_SHUTDOWN),=0A+=09=09=09=09=09errmsg("postmaster=20= exited=20during=20data=20checksums=20processing"),=0A+=09=09=09=09=09= errhint("Data=20checksums=20processing=20must=20be=20restarted=20= manually=20after=20cluster=20restart."));=0A+=0A+=09=09= LWLockAcquire(DataChecksumsWorkerLock,=20LW_SHARED);=0A+=09=09if=20= (DataChecksumsWorkerShmem->launch_operation=20!=3D=20operation)=0A+=09=09= =09abort_requested=20=3D=20true;=0A+=09=09= LWLockRelease(DataChecksumsWorkerLock);=0A+=09=09if=20(abort_requested)=0A= +=09=09=09break;=0A+=09}=0A+=0A+=09pgstat_report_activity(STATE_IDLE,=20= NULL);=0A+=09return;=0A+}=0A+=0A+/*=0A+=20*=20= DataChecksumsWorkerLauncherMain=0A+=20*=0A+=20*=20Main=20function=20for=20= launching=20dynamic=20background=20workers=20for=20processing=20data=0A+=20= *=20checksums=20in=20databases.=20This=20function=20has=20the=20bgworker=20= management,=20with=0A+=20*=20ProcessAllDatabases=20being=20responsible=20= for=20looping=20over=20the=20databases=20and=0A+=20*=20initiating=20= processing.=0A+=20*/=0A+void=0A+DataChecksumsWorkerLauncherMain(Datum=20= arg)=0A+{=0A+=09on_shmem_exit(launcher_exit,=200);=0A+=0A+=09= ereport(DEBUG1,=0A+=09=09=09errmsg("background=20worker=20= \"datachecksums=20launcher\"=20started"));=0A+=0A+=09pqsignal(SIGTERM,=20= die);=0A+=09pqsignal(SIGINT,=20launcher_cancel_handler);=0A+=0A+=09= BackgroundWorkerUnblockSignals();=0A+=0A+=09MyBackendType=20=3D=20= B_DATACHECKSUMSWORKER_LAUNCHER;=0A+=09init_ps_display(NULL);=0A+=0A+=09= INJECTION_POINT("datachecksumsworker-launcher-delay",=20NULL);=0A+=0A+=09= LWLockAcquire(DataChecksumsWorkerLock,=20LW_EXCLUSIVE);=0A+=0A+=09if=20= (DataChecksumsWorkerShmem->launcher_running)=0A+=09{=0A+=09=09= ereport(LOG,=0A+=09=09=09=09errmsg("background=20worker=20= \"datachecksums=20launcher\"=20already=20running,=20exiting"));=0A+=09=09= /*=20Launcher=20was=20already=20running,=20let=20it=20finish=20*/=0A+=09=09= LWLockRelease(DataChecksumsWorkerLock);=0A+=09=09return;=0A+=09}=0A+=0A+=09= launcher_running=20=3D=20true;=0A+=0A+=09/*=0A+=09=20*=20Initialize=20a=20= connection=20to=20shared=20catalogs=20only.=0A+=09=20*/=0A+=09= BackgroundWorkerInitializeConnectionByOid(InvalidOid,=20InvalidOid,=20= 0);=0A+=0A+=09operation=20=3D=20= DataChecksumsWorkerShmem->launch_operation;=0A+=09= DataChecksumsWorkerShmem->launcher_running=20=3D=20true;=0A+=09= DataChecksumsWorkerShmem->operation=20=3D=20operation;=0A+=09= DataChecksumsWorkerShmem->cost_delay=20=3D=20= DataChecksumsWorkerShmem->launch_cost_delay;=0A+=09= DataChecksumsWorkerShmem->cost_limit=20=3D=20= DataChecksumsWorkerShmem->launch_cost_limit;=0A+=09= LWLockRelease(DataChecksumsWorkerLock);=0A+=0A+=09/*=0A+=09=20*=20The=20= target=20state=20can=20change=20while=20we=20are=20busy=20= enabling/disabling=0A+=09=20*=20checksums,=20if=20the=20user=20calls=20= pg_disable/enable_data_checksums()=20before=0A+=09=20*=20we=20are=20= finished=20with=20the=20previous=20request.=20In=20that=20case,=20we=20= will=20loop=0A+=09=20*=20back=20here,=20to=20process=20the=20new=20= request.=0A+=09=20*/=0A+again:=0A+=0A+=09= pgstat_progress_start_command(PROGRESS_COMMAND_DATACHECKSUMS,=0A+=09=09=09= =09=09=09=09=09=20=20InvalidOid);=0A+=0A+=09if=20(operation=20=3D=3D=20= ENABLE_DATACHECKSUMS)=0A+=09{=0A+=09=09/*=0A+=09=09=20*=20If=20we=20are=20= asked=20to=20enable=20checksums=20in=20a=20cluster=20which=20already=20= has=0A+=09=09=20*=20checksums=20enabled,=20exit=20immediately=20as=20= there=20is=20nothing=20more=20to=20do.=0A+=09=09=20*=20Hold=20interrupts=20= to=20make=20sure=20state=20doesn't=20change=20during=20checking.=0A+=09=09= =20*/=0A+=09=09if=20(DataChecksumsNeedVerify())=0A+=09=09=09goto=20done;=0A= +=0A+=09=09/*=0A+=09=09=20*=20Set=20the=20state=20to=20inprogress-on=20= and=20wait=20on=20the=20procsignal=20barrier.=0A+=09=09=20*/=0A+=09=09= pgstat_progress_update_param(PROGRESS_DATACHECKSUMS_PHASE,=0A+=09=09=09=09= =09=09=09=09=09=20PROGRESS_DATACHECKSUMS_PHASE_ENABLING);=0A+=09=09= SetDataChecksumsOnInProgress();=0A+=0A+=09=09/*=0A+=09=09=20*=20All=20= backends=20are=20now=20in=20inprogress-on=20state=20and=20are=20writing=20= data=0A+=09=09=20*=20checksums.=20=20Start=20processing=20all=20data=20= at=20rest.=0A+=09=09=20*/=0A+=09=09if=20(!ProcessAllDatabases())=0A+=09=09= {=0A+=09=09=09/*=0A+=09=09=09=20*=20If=20the=20target=20state=20changed=20= during=20processing=20then=20it's=20not=20a=0A+=09=09=09=20*=20failure,=20= so=20restart=20processing=20instead.=0A+=09=09=09=20*/=0A+=09=09=09= LWLockAcquire(DataChecksumsWorkerLock,=20LW_EXCLUSIVE);=0A+=09=09=09if=20= (DataChecksumsWorkerShmem->launch_operation=20!=3D=20operation)=0A+=09=09= =09{=0A+=09=09=09=09LWLockRelease(DataChecksumsWorkerLock);=0A+=09=09=09=09= goto=20done;=0A+=09=09=09}=0A+=09=09=09= LWLockRelease(DataChecksumsWorkerLock);=0A+=09=09=09ereport(ERROR,=0A+=09= =09=09=09=09errcode(ERRCODE_INSUFFICIENT_RESOURCES),=0A+=09=09=09=09=09= errmsg("unable=20to=20enable=20data=20checksums=20in=20cluster"));=0A+=09= =09}=0A+=0A+=09=09/*=0A+=09=09=20*=20Data=20checksums=20have=20been=20= set=20on=20all=20pages,=20set=20the=20state=20to=20on=20in=0A+=09=09=20*=20= order=20to=20instruct=20backends=20to=20validate=20checksums=20on=20= reading.=0A+=09=09=20*/=0A+=09=09SetDataChecksumsOn();=0A+=09}=0A+=09= else=20if=20(operation=20=3D=3D=20DISABLE_DATACHECKSUMS)=0A+=09{=0A+=09=09= pgstat_progress_update_param(PROGRESS_DATACHECKSUMS_PHASE,=0A+=09=09=09=09= =09=09=09=09=09=20PROGRESS_DATACHECKSUMS_PHASE_DISABLING);=0A+=09=09= SetDataChecksumsOff();=0A+=09}=0A+=09else=0A+=09{=0A+=09=09= Assert(false);=0A+=09}=0A+=0A+done:=0A+=0A+=09/*=0A+=09=20*=20This=20= state=20will=20only=20be=20displayed=20for=20a=20fleeting=20moment,=20= but=20for=20the=0A+=09=20*=20sake=20of=20correctness=20it=20is=20still=20= added=20before=20ending=20the=20command.=0A+=09=20*/=0A+=09= pgstat_progress_update_param(PROGRESS_DATACHECKSUMS_PHASE,=0A+=09=09=09=09= =09=09=09=09=20PROGRESS_DATACHECKSUMS_PHASE_DONE);=0A+=0A+=09/*=0A+=09=20= *=20All=20done.=20But=20before=20we=20exit,=20check=20if=20the=20target=20= state=20was=20changed=0A+=09=20*=20while=20we=20were=20running.=20In=20= that=20case=20we=20will=20have=20to=20start=20all=20over=0A+=09=20*=20= again.=0A+=09=20*/=0A+=09LWLockAcquire(DataChecksumsWorkerLock,=20= LW_EXCLUSIVE);=0A+=09if=20(DataChecksumsWorkerShmem->launch_operation=20= !=3D=20operation)=0A+=09{=0A+=09=09DataChecksumsWorkerShmem->operation=20= =3D=20DataChecksumsWorkerShmem->launch_operation;=0A+=09=09operation=20=3D= =20DataChecksumsWorkerShmem->launch_operation;=0A+=09=09= DataChecksumsWorkerShmem->cost_delay=20=3D=20= DataChecksumsWorkerShmem->launch_cost_delay;=0A+=09=09= DataChecksumsWorkerShmem->cost_limit=20=3D=20= DataChecksumsWorkerShmem->launch_cost_limit;=0A+=09=09= LWLockRelease(DataChecksumsWorkerLock);=0A+=09=09goto=20again;=0A+=09}=0A= +=0A+=09/*=20Shut=20down=20progress=20reporting=20as=20we=20are=20done=20= */=0A+=09pgstat_progress_end_command();=0A+=0A+=09launcher_running=20=3D=20= false;=0A+=09DataChecksumsWorkerShmem->launcher_running=20=3D=20false;=0A= +=09LWLockRelease(DataChecksumsWorkerLock);=0A+}=0A+=0A+/*=0A+=20*=20= ProcessAllDatabases=0A+=20*=09=09Compute=20the=20list=20of=20all=20= databases=20and=20process=20checksums=20in=20each=0A+=20*=0A+=20*=20This=20= will=20repeatedly=20generate=20a=20list=20of=20databases=20to=20process=20= for=20enabling=0A+=20*=20checksums.=20Until=20no=20new=20databases=20are=20= found,=20this=20will=20loop=20around=20computing=0A+=20*=20a=20new=20= list=20and=20comparing=20it=20to=20the=20already=20seen=20ones.=0A+=20*=0A= +=20*/=0A+static=20bool=0A+ProcessAllDatabases(void)=0A+{=0A+=09List=09=20= =20=20*DatabaseList;=0A+=09HTAB=09=20=20=20*ProcessedDatabases=20=3D=20= NULL;=0A+=09HASHCTL=09=09hash_ctl;=0A+=09bool=09=09found_failed=20=3D=20= false;=0A+=0A+=09/*=20Initialize=20a=20hash=20tracking=20all=20processed=20= databases=20*/=0A+=09memset(&hash_ctl,=200,=20sizeof(hash_ctl));=0A+=09= hash_ctl.keysize=20=3D=20sizeof(Oid);=0A+=09hash_ctl.entrysize=20=3D=20= sizeof(DataChecksumsWorkerResultEntry);=0A+=09ProcessedDatabases=20=3D=20= hash_create("Processed=20databases",=0A+=09=09=09=09=09=09=09=09=09=20= 64,=0A+=09=09=09=09=09=09=09=09=09=20&hash_ctl,=0A+=09=09=09=09=09=09=09=09= =09=20HASH_ELEM=20|=20HASH_BLOBS);=0A+=0A+=09/*=0A+=09=20*=20Set=20up=20= so=20first=20run=20processes=20shared=20catalogs,=20but=20not=20once=20= in=20every=0A+=09=20*=20db.=0A+=09=20*/=0A+=09= DataChecksumsWorkerShmem->process_shared_catalogs=20=3D=20true;=0A+=0A+=09= /*=0A+=09=20*=20Get=20a=20list=20of=20all=20databases=20to=20process.=20= This=20may=20include=20databases=20that=0A+=09=20*=20were=20created=20= during=20our=20runtime.=20=20Since=20a=20database=20can=20be=20created=20= as=20a=0A+=09=20*=20copy=20of=20any=20other=20database=20(which=20may=20= not=20have=20existed=20in=20our=20last=0A+=09=20*=20run),=20we=20have=20= to=20repeat=20this=20loop=20until=20no=20new=20databases=20show=20up=20= in=20the=0A+=09=20*=20list.=20=20Here=20the=20initial=20list=20for=20the=20= loop=20processing=20is=20generated=20after=0A+=09=20*=20waiting=20for=20= all=20existing=20transactions=20to=20finish=20to=20ensure=20that=20we=20= can=0A+=09=20*=20see=20any=20database=20which=20was=20created=20even=20= if=20the=20transaction=20in=20which=20it=0A+=09=20*=20was=20created=20= started=20before=20checksums=20were=20being=20processed.=0A+=09=20*/=0A+=09= WaitForAllTransactionsToFinish();=0A+=09DatabaseList=20=3D=20= BuildDatabaseList();=0A+=0A+=09/*=20Allow=20a=20test=20case=20to=20= modify=20the=20initial=20list=20of=20databases=20*/=0A+=09= INJECTION_POINT("datachecksumsworker-initial-dblist",=20DatabaseList);=0A= +=0A+=09/*=0A+=09=20*=20Update=20progress=20reporting=20with=20the=20= total=20number=20of=20databases=20we=20need=20to=0A+=09=20*=20process.=20= =20This=20number=20should=20not=20be=20changed=20during=20processing,=20= the=0A+=09=20*=20columns=20for=20processed=20databases=20is=20instead=20= increased=20such=20that=20it=20can=0A+=09=20*=20be=20compared=20against=20= the=20total.=0A+=09=20*/=0A+=09{=0A+=09=09const=20int=09index[]=20=3D=20= {=0A+=09=09=09PROGRESS_DATACHECKSUMS_DBS_TOTAL,=0A+=09=09=09= PROGRESS_DATACHECKSUMS_DBS_DONE,=0A+=09=09=09= PROGRESS_DATACHECKSUMS_RELS_TOTAL,=0A+=09=09=09= PROGRESS_DATACHECKSUMS_RELS_DONE,=0A+=09=09=09= PROGRESS_DATACHECKSUMS_BLOCKS_TOTAL,=0A+=09=09=09= PROGRESS_DATACHECKSUMS_BLOCKS_DONE,=0A+=09=09};=0A+=0A+=09=09int64=09=09= vals[6];=0A+=0A+=09=09vals[0]=20=3D=20list_length(DatabaseList);=0A+=09=09= vals[1]=20=3D=200;=0A+=0A+=09=09/*=20translated=20to=20NULL=20*/=0A+=09=09= vals[2]=20=3D=20-1;=0A+=09=09vals[3]=20=3D=20-1;=0A+=09=09vals[4]=20=3D=20= -1;=0A+=09=09vals[5]=20=3D=20-1;=0A+=0A+=09=09= pgstat_progress_update_multi_param(6,=20index,=20vals);=0A+=09}=0A+=0A+=09= while=20(true)=0A+=09{=0A+=09=09int=09=09=09processed_databases=20=3D=20= 0;=0A+=0A+=09=09foreach_ptr(DataChecksumsWorkerDatabase,=20db,=20= DatabaseList)=0A+=09=09{=0A+=09=09=09DataChecksumsWorkerResult=20result;=0A= +=09=09=09DataChecksumsWorkerResultEntry=20*entry;=0A+=09=09=09bool=09=09= found;=0A+=0A+=09=09=09/*=0A+=09=09=09=20*=20Check=20if=20this=20= database=20has=20been=20processed=20already,=20and=20if=20so=0A+=09=09=09= =20*=20whether=20it=20should=20be=20retried=20or=20skipped.=0A+=09=09=09=20= */=0A+=09=09=09entry=20=3D=20(DataChecksumsWorkerResultEntry=20*)=20= hash_search(ProcessedDatabases,=20&db->dboid,=0A+=09=09=09=09=09=09=09=09= =09=09=09=09=09=09=09=09=20=20=20HASH_FIND,=20NULL);=0A+=0A+=09=09=09if=20= (entry)=0A+=09=09=09{=0A+=09=09=09=09if=20(entry->result=20=3D=3D=20= DATACHECKSUMSWORKER_RETRYDB)=0A+=09=09=09=09{=0A+=09=09=09=09=09/*=0A+=09= =09=09=09=09=20*=20Limit=20the=20number=20of=20retries=20to=20avoid=20= infinite=20looping=0A+=09=09=09=09=09=20*=20in=20case=20there=20simply=20= won't=20be=20enough=20workers=20in=20the=0A+=09=09=09=09=09=20*=20= cluster=20to=20finish=20this=20operation.=0A+=09=09=09=09=09=20*/=0A+=09=09= =09=09=09if=20(entry->retries=20>=20DATACHECKSUMSWORKER_MAX_DB_RETRIES)=0A= +=09=09=09=09=09=09entry->result=20=3D=20DATACHECKSUMSWORKER_FAILED;=0A+=09= =09=09=09}=0A+=0A+=09=09=09=09/*=20Skip=20if=20this=20database=20has=20= been=20processed=20already=20*/=0A+=09=09=09=09if=20(entry->result=20!=3D=20= DATACHECKSUMSWORKER_RETRYDB)=0A+=09=09=09=09=09continue;=0A+=09=09=09}=0A= +=0A+=09=09=09result=20=3D=20ProcessDatabase(db);=0A+=09=09=09= processed_databases++;=0A+=0A+=09=09=09/*=0A+=09=09=09=20*=20Update=20= the=20number=20of=20processed=20databases=20in=20the=20progress=0A+=09=09= =09=20*=20report.=0A+=09=09=09=20*/=0A+=09=09=09= pgstat_progress_update_param(PROGRESS_DATACHECKSUMS_DBS_DONE,=0A+=09=09=09= =09=09=09=09=09=09=09=20processed_databases);=0A+=0A+=09=09=09/*=20Allow=20= a=20test=20process=20to=20alter=20the=20result=20of=20the=20operation=20= */=0A+=09=09=09INJECTION_POINT("datachecksumsworker-fail-db",=20= &result);=0A+=0A+=09=09=09if=20(result=20=3D=3D=20= DATACHECKSUMSWORKER_SUCCESSFUL)=0A+=09=09=09{=0A+=09=09=09=09/*=0A+=09=09= =09=09=20*=20If=20one=20database=20has=20completed=20shared=20catalogs,=20= we=20don't=0A+=09=09=09=09=20*=20have=20to=20process=20them=20again.=0A+=09= =09=09=09=20*/=0A+=09=09=09=09if=20= (DataChecksumsWorkerShmem->process_shared_catalogs)=0A+=09=09=09=09=09= DataChecksumsWorkerShmem->process_shared_catalogs=20=3D=20false;=0A+=09=09= =09}=0A+=09=09=09else=20if=20(result=20=3D=3D=20= DATACHECKSUMSWORKER_ABORTED)=0A+=09=09=09{=0A+=09=09=09=09/*=20Abort=20= flag=20set,=20so=20exit=20the=20whole=20process=20*/=0A+=09=09=09=09= return=20false;=0A+=09=09=09}=0A+=0A+=09=09=09entry=20=3D=20= hash_search(ProcessedDatabases,=20&db->dboid,=20HASH_ENTER,=20&found);=0A= +=09=09=09entry->dboid=20=3D=20db->dboid;=0A+=09=09=09entry->result=20=3D=20= result;=0A+=09=09=09if=20(!found)=0A+=09=09=09=09entry->retries=20=3D=20= 0;=0A+=09=09=09else=0A+=09=09=09=09entry->retries++;=0A+=09=09}=0A+=0A+=09= =09elog(DEBUG1,=0A+=09=09=09=20"%i=20databases=20processed=20for=20data=20= checksum=20enabling,=20%s",=0A+=09=09=09=20processed_databases,=0A+=09=09= =09=20(processed_databases=20?=20"process=20with=20restart"=20:=20= "process=20completed"));=0A+=0A+=09=09FreeDatabaseList(DatabaseList);=0A= +=0A+=09=09/*=0A+=09=09=20*=20If=20no=20databases=20were=20processed=20= in=20this=20run=20of=20the=20loop,=20we=20have=20now=0A+=09=09=20*=20= finished=20all=20databases=20and=20no=20concurrently=20created=20ones=20= can=20exist.=0A+=09=09=20*/=0A+=09=09if=20(processed_databases=20=3D=3D=20= 0)=0A+=09=09=09break;=0A+=0A+=09=09/*=0A+=09=09=20*=20Re-generate=20the=20= list=20of=20databases=20for=20another=20pass=20in=20case=20any=20new=0A+=09= =09=20*=20databases=20were=20created=20while=20we=20were=20running.=20=20= Since=20the=20initial=0A+=09=09=20*=20list=20was=20generated=20efter=20= waiting=20for=20all=20transaction=20to=20finish=20we=0A+=09=09=20*=20= know=20that=20all=20new=20databases=20found=20here=20must=20have=20been=20= created=20while=0A+=09=09=20*=20seeing=20the=20new=20checksum=20state.=20= =20By=20waiting=20for=20all=20transactions=0A+=09=09=20*=20here=20as=20= well=20we=20know=20that=20any=20database=20created=20using=20an=20= existing=20db=0A+=09=09=20*=20as=20a=20template=20(which=20may=20have=20= been=20used=20before=20it=20had=20checksums=0A+=09=09=20*=20enabled)=20= will=20be=20committed.=0A+=09=09=20*/=0A+=09=09= WaitForAllTransactionsToFinish();=0A+=09=09DatabaseList=20=3D=20= BuildDatabaseList();=0A+=09}=0A+=0A+=09/*=0A+=09=20*=20= ProcessedDatabases=20now=20has=20all=20databases=20and=20the=20results=20= of=20their=0A+=09=20*=20processing.=20Failure=20to=20enable=20checksums=20= for=20a=20database=20can=20be=20because=0A+=09=20*=20they=20actually=20= failed=20for=20some=20reason,=20or=20because=20the=20database=20was=0A+=09= =20*=20dropped=20between=20us=20getting=20the=20database=20list=20and=20= trying=20to=20process=20it.=0A+=09=20*=20Get=20a=20fresh=20list=20of=20= databases=20to=20detect=20the=20second=20case=20where=20the=0A+=09=20*=20= database=20was=20dropped=20before=20we=20had=20started=20processing=20= it.=20If=20a=20database=0A+=09=20*=20still=20exists,=20but=20enabling=20= checksums=20failed=20then=20we=20fail=20the=20entire=0A+=09=20*=20= checksumming=20process=20and=20exit=20with=20an=20error.=0A+=09=20*/=0A+=09= DatabaseList=20=3D=20BuildDatabaseList();=0A+=0A+=09= foreach_ptr(DataChecksumsWorkerDatabase,=20db,=20DatabaseList)=0A+=09{=0A= +=09=09DataChecksumsWorkerResultEntry=20*entry;=0A+=09=09bool=09=09= found;=0A+=0A+=09=09entry=20=3D=20hash_search(ProcessedDatabases,=20= (void=20*)=20&db->dboid,=0A+=09=09=09=09=09=09=09HASH_FIND,=20&found);=0A= +=0A+=09=09/*=0A+=09=09=20*=20We=20are=20only=20interested=20in=20the=20= processed=20databases=20which=20failed,=20and=0A+=09=09=20*=20where=20= the=20failed=20database=20still=20exists.=20=20This=20indicates=20that=0A= +=09=09=20*=20enabling=20checksums=20actually=20failed,=20and=20not=20= that=20the=20failure=20was=0A+=09=09=20*=20due=20to=20the=20db=20being=20= concurrently=20dropped.=0A+=09=09=20*/=0A+=09=09if=20(found=20&&=20= entry->result=20=3D=3D=20DATACHECKSUMSWORKER_FAILED)=0A+=09=09{=0A+=09=09= =09ereport(WARNING,=0A+=09=09=09=09=09errmsg("failed=20to=20enable=20= data=20checksums=20in=20\"%s\"",=20db->dbname));=0A+=09=09=09= found_failed=20=3D=20found;=0A+=09=09=09continue;=0A+=09=09}=0A+=09}=0A+=0A= +=09FreeDatabaseList(DatabaseList);=0A+=0A+=09if=20(found_failed)=0A+=09= {=0A+=09=09/*=20Disable=20checksums=20on=20cluster,=20because=20we=20= failed=20*/=0A+=09=09SetDataChecksumsOff();=0A+=09=09ereport(ERROR,=0A+=09= =09=09=09errcode(ERRCODE_INSUFFICIENT_RESOURCES),=0A+=09=09=09=09= errmsg("data=20checksums=20failed=20to=20get=20enabled=20in=20all=20= databases,=20aborting"),=0A+=09=09=09=09errhint("The=20server=20log=20= might=20have=20more=20information=20on=20the=20cause=20of=20the=20= error."));=0A+=09}=0A+=0A+=09= pgstat_progress_update_param(PROGRESS_DATACHECKSUMS_PHASE,=0A+=09=09=09=09= =09=09=09=09=20PROGRESS_DATACHECKSUMS_PHASE_WAITING_BARRIER);=0A+=09= return=20true;=0A+}=0A+=0A+/*=0A+=20*=20DataChecksumsWorkerShmemSize=0A+=20= *=09=09Compute=20required=20space=20for=20datachecksumsworker-related=20= shared=20memory=0A+=20*/=0A+Size=0A+DataChecksumsWorkerShmemSize(void)=0A= +{=0A+=09Size=09=09size;=0A+=0A+=09size=20=3D=20= sizeof(DataChecksumsWorkerShmemStruct);=0A+=09size=20=3D=20= MAXALIGN(size);=0A+=0A+=09return=20size;=0A+}=0A+=0A+/*=0A+=20*=20= DataChecksumsWorkerShmemInit=0A+=20*=09=09Allocate=20and=20initialize=20= datachecksumsworker-related=20shared=20memory=0A+=20*/=0A+void=0A= +DataChecksumsWorkerShmemInit(void)=0A+{=0A+=09bool=09=09found;=0A+=0A+=09= DataChecksumsWorkerShmem=20=3D=20(DataChecksumsWorkerShmemStruct=20*)=0A= +=09=09ShmemInitStruct("DataChecksumsWorker=20Data",=0A+=09=09=09=09=09=09= DataChecksumsWorkerShmemSize(),=0A+=09=09=09=09=09=09&found);=0A+=0A+=09= if=20(!found)=0A+=09{=0A+=09=09MemSet(DataChecksumsWorkerShmem,=200,=20= DataChecksumsWorkerShmemSize());=0A+=0A+=09=09/*=0A+=09=09=20*=20Even=20= if=20this=20is=20a=20redundant=20assignment,=20we=20want=20to=20be=20= explicit=0A+=09=09=20*=20about=20our=20intent=20for=20readability.=0A+=09= =09=20*/=0A+=09=09DataChecksumsWorkerShmem->launch_operation=20=3D=20= false;=0A+=09=09DataChecksumsWorkerShmem->launcher_running=20=3D=20= false;=0A+=09=09DataChecksumsWorkerShmem->worker_running=20=3D=20false;=0A= +=09}=0A+}=0A+=0A+/*=0A+=20*=20BuildDatabaseList=0A+=20*=09=09Compile=20= a=20list=20of=20all=20currently=20available=20databases=20in=20the=20= cluster=0A+=20*=0A+=20*=20This=20creates=20the=20list=20of=20databases=20= for=20the=20datachecksumsworker=20workers=20to=0A+=20*=20add=20checksums=20= to.=20If=20the=20caller=20wants=20to=20ensure=20that=20no=20concurrently=0A= +=20*=20running=20CREATE=20DATABASE=20calls=20exist,=20this=20needs=20to=20= be=20preceded=20by=20a=20call=0A+=20*=20to=20= WaitForAllTransactionsToFinish().=0A+=20*/=0A+static=20List=20*=0A= +BuildDatabaseList(void)=0A+{=0A+=09List=09=20=20=20*DatabaseList=20=3D=20= NIL;=0A+=09Relation=09rel;=0A+=09TableScanDesc=20scan;=0A+=09HeapTuple=09= tup;=0A+=09MemoryContext=20ctx=20=3D=20CurrentMemoryContext;=0A+=09= MemoryContext=20oldctx;=0A+=0A+=09StartTransactionCommand();=0A+=0A+=09= rel=20=3D=20table_open(DatabaseRelationId,=20AccessShareLock);=0A+=09= scan=20=3D=20table_beginscan_catalog(rel,=200,=20NULL);=0A+=0A+=09while=20= (HeapTupleIsValid(tup=20=3D=20heap_getnext(scan,=20= ForwardScanDirection)))=0A+=09{=0A+=09=09Form_pg_database=20pgdb=20=3D=20= (Form_pg_database)=20GETSTRUCT(tup);=0A+=09=09= DataChecksumsWorkerDatabase=20*db;=0A+=0A+=09=09oldctx=20=3D=20= MemoryContextSwitchTo(ctx);=0A+=0A+=09=09db=20=3D=20= (DataChecksumsWorkerDatabase=20*)=20= palloc0(sizeof(DataChecksumsWorkerDatabase));=0A+=0A+=09=09db->dboid=20=3D= =20pgdb->oid;=0A+=09=09db->dbname=20=3D=20= pstrdup(NameStr(pgdb->datname));=0A+=0A+=09=09DatabaseList=20=3D=20= lappend(DatabaseList,=20db);=0A+=0A+=09=09MemoryContextSwitchTo(oldctx);=0A= +=09}=0A+=0A+=09table_endscan(scan);=0A+=09table_close(rel,=20= AccessShareLock);=0A+=0A+=09CommitTransactionCommand();=0A+=0A+=09return=20= DatabaseList;=0A+}=0A+=0A+static=20void=0A+FreeDatabaseList(List=20= *dblist)=0A+{=0A+=09if=20(!dblist)=0A+=09=09return;=0A+=0A+=09= foreach_ptr(DataChecksumsWorkerDatabase,=20db,=20dblist)=0A+=09{=0A+=09=09= if=20(db->dbname=20!=3D=20NULL)=0A+=09=09=09pfree(db->dbname);=0A+=09}=0A= +=0A+=09list_free_deep(dblist);=0A+}=0A+=0A+/*=0A+=20*=20= BuildRelationList=0A+=20*=09=09Compile=20a=20list=20of=20relations=20in=20= the=20database=0A+=20*=0A+=20*=20Returns=20a=20list=20of=20OIDs=20for=20= the=20request=20relation=20types.=20If=20temp_relations=0A+=20*=20is=20= True=20then=20only=20temporary=20relations=20are=20returned.=20If=20= temp_relations=20is=0A+=20*=20False=20then=20non-temporary=20relations=20= which=20have=20data=20checksums=20are=20returned.=0A+=20*=20If=20= include_shared=20is=20True=20then=20shared=20relations=20are=20included=20= as=20well=20in=20a=0A+=20*=20non-temporary=20list.=20include_shared=20= has=20no=20relevance=20when=20building=20a=20list=20of=0A+=20*=20= temporary=20relations.=0A+=20*/=0A+static=20List=20*=0A= +BuildRelationList(bool=20temp_relations,=20bool=20include_shared)=0A+{=0A= +=09List=09=20=20=20*RelationList=20=3D=20NIL;=0A+=09Relation=09rel;=0A+=09= TableScanDesc=20scan;=0A+=09HeapTuple=09tup;=0A+=09MemoryContext=20ctx=20= =3D=20CurrentMemoryContext;=0A+=09MemoryContext=20oldctx;=0A+=0A+=09= StartTransactionCommand();=0A+=0A+=09rel=20=3D=20= table_open(RelationRelationId,=20AccessShareLock);=0A+=09scan=20=3D=20= table_beginscan_catalog(rel,=200,=20NULL);=0A+=0A+=09while=20= (HeapTupleIsValid(tup=20=3D=20heap_getnext(scan,=20= ForwardScanDirection)))=0A+=09{=0A+=09=09Form_pg_class=20pgc=20=3D=20= (Form_pg_class)=20GETSTRUCT(tup);=0A+=0A+=09=09/*=0A+=09=09=20*=20Only=20= include=20temporary=20relations=20when=20asked=20for=20a=20temp=20= relation=0A+=09=09=20*=20list.=0A+=09=09=20*/=0A+=09=09if=20= (pgc->relpersistence=20=3D=3D=20RELPERSISTENCE_TEMP)=0A+=09=09{=0A+=09=09= =09if=20(!temp_relations)=0A+=09=09=09=09continue;=0A+=09=09}=0A+=09=09= else=0A+=09=09{=0A+=09=09=09/*=0A+=09=09=09=20*=20If=20we=20are=20only=20= interested=20in=20temp=20relations=20then=20continue=0A+=09=09=09=20*=20= immediately=20as=20the=20current=20relation=20isn't=20a=20temp=20= relation.=0A+=09=09=09=20*/=0A+=09=09=09if=20(temp_relations)=0A+=09=09=09= =09continue;=0A+=0A+=09=09=09if=20(!RELKIND_HAS_STORAGE(pgc->relkind))=0A= +=09=09=09=09continue;=0A+=0A+=09=09=09if=20(pgc->relisshared=20&&=20= !include_shared)=0A+=09=09=09=09continue;=0A+=09=09}=0A+=0A+=09=09oldctx=20= =3D=20MemoryContextSwitchTo(ctx);=0A+=09=09RelationList=20=3D=20= lappend_oid(RelationList,=20pgc->oid);=0A+=09=09= MemoryContextSwitchTo(oldctx);=0A+=09}=0A+=0A+=09table_endscan(scan);=0A= +=09table_close(rel,=20AccessShareLock);=0A+=0A+=09= CommitTransactionCommand();=0A+=0A+=09return=20RelationList;=0A+}=0A+=0A= +/*=0A+=20*=20DataChecksumsWorkerMain=0A+=20*=0A+=20*=20Main=20function=20= for=20enabling=20checksums=20in=20a=20single=20database,=20This=20is=20= the=0A+=20*=20function=20set=20as=20the=20bgw_function_name=20in=20the=20= dynamic=20background=20worker=0A+=20*=20process=20initiated=20for=20each=20= database=20by=20the=20worker=20launcher.=20After=20enabling=0A+=20*=20= data=20checksums=20in=20each=20applicable=20relation=20in=20the=20= database,=20it=20will=20wait=20for=0A+=20*=20all=20temporary=20relations=20= that=20were=20present=20when=20the=20function=20started=20to=0A+=20*=20= disappear=20before=20returning.=20This=20is=20required=20since=20we=20= cannot=20rewrite=0A+=20*=20existing=20temporary=20relations=20with=20= data=20checksums.=0A+=20*/=0A+void=0A+DataChecksumsWorkerMain(Datum=20= arg)=0A+{=0A+=09Oid=09=09=09dboid=20=3D=20DatumGetObjectId(arg);=0A+=09= List=09=20=20=20*RelationList=20=3D=20NIL;=0A+=09List=09=20=20=20= *InitialTempTableList=20=3D=20NIL;=0A+=09BufferAccessStrategy=20= strategy;=0A+=09bool=09=09aborted=20=3D=20false;=0A+=09int64=09=09= rels_done;=0A+=0A+=09operation=20=3D=20ENABLE_DATACHECKSUMS;=0A+=0A+=09= pqsignal(SIGTERM,=20die);=0A+=0A+=09BackgroundWorkerUnblockSignals();=0A= +=0A+=09MyBackendType=20=3D=20B_DATACHECKSUMSWORKER_WORKER;=0A+=09= init_ps_display(NULL);=0A+=0A+=09= BackgroundWorkerInitializeConnectionByOid(dboid,=20InvalidOid,=0A+=09=09=09= =09=09=09=09=09=09=09=09=20=20BGWORKER_BYPASS_ALLOWCONN);=0A+=0A+=09/*=20= worker=20will=20have=20a=20separate=20entry=20in=20= pg_stat_progress_data_checksums=20*/=0A+=09= pgstat_progress_start_command(PROGRESS_COMMAND_DATACHECKSUMS,=0A+=09=09=09= =09=09=09=09=09=20=20InvalidOid);=0A+=0A+=09/*=0A+=09=20*=20Get=20a=20= list=20of=20all=20temp=20tables=20present=20as=20we=20start=20in=20this=20= database.=20We=0A+=09=20*=20need=20to=20wait=20until=20they=20are=20all=20= gone=20until=20we=20are=20done,=20since=20we=20cannot=0A+=09=20*=20= access=20these=20relations=20and=20modify=20them.=0A+=09=20*/=0A+=09= InitialTempTableList=20=3D=20BuildRelationList(true,=20false);=0A+=0A+=09= /*=0A+=09=20*=20Enable=20vacuum=20cost=20delay,=20if=20any.=20=20While=20= this=20process=20isn't=20doing=20any=0A+=09=20*=20vacuuming,=20we=20are=20= re-using=20the=20infrastructure=20that=20vacuum=20cost=20delay=0A+=09=20= *=20provides=20rather=20than=20inventing=20something=20bespoke.=20This=20= is=20an=20internal=0A+=09=20*=20implementation=20detail=20and=20care=20= should=20be=20taken=20to=20avoid=20it=20bleeding=0A+=09=20*=20through=20= to=20the=20user=20to=20avoid=20confusion.=0A+=09=20*/=0A+=09= Assert(DataChecksumsWorkerShmem->operation=20=3D=3D=20= ENABLE_DATACHECKSUMS);=0A+=09VacuumCostDelay=20=3D=20= DataChecksumsWorkerShmem->cost_delay;=0A+=09VacuumCostLimit=20=3D=20= DataChecksumsWorkerShmem->cost_limit;=0A+=09VacuumCostActive=20=3D=20= (VacuumCostDelay=20>=200);=0A+=09VacuumCostBalance=20=3D=200;=0A+=09= VacuumCostPageHit=20=3D=200;=0A+=09VacuumCostPageMiss=20=3D=200;=0A+=09= VacuumCostPageDirty=20=3D=200;=0A+=0A+=09/*=0A+=09=20*=20Create=20and=20= set=20the=20vacuum=20strategy=20as=20our=20buffer=20strategy.=0A+=09=20= */=0A+=09strategy=20=3D=20GetAccessStrategy(BAS_VACUUM);=0A+=0A+=09= RelationList=20=3D=20BuildRelationList(false,=0A+=09=09=09=09=09=09=09=09= =09=20DataChecksumsWorkerShmem->process_shared_catalogs);=0A+=0A+=09/*=20= Update=20the=20total=20number=20of=20relations=20to=20be=20processed=20= in=20this=20DB.=20*/=0A+=09{=0A+=09=09const=20int=09index[]=20=3D=20{=0A= +=09=09=09PROGRESS_DATACHECKSUMS_RELS_TOTAL,=0A+=09=09=09= PROGRESS_DATACHECKSUMS_RELS_DONE=0A+=09=09};=0A+=0A+=09=09int64=09=09= vals[2];=0A+=0A+=09=09vals[0]=20=3D=20list_length(RelationList);=0A+=09=09= vals[1]=20=3D=200;=0A+=0A+=09=09pgstat_progress_update_multi_param(2,=20= index,=20vals);=0A+=09}=0A+=0A+=09/*=20Process=20the=20relations=20*/=0A= +=09rels_done=20=3D=200;=0A+=09foreach_oid(reloid,=20RelationList)=0A+=09= {=0A+=09=09if=20(!ProcessSingleRelationByOid(reloid,=20strategy))=0A+=09=09= {=0A+=09=09=09aborted=20=3D=20true;=0A+=09=09=09break;=0A+=09=09}=0A+=0A= +=09=09pgstat_progress_update_param(PROGRESS_DATACHECKSUMS_RELS_DONE,=0A= +=09=09=09=09=09=09=09=09=09=20++rels_done);=0A+=09}=0A+=09= list_free(RelationList);=0A+=0A+=09if=20(aborted)=0A+=09{=0A+=09=09= DataChecksumsWorkerShmem->success=20=3D=20DATACHECKSUMSWORKER_ABORTED;=0A= +=09=09ereport(DEBUG1,=0A+=09=09=09=09errmsg("data=20checksum=20= processing=20aborted=20in=20database=20OID=20%u",=0A+=09=09=09=09=09=20=20= =20dboid));=0A+=09=09return;=0A+=09}=0A+=0A+=09/*=20The=20worker=20is=20= about=20to=20wait=20for=20temporary=20tables=20to=20go=20away.=20*/=0A+=09= pgstat_progress_update_param(PROGRESS_DATACHECKSUMS_PHASE,=0A+=09=09=09=09= =09=09=09=09=20PROGRESS_DATACHECKSUMS_PHASE_WAITING_TEMPREL);=0A+=0A+=09= /*=0A+=09=20*=20Wait=20for=20all=20temp=20tables=20that=20existed=20when=20= we=20started=20to=20go=20away.=20This=0A+=09=20*=20is=20necessary=20= since=20we=20cannot=20"reach"=20them=20to=20enable=20checksums.=20Any=20= temp=0A+=09=20*=20tables=20created=20after=20we=20started=20will=20= already=20have=20checksums=20in=20them=0A+=09=20*=20(due=20to=20the=20= "inprogress-on"=20state),=20so=20no=20need=20to=20wait=20for=20those.=0A= +=09=20*/=0A+=09for=20(;;)=0A+=09{=0A+=09=09List=09=20=20=20= *CurrentTempTables;=0A+=09=09int=09=09=09numleft;=0A+=09=09char=09=09= activity[64];=0A+=0A+=09=09CurrentTempTables=20=3D=20= BuildRelationList(true,=20false);=0A+=09=09numleft=20=3D=200;=0A+=09=09= foreach_oid(tmptbloid,=20InitialTempTableList)=0A+=09=09{=0A+=09=09=09if=20= (list_member_oid(CurrentTempTables,=20tmptbloid))=0A+=09=09=09=09= numleft++;=0A+=09=09}=0A+=09=09list_free(CurrentTempTables);=0A+=0A+=09=09= INJECTION_POINT("datachecksumsworker-fake-temptable-wait",=20&numleft);=0A= +=0A+=09=09if=20(numleft=20=3D=3D=200)=0A+=09=09=09break;=0A+=0A+=09=09= /*=0A+=09=09=20*=20At=20least=20one=20temp=20table=20is=20left=20to=20= wait=20for,=20indicate=20in=20pgstat=0A+=09=09=20*=20activity=20and=20= progress=20reporting.=0A+=09=09=20*/=0A+=09=09snprintf(activity,=0A+=09=09= =09=09=20sizeof(activity),=0A+=09=09=09=09=20"Waiting=20for=20%d=20temp=20= tables=20to=20be=20removed",=20numleft);=0A+=09=09= pgstat_report_activity(STATE_RUNNING,=20activity);=0A+=0A+=09=09/*=20= Retry=20every=203=20seconds=20*/=0A+=09=09ResetLatch(MyLatch);=0A+=09=09= (void)=20WaitLatch(MyLatch,=0A+=09=09=09=09=09=09=20WL_LATCH_SET=20|=20= WL_TIMEOUT=20|=20WL_EXIT_ON_PM_DEATH,=0A+=09=09=09=09=09=09=203000,=0A+=09= =09=09=09=09=09=20WAIT_EVENT_CHECKSUM_ENABLE_TEMPTABLE_WAIT);=0A+=0A+=09=09= LWLockAcquire(DataChecksumsWorkerLock,=20LW_EXCLUSIVE);=0A+=09=09aborted=20= =3D=20DataChecksumsWorkerShmem->launch_operation=20!=3D=20operation;=0A+=09= =09LWLockRelease(DataChecksumsWorkerLock);=0A+=0A+=09=09if=20(aborted=20= ||=20abort_requested)=0A+=09=09{=0A+=09=09=09= DataChecksumsWorkerShmem->success=20=3D=20DATACHECKSUMSWORKER_ABORTED;=0A= +=09=09=09ereport(DEBUG1,=0A+=09=09=09=09=09errmsg("data=20checksum=20= processing=20aborted=20in=20database=20OID=20%u",=0A+=09=09=09=09=09=09=20= =20=20dboid));=0A+=09=09=09return;=0A+=09=09}=0A+=09}=0A+=0A+=09= list_free(InitialTempTableList);=0A+=0A+=09/*=20worker=20done=20*/=0A+=09= pgstat_progress_end_command();=0A+=0A+=09= DataChecksumsWorkerShmem->success=20=3D=20= DATACHECKSUMSWORKER_SUCCESSFUL;=0A+}=0Adiff=20--git=20= a/src/backend/postmaster/meson.build=20= b/src/backend/postmaster/meson.build=0Aindex=20e1f70726604..78ae5cc6611=20= 100644=0A---=20a/src/backend/postmaster/meson.build=0A+++=20= b/src/backend/postmaster/meson.build=0A@@=20-6,6=20+6,7=20@@=20= backend_sources=20+=3D=20files(=0A=20=20=20'bgworker.c',=0A=20=20=20= 'bgwriter.c',=0A=20=20=20'checkpointer.c',=0A+=20=20= 'datachecksumsworker.c',=0A=20=20=20'fork_process.c',=0A=20=20=20= 'interrupt.c',=0A=20=20=20'launch_backend.c',=0Adiff=20--git=20= a/src/backend/postmaster/postmaster.c=20= b/src/backend/postmaster/postmaster.c=0Aindex=20d6133bfebc6..c6fa6460b41=20= 100644=0A---=20a/src/backend/postmaster/postmaster.c=0A+++=20= b/src/backend/postmaster/postmaster.c=0A@@=20-2999,6=20+2999,11=20@@=20= PostmasterStateMachine(void)=0A=20=09=09=09=09=09=09=09=09=09B_INVALID,=0A= =20=09=09=09=09=09=09=09=09=09B_STANDALONE_BACKEND);=0A=20=0A+=09=09=09= /*=20also=20add=20checksumming=20processes=20*/=0A+=09=09=09remainMask=20= =3D=20btmask_add(remainMask,=0A+=09=09=09=09=09=09=09=09=09= B_DATACHECKSUMSWORKER_LAUNCHER,=0A+=09=09=09=09=09=09=09=09=09= B_DATACHECKSUMSWORKER_WORKER);=0A+=0A=20=09=09=09/*=20All=20types=20= should=20be=20included=20in=20targetMask=20or=20remainMask=20*/=0A=20=09=09= =09Assert((remainMask.mask=20|=20targetMask.mask)=20=3D=3D=20= BTYPE_MASK_ALL.mask);=0A=20=09=09}=0Adiff=20--git=20= a/src/backend/postmaster/walsummarizer.c=20= b/src/backend/postmaster/walsummarizer.c=0Aindex=20= c3d56c866d3..afef118d1a4=20100644=0A---=20= a/src/backend/postmaster/walsummarizer.c=0A+++=20= b/src/backend/postmaster/walsummarizer.c=0A@@=20-1430,8=20+1430,11=20@@=20= SummarizeXlogRecord(XLogReaderState=20*xlogreader,=20bool=20= *new_fast_forward)=0A=20=0A=20=09if=20(info=20=3D=3D=20= XLOG_CHECKPOINT_REDO)=0A=20=09{=0A+=09=09xl_checkpoint_redo=20xlrec;=0A+=0A= =20=09=09/*=20Payload=20is=20wal_level=20at=20the=20time=20record=20was=20= written.=20*/=0A-=09=09memcpy(&record_wal_level,=20= XLogRecGetData(xlogreader),=20sizeof(int));=0A+=09=09memcpy(&xlrec,=20= XLogRecGetData(xlogreader),=20sizeof(xl_checkpoint_redo));=0A+=09=09= record_wal_level=20=3D=20xlrec.wal_level;=0A=20=09}=0A=20=09else=20if=20= (info=20=3D=3D=20XLOG_CHECKPOINT_SHUTDOWN)=0A=20=09{=0Adiff=20--git=20= a/src/backend/replication/logical/decode.c=20= b/src/backend/replication/logical/decode.c=0Aindex=20= 32af1249610..7e606176d51=20100644=0A---=20= a/src/backend/replication/logical/decode.c=0A+++=20= b/src/backend/replication/logical/decode.c=0A@@=20-181,6=20+181,7=20@@=20= xlog_decode(LogicalDecodingContext=20*ctx,=20XLogRecordBuffer=20*buf)=0A=20= =09=09case=20XLOG_FPW_CHANGE:=0A=20=09=09case=20XLOG_FPI_FOR_HINT:=0A=20=09= =09case=20XLOG_FPI:=0A+=09=09case=20XLOG_CHECKSUMS:=0A=20=09=09case=20= XLOG_OVERWRITE_CONTRECORD:=0A=20=09=09case=20XLOG_CHECKPOINT_REDO:=0A=20=09= =09=09break;=0Adiff=20--git=20a/src/backend/storage/ipc/ipci.c=20= b/src/backend/storage/ipc/ipci.c=0Aindex=201f7e933d500..359dc54d27e=20= 100644=0A---=20a/src/backend/storage/ipc/ipci.c=0A+++=20= b/src/backend/storage/ipc/ipci.c=0A@@=20-31,6=20+31,7=20@@=0A=20#include=20= "postmaster/autovacuum.h"=0A=20#include=20= "postmaster/bgworker_internals.h"=0A=20#include=20= "postmaster/bgwriter.h"=0A+#include=20"postmaster/datachecksumsworker.h"=0A= =20#include=20"postmaster/walsummarizer.h"=0A=20#include=20= "replication/logicallauncher.h"=0A=20#include=20"replication/origin.h"=0A= @@=20-141,6=20+142,7=20@@=20CalculateShmemSize(void)=0A=20=09size=20=3D=20= add_size(size,=20AioShmemSize());=0A=20=09size=20=3D=20add_size(size,=20= WaitLSNShmemSize());=0A=20=09size=20=3D=20add_size(size,=20= LogicalDecodingCtlShmemSize());=0A+=09size=20=3D=20add_size(size,=20= DataChecksumsWorkerShmemSize());=0A=20=0A=20=09/*=20include=20additional=20= requested=20shmem=20from=20preload=20libraries=20*/=0A=20=09size=20=3D=20= add_size(size,=20total_addin_request);=0A@@=20-315,6=20+317,7=20@@=20= CreateOrAttachShmemStructs(void)=0A=20=09PgArchShmemInit();=0A=20=09= ApplyLauncherShmemInit();=0A=20=09SlotSyncShmemInit();=0A+=09= DataChecksumsWorkerShmemInit();=0A=20=0A=20=09/*=0A=20=09=20*=20Set=20up=20= other=20modules=20that=20need=20some=20shared=20memory=20space=0Adiff=20= --git=20a/src/backend/storage/ipc/procsignal.c=20= b/src/backend/storage/ipc/procsignal.c=0Aindex=20= 8e56922dcea..9be31ad7634=20100644=0A---=20= a/src/backend/storage/ipc/procsignal.c=0A+++=20= b/src/backend/storage/ipc/procsignal.c=0A@@=20-18,12=20+18,14=20@@=0A=20= #include=20=0A=20=0A=20#include=20"access/parallel.h"=0A= +#include=20"access/xlog.h"=0A=20#include=20"commands/async.h"=0A=20= #include=20"miscadmin.h"=0A=20#include=20"pgstat.h"=0A=20#include=20= "port/pg_bitutils.h"=0A=20#include=20"replication/logicalworker.h"=0A=20= #include=20"replication/walsender.h"=0A+#include=20"storage/checksum.h"=0A= =20#include=20"storage/condition_variable.h"=0A=20#include=20= "storage/ipc.h"=0A=20#include=20"storage/latch.h"=0A@@=20-579,6=20= +581,18=20@@=20ProcessProcSignalBarrier(void)=0A=20=09=09=09=09=09case=20= PROCSIGNAL_BARRIER_UPDATE_XLOG_LOGICAL_INFO:=0A=20=09=09=09=09=09=09= processed=20=3D=20ProcessBarrierUpdateXLogLogicalInfo();=0A=20=09=09=09=09= =09=09break;=0A+=09=09=09=09=09case=20= PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_ON:=0A+=09=09=09=09=09=09= processed=20=3D=20= AbsorbDataChecksumsBarrier(PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION);=0A+=09= =09=09=09=09=09break;=0A+=09=09=09=09=09case=20= PROCSIGNAL_BARRIER_CHECKSUM_ON:=0A+=09=09=09=09=09=09processed=20=3D=20= AbsorbDataChecksumsBarrier(PG_DATA_CHECKSUM_VERSION);=0A+=09=09=09=09=09=09= break;=0A+=09=09=09=09=09case=20= PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_OFF:=0A+=09=09=09=09=09=09= processed=20=3D=20= AbsorbDataChecksumsBarrier(PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION);=0A+=09= =09=09=09=09=09break;=0A+=09=09=09=09=09case=20= PROCSIGNAL_BARRIER_CHECKSUM_OFF:=0A+=09=09=09=09=09=09processed=20=3D=20= AbsorbDataChecksumsBarrier(PG_DATA_CHECKSUM_OFF);=0A+=09=09=09=09=09=09= break;=0A=20=09=09=09=09}=0A=20=0A=20=09=09=09=09/*=0Adiff=20--git=20= a/src/backend/storage/page/README=20b/src/backend/storage/page/README=0A= index=20e30d7ac59ad..73c36a63908=20100644=0A---=20= a/src/backend/storage/page/README=0A+++=20= b/src/backend/storage/page/README=0A@@=20-10,7=20+10,9=20@@=20= http://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf,=20discussed=0A= =202010/12/22=20on=20-hackers=20list.=0A=20=0A=20Current=20= implementation=20requires=20this=20be=20enabled=20system-wide=20at=20= initdb=20time,=20or=0A-by=20using=20the=20pg_checksums=20tool=20on=20an=20= offline=20cluster.=0A+by=20using=20the=20pg_checksums=20tool=20on=20an=20= offline=20cluster.=20=20Checksums=20can=20also=20be=0A+enabled=20at=20= runtime=20using=20pg_enable_data_checksums(),=20and=20disabled=20by=20= using=0A+pg_disable_data_checksums().=0A=20=0A=20The=20checksum=20is=20= not=20valid=20at=20all=20times=20on=20a=20data=20page!!=0A=20The=20= checksum=20is=20valid=20when=20the=20page=20leaves=20the=20shared=20pool=20= and=20is=20checked=0Adiff=20--git=20a/src/backend/storage/page/bufpage.c=20= b/src/backend/storage/page/bufpage.c=0Aindex=20de85911e3ac..72e1c042cf1=20= 100644=0A---=20a/src/backend/storage/page/bufpage.c=0A+++=20= b/src/backend/storage/page/bufpage.c=0A@@=20-107,7=20+107,8=20@@=20= PageIsVerified(PageData=20*page,=20BlockNumber=20blkno,=20int=20flags,=20= bool=20*checksum_fail=0A=20=09=20*/=0A=20=09if=20(!PageIsNew(page))=0A=20= =09{=0A-=09=09if=20(DataChecksumsEnabled())=0A+=09=09HOLD_INTERRUPTS();=0A= +=09=09if=20(DataChecksumsNeedVerify())=0A=20=09=09{=0A=20=09=09=09= checksum=20=3D=20pg_checksum_page(page,=20blkno);=0A=20=0A@@=20-118,6=20= +119,7=20@@=20PageIsVerified(PageData=20*page,=20BlockNumber=20blkno,=20= int=20flags,=20bool=20*checksum_fail=0A=20=09=09=09=09=09= *checksum_failure_p=20=3D=20true;=0A=20=09=09=09}=0A=20=09=09}=0A+=09=09= RESUME_INTERRUPTS();=0A=20=0A=20=09=09/*=0A=20=09=09=20*=20The=20= following=20checks=20don't=20prove=20the=20header=20is=20correct,=20only=20= that=0A@@=20-151,8=20+153,8=20@@=20PageIsVerified(PageData=20*page,=20= BlockNumber=20blkno,=20int=20flags,=20bool=20*checksum_fail=0A=20=09=09= if=20((flags=20&=20(PIV_LOG_WARNING=20|=20PIV_LOG_LOG))=20!=3D=200)=0A=20= =09=09=09ereport(flags=20&=20PIV_LOG_WARNING=20?=20WARNING=20:=20LOG,=0A=20= =09=09=09=09=09(errcode(ERRCODE_DATA_CORRUPTED),=0A-=09=09=09=09=09=20= errmsg("page=20verification=20failed,=20calculated=20checksum=20%u=20but=20= expected=20%u",=0A-=09=09=09=09=09=09=09checksum,=20p->pd_checksum)));=0A= +=09=09=09=09=09=20errmsg("page=20verification=20failed,=20calculated=20= checksum=20%u=20but=20expected=20%u=20(page=20LSN=20%X/%08X)",=0A+=09=09=09= =09=09=09=09checksum,=20p->pd_checksum,=20= LSN_FORMAT_ARGS(PageXLogRecPtrGet(p->pd_lsn)))));=0A=20=0A=20=09=09if=20= (header_sane=20&&=20(flags=20&=20PIV_IGNORE_CHECKSUM_FAILURE))=0A=20=09=09= =09return=20true;=0A@@=20-1510,9=20+1512,13=20@@=20= PageSetChecksumCopy(Page=20page,=20BlockNumber=20blkno)=0A=20{=0A=20=09= static=20char=20*pageCopy=20=3D=20NULL;=0A=20=0A+=09HOLD_INTERRUPTS();=0A= =20=09/*=20If=20we=20don't=20need=20a=20checksum,=20just=20return=20the=20= passed-in=20data=20*/=0A-=09if=20(PageIsNew(page)=20||=20= !DataChecksumsEnabled())=0A+=09if=20(PageIsNew(page)=20||=20= !DataChecksumsNeedWrite())=0A+=09{=0A+=09=09RESUME_INTERRUPTS();=0A=20=09= =09return=20page;=0A+=09}=0A=20=0A=20=09/*=0A=20=09=20*=20We=20allocate=20= the=20copy=20space=20once=20and=20use=20it=20over=20on=20each=20= subsequent=0A@@=20-1528,6=20+1534,7=20@@=20PageSetChecksumCopy(Page=20= page,=20BlockNumber=20blkno)=0A=20=0A=20=09memcpy(pageCopy,=20page,=20= BLCKSZ);=0A=20=09((PageHeader)=20pageCopy)->pd_checksum=20=3D=20= pg_checksum_page(pageCopy,=20blkno);=0A+=09RESUME_INTERRUPTS();=0A=20=09= return=20pageCopy;=0A=20}=0A=20=0A@@=20-1540,9=20+1547,14=20@@=20= PageSetChecksumCopy(Page=20page,=20BlockNumber=20blkno)=0A=20void=0A=20= PageSetChecksumInplace(Page=20page,=20BlockNumber=20blkno)=0A=20{=0A+=09= HOLD_INTERRUPTS();=0A=20=09/*=20If=20we=20don't=20need=20a=20checksum,=20= just=20return=20*/=0A-=09if=20(PageIsNew(page)=20||=20= !DataChecksumsEnabled())=0A+=09if=20(PageIsNew(page)=20||=20= !DataChecksumsNeedWrite())=0A+=09{=0A+=09=09RESUME_INTERRUPTS();=0A=20=09= =09return;=0A+=09}=0A=20=0A=20=09((PageHeader)=20page)->pd_checksum=20=3D=20= pg_checksum_page(page,=20blkno);=0A+=09RESUME_INTERRUPTS();=0A=20}=0A= diff=20--git=20a/src/backend/utils/activity/pgstat_backend.c=20= b/src/backend/utils/activity/pgstat_backend.c=0Aindex=20= 1350f5f62f1..47d34be1bab=20100644=0A---=20= a/src/backend/utils/activity/pgstat_backend.c=0A+++=20= b/src/backend/utils/activity/pgstat_backend.c=0A@@=20-380,6=20+380,8=20= @@=20pgstat_tracks_backend_bktype(BackendType=20bktype)=0A=20=09=09case=20= B_CHECKPOINTER:=0A=20=09=09case=20B_IO_WORKER:=0A=20=09=09case=20= B_STARTUP:=0A+=09=09case=20B_DATACHECKSUMSWORKER_LAUNCHER:=0A+=09=09case=20= B_DATACHECKSUMSWORKER_WORKER:=0A=20=09=09=09return=20false;=0A=20=0A=20=09= =09case=20B_AUTOVAC_WORKER:=0Adiff=20--git=20= a/src/backend/utils/activity/pgstat_io.c=20= b/src/backend/utils/activity/pgstat_io.c=0Aindex=20= 28de24538dc..2be26e92283=20100644=0A---=20= a/src/backend/utils/activity/pgstat_io.c=0A+++=20= b/src/backend/utils/activity/pgstat_io.c=0A@@=20-362,6=20+362,8=20@@=20= pgstat_tracks_io_bktype(BackendType=20bktype)=0A=20=09=09case=20= B_LOGGER:=0A=20=09=09=09return=20false;=0A=20=0A+=09=09case=20= B_DATACHECKSUMSWORKER_LAUNCHER:=0A+=09=09case=20= B_DATACHECKSUMSWORKER_WORKER:=0A=20=09=09case=20B_AUTOVAC_LAUNCHER:=0A=20= =09=09case=20B_AUTOVAC_WORKER:=0A=20=09=09case=20B_BACKEND:=0Adiff=20= --git=20a/src/backend/utils/activity/wait_event_names.txt=20= b/src/backend/utils/activity/wait_event_names.txt=0Aindex=20= 4aa864fe3c3..084f6a4f6d3=20100644=0A---=20= a/src/backend/utils/activity/wait_event_names.txt=0A+++=20= b/src/backend/utils/activity/wait_event_names.txt=0A@@=20-119,6=20+119,9=20= @@=20CHECKPOINT_DELAY_COMPLETE=09"Waiting=20for=20a=20backend=20that=20= blocks=20a=20checkpoint=20from=20c=0A=20CHECKPOINT_DELAY_START=09= "Waiting=20for=20a=20backend=20that=20blocks=20a=20checkpoint=20from=20= starting."=0A=20CHECKPOINT_DONE=09"Waiting=20for=20a=20checkpoint=20to=20= complete."=0A=20CHECKPOINT_START=09"Waiting=20for=20a=20checkpoint=20to=20= start."=0A+CHECKSUM_ENABLE_STARTCONDITION=09"Waiting=20for=20data=20= checksums=20enabling=20to=20start."=0A+CHECKSUM_ENABLE_FINISHCONDITION=09= "Waiting=20for=20data=20checksums=20to=20be=20enabled."=0A= +CHECKSUM_ENABLE_TEMPTABLE_WAIT=09"Waiting=20for=20temporary=20tables=20= to=20be=20dropped=20for=20data=20checksums=20to=20be=20enabled."=0A=20= EXECUTE_GATHER=09"Waiting=20for=20activity=20from=20a=20child=20process=20= while=20executing=20a=20Gather=20plan=20node."=0A=20= HASH_BATCH_ALLOCATE=09"Waiting=20for=20an=20elected=20Parallel=20Hash=20= participant=20to=20allocate=20a=20hash=20table."=0A=20HASH_BATCH_ELECT=09= "Waiting=20to=20elect=20a=20Parallel=20Hash=20participant=20to=20= allocate=20a=20hash=20table."=0A@@=20-366,6=20+369,7=20@@=20= SerialControl=09"Waiting=20to=20read=20or=20update=20shared=20= pg_serial=20s=0A=20AioWorkerSubmissionQueue=09= "Waiting=20to=20access=20AIO=20worker=20submission=20queue."=0A=20= WaitLSN=09"Waiting=20to=20read=20or=20update=20shared=20Wait-for-LSN=20= state."=0A=20LogicalDecodingControl=09"Waiting=20to=20read=20or=20update=20= logical=20decoding=20status=20information."=0A+DataChecksumsWorker=09= "Waiting=20for=20data=20checksumsworker."=0A=20=0A=20#=0A=20#=20END=20OF=20= PREDEFINED=20LWLOCKS=20(DO=20NOT=20CHANGE=20THIS=20LINE)=0Adiff=20--git=20= a/src/backend/utils/adt/pgstatfuncs.c=20= b/src/backend/utils/adt/pgstatfuncs.c=0Aindex=2073ca0bb0b7f..b9a37af242c=20= 100644=0A---=20a/src/backend/utils/adt/pgstatfuncs.c=0A+++=20= b/src/backend/utils/adt/pgstatfuncs.c=0A@@=20-295,6=20+295,8=20@@=20= pg_stat_get_progress_info(PG_FUNCTION_ARGS)=0A=20=09=09cmdtype=20=3D=20= PROGRESS_COMMAND_BASEBACKUP;=0A=20=09else=20if=20(pg_strcasecmp(cmd,=20= "COPY")=20=3D=3D=200)=0A=20=09=09cmdtype=20=3D=20PROGRESS_COMMAND_COPY;=0A= +=09else=20if=20(pg_strcasecmp(cmd,=20"DATACHECKSUMS")=20=3D=3D=200)=0A+=09= =09cmdtype=20=3D=20PROGRESS_COMMAND_DATACHECKSUMS;=0A=20=09else=0A=20=09=09= ereport(ERROR,=0A=20=09=09=09=09= (errcode(ERRCODE_INVALID_PARAMETER_VALUE),=0A@@=20-1167,9=20+1169,6=20@@=20= pg_stat_get_db_checksum_failures(PG_FUNCTION_ARGS)=0A=20=09int64=09=09= result;=0A=20=09PgStat_StatDBEntry=20*dbentry;=0A=20=0A-=09if=20= (!DataChecksumsEnabled())=0A-=09=09PG_RETURN_NULL();=0A-=0A=20=09if=20= ((dbentry=20=3D=20pgstat_fetch_stat_dbentry(dbid))=20=3D=3D=20NULL)=0A=20= =09=09result=20=3D=200;=0A=20=09else=0A@@=20-1185,9=20+1184,6=20@@=20= pg_stat_get_db_checksum_last_failure(PG_FUNCTION_ARGS)=0A=20=09= TimestampTz=20result;=0A=20=09PgStat_StatDBEntry=20*dbentry;=0A=20=0A-=09= if=20(!DataChecksumsEnabled())=0A-=09=09PG_RETURN_NULL();=0A-=0A=20=09if=20= ((dbentry=20=3D=20pgstat_fetch_stat_dbentry(dbid))=20=3D=3D=20NULL)=0A=20= =09=09result=20=3D=200;=0A=20=09else=0Adiff=20--git=20= a/src/backend/utils/init/miscinit.c=20= b/src/backend/utils/init/miscinit.c=0Aindex=20563f20374ff..4b4804a7ce5=20= 100644=0A---=20a/src/backend/utils/init/miscinit.c=0A+++=20= b/src/backend/utils/init/miscinit.c=0A@@=20-844,7=20+844,8=20@@=20= InitializeSessionUserIdStandalone(void)=0A=20=09=20*=20workers,=20in=20= slot=20sync=20worker=20and=20in=20background=20workers.=0A=20=09=20*/=0A=20= =09Assert(!IsUnderPostmaster=20||=20AmAutoVacuumWorkerProcess()=20||=0A-=09= =09=20=20=20AmLogicalSlotSyncWorkerProcess()=20||=20= AmBackgroundWorkerProcess());=0A+=09=09=20=20=20= AmLogicalSlotSyncWorkerProcess()=20||=20AmBackgroundWorkerProcess()=20||=0A= +=09=09=20=20=20AmDataChecksumsWorkerProcess());=0A=20=0A=20=09/*=20call=20= only=20once=20*/=0A=20=09Assert(!OidIsValid(AuthenticatedUserId));=0A= diff=20--git=20a/src/backend/utils/init/postinit.c=20= b/src/backend/utils/init/postinit.c=0Aindex=203f401faf3de..34113a7e8bb=20= 100644=0A---=20a/src/backend/utils/init/postinit.c=0A+++=20= b/src/backend/utils/init/postinit.c=0A@@=20-749,6=20+749,24=20@@=20= InitPostgres(const=20char=20*in_dbname,=20Oid=20dboid,=0A=20=0A=20=09= ProcSignalInit(MyCancelKey,=20MyCancelKeyLength);=0A=20=0A+=09/*=0A+=09=20= *=20Initialize=20a=20local=20cache=20of=20the=20data_checksum_version,=20= to=20be=20updated=20by=0A+=09=20*=20the=20procsignal-based=20barriers.=0A= +=09=20*=0A+=09=20*=20This=20intentionally=20happens=20after=20= initializing=20the=20procsignal,=20otherwise=0A+=09=20*=20we=20might=20= miss=20a=20state=20change.=20This=20means=20we=20can=20get=20a=20barrier=20= for=20the=0A+=09=20*=20state=20we've=20just=20initialized.=0A+=09=20*=0A= +=09=20*=20The=20postmaster=20(which=20is=20what=20gets=20forked=20into=20= the=20new=20child=20process)=0A+=09=20*=20does=20not=20handle=20= barriers,=20therefore=20it=20may=20not=20have=20the=20current=20value=0A= +=09=20*=20of=20LocalDataChecksumVersion=20value=20(it'll=20have=20the=20= value=20read=20from=20the=0A+=09=20*=20control=20file,=20which=20may=20= be=20arbitrarily=20old).=0A+=09=20*=0A+=09=20*=20NB:=20Even=20if=20the=20= postmaster=20handled=20barriers,=20the=20value=20might=20still=20be=0A+=09= =20*=20stale,=20as=20it=20might=20have=20changed=20after=20this=20= process=20forked.=0A+=09=20*/=0A+=09InitLocalDataChecksumVersion();=0A+=0A= =20=09/*=0A=20=09=20*=20Also=20set=20up=20timeout=20handlers=20needed=20= for=20backend=20operation.=20=20We=20need=0A=20=09=20*=20these=20in=20= every=20case=20except=20bootstrap.=0A@@=20-877,7=20+895,7=20@@=20= InitPostgres(const=20char=20*in_dbname,=20Oid=20dboid,=0A=20=09=09=09=09=09= =20errhint("You=20should=20immediately=20run=20CREATE=20USER=20\"%s\"=20= SUPERUSER;.",=0A=20=09=09=09=09=09=09=09=20username=20!=3D=20NULL=20?=20= username=20:=20"postgres")));=0A=20=09}=0A-=09else=20if=20= (AmBackgroundWorkerProcess())=0A+=09else=20if=20= (AmBackgroundWorkerProcess()=20||=20AmDataChecksumsWorkerProcess())=0A=20= =09{=0A=20=09=09if=20(username=20=3D=3D=20NULL=20&&=20= !OidIsValid(useroid))=0A=20=09=09{=0Adiff=20--git=20= a/src/backend/utils/misc/guc_parameters.dat=20= b/src/backend/utils/misc/guc_parameters.dat=0Aindex=20= f0260e6e412..8fed24f0192=20100644=0A---=20= a/src/backend/utils/misc/guc_parameters.dat=0A+++=20= b/src/backend/utils/misc/guc_parameters.dat=0A@@=20-531,11=20+531,12=20= @@=0A=20=20=20max=20=3D>=20'1.0',=0A=20},=0A=20=0A-{=20name=20=3D>=20= 'data_checksums',=20type=20=3D>=20'bool',=20context=20=3D>=20= 'PGC_INTERNAL',=20group=20=3D>=20'PRESET_OPTIONS',=0A+{=20name=20=3D>=20= 'data_checksums',=20type=20=3D>=20'enum',=20context=20=3D>=20= 'PGC_INTERNAL',=20group=20=3D>=20'PRESET_OPTIONS',=0A=20=20=20short_desc=20= =3D>=20'Shows=20whether=20data=20checksums=20are=20turned=20on=20for=20= this=20cluster.',=0A=20=20=20flags=20=3D>=20'GUC_NOT_IN_SAMPLE=20|=20= GUC_DISALLOW_IN_FILE=20|=20GUC_RUNTIME_COMPUTED',=0A=20=20=20variable=20= =3D>=20'data_checksums',=0A-=20=20boot_val=20=3D>=20'false',=0A+=20=20= boot_val=20=3D>=20'PG_DATA_CHECKSUM_OFF',=0A+=20=20options=20=3D>=20= 'data_checksums_options',=0A=20},=0A=20=0A=20#=20Can't=20be=20set=20by=20= ALTER=20SYSTEM=20as=20it=20can=20lead=20to=20recursive=20definition=0A= diff=20--git=20a/src/backend/utils/misc/guc_tables.c=20= b/src/backend/utils/misc/guc_tables.c=0Aindex=2013c569d8790..b657eab1119=20= 100644=0A---=20a/src/backend/utils/misc/guc_tables.c=0A+++=20= b/src/backend/utils/misc/guc_tables.c=0A@@=20-491,6=20+491,14=20@@=20= static=20const=20struct=20config_enum_entry=20file_copy_method_options[]=20= =3D=20{=0A=20=09{NULL,=200,=20false}=0A=20};=0A=20=0A+static=20const=20= struct=20config_enum_entry=20data_checksums_options[]=20=3D=20{=0A+=09= {"on",=20PG_DATA_CHECKSUM_VERSION,=20true},=0A+=09{"off",=20= PG_DATA_CHECKSUM_OFF,=20true},=0A+=09{"inprogress-on",=20= PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION,=20true},=0A+=09= {"inprogress-off",=20PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION,=20true},=0A= +=09{NULL,=200,=20false}=0A+};=0A+=0A=20/*=0A=20=20*=20Options=20for=20= enum=20values=20stored=20in=20other=20modules=0A=20=20*/=0A@@=20-618,7=20= +626,6=20@@=20static=20int=09shared_memory_size_in_huge_pages;=0A=20= static=20int=09wal_block_size;=0A=20static=20int=09num_os_semaphores;=0A=20= static=20int=09effective_wal_level=20=3D=20WAL_LEVEL_REPLICA;=0A-static=20= bool=20data_checksums;=0A=20static=20bool=20integer_datetimes;=0A=20=0A=20= #ifdef=20USE_ASSERT_CHECKING=0Adiff=20--git=20= a/src/bin/pg_checksums/pg_checksums.c=20= b/src/bin/pg_checksums/pg_checksums.c=0Aindex=20301e256fbb1..2a38f1d688b=20= 100644=0A---=20a/src/bin/pg_checksums/pg_checksums.c=0A+++=20= b/src/bin/pg_checksums/pg_checksums.c=0A@@=20-585,7=20+585,7=20@@=20= main(int=20argc,=20char=20*argv[])=0A=20=09=09ControlFile->state=20!=3D=20= DB_SHUTDOWNED_IN_RECOVERY)=0A=20=09=09pg_fatal("cluster=20must=20be=20= shut=20down");=0A=20=0A-=09if=20(ControlFile->data_checksum_version=20=3D=3D= =200=20&&=0A+=09if=20(ControlFile->data_checksum_version=20!=3D=20= PG_DATA_CHECKSUM_VERSION=20&&=0A=20=09=09mode=20=3D=3D=20PG_MODE_CHECK)=0A= =20=09=09pg_fatal("data=20checksums=20are=20not=20enabled=20in=20= cluster");=0A=20=0A@@=20-593,7=20+593,7=20@@=20main(int=20argc,=20char=20= *argv[])=0A=20=09=09mode=20=3D=3D=20PG_MODE_DISABLE)=0A=20=09=09= pg_fatal("data=20checksums=20are=20already=20disabled=20in=20cluster");=0A= =20=0A-=09if=20(ControlFile->data_checksum_version=20>=200=20&&=0A+=09if=20= (ControlFile->data_checksum_version=20=3D=3D=20PG_DATA_CHECKSUM_VERSION=20= &&=0A=20=09=09mode=20=3D=3D=20PG_MODE_ENABLE)=0A=20=09=09pg_fatal("data=20= checksums=20are=20already=20enabled=20in=20cluster");=0A=20=0Adiff=20= --git=20a/src/bin/pg_controldata/pg_controldata.c=20= b/src/bin/pg_controldata/pg_controldata.c=0Aindex=20= a4060309ae0..47afa82d9c5=20100644=0A---=20= a/src/bin/pg_controldata/pg_controldata.c=0A+++=20= b/src/bin/pg_controldata/pg_controldata.c=0A@@=20-287,6=20+287,8=20@@=20= main(int=20argc,=20char=20*argv[])=0A=20=09=09=20=20=20= ControlFile->checkPointCopy.oldestCommitTsXid);=0A=20=09printf(_("Latest=20= checkpoint's=20newestCommitTsXid:%u\n"),=0A=20=09=09=20=20=20= ControlFile->checkPointCopy.newestCommitTsXid);=0A+=09printf(_("Latest=20= checkpoint's=20data_checksum_version:%u\n"),=0A+=09=09=20=20=20= ControlFile->checkPointCopy.dataChecksumVersion);=0A=20=09printf(_("Time=20= of=20latest=20checkpoint:=20=20=20=20=20=20=20=20=20=20=20=20%s\n"),=0A=20= =09=09=20=20=20ckpttime_str);=0A=20=09printf(_("Fake=20LSN=20counter=20= for=20unlogged=20rels:=20=20=20%X/%08X\n"),=0Adiff=20--git=20= a/src/bin/pg_upgrade/controldata.c=20b/src/bin/pg_upgrade/controldata.c=0A= index=20aa6e8b4de5d..b7fd4b83757=20100644=0A---=20= a/src/bin/pg_upgrade/controldata.c=0A+++=20= b/src/bin/pg_upgrade/controldata.c=0A@@=20-15,6=20+15,7=20@@=0A=20= #include=20"access/xlog_internal.h"=0A=20#include=20"common/string.h"=0A=20= #include=20"pg_upgrade.h"=0A+#include=20"storage/bufpage.h"=0A=20=0A=20=0A= =20/*=0A@@=20-736,6=20+737,14=20@@=20check_control_data(ControlData=20= *oldctrl,=0A=20=09=20*=20check_for_isn_and_int8_passing_mismatch().=0A=20= =09=20*/=0A=20=0A+=09/*=0A+=09=20*=20If=20data=20checksums=20are=20in=20= any=20in-progress=20state=20then=20disallow=20the=0A+=09=20*=20upgrade.=20= The=20user=20should=20either=20let=20the=20process=20finish,=20or=20turn=20= off=0A+=09=20*=20data=20checksums,=20before=20retrying.=0A+=09=20*/=0A+=09= if=20(oldctrl->data_checksum_version=20>=20PG_DATA_CHECKSUM_VERSION)=0A+=09= =09pg_fatal("checksums=20are=20being=20enabled=20in=20the=20old=20= cluster");=0A+=0A=20=09/*=0A=20=09=20*=20We=20might=20eventually=20allow=20= upgrades=20from=20checksum=20to=20no-checksum=0A=20=09=20*=20clusters.=0A= diff=20--git=20a/src/include/access/xlog.h=20b/src/include/access/xlog.h=0A= index=20fdfb572467b..352ce0477d4=20100644=0A---=20= a/src/include/access/xlog.h=0A+++=20b/src/include/access/xlog.h=0A@@=20= -57,6=20+57,7=20@@=20extern=20PGDLLIMPORT=20int=20CommitDelay;=0A=20= extern=20PGDLLIMPORT=20int=20CommitSiblings;=0A=20extern=20PGDLLIMPORT=20= bool=20track_wal_io_timing;=0A=20extern=20PGDLLIMPORT=20int=20= wal_decode_buffer_size;=0A+extern=20PGDLLIMPORT=20int=20data_checksums;=0A= =20=0A=20extern=20PGDLLIMPORT=20int=20CheckPointSegments;=0A=20=0A@@=20= -119,7=20+120,7=20@@=20extern=20PGDLLIMPORT=20bool=20XLogLogicalInfo;=0A=20= =20*=20of=20the=20bits=20make=20it=20to=20disk,=20but=20the=20checksum=20= wouldn't=20match.=20=20Also=20WAL-log=0A=20=20*=20them=20if=20forced=20= by=20wal_log_hints=3Don.=0A=20=20*/=0A-#define=20XLogHintBitIsNeeded()=20= (DataChecksumsEnabled()=20||=20wal_log_hints)=0A+#define=20= XLogHintBitIsNeeded()=20(wal_log_hints=20||=20DataChecksumsNeedWrite())=0A= =20=0A=20/*=20Do=20we=20need=20to=20WAL-log=20information=20required=20= only=20for=20Hot=20Standby=20and=20logical=20replication?=20*/=0A=20= #define=20XLogStandbyInfoActive()=20(wal_level=20>=3D=20= WAL_LEVEL_REPLICA)=0A@@=20-242,7=20+243,16=20@@=20extern=20XLogRecPtr=20= GetXLogWriteRecPtr(void);=0A=20=0A=20extern=20uint64=20= GetSystemIdentifier(void);=0A=20extern=20char=20= *GetMockAuthenticationNonce(void);=0A-extern=20bool=20= DataChecksumsEnabled(void);=0A+extern=20bool=20= DataChecksumsNeedWrite(void);=0A+extern=20bool=20= DataChecksumsNeedVerify(void);=0A+extern=20bool=20= DataChecksumsOnInProgress(void);=0A+extern=20bool=20= DataChecksumsOffInProgress(void);=0A+extern=20void=20= SetDataChecksumsOnInProgress(void);=0A+extern=20void=20= SetDataChecksumsOn(void);=0A+extern=20void=20SetDataChecksumsOff(void);=0A= +extern=20bool=20AbsorbDataChecksumsBarrier(int=20target_state);=0A= +extern=20const=20char=20*show_data_checksums(void);=0A+extern=20void=20= InitLocalDataChecksumVersion(void);=0A=20extern=20bool=20= GetDefaultCharSignedness(void);=0A=20extern=20XLogRecPtr=20= GetFakeLSNForUnloggedRel(void);=0A=20extern=20Size=20= XLOGShmemSize(void);=0Adiff=20--git=20= a/src/include/access/xlog_internal.h=20= b/src/include/access/xlog_internal.h=0Aindex=20fbc5d77ab39..53d1e2bebfd=20= 100644=0A---=20a/src/include/access/xlog_internal.h=0A+++=20= b/src/include/access/xlog_internal.h=0A@@=20-25,6=20+25,7=20@@=0A=20= #include=20"lib/stringinfo.h"=0A=20#include=20"pgtime.h"=0A=20#include=20= "storage/block.h"=0A+#include=20"storage/checksum.h"=0A=20#include=20= "storage/relfilelocator.h"=0A=20=0A=20=0A@@=20-289,6=20+290,12=20@@=20= typedef=20struct=20xl_restore_point=0A=20=09char=09=09= rp_name[MAXFNAMELEN];=0A=20}=20xl_restore_point;=0A=20=0A+/*=20= Information=20logged=20when=20data=20checksum=20level=20is=20changed=20= */=0A+typedef=20struct=20xl_checksum_state=0A+{=0A+=09ChecksumType=20= new_checksumtype;=0A+}=20xl_checksum_state;=0A+=0A=20/*=20Overwrite=20of=20= prior=20contrecord=20*/=0A=20typedef=20struct=20xl_overwrite_contrecord=0A= =20{=0A@@=20-305,6=20+312,12=20@@=20typedef=20struct=20= xl_end_of_recovery=0A=20=09int=09=09=09wal_level;=0A=20}=20= xl_end_of_recovery;=0A=20=0A+typedef=20struct=20xl_checkpoint_redo=0A+{=0A= +=09int=09=09=09wal_level;=0A+=09uint32=09=09data_checksum_version;=0A+}=20= xl_checkpoint_redo;=0A+=0A=20/*=0A=20=20*=20The=20functions=20in=20= xloginsert.c=20construct=20a=20chain=20of=20XLogRecData=20structs=0A=20=20= *=20to=20represent=20the=20final=20WAL=20record.=0Adiff=20--git=20= a/src/include/catalog/pg_control.h=20b/src/include/catalog/pg_control.h=0A= index=207503db1af51..7290ece5205=20100644=0A---=20= a/src/include/catalog/pg_control.h=0A+++=20= b/src/include/catalog/pg_control.h=0A@@=20-63,6=20+63,9=20@@=20typedef=20= struct=20CheckPoint=0A=20=09=20*=20set=20to=20InvalidTransactionId.=0A=20= =09=20*/=0A=20=09TransactionId=20oldestActiveXid;=0A+=0A+=09/*=20data=20= checksums=20state=20at=20the=20time=20of=20the=20checkpoint=20=20*/=0A+=09= uint32=09=09dataChecksumVersion;=0A=20}=20CheckPoint;=0A=20=0A=20/*=20= XLOG=20info=20values=20for=20XLOG=20rmgr=20*/=0A@@=20-82,6=20+85,7=20@@=20= typedef=20struct=20CheckPoint=0A=20#define=20XLOG_OVERWRITE_CONTRECORD=09= =090xD0=0A=20#define=20XLOG_CHECKPOINT_REDO=09=09=090xE0=0A=20#define=20= XLOG_LOGICAL_DECODING_STATUS_CHANGE=090xF0=0A+#define=20XLOG_CHECKSUMS=09= =09=09=09=090xC0=0A=20=0A=20=0A=20/*=0A@@=20-223,7=20+227,7=20@@=20= typedef=20struct=20ControlFileData=0A=20=09bool=09=09float8ByVal;=09/*=20= float8,=20int8,=20etc=20pass-by-value?=20*/=0A=20=0A=20=09/*=20Are=20= data=20pages=20protected=20by=20checksums?=20Zero=20if=20no=20checksum=20= version=20*/=0A-=09uint32=09=09data_checksum_version;=0A+=09uint32=09=09= data_checksum_version;=09/*=20persistent=20*/=0A=20=0A=20=09/*=0A=20=09=20= *=20True=20if=20the=20default=20signedness=20of=20char=20is=20"signed"=20= on=20a=20platform=20where=0Adiff=20--git=20= a/src/include/catalog/pg_proc.dat=20b/src/include/catalog/pg_proc.dat=0A= index=205e5e33f64fc..d1e8d3cd1e3=20100644=0A---=20= a/src/include/catalog/pg_proc.dat=0A+++=20= b/src/include/catalog/pg_proc.dat=0A@@=20-12405,6=20+12405,18=20@@=0A=20=20= =20proname=20=3D>=20'jsonb_subscript_handler',=20prorettype=20=3D>=20= 'internal',=0A=20=20=20proargtypes=20=3D>=20'internal',=20prosrc=20=3D>=20= 'jsonb_subscript_handler'=20},=0A=20=0A+#=20data=20checksum=20management=20= functions=0A+{=20oid=20=3D>=20'9258',=0A+=20=20descr=20=3D>=20'disable=20= data=20checksums',=0A+=20=20proname=20=3D>=20= 'pg_disable_data_checksums',=20provolatile=20=3D>=20's',=20prorettype=20= =3D>=20'void',=0A+=20=20proparallel=20=3D>=20'r',=20prosrc=20=3D>=20= 'disable_data_checksums',=20proargtypes=20=3D>=20''=20},=0A+{=20oid=20=3D>= =20'9257',=0A+=20=20descr=20=3D>=20'enable=20data=20checksums',=0A+=20=20= proname=20=3D>=20'pg_enable_data_checksums',=20provolatile=20=3D>=20'v',=20= prorettype=20=3D>=20'void',=0A+=20=20proparallel=20=3D>=20'r',=20= proargtypes=20=3D>=20'int4=20int4',=20proallargtypes=20=3D>=20= '{int4,int4}',=0A+=20=20proargmodes=20=3D>=20'{i,i}',=20proargnames=20=3D>= =20'{cost_delay,cost_limit}',=0A+=20=20prosrc=20=3D>=20= 'enable_data_checksums'=20},=0A+=0A=20#=20collation=20management=20= functions=0A=20{=20oid=20=3D>=20'3445',=20descr=20=3D>=20'import=20= collations=20from=20operating=20system',=0A=20=20=20proname=20=3D>=20= 'pg_import_system_collations',=20procost=20=3D>=20'100',=0Adiff=20--git=20= a/src/include/commands/progress.h=20b/src/include/commands/progress.h=0A= index=20359221dc296..cdaf20a9243=20100644=0A---=20= a/src/include/commands/progress.h=0A+++=20= b/src/include/commands/progress.h=0A@@=20-179,4=20+179,20=20@@=0A=20= #define=20PROGRESS_COPY_TYPE_PIPE=203=0A=20#define=20= PROGRESS_COPY_TYPE_CALLBACK=204=0A=20=0A+/*=20Progress=20parameters=20= for=20PROGRESS_DATACHECKSUMS=20*/=0A+#define=20= PROGRESS_DATACHECKSUMS_PHASE=09=090=0A+#define=20= PROGRESS_DATACHECKSUMS_DBS_TOTAL=091=0A+#define=20= PROGRESS_DATACHECKSUMS_DBS_DONE=09=092=0A+#define=20= PROGRESS_DATACHECKSUMS_RELS_TOTAL=093=0A+#define=20= PROGRESS_DATACHECKSUMS_RELS_DONE=094=0A+#define=20= PROGRESS_DATACHECKSUMS_BLOCKS_TOTAL=095=0A+#define=20= PROGRESS_DATACHECKSUMS_BLOCKS_DONE=096=0A+=0A+/*=20Phases=20of=20= datachecksumsworker=20operation=20*/=0A+#define=20= PROGRESS_DATACHECKSUMS_PHASE_ENABLING=09=09=090=0A+#define=20= PROGRESS_DATACHECKSUMS_PHASE_DISABLING=09=09=091=0A+#define=20= PROGRESS_DATACHECKSUMS_PHASE_WAITING_TEMPREL=092=0A+#define=20= PROGRESS_DATACHECKSUMS_PHASE_WAITING_BARRIER=093=0A+#define=20= PROGRESS_DATACHECKSUMS_PHASE_DONE=09=09=09=094=0A+=0A=20#endif=0Adiff=20= --git=20a/src/include/miscadmin.h=20b/src/include/miscadmin.h=0Aindex=20= db559b39c4d..13d111b0e57=20100644=0A---=20a/src/include/miscadmin.h=0A= +++=20b/src/include/miscadmin.h=0A@@=20-367,6=20+367,9=20@@=20typedef=20= enum=20BackendType=0A=20=09B_WAL_SUMMARIZER,=0A=20=09B_WAL_WRITER,=0A=20=0A= +=09B_DATACHECKSUMSWORKER_LAUNCHER,=0A+=09B_DATACHECKSUMSWORKER_WORKER,=0A= +=0A=20=09/*=0A=20=09=20*=20Logger=20is=20not=20connected=20to=20shared=20= memory=20and=20does=20not=20have=20a=20PGPROC=0A=20=09=20*=20entry.=0A@@=20= -392,6=20+395,9=20@@=20extern=20PGDLLIMPORT=20BackendType=20= MyBackendType;=0A=20#define=20AmWalSummarizerProcess()=09(MyBackendType=20= =3D=3D=20B_WAL_SUMMARIZER)=0A=20#define=20AmWalWriterProcess()=09=09= (MyBackendType=20=3D=3D=20B_WAL_WRITER)=0A=20#define=20= AmIoWorkerProcess()=09=09=09(MyBackendType=20=3D=3D=20B_IO_WORKER)=0A= +#define=20AmDataChecksumsWorkerProcess()=20\=0A+=09(MyBackendType=20=3D=3D= =20B_DATACHECKSUMSWORKER_LAUNCHER=20||=20\=0A+=09=20MyBackendType=20=3D=3D= =20B_DATACHECKSUMSWORKER_WORKER)=0A=20=0A=20#define=20= AmSpecialWorkerProcess()=20\=0A=20=09(AmAutoVacuumLauncherProcess()=20||=20= \=0Adiff=20--git=20a/src/include/postmaster/datachecksumsworker.h=20= b/src/include/postmaster/datachecksumsworker.h=0Anew=20file=20mode=20= 100644=0Aindex=2000000000000..dc4bf24be7c=0A---=20/dev/null=0A+++=20= b/src/include/postmaster/datachecksumsworker.h=0A@@=20-0,0=20+1,50=20@@=0A= +/*-----------------------------------------------------------------------= --=0A+=20*=0A+=20*=20datachecksumsworker.h=0A+=20*=09=20=20header=20file=20= for=20data=20checksum=20helper=20background=20worker=0A+=20*=0A+=20*=0A+=20= *=20Portions=20Copyright=20(c)=201996-2026,=20PostgreSQL=20Global=20= Development=20Group=0A+=20*=20Portions=20Copyright=20(c)=201994,=20= Regents=20of=20the=20University=20of=20California=0A+=20*=0A+=20*=20= src/include/postmaster/datachecksumsworker.h=0A+=20*=0A+=20= *-------------------------------------------------------------------------= =0A+=20*/=0A+#ifndef=20DATACHECKSUMSWORKER_H=0A+#define=20= DATACHECKSUMSWORKER_H=0A+=0A+/*=20Shared=20memory=20*/=0A+extern=20Size=20= DataChecksumsWorkerShmemSize(void);=0A+extern=20void=20= DataChecksumsWorkerShmemInit(void);=0A+=0A+/*=20Possible=20operations=20= the=20Datachecksumsworker=20can=20perform=20*/=0A+typedef=20enum=20= DataChecksumsWorkerOperation=0A+{=0A+=09ENABLE_DATACHECKSUMS,=0A+=09= DISABLE_DATACHECKSUMS,=0A+=09/*=20TODO:=20VERIFY_DATACHECKSUMS,=20*/=0A= +}=20DataChecksumsWorkerOperation;=0A+=0A+/*=0A+=20*=20Possible=20states=20= for=20a=20database=20entry=20which=20has=20been=20processed.=20Exported=0A= +=20*=20here=20since=20we=20want=20to=20be=20able=20to=20reference=20= this=20from=20injection=20point=20tests.=0A+=20*/=0A+typedef=20enum=0A+{=0A= +=09DATACHECKSUMSWORKER_SUCCESSFUL=20=3D=200,=0A+=09= DATACHECKSUMSWORKER_ABORTED,=0A+=09DATACHECKSUMSWORKER_FAILED,=0A+=09= DATACHECKSUMSWORKER_RETRYDB,=0A+}=20DataChecksumsWorkerResult;=0A+=0A+/*=20= Start=20the=20background=20processes=20for=20enabling=20or=20disabling=20= checksums=20*/=0A+void=09=09= StartDataChecksumsWorkerLauncher(DataChecksumsWorkerOperation=20op,=0A+=09= =09=09=09=09=09=09=09=09=09=09=20int=20cost_delay,=0A+=09=09=09=09=09=09=09= =09=09=09=09=20int=20cost_limit);=0A+=0A+/*=20Background=20worker=20= entrypoints=20*/=0A+void=09=09DataChecksumsWorkerLauncherMain(Datum=20= arg);=0A+void=09=09DataChecksumsWorkerMain(Datum=20arg);=0A+=0A+#endif=09= =09=09=09=09=09=09/*=20DATACHECKSUMSWORKER_H=20*/=0Adiff=20--git=20= a/src/include/postmaster/proctypelist.h=20= b/src/include/postmaster/proctypelist.h=0Aindex=20= 0b99eaabfd0..3c9aa1478fd=20100644=0A---=20= a/src/include/postmaster/proctypelist.h=0A+++=20= b/src/include/postmaster/proctypelist.h=0A@@=20-38,6=20+38,8=20@@=20= PG_PROCTYPE(B_BACKEND,=20gettext_noop("client=20backend"),=20= BackendMain,=20true)=0A=20PG_PROCTYPE(B_BG_WORKER,=20= gettext_noop("background=20worker"),=20BackgroundWorkerMain,=20true)=0A=20= PG_PROCTYPE(B_BG_WRITER,=20gettext_noop("background=20writer"),=20= BackgroundWriterMain,=20true)=0A=20PG_PROCTYPE(B_CHECKPOINTER,=20= gettext_noop("checkpointer"),=20CheckpointerMain,=20true)=0A= +PG_PROCTYPE(B_DATACHECKSUMSWORKER_LAUNCHER,=20= gettext_noop("datachecksum=20launcher"),=20NULL,=20false)=0A= +PG_PROCTYPE(B_DATACHECKSUMSWORKER_WORKER,=20gettext_noop("datachecksum=20= worker"),=20NULL,=20false)=0A=20PG_PROCTYPE(B_DEAD_END_BACKEND,=20= gettext_noop("dead-end=20client=20backend"),=20BackendMain,=20true)=0A=20= PG_PROCTYPE(B_INVALID,=20gettext_noop("unrecognized"),=20NULL,=20false)=0A= =20PG_PROCTYPE(B_IO_WORKER,=20gettext_noop("io=20worker"),=20= IoWorkerMain,=20true)=0Adiff=20--git=20a/src/include/storage/bufpage.h=20= b/src/include/storage/bufpage.h=0Aindex=20ae3725b3b81..8a668d14803=20= 100644=0A---=20a/src/include/storage/bufpage.h=0A+++=20= b/src/include/storage/bufpage.h=0A@@=20-16,6=20+16,7=20@@=0A=20=0A=20= #include=20"access/xlogdefs.h"=0A=20#include=20"storage/block.h"=0A= +#include=20"storage/checksum.h"=0A=20#include=20"storage/off.h"=0A=20=0A= =20/*=20GUC=20variable=20*/=0A@@=20-204,7=20+205,6=20@@=20typedef=20= PageHeaderData=20*PageHeader;=0A=20=20*=20handling=20pages.=0A=20=20*/=0A= =20#define=20PG_PAGE_LAYOUT_VERSION=09=094=0A-#define=20= PG_DATA_CHECKSUM_VERSION=091=0A=20=0A=20/*=20= ----------------------------------------------------------------=0A=20=20= *=09=09=09=09=09=09page=20support=20functions=0Adiff=20--git=20= a/src/include/storage/checksum.h=20b/src/include/storage/checksum.h=0A= index=20ff417d5ae3e..b6bbba82949=20100644=0A---=20= a/src/include/storage/checksum.h=0A+++=20= b/src/include/storage/checksum.h=0A@@=20-15,6=20+15,20=20@@=0A=20=0A=20= #include=20"storage/block.h"=0A=20=0A+/*=0A+=20*=20Checksum=20version=20= 0=20is=20used=20for=20when=20data=20checksums=20are=20disabled=20(OFF).=0A= +=20*=20PG_DATA_CHECKSUM_VERSION=20defines=20that=20data=20checksums=20= are=20enabled=20in=20the=0A+=20*=20cluster=20and=20= PG_DATA_CHECKSUM_INPROGRESS_{ON|OFF}_VERSION=20defines=20that=20data=0A+=20= *=20checksums=20are=20either=20currently=20being=20enabled=20or=20= disabled.=0A+=20*/=0A+typedef=20enum=20ChecksumType=0A+{=0A+=09= PG_DATA_CHECKSUM_OFF=20=3D=200,=0A+=09PG_DATA_CHECKSUM_VERSION,=0A+=09= PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION,=0A+=09= PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION=0A+}=20ChecksumType;=0A+=0A=20/*=0A= =20=20*=20Compute=20the=20checksum=20for=20a=20Postgres=20page.=20=20The=20= page=20must=20be=20aligned=20on=20a=0A=20=20*=204-byte=20boundary.=0A= diff=20--git=20a/src/include/storage/lwlocklist.h=20= b/src/include/storage/lwlocklist.h=0Aindex=20e94ebce95b9..0c02f2ae38f=20= 100644=0A---=20a/src/include/storage/lwlocklist.h=0A+++=20= b/src/include/storage/lwlocklist.h=0A@@=20-87,6=20+87,7=20@@=20= PG_LWLOCK(52,=20SerialControl)=0A=20PG_LWLOCK(53,=20= AioWorkerSubmissionQueue)=0A=20PG_LWLOCK(54,=20WaitLSN)=0A=20= PG_LWLOCK(55,=20LogicalDecodingControl)=0A+PG_LWLOCK(56,=20= DataChecksumsWorker)=0A=20=0A=20/*=0A=20=20*=20There=20also=20exist=20= several=20built-in=20LWLock=20tranches.=20=20As=20with=20the=20= predefined=0Adiff=20--git=20a/src/include/storage/proc.h=20= b/src/include/storage/proc.h=0Aindex=2081f1960a635..45b4b312ab9=20100644=0A= ---=20a/src/include/storage/proc.h=0A+++=20b/src/include/storage/proc.h=0A= @@=20-462,12=20+462,11=20@@=20extern=20PGDLLIMPORT=20PGPROC=20= *PreparedXactProcs;=0A=20=20*=20Background=20writer,=20checkpointer,=20= WAL=20writer,=20WAL=20summarizer,=20and=20archiver=0A=20=20*=20run=20= during=20normal=20operation.=20=20Startup=20process=20and=20WAL=20= receiver=20also=20consume=0A=20=20*=202=20slots,=20but=20WAL=20writer=20= is=20launched=20only=20after=20startup=20has=20exited,=20so=20we=0A-=20*=20= only=20need=206=20slots.=0A+=20*=20only=20need=206=20slots=20to=20cover=20= these.=0A=20=20*/=0A=20#define=20MAX_IO_WORKERS=20=20=20=20=20=20=20=20=20= =2032=0A=20#define=20NUM_AUXILIARY_PROCS=09=09(6=20+=20MAX_IO_WORKERS)=0A= =20=0A-=0A=20/*=20configurable=20options=20*/=0A=20extern=20PGDLLIMPORT=20= int=20DeadlockTimeout;=0A=20extern=20PGDLLIMPORT=20int=20= StatementTimeout;=0Adiff=20--git=20a/src/include/storage/procsignal.h=20= b/src/include/storage/procsignal.h=0Aindex=20e52b8eb7697..5430386ee70=20= 100644=0A---=20a/src/include/storage/procsignal.h=0A+++=20= b/src/include/storage/procsignal.h=0A@@=20-56,6=20+56,10=20@@=20typedef=20= enum=0A=20=09PROCSIGNAL_BARRIER_SMGRRELEASE,=20/*=20ask=20smgr=20to=20= close=20files=20*/=0A=20=09PROCSIGNAL_BARRIER_UPDATE_XLOG_LOGICAL_INFO,=09= /*=20ask=20to=20update=0A=20=09=09=09=09=09=09=09=09=09=09=09=09=09=20*=20= XLogLogicalInfo=20*/=0A+=09PROCSIGNAL_BARRIER_CHECKSUM_OFF,=0A+=09= PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_ON,=0A+=09= PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_OFF,=0A+=09= PROCSIGNAL_BARRIER_CHECKSUM_ON,=0A=20}=20ProcSignalBarrierType;=0A=20=0A=20= /*=0Adiff=20--git=20a/src/include/utils/backend_progress.h=20= b/src/include/utils/backend_progress.h=0Aindex=20= 19f63b41431..34913667cab=20100644=0A---=20= a/src/include/utils/backend_progress.h=0A+++=20= b/src/include/utils/backend_progress.h=0A@@=20-28,6=20+28,7=20@@=20= typedef=20enum=20ProgressCommandType=0A=20=09= PROGRESS_COMMAND_CREATE_INDEX,=0A=20=09PROGRESS_COMMAND_BASEBACKUP,=0A=20= =09PROGRESS_COMMAND_COPY,=0A+=09PROGRESS_COMMAND_DATACHECKSUMS,=0A=20}=20= ProgressCommandType;=0A=20=0A=20#define=20PGSTAT_NUM_PROGRESS_PARAM=0920=0A= diff=20--git=20a/src/test/modules/Makefile=20b/src/test/modules/Makefile=0A= index=2044c7163c1cd..319b42db4a2=20100644=0A---=20= a/src/test/modules/Makefile=0A+++=20b/src/test/modules/Makefile=0A@@=20= -20,6=20+20,7=20@@=20SUBDIRS=20=3D=20\=0A=20=09=09=20=20test_bitmapset=20= \=0A=20=09=09=20=20test_bloomfilter=20\=0A=20=09=09=20=20test_cloexec=20= \=0A+=09=09=20=20test_checksums=20\=0A=20=09=09=20=20test_copy_callbacks=20= \=0A=20=09=09=20=20test_custom_rmgrs=20\=0A=20=09=09=20=20= test_custom_stats=20\=0Adiff=20--git=20a/src/test/modules/meson.build=20= b/src/test/modules/meson.build=0Aindex=202634a519935..194f635713e=20= 100644=0A---=20a/src/test/modules/meson.build=0A+++=20= b/src/test/modules/meson.build=0A@@=20-20,6=20+20,7=20@@=20= subdir('test_binaryheap')=0A=20subdir('test_bitmapset')=0A=20= subdir('test_bloomfilter')=0A=20subdir('test_cloexec')=0A= +subdir('test_checksums')=0A=20subdir('test_copy_callbacks')=0A=20= subdir('test_cplusplusext')=0A=20subdir('test_custom_rmgrs')=0Adiff=20= --git=20a/src/test/modules/test_checksums/.gitignore=20= b/src/test/modules/test_checksums/.gitignore=0Anew=20file=20mode=20= 100644=0Aindex=2000000000000..871e943d50e=0A---=20/dev/null=0A+++=20= b/src/test/modules/test_checksums/.gitignore=0A@@=20-0,0=20+1,2=20@@=0A= +#=20Generated=20by=20test=20suite=0A+/tmp_check/=0Adiff=20--git=20= a/src/test/modules/test_checksums/Makefile=20= b/src/test/modules/test_checksums/Makefile=0Anew=20file=20mode=20100644=0A= index=2000000000000..fe054cfb88e=0A---=20/dev/null=0A+++=20= b/src/test/modules/test_checksums/Makefile=0A@@=20-0,0=20+1,40=20@@=0A= +#------------------------------------------------------------------------= -=0A+#=0A+#=20Makefile=20for=20src/test/checksum=0A+#=0A+#=20Portions=20= Copyright=20(c)=201996-2026,=20PostgreSQL=20Global=20Development=20Group=0A= +#=20Portions=20Copyright=20(c)=201994,=20Regents=20of=20the=20= University=20of=20California=0A+#=0A+#=20src/test/checksum/Makefile=0A+#=0A= +#------------------------------------------------------------------------= -=0A+=0A+EXTRA_INSTALL=20=3D=20src/test/modules/injection_points=0A+=0A= +export=20enable_injection_points=0A+=0A+MODULE_big=20=3D=20= test_checksums=0A+OBJS=20=3D=20\=0A+=09$(WIN32RES)=20\=0A+=09= test_checksums.o=0A+PGFILEDESC=20=3D=20"test_checksums=20-=20test=20code=20= for=20data=20checksums"=0A+=0A+EXTENSION=20=3D=20test_checksums=0A+DATA=20= =3D=20test_checksums--1.0.sql=0A+=0A+ifdef=20USE_PGXS=0A+PG_CONFIG=20=3D=20= pg_config=0A+PGXS=20:=3D=20$(shell=20$(PG_CONFIG)=20--pgxs)=0A+include=20= $(PGXS)=0A+else=0A+subdir=20=3D=20src/test/modules/test_checksums=0A= +top_builddir=20=3D=20../../../..=0A+include=20= $(top_builddir)/src/Makefile.global=0A+include=20= $(top_srcdir)/contrib/contrib-global.mk=0A+endif=0A+=0A+check:=0A+=09= $(prove_check)=0A+=0A+installcheck:=0A+=09$(prove_installcheck)=0Adiff=20= --git=20a/src/test/modules/test_checksums/README=20= b/src/test/modules/test_checksums/README=0Anew=20file=20mode=20100644=0A= index=2000000000000..0f0317060b3=0A---=20/dev/null=0A+++=20= b/src/test/modules/test_checksums/README=0A@@=20-0,0=20+1,22=20@@=0A= +src/test/checksum/README=0A+=0A+Regression=20tests=20for=20data=20= checksums=0A+=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=0A+=0A+This=20directory=20= contains=20a=20test=20suite=20for=20enabling=20data=20checksums=0A+in=20= a=20running=20cluster.=0A+=0A+Running=20the=20tests=0A+=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=0A+=0A+=20=20=20=20make=20check=0A+=0A+or=0A= +=0A+=20=20=20=20make=20installcheck=0A+=0A+NOTE:=20This=20creates=20a=20= temporary=20installation=20(in=20the=20case=20of=20"check"),=0A+with=20= multiple=20nodes,=20be=20they=20master=20or=20standby(s)=20for=20the=20= purpose=20of=0A+the=20tests.=0A+=0A+NOTE:=20This=20requires=20the=20= --enable-tap-tests=20argument=20to=20configure.=0Adiff=20--git=20= a/src/test/modules/test_checksums/meson.build=20= b/src/test/modules/test_checksums/meson.build=0Anew=20file=20mode=20= 100644=0Aindex=2000000000000..3e6a155f35f=0A---=20/dev/null=0A+++=20= b/src/test/modules/test_checksums/meson.build=0A@@=20-0,0=20+1,36=20@@=0A= +#=20Copyright=20(c)=202026,=20PostgreSQL=20Global=20Development=20Group=0A= +=0A+test_checksums_sources=20=3D=20files(=0A+=09'test_checksums.c',=0A= +)=0A+=0A+test_checksums=20=3D=20shared_module('test_checksums',=0A+=09= test_checksums_sources,=0A+=09kwargs:=20pg_test_mod_args,=0A+)=0A= +test_install_libs=20+=3D=20test_checksums=0A+=0A+test_install_data=20+=3D= =20files(=0A+=09'test_checksums.control',=0A+=09= 'test_checksums--1.0.sql',=0A+)=0A+=0A+tests=20+=3D=20{=0A+=20=20'name':=20= 'test_checksums',=0A+=20=20'sd':=20meson.current_source_dir(),=0A+=20=20= 'bd':=20meson.current_build_dir(),=0A+=20=20'tap':=20{=0A+=20=20=20=20= 'env':=20{=0A+=20=20=20=20=20=20=20'enable_injection_points':=20= get_option('injection_points')=20?=20'yes'=20:=20'no',=0A+=09},=0A+=20=20= =20=20'tests':=20[=0A+=20=20=20=20=20=20't/001_basic.pl',=0A+=20=20=20=20= =20=20't/002_restarts.pl',=0A+=20=20=20=20=20=20= 't/003_standby_restarts.pl',=0A+=20=20=20=20=20=20't/004_offline.pl',=0A= +=20=20=20=20=20=20't/005_injection.pl',=0A+=20=20=20=20=20=20= 't/006_pgbench_single.pl',=0A+=20=20=20=20=20=20= 't/007_pgbench_standby.pl',=0A+=20=20=20=20],=0A+=20=20},=0A+}=0Adiff=20= --git=20a/src/test/modules/test_checksums/t/001_basic.pl=20= b/src/test/modules/test_checksums/t/001_basic.pl=0Anew=20file=20mode=20= 100644=0Aindex=2000000000000..397f6411cb5=0A---=20/dev/null=0A+++=20= b/src/test/modules/test_checksums/t/001_basic.pl=0A@@=20-0,0=20+1,63=20= @@=0A+=0A+#=20Copyright=20(c)=202026,=20PostgreSQL=20Global=20= Development=20Group=0A+=0A+#=20Test=20suite=20for=20testing=20enabling=20= data=20checksums=20in=20an=20online=20cluster=0A+use=20strict;=0A+use=20= warnings=20FATAL=20=3D>=20'all';=0A+=0A+use=20PostgreSQL::Test::Cluster;=0A= +use=20PostgreSQL::Test::Utils;=0A+use=20Test::More;=0A+=0A+use=20= FindBin;=0A+use=20lib=20$FindBin::RealBin;=0A+=0A+use=20= DataChecksums::Utils;=0A+=0A+#=20Initialize=20node=20with=20checksums=20= disabled.=0A+my=20$node=20=3D=20PostgreSQL::Test::Cluster->new('main');=0A= +$node->init(no_data_checksums=20=3D>=201);=0A+$node->start;=0A+=0A+#=20= Create=20some=20content=20to=20have=20un-checksummed=20data=20in=20the=20= cluster=0A+$node->safe_psql('postgres',=0A+=09"CREATE=20TABLE=20t=20AS=20= SELECT=20generate_series(1,10000)=20AS=20a;");=0A+=0A+#=20Ensure=20that=20= checksums=20are=20turned=20off=0A+test_checksum_state($node,=20'off');=0A= +=0A+#=20Enable=20data=20checksums=20and=20wait=20for=20the=20state=20= transition=20to=20'on'=0A+enable_data_checksums($node,=20wait=20=3D>=20= 'on');=0A+=0A+#=20Run=20a=20dummy=20query=20just=20to=20make=20sure=20we=20= can=20read=20back=20data=0A+my=20$result=20=3D=0A+=20=20= $node->safe_psql('postgres',=20"SELECT=20count(*)=20FROM=20t=20WHERE=20a=20= >=201=20");=0A+is($result,=20'9999',=20'ensure=20checksummed=20pages=20= can=20be=20read=20back');=0A+=0A+#=20Enable=20data=20checksums=20again=20= which=20should=20be=20a=20no-op=20so=20we=20explicitly=20don't=0A+#=20= wait=20for=20any=20state=20transition=20as=20none=20should=20happen=20= here=0A+enable_data_checksums($node);=0A+test_checksum_state($node,=20= 'on');=0A+#=20..and=20make=20sure=20we=20can=20still=20read/write=20data=0A= +$node->safe_psql('postgres',=20"UPDATE=20t=20SET=20a=20=3D=20a=20+=20= 1;");=0A+$result=20=3D=20$node->safe_psql('postgres',=20"SELECT=20= count(*)=20FROM=20t=20WHERE=20a=20>=201");=0A+is($result,=20'10000',=20= 'ensure=20checksummed=20pages=20can=20be=20read=20back');=0A+=0A+#=20= Disable=20checksums=20again=20and=20wait=20for=20the=20state=20= transition=0A+disable_data_checksums($node,=20wait=20=3D>=20'on');=0A+=0A= +#=20Test=20reading=20data=20again=0A+$result=20=3D=20= $node->safe_psql('postgres',=20"SELECT=20count(*)=20FROM=20t=20WHERE=20a=20= >=201");=0A+is($result,=20'10000',=20'ensure=20previously=20checksummed=20= pages=20can=20be=20read=20back');=0A+=0A+#=20Re-enable=20checksums=20and=20= make=20sure=20that=20the=20underlying=20data=20has=20changed=20to=0A+#=20= ensure=20that=20checksums=20will=20be=20different.=0A= +$node->safe_psql('postgres',=20"UPDATE=20t=20SET=20a=20=3D=20a=20+=20= 1;");=0A+enable_data_checksums($node,=20wait=20=3D>=20'on');=0A+=0A+#=20= Run=20a=20dummy=20query=20just=20to=20make=20sure=20we=20can=20read=20= back=20the=20data=0A+$result=20=3D=20$node->safe_psql('postgres',=20= "SELECT=20count(*)=20FROM=20t=20WHERE=20a=20>=201");=0A+is($result,=20= '10000',=20'ensure=20checksummed=20pages=20can=20be=20read=20back');=0A+=0A= +$node->stop;=0A+done_testing();=0Adiff=20--git=20= a/src/test/modules/test_checksums/t/002_restarts.pl=20= b/src/test/modules/test_checksums/t/002_restarts.pl=0Anew=20file=20mode=20= 100644=0Aindex=2000000000000..a4d35e9f95c=0A---=20/dev/null=0A+++=20= b/src/test/modules/test_checksums/t/002_restarts.pl=0A@@=20-0,0=20+1,110=20= @@=0A+=0A+#=20Copyright=20(c)=202026,=20PostgreSQL=20Global=20= Development=20Group=0A+=0A+#=20Test=20suite=20for=20testing=20enabling=20= data=20checksums=20in=20an=20online=20cluster=20with=20a=0A+#=20restart=20= which=20breaks=20processing.=0A+use=20strict;=0A+use=20warnings=20FATAL=20= =3D>=20'all';=0A+=0A+use=20PostgreSQL::Test::Cluster;=0A+use=20= PostgreSQL::Test::Utils;=0A+use=20Test::More;=0A+=0A+use=20FindBin;=0A= +use=20lib=20$FindBin::RealBin;=0A+=0A+use=20DataChecksums::Utils;=0A+=0A= +#=20Initialize=20node=20with=20checksums=20disabled.=0A+my=20$node=20=3D=20= PostgreSQL::Test::Cluster->new('main');=0A+$node->init(no_data_checksums=20= =3D>=201);=0A+$node->start;=0A+=0A+#=20Initialize=20result=20storage=20= for=20queries=0A+my=20$result;=0A+=0A+#=20Create=20some=20content=20to=20= have=20un-checksummed=20data=20in=20the=20cluster=0A= +$node->safe_psql('postgres',=0A+=09"CREATE=20TABLE=20t=20AS=20SELECT=20= generate_series(1,10000)=20AS=20a;");=0A+=0A+#=20Ensure=20that=20= checksums=20are=20disabled=0A+test_checksum_state($node,=20'off');=0A+=0A= +SKIP:=0A+{=0A+=09skip=20'Data=20checksum=20delay=20tests=20not=20= enabled=20in=20PG_TEST_EXTRA',=206=0A+=09=20=20if=20= (!$ENV{PG_TEST_EXTRA}=0A+=09=09||=20$ENV{PG_TEST_EXTRA}=20!~=20= /\bchecksum_extended\b/);=0A+=0A+=09#=20Create=20a=20barrier=20for=20= checksumming=20to=20block=20on,=20in=20this=20case=20a=20pre-=0A+=09#=20= existing=20temporary=20table=20which=20is=20kept=20open=20while=20= processing=20is=20started.=0A+=09#=20We=20can=20accomplish=20this=20by=20= setting=20up=20an=20interactive=20psql=20process=20which=0A+=09#=20keeps=20= the=20temporary=20table=20created=20as=20we=20enable=20checksums=20in=20= another=20psql=0A+=09#=20process.=0A+=09#=0A+=09#=20This=20is=20a=20= similar=20test=20to=20the=20synthetic=20variant=20in=20005_injection.pl=0A= +=09#=20which=20fakes=20this=20scenario.=0A+=09my=20$bsession=20=3D=20= $node->background_psql('postgres');=0A+=09$bsession->query_safe('CREATE=20= TEMPORARY=20TABLE=20tt=20(a=20integer);');=0A+=0A+=09#=20In=20another=20= session,=20make=20sure=20we=20can=20see=20the=20blocking=20temp=20table=20= but=0A+=09#=20start=20processing=20anyways=20and=20check=20that=20we=20= are=20blocked=20with=20a=20proper=0A+=09#=20wait=20event.=0A+=09$result=20= =3D=20$node->safe_psql('postgres',=0A+=09=09"SELECT=20relpersistence=20= FROM=20pg_catalog.pg_class=20WHERE=20relname=20=3D=20'tt';"=0A+=09);=0A+=09= is($result,=20't',=20'ensure=20we=20can=20see=20the=20temporary=20= table');=0A+=0A+=09#=20Enabling=20data=20checksums=20shouldn't=20work=20= as=20the=20process=20is=20blocked=20on=20the=0A+=09#=20temporary=20table=20= held=20open=20by=20$bsession.=20Ensure=20that=20we=20reach=20inprogress-=0A= +=09#=20on=20before=20we=20do=20more=20tests.=0A+=09= enable_data_checksums($node,=20wait=20=3D>=20'inprogress-on');=0A+=0A+=09= #=20Wait=20for=20processing=20to=20finish=20and=20the=20worker=20waiting=20= for=20leftover=20temp=0A+=09#=20relations=20to=20be=20able=20to=20= actually=20finish=0A+=09$result=20=3D=20$node->poll_query_until(=0A+=09=09= 'postgres',=0A+=09=09"SELECT=20wait_event=20FROM=20= pg_catalog.pg_stat_activity=20"=0A+=09=09=20=20.=20"WHERE=20backend_type=20= =3D=20'datachecksum=20worker';",=0A+=09=09= 'ChecksumEnableTemptableWait');=0A+=0A+=09#=20The=20datachecksumsworker=20= waits=20for=20temporary=20tables=20to=20disappear=20for=203=0A+=09#=20= seconds=20before=20retrying,=20so=20sleep=20for=204=20seconds=20to=20be=20= guaranteed=20to=20see=0A+=09#=20a=20retry=20cycle=0A+=09sleep(4);=0A+=0A= +=09#=20Re-check=20the=20wait=20event=20to=20ensure=20we=20are=20blocked=20= on=20the=20right=20thing.=0A+=09$result=20=3D=20= $node->safe_psql('postgres',=0A+=09=09=09"SELECT=20wait_event=20FROM=20= pg_catalog.pg_stat_activity=20"=0A+=09=09=20=20.=20"WHERE=20backend_type=20= =3D=20'datachecksum=20worker';");=0A+=09is($result,=20= 'ChecksumEnableTemptableWait',=0A+=09=09'ensure=20the=20correct=20wait=20= condition=20is=20set');=0A+=09test_checksum_state($node,=20= 'inprogress-on');=0A+=0A+=09#=20Stop=20the=20cluster=20while=20bsession=20= is=20still=20attached.=20=20We=20can't=20close=20the=0A+=09#=20session=20= first=20since=20the=20brief=20period=20between=20closing=20and=20= stopping=20might=0A+=09#=20be=20enough=20for=20checksums=20to=20get=20= enabled.=0A+=09$node->stop;=0A+=09$bsession->quit;=0A+=09$node->start;=0A= +=0A+=09#=20Ensure=20the=20checksums=20aren't=20enabled=20across=20the=20= restart.=20=20This=20leaves=20the=0A+=09#=20cluster=20in=20the=20same=20= state=20as=20before=20we=20entered=20the=20SKIP=20block.=0A+=09= test_checksum_state($node,=20'off');=0A+}=0A+=0A= +enable_data_checksums($node,=20wait=20=3D>=20'on');=0A+=0A+$result=20=3D=20= $node->safe_psql('postgres',=20"SELECT=20count(*)=20FROM=20t=20WHERE=20a=20= >=201");=0A+is($result,=20'9999',=20'ensure=20checksummed=20pages=20can=20= be=20read=20back');=0A+=0A+$result=20=3D=20$node->poll_query_until(=0A+=09= 'postgres',=0A+=09"SELECT=20count(*)=20FROM=20pg_stat_activity=20WHERE=20= backend_type=20LIKE=20'datachecksumsworker%';",=0A+=09'0');=0A= +is($result,=201,=20'await=20datachecksums=20worker/launcher=20= termination');=0A+=0A+disable_data_checksums($node,=20wait=20=3D>=201);=0A= +=0A+$node->stop;=0A+done_testing();=0Adiff=20--git=20= a/src/test/modules/test_checksums/t/003_standby_restarts.pl=20= b/src/test/modules/test_checksums/t/003_standby_restarts.pl=0Anew=20file=20= mode=20100644=0Aindex=2000000000000..e9bf21fe5d5=0A---=20/dev/null=0A+++=20= b/src/test/modules/test_checksums/t/003_standby_restarts.pl=0A@@=20-0,0=20= +1,114=20@@=0A+=0A+#=20Copyright=20(c)=202026,=20PostgreSQL=20Global=20= Development=20Group=0A+=0A+#=20Test=20suite=20for=20testing=20enabling=20= data=20checksums=20in=20an=20online=20cluster=20with=0A+#=20streaming=20= replication=0A+use=20strict;=0A+use=20warnings=20FATAL=20=3D>=20'all';=0A= +use=20PostgreSQL::Test::Cluster;=0A+use=20PostgreSQL::Test::Utils;=0A= +use=20Test::More;=0A+=0A+use=20FindBin;=0A+use=20lib=20= $FindBin::RealBin;=0A+=0A+use=20DataChecksums::Utils;=0A+=0A+#=20= Initialize=20primary=20node=0A+my=20$node_primary=20=3D=20= PostgreSQL::Test::Cluster->new('primary');=0A= +$node_primary->init(allows_streaming=20=3D>=201,=20no_data_checksums=20= =3D>=201);=0A+$node_primary->start;=0A+=0A+my=20$slotname=20=3D=20= 'physical_slot';=0A+$node_primary->safe_psql('postgres',=0A+=09"SELECT=20= pg_create_physical_replication_slot('$slotname')");=0A+=0A+#=20Take=20= backup=0A+my=20$backup_name=20=3D=20'my_backup';=0A= +$node_primary->backup($backup_name);=0A+=0A+#=20Create=20streaming=20= standby=20linking=20to=20primary=0A+my=20$node_standby_1=20=3D=20= PostgreSQL::Test::Cluster->new('standby_1');=0A= +$node_standby_1->init_from_backup($node_primary,=20$backup_name,=0A+=09= has_streaming=20=3D>=201);=0A+$node_standby_1->append_conf(=0A+=09= 'postgresql.conf',=20qq[=0A+primary_slot_name=20=3D=20'$slotname'=0A+]);=0A= +$node_standby_1->start;=0A+=0A+#=20Create=20some=20content=20on=20the=20= primary=20to=20have=20un-checksummed=20data=20in=20the=20cluster=0A= +$node_primary->safe_psql('postgres',=0A+=09"CREATE=20TABLE=20t=20AS=20= SELECT=20generate_series(1,10000)=20AS=20a;");=0A+=0A+#=20Wait=20for=20= standbys=20to=20catch=20up=0A= +$node_primary->wait_for_catchup($node_standby_1,=20'replay',=0A+=09= $node_primary->lsn('insert'));=0A+=0A+#=20Check=20that=20checksums=20are=20= turned=20off=20on=20all=20nodes=0A+test_checksum_state($node_primary,=20= 'off');=0A+test_checksum_state($node_standby_1,=20'off');=0A+=0A+#=20= --------------------------------------------------------------------------= -=0A+#=20Enable=20checksums=20for=20the=20cluster,=20and=20make=20sure=20= that=20both=20the=20primary=20and=0A+#=20standby=20change=20state.=0A+#=0A= +=0A+#=20Ensure=20that=20the=20primary=20switches=20to=20"inprogress-on"=0A= +enable_data_checksums($node_primary,=20wait=20=3D>=20'inprogress-on');=0A= +#=20Wait=20for=20checksum=20enable=20to=20be=20replayed=0A= +$node_primary->wait_for_catchup($node_standby_1,=20'replay');=0A+=0A+#=20= Ensure=20that=20the=20standby=20has=20switched=20to=20"inprogress-on"=20= or=20"on".=20=20Normally=20it=0A+#=20would=20be=20"inprogress-on",=20but=20= it=20is=20theoretically=20possible=20for=20the=20primary=20to=0A+#=20= complete=20the=20checksum=20enabling=20*and*=20have=20the=20standby=20= replay=20that=20record=0A+#=20before=20we=20reach=20the=20check=20below.=0A= +my=20$result=20=3D=20$node_standby_1->poll_query_until(=0A+=09= 'postgres',=0A+=09"SELECT=20setting=20=3D=20'off'=20FROM=20= pg_catalog.pg_settings=20WHERE=20name=20=3D=20'data_checksums';",=0A+=09= 'f');=0A+is($result,=201,=20'ensure=20standby=20has=20absorbed=20the=20= inprogress-on=20barrier');=0A+$result=20=3D=20= $node_standby_1->safe_psql('postgres',=0A+=09"SELECT=20setting=20FROM=20= pg_catalog.pg_settings=20WHERE=20name=20=3D=20'data_checksums';"=0A+);=0A= +=0A+is(($result=20eq=20'inprogress-on'=20||=20$result=20eq=20'on'),=0A+=09= 1,=20'ensure=20checksums=20are=20on,=20or=20in=20progress,=20on=20= standby_1');=0A+=0A+#=20Insert=20some=20more=20data=20which=20should=20= be=20checksummed=20on=20INSERT=0A+$node_primary->safe_psql('postgres',=0A= +=09"INSERT=20INTO=20t=20VALUES=20(generate_series(1,=2010000));");=0A+=0A= +#=20Wait=20for=20checksums=20enabled=20on=20the=20primary=20and=20= standby=0A+wait_for_checksum_state($node_primary,=20'on');=0A= +wait_for_checksum_state($node_standby_1,=20'on');=0A+=0A+$result=20=3D=0A= +=20=20$node_primary->safe_psql('postgres',=20"SELECT=20count(a)=20FROM=20= t=20WHERE=20a=20>=201");=0A+is($result,=20'19998',=20'ensure=20we=20can=20= safely=20read=20all=20data=20with=20checksums');=0A+=0A+$result=20=3D=20= $node_primary->poll_query_until(=0A+=09'postgres',=0A+=09"SELECT=20= count(*)=20FROM=20pg_stat_activity=20WHERE=20backend_type=20LIKE=20= 'datachecksumsworker%';",=0A+=09'0');=0A+is($result,=201,=20'await=20= datachecksums=20worker/launcher=20termination');=0A+=0A+#=0A+#=20Disable=20= checksums=20and=20ensure=20it's=20propagated=20to=20standby=20and=20that=20= we=20can=0A+#=20still=20read=20all=20data=0A+#=0A+=0A+#=20Disable=20= checksums=20and=20wait=20for=20the=20operation=20to=20be=20replayed=0A= +disable_data_checksums($node_primary);=0A= +$node_primary->wait_for_catchup($node_standby_1,=20'replay');=0A+#=20= Ensure=20that=20the=20primary=20abd=20standby=20has=20switched=20to=20= off=0A+wait_for_checksum_state($node_primary,=20'off');=0A= +wait_for_checksum_state($node_standby_1,=20'off');=0A+#=20Doublecheck=20= reading=20data=20without=20errors=0A+$result=20=3D=0A+=20=20= $node_primary->safe_psql('postgres',=20"SELECT=20count(a)=20FROM=20t=20= WHERE=20a=20>=201");=0A+is($result,=20"19998",=20'ensure=20we=20can=20= safely=20read=20all=20data=20without=20checksums');=0A+=0A= +$node_standby_1->stop;=0A+$node_primary->stop;=0A+done_testing();=0A= diff=20--git=20a/src/test/modules/test_checksums/t/004_offline.pl=20= b/src/test/modules/test_checksums/t/004_offline.pl=0Anew=20file=20mode=20= 100644=0Aindex=2000000000000..f33e278de28=0A---=20/dev/null=0A+++=20= b/src/test/modules/test_checksums/t/004_offline.pl=0A@@=20-0,0=20+1,82=20= @@=0A+=0A+#=20Copyright=20(c)=202026,=20PostgreSQL=20Global=20= Development=20Group=0A+=0A+#=20Test=20suite=20for=20testing=20enabling=20= data=20checksums=20offline=20from=20various=20states=0A+#=20of=20= checksum=20processing=0A+use=20strict;=0A+use=20warnings=20FATAL=20=3D>=20= 'all';=0A+=0A+use=20PostgreSQL::Test::Cluster;=0A+use=20= PostgreSQL::Test::Utils;=0A+use=20Test::More;=0A+=0A+use=20FindBin;=0A= +use=20lib=20$FindBin::RealBin;=0A+=0A+use=20DataChecksums::Utils;=0A+=0A= +#=20Initialize=20node=20with=20checksums=20disabled.=0A+my=20$node=20=3D=20= PostgreSQL::Test::Cluster->new('main');=0A+$node->init(no_data_checksums=20= =3D>=201);=0A+$node->start;=0A+=0A+#=20Create=20some=20content=20to=20= have=20un-checksummed=20data=20in=20the=20cluster=0A= +$node->safe_psql('postgres',=0A+=09"CREATE=20TABLE=20t=20AS=20SELECT=20= generate_series(1,10000)=20AS=20a;");=0A+=0A+#=20Ensure=20that=20= checksums=20are=20disabled=0A+test_checksum_state($node,=20'off');=0A+=0A= +#=20Enable=20checksums=20offline=20using=20pg_checksums=0A+$node->stop;=0A= +$node->checksum_enable_offline;=0A+$node->start;=0A+=0A+#=20Ensure=20= that=20checksums=20are=20enabled=0A+test_checksum_state($node,=20'on');=0A= +=0A+#=20Run=20a=20dummy=20query=20just=20to=20make=20sure=20we=20can=20= read=20back=20some=20data=0A+my=20$result=20=3D=0A+=20=20= $node->safe_psql('postgres',=20"SELECT=20count(*)=20FROM=20t=20WHERE=20a=20= >=201");=0A+is($result,=20'9999',=20'ensure=20checksummed=20pages=20can=20= be=20read=20back');=0A+=0A+#=20Disable=20checksums=20offline=20again=20= using=20pg_checksums=0A+$node->stop;=0A+$node->checksum_disable_offline;=0A= +$node->start;=0A+=0A+#=20Ensure=20that=20checksums=20are=20disabled=0A= +test_checksum_state($node,=20'off');=0A+=0A+#=20Create=20a=20barrier=20= for=20checksumming=20to=20block=20on,=20in=20this=20case=20a=20= pre-existing=0A+#=20temporary=20table=20which=20is=20kept=20open=20while=20= processing=20is=20started.=20We=20can=0A+#=20accomplish=20this=20by=20= setting=20up=20an=20interactive=20psql=20process=20which=20keeps=20the=0A= +#=20temporary=20table=20created=20as=20we=20enable=20checksums=20in=20= another=20psql=20process.=0A+=0A+my=20$bsession=20=3D=20= $node->background_psql('postgres');=0A+$bsession->query_safe('CREATE=20= TEMPORARY=20TABLE=20tt=20(a=20integer);');=0A+=0A+#=20In=20another=20= session,=20make=20sure=20we=20can=20see=20the=20blocking=20temp=20table=20= but=20start=0A+#=20processing=20anyways=20and=20check=20that=20we=20are=20= blocked=20with=20a=20proper=20wait=20event.=0A+$result=20=3D=20= $node->safe_psql('postgres',=0A+=09"SELECT=20relpersistence=20FROM=20= pg_catalog.pg_class=20WHERE=20relname=20=3D=20'tt';");=0A+is($result,=20= 't',=20'ensure=20we=20can=20see=20the=20temporary=20table');=0A+=0A= +enable_data_checksums($node,=20wait=20=3D>=20'inprogress-on');=0A+=0A+#=20= Turn=20the=20cluster=20off=20and=20enable=20checksums=20offline,=20then=20= start=20back=20up=0A+$bsession->quit;=0A+$node->stop;=0A= +$node->checksum_enable_offline;=0A+$node->start;=0A+=0A+#=20Ensure=20= that=20checksums=20are=20now=20enabled=20even=20though=20processing=20= wasn't=0A+#=20restarted=0A+test_checksum_state($node,=20'on');=0A+=0A+#=20= Run=20a=20dummy=20query=20just=20to=20make=20sure=20we=20can=20read=20= back=20some=20data=0A+$result=20=3D=20$node->safe_psql('postgres',=20= "SELECT=20count(*)=20FROM=20t=20WHERE=20a=20>=201");=0A+is($result,=20= '9999',=20'ensure=20checksummed=20pages=20can=20be=20read=20back');=0A+=0A= +$node->stop;=0A+done_testing();=0Adiff=20--git=20= a/src/test/modules/test_checksums/t/005_injection.pl=20= b/src/test/modules/test_checksums/t/005_injection.pl=0Anew=20file=20mode=20= 100644=0Aindex=2000000000000..6d374793eec=0A---=20/dev/null=0A+++=20= b/src/test/modules/test_checksums/t/005_injection.pl=0A@@=20-0,0=20= +1,109=20@@=0A+=0A+#=20Copyright=20(c)=202026,=20PostgreSQL=20Global=20= Development=20Group=0A+=0A+#=20Test=20suite=20for=20testing=20enabling=20= data=20checksums=20in=20an=20online=20cluster=20with=0A+#=20injection=20= point=20tests=20injecting=20failures=20into=20the=20processing=0A+=0A= +use=20strict;=0A+use=20warnings=20FATAL=20=3D>=20'all';=0A+=0A+use=20= PostgreSQL::Test::Cluster;=0A+use=20PostgreSQL::Test::Utils;=0A+use=20= Test::More;=0A+=0A+use=20FindBin;=0A+use=20lib=20$FindBin::RealBin;=0A+=0A= +use=20DataChecksums::Utils;=0A+=0A+if=20($ENV{enable_injection_points}=20= ne=20'yes')=0A+{=0A+=09plan=20skip_all=20=3D>=20'Injection=20points=20= not=20supported=20by=20this=20build';=0A+}=0A+=0A+#=20= --------------------------------------------------------------------------= -=0A+#=20Test=20cluster=20setup=0A+#=0A+=0A+#=20Initiate=20testcluster=0A= +my=20$node=20=3D=20PostgreSQL::Test::Cluster->new('main');=0A= +$node->init(no_data_checksums=20=3D>=201);=0A+$node->start;=0A+=0A+#=20= Set=20up=20test=20environment=0A+$node->safe_psql('postgres',=20'CREATE=20= EXTENSION=20test_checksums;');=0A+=0A+#=20= --------------------------------------------------------------------------= -=0A+#=20Inducing=20failures=20and=20crashes=20in=20processing=0A+=0A+#=20= Force=20enabling=20checksums=20to=20fail=20by=20marking=20one=20of=20the=20= databases=20as=20having=0A+#=20failed=20in=20processing.=0A= +disable_data_checksums($node,=20wait=20=3D>=201);=0A= +$node->safe_psql('postgres',=20'SELECT=20= dcw_inject_fail_database(true);');=0A+enable_data_checksums($node,=20= wait=20=3D>=20'off');=0A+$node->safe_psql('postgres',=20'SELECT=20= dcw_inject_fail_database(false);');=0A+=0A+#=20Make=20sure=20that=20= disabling=20after=20a=20failure=20works=0A= +disable_data_checksums($node);=0A+test_checksum_state($node,=20'off');=0A= +=0A+#=20= --------------------------------------------------------------------------= -=0A+#=20Timing=20and=20retry=20related=20tests=0A+#=0A+=0A+#=20Force=20= the=20enable=20checksums=20processing=20to=20make=20multiple=20passes=20= by=20removing=0A+#=20one=20database=20from=20the=20list=20in=20the=20= first=20pass.=20=20This=20will=20simulate=20a=20CREATE=0A+#=20DATABASE=20= during=20processing.=20=20Doing=20this=20via=20fault=20injection=20makes=20= the=20test=0A+#=20not=20be=20subject=20to=20exact=20timing.=0A= +$node->safe_psql('postgres',=20'SELECT=20dcw_prune_dblist(true);');=0A= +enable_data_checksums($node,=20wait=20=3D>=20'on');=0A+=0A+SKIP:=0A+{=0A= +=09skip=20'Data=20checksum=20delay=20tests=20not=20enabled=20in=20= PG_TEST_EXTRA',=204=0A+=09=20=20if=20(!$ENV{PG_TEST_EXTRA}=0A+=09=09||=20= $ENV{PG_TEST_EXTRA}=20!~=20/\bchecksum_extended\b/);=0A+=0A+=09#=20= Inject=20a=20delay=20in=20the=20barrier=20for=20enabling=20checksums=0A+=09= disable_data_checksums($node,=20wait=20=3D>=201);=0A+=09= $node->safe_psql('postgres',=20'SELECT=20dcw_inject_delay_barrier();');=0A= +=09enable_data_checksums($node,=20wait=20=3D>=20'on');=0A+=0A+=09#=20= Fake=20the=20existence=20of=20a=20temporary=20table=20at=20the=20start=20= of=20processing,=20which=0A+=09#=20will=20force=20the=20processing=20to=20= wait=20and=20retry=20in=20order=20to=20wait=20for=20it=20to=0A+=09#=20= disappear.=0A+=09disable_data_checksums($node,=20wait=20=3D>=201);=0A+=09= $node->safe_psql('postgres',=20'SELECT=20dcw_fake_temptable(true);');=0A= +=09enable_data_checksums($node,=20wait=20=3D>=20'on');=0A+}=0A+=0A+#=20= Test=20starting=20the=20data=20checksums=20launcher=20from=20two=20= sessions=20concurrently.=20In=0A+#=20order=20reliably=20run=20things=20= concurrently=20we=20use=20background=20psql's=20which=20block=0A+#=20on=20= various=20fault=20injected=20delays.=0A+enable_data_checksums($node,=20= wait=20=3D>=20'on');=0A+=0A+my=20$A=20=3D=20= $node->background_psql('postgres');=0A+$A->query_safe('SELECT=20= dcw_inject_launcher_delay(true);');=0A+$A->query_safe('SELECT=20= pg_enable_data_checksums();');=0A+$node->safe_psql('postgres',=20'SELECT=20= dcw_inject_launcher_delay(false);');=0A+my=20($ret,=20$stdout,=20= $stderr)=20=3D=0A+=20=20$node->psql('postgres',=20'SELECT=20= pg_disable_data_checksums();');=0A+wait_for_checksum_state($node,=20= 'off');=0A+=0A+=0A+my=20$B=20=3D=20$node->background_psql('postgres');=0A= +$B->query_safe('SELECT=20dcw_inject_startup_delay(true);');=0A= +$B->query_safe('SELECT=20pg_enable_data_checksums();');=0A= +$node->safe_psql('postgres',=20'SELECT=20= dcw_inject_startup_delay(false);');=0A+($ret,=20$stdout,=20$stderr)=20=3D=0A= +=20=20$node->psql('postgres',=20'SELECT=20= pg_enable_data_checksums();');=0A+like($stderr,=20qr/data=20checksum=20= processing=20already=20running/,=0A+=09'ERROR=20on=20stderr');=0A= +wait_for_checksum_state($node,=20'on');=0A+=0A+$A->quit;=0A+$B->quit;=0A= +=0A+=0A+$node->stop;=0A+done_testing();=0Adiff=20--git=20= a/src/test/modules/test_checksums/t/006_pgbench_single.pl=20= b/src/test/modules/test_checksums/t/006_pgbench_single.pl=0Anew=20file=20= mode=20100644=0Aindex=2000000000000..6bb0e67c7e1=0A---=20/dev/null=0A+++=20= b/src/test/modules/test_checksums/t/006_pgbench_single.pl=0A@@=20-0,0=20= +1,252=20@@=0A+=0A+#=20Copyright=20(c)=202026,=20PostgreSQL=20Global=20= Development=20Group=0A+=0A+#=20Test=20suite=20for=20testing=20enabling=20= data=20checksums=20in=20an=20online=20cluster=20with=0A+#=20concurrent=20= activity=20via=20pgbench=20runs=0A+=0A+use=20strict;=0A+use=20warnings=20= FATAL=20=3D>=20'all';=0A+=0A+use=20PostgreSQL::Test::Cluster;=0A+use=20= PostgreSQL::Test::Utils;=0A+use=20Test::More;=0A+=0A+use=20FindBin;=0A= +use=20lib=20$FindBin::RealBin;=0A+=0A+use=20DataChecksums::Utils;=0A+=0A= +if=20(!$ENV{PG_TEST_EXTRA}=20||=20$ENV{PG_TEST_EXTRA}=20!~=20= /\bchecksum_extended\b/)=0A+{=0A+=09plan=20skip_all=20=3D>=20'Extended=20= tests=20not=20enabled';=0A+}=0A+=0A+if=20($ENV{enable_injection_points}=20= ne=20'yes')=0A+{=0A+=09plan=20skip_all=20=3D>=20'Injection=20points=20= not=20supported=20by=20this=20build';=0A+}=0A+=0A+my=20$node;=0A+my=20= $node_loglocation=20=3D=200;=0A+=0A+#=20The=20number=20of=20full=20test=20= iterations=20which=20will=20be=20performed.=20The=20exact=20number=0A+#=20= of=20tests=20performed=20and=20the=20wall=20time=20taken=20is=20= non-deterministic=20as=20the=20test=0A+#=20performs=20a=20lot=20of=20= randomized=20actions,=20but=2010=20iterations=20will=20be=20a=20long=20= test=0A+#=20run=20regardless.=0A+my=20$TEST_ITERATIONS=20=3D=2010;=0A+=0A= +#=20Variables=20which=20record=20the=20current=20state=20of=20the=20= cluster=0A+my=20$data_checksum_state=20=3D=20'off';=0A+my=20$pgbench=20=3D= =20undef;=0A+=0A+#=20Start=20a=20pgbench=20run=20in=20the=20background=20= against=20the=20server=20specified=20via=20the=0A+#=20port=20passed=20as=20= parameter.=0A+sub=20background_rw_pgbench=0A+{=0A+=09my=20$port=20=3D=20= shift;=0A+=0A+=09#=20If=20a=20previous=20pgbench=20is=20still=20running,=20= start=20by=20shutting=20it=20down.=0A+=09$pgbench->finish=20if=20= $pgbench;=0A+=0A+=09#=20Randomize=20the=20number=20of=20pgbench=20= clients=20a=20bit=20(range=201-16)=0A+=09my=20$clients=20=3D=201=20+=20= int(rand(15));=0A+=0A+=09my=20@cmd=20=3D=20('pgbench',=20'-p',=20$port,=20= '-T',=20'600',=20'-c',=20$clients);=0A+=0A+=09#=20Randomize=20whether=20= we=20spawn=20connections=20or=20not=0A+=09push(@cmd,=20'-C')=20if=20= (cointoss);=0A+=09#=20Finally=20add=20the=20database=20name=20to=20use=0A= +=09push(@cmd,=20'postgres');=0A+=0A+=09$pgbench=20=3D=20= IPC::Run::start(=0A+=09=09\@cmd,=0A+=09=09'<'=20=3D>=20'/dev/null',=0A+=09= =09'>'=20=3D>=20'/dev/null',=0A+=09=09'2>'=20=3D>=20'/dev/null',=0A+=09=09= IPC::Run::timer($PostgreSQL::Test::Utils::timeout_default));=0A+}=0A+=0A= +#=20Invert=20the=20state=20of=20data=20checksums=20in=20the=20cluster,=20= if=20data=20checksums=20are=20on=0A+#=20then=20disable=20them=20and=20= vice=20versa.=20Also=20performs=20proper=20validation=20of=20the=0A+#=20= before=20and=20after=20state.=0A+sub=20flip_data_checksums=0A+{=0A+=09#=20= First,=20make=20sure=20the=20cluster=20is=20in=20the=20state=20we=20= expect=20it=20to=20be=0A+=09test_checksum_state($node,=20= $data_checksum_state);=0A+=0A+=09if=20($data_checksum_state=20eq=20= 'off')=0A+=09{=0A+=09=09#=20Coin-toss=20to=20see=20if=20we=20are=20= injecting=20a=20retry=20due=20to=20a=20temptable=0A+=09=09= $node->safe_psql('postgres',=20'SELECT=20dcw_fake_temptable();')=0A+=09=09= =20=20if=20cointoss();=0A+=0A+=09=09#=20log=20LSN=20right=20before=20we=20= start=20changing=20checksums=0A+=09=09my=20$result=20=3D=0A+=09=09=20=20= $node->safe_psql('postgres',=20"SELECT=20pg_current_wal_lsn()");=0A+=09=09= note("LSN=20before=20enabling:=20"=20.=20$result=20.=20"\n");=0A+=0A+=09=09= #=20Ensure=20that=20the=20primary=20switches=20to=20"inprogress-on"=0A+=09= =09enable_data_checksums($node,=20wait=20=3D>=20'inprogress-on');=0A+=0A= +=09=09random_sleep();=0A+=0A+=09=09#=20Wait=20for=20checksums=20enabled=20= on=20the=20primary=0A+=09=09wait_for_checksum_state($node,=20'on');=0A+=0A= +=09=09#=20log=20LSN=20right=20after=20the=20primary=20flips=20checksums=20= to=20"on"=0A+=09=09$result=20=3D=20$node->safe_psql('postgres',=20= "SELECT=20pg_current_wal_lsn()");=0A+=09=09note("LSN=20after=20enabling:=20= "=20.=20$result=20.=20"\n");=0A+=0A+=09=09random_sleep();=0A+=0A+=09=09= $node->safe_psql('postgres',=20'SELECT=20dcw_fake_temptable(false);');=0A= +=09=09$data_checksum_state=20=3D=20'on';=0A+=09}=0A+=09elsif=20= ($data_checksum_state=20eq=20'on')=0A+=09{=0A+=09=09random_sleep();=0A+=0A= +=09=09#=20log=20LSN=20right=20before=20we=20start=20changing=20= checksums=0A+=09=09my=20$result=20=3D=0A+=09=09=20=20= $node->safe_psql('postgres',=20"SELECT=20pg_current_wal_lsn()");=0A+=09=09= note("LSN=20before=20disabling:=20"=20.=20$result=20.=20"\n");=0A+=0A+=09= =09disable_data_checksums($node);=0A+=0A+=09=09#=20Wait=20for=20= checksums=20disabled=20on=20the=20primary=0A+=09=09= wait_for_checksum_state($node,=20'off');=0A+=0A+=09=09#=20log=20LSN=20= right=20after=20the=20primary=20flips=20checksums=20to=20"off"=0A+=09=09= $result=20=3D=20$node->safe_psql('postgres',=20"SELECT=20= pg_current_wal_lsn()");=0A+=09=09note("LSN=20after=20disabling:=20"=20.=20= $result=20.=20"\n");=0A+=0A+=09=09random_sleep();=0A+=0A+=09=09= $data_checksum_state=20=3D=20'off';=0A+=09}=0A+=09else=0A+=09{=0A+=09=09= #=20This=20should=20only=20happen=20due=20to=20programmer=20error=20when=20= hacking=20on=20the=0A+=09=09#=20test=20code,=20but=20since=20that=20= might=20pass=20subtly=20we=20error=20out.=0A+=09=09= bail('data_checksum_state=20variable=20has=20invalid=20state:'=0A+=09=09=09= =20=20.=20$data_checksum_state);=0A+=09}=0A+}=0A+=0A+#=20Create=20and=20= start=20a=20cluster=20with=20one=20node=0A+$node=20=3D=20= PostgreSQL::Test::Cluster->new('main');=0A+$node->init(allows_streaming=20= =3D>=201,=20no_data_checksums=20=3D>=201);=0A+#=20max_connections=20need=20= to=20be=20bumped=20in=20order=20to=20accommodate=20for=20pgbench=20= clients=0A+#=20and=20log_statement=20is=20dialled=20down=20since=20it=20= otherwise=20will=20generate=20enormous=0A+#=20amounts=20of=20logging.=20= Page=20verification=20failures=20are=20still=20logged.=0A= +$node->append_conf(=0A+=09'postgresql.conf',=0A+=09qq[=0A= +max_connections=20=3D=20100=0A+log_statement=20=3D=20none=0A+]);=0A= +$node->start;=0A+$node->safe_psql('postgres',=20'CREATE=20EXTENSION=20= test_checksums;');=0A+#=20Create=20some=20content=20to=20have=20= un-checksummed=20data=20in=20the=20cluster=0A= +$node->safe_psql('postgres',=0A+=09"CREATE=20TABLE=20t=20AS=20SELECT=20= generate_series(1,=20100000)=20AS=20a;");=0A+#=20Initialize=20pgbench=0A= +$node->command_ok([=20'pgbench',=20'-i',=20'-s',=20'100',=20'-q',=20= 'postgres'=20]);=0A+#=20Start=20the=20test=20suite=20with=20pgbench=20= running.=0A+background_rw_pgbench($node->port);=0A+=0A+#=20Main=20test=20= suite.=20This=20loop=20will=20start=20a=20pgbench=20run=20on=20the=20= cluster=20and=20while=0A+#=20that's=20running=20flip=20the=20state=20of=20= data=20checksums=20concurrently.=20It=20will=20then=0A+#=20randomly=20= restart=20the=20cluster=20and=20then=20check=20for=0A+#=20the=20desired=20= state.=20=20The=20idea=20behind=20doing=20things=20randomly=20is=20to=20= stress=20out=0A+#=20any=20timing=20related=20issues=20by=20subjecting=20= the=20cluster=20for=20varied=20workloads.=0A+#=20A=20TODO=20is=20to=20= generate=20a=20trace=20such=20that=20any=20test=20failure=20can=20be=20= traced=20to=0A+#=20its=20order=20of=20operations=20for=20debugging.=0A= +for=20(my=20$i=20=3D=200;=20$i=20<=20$TEST_ITERATIONS;=20$i++)=0A+{=0A+=09= note("iteration=20",=20($i=20+=201),=20"=20of=20",=20$TEST_ITERATIONS);=0A= +=0A+=09if=20(!$node->is_alive)=0A+=09{=0A+=09=09#=20Start,=20to=20do=20= recovery,=20and=20stop=0A+=09=09$node->start;=0A+=09=09= $node->stop('fast');=0A+=0A+=09=09#=20Since=20the=20log=20isn't=20being=20= written=20to=20now,=20parse=20the=20log=20and=20check=0A+=09=09#=20for=20= instances=20of=20checksum=20verification=20failures.=0A+=09=09my=20$log=20= =3D=20PostgreSQL::Test::Utils::slurp_file($node->logfile,=0A+=09=09=09= $node_loglocation);=0A+=09=09unlike(=0A+=09=09=09$log,=0A+=09=09=09= qr/page=20verification=20failed/,=0A+=09=09=09"no=20checksum=20= validation=20errors=20in=20primary=20log=20(during=20WAL=20recovery)"=0A= +=09=09);=0A+=09=09$node_loglocation=20=3D=20-s=20$node->logfile;=0A+=0A= +=09=09#=20Randomize=20the=20WAL=20size,=20to=20trigger=20checkpoints=20= less/more=20often=0A+=09=09my=20$sb=20=3D=2064=20+=20int(rand(1024));=0A= +=09=09$node->append_conf('postgresql.conf',=20qq[max_wal_size=20=3D=20= $sb]);=0A+=09=09note("changing=20max_wal_size=20to=20"=20.=20$sb);=0A+=0A= +=09=09$node->start;=0A+=0A+=09=09#=20Start=20a=20pgbench=20in=20the=20= background=20against=20the=20primary=0A+=09=09= background_rw_pgbench($node->port);=0A+=09}=0A+=0A+=09= $node->safe_psql('postgres',=20"UPDATE=20t=20SET=20a=20=3D=20a=20+=20= 1;");=0A+=0A+=09flip_data_checksums();=0A+=09random_sleep();=0A+=09my=20= $result=20=3D=0A+=09=20=20$node->safe_psql('postgres',=20"SELECT=20= count(*)=20FROM=20t=20WHERE=20a=20>=201");=0A+=09is($result,=20'100000',=20= 'ensure=20data=20pages=20can=20be=20read=20back=20on=20primary');=0A+=0A= +=09random_sleep();=0A+=0A+=09#=20Potentially=20powercycle=20the=20node=0A= +=09if=20(cointoss())=0A+=09{=0A+=09=09$node->stop(stopmode());=0A+=0A+=09= =09PostgreSQL::Test::Utils::system_log("pg_controldata",=0A+=09=09=09= $node->data_dir);=0A+=0A+=09=09my=20$log=20=3D=20= PostgreSQL::Test::Utils::slurp_file($node->logfile,=0A+=09=09=09= $node_loglocation);=0A+=09=09unlike(=0A+=09=09=09$log,=0A+=09=09=09= qr/page=20verification=20failed/,=0A+=09=09=09"no=20checksum=20= validation=20errors=20in=20primary=20log=20(outside=20WAL=20recovery)"=0A= +=09=09);=0A+=09=09$node_loglocation=20=3D=20-s=20$node->logfile;=0A+=09= }=0A+=0A+=09random_sleep();=0A+}=0A+=0A+#=20Make=20sure=20the=20node=20= is=20running=0A+if=20(!$node->is_alive)=0A+{=0A+=09$node->start;=0A+}=0A= +=0A+#=20Testrun=20is=20over,=20ensure=20that=20data=20reads=20back=20as=20= expected=20and=20perform=20a=20final=0A+#=20verification=20of=20the=20= data=20checksum=20state.=0A+my=20$result=20=3D=0A+=20=20= $node->safe_psql('postgres',=20"SELECT=20count(*)=20FROM=20t=20WHERE=20a=20= >=201");=0A+is($result,=20'100000',=20'ensure=20data=20pages=20can=20be=20= read=20back=20on=20primary');=0A+test_checksum_state($node,=20= $data_checksum_state);=0A+=0A+#=20Perform=20one=20final=20pass=20over=20= the=20logs=20and=20hunt=20for=20unexpected=20errors=0A+my=20$log=20=3D=0A= +=20=20PostgreSQL::Test::Utils::slurp_file($node->logfile,=20= $node_loglocation);=0A+unlike(=0A+=09$log,=0A+=09qr/page=20verification=20= failed/,=0A+=09"no=20checksum=20validation=20errors=20in=20primary=20= log");=0A+$node_loglocation=20=3D=20-s=20$node->logfile;=0A+=0A= +$node->teardown_node;=0A+=0A+done_testing();=0Adiff=20--git=20= a/src/test/modules/test_checksums/t/007_pgbench_standby.pl=20= b/src/test/modules/test_checksums/t/007_pgbench_standby.pl=0Anew=20file=20= mode=20100644=0Aindex=2000000000000..a7365496853=0A---=20/dev/null=0A+++=20= b/src/test/modules/test_checksums/t/007_pgbench_standby.pl=0A@@=20-0,0=20= +1,376=20@@=0A+=0A+#=20Copyright=20(c)=202026,=20PostgreSQL=20Global=20= Development=20Group=0A+=0A+#=20Test=20suite=20for=20testing=20enabling=20= data=20checksums=20in=20an=20online=20cluster,=0A+#=20comprising=20of=20= a=20primary=20and=20a=20replicated=20standby,=20with=20concurrent=20= activity=0A+#=20via=20pgbench=20runs=0A+=0A+use=20strict;=0A+use=20= warnings=20FATAL=20=3D>=20'all';=0A+=0A+use=20PostgreSQL::Test::Cluster;=0A= +use=20PostgreSQL::Test::Utils;=0A+use=20Test::More;=0A+=0A+use=20= FindBin;=0A+use=20lib=20$FindBin::RealBin;=0A+=0A+use=20= DataChecksums::Utils;=0A+=0A+if=20(!$ENV{PG_TEST_EXTRA}=20||=20= $ENV{PG_TEST_EXTRA}=20!~=20/\bchecksum_extended\b/)=0A+{=0A+=09plan=20= skip_all=20=3D>=20'Extended=20checksum=20tests=20not=20enabled';=0A+}=0A= +=0A+if=20($ENV{enable_injection_points}=20ne=20'yes')=0A+{=0A+=09plan=20= skip_all=20=3D>=20'Injection=20points=20not=20supported=20by=20this=20= build';=0A+}=0A+=0A+my=20$node_primary_slot=20=3D=20'physical_slot';=0A= +my=20$node_primary_backup=20=3D=20'primary_backup';=0A+my=20= $node_primary;=0A+my=20$node_primary_loglocation=20=3D=200;=0A+my=20= $node_standby_1;=0A+my=20$node_standby_1_loglocation=20=3D=200;=0A+=0A+#=20= The=20number=20of=20full=20test=20iterations=20which=20will=20be=20= performed.=20The=20exact=20number=0A+#=20of=20tests=20performed=20and=20= the=20wall=20time=20taken=20is=20non-deterministic=20as=20the=20test=0A= +#=20performs=20a=20lot=20of=20randomized=20actions,=20but=205=20= iterations=20will=20be=20a=20long=20test=0A+#=20run=20regardless.=0A+my=20= $TEST_ITERATIONS=20=3D=205;=0A+=0A+#=20Variables=20which=20record=20the=20= current=20state=20of=20the=20cluster=0A+my=20$data_checksum_state=20=3D=20= 'off';=0A+=0A+my=20$pgbench_primary=20=3D=20undef;=0A+my=20= $pgbench_standby=20=3D=20undef;=0A+=0A+#=20Start=20a=20pgbench=20run=20= in=20the=20background=20against=20the=20server=20specified=20via=20the=0A= +#=20port=20passed=20as=20parameter=0A+sub=20background_pgbench=0A+{=0A+=09= my=20($port,=20$standby)=20=3D=20@_;=0A+=0A+=09#=20Terminate=20any=20= currently=20running=20pgbench=20process=20before=20continuing=0A+=09= $pgbench_primary->finish=20if=20$pgbench_primary;=0A+=0A+=09#=20= Randomize=20the=20number=20of=20pgbench=20clients=20a=20bit=20(range=20= 1-16)=0A+=09my=20$clients=20=3D=201=20+=20int(rand(15));=0A+=0A+=09my=20= @cmd=20=3D=20('pgbench',=20'-p',=20$port,=20'-T',=20'600',=20'-c',=20= $clients);=0A+=09#=20Randomize=20whether=20we=20spawn=20connections=20or=20= not=0A+=09push(@cmd,=20'-C')=20if=20(cointoss());=0A+=09#=20If=20we=20= run=20on=20a=20standby=20it=20needs=20to=20be=20a=20read-only=20= benchmark=0A+=09push(@cmd,=20'-S')=20if=20($standby);=0A+=09#=20Finally=20= add=20the=20database=20name=20to=20use=0A+=09push(@cmd,=20'postgres');=0A= +=0A+=09$pgbench_primary=20=3D=20IPC::Run::start(=0A+=09=09\@cmd,=0A+=09=09= '<'=20=3D>=20'/dev/null',=0A+=09=09'>'=20=3D>=20'/dev/null',=0A+=09=09= '2>'=20=3D>=20'/dev/null',=0A+=09=09= IPC::Run::timer($PostgreSQL::Test::Utils::timeout_default));=0A+}=0A+=0A= +#=20Invert=20the=20state=20of=20data=20checksums=20in=20the=20cluster,=20= if=20data=20checksums=20are=20on=0A+#=20then=20disable=20them=20and=20= vice=20versa.=20Also=20performs=20proper=20validation=20of=20the=0A+#=20= before=20and=20after=20state.=0A+sub=20flip_data_checksums=0A+{=0A+=09#=20= First,=20make=20sure=20the=20cluster=20is=20in=20the=20state=20we=20= expect=20it=20to=20be=0A+=09test_checksum_state($node_primary,=20= $data_checksum_state);=0A+=09test_checksum_state($node_standby_1,=20= $data_checksum_state);=0A+=0A+=09if=20($data_checksum_state=20eq=20= 'off')=0A+=09{=0A+=09=09#=20Coin-toss=20to=20see=20if=20we=20are=20= injecting=20a=20retry=20due=20to=20a=20temptable=0A+=09=09= $node_primary->safe_psql('postgres',=20'SELECT=20dcw_fake_temptable();')=0A= +=09=09=20=20if=20cointoss();=0A+=0A+=09=09#=20log=20LSN=20right=20= before=20we=20start=20changing=20checksums=0A+=09=09my=20$result=20=3D=0A= +=09=09=20=20$node_primary->safe_psql('postgres',=20"SELECT=20= pg_current_wal_lsn()");=0A+=09=09note("LSN=20before=20enabling:=20"=20.=20= $result=20.=20"\n");=0A+=0A+=09=09#=20Ensure=20that=20the=20primary=20= switches=20to=20"inprogress-on"=0A+=09=09= enable_data_checksums($node_primary,=20wait=20=3D>=20'inprogress-on');=0A= +=0A+=09=09random_sleep();=0A+=0A+=09=09#=20Wait=20for=20checksum=20= enable=20to=20be=20replayed=0A+=09=09= $node_primary->wait_for_catchup($node_standby_1,=20'replay');=0A+=0A+=09=09= #=20Ensure=20that=20the=20standby=20has=20switched=20to=20= "inprogress-on"=20or=20"on".=0A+=09=09#=20Normally=20it=20would=20be=20= "inprogress-on",=20but=20it=20is=20theoretically=0A+=09=09#=20possible=20= for=20the=20primary=20to=20complete=20the=20checksum=20enabling=20*and*=20= have=0A+=09=09#=20the=20standby=20replay=20that=20record=20before=20we=20= reach=20the=20check=20below.=0A+=09=09$result=20=3D=20= $node_standby_1->poll_query_until(=0A+=09=09=09'postgres',=0A+=09=09=09= "SELECT=20setting=20=3D=20'off'=20"=0A+=09=09=09=20=20.=20"FROM=20= pg_catalog.pg_settings=20"=0A+=09=09=09=20=20.=20"WHERE=20name=20=3D=20= 'data_checksums';",=0A+=09=09=09'f');=0A+=09=09is($result,=201,=0A+=09=09= =09'ensure=20standby=20has=20absorbed=20the=20inprogress-on=20barrier');=0A= +=09=09$result=20=3D=20$node_standby_1->safe_psql('postgres',=0A+=09=09=09= =09"SELECT=20setting=20"=0A+=09=09=09=20=20.=20"FROM=20= pg_catalog.pg_settings=20"=0A+=09=09=09=20=20.=20"WHERE=20name=20=3D=20= 'data_checksums';");=0A+=0A+=09=09is(($result=20eq=20'inprogress-on'=20= ||=20$result=20eq=20'on'),=0A+=09=09=091,=20'ensure=20checksums=20are=20= on,=20or=20in=20progress,=20on=20standby_1');=0A+=0A+=09=09#=20Wait=20= for=20checksums=20enabled=20on=20the=20primary=20and=20standby=0A+=09=09= wait_for_checksum_state($node_primary,=20'on');=0A+=0A+=09=09#=20log=20= LSN=20right=20after=20the=20primary=20flips=20checksums=20to=20"on"=0A+=09= =09$result=20=3D=0A+=09=09=20=20$node_primary->safe_psql('postgres',=20= "SELECT=20pg_current_wal_lsn()");=0A+=09=09note("LSN=20after=20enabling:=20= "=20.=20$result=20.=20"\n");=0A+=0A+=09=09random_sleep();=0A+=09=09= wait_for_checksum_state($node_standby_1,=20'on');=0A+=0A+=09=09= $node_primary->safe_psql('postgres',=0A+=09=09=09'SELECT=20= dcw_fake_temptable(false);');=0A+=09=09$data_checksum_state=20=3D=20= 'on';=0A+=09}=0A+=09elsif=20($data_checksum_state=20eq=20'on')=0A+=09{=0A= +=09=09random_sleep();=0A+=0A+=09=09#=20log=20LSN=20right=20before=20we=20= start=20changing=20checksums=0A+=09=09my=20$result=20=3D=0A+=09=09=20=20= $node_primary->safe_psql('postgres',=20"SELECT=20pg_current_wal_lsn()");=0A= +=09=09note("LSN=20before=20disabling:=20"=20.=20$result=20.=20"\n");=0A= +=0A+=09=09disable_data_checksums($node_primary);=0A+=09=09= $node_primary->wait_for_catchup($node_standby_1,=20'replay');=0A+=0A+=09=09= #=20Wait=20for=20checksums=20disabled=20on=20the=20primary=20and=20= standby=0A+=09=09wait_for_checksum_state($node_primary,=20'off');=0A+=09=09= wait_for_checksum_state($node_standby_1,=20'off');=0A+=0A+=09=09#=20log=20= LSN=20right=20after=20the=20primary=20flips=20checksums=20to=20"off"=0A+=09= =09$result=20=3D=0A+=09=09=20=20$node_primary->safe_psql('postgres',=20= "SELECT=20pg_current_wal_lsn()");=0A+=09=09note("LSN=20after=20= disabling:=20"=20.=20$result=20.=20"\n");=0A+=0A+=09=09random_sleep();=0A= +=09=09wait_for_checksum_state($node_standby_1,=20'off');=0A+=0A+=09=09= $data_checksum_state=20=3D=20'off';=0A+=09}=0A+=09else=0A+=09{=0A+=09=09= #=20This=20should=20only=20happen=20due=20to=20programmer=20error=20when=20= hacking=20on=20the=0A+=09=09#=20test=20code,=20but=20since=20that=20= might=20pass=20subtly=20we=20error=20out.=0A+=09=09= bail('data_checksum_state=20variable=20has=20invalid=20state:'=0A+=09=09=09= =20=20.=20$data_checksum_state);=0A+=09}=0A+}=0A+=0A+#=20Create=20and=20= start=20a=20cluster=20with=20one=20primary=20and=20one=20standby=20node,=20= and=20ensure=0A+#=20they=20are=20caught=20up=20and=20in=20sync.=0A= +$node_primary=20=3D=20PostgreSQL::Test::Cluster->new('main');=0A= +$node_primary->init(allows_streaming=20=3D>=201,=20no_data_checksums=20= =3D>=201);=0A+#=20max_connections=20need=20to=20be=20bumped=20in=20order=20= to=20accommodate=20for=20pgbench=20clients=0A+#=20and=20log_statement=20= is=20dialled=20down=20since=20it=20otherwise=20will=20generate=20= enormous=0A+#=20amounts=20of=20logging.=20Page=20verification=20failures=20= are=20still=20logged.=0A+$node_primary->append_conf(=0A+=09= 'postgresql.conf',=0A+=09qq[=0A+max_connections=20=3D=2030=0A= +log_statement=20=3D=20none=0A+]);=0A+$node_primary->start;=0A= +$node_primary->safe_psql('postgres',=20'CREATE=20EXTENSION=20= test_checksums;');=0A+#=20Create=20some=20content=20to=20have=20= un-checksummed=20data=20in=20the=20cluster=0A= +$node_primary->safe_psql('postgres',=0A+=09"CREATE=20TABLE=20t=20AS=20= SELECT=20generate_series(1,=20100000)=20AS=20a;");=0A= +$node_primary->safe_psql('postgres',=0A+=09"SELECT=20= pg_create_physical_replication_slot('$node_primary_slot');");=0A= +$node_primary->backup($node_primary_backup);=0A+=0A+$node_standby_1=20=3D= =20PostgreSQL::Test::Cluster->new('standby_1');=0A= +$node_standby_1->init_from_backup($node_primary,=20= $node_primary_backup,=0A+=09has_streaming=20=3D>=201);=0A= +$node_standby_1->append_conf(=0A+=09'postgresql.conf',=20qq[=0A= +primary_slot_name=20=3D=20'$node_primary_slot'=0A+]);=0A= +$node_standby_1->start;=0A+=0A+#=20Initialize=20pgbench=20and=20wait=20= for=20the=20objects=20to=20be=20created=20on=20the=20standby=0A= +$node_primary->command_ok([=20'pgbench',=20'-i',=20'-s',=20'100',=20= '-q',=20'postgres'=20]);=0A= +$node_primary->wait_for_catchup($node_standby_1,=20'replay');=0A+=0A+#=20= Start=20the=20test=20suite=20with=20pgbench=20running=20on=20all=20nodes=0A= +background_pgbench($node_standby_1->port,=201);=0A= +background_pgbench($node_primary->port,=200);=0A+=0A+#=20Main=20test=20= suite.=20This=20loop=20will=20start=20a=20pgbench=20run=20on=20the=20= cluster=20and=20while=0A+#=20that's=20running=20flip=20the=20state=20of=20= data=20checksums=20concurrently.=20It=20will=20then=0A+#=20randomly=20= restart=20the=20cluster=20and=20then=20check=20for=0A+#=20the=20desired=20= state.=20=20The=20idea=20behind=20doing=20things=20randomly=20is=20to=20= stress=20out=0A+#=20any=20timing=20related=20issues=20by=20subjecting=20= the=20cluster=20for=20varied=20workloads.=0A+#=20A=20TODO=20is=20to=20= generate=20a=20trace=20such=20that=20any=20test=20failure=20can=20be=20= traced=20to=0A+#=20its=20order=20of=20operations=20for=20debugging.=0A= +for=20(my=20$i=20=3D=200;=20$i=20<=20$TEST_ITERATIONS;=20$i++)=0A+{=0A+=09= note("iteration=20",=20($i=20+=201),=20"=20of=20",=20$TEST_ITERATIONS);=0A= +=0A+=09if=20(!$node_primary->is_alive)=0A+=09{=0A+=09=09#=20start,=20to=20= do=20recovery,=20and=20stop=0A+=09=09$node_primary->start;=0A+=09=09= $node_primary->stop('fast');=0A+=0A+=09=09#=20Since=20the=20log=20isn't=20= being=20written=20to=20now,=20parse=20the=20log=20and=20check=0A+=09=09#=20= for=20instances=20of=20checksum=20verification=20failures.=0A+=09=09my=20= $log=20=3D=20PostgreSQL::Test::Utils::slurp_file($node_primary->logfile,=0A= +=09=09=09$node_primary_loglocation);=0A+=09=09unlike(=0A+=09=09=09$log,=0A= +=09=09=09qr/page=20verification=20failed/,=0A+=09=09=09"no=20checksum=20= validation=20errors=20in=20primary=20log=20(during=20WAL=20recovery)"=0A= +=09=09);=0A+=09=09$node_primary_loglocation=20=3D=20-s=20= $node_primary->logfile;=0A+=0A+=09=09#=20randomize=20the=20WAL=20size,=20= to=20trigger=20checkpoints=20less/more=20often=0A+=09=09my=20$sb=20=3D=20= 64=20+=20int(rand(960));=0A+=09=09= $node_primary->append_conf('postgresql.conf',=20qq[max_wal_size=20=3D=20= $sb]);=0A+=0A+=09=09note("changing=20primary=20max_wal_size=20to=20"=20.=20= $sb);=0A+=0A+=09=09$node_primary->start;=0A+=0A+=09=09#=20Start=20a=20= pgbench=20in=20the=20background=20against=20the=20primary=0A+=09=09= background_pgbench($node_primary->port,=200);=0A+=09}=0A+=0A+=09if=20= (!$node_standby_1->is_alive)=0A+=09{=0A+=09=09$node_standby_1->start;=0A= +=09=09$node_standby_1->stop('fast');=0A+=0A+=09=09#=20Since=20the=20log=20= isn't=20being=20written=20to=20now,=20parse=20the=20log=20and=20check=0A= +=09=09#=20for=20instances=20of=20checksum=20verification=20failures.=0A= +=09=09my=20$log=20=3D=0A+=09=09=20=20= PostgreSQL::Test::Utils::slurp_file($node_standby_1->logfile,=0A+=09=09=09= $node_standby_1_loglocation);=0A+=09=09unlike(=0A+=09=09=09$log,=0A+=09=09= =09qr/page=20verification=20failed/,=0A+=09=09=09"no=20checksum=20= validation=20errors=20in=20standby_1=20log=20(during=20WAL=20recovery)"=0A= +=09=09);=0A+=09=09$node_standby_1_loglocation=20=3D=20-s=20= $node_standby_1->logfile;=0A+=0A+=09=09#=20randomize=20the=20WAL=20size,=20= to=20trigger=20checkpoints=20less/more=20often=0A+=09=09my=20$sb=20=3D=20= 64=20+=20int(rand(960));=0A+=09=09= $node_standby_1->append_conf('postgresql.conf',=0A+=09=09=09= qq[max_wal_size=20=3D=20$sb]);=0A+=0A+=09=09note("changing=20standby=20= max_wal_size=20to=20"=20.=20$sb);=0A+=0A+=09=09$node_standby_1->start;=0A= +=0A+=09=09#=20Start=20a=20read-only=20pgbench=20in=20the=20background=20= on=20the=20standby=0A+=09=09background_pgbench($node_standby_1->port,=20= 1);=0A+=09}=0A+=0A+=09$node_primary->safe_psql('postgres',=20"UPDATE=20t=20= SET=20a=20=3D=20a=20+=201;");=0A+=09= $node_primary->wait_for_catchup($node_standby_1,=20'write');=0A+=0A+=09= flip_data_checksums();=0A+=09random_sleep();=0A+=09my=20$result=20=3D=20= $node_primary->safe_psql('postgres',=0A+=09=09"SELECT=20count(*)=20FROM=20= t=20WHERE=20a=20>=201");=0A+=09is($result,=20'100000',=20'ensure=20data=20= pages=20can=20be=20read=20back=20on=20primary');=0A+=09random_sleep();=0A= +=0A+=09#=20Potentially=20powercycle=20the=20cluster=20(the=20nodes=20= independently)=0A+=09#=20XXX=20should=20maybe=20try=20stopping=20nodes=20= in=20the=20opposite=20order=20too?=0A+=09if=20(cointoss())=0A+=09{=0A+=09= =09$node_primary->stop(stopmode());=0A+=0A+=09=09#=20print=20the=20= contents=20of=20the=20control=20file=20on=20the=20primary=0A+=09=09= PostgreSQL::Test::Utils::system_log("pg_controldata",=0A+=09=09=09= $node_primary->data_dir);=0A+=0A+=09=09#=20slurp=20the=20file=20after=20= shutdown,=20so=20that=20it=20doesn't=20interfere=20with=20the=20recovery=0A= +=09=09my=20$log=20=3D=20= PostgreSQL::Test::Utils::slurp_file($node_primary->logfile,=0A+=09=09=09= $node_primary_loglocation);=0A+=09=09unlike(=0A+=09=09=09$log,=0A+=09=09=09= qr/page=20verification=20failed/,=0A+=09=09=09"no=20checksum=20= validation=20errors=20in=20primary=20log=20(outside=20WAL=20recovery)"=0A= +=09=09);=0A+=09=09$node_primary_loglocation=20=3D=20-s=20= $node_primary->logfile;=0A+=09}=0A+=0A+=09random_sleep();=0A+=0A+=09if=20= (cointoss())=0A+=09{=0A+=09=09$node_standby_1->stop(stopmode());=0A+=0A+=09= =09#=20print=20the=20contents=20of=20the=20control=20file=20on=20the=20= standby=0A+=09=09PostgreSQL::Test::Utils::system_log("pg_controldata",=0A= +=09=09=09$node_standby_1->data_dir);=0A+=0A+=09=09#=20slurp=20the=20= file=20after=20shutdown,=20so=20that=20it=20doesn't=20interfere=20with=20= the=20recovery=0A+=09=09my=20$log=20=3D=0A+=09=09=20=20= PostgreSQL::Test::Utils::slurp_file($node_standby_1->logfile,=0A+=09=09=09= $node_standby_1_loglocation);=0A+=09=09unlike(=0A+=09=09=09$log,=0A+=09=09= =09qr/page=20verification=20failed/,=0A+=09=09=09"no=20checksum=20= validation=20errors=20in=20standby_1=20log=20(outside=20WAL=20recovery)"=0A= +=09=09);=0A+=09=09$node_standby_1_loglocation=20=3D=20-s=20= $node_standby_1->logfile;=0A+=09}=0A+}=0A+=0A+#=20make=20sure=20the=20= nodes=20are=20running=0A+if=20(!$node_primary->is_alive)=0A+{=0A+=09= $node_primary->start;=0A+}=0A+=0A+if=20(!$node_standby_1->is_alive)=0A+{=0A= +=09$node_standby_1->start;=0A+}=0A+=0A+#=20Testrun=20is=20over,=20= ensure=20that=20data=20reads=20back=20as=20expected=20and=20perform=20a=20= final=0A+#=20verification=20of=20the=20data=20checksum=20state.=0A+my=20= $result=20=3D=0A+=20=20$node_primary->safe_psql('postgres',=20"SELECT=20= count(*)=20FROM=20t=20WHERE=20a=20>=201");=0A+is($result,=20'100000',=20= 'ensure=20data=20pages=20can=20be=20read=20back=20on=20primary');=0A= +test_checksum_state($node_primary,=20$data_checksum_state);=0A= +test_checksum_state($node_standby_1,=20$data_checksum_state);=0A+=0A+#=20= Perform=20one=20final=20pass=20over=20the=20logs=20and=20hunt=20for=20= unexpected=20errors=0A+my=20$log=20=3D=20= PostgreSQL::Test::Utils::slurp_file($node_primary->logfile,=0A+=09= $node_primary_loglocation);=0A+unlike(=0A+=09$log,=0A+=09qr/page=20= verification=20failed/,=0A+=09"no=20checksum=20validation=20errors=20in=20= primary=20log");=0A+$node_primary_loglocation=20=3D=20-s=20= $node_primary->logfile;=0A+$log=20=3D=20= PostgreSQL::Test::Utils::slurp_file($node_standby_1->logfile,=0A+=09= $node_standby_1_loglocation);=0A+unlike(=0A+=09$log,=0A+=09qr/page=20= verification=20failed/,=0A+=09"no=20checksum=20validation=20errors=20in=20= standby_1=20log");=0A+$node_standby_1_loglocation=20=3D=20-s=20= $node_standby_1->logfile;=0A+=0A+$node_standby_1->teardown_node;=0A= +$node_primary->teardown_node;=0A+=0A+done_testing();=0Adiff=20--git=20= a/src/test/modules/test_checksums/t/DataChecksums/Utils.pm=20= b/src/test/modules/test_checksums/t/DataChecksums/Utils.pm=0Anew=20file=20= mode=20100644=0Aindex=2000000000000..d299a1ba01d=0A---=20/dev/null=0A+++=20= b/src/test/modules/test_checksums/t/DataChecksums/Utils.pm=0A@@=20-0,0=20= +1,261=20@@=0A+=0A+#=20Copyright=20(c)=202026,=20PostgreSQL=20Global=20= Development=20Group=0A+=0A+=3Dpod=0A+=0A+=3Dhead1=20NAME=0A+=0A= +DataChecksums::Utils=20-=20Utility=20functions=20for=20testing=20data=20= checksums=20in=20a=20running=20cluster=0A+=0A+=3Dhead1=20SYNOPSIS=0A+=0A= +=20=20use=20PostgreSQL::Test::Cluster;=0A+=20=20use=20= DataChecksums::Utils=20qw(=20..=20);=0A+=0A+=20=20#=20Create,=20and=20= start,=20a=20new=20cluster=0A+=20=20my=20$node=20=3D=20= PostgreSQL::Test::Cluster->new('primary');=0A+=20=20$node->init;=0A+=20=20= $node->start;=0A+=0A+=20=20test_checksum_state($node,=20'off');=0A+=0A+=20= =20enable_data_checksums($node);=0A+=0A+=20=20= wait_for_checksum_state($node,=20'on');=0A+=0A+=0A+=3Dcut=0A+=0A+package=20= DataChecksums::Utils;=0A+=0A+use=20strict;=0A+use=20warnings=20FATAL=20= =3D>=20'all';=0A+use=20Exporter=20'import';=0A+use=20= PostgreSQL::Test::Cluster;=0A+use=20PostgreSQL::Test::Utils;=0A+use=20= Test::More;=0A+=0A+our=20@EXPORT=20=3D=20qw(=0A+=20=20cointoss=0A+=20=20= disable_data_checksums=0A+=20=20enable_data_checksums=0A+=20=20= random_sleep=0A+=20=20stopmode=0A+=20=20test_checksum_state=0A+=20=20= wait_for_checksum_state=0A+=20=20wait_for_cluster_crash=0A+);=0A+=0A= +=3Dpod=0A+=0A+=3Dhead1=20METHODS=0A+=0A+=3Dover=0A+=0A+=3Ditem=20= test_checksum_state(node,=20state)=0A+=0A+Test=20that=20the=20current=20= value=20of=20the=20data=20checksum=20GUC=20in=20the=20server=20running=0A= +at=20B=20matches=20B.=20=20If=20the=20values=20differ,=20a=20= test=20failure=20is=20logged.=0A+Returns=20True=20if=20the=20values=20= match,=20otherwise=20False.=0A+=0A+=3Dcut=0A+=0A+sub=20= test_checksum_state=0A+{=0A+=09my=20($postgresnode,=20$state)=20=3D=20= @_;=0A+=0A+=09my=20$result=20=3D=20$postgresnode->safe_psql('postgres',=0A= +=09=09"SELECT=20setting=20FROM=20pg_catalog.pg_settings=20WHERE=20name=20= =3D=20'data_checksums';"=0A+=09);=0A+=09is($result,=20$state,=20'ensure=20= checksums=20are=20set=20to=20'=20.=20$state);=0A+=09return=20$result=20= eq=20$state;=0A+}=0A+=0A+=3Ditem=20wait_for_checksum_state(node,=20= state)=0A+=0A+Test=20the=20value=20of=20the=20data=20checksum=20GUC=20in=20= the=20server=20running=20at=20B=0A+repeatedly=20until=20it=20= matches=20B=20or=20times=20out.=20=20Processing=20will=20run=20= for=0A+$PostgreSQL::Test::Utils::timeout_default=20seconds=20before=20= timing=20out.=20=20If=20the=0A+values=20differ=20when=20the=20process=20= times=20out,=20False=20is=20returned=20and=20a=20test=20failure=0A+is=20= logged,=20otherwise=20True.=0A+=0A+=3Dcut=0A+=0A+sub=20= wait_for_checksum_state=0A+{=0A+=09my=20($postgresnode,=20$state)=20=3D=20= @_;=0A+=0A+=09my=20$res=20=3D=20$postgresnode->poll_query_until(=0A+=09=09= 'postgres',=0A+=09=09"SELECT=20setting=20FROM=20pg_catalog.pg_settings=20= WHERE=20name=20=3D=20'data_checksums';",=0A+=09=09$state);=0A+=09= is($res,=201,=20'ensure=20data=20checksums=20are=20transitioned=20to=20'=20= .=20$state);=0A+=09return=20$res=20=3D=3D=201;=0A+}=0A+=0A+=3Ditem=20= wait_for_cluster_crash(node,=20params)=0A+=0A+Repeatedly=20test=20if=20= the=20cluster=20running=20at=20B=20for=20responds=20to=20= connections=0A+and=20return=20when=20it=20no=20longer=20does=20so,=20or=20= when=20it=20times=20out.=20=20Processing=20will=0A+run=20for=20= $PostgreSQL::Test::Utils::timeout_default=20seconds=20unless=20a=20= timeout=0A+value=20is=20specified=20as=20a=20parameter.=20=20Returns=20= True=20if=20the=20cluster=20crashed,=20else=0A+False=20if=20the=20= process=20timed=20out.=0A+=0A+=3Dover=0A+=0A+=3Ditem=20timeout=0A+=0A= +Approximate=20number=20of=20seconds=20to=20wait=20for=20cluster=20to=20= crash,=20default=20is=0A+$PostgreSQL::Test::Utils::timeout_default.=20=20= There=20are=20no=20real-time=20guarantee=0A+that=20the=20total=20process=20= time=20won't=20exceed=20the=20timeout.=0A+=0A+=3Dback=0A+=0A+=3Dcut=0A+=0A= +sub=20wait_for_cluster_crash=0A+{=0A+=09my=20$postgresnode=20=3D=20= shift;=0A+=09my=20%params=20=3D=20@_;=0A+=09my=20$crash=20=3D=200;=0A+=0A= +=09$params{timeout}=20=3D=20$PostgreSQL::Test::Utils::timeout_default=0A= +=09=20=20unless=20(defined($params{timeout}));=0A+=0A+=09for=20(my=20= $naps=20=3D=200;=20$naps=20<=20$params{timeout};=20$naps++)=0A+=09{=0A+=09= =09if=20(!$postgresnode->is_alive)=0A+=09=09{=0A+=09=09=09$crash=20=3D=20= 1;=0A+=09=09=09last;=0A+=09=09}=0A+=09=09sleep(1);=0A+=09}=0A+=0A+=09= return=20$crash=20=3D=3D=201;=0A+}=0A+=0A+=3Ditem=20= enable_data_checksums($node,=20%params)=0A+=0A+Function=20for=20enabling=20= data=20checksums=20in=20the=20cluster=20running=20at=20B.=0A+=0A= +=3Dover=0A+=0A+=3Ditem=20cost_delay=0A+=0A+The=20B=20to=20= use=20when=20enabling=20data=20checksums,=20default=20is=200.=0A+=0A= +=3Ditem=20cost_limit=0A+=0A+The=20B=20to=20use=20when=20= enabling=20data=20checksums,=20default=20is=20100.=0A+=0A+=3Ditem=20wait=0A= +=0A+If=20defined,=20the=20function=20will=20wait=20for=20the=20state=20= defined=20in=20this=20parameter,=0A+waiting=20timing=20out,=20before=20= returning.=20=20The=20function=20will=20wait=20for=0A= +$PostgreSQL::Test::Utils::timeout_default=20seconds=20before=20timing=20= out.=0A+=0A+=3Dback=0A+=0A+=3Dcut=0A+=0A+sub=20enable_data_checksums=0A= +{=0A+=09my=20$postgresnode=20=3D=20shift;=0A+=09my=20%params=20=3D=20= @_;=0A+=0A+=09#=20Set=20sane=20defaults=20for=20the=20parameters=0A+=09= $params{cost_delay}=20=3D=200=20unless=20(defined($params{cost_delay}));=0A= +=09$params{cost_limit}=20=3D=20100=20unless=20= (defined($params{cost_limit}));=0A+=0A+=09my=20$query=20=3D=20<<'EOQ';=0A= +SELECT=20pg_enable_data_checksums(%s,=20%s);=0A+EOQ=0A+=0A+=09= $postgresnode->safe_psql('postgres',=0A+=09=09sprintf($query,=20= $params{cost_delay},=20$params{cost_limit}));=0A+=0A+=09= wait_for_checksum_state($postgresnode,=20$params{wait})=0A+=09=20=20if=20= (defined($params{wait}));=0A+}=0A+=0A+=3Ditem=20= disable_data_checksums($node,=20%params)=0A+=0A+Function=20for=20= disabling=20data=20checksums=20in=20the=20cluster=20running=20at=20= B.=0A+=0A+=3Dover=0A+=0A+=3Ditem=20wait=0A+=0A+If=20defined,=20the=20= function=20will=20wait=20for=20the=20state=20to=20turn=20to=20B,=20= or=0A+waiting=20timing=20out,=20before=20returning.=20=20The=20function=20= will=20wait=20for=0A+$PostgreSQL::Test::Utils::timeout_default=20seconds=20= before=20timing=20out.=0A+Unlike=20in=20C=20the=20= value=20of=20the=20parameter=20is=20discarded.=0A+=0A+=3Dback=0A+=0A= +=3Dcut=0A+=0A+sub=20disable_data_checksums=0A+{=0A+=09my=20= $postgresnode=20=3D=20shift;=0A+=09my=20%params=20=3D=20@_;=0A+=0A+=09= $postgresnode->safe_psql('postgres',=0A+=09=09'SELECT=20= pg_disable_data_checksums();');=0A+=0A+=09= wait_for_checksum_state($postgresnode,=20'off')=20if=20= (defined($params{wait}));=0A+}=0A+=0A+=3Ditem=20cointoss=0A+=0A+Helper=20= for=20retrieving=20a=20binary=20value=20with=20random=20distribution=20= for=20deciding=0A+whether=20to=20turn=20things=20off=20during=20testing.=0A= +=0A+=3Dback=0A+=0A+=3Dcut=0A+=0A+sub=20cointoss=0A+{=0A+=09return=20= int(rand()=20<=200.5);=0A+}=0A+=0A+=3Ditem=20random_sleep(max)=0A+=0A= +Helper=20for=20injecting=20random=20sleeps=20here=20and=20there=20in=20= the=20testrun.=20The=20sleep=0A+duration=20will=20be=20in=20the=20range=20= (0,B),=20but=20won't=20be=20predictable=20in=20order=20to=0A+avoid=20= sleep=20patterns=20that=20manage=20to=20avoid=20race=20conditions=20and=20= timing=20bugs.=0A+The=20default=20B=20is=203=20seconds.=0A+=0A= +=3Dback=0A+=0A+=3Dcut=0A+=0A+sub=20random_sleep=0A+{=0A+=09my=20$max=20= =3D=20shift;=0A+=09sleep(int(rand(defined($max)=20?=20$max=20:=203)))=20= if=20cointoss;=0A+}=0A+=0A+=3Ditem=20stopmode=0A+=0A+Small=20helper=20= function=20for=20randomly=20selecting=20a=20valid=20stopmode.=0A+=0A= +=3Dback=0A+=0A+=3Dcut=0A+=0A+sub=20stopmode=0A+{=0A+=09return=20= 'immediate'=20if=20(cointoss);=0A+=09return=20'fast';=0A+}=0A+=0A+=3Dpod=0A= +=0A+=3Dback=0A+=0A+=3Dcut=0A+=0A+1;=0Adiff=20--git=20= a/src/test/modules/test_checksums/test_checksums--1.0.sql=20= b/src/test/modules/test_checksums/test_checksums--1.0.sql=0Anew=20file=20= mode=20100644=0Aindex=2000000000000..8322b64a6ae=0A---=20/dev/null=0A+++=20= b/src/test/modules/test_checksums/test_checksums--1.0.sql=0A@@=20-0,0=20= +1,32=20@@=0A+/*=20= src/test/modules/test_checksums/test_checksums--1.0.sql=20*/=0A+=0A+--=20= complain=20if=20script=20is=20sourced=20in=20psql,=20rather=20than=20via=20= CREATE=20EXTENSION=0A+\echo=20Use=20"CREATE=20EXTENSION=20= test_checksums"=20to=20load=20this=20file.=20\quit=0A+=0A+CREATE=20= FUNCTION=20dcw_inject_delay_barrier(attach=20boolean=20DEFAULT=20true)=0A= +=09RETURNS=20pg_catalog.void=0A+=09AS=20'MODULE_PATHNAME'=20LANGUAGE=20= C;=0A+=0A+CREATE=20FUNCTION=20dcw_inject_launcher_delay(attach=20boolean=20= DEFAULT=20true)=0A+=09RETURNS=20pg_catalog.void=0A+=09AS=20= 'MODULE_PATHNAME'=20LANGUAGE=20C;=0A+=0A+CREATE=20FUNCTION=20= dcw_inject_startup_delay(attach=20boolean=20DEFAULT=20true)=0A+=09= RETURNS=20pg_catalog.void=0A+=09AS=20'MODULE_PATHNAME'=20LANGUAGE=20C;=0A= +=0A+CREATE=20FUNCTION=20dcw_inject_fail_database(attach=20boolean=20= DEFAULT=20true)=0A+=09RETURNS=20pg_catalog.void=0A+=09AS=20= 'MODULE_PATHNAME'=20LANGUAGE=20C;=0A+=0A+CREATE=20FUNCTION=20= dcw_prune_dblist(attach=20boolean=20DEFAULT=20true)=0A+=09RETURNS=20= pg_catalog.void=0A+=09AS=20'MODULE_PATHNAME'=20LANGUAGE=20C;=0A+=0A= +CREATE=20FUNCTION=20dcw_fake_temptable(attach=20boolean=20DEFAULT=20= true)=0A+=09RETURNS=20pg_catalog.void=0A+=09AS=20'MODULE_PATHNAME'=20= LANGUAGE=20C;=0A+=0A+CREATE=20FUNCTION=20= dc_crash_after_checksum_enable(attach=20boolean=20DEFAULT=20true)=0A+=09= RETURNS=20pg_catalog.void=0A+=09AS=20'MODULE_PATHNAME'=20LANGUAGE=20C;=0A= diff=20--git=20a/src/test/modules/test_checksums/test_checksums.c=20= b/src/test/modules/test_checksums/test_checksums.c=0Anew=20file=20mode=20= 100644=0Aindex=2000000000000..8e4c676df76=0A---=20/dev/null=0A+++=20= b/src/test/modules/test_checksums/test_checksums.c=0A@@=20-0,0=20+1,246=20= @@=0A= +/*-----------------------------------------------------------------------= ---=0A+=20*=0A+=20*=20test_checksums.c=0A+=20*=09=09Test=20data=20= checksums=0A+=20*=0A+=20*=20Copyright=20(c)=202026,=20PostgreSQL=20= Global=20Development=20Group=0A+=20*=0A+=20*=20IDENTIFICATION=0A+=20*=09=09= src/test/modules/test_checksums/test_checksums.c=0A+=20*=0A+=20*=20= -------------------------------------------------------------------------=0A= +=20*/=0A+#include=20"postgres.h"=0A+=0A+#include=20"funcapi.h"=0A= +#include=20"miscadmin.h"=0A+#include=20= "postmaster/datachecksumsworker.h"=0A+#include=20"storage/latch.h"=0A= +#include=20"utils/injection_point.h"=0A+#include=20"utils/wait_event.h"=0A= +=0A+#define=20USEC_PER_SEC=20=20=20=201000000=0A+=0A+PG_MODULE_MAGIC;=0A= +=0A+extern=20PGDLLEXPORT=20void=20dc_delay_barrier(const=20char=20= *name,=20const=20void=20*private_data,=20void=20*arg);=0A+extern=20= PGDLLEXPORT=20void=20dc_fail_database(const=20char=20*name,=20const=20= void=20*private_data,=20void=20*arg);=0A+extern=20PGDLLEXPORT=20void=20= dc_dblist(const=20char=20*name,=20const=20void=20*private_data,=20void=20= *arg);=0A+extern=20PGDLLEXPORT=20void=20dc_fake_temptable(const=20char=20= *name,=20const=20void=20*private_data,=20void=20*arg);=0A+=0A+extern=20= PGDLLEXPORT=20void=20crash(const=20char=20*name,=20const=20void=20= *private_data,=20void=20*arg);=0A+=0A+/*=0A+=20*=20Test=20for=20delaying=20= emission=20of=20procsignalbarriers.=0A+=20*/=0A+void=0A= +dc_delay_barrier(const=20char=20*name,=20const=20void=20*private_data,=20= void=20*arg)=0A+{=0A+=09(void)=20name;=0A+=09(void)=20private_data;=0A+=0A= +=09(void)=20WaitLatch(MyLatch,=0A+=09=09=09=09=09=20WL_LATCH_SET=20|=20= WL_TIMEOUT=20|=20WL_EXIT_ON_PM_DEATH,=0A+=09=09=09=09=09=20(3=20*=20= 1000),=0A+=09=09=09=09=09=20WAIT_EVENT_PG_SLEEP);=0A+}=0A+=0A= +PG_FUNCTION_INFO_V1(dcw_inject_delay_barrier);=0A+Datum=0A= +dcw_inject_delay_barrier(PG_FUNCTION_ARGS)=0A+{=0A+#ifdef=20= USE_INJECTION_POINTS=0A+=09bool=09=09attach=20=3D=20PG_GETARG_BOOL(0);=0A= +=0A+=09if=20(attach)=0A+=09=09= InjectionPointAttach("datachecksums-enable-checksums-delay",=0A+=09=09=09= =09=09=09=09=20"test_checksums",=0A+=09=09=09=09=09=09=09=20= "dc_delay_barrier",=0A+=09=09=09=09=09=09=09=20NULL,=0A+=09=09=09=09=09=09= =09=200);=0A+=09else=0A+=09=09= InjectionPointDetach("datachecksums-enable-checksums-delay");=0A+#else=0A= +=09elog(ERROR,=0A+=09=09=20"test=20is=20not=20working=20as=20intended=20= when=20injection=20points=20are=20disabled");=0A+#endif=0A+=09= PG_RETURN_VOID();=0A+}=0A+=0A= +PG_FUNCTION_INFO_V1(dcw_inject_launcher_delay);=0A+Datum=0A= +dcw_inject_launcher_delay(PG_FUNCTION_ARGS)=0A+{=0A+#ifdef=20= USE_INJECTION_POINTS=0A+=09bool=09=09attach=20=3D=20PG_GETARG_BOOL(0);=0A= +=0A+=09if=20(attach)=0A+=09=09= InjectionPointAttach("datachecksumsworker-launcher-delay",=0A+=09=09=09=09= =09=09=09=20"test_checksums",=0A+=09=09=09=09=09=09=09=20= "dc_delay_barrier",=0A+=09=09=09=09=09=09=09=20NULL,=0A+=09=09=09=09=09=09= =09=200);=0A+=09else=0A+=09=09= InjectionPointDetach("datachecksumsworker-launcher-delay");=0A+#else=0A+=09= elog(ERROR,=0A+=09=09=20"test=20is=20not=20working=20as=20intended=20= when=20injection=20points=20are=20disabled");=0A+#endif=0A+=09= PG_RETURN_VOID();=0A+}=0A+=0A= +PG_FUNCTION_INFO_V1(dcw_inject_startup_delay);=0A+Datum=0A= +dcw_inject_startup_delay(PG_FUNCTION_ARGS)=0A+{=0A+#ifdef=20= USE_INJECTION_POINTS=0A+=09bool=09=09attach=20=3D=20PG_GETARG_BOOL(0);=0A= +=0A+=09if=20(attach)=0A+=09=09= InjectionPointAttach("datachecksumsworker-startup-delay",=0A+=09=09=09=09= =09=09=09=20"test_checksums",=0A+=09=09=09=09=09=09=09=20= "dc_delay_barrier",=0A+=09=09=09=09=09=09=09=20NULL,=0A+=09=09=09=09=09=09= =09=200);=0A+=09else=0A+=09=09= InjectionPointDetach("datachecksumsworker-startup-delay");=0A+#else=0A+=09= elog(ERROR,=0A+=09=09=20"test=20is=20not=20working=20as=20intended=20= when=20injection=20points=20are=20disabled");=0A+#endif=0A+=09= PG_RETURN_VOID();=0A+}=0A+=0A+void=0A+dc_fail_database(const=20char=20= *name,=20const=20void=20*private_data,=20void=20*arg)=0A+{=0A+=09static=20= bool=20first_pass=20=3D=20true;=0A+=09DataChecksumsWorkerResult=20*res=20= =3D=20(DataChecksumsWorkerResult=20*)=20arg;=0A+=0A+=09if=20(first_pass)=0A= +=09=09*res=20=3D=20DATACHECKSUMSWORKER_FAILED;=0A+=09first_pass=20=3D=20= false;=0A+}=0A+=0A+PG_FUNCTION_INFO_V1(dcw_inject_fail_database);=0A= +Datum=0A+dcw_inject_fail_database(PG_FUNCTION_ARGS)=0A+{=0A+#ifdef=20= USE_INJECTION_POINTS=0A+=09bool=09=09attach=20=3D=20PG_GETARG_BOOL(0);=0A= +=0A+=09if=20(attach)=0A+=09=09= InjectionPointAttach("datachecksumsworker-fail-db",=0A+=09=09=09=09=09=09= =09=20"test_checksums",=0A+=09=09=09=09=09=09=09=20"dc_fail_database",=0A= +=09=09=09=09=09=09=09=20NULL,=0A+=09=09=09=09=09=09=09=200);=0A+=09else=0A= +=09=09InjectionPointDetach("datachecksumsworker-fail-db");=0A+#else=0A+=09= elog(ERROR,=0A+=09=09=20"test=20is=20not=20working=20as=20intended=20= when=20injection=20points=20are=20disabled");=0A+#endif=0A+=09= PG_RETURN_VOID();=0A+}=0A+=0A+/*=0A+=20*=20Test=20to=20remove=20an=20= entry=20from=20the=20Databaselist=20to=20force=20re-processing=20since=0A= +=20*=20not=20all=20databases=20could=20be=20processed=20in=20the=20= first=20iteration=20of=20the=20loop.=0A+=20*/=0A+void=0A+dc_dblist(const=20= char=20*name,=20const=20void=20*private_data,=20void=20*arg)=0A+{=0A+=09= static=20bool=20first_pass=20=3D=20true;=0A+=09List=09=20=20=20= *DatabaseList=20=3D=20(List=20*)=20arg;=0A+=0A+=09if=20(first_pass)=0A+=09= =09DatabaseList=20=3D=20list_delete_last(DatabaseList);=0A+=09first_pass=20= =3D=20false;=0A+}=0A+=0A+PG_FUNCTION_INFO_V1(dcw_prune_dblist);=0A+Datum=0A= +dcw_prune_dblist(PG_FUNCTION_ARGS)=0A+{=0A+#ifdef=20= USE_INJECTION_POINTS=0A+=09bool=09=09attach=20=3D=20PG_GETARG_BOOL(0);=0A= +=0A+=09if=20(attach)=0A+=09=09= InjectionPointAttach("datachecksumsworker-initial-dblist",=0A+=09=09=09=09= =09=09=09=20"test_checksums",=0A+=09=09=09=09=09=09=09=20"dc_dblist",=0A= +=09=09=09=09=09=09=09=20NULL,=0A+=09=09=09=09=09=09=09=200);=0A+=09else=0A= +=09=09InjectionPointDetach("datachecksumsworker-initial-dblist");=0A= +#else=0A+=09elog(ERROR,=0A+=09=09=20"test=20is=20not=20working=20as=20= intended=20when=20injection=20points=20are=20disabled");=0A+#endif=0A+=09= PG_RETURN_VOID();=0A+}=0A+=0A+/*=0A+=20*=20Test=20to=20force=20waiting=20= for=20existing=20temptables.=0A+=20*/=0A+void=0A+dc_fake_temptable(const=20= char=20*name,=20const=20void=20*private_data,=20void=20*arg)=0A+{=0A+=09= static=20bool=20first_pass=20=3D=20true;=0A+=09int=09=09=20=20=20= *numleft=20=3D=20(int=20*)=20arg;=0A+=0A+=09if=20(first_pass)=0A+=09=09= *numleft=20=3D=201;=0A+=09first_pass=20=3D=20false;=0A+}=0A+=0A= +PG_FUNCTION_INFO_V1(dcw_fake_temptable);=0A+Datum=0A= +dcw_fake_temptable(PG_FUNCTION_ARGS)=0A+{=0A+#ifdef=20= USE_INJECTION_POINTS=0A+=09bool=09=09attach=20=3D=20PG_GETARG_BOOL(0);=0A= +=0A+=09if=20(attach)=0A+=09=09= InjectionPointAttach("datachecksumsworker-fake-temptable-wait",=0A+=09=09= =09=09=09=09=09=20"test_checksums",=0A+=09=09=09=09=09=09=09=20= "dc_fake_temptable",=0A+=09=09=09=09=09=09=09=20NULL,=0A+=09=09=09=09=09=09= =09=200);=0A+=09else=0A+=09=09= InjectionPointDetach("datachecksumsworker-fake-temptable-wait");=0A= +#else=0A+=09elog(ERROR,=0A+=09=09=20"test=20is=20not=20working=20as=20= intended=20when=20injection=20points=20are=20disabled");=0A+#endif=0A+=09= PG_RETURN_VOID();=0A+}=0A+=0A+void=0A+crash(const=20char=20*name,=20= const=20void=20*private_data,=20void=20*arg)=0A+{=0A+=09abort();=0A+}=0A= +=0A+/*=0A+=20*=20dc_crash_after_checksum_enable=0A+=20*=0A+=20*=20= Ensure=20that=20the=20server=20crashes=20just=20before=20the=20= checkpoint=20is=20issued=20after=0A+=20*=20enabling=20or=20disabling=20= checksums.=0A+=20*/=0A= +PG_FUNCTION_INFO_V1(dc_crash_after_checksum_enable);=0A+Datum=0A= +dc_crash_after_checksum_enable(PG_FUNCTION_ARGS)=0A+{=0A+#ifdef=20= USE_INJECTION_POINTS=0A+=09= InjectionPointAttach("datachecksums-enable-checksums-done",=0A+=09=09=09=09= =09=09=20"test_checksums",=20"crash",=20NULL,=200);=0A+=09= InjectionPointAttach("datachecksums-disable-checksums-done",=0A+=09=09=09= =09=09=09=20"test_checksums",=20"crash",=20NULL,=200);=0A+#else=0A+=09= elog(ERROR,=0A+=09=09=20"test=20is=20not=20working=20as=20intended=20= when=20injection=20points=20are=20disabled");=0A+#endif=0A+=09= PG_RETURN_VOID();=0A+}=0Adiff=20--git=20= a/src/test/modules/test_checksums/test_checksums.control=20= b/src/test/modules/test_checksums/test_checksums.control=0Anew=20file=20= mode=20100644=0Aindex=2000000000000..84b4cc035a7=0A---=20/dev/null=0A+++=20= b/src/test/modules/test_checksums/test_checksums.control=0A@@=20-0,0=20= +1,4=20@@=0A+comment=20=3D=20'Test=20code=20for=20data=20checksums'=0A= +default_version=20=3D=20'1.0'=0A+module_pathname=20=3D=20= '$libdir/test_checksums'=0A+relocatable=20=3D=20true=0Adiff=20--git=20= a/src/test/perl/PostgreSQL/Test/Cluster.pm=20= b/src/test/perl/PostgreSQL/Test/Cluster.pm=0Aindex=20= e267ba868fe..9da65f2a55a=20100644=0A---=20= a/src/test/perl/PostgreSQL/Test/Cluster.pm=0A+++=20= b/src/test/perl/PostgreSQL/Test/Cluster.pm=0A@@=20-3863,6=20+3863,42=20= @@=20sub=20advance_wal=0A=20=09}=0A=20}=0A=20=0A+=3Ditem=20= $node->checksum_enable_offline()=0A+=0A+Enable=20data=20page=20checksums=20= in=20an=20offline=20cluster=20with=20B.=20The=0A+caller=20= is=20responsible=20for=20ensuring=20that=20the=20cluster=20is=20in=20the=20= right=20state=20for=0A+this=20operation.=0A+=0A+=3Dcut=0A+=0A+sub=20= checksum_enable_offline=0A+{=0A+=09my=20($self)=20=3D=20@_;=0A+=0A+=09= print=20"#=20Enabling=20checksums=20in=20\"$self->data_dir\"\n";=0A+=09= PostgreSQL::Test::Utils::system_or_bail('pg_checksums',=20'-D',=0A+=09=09= $self->data_dir,=20'-e');=0A+=09return;=0A+}=0A+=0A+=3Ditem=20= checksum_disable_offline=0A+=0A+Disable=20data=20page=20checksums=20in=20= an=20offline=20cluster=20with=20B.=20The=0A+caller=20is=20= responsible=20for=20ensuring=20that=20the=20cluster=20is=20in=20the=20= right=20state=20for=0A+this=20operation.=0A+=0A+=3Dcut=0A+=0A+sub=20= checksum_disable_offline=0A+{=0A+=09my=20($self)=20=3D=20@_;=0A+=0A+=09= print=20"#=20Disabling=20checksums=20in=20\"$self->data_dir\"\n";=0A+=09= PostgreSQL::Test::Utils::system_or_bail('pg_checksums',=20'-D',=0A+=09=09= $self->data_dir,=20'-d');=0A+=09return;=0A+}=0A+=0A=20=3Dpod=0A=20=0A=20= =3Dback=0Adiff=20--git=20a/src/test/regress/expected/rules.out=20= b/src/test/regress/expected/rules.out=0Aindex=20f4ee2bd7459..697e0329da4=20= 100644=0A---=20a/src/test/regress/expected/rules.out=0A+++=20= b/src/test/regress/expected/rules.out=0A@@=20-2089,6=20+2089,41=20@@=20= pg_stat_progress_create_index|=20SELECT=20s.pid,=0A=20=20=20=20=20= s.param15=20AS=20partitions_done=0A=20=20=20=20FROM=20= (pg_stat_get_progress_info('CREATE=20INDEX'::text)=20s(pid,=20datid,=20= relid,=20param1,=20param2,=20param3,=20param4,=20param5,=20param6,=20= param7,=20param8,=20param9,=20param10,=20param11,=20param12,=20param13,=20= param14,=20param15,=20param16,=20param17,=20param18,=20param19,=20= param20)=0A=20=20=20=20=20=20LEFT=20JOIN=20pg_database=20d=20ON=20= ((s.datid=20=3D=20d.oid)));=0A+pg_stat_progress_data_checksums|=20SELECT=20= s.pid,=0A+=20=20=20=20s.datid,=0A+=20=20=20=20d.datname,=0A+=20=20=20=20=20= =20=20=20CASE=20s.param1=0A+=20=20=20=20=20=20=20=20=20=20=20=20WHEN=200=20= THEN=20'enabling'::text=0A+=20=20=20=20=20=20=20=20=20=20=20=20WHEN=201=20= THEN=20'disabling'::text=0A+=20=20=20=20=20=20=20=20=20=20=20=20WHEN=202=20= THEN=20'waiting=20on=20temporary=20tables'::text=0A+=20=20=20=20=20=20=20= =20=20=20=20=20WHEN=203=20THEN=20'waiting=20on=20barrier'::text=0A+=20=20= =20=20=20=20=20=20=20=20=20=20WHEN=204=20THEN=20'done'::text=0A+=20=20=20= =20=20=20=20=20=20=20=20=20ELSE=20NULL::text=0A+=20=20=20=20=20=20=20=20= END=20AS=20phase,=0A+=20=20=20=20=20=20=20=20CASE=20s.param2=0A+=20=20=20= =20=20=20=20=20=20=20=20=20WHEN=20'-1'::integer=20THEN=20NULL::bigint=0A= +=20=20=20=20=20=20=20=20=20=20=20=20ELSE=20s.param2=0A+=20=20=20=20=20=20= =20=20END=20AS=20databases_total,=0A+=20=20=20=20s.param3=20AS=20= databases_done,=0A+=20=20=20=20=20=20=20=20CASE=20s.param4=0A+=20=20=20=20= =20=20=20=20=20=20=20=20WHEN=20'-1'::integer=20THEN=20NULL::bigint=0A+=20= =20=20=20=20=20=20=20=20=20=20=20ELSE=20s.param4=0A+=20=20=20=20=20=20=20= =20END=20AS=20relations_total,=0A+=20=20=20=20=20=20=20=20CASE=20= s.param5=0A+=20=20=20=20=20=20=20=20=20=20=20=20WHEN=20'-1'::integer=20= THEN=20NULL::bigint=0A+=20=20=20=20=20=20=20=20=20=20=20=20ELSE=20= s.param5=0A+=20=20=20=20=20=20=20=20END=20AS=20relations_done,=0A+=20=20=20= =20=20=20=20=20CASE=20s.param6=0A+=20=20=20=20=20=20=20=20=20=20=20=20= WHEN=20'-1'::integer=20THEN=20NULL::bigint=0A+=20=20=20=20=20=20=20=20=20= =20=20=20ELSE=20s.param6=0A+=20=20=20=20=20=20=20=20END=20AS=20= blocks_total,=0A+=20=20=20=20=20=20=20=20CASE=20s.param7=0A+=20=20=20=20=20= =20=20=20=20=20=20=20WHEN=20'-1'::integer=20THEN=20NULL::bigint=0A+=20=20= =20=20=20=20=20=20=20=20=20=20ELSE=20s.param7=0A+=20=20=20=20=20=20=20=20= END=20AS=20blocks_done=0A+=20=20=20FROM=20= (pg_stat_get_progress_info('DATACHECKSUMS'::text)=20s(pid,=20datid,=20= relid,=20param1,=20param2,=20param3,=20param4,=20param5,=20param6,=20= param7,=20param8,=20param9,=20param10,=20param11,=20param12,=20param13,=20= param14,=20param15,=20param16,=20param17,=20param18,=20param19,=20= param20)=0A+=20=20=20=20=20LEFT=20JOIN=20pg_database=20d=20ON=20= ((s.datid=20=3D=20d.oid)))=0A+=20=20ORDER=20BY=20s.datid;=0A=20= pg_stat_progress_vacuum|=20SELECT=20s.pid,=0A=20=20=20=20=20s.datid,=0A=20= =20=20=20=20d.datname,=0Adiff=20--git=20= a/src/test/regress/expected/stats.out=20= b/src/test/regress/expected/stats.out=0Aindex=20cd00f35bf7a..aa86c11bd99=20= 100644=0A---=20a/src/test/regress/expected/stats.out=0A+++=20= b/src/test/regress/expected/stats.out=0A@@=20-51,6=20+51,22=20@@=20= client=20backend|relation|vacuum=0A=20client=20backend|temp=20= relation|normal=0A=20client=20backend|wal|init=0A=20client=20= backend|wal|normal=0A+datachecksum=20launcher|relation|bulkread=0A= +datachecksum=20launcher|relation|bulkwrite=0A+datachecksum=20= launcher|relation|init=0A+datachecksum=20launcher|relation|normal=0A= +datachecksum=20launcher|relation|vacuum=0A+datachecksum=20launcher|temp=20= relation|normal=0A+datachecksum=20launcher|wal|init=0A+datachecksum=20= launcher|wal|normal=0A+datachecksum=20worker|relation|bulkread=0A= +datachecksum=20worker|relation|bulkwrite=0A+datachecksum=20= worker|relation|init=0A+datachecksum=20worker|relation|normal=0A= +datachecksum=20worker|relation|vacuum=0A+datachecksum=20worker|temp=20= relation|normal=0A+datachecksum=20worker|wal|init=0A+datachecksum=20= worker|wal|normal=0A=20io=20worker|relation|bulkread=0A=20io=20= worker|relation|bulkwrite=0A=20io=20worker|relation|init=0A@@=20-95,7=20= +111,7=20@@=20walsummarizer|wal|init=0A=20walsummarizer|wal|normal=0A=20= walwriter|wal|init=0A=20walwriter|wal|normal=0A-(79=20rows)=0A+(95=20= rows)=0A=20\a=0A=20--=20ensure=20that=20both=20seqscan=20and=20indexscan=20= plans=20are=20allowed=0A=20SET=20enable_seqscan=20TO=20on;=0Adiff=20= --git=20a/src/tools/pgindent/typedefs.list=20= b/src/tools/pgindent/typedefs.list=0Aindex=209f5ee8fd482..f4cce543113=20= 100644=0A---=20a/src/tools/pgindent/typedefs.list=0A+++=20= b/src/tools/pgindent/typedefs.list=0A@@=20-428,6=20+428,8=20@@=20= CheckPointStmt=0A=20CheckpointStatsData=0A=20CheckpointerRequest=0A=20= CheckpointerShmemStruct=0A+ChecksumBarrierCondition=0A+ChecksumType=0A=20= Chromosome=0A=20CkptSortItem=0A=20CkptTsStatus=0A@@=20-598,6=20+600,7=20= @@=20CustomScan=0A=20CustomScanMethods=0A=20CustomScanState=0A=20= CycleCtr=0A+DataChecksumsWorkerOperation=0A=20DBState=0A=20DCHCacheEntry=0A= =20DEADLOCK_INFO=0A@@=20-621,6=20+624,10=20@@=20DataPageDeleteStack=0A=20= DataTypesUsageChecks=0A=20DataTypesUsageVersionCheck=0A=20DatabaseInfo=0A= +DataChecksumsWorkerDatabase=0A+DataChecksumsWorkerResult=0A= +DataChecksumsWorkerResultEntry=0A+DataChecksumsWorkerShmemStruct=0A=20= DateADT=0A=20DateTimeErrorExtra=0A=20Datum=0A@@=20-4314,6=20+4321,8=20@@=20= xl_btree_split=0A=20xl_btree_unlink_page=0A=20xl_btree_update=0A=20= xl_btree_vacuum=0A+xl_checkpoint_redo=0A+xl_checksum_state=0A=20= xl_clog_truncate=0A=20xl_commit_ts_truncate=0A=20= xl_dbase_create_file_copy_rec=0A--=20=0A2.39.3=20(Apple=20Git-146)=0A=0A= --Apple-Mail=_B8DC27FC-A3EC-4551-9E38-E86153E9E5F4 Content-Disposition: attachment; filename=v20250203-2-Review-fixes.txt Content-Type: text/plain; x-unix-mode=0644; name="v20250203-2-Review-fixes.txt" Content-Transfer-Encoding: quoted-printable =46rom 15eda37bed7543a314b5dc811b155545cb4b0778 Mon Sep 17 00:00:00 2001 From: Daniel Gustafsson Date: Thu, 8 Jan 2026 14:45:16 +0100 Subject: [PATCH v20250203 2/2] Review comments --- doc/src/sgml/func/func-admin.sgml | 12 +- doc/src/sgml/glossary.sgml | 4 +- doc/src/sgml/wal.sgml | 6 + src/backend/access/rmgrdesc/xlogdesc.c | 46 ++- src/backend/access/transam/xlog.c | 292 ++++++++-------- src/backend/access/transam/xlogfuncs.c | 13 +- src/backend/backup/basebackup.c | 25 +- src/backend/catalog/system_functions.sql | 14 +- src/backend/catalog/system_views.sql | 5 +- src/backend/postmaster/datachecksumsworker.c | 326 +++++++++--------- src/backend/postmaster/walsummarizer.c | 5 +- src/backend/storage/page/bufpage.c | 12 + src/backend/utils/init/postinit.c | 2 +- src/bin/pg_controldata/pg_controldata.c | 2 +- src/include/access/xlog.h | 6 +- src/include/access/xlog_internal.h | 8 +- src/include/catalog/pg_control.h | 6 +- src/include/catalog/pg_proc.dat | 15 +- src/include/commands/progress.h | 5 +- src/include/postmaster/datachecksumsworker.h | 5 +- src/include/storage/checksum.h | 3 +- src/include/storage/proc.h | 5 +- src/test/modules/test_checksums/Makefile | 2 +- src/test/modules/test_checksums/meson.build | 2 +- .../modules/test_checksums/t/001_basic.pl | 2 +- .../modules/test_checksums/t/002_restarts.pl | 2 +- .../test_checksums/t/003_standby_restarts.pl | 2 +- .../modules/test_checksums/t/004_offline.pl | 2 +- .../modules/test_checksums/t/005_injection.pl | 77 ++--- .../test_checksums/t/006_pgbench_single.pl | 30 +- .../test_checksums/t/007_pgbench_standby.pl | 72 ++-- .../test_checksums/t/DataChecksums/Utils.pm | 34 +- .../test_checksums/test_checksums--1.0.sql | 14 +- .../modules/test_checksums/test_checksums.c | 79 +++-- src/test/perl/PostgreSQL/Test/Cluster.pm | 9 - src/test/regress/expected/rules.out | 7 +- src/tools/pgindent/typedefs.list | 1 + 37 files changed, 558 insertions(+), 594 deletions(-) diff --git a/doc/src/sgml/func/func-admin.sgml = b/doc/src/sgml/func/func-admin.sgml index 315f4093398..7762b774bf2 100644 --- a/doc/src/sgml/func/func-admin.sgml +++ b/doc/src/sgml/func/func-admin.sgml @@ -3122,7 +3122,7 @@ SELECT = convert_from(pg_read_binary_file('file_in_utf8.txt'), 'UTF8'); pg_enable_data_checksums - pg_enable_data_checksums ( = cost_delay int, = cost_limit int, = fast bool ) + pg_enable_data_checksums ( = cost_delay int, = cost_limit int ) void @@ -3137,9 +3137,6 @@ SELECT = convert_from(pg_read_binary_file('file_in_utf8.txt'), 'UTF8'); If cost_delay and = cost_limit are specified, the process is throttled using the same principles = as Cost-based = Vacuum Delay. - If fast is specified as = true - then a fast checkpoint will be issued when data checksums have = been - enabled, which may cause a spike in I/O. @@ -3149,7 +3146,7 @@ SELECT = convert_from(pg_read_binary_file('file_in_utf8.txt'), 'UTF8'); pg_disable_data_checksums - pg_disable_data_checksums ( = fast bool ) + pg_disable_data_checksums () void @@ -3159,11 +3156,6 @@ SELECT = convert_from(pg_read_binary_file('file_in_utf8.txt'), 'UTF8'); stopped validating data checksums, the data checksum mode will = be set to off. - - If fast is specified as = true - then a fast checkpoint will be issued when data checksums have = been - enabled, which may cause a spike in I/O. - diff --git a/doc/src/sgml/glossary.sgml b/doc/src/sgml/glossary.sgml index 4895ae76b81..4f3120b7826 100644 --- a/doc/src/sgml/glossary.sgml +++ b/doc/src/sgml/glossary.sgml @@ -580,7 +580,7 @@ Data Checksums Worker - An auxiliary = process + A background = worker which enables data checksums in a specific database. @@ -590,7 +590,7 @@ Data Checksums Worker Launcher - An auxiliary = process + A background = worker which starts = processes for enabling data checksums in each database, or disables data = checksums cluster-wide. diff --git a/doc/src/sgml/wal.sgml b/doc/src/sgml/wal.sgml index 4b5df81caf2..df41fb54cb6 100644 --- a/doc/src/sgml/wal.sgml +++ b/doc/src/sgml/wal.sgml @@ -342,6 +342,12 @@ the mode will automatically be set to off. =20 + + Disabling data checksums while data checksums are actively being = enabled + will abort the current processing. There is no way to restart from = where + processing was interrupted. + + Impact on system of online operations diff --git a/src/backend/access/rmgrdesc/xlogdesc.c = b/src/backend/access/rmgrdesc/xlogdesc.c index 1a70cdccb5c..8a854312ea1 100644 --- a/src/backend/access/rmgrdesc/xlogdesc.c +++ b/src/backend/access/rmgrdesc/xlogdesc.c @@ -55,6 +55,25 @@ get_wal_level_string(int wal_level) return wal_level_str; } =20 +static const char * +get_checksum_version_string(ChecksumType checksum) +{ + switch (checksum) + { + case PG_DATA_CHECKSUM_VERSION: + return "on"; + case PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION: + return "inprogress-off"; + case PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION: + return "inprogress-on"; + case PG_DATA_CHECKSUM_OFF: + return "off"; + } + + Assert(false); + return "?"; +} + void xlog_desc(StringInfo buf, XLogReaderState *record) { @@ -70,7 +89,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record) "tli %u; prev tli %u; = fpw %s; wal_level %s; logical decoding %s; xid %u:%u; oid %u; multi %u; = offset %" PRIu64 "; " "oldest xid %u in DB = %u; oldest multi %u in DB %u; " "oldest/newest commit = timestamp xid: %u/%u; " - "oldest running xid %u; = %s", + "oldest running xid %u; = " + "checksums %s; %s", = LSN_FORMAT_ARGS(checkpoint->redo), = checkpoint->ThisTimeLineID, = checkpoint->PrevTimeLineID, @@ -89,6 +109,7 @@ xlog_desc(StringInfo buf, XLogReaderState *record) = checkpoint->oldestCommitTsXid, = checkpoint->newestCommitTsXid, = checkpoint->oldestActiveXid, + = get_checksum_version_string(checkpoint->dataChecksumVersion), (info =3D=3D = XLOG_CHECKPOINT_SHUTDOWN) ? "shutdown" : "online"); } else if (info =3D=3D XLOG_NEXTOID) @@ -164,10 +185,12 @@ xlog_desc(StringInfo buf, XLogReaderState *record) } else if (info =3D=3D XLOG_CHECKPOINT_REDO) { - int wal_level; + xl_checkpoint_redo xlrec; =20 - memcpy(&wal_level, rec, sizeof(int)); - appendStringInfo(buf, "wal_level %s", = get_wal_level_string(wal_level)); + memcpy(&xlrec, rec, sizeof(xl_checkpoint_redo)); + appendStringInfo(buf, "wal_level %s; checksums %s", + = get_wal_level_string(xlrec.wal_level), + = get_checksum_version_string(xlrec.data_checksum_version)); } else if (info =3D=3D XLOG_LOGICAL_DECODING_STATUS_CHANGE) { @@ -181,20 +204,7 @@ xlog_desc(StringInfo buf, XLogReaderState *record) xl_checksum_state xlrec; =20 memcpy(&xlrec, rec, sizeof(xl_checksum_state)); - switch (xlrec.new_checksumtype) - { - case PG_DATA_CHECKSUM_VERSION: - appendStringInfoString(buf, "on"); - break; - case PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION: - appendStringInfoString(buf, = "inprogress-off"); - break; - case PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION: - appendStringInfoString(buf, = "inprogress-on"); - break; - default: - appendStringInfoString(buf, "off"); - } + appendStringInfoString(buf, = get_checksum_version_string(xlrec.new_checksumtype)); } } =20 diff --git a/src/backend/access/transam/xlog.c = b/src/backend/access/transam/xlog.c index 6d437c2bc11..28928e2195d 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -288,11 +288,6 @@ static XLogRecPtr RedoRecPtr; */ static bool doPageWrites; =20 -/* - * Force creating a restartpoint on the next CHECKPOINT after = XLOG_CHECKSUMS. - */ -static bool checksumRestartPoint =3D false; - /*---------- * Shared-memory data structures for XLOG control * @@ -596,6 +591,10 @@ static ControlFileData *ControlFile =3D NULL; * looked up in the checksum_barriers struct. The struct member for = the target * state defines which state the backend must currently be in, and = which it * must not be in. + * + * The reason for this explicit checking is to ensure that processing = cannot + * be started such that it breaks the assumptions of the state machine. = See + * datachecksumsworker.c for a lengthy discussion on these states. */ typedef struct ChecksumBarrierCondition { @@ -611,13 +610,12 @@ typedef struct ChecksumBarrierCondition int barrier_ne_sz; } ChecksumBarrierCondition; =20 -static const ChecksumBarrierCondition checksum_barriers[] =3D +static const ChecksumBarrierCondition checksum_barriers[4] =3D { {PG_DATA_CHECKSUM_OFF, {PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION, = PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION}, 2, {PG_DATA_CHECKSUM_VERSION}, = 1}, {PG_DATA_CHECKSUM_VERSION, = {PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION}, 1, {0}, 0}, - {PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION, = {PG_DATA_CHECKSUM_ANY_VERSION}, 1, {PG_DATA_CHECKSUM_VERSION}, 1}, + {PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION, {0}, 0, = {PG_DATA_CHECKSUM_VERSION}, 1}, {PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION, = {PG_DATA_CHECKSUM_VERSION}, 1, {0}, 0}, - {-1} }; =20 /* @@ -699,20 +697,9 @@ static bool updateMinRecoveryPoint =3D true; * this is only updated when absorbing a procsignal barrier during = interrupt * processing. The reason for keeping a copy in backend-private memory = is to * avoid locking for interrogating the data checksum state. Possible = values - * are the data checksum versions defined in storage/bufpage.h as well = as zero - * when data checksums are disabled. - */ -static uint32 LocalDataChecksumVersion =3D 0; - -/* - * Flag to remember if the procsignalbarrier being absorbed for = checksums is - * the first one. The first procsignalbarrier can in rare cases be for = the - * state we've initialized, i.e. a duplicate. This may happen for any - * data_checksum_version value when the process is spawned between the = update - * of XLogCtl->data_checksum_version and the barrier being emitted. = This can - * only happen on the very first barrier so mark that with this flag. + * are the data checksum versions defined in storage/checksum.h. */ -static bool InitialDataChecksumTransition =3D true; +static ChecksumType LocalDataChecksumVersion =3D 0; =20 /* * Variable backing the GUC, keep it in sync with = LocalDataChecksumVersion. @@ -921,9 +908,7 @@ XLogInsertRecord(XLogRecData *rdata, Assert(RedoRecPtr < Insert->RedoRecPtr); RedoRecPtr =3D Insert->RedoRecPtr; } - doPageWrites =3D (Insert->fullPageWrites || - Insert->runningBackups > = 0 || - = DataChecksumsNeedWrite()); + doPageWrites =3D (Insert->fullPageWrites || = Insert->runningBackups > 0); =20 if (doPageWrites && (!prevDoPageWrites || @@ -4808,10 +4793,9 @@ DataChecksumsOffInProgress(void) * state transition. */ void -SetDataChecksumsOnInProgress(bool immediate_checkpoint) +SetDataChecksumsOnInProgress(void) { uint64 barrier; - int flags; =20 Assert(ControlFile !=3D NULL); =20 @@ -4819,19 +4803,20 @@ SetDataChecksumsOnInProgress(bool = immediate_checkpoint) * The state transition is performed in a critical section with * checkpoints held off to provide crash safety. */ - MyProc->delayChkptFlags |=3D DELAY_CHKPT_START; START_CRIT_SECTION(); + MyProc->delayChkptFlags |=3D DELAY_CHKPT_START; =20 XLogChecksums(PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION); =20 SpinLockAcquire(&XLogCtl->info_lck); + Assert(XLogCtl->data_checksum_version =3D=3D = PG_DATA_CHECKSUM_OFF); XLogCtl->data_checksum_version =3D = PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION; SpinLockRelease(&XLogCtl->info_lck); =20 barrier =3D = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_ON); =20 - END_CRIT_SECTION(); MyProc->delayChkptFlags &=3D ~DELAY_CHKPT_START; + END_CRIT_SECTION(); =20 /* * Await state change in all backends to ensure that all = backends are in @@ -4839,18 +4824,6 @@ SetDataChecksumsOnInProgress(bool = immediate_checkpoint) * checksums. */ WaitForProcSignalBarrier(barrier); - - /* - * force checkpoint to persist the current checksum state in = control file - * etc. - * - * XXX is this needed? There's already a checkpoint at the end = of - * ProcessAllDatabases, maybe this is redundant? - */ - flags =3D CHECKPOINT_FORCE | CHECKPOINT_WAIT; - if (immediate_checkpoint) - flags =3D flags | CHECKPOINT_FAST; - RequestCheckpoint(flags); } =20 /* @@ -4877,10 +4850,9 @@ SetDataChecksumsOnInProgress(bool = immediate_checkpoint) * state transition. */ void -SetDataChecksumsOn(bool immediate_checkpoint) +SetDataChecksumsOn(void) { uint64 barrier; - int flags; =20 Assert(ControlFile !=3D NULL); =20 @@ -4893,14 +4865,14 @@ SetDataChecksumsOn(bool immediate_checkpoint) if (XLogCtl->data_checksum_version !=3D = PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION) { SpinLockRelease(&XLogCtl->info_lck); - elog(ERROR, "checksums not in \"inprogress-on\" mode"); + elog(PANIC, "checksums not in \"inprogress-on\" mode"); } =20 SpinLockRelease(&XLogCtl->info_lck); =20 - MyProc->delayChkptFlags |=3D DELAY_CHKPT_START; INJECTION_POINT("datachecksums-enable-checksums-delay", NULL); START_CRIT_SECTION(); + MyProc->delayChkptFlags |=3D DELAY_CHKPT_START; =20 XLogChecksums(PG_DATA_CHECKSUM_VERSION); =20 @@ -4910,8 +4882,8 @@ SetDataChecksumsOn(bool immediate_checkpoint) =20 barrier =3D = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_ON); =20 - END_CRIT_SECTION(); MyProc->delayChkptFlags &=3D ~DELAY_CHKPT_START; + END_CRIT_SECTION(); =20 /* * Await state transition of "on" in all backends. When done we = know that @@ -4920,13 +4892,7 @@ SetDataChecksumsOn(bool immediate_checkpoint) */ WaitForProcSignalBarrier(barrier); =20 - INJECTION_POINT("datachecksums-enable-checksums-pre-checkpoint", = NULL); - - /* XXX is this needed? */ - flags =3D CHECKPOINT_FORCE | CHECKPOINT_WAIT; - if (immediate_checkpoint) - flags =3D flags | CHECKPOINT_FAST; - RequestCheckpoint(flags); + INJECTION_POINT("datachecksums-enable-checksums-done", NULL); } =20 /* @@ -4943,10 +4909,9 @@ SetDataChecksumsOn(bool immediate_checkpoint) * state transition. */ void -SetDataChecksumsOff(bool immediate_checkpoint) +SetDataChecksumsOff(void) { uint64 barrier; - int flags; =20 Assert(ControlFile !=3D NULL); =20 @@ -4970,8 +4935,8 @@ SetDataChecksumsOff(bool immediate_checkpoint) { SpinLockRelease(&XLogCtl->info_lck); =20 - MyProc->delayChkptFlags |=3D DELAY_CHKPT_START; START_CRIT_SECTION(); + MyProc->delayChkptFlags |=3D DELAY_CHKPT_START; =20 XLogChecksums(PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION); =20 @@ -4981,8 +4946,8 @@ SetDataChecksumsOff(bool immediate_checkpoint) =20 barrier =3D = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_OFF); =20 - END_CRIT_SECTION(); MyProc->delayChkptFlags &=3D ~DELAY_CHKPT_START; + END_CRIT_SECTION(); =20 /* * Update local state in all backends to ensure that any = backend in @@ -4990,20 +4955,6 @@ SetDataChecksumsOff(bool immediate_checkpoint) */ WaitForProcSignalBarrier(barrier); =20 - /* - * force checkpoint to persist the current checksum = state in control - * file etc. - * - * XXX is this safe? What if the crash/shutdown happens = while waiting - * for the checkpoint? Also, should we persist the = checksum first and - * only then flip the flag in XLogCtl? - */ - = INJECTION_POINT("datachecksums-disable-checksums-pre-checkpoint", NULL); - flags =3D CHECKPOINT_FORCE | CHECKPOINT_WAIT; - if (immediate_checkpoint) - flags =3D flags | CHECKPOINT_FAST; - RequestCheckpoint(flags); - /* * At this point we know that no backends are verifying = data checksums * during reading. Next, we can safely move to state = "off" to also @@ -5020,13 +4971,11 @@ SetDataChecksumsOff(bool immediate_checkpoint) SpinLockRelease(&XLogCtl->info_lck); } =20 - /* - * Ensure that we don't incur a checkpoint during disabling = checksums. - */ - MyProc->delayChkptFlags |=3D DELAY_CHKPT_START; START_CRIT_SECTION(); + /* Ensure that we don't incur a checkpoint during disabling = checksums */ + MyProc->delayChkptFlags |=3D DELAY_CHKPT_START; =20 - XLogChecksums(0); + XLogChecksums(PG_DATA_CHECKSUM_OFF); =20 SpinLockAcquire(&XLogCtl->info_lck); XLogCtl->data_checksum_version =3D 0; @@ -5034,15 +4983,10 @@ SetDataChecksumsOff(bool immediate_checkpoint) =20 barrier =3D = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_OFF); =20 - END_CRIT_SECTION(); MyProc->delayChkptFlags &=3D ~DELAY_CHKPT_START; + END_CRIT_SECTION(); =20 WaitForProcSignalBarrier(barrier); - - flags =3D CHECKPOINT_FORCE | CHECKPOINT_WAIT; - if (immediate_checkpoint) - flags =3D flags | CHECKPOINT_FAST; - RequestCheckpoint(flags); } =20 /* @@ -5057,36 +5001,55 @@ SetDataChecksumsOff(bool immediate_checkpoint) bool AbsorbDataChecksumsBarrier(int target_state) { - const ChecksumBarrierCondition *condition =3D checksum_barriers; + const ChecksumBarrierCondition *condition =3D NULL; int current =3D LocalDataChecksumVersion; bool found =3D false; =20 + /* + * If the target state matches the current state then the = barrier has been + * repeated. + */ + if (current =3D=3D target_state) + return true; + + /* + * If the cluster is in recovery we skip the validation of = current state + * since the replay is trusted. + */ + if (InRecovery) + { + SetLocalDataChecksumVersion(target_state); + return true; + } + /* * Find the barrier condition definition for the target state. = Not finding * a condition would be a grave programmer error as the states = are a * discrete set. */ - while (condition->target !=3D target_state && condition->target = !=3D -1) - condition++; - if (unlikely(condition->target =3D=3D -1)) - ereport(ERROR, - = errcode(ERRCODE_INVALID_PARAMETER_VALUE), - errmsg("invalid target state %i for data = checksum barrier", - target_state)); + for (int i =3D 0; i < lengthof(checksum_barriers); i++) + { + if (checksum_barriers[i].target =3D=3D target_state) + condition =3D &checksum_barriers[i]; + } + Assert(condition); =20 /* * The current state MUST be equal to one of the EQ states = defined in this - * barrier condition, or equal to the target_state if - and only = if - - * InitialDataChecksumTransition is true. + * barrier condition. If the EQ states array is zero then that = imples that + * the current state can match any state, so fastpath check for = that + * first. */ - for (int i =3D 0; i < condition->barrier_eq_sz; i++) + if (condition->barrier_eq_sz =3D=3D 0) + found =3D true; + else { - if (current =3D=3D condition->barrier_eq[i] || - condition->barrier_eq[i] =3D=3D = PG_DATA_CHECKSUM_ANY_VERSION) - found =3D true; + for (int i =3D 0; i < condition->barrier_eq_sz; i++) + { + if (current =3D=3D condition->barrier_eq[i]) + found =3D true; + } } - if (InitialDataChecksumTransition && current =3D=3D = target_state) - found =3D true; =20 /* * The current state MUST NOT be equal to any of the NE states = defined in @@ -5099,7 +5062,7 @@ AbsorbDataChecksumsBarrier(int target_state) } =20 /* - * If the relevent state criteria aren't satisfied, throw an = error which + * If the relevant state criteria aren't satisfied, throw an = error which * will be caught by the procsignal machinery for a later retry. */ if (!found) @@ -5109,7 +5072,6 @@ AbsorbDataChecksumsBarrier(int target_state) current, target_state)); =20 SetLocalDataChecksumVersion(target_state); - InitialDataChecksumTransition =3D false; return true; } =20 @@ -5141,14 +5103,17 @@ SetLocalDataChecksumVersion(uint32 = data_checksum_version) const char * show_data_checksums(void) { - if (LocalDataChecksumVersion =3D=3D PG_DATA_CHECKSUM_VERSION) - return "on"; - else if (LocalDataChecksumVersion =3D=3D = PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION) - return "inprogress-on"; - else if (LocalDataChecksumVersion =3D=3D = PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION) - return "inprogress-off"; - else - return "off"; + switch (LocalDataChecksumVersion) + { + case PG_DATA_CHECKSUM_VERSION: + return "on"; + case PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION: + return "inprogress-on"; + case PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION: + return "inprogress-off"; + case PG_DATA_CHECKSUM_OFF: + return "off"; + } } =20 /* @@ -6305,6 +6270,11 @@ StartupXLOG(void) XLogCtl->SharedRecoveryState =3D = RECOVERY_STATE_ARCHIVE; else XLogCtl->SharedRecoveryState =3D = RECOVERY_STATE_CRASH; + + XLogCtl->data_checksum_version =3D = checkPoint.dataChecksumVersion; + = SetLocalDataChecksumVersion(checkPoint.dataChecksumVersion); + ControlFile->data_checksum_version =3D = checkPoint.dataChecksumVersion; + SpinLockRelease(&XLogCtl->info_lck); =20 /* @@ -6780,7 +6750,7 @@ StartupXLOG(void) */ if (XLogCtl->data_checksum_version =3D=3D = PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION) { - XLogChecksums(0); + XLogChecksums(PG_DATA_CHECKSUM_OFF); =20 SpinLockAcquire(&XLogCtl->info_lck); XLogCtl->data_checksum_version =3D 0; @@ -6788,8 +6758,8 @@ StartupXLOG(void) SpinLockRelease(&XLogCtl->info_lck); =20 ereport(WARNING, - (errmsg("data checksums state has been = set of off"), - errhint("If checksums were being = enabled during shutdown then processing must be manually restarted."))); + errmsg("data checksums state has been = set to off"), + errhint("If checksums were being enabled = during shutdown then processing must be manually restarted.")); } =20 /* @@ -6800,7 +6770,7 @@ StartupXLOG(void) */ if (XLogCtl->data_checksum_version =3D=3D = PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION) { - XLogChecksums(0); + XLogChecksums(PG_DATA_CHECKSUM_OFF); =20 SpinLockAcquire(&XLogCtl->info_lck); XLogCtl->data_checksum_version =3D 0; @@ -7685,7 +7655,7 @@ CreateCheckPoint(int flags) * Get the current data_checksum_version value from xlogctl, = valid at the * time of the checkpoint. */ - checkPoint.data_checksum_version =3D = XLogCtl->data_checksum_version; + checkPoint.dataChecksumVersion =3D = XLogCtl->data_checksum_version; =20 if (shutdown) { @@ -7739,9 +7709,18 @@ CreateCheckPoint(int flags) */ if (!shutdown) { + xl_checkpoint_redo redo_rec; + + WALInsertLockAcquire(); + redo_rec.wal_level =3D wal_level; + SpinLockAcquire(&XLogCtl->info_lck); + redo_rec.data_checksum_version =3D = XLogCtl->data_checksum_version; + SpinLockRelease(&XLogCtl->info_lck); + WALInsertLockRelease(); + /* Include WAL level in record for WAL summarizer's = benefit. */ XLogBeginInsert(); - XLogRegisterData(&wal_level, sizeof(wal_level)); + XLogRegisterData(&redo_rec, sizeof(xl_checkpoint_redo)); (void) XLogInsert(RM_XLOG_ID, XLOG_CHECKPOINT_REDO); =20 /* @@ -7797,6 +7776,10 @@ CreateCheckPoint(int flags) checkPoint.nextOid +=3D TransamVariables->oidCount; LWLockRelease(OidGenLock); =20 + SpinLockAcquire(&XLogCtl->info_lck); + checkPoint.dataChecksumVersion =3D = XLogCtl->data_checksum_version; + SpinLockRelease(&XLogCtl->info_lck); + checkPoint.logicalDecodingEnabled =3D = IsLogicalDecodingEnabled(); =20 MultiXactGetCheckptMulti(shutdown, @@ -7947,7 +7930,7 @@ CreateCheckPoint(int flags) ControlFile->minRecoveryPointTLI =3D 0; =20 /* make sure we start with the checksum version as of the = checkpoint */ - ControlFile->data_checksum_version =3D = checkPoint.data_checksum_version; + ControlFile->data_checksum_version =3D = checkPoint.dataChecksumVersion; =20 /* * Persist unloggedLSN value. It's reset on crash recovery, so = this goes @@ -8094,7 +8077,9 @@ CreateEndOfRecoveryRecord(void) ControlFile->minRecoveryPointTLI =3D xlrec.ThisTimeLineID; =20 /* start with the latest checksum version (as of the end of = recovery) */ + SpinLockAcquire(&XLogCtl->info_lck); ControlFile->data_checksum_version =3D = XLogCtl->data_checksum_version; + SpinLockRelease(&XLogCtl->info_lck); =20 UpdateControlFile(); LWLockRelease(ControlFileLock); @@ -8439,7 +8424,7 @@ CreateRestartPoint(int flags) } =20 /* we shall start with the latest checksum version */ - ControlFile->data_checksum_version =3D = lastCheckPoint.data_checksum_version; + ControlFile->data_checksum_version =3D = lastCheckPoint.dataChecksumVersion; =20 UpdateControlFile(); } @@ -8869,8 +8854,6 @@ XLogChecksums(uint32 new_type) XLogBeginInsert(); XLogRegisterData((char *) &xlrec, sizeof(xl_checksum_state)); =20 - INJECTION_POINT("datachecksums-xlogchecksums-pre-xloginsert", = &new_type); - recptr =3D XLogInsert(RM_XLOG_ID, XLOG_CHECKSUMS); XLogFlush(recptr); } @@ -9001,6 +8984,11 @@ xlog_redo(XLogReaderState *record) MultiXactAdvanceOldest(checkPoint.oldestMulti, = checkPoint.oldestMultiDB); =20 + SpinLockAcquire(&XLogCtl->info_lck); + XLogCtl->data_checksum_version =3D = checkPoint.dataChecksumVersion; + = SetLocalDataChecksumVersion(checkPoint.dataChecksumVersion); + SpinLockRelease(&XLogCtl->info_lck); + /* * No need to set oldestClogXid here as well; it'll be = set when we * redo an xl_clog_truncate if it changed since = initialization. @@ -9060,6 +9048,7 @@ xlog_redo(XLogReaderState *record) /* ControlFile->checkPointCopy always tracks the latest = ckpt XID */ LWLockAcquire(ControlFileLock, LW_EXCLUSIVE); ControlFile->checkPointCopy.nextXid =3D = checkPoint.nextXid; + ControlFile->data_checksum_version =3D = checkPoint.dataChecksumVersion; LWLockRelease(ControlFileLock); =20 /* @@ -9124,8 +9113,14 @@ xlog_redo(XLogReaderState *record) /* ControlFile->checkPointCopy always tracks the latest = ckpt XID */ LWLockAcquire(ControlFileLock, LW_EXCLUSIVE); ControlFile->checkPointCopy.nextXid =3D = checkPoint.nextXid; + ControlFile->data_checksum_version =3D = checkPoint.dataChecksumVersion; LWLockRelease(ControlFileLock); =20 + SpinLockAcquire(&XLogCtl->info_lck); + XLogCtl->data_checksum_version =3D = checkPoint.dataChecksumVersion; + = SetLocalDataChecksumVersion(checkPoint.dataChecksumVersion); + SpinLockRelease(&XLogCtl->info_lck); + /* TLI should not change in an on-line checkpoint */ (void) GetCurrentReplayRecPtr(&replayTLI); if (checkPoint.ThisTimeLineID !=3D replayTLI) @@ -9292,7 +9287,43 @@ xlog_redo(XLogReaderState *record) } else if (info =3D=3D XLOG_CHECKPOINT_REDO) { - /* nothing to do here, just for informational purposes = */ + xl_checkpoint_redo redo_rec; + bool new_state =3D false; + uint64 barrier; + + memcpy(&redo_rec, XLogRecGetData(record), = sizeof(xl_checkpoint_redo)); + + SpinLockAcquire(&XLogCtl->info_lck); + XLogCtl->data_checksum_version =3D = redo_rec.data_checksum_version; + if (redo_rec.data_checksum_version !=3D = ControlFile->data_checksum_version) + new_state =3D true; + SpinLockRelease(&XLogCtl->info_lck); + + if (new_state) + { + switch (redo_rec.data_checksum_version) + { + case = PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION: + barrier =3D = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_ON); + = WaitForProcSignalBarrier(barrier); + break; + + case = PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION: + barrier =3D = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_INPROGRESS_OFF); + = WaitForProcSignalBarrier(barrier); + break; + + case PG_DATA_CHECKSUM_VERSION: + barrier =3D = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_ON); + = WaitForProcSignalBarrier(barrier); + break; + + case PG_DATA_CHECKSUM_OFF: + barrier =3D = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_OFF); + = WaitForProcSignalBarrier(barrier); + break; + } + } } else if (info =3D=3D XLOG_LOGICAL_DECODING_STATUS_CHANGE) { @@ -9382,34 +9413,11 @@ xlog_redo(XLogReaderState *record) WaitForProcSignalBarrier(barrier); break; =20 - default: - Assert(state.new_checksumtype =3D=3D 0); + case PG_DATA_CHECKSUM_OFF: barrier =3D = EmitProcSignalBarrier(PROCSIGNAL_BARRIER_CHECKSUM_OFF); WaitForProcSignalBarrier(barrier); break; } - - /* - * Force creating a restartpoint for the first = CHECKPOINT after seeing - * XLOG_CHECKSUMS in WAL - */ - checksumRestartPoint =3D true; - } - - if (checksumRestartPoint && - (info =3D=3D XLOG_CHECKPOINT_ONLINE || - info =3D=3D XLOG_CHECKPOINT_REDO || - info =3D=3D XLOG_CHECKPOINT_SHUTDOWN)) - { - int flags; - - elog(LOG, "forcing creation of a restartpoint after = XLOG_CHECKSUMS"); - - /* We explicitly want an immediate checkpoint here */ - flags =3D CHECKPOINT_FORCE | CHECKPOINT_WAIT | = CHECKPOINT_FAST; - RequestCheckpoint(flags); - - checksumRestartPoint =3D false; } } =20 diff --git a/src/backend/access/transam/xlogfuncs.c = b/src/backend/access/transam/xlogfuncs.c index b991d1997d6..742ddcf124a 100644 --- a/src/backend/access/transam/xlogfuncs.c +++ b/src/backend/access/transam/xlogfuncs.c @@ -757,17 +757,12 @@ pg_promote(PG_FUNCTION_ARGS) Datum disable_data_checksums(PG_FUNCTION_ARGS) { - bool fast =3D PG_GETARG_BOOL(0); - - ereport(LOG, - errmsg("disable_data_checksums, fast: %d", = fast)); - if (!superuser()) ereport(ERROR, errcode(ERRCODE_INSUFFICIENT_PRIVILEGE), errmsg("must be superuser to change data = checksum state")); =20 - StartDataChecksumsWorkerLauncher(DISABLE_DATACHECKSUMS, 0, 0, = fast); + StartDataChecksumsWorkerLauncher(DISABLE_DATACHECKSUMS, 0, 0); PG_RETURN_VOID(); } =20 @@ -781,10 +776,6 @@ enable_data_checksums(PG_FUNCTION_ARGS) { int cost_delay =3D PG_GETARG_INT32(0); int cost_limit =3D PG_GETARG_INT32(1); - bool fast =3D PG_GETARG_BOOL(2); - - ereport(LOG, - errmsg("enable_data_checksums, cost_delay: %d = cost_limit: %d fast: %d", cost_delay, cost_limit, fast)); =20 if (!superuser()) ereport(ERROR, @@ -801,7 +792,7 @@ enable_data_checksums(PG_FUNCTION_ARGS) = errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("cost limit must be greater than = zero")); =20 - StartDataChecksumsWorkerLauncher(ENABLE_DATACHECKSUMS, = cost_delay, cost_limit, fast); + StartDataChecksumsWorkerLauncher(ENABLE_DATACHECKSUMS, = cost_delay, cost_limit); =20 PG_RETURN_VOID(); } diff --git a/src/backend/backup/basebackup.c = b/src/backend/backup/basebackup.c index 7a668355f4d..2eb0a3af453 100644 --- a/src/backend/backup/basebackup.c +++ b/src/backend/backup/basebackup.c @@ -1613,9 +1613,7 @@ sendFile(bbsink *sink, const char *readfilename, = const char *tarfilename, * enabled for this cluster, and if this is a relation file, = then verify * the checksum. */ - if (!noverify_checksums && - DataChecksumsNeedWrite() && - RelFileNumberIsValid(relfilenumber)) + if (!noverify_checksums && RelFileNumberIsValid(relfilenumber)) verify_checksum =3D true; =20 /* @@ -1748,7 +1746,7 @@ sendFile(bbsink *sink, const char *readfilename, = const char *tarfilename, * If the amount of data we were able to read was not a = multiple of * BLCKSZ, we cannot verify checksums, which are = block-level. */ - if (verify_checksum && (cnt % BLCKSZ !=3D 0)) + if (verify_checksum && DataChecksumsNeedVerify() && (cnt = % BLCKSZ !=3D 0)) { ereport(WARNING, (errmsg("could not verify = checksum in file \"%s\", block " @@ -1843,9 +1841,10 @@ sendFile(bbsink *sink, const char *readfilename, = const char *tarfilename, * 'blkno' is the block number of the first page in the bbsink's buffer * relative to the start of the relation. * - * 'verify_checksum' indicates whether we should try to verify = checksums - * for the blocks we read. If we do this, we'll update = *checksum_failures - * and issue warnings as appropriate. + * 'verify_checksum' determins if the user has asked to verify = checksums, but + * since data checksums can be disabled, or become disabled, we need to = check + * state before verifying individual pages. If we do this, we'll = update + * *checksum_failures and issue warnings as appropriate. */ static off_t read_file_data_into_buffer(bbsink *sink, const char *readfilename, int = fd, @@ -1871,6 +1870,13 @@ read_file_data_into_buffer(bbsink *sink, const = char *readfilename, int fd, int reread_cnt; uint16 expected_checksum; =20 + /* + * The data checksum state can change at any point, so = we need to + * re-check before each page. + */ + if (!DataChecksumsNeedVerify()) + return cnt; + page =3D sink->bbs_buffer + BLCKSZ * i; =20 /* If the page is OK, go on to the next one. */ @@ -1893,7 +1899,12 @@ read_file_data_into_buffer(bbsink *sink, const = char *readfilename, int fd, * allows us to wait until we can be certain that no = write to the * block is in progress. Since we don't have any such = thing right now, * we just do this and hope for the best. + * + * The data checksum state may also have changed = concurrently so check + * again. */ + if (!DataChecksumsNeedVerify()) + return cnt; reread_cnt =3D basebackup_read_file(fd, sink->bbs_buffer + = BLCKSZ * i, BLCKSZ, = offset + BLCKSZ * i, diff --git a/src/backend/catalog/system_functions.sql = b/src/backend/catalog/system_functions.sql index 2ab882c363d..b4c58652af1 100644 --- a/src/backend/catalog/system_functions.sql +++ b/src/backend/catalog/system_functions.sql @@ -659,20 +659,12 @@ AS 'pg_replication_origin_session_setup'; =20 CREATE OR REPLACE FUNCTION pg_enable_data_checksums(cost_delay integer DEFAULT 0, - cost_limit integer DEFAULT 100, - fast boolean DEFAULT = false) + cost_limit integer DEFAULT 100) RETURNS void STRICT VOLATILE LANGUAGE internal PARALLEL RESTRICTED AS 'enable_data_checksums'; =20 -CREATE OR REPLACE FUNCTION - pg_disable_data_checksums(fast boolean DEFAULT false) -RETURNS void -STRICT VOLATILE LANGUAGE internal -PARALLEL RESTRICTED -AS 'disable_data_checksums'; - -- -- The default permissions for functions mean that anyone can execute = them. -- A number of functions shouldn't be executable by just anyone, but = rather @@ -798,9 +790,9 @@ REVOKE EXECUTE ON FUNCTION pg_ls_logicalmapdir() = FROM PUBLIC; =20 REVOKE EXECUTE ON FUNCTION pg_ls_replslotdir(text) FROM PUBLIC; =20 -REVOKE EXECUTE ON FUNCTION pg_enable_data_checksums(integer, integer, = boolean) FROM public; +REVOKE EXECUTE ON FUNCTION pg_enable_data_checksums(integer, integer) = FROM public; =20 -REVOKE EXECUTE ON FUNCTION pg_disable_data_checksums(boolean) FROM = public; +REVOKE EXECUTE ON FUNCTION pg_disable_data_checksums() FROM public; =20 -- -- We also set up some things as accessible to standard roles. diff --git a/src/backend/catalog/system_views.sql = b/src/backend/catalog/system_views.sql index 919c97e1040..47c661fe678 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -1391,9 +1391,8 @@ CREATE VIEW pg_stat_progress_data_checksums AS CASE S.param1 WHEN 0 THEN 'enabling' WHEN 1 THEN 'disabling' WHEN 2 THEN 'waiting on temporary tables' - WHEN 3 THEN 'waiting on checkpoint' - WHEN 4 THEN 'waiting on = barrier' - WHEN 5 THEN 'done' + WHEN 3 THEN 'waiting on = barrier' + WHEN 4 THEN 'done' END AS phase, CASE S.param2 WHEN -1 THEN NULL ELSE S.param2 END AS = databases_total, S.param3 AS databases_done, diff --git a/src/backend/postmaster/datachecksumsworker.c = b/src/backend/postmaster/datachecksumsworker.c index 57311760b2b..24e6af63f6a 100644 --- a/src/backend/postmaster/datachecksumsworker.c +++ b/src/backend/postmaster/datachecksumsworker.c @@ -14,35 +14,43 @@ * Checksums can be either enabled or disabled cluster-wide, with = on/off being * the end state for data_checksums. * - * Enabling checksums - * ------------------ + * 1. Enabling checksums + * --------------------- * When enabling checksums in an online cluster, data_checksums will be = set to * "inprogress-on" which signals that write operations MUST compute and = write * the checksum on the data page, but during reading the checksum SHALL = NOT be - * verified. This ensures that all objects created during checksumming = will - * have checksums set, but no reads will fail due to incorrect = checksum. The - * DataChecksumsWorker will compile a list of databases which exist at = the - * start of checksumming, and all of these which haven't been dropped = during - * the processing MUST have been processed successfully in order for = checksums - * to be enabled. Any new relation created during processing will see = the - * in-progress state and will automatically be checksummed. + * verified. This ensures that all objects created during when = checksums are + * being enabled will have checksums set, but reads wont fail due to = missing or + * invalid checksums. Invalid checksums can be present in case the = cluster had + * checksums enabled, then disabled them and updated the page while = they were + * disabled. + * + * The DataChecksumsWorker will compile a list of databases which exist = at the + * start of checksumming, and once all are processed will regenerate = the list + * and start over processing any new entries. Once there are no new = entries on + * the list, processing will end. All databases MUST BE successfully = processed + * in order for data checksums to be enabled, the only exception are = databases + * which are dropped before having been processed. + + * Any new relation in a processed database, created during processing, = will + * see the in-progress state and will automatically be checksummed. * * For each database, all relations which have storage are read and = every data * page is marked dirty to force a write with the checksum. This will = generate * a lot of WAL as the entire database is read and written. * - * If the processing is interrupted by a cluster restart, it will be = restarted - * from the beginning again as state isn't persisted. + * If the processing is interrupted by a cluster crash or restart, it = needs to + * be restarted from the beginning again as state isn't persisted. * - * Disabling checksums - * ------------------- + * 2. Disabling checksums + * ---------------------- * When disabling checksums, data_checksums will be set to = "inprogress-off" * which signals that checksums are written but no longer verified. = This ensure * that backends which have yet to move from the "on" state will still = be able * to process data checksum validation. * - * Synchronization and Correctness - * ------------------------------- + * 3. Synchronization and Correctness + * ---------------------------------- * The processes involved in enabling, or disabling, data checksums in = an * online cluster must be properly synchronized with the normal = backends * serving concurrent queries to ensure correctness. Correctness is = defined @@ -57,7 +65,7 @@ * backends must wait on the procsignalbarrier to be acknowledged = by all * before proceeding to validate data checksums. * - * There are two levels of synchronization required for changing = data_checksums + * There are two steps of synchronization required for changing = data_checksums * in an online cluster: (i) changing state in the active backends = ("on", * "off", "inprogress-on" and "inprogress-off"), and (ii) ensuring no * incompatible objects and processes are left in a database when = workers end. @@ -65,81 +73,89 @@ * latter with ensuring that any concurrent activity cannot break the = data * checksum contract during processing. * - * Synchronizing the state change is done with procsignal barriers, = where the - * WAL logging backend updating the global state in the controlfile = will wait - * for all other backends to absorb the barrier. Barrier absorption = will happen - * during interrupt processing, which means that connected backends = will change - * state at different times. To prevent data checksum state changes = when - * writing and verifying checksums, interrupts shall be held off before - * interrogating state and resumed when the IO operation has been = performed. + * Synchronizing the state change is done with procsignal barriers, = before + * updating the controlfile with the state all other backends must = absorb the + * barrier. Barrier absorption will happen during interrupt = processing, which + * means that connected backends will change state at different times. = If + * waiting for a barrier is done during startup, for example during = replay, it + * is important to realize that any locks held by the startup process = might + * cause deadblocks if backends end up waiting for those locks while = startup + * is waiting for a procsignalbarrier. + * + * 3.1 When Enabling Data Checksums + * -------------------------------- + * A process which fails to observe data checksums being enabled can = induce two + * types of errors: failing to write the checksum when modifying the = page and + * failing to validate the data checksum on the page when reading it. * - * When Enabling Data Checksums - * ---------------------------- - * A process which fails to observe data checksums being enabled can = induce - * two types of errors: failing to write the checksum when modifying = the page - * and failing to validate the data checksum on the page when reading = it. + * When processing starts all backends belong to one of the below sets, = with + * one set being empty: * - * When processing starts all backends belong to one of the below = sets, with - * one set being empty: + * Bd: Backends in "off" state + * Bi: Backends in "inprogress-on" state * - * Bd: Backends in "off" state - * Bi: Backends in "inprogress-on" state + * If processing is started in an online cluster then all backends are = in Bd. + * If processing was halted by the cluster shutting down (due to a = crash or + * intentional restart), the controlfile state "inprogress-on" will be = observed + * on system startup and all backends will be placed in Bd. The = controlfile + * state will also be set of "off". * - * If processing is started in an online cluster then all backends = are in Bd. - * If processing was halted by the cluster shutting down, the = controlfile - * state "inprogress-on" will be observed on system startup and all = backends - * will be placed in Bd. Backends transition Bd -> Bi via a = procsignalbarrier - * which is emitted by the DataChecksumsLauncher. When all backends = have - * acknowledged the barrier then Bd will be empty and the next phase = can - * begin: calculating and writing data checksums with = DataChecksumsWorkers. - * When the DataChecksumsWorker processes have finished writing = checksums on - * all pages and enables data checksums cluster-wide via another - * procsignalbarrier, there are four sets of backends where Bd shall = be an - * empty set: + * Backends transition Bd -> Bi via a procsignalbarrier which is = emitted by the + * DataChecksumsLauncher. When all backends have acknowledged the = barrier then + * Bd will be empty and the next phase can begin: calculating and = writing data + * checksums with DataChecksumsWorkers. When the DataChecksumsWorker = processes + * have finished writing checksums on all pages, data checksums are = enabled + * cluster-wide via another procsignalbarrier. There are four sets of = backends + * where Bd shall be an empty set: * - * Bg: Backend updating the global state and emitting the = procsignalbarrier - * Bd: Backends in "off" state - * Be: Backends in "on" state - * Bi: Backends in "inprogress-on" state + * Bg: Backend updating the global state and emitting the = procsignalbarrier + * Bd: Backends in "off" state + * Be: Backends in "on" state + * Bi: Backends in "inprogress-on" state * - * Backends in Bi and Be will write checksums when modifying a page, = but only - * backends in Be will verify the checksum during reading. The Bg = backend is - * blocked waiting for all backends in Bi to process interrupts and = move to - * Be. Any backend starting while Bg is waiting on the = procsignalbarrier will - * observe the global state being "on" and will thus automatically = belong to - * Be. Checksums are enabled cluster-wide when Bi is an empty set. = Bi and Be - * are compatible sets while still operating based on their local = state as - * both write data checksums. + * Backends in Bi and Be will write checksums when modifying a page, = but only + * backends in Be will verify the checksum during reading. The Bg = backend is + * blocked waiting for all backends in Bi to process interrupts and = move to + * Be. Any backend starting while Bg is waiting on the = procsignalbarrier will + * observe the global state being "on" and will thus automatically = belong to + * Be. Checksums are enabled cluster-wide when Bi is an empty set. Bi = and Be + * are compatible sets while still operating based on their local state = as + * both write data checksums. * - * When Disabling Data Checksums - * ----------------------------- - * A process which fails to observe that data checksums have been = disabled - * can induce two types of errors: writing the checksum when = modifying the - * page and validating a data checksum which is no longer correct due = to - * modifications to the page. + * 3.2 When Disabling Data Checksums + * --------------------------------- + * A process which fails to observe that data checksums have been = disabled + * can induce two types of errors: writing the checksum when modifying = the + * page and validating a data checksum which is no longer correct due = to + * modifications to the page. The former is not an error per se as data + * integrity is maintained, but it is wasteful. The latter will cause = errors + * in user operations. Assuming the following sets of backends: * - * Bg: Backend updating the global state and emitting the = procsignalbarrier - * Bd: Backends in "off" state - * Be: Backends in "on" state - * Bo: Backends in "inprogress-off" state + * Bg: Backend updating the global state and emitting the = procsignalbarrier + * Bd: Backends in "off" state + * Be: Backends in "on" state + * Bo: Backends in "inprogress-off" state + * Bi: Backends in "inprogress-on" state * - * Backends transition from the Be state to Bd like so: Be -> Bo -> = Bd + * Backends transition from the Be state to Bd like so: Be -> Bo -> Bd. = From + * all other states, the transition can be straight to Bd. * - * The goal is to transition all backends to Bd making the others = empty sets. - * Backends in Bo write data checksums, but don't validate them, such = that - * backends still in Be can continue to validate pages until the = barrier has - * been absorbed such that they are in Bo. Once all backends are in = Bo, the - * barrier to transition to "off" can be raised and all backends can = safely - * stop writing data checksums as no backend is enforcing data = checksum - * validation any longer. + * The goal is to transition all backends to Bd making the others empty = sets. + * Backends in Bo write data checksums, but don't validate them, such = that + * backends still in Be can continue to validate pages until the = barrier has + * been absorbed such that they are in Bo. Once all backends are in Bo, = the + * barrier to transition to "off" can be raised and all backends can = safely + * stop writing data checksums as no backend is enforcing data checksum + * validation any longer. * - * Potential optimizations - * ----------------------- + * 4. Future opportunities for optimizations + * ----------------------------------------- * Below are some potential optimizations and improvements which were = brought * up during reviews of this feature, but which weren't implemented in = the * initial version. These are ideas listed without any validation on = their - * feasibility or potential payoff. More discussion on these can be = found on - * the -hackers threads linked to in the commit message of this = feature. + * feasibility or potential payoff. More discussion on (most of) these = can be + * found on the -hackers threads linked to in the commit message of = this + * feature. * * * Launching datachecksumsworker for resuming operation from the = startup * process: Currently users have to restart processing manually = after a @@ -149,17 +165,16 @@ * * Avoid dirtying the page when checksums already match: Iff the = checksum * on the page happens to already match we still dirty the page. It = should * be enough to only do the log_newpage_buffer() call in that case. - * * Invent a lightweight WAL record that doesn't contain the = full-page - * image but just the block number: On replay, the redo routine = would read - * the page from disk. * * Teach pg_checksums to avoid checksummed pages when pg_checksums = is used * to enable checksums on a cluster which is in inprogress-on state = and * may have checksummed pages (make pg_checksums be able to resume = an - * online operation). + * online operation). This should only be attempted for wal_level = minimal. * * Restartability (not necessarily with page granularity). + * * Avoid processing databases which were created during = inprogress-on. + * Right now all databases are processed regardless to be safe. * * - * Portions Copyright (c) 1996-2025, PostgreSQL Global Development = Group + * Portions Copyright (c) 1996-2026, PostgreSQL Global Development = Group * Portions Copyright (c) 1994, Regents of the University of California * * @@ -221,16 +236,19 @@ typedef struct DataChecksumsWorkerShmemStruct DataChecksumsWorkerOperation launch_operation; int launch_cost_delay; int launch_cost_limit; - bool launch_fast; =20 /* - * Is a launcher process is currently running? - * - * This is set by the launcher process, after it has read the = above - * launch_* parameters. + * Is a launcher process is currently running? This is set by = the main + * launcher process, after it has read the above launch_* = parameters. */ bool launcher_running; =20 + /* + * If a worker process currently running? This is set by the = worker + * launcher when it starts waiting for a worker process to = finish. + */ + bool worker_running; + /* * These fields indicate the target state that the launcher is = currently * working towards. They can be different from the corresponding = launch_* @@ -245,7 +263,6 @@ typedef struct DataChecksumsWorkerShmemStruct DataChecksumsWorkerOperation operation; int cost_delay; int cost_limit; - bool immediate_checkpoint; =20 /* * Signaling between the launcher and the worker process. @@ -304,7 +321,7 @@ static List *BuildDatabaseList(void); static List *BuildRelationList(bool temp_relations, bool = include_shared); static void FreeDatabaseList(List *dblist); static DataChecksumsWorkerResult = ProcessDatabase(DataChecksumsWorkerDatabase *db); -static bool ProcessAllDatabases(bool immediate_checkpoint); +static bool ProcessAllDatabases(void); static bool ProcessSingleRelationFork(Relation reln, ForkNumber = forkNum, BufferAccessStrategy strategy); static void launcher_cancel_handler(SIGNAL_ARGS); static void WaitForAllTransactionsToFinish(void); @@ -319,8 +336,7 @@ static void WaitForAllTransactionsToFinish(void); void StartDataChecksumsWorkerLauncher(DataChecksumsWorkerOperation op, int = cost_delay, - int = cost_limit, - bool = fast) + int = cost_limit) { BackgroundWorker bgw; BackgroundWorkerHandle *bgw_handle; @@ -338,7 +354,8 @@ = StartDataChecksumsWorkerLauncher(DataChecksumsWorkerOperation op, DataChecksumsWorkerShmem->launch_operation =3D op; DataChecksumsWorkerShmem->launch_cost_delay =3D cost_delay; DataChecksumsWorkerShmem->launch_cost_limit =3D cost_limit; - DataChecksumsWorkerShmem->launch_fast =3D fast; + + INJECTION_POINT("datachecksumsworker-startup-delay", NULL); =20 /* is the launcher already running? */ launcher_running =3D DataChecksumsWorkerShmem->launcher_running; @@ -384,6 +401,9 @@ = StartDataChecksumsWorkerLauncher(DataChecksumsWorkerOperation op, = errcode(ERRCODE_INSUFFICIENT_RESOURCES), errmsg("failed to start = background worker to process data checksums")); } + else + ereport(ERROR, + errmsg("data checksum processing already = running")); } =20 /* @@ -409,15 +429,7 @@ ProcessSingleRelationFork(Relation reln, ForkNumber = forkNum, BufferAccessStrateg snprintf(activity, sizeof(activity) - 1, "processing: %s.%s (%s, = %dblocks)", relns, RelationGetRelationName(reln), = forkNames[forkNum], numblocks); pgstat_report_activity(STATE_RUNNING, activity); - - /* - * As of now we only update the block counter for main forks in = order to - * not cause too frequent calls. TODO: investigate whether we = should do it - * more frequent? - */ - if (forkNum =3D=3D MAIN_FORKNUM) - = pgstat_progress_update_param(PROGRESS_DATACHECKSUMS_BLOCKS_TOTAL, - = numblocks); + = pgstat_progress_update_param(PROGRESS_DATACHECKSUMS_BLOCKS_TOTAL, = numblocks); =20 /* * We are looping over the blocks which existed at the time of = process @@ -456,8 +468,10 @@ ProcessSingleRelationFork(Relation reln, ForkNumber = forkNum, BufferAccessStrateg * a lock, because if we miss it being set, we will try = again soon. */ Assert(operation =3D=3D ENABLE_DATACHECKSUMS); + LWLockAcquire(DataChecksumsWorkerLock, LW_SHARED); if (DataChecksumsWorkerShmem->launch_operation =3D=3D = DISABLE_DATACHECKSUMS) abort_requested =3D true; + LWLockRelease(DataChecksumsWorkerLock); =20 if (abort_requested) return false; @@ -471,6 +485,10 @@ ProcessSingleRelationFork(Relation reln, ForkNumber = forkNum, BufferAccessStrateg = pgstat_progress_update_param(PROGRESS_DATACHECKSUMS_BLOCKS_DONE, = (blknum + 1)); =20 + /* + * Processing is re-using the vacuum cost delay for = process + * throttling, hence why we call vacuum APIs here. + */ vacuum_delay_point(false); } =20 @@ -607,6 +625,8 @@ ProcessDatabase(DataChecksumsWorkerDatabase *db) errmsg("initiating data checksum processing in = database \"%s\"", db->dbname)); =20 + DataChecksumsWorkerShmem->worker_running =3D true; + snprintf(activity, sizeof(activity) - 1, "Waiting for worker in database %s (pid %ld)", = db->dbname, (long) pid); pgstat_report_activity(STATE_RUNNING, activity); @@ -625,6 +645,7 @@ ProcessDatabase(DataChecksumsWorkerDatabase *db) db->dbname)); =20 pgstat_report_activity(STATE_IDLE, NULL); + DataChecksumsWorkerShmem->worker_running =3D false; =20 return DataChecksumsWorkerShmem->success; } @@ -633,8 +654,9 @@ ProcessDatabase(DataChecksumsWorkerDatabase *db) * launcher_exit * * Internal routine for cleaning up state when the launcher process = exits. We - * need to clean up the abort flag to ensure that processing can be = restarted - * again after it was previously aborted. + * need to clean up the abort flag to ensure that processing started = again if + * it was previously aborted (note: started again, *not* restarted from = where + * it left off). */ static void launcher_exit(int code, Datum arg) @@ -644,6 +666,14 @@ launcher_exit(int code, Datum arg) LWLockAcquire(DataChecksumsWorkerLock, LW_EXCLUSIVE); launcher_running =3D false; DataChecksumsWorkerShmem->launcher_running =3D false; + + /* + * TODO: how to really handle the worker still running = when the + * launcher exits? + */ + if (DataChecksumsWorkerShmem->worker_running) + ereport(LOG, + errmsg("data checksums launcher = exiting while worker is still running")); LWLockRelease(DataChecksumsWorkerLock); } } @@ -720,8 +750,8 @@ WaitForAllTransactionsToFinish(void) if (rc & WL_POSTMASTER_DEATH) ereport(FATAL, errcode(ERRCODE_ADMIN_SHUTDOWN), - errmsg("postmaster exited during = data checksum processing"), - errhint("Restart the database = and restart data checksum processing by calling = pg_enable_data_checksums().")); + errmsg("postmaster exited during = data checksums processing"), + errhint("Data checksums = processing must be restarted manually after cluster restart.")); =20 LWLockAcquire(DataChecksumsWorkerLock, LW_SHARED); if (DataChecksumsWorkerShmem->launch_operation !=3D = operation) @@ -749,7 +779,7 @@ DataChecksumsWorkerLauncherMain(Datum arg) on_shmem_exit(launcher_exit, 0); =20 ereport(DEBUG1, - errmsg("background worker \"datachecksum = launcher\" started")); + errmsg("background worker \"datachecksums = launcher\" started")); =20 pqsignal(SIGTERM, die); pqsignal(SIGINT, launcher_cancel_handler); @@ -759,10 +789,14 @@ DataChecksumsWorkerLauncherMain(Datum arg) MyBackendType =3D B_DATACHECKSUMSWORKER_LAUNCHER; init_ps_display(NULL); =20 + INJECTION_POINT("datachecksumsworker-launcher-delay", NULL); + LWLockAcquire(DataChecksumsWorkerLock, LW_EXCLUSIVE); =20 if (DataChecksumsWorkerShmem->launcher_running) { + ereport(LOG, + errmsg("background worker = \"datachecksums launcher\" already running, exiting")); /* Launcher was already running, let it finish */ LWLockRelease(DataChecksumsWorkerLock); return; @@ -780,7 +814,6 @@ DataChecksumsWorkerLauncherMain(Datum arg) DataChecksumsWorkerShmem->operation =3D operation; DataChecksumsWorkerShmem->cost_delay =3D = DataChecksumsWorkerShmem->launch_cost_delay; DataChecksumsWorkerShmem->cost_limit =3D = DataChecksumsWorkerShmem->launch_cost_limit; - DataChecksumsWorkerShmem->immediate_checkpoint =3D = DataChecksumsWorkerShmem->launch_fast; LWLockRelease(DataChecksumsWorkerLock); =20 /* @@ -801,26 +834,21 @@ again: * checksums enabled, exit immediately as there is = nothing more to do. * Hold interrupts to make sure state doesn't change = during checking. */ - HOLD_INTERRUPTS(); if (DataChecksumsNeedVerify()) - { - RESUME_INTERRUPTS(); goto done; - } - RESUME_INTERRUPTS(); =20 /* * Set the state to inprogress-on and wait on the = procsignal barrier. */ = pgstat_progress_update_param(PROGRESS_DATACHECKSUMS_PHASE, = PROGRESS_DATACHECKSUMS_PHASE_ENABLING); - = SetDataChecksumsOnInProgress(DataChecksumsWorkerShmem->immediate_checkpoin= t); + SetDataChecksumsOnInProgress(); =20 /* * All backends are now in inprogress-on state and are = writing data * checksums. Start processing all data at rest. */ - if = (!ProcessAllDatabases(DataChecksumsWorkerShmem->immediate_checkpoint)) + if (!ProcessAllDatabases()) { /* * If the target state changed during processing = then it's not a @@ -842,20 +870,13 @@ again: * Data checksums have been set on all pages, set the = state to on in * order to instruct backends to validate checksums on = reading. */ - = SetDataChecksumsOn(DataChecksumsWorkerShmem->immediate_checkpoint); + SetDataChecksumsOn(); } else if (operation =3D=3D DISABLE_DATACHECKSUMS) { - int flags; - = pgstat_progress_update_param(PROGRESS_DATACHECKSUMS_PHASE, = PROGRESS_DATACHECKSUMS_PHASE_DISABLING); - = SetDataChecksumsOff(DataChecksumsWorkerShmem->immediate_checkpoint); - - flags =3D CHECKPOINT_FORCE | CHECKPOINT_WAIT; - if (DataChecksumsWorkerShmem->immediate_checkpoint) - flags =3D flags | CHECKPOINT_FAST; - RequestCheckpoint(flags); + SetDataChecksumsOff(); } else { @@ -903,18 +924,14 @@ done: * checksums. Until no new databases are found, this will loop around = computing * a new list and comparing it to the already seen ones. * - * If immediate_checkpoint is set to true then a CHECKPOINT_FAST will = be - * issued. This is useful for testing but should be avoided in = production use - * as it may affect cluster performance drastically. */ static bool -ProcessAllDatabases(bool immediate_checkpoint) +ProcessAllDatabases(void) { List *DatabaseList; HTAB *ProcessedDatabases =3D NULL; HASHCTL hash_ctl; bool found_failed =3D false; - int flags; =20 /* Initialize a hash tracking all processed databases */ memset(&hash_ctl, 0, sizeof(hash_ctl)); @@ -936,8 +953,12 @@ ProcessAllDatabases(bool immediate_checkpoint) * were created during our runtime. Since a database can be = created as a * copy of any other database (which may not have existed in our = last * run), we have to repeat this loop until no new databases show = up in the - * list. + * list. Here the initial list for the loop processing is = generated after + * waiting for all existing transactions to finish to ensure = that we can + * see any database which was created even if the transaction in = which it + * was created started before checksums were being processed. */ + WaitForAllTransactionsToFinish(); DatabaseList =3D BuildDatabaseList(); =20 /* Allow a test case to modify the initial list of databases */ @@ -1060,9 +1081,14 @@ ProcessAllDatabases(bool immediate_checkpoint) break; =20 /* - * Re-generate the list of databases for another pass. = Since we wait - * for all pre-existing transactions finish, this way we = can be - * certain that there are no databases left without = checksums. + * Re-generate the list of databases for another pass in = case any new + * databases were created while we were running. Since = the initial + * list was generated efter waiting for all transaction = to finish we + * know that all new databases found here must have been = created while + * seeing the new checksum state. By waiting for all = transactions + * here as well we know that any database created using = an existing db + * as a template (which may have been used before it had = checksums + * enabled) will be committed. */ WaitForAllTransactionsToFinish(); DatabaseList =3D BuildDatabaseList(); @@ -1078,7 +1104,6 @@ ProcessAllDatabases(bool immediate_checkpoint) * still exists, but enabling checksums failed then we fail the = entire * checksumming process and exit with an error. */ - WaitForAllTransactionsToFinish(); DatabaseList =3D BuildDatabaseList(); =20 foreach_ptr(DataChecksumsWorkerDatabase, db, DatabaseList) @@ -1109,35 +1134,13 @@ ProcessAllDatabases(bool immediate_checkpoint) if (found_failed) { /* Disable checksums on cluster, because we failed */ - = SetDataChecksumsOff(DataChecksumsWorkerShmem->immediate_checkpoint); - /* Force a checkpoint to make everything consistent */ - flags =3D CHECKPOINT_FORCE | CHECKPOINT_WAIT; - if (immediate_checkpoint) - flags =3D flags | CHECKPOINT_FAST; - RequestCheckpoint(flags); + SetDataChecksumsOff(); ereport(ERROR, errcode(ERRCODE_INSUFFICIENT_RESOURCES), errmsg("data checksums failed to get = enabled in all databases, aborting"), errhint("The server log might have more = information on the cause of the error.")); } =20 - /* - * When enabling checksums, we have to wait for a checkpoint for = the - * checksums to change from in-progress to on. - */ - pgstat_progress_update_param(PROGRESS_DATACHECKSUMS_PHASE, - = PROGRESS_DATACHECKSUMS_PHASE_WAITING_CHECKPOINT); - - /* - * Force a checkpoint to get everything out to disk. The use of = immediate - * checkpoints is for running tests, as they would otherwise not = execute - * in such a way that they can reliably be placed under timeout = control. - */ - flags =3D CHECKPOINT_FORCE | CHECKPOINT_WAIT; - if (immediate_checkpoint) - flags =3D flags | CHECKPOINT_FAST; - RequestCheckpoint(flags); - pgstat_progress_update_param(PROGRESS_DATACHECKSUMS_PHASE, = PROGRESS_DATACHECKSUMS_PHASE_WAITING_BARRIER); return true; @@ -1178,12 +1181,11 @@ DataChecksumsWorkerShmemInit(void) =20 /* * Even if this is a redundant assignment, we want to be = explicit - * about our intent for readability, since we want to be = able to query - * this state in case of restartability. + * about our intent for readability. */ DataChecksumsWorkerShmem->launch_operation =3D false; DataChecksumsWorkerShmem->launcher_running =3D false; - DataChecksumsWorkerShmem->launch_fast =3D false; + DataChecksumsWorkerShmem->worker_running =3D false; } } =20 @@ -1364,7 +1366,11 @@ DataChecksumsWorkerMain(Datum arg) InitialTempTableList =3D BuildRelationList(true, false); =20 /* - * Enable vacuum cost delay, if any. + * Enable vacuum cost delay, if any. While this process isn't = doing any + * vacuuming, we are re-using the infrastructure that vacuum = cost delay + * provides rather than inventing something bespoke. This is an = internal + * implementation detail and care should be taken to avoid it = bleeding + * through to the user to avoid confusion. */ Assert(DataChecksumsWorkerShmem->operation =3D=3D = ENABLE_DATACHECKSUMS); VacuumCostDelay =3D DataChecksumsWorkerShmem->cost_delay; diff --git a/src/backend/postmaster/walsummarizer.c = b/src/backend/postmaster/walsummarizer.c index c3d56c866d3..afef118d1a4 100644 --- a/src/backend/postmaster/walsummarizer.c +++ b/src/backend/postmaster/walsummarizer.c @@ -1430,8 +1430,11 @@ SummarizeXlogRecord(XLogReaderState *xlogreader, = bool *new_fast_forward) =20 if (info =3D=3D XLOG_CHECKPOINT_REDO) { + xl_checkpoint_redo xlrec; + /* Payload is wal_level at the time record was written. = */ - memcpy(&record_wal_level, XLogRecGetData(xlogreader), = sizeof(int)); + memcpy(&xlrec, XLogRecGetData(xlogreader), = sizeof(xl_checkpoint_redo)); + record_wal_level =3D xlrec.wal_level; } else if (info =3D=3D XLOG_CHECKPOINT_SHUTDOWN) { diff --git a/src/backend/storage/page/bufpage.c = b/src/backend/storage/page/bufpage.c index 8b67ba40f0f..72e1c042cf1 100644 --- a/src/backend/storage/page/bufpage.c +++ b/src/backend/storage/page/bufpage.c @@ -107,6 +107,7 @@ PageIsVerified(PageData *page, BlockNumber blkno, = int flags, bool *checksum_fail */ if (!PageIsNew(page)) { + HOLD_INTERRUPTS(); if (DataChecksumsNeedVerify()) { checksum =3D pg_checksum_page(page, blkno); @@ -118,6 +119,7 @@ PageIsVerified(PageData *page, BlockNumber blkno, = int flags, bool *checksum_fail *checksum_failure_p =3D true; } } + RESUME_INTERRUPTS(); =20 /* * The following checks don't prove the header is = correct, only that @@ -1510,9 +1512,13 @@ PageSetChecksumCopy(Page page, BlockNumber blkno) { static char *pageCopy =3D NULL; =20 + HOLD_INTERRUPTS(); /* If we don't need a checksum, just return the passed-in data = */ if (PageIsNew(page) || !DataChecksumsNeedWrite()) + { + RESUME_INTERRUPTS(); return page; + } =20 /* * We allocate the copy space once and use it over on each = subsequent @@ -1528,6 +1534,7 @@ PageSetChecksumCopy(Page page, BlockNumber blkno) =20 memcpy(pageCopy, page, BLCKSZ); ((PageHeader) pageCopy)->pd_checksum =3D = pg_checksum_page(pageCopy, blkno); + RESUME_INTERRUPTS(); return pageCopy; } =20 @@ -1540,9 +1547,14 @@ PageSetChecksumCopy(Page page, BlockNumber blkno) void PageSetChecksumInplace(Page page, BlockNumber blkno) { + HOLD_INTERRUPTS(); /* If we don't need a checksum, just return */ if (PageIsNew(page) || !DataChecksumsNeedWrite()) + { + RESUME_INTERRUPTS(); return; + } =20 ((PageHeader) page)->pd_checksum =3D pg_checksum_page(page, = blkno); + RESUME_INTERRUPTS(); } diff --git a/src/backend/utils/init/postinit.c = b/src/backend/utils/init/postinit.c index d3dc55e3820..34113a7e8bb 100644 --- a/src/backend/utils/init/postinit.c +++ b/src/backend/utils/init/postinit.c @@ -755,7 +755,7 @@ InitPostgres(const char *in_dbname, Oid dboid, * * This intentionally happens after initializing the procsignal, = otherwise * we might miss a state change. This means we can get a barrier = for the - * state we've just initialized - but it can happen only once. + * state we've just initialized. * * The postmaster (which is what gets forked into the new child = process) * does not handle barriers, therefore it may not have the = current value diff --git a/src/bin/pg_controldata/pg_controldata.c = b/src/bin/pg_controldata/pg_controldata.c index ca4957b2fa1..47afa82d9c5 100644 --- a/src/bin/pg_controldata/pg_controldata.c +++ b/src/bin/pg_controldata/pg_controldata.c @@ -288,7 +288,7 @@ main(int argc, char *argv[]) printf(_("Latest checkpoint's newestCommitTsXid:%u\n"), ControlFile->checkPointCopy.newestCommitTsXid); printf(_("Latest checkpoint's data_checksum_version:%u\n"), - ControlFile->checkPointCopy.data_checksum_version); + ControlFile->checkPointCopy.dataChecksumVersion); printf(_("Time of latest checkpoint: %s\n"), ckpttime_str); printf(_("Fake LSN counter for unlogged rels: %X/%08X\n"), diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index 49518ea6b2a..352ce0477d4 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -247,9 +247,9 @@ extern bool DataChecksumsNeedWrite(void); extern bool DataChecksumsNeedVerify(void); extern bool DataChecksumsOnInProgress(void); extern bool DataChecksumsOffInProgress(void); -extern void SetDataChecksumsOnInProgress(bool immediate_checkpoint); -extern void SetDataChecksumsOn(bool immediate_checkpoint); -extern void SetDataChecksumsOff(bool immediate_checkpoint); +extern void SetDataChecksumsOnInProgress(void); +extern void SetDataChecksumsOn(void); +extern void SetDataChecksumsOff(void); extern bool AbsorbDataChecksumsBarrier(int target_state); extern const char *show_data_checksums(void); extern void InitLocalDataChecksumVersion(void); diff --git a/src/include/access/xlog_internal.h = b/src/include/access/xlog_internal.h index 98a8a4ac384..53d1e2bebfd 100644 --- a/src/include/access/xlog_internal.h +++ b/src/include/access/xlog_internal.h @@ -293,7 +293,7 @@ typedef struct xl_restore_point /* Information logged when data checksum level is changed */ typedef struct xl_checksum_state { - uint32 new_checksumtype; + ChecksumType new_checksumtype; } xl_checksum_state; =20 /* Overwrite of prior contrecord */ @@ -312,6 +312,12 @@ typedef struct xl_end_of_recovery int wal_level; } xl_end_of_recovery; =20 +typedef struct xl_checkpoint_redo +{ + int wal_level; + uint32 data_checksum_version; +} xl_checkpoint_redo; + /* * The functions in xloginsert.c construct a chain of XLogRecData = structs * to represent the final WAL record. diff --git a/src/include/catalog/pg_control.h = b/src/include/catalog/pg_control.h index 4201003717b..7290ece5205 100644 --- a/src/include/catalog/pg_control.h +++ b/src/include/catalog/pg_control.h @@ -64,8 +64,8 @@ typedef struct CheckPoint */ TransactionId oldestActiveXid; =20 - /* data checksums at the time of the checkpoint */ - uint32 data_checksum_version; + /* data checksums state at the time of the checkpoint */ + uint32 dataChecksumVersion; } CheckPoint; =20 /* XLOG info values for XLOG rmgr */ @@ -85,7 +85,7 @@ typedef struct CheckPoint #define XLOG_OVERWRITE_CONTRECORD 0xD0 #define XLOG_CHECKPOINT_REDO 0xE0 #define XLOG_LOGICAL_DECODING_STATUS_CHANGE 0xF0 -#define XLOG_CHECKSUMS 0xF1 +#define XLOG_CHECKSUMS 0xC0 =20 =20 /* diff --git a/src/include/catalog/pg_proc.dat = b/src/include/catalog/pg_proc.dat index 92d56e60719..d1e8d3cd1e3 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -12408,20 +12408,13 @@ # data checksum management functions { oid =3D> '9258', descr =3D> 'disable data checksums', - proname =3D> 'pg_disable_data_checksums', provolatile =3D> 'v', = prorettype =3D> 'void', - proparallel =3D> 'r', - proargtypes =3D> 'bool', proallargtypes =3D> '{bool}', - proargmodes =3D> '{i}', - proargnames =3D> '{fast}', - prosrc =3D> 'disable_data_checksums' }, - + proname =3D> 'pg_disable_data_checksums', provolatile =3D> 's', = prorettype =3D> 'void', + proparallel =3D> 'r', prosrc =3D> 'disable_data_checksums', = proargtypes =3D> '' }, { oid =3D> '9257', descr =3D> 'enable data checksums', proname =3D> 'pg_enable_data_checksums', provolatile =3D> 'v', = prorettype =3D> 'void', - proparallel =3D> 'r', - proargtypes =3D> 'int4 int4 bool', proallargtypes =3D> = '{int4,int4,bool}', - proargmodes =3D> '{i,i,i}', - proargnames =3D> '{cost_delay,cost_limit,fast}', + proparallel =3D> 'r', proargtypes =3D> 'int4 int4', proallargtypes =3D>= '{int4,int4}', + proargmodes =3D> '{i,i}', proargnames =3D> '{cost_delay,cost_limit}', prosrc =3D> 'enable_data_checksums' }, =20 # collation management functions diff --git a/src/include/commands/progress.h = b/src/include/commands/progress.h index 6abe7456d3c..cdaf20a9243 100644 --- a/src/include/commands/progress.h +++ b/src/include/commands/progress.h @@ -192,8 +192,7 @@ #define PROGRESS_DATACHECKSUMS_PHASE_ENABLING 0 #define PROGRESS_DATACHECKSUMS_PHASE_DISABLING 1 #define PROGRESS_DATACHECKSUMS_PHASE_WAITING_TEMPREL 2 -#define PROGRESS_DATACHECKSUMS_PHASE_WAITING_CHECKPOINT 3 -#define PROGRESS_DATACHECKSUMS_PHASE_WAITING_BARRIER 4 -#define PROGRESS_DATACHECKSUMS_PHASE_DONE = 5 +#define PROGRESS_DATACHECKSUMS_PHASE_WAITING_BARRIER 3 +#define PROGRESS_DATACHECKSUMS_PHASE_DONE = 4 =20 #endif diff --git a/src/include/postmaster/datachecksumsworker.h = b/src/include/postmaster/datachecksumsworker.h index 0daef709ec8..dc4bf24be7c 100644 --- a/src/include/postmaster/datachecksumsworker.h +++ b/src/include/postmaster/datachecksumsworker.h @@ -4,7 +4,7 @@ * header file for data checksum helper background worker * * - * Portions Copyright (c) 1996-2025, PostgreSQL Global Development = Group + * Portions Copyright (c) 1996-2026, PostgreSQL Global Development = Group * Portions Copyright (c) 1994, Regents of the University of California * * src/include/postmaster/datachecksumsworker.h @@ -41,8 +41,7 @@ typedef enum /* Start the background processes for enabling or disabling checksums = */ void = StartDataChecksumsWorkerLauncher(DataChecksumsWorkerOperation op, = int cost_delay, - = int cost_limit, - = bool fast); + = int cost_limit); =20 /* Background worker entrypoints */ void DataChecksumsWorkerLauncherMain(Datum arg); diff --git a/src/include/storage/checksum.h = b/src/include/storage/checksum.h index d22687e2e4c..b6bbba82949 100644 --- a/src/include/storage/checksum.h +++ b/src/include/storage/checksum.h @@ -26,8 +26,7 @@ typedef enum ChecksumType PG_DATA_CHECKSUM_OFF =3D 0, PG_DATA_CHECKSUM_VERSION, PG_DATA_CHECKSUM_INPROGRESS_ON_VERSION, - PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION, - PG_DATA_CHECKSUM_ANY_VERSION + PG_DATA_CHECKSUM_INPROGRESS_OFF_VERSION } ChecksumType; =20 /* diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h index bd2af5753a2..45b4b312ab9 100644 --- a/src/include/storage/proc.h +++ b/src/include/storage/proc.h @@ -462,11 +462,10 @@ extern PGDLLIMPORT PGPROC *PreparedXactProcs; * Background writer, checkpointer, WAL writer, WAL summarizer, and = archiver * run during normal operation. Startup process and WAL receiver also = consume * 2 slots, but WAL writer is launched only after startup has exited, = so we - * only need 6 slots to cover these. The DataChecksums worker and = launcher - * can consume 2 slots when data checksums are enabled or disabled. + * only need 6 slots to cover these. */ #define MAX_IO_WORKERS 32 -#define NUM_AUXILIARY_PROCS (8 + MAX_IO_WORKERS) +#define NUM_AUXILIARY_PROCS (6 + MAX_IO_WORKERS) =20 /* configurable options */ extern PGDLLIMPORT int DeadlockTimeout; diff --git a/src/test/modules/test_checksums/Makefile = b/src/test/modules/test_checksums/Makefile index a5b6259a728..fe054cfb88e 100644 --- a/src/test/modules/test_checksums/Makefile +++ b/src/test/modules/test_checksums/Makefile @@ -2,7 +2,7 @@ # # Makefile for src/test/checksum # -# Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group +# Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group # Portions Copyright (c) 1994, Regents of the University of California # # src/test/checksum/Makefile diff --git a/src/test/modules/test_checksums/meson.build = b/src/test/modules/test_checksums/meson.build index ffc737ca87a..3e6a155f35f 100644 --- a/src/test/modules/test_checksums/meson.build +++ b/src/test/modules/test_checksums/meson.build @@ -1,4 +1,4 @@ -# Copyright (c) 2025, PostgreSQL Global Development Group +# Copyright (c) 2026, PostgreSQL Global Development Group =20 test_checksums_sources =3D files( 'test_checksums.c', diff --git a/src/test/modules/test_checksums/t/001_basic.pl = b/src/test/modules/test_checksums/t/001_basic.pl index 728a5c4510c..397f6411cb5 100644 --- a/src/test/modules/test_checksums/t/001_basic.pl +++ b/src/test/modules/test_checksums/t/001_basic.pl @@ -1,5 +1,5 @@ =20 -# Copyright (c) 2025, PostgreSQL Global Development Group +# Copyright (c) 2026, PostgreSQL Global Development Group =20 # Test suite for testing enabling data checksums in an online cluster use strict; diff --git a/src/test/modules/test_checksums/t/002_restarts.pl = b/src/test/modules/test_checksums/t/002_restarts.pl index 6c17f304eac..a4d35e9f95c 100644 --- a/src/test/modules/test_checksums/t/002_restarts.pl +++ b/src/test/modules/test_checksums/t/002_restarts.pl @@ -1,5 +1,5 @@ =20 -# Copyright (c) 2024, PostgreSQL Global Development Group +# Copyright (c) 2026, PostgreSQL Global Development Group =20 # Test suite for testing enabling data checksums in an online cluster = with a # restart which breaks processing. diff --git a/src/test/modules/test_checksums/t/003_standby_restarts.pl = b/src/test/modules/test_checksums/t/003_standby_restarts.pl index f724d4ea74c..e9bf21fe5d5 100644 --- a/src/test/modules/test_checksums/t/003_standby_restarts.pl +++ b/src/test/modules/test_checksums/t/003_standby_restarts.pl @@ -1,5 +1,5 @@ =20 -# Copyright (c) 2024, PostgreSQL Global Development Group +# Copyright (c) 2026, PostgreSQL Global Development Group =20 # Test suite for testing enabling data checksums in an online cluster = with # streaming replication diff --git a/src/test/modules/test_checksums/t/004_offline.pl = b/src/test/modules/test_checksums/t/004_offline.pl index e9fbcf77eab..f33e278de28 100644 --- a/src/test/modules/test_checksums/t/004_offline.pl +++ b/src/test/modules/test_checksums/t/004_offline.pl @@ -1,5 +1,5 @@ =20 -# Copyright (c) 2024, PostgreSQL Global Development Group +# Copyright (c) 2026, PostgreSQL Global Development Group =20 # Test suite for testing enabling data checksums offline from various = states # of checksum processing diff --git a/src/test/modules/test_checksums/t/005_injection.pl = b/src/test/modules/test_checksums/t/005_injection.pl index ae801cd336f..6d374793eec 100644 --- a/src/test/modules/test_checksums/t/005_injection.pl +++ b/src/test/modules/test_checksums/t/005_injection.pl @@ -1,5 +1,5 @@ =20 -# Copyright (c) 2025, PostgreSQL Global Development Group +# Copyright (c) 2026, PostgreSQL Global Development Group =20 # Test suite for testing enabling data checksums in an online cluster = with # injection point tests injecting failures into the processing @@ -43,53 +43,8 @@ $node->safe_psql('postgres', 'SELECT = dcw_inject_fail_database(true);'); enable_data_checksums($node, wait =3D> 'off'); $node->safe_psql('postgres', 'SELECT = dcw_inject_fail_database(false);'); =20 -# Force the server to crash after enabling data checksums but before = issuing -# the checkpoint. Since the switch has been WAL logged the server = should come -# up with checksums enabled after replay. -test_checksum_state($node, 'off'); -$node->safe_psql('postgres', 'SELECT dc_crash_before_checkpoint();'); -enable_data_checksums($node, fast =3D> 'true'); -my $ret =3D wait_for_cluster_crash($node); -ok($ret =3D=3D 1, "Cluster crash detection timeout"); -ok(!$node->is_alive, "Cluster crashed due to abort() before = checkpointing"); -$node->_update_pid(-1); -$node->start; -test_checksum_state($node, 'on'); - -# Another test just like the previous, but for disabling data checksums = (and -# crashing just before checkpointing). The previous injection points = were all -# detached from through the crash so they need to be reattached. -$node->safe_psql('postgres', 'SELECT dc_crash_before_checkpoint();'); +# Make sure that disabling after a failure works disable_data_checksums($node); -$ret =3D wait_for_cluster_crash($node); -ok($ret =3D=3D 1, "Cluster crash detection timeout"); -ok(!$node->is_alive, "Cluster crashed due to abort() before = checkpointing"); -$node->_update_pid(-1); -$node->start; -test_checksum_state($node, 'off'); - -# Now inject a crash before inserting the WAL record for data checksum = state -# change, when the server comes back up again the state should not have = been -# set to the new value since the process didn't succeed. -$node->safe_psql('postgres', 'SELECT dc_crash_before_xlog();'); -enable_data_checksums($node); -$ret =3D wait_for_cluster_crash($node); -ok($ret =3D=3D 1, "Cluster crash detection timeout"); -ok(!$node->is_alive, "Cluster crashed"); -$node->_update_pid(-1); -$node->start; -test_checksum_state($node, 'off'); - -# This re-runs the same test again but with first disabling data = checksums and -# then enabling again, crashing right before inserting the WAL record. = When -# it comes back up the checksums must not be enabled. -$node->safe_psql('postgres', 'SELECT dc_crash_before_xlog();'); -enable_data_checksums($node); -$ret =3D wait_for_cluster_crash($node); -ok($ret =3D=3D 1, "Cluster crash detection timeout"); -ok(!$node->is_alive, "Cluster crashed"); -$node->_update_pid(-1); -$node->start; test_checksum_state($node, 'off'); =20 # = --------------------------------------------------------------------------= - @@ -122,5 +77,33 @@ SKIP: enable_data_checksums($node, wait =3D> 'on'); } =20 +# Test starting the data checksums launcher from two sessions = concurrently. In +# order reliably run things concurrently we use background psql's which = block +# on various fault injected delays. +enable_data_checksums($node, wait =3D> 'on'); + +my $A =3D $node->background_psql('postgres'); +$A->query_safe('SELECT dcw_inject_launcher_delay(true);'); +$A->query_safe('SELECT pg_enable_data_checksums();'); +$node->safe_psql('postgres', 'SELECT = dcw_inject_launcher_delay(false);'); +my ($ret, $stdout, $stderr) =3D + $node->psql('postgres', 'SELECT pg_disable_data_checksums();'); +wait_for_checksum_state($node, 'off'); + + +my $B =3D $node->background_psql('postgres'); +$B->query_safe('SELECT dcw_inject_startup_delay(true);'); +$B->query_safe('SELECT pg_enable_data_checksums();'); +$node->safe_psql('postgres', 'SELECT = dcw_inject_startup_delay(false);'); +($ret, $stdout, $stderr) =3D + $node->psql('postgres', 'SELECT pg_enable_data_checksums();'); +like($stderr, qr/data checksum processing already running/, + 'ERROR on stderr'); +wait_for_checksum_state($node, 'on'); + +$A->quit; +$B->quit; + + $node->stop; done_testing(); diff --git a/src/test/modules/test_checksums/t/006_pgbench_single.pl = b/src/test/modules/test_checksums/t/006_pgbench_single.pl index 96f3b2cd8a6..6bb0e67c7e1 100644 --- a/src/test/modules/test_checksums/t/006_pgbench_single.pl +++ b/src/test/modules/test_checksums/t/006_pgbench_single.pl @@ -1,5 +1,5 @@ =20 -# Copyright (c) 2025, PostgreSQL Global Development Group +# Copyright (c) 2026, PostgreSQL Global Development Group =20 # Test suite for testing enabling data checksums in an online cluster = with # concurrent activity via pgbench runs @@ -39,10 +39,6 @@ my $TEST_ITERATIONS =3D 10; my $data_checksum_state =3D 'off'; my $pgbench =3D undef; =20 -# determines whether enable_data_checksums/disable_data_checksums = forces an -# immediate checkpoint -my @flip_modes =3D ('true', 'false'); - # Start a pgbench run in the background against the server specified = via the # port passed as parameter. sub background_rw_pgbench @@ -50,10 +46,7 @@ sub background_rw_pgbench my $port =3D shift; =20 # If a previous pgbench is still running, start by shutting it = down. - if ($pgbench) - { - $pgbench->finish; - } + $pgbench->finish if $pgbench; =20 # Randomize the number of pgbench clients a bit (range 1-16) my $clients =3D 1 + int(rand(15)); @@ -92,13 +85,8 @@ sub flip_data_checksums $node->safe_psql('postgres', "SELECT = pg_current_wal_lsn()"); note("LSN before enabling: " . $result . "\n"); =20 - my $mode =3D $flip_modes[ int(rand(@flip_modes)) ]; - # Ensure that the primary switches to "inprogress-on" - enable_data_checksums( - $node, - wait =3D> 'inprogress-on', - 'fast' =3D> $mode); + enable_data_checksums($node, wait =3D> 'inprogress-on'); =20 random_sleep(); =20 @@ -123,9 +111,7 @@ sub flip_data_checksums $node->safe_psql('postgres', "SELECT = pg_current_wal_lsn()"); note("LSN before disabling: " . $result . "\n"); =20 - my $mode =3D $flip_modes[ int(rand(@flip_modes)) ]; - - disable_data_checksums($node, 'fast' =3D> $mode); + disable_data_checksums($node); =20 # Wait for checksums disabled on the primary wait_for_checksum_state($node, 'off'); @@ -141,8 +127,7 @@ sub flip_data_checksums else { # This should only happen due to programmer error when = hacking on the - # test code, but since that might pass subtly by let's = ensure it gets - # caught with a test error if so. + # test code, but since that might pass subtly we error = out. bail('data_checksum_state variable has invalid state:' . $data_checksum_state); } @@ -172,7 +157,7 @@ background_rw_pgbench($node->port); =20 # Main test suite. This loop will start a pgbench run on the cluster = and while # that's running flip the state of data checksums concurrently. It will = then -# randomly restart thec cluster (in fast or immediate) mode and then = check for +# randomly restart the cluster and then check for # the desired state. The idea behind doing things randomly is to = stress out # any timing related issues by subjecting the cluster for varied = workloads. # A TODO is to generate a trace such that any test failure can be = traced to @@ -183,8 +168,6 @@ for (my $i =3D 0; $i < $TEST_ITERATIONS; $i++) =20 if (!$node->is_alive) { - random_sleep(); - # Start, to do recovery, and stop $node->start; $node->stop('fast'); @@ -203,6 +186,7 @@ for (my $i =3D 0; $i < $TEST_ITERATIONS; $i++) # Randomize the WAL size, to trigger checkpoints = less/more often my $sb =3D 64 + int(rand(1024)); $node->append_conf('postgresql.conf', qq[max_wal_size =3D = $sb]); + note("changing max_wal_size to " . $sb); =20 $node->start; =20 diff --git a/src/test/modules/test_checksums/t/007_pgbench_standby.pl = b/src/test/modules/test_checksums/t/007_pgbench_standby.pl index 8b8e031cbf6..a7365496853 100644 --- a/src/test/modules/test_checksums/t/007_pgbench_standby.pl +++ b/src/test/modules/test_checksums/t/007_pgbench_standby.pl @@ -1,5 +1,5 @@ =20 -# Copyright (c) 2025, PostgreSQL Global Development Group +# Copyright (c) 2026, PostgreSQL Global Development Group =20 # Test suite for testing enabling data checksums in an online cluster, # comprising of a primary and a replicated standby, with concurrent = activity @@ -17,6 +17,16 @@ use lib $FindBin::RealBin; =20 use DataChecksums::Utils; =20 +if (!$ENV{PG_TEST_EXTRA} || $ENV{PG_TEST_EXTRA} !~ = /\bchecksum_extended\b/) +{ + plan skip_all =3D> 'Extended checksum tests not enabled'; +} + +if ($ENV{enable_injection_points} ne 'yes') +{ + plan skip_all =3D> 'Injection points not supported by this = build'; +} + my $node_primary_slot =3D 'physical_slot'; my $node_primary_backup =3D 'primary_backup'; my $node_primary; @@ -36,27 +46,6 @@ my $data_checksum_state =3D 'off'; my $pgbench_primary =3D undef; my $pgbench_standby =3D undef; =20 -# Variables holding state for managing the cluster and aux processes in -# various ways -my ($pgb_primary_stdin, $pgb_primary_stdout, $pgb_primary_stderr) =3D - ('', '', ''); -my ($pgb_standby_1_stdin, $pgb_standby_1_stdout, $pgb_standby_1_stderr) = =3D - ('', '', ''); - -if (!$ENV{PG_TEST_EXTRA} || $ENV{PG_TEST_EXTRA} !~ = /\bchecksum_extended\b/) -{ - plan skip_all =3D> 'Extended tests not enabled'; -} - -if ($ENV{enable_injection_points} ne 'yes') -{ - plan skip_all =3D> 'Injection points not supported by this = build'; -} - -# determines whether enable_data_checksums/disable_data_checksums = forces an -# immediate checkpoint -my @flip_modes =3D ('true', 'false'); - # Start a pgbench run in the background against the server specified = via the # port passed as parameter sub background_pgbench @@ -66,6 +55,7 @@ sub background_pgbench # Terminate any currently running pgbench process before = continuing $pgbench_primary->finish if $pgbench_primary; =20 + # Randomize the number of pgbench clients a bit (range 1-16) my $clients =3D 1 + int(rand(15)); =20 my @cmd =3D ('pgbench', '-p', $port, '-T', '600', '-c', = $clients); @@ -77,7 +67,7 @@ sub background_pgbench push(@cmd, 'postgres'); =20 $pgbench_primary =3D IPC::Run::start( - [ 'pgbench', '-p', $port, '-T', '600', '-c', $clients, = 'postgres' ], + \@cmd, '<' =3D> '/dev/null', '>' =3D> '/dev/null', '2>' =3D> '/dev/null', @@ -89,14 +79,14 @@ sub background_pgbench # before and after state. sub flip_data_checksums { + # First, make sure the cluster is in the state we expect it to = be test_checksum_state($node_primary, $data_checksum_state); test_checksum_state($node_standby_1, $data_checksum_state); =20 if ($data_checksum_state eq 'off') { # Coin-toss to see if we are injecting a retry due to a = temptable - $node_primary->safe_psql('postgres', - 'SELECT dcw_fake_temptable(true);') + $node_primary->safe_psql('postgres', 'SELECT = dcw_fake_temptable();') if cointoss(); =20 # log LSN right before we start changing checksums @@ -104,14 +94,11 @@ sub flip_data_checksums $node_primary->safe_psql('postgres', "SELECT = pg_current_wal_lsn()"); note("LSN before enabling: " . $result . "\n"); =20 - my $mode =3D $flip_modes[ int(rand(@flip_modes)) ]; - # Ensure that the primary switches to "inprogress-on" - enable_data_checksums( - $node_primary, - wait =3D> 'inprogress-on', - 'fast' =3D> $mode); + enable_data_checksums($node_primary, wait =3D> = 'inprogress-on'); + random_sleep(); + # Wait for checksum enable to be replayed $node_primary->wait_for_catchup($node_standby_1, = 'replay'); =20 @@ -127,7 +114,6 @@ sub flip_data_checksums 'f'); is($result, 1, 'ensure standby has absorbed the inprogress-on = barrier'); - random_sleep(); $result =3D $node_standby_1->safe_psql('postgres', "SELECT setting " . "FROM pg_catalog.pg_settings " @@ -160,9 +146,7 @@ sub flip_data_checksums $node_primary->safe_psql('postgres', "SELECT = pg_current_wal_lsn()"); note("LSN before disabling: " . $result . "\n"); =20 - my $mode =3D $flip_modes[ int(rand(@flip_modes)) ]; - - disable_data_checksums($node_primary, 'fast' =3D> = $mode); + disable_data_checksums($node_primary); $node_primary->wait_for_catchup($node_standby_1, = 'replay'); =20 # Wait for checksums disabled on the primary and standby @@ -182,9 +166,9 @@ sub flip_data_checksums else { # This should only happen due to programmer error when = hacking on the - # test code, but since that might pass subtly by let's = ensure it gets - # caught with a test error if so. - is(1, 0, 'data_checksum_state variable has invalid = state'); + # test code, but since that might pass subtly we error = out. + bail('data_checksum_state variable has invalid state:' + . $data_checksum_state); } } =20 @@ -229,7 +213,7 @@ background_pgbench($node_primary->port, 0); =20 # Main test suite. This loop will start a pgbench run on the cluster = and while # that's running flip the state of data checksums concurrently. It will = then -# randomly restart the cluster (in fast or immediate) mode and then = check for +# randomly restart the cluster and then check for # the desired state. The idea behind doing things randomly is to = stress out # any timing related issues by subjecting the cluster for varied = workloads. # A TODO is to generate a trace such that any test failure can be = traced to @@ -240,8 +224,6 @@ for (my $i =3D 0; $i < $TEST_ITERATIONS; $i++) =20 if (!$node_primary->is_alive) { - random_sleep(); - # start, to do recovery, and stop $node_primary->start; $node_primary->stop('fast'); @@ -271,8 +253,6 @@ for (my $i =3D 0; $i < $TEST_ITERATIONS; $i++) =20 if (!$node_standby_1->is_alive) { - random_sleep(); - $node_standby_1->start; $node_standby_1->stop('fast'); =20 @@ -297,20 +277,18 @@ for (my $i =3D 0; $i < $TEST_ITERATIONS; $i++) =20 $node_standby_1->start; =20 - # Start a select-only pgbench in the background on the = standby + # Start a read-only pgbench in the background on the = standby background_pgbench($node_standby_1->port, 1); } =20 $node_primary->safe_psql('postgres', "UPDATE t SET a =3D a + = 1;"); + $node_primary->wait_for_catchup($node_standby_1, 'write'); =20 flip_data_checksums(); random_sleep(); my $result =3D $node_primary->safe_psql('postgres', "SELECT count(*) FROM t WHERE a > 1"); is($result, '100000', 'ensure data pages can be read back on = primary'); - random_sleep(); - $node_primary->wait_for_catchup($node_standby_1, 'write'); - random_sleep(); =20 # Potentially powercycle the cluster (the nodes independently) diff --git a/src/test/modules/test_checksums/t/DataChecksums/Utils.pm = b/src/test/modules/test_checksums/t/DataChecksums/Utils.pm index cf670be944c..d299a1ba01d 100644 --- a/src/test/modules/test_checksums/t/DataChecksums/Utils.pm +++ b/src/test/modules/test_checksums/t/DataChecksums/Utils.pm @@ -1,5 +1,5 @@ =20 -# Copyright (c) 2025, PostgreSQL Global Development Group +# Copyright (c) 2026, PostgreSQL Global Development Group =20 =3Dpod =20 @@ -149,12 +149,6 @@ The B to use when enabling data = checksums, default is 0. =20 The B to use when enabling data checksums, default is 100. =20 -=3Ditem fast - -If set to C an immediate checkpoint will be issued after data -checksums are enabled. Setting this to false will lead to slower = tests. -The default is C. - =3Ditem wait =20 If defined, the function will wait for the state defined in this = parameter, @@ -173,16 +167,13 @@ sub enable_data_checksums # Set sane defaults for the parameters $params{cost_delay} =3D 0 unless (defined($params{cost_delay})); $params{cost_limit} =3D 100 unless = (defined($params{cost_limit})); - $params{fast} =3D 'true' unless (defined($params{fast})); =20 my $query =3D <<'EOQ'; -SELECT pg_enable_data_checksums(%s, %s, %s); +SELECT pg_enable_data_checksums(%s, %s); EOQ =20 - $postgresnode->safe_psql( - 'postgres', - sprintf($query, - $params{cost_delay}, $params{cost_limit}, = $params{fast})); + $postgresnode->safe_psql('postgres', + sprintf($query, $params{cost_delay}, = $params{cost_limit})); =20 wait_for_checksum_state($postgresnode, $params{wait}) if (defined($params{wait})); @@ -201,13 +192,6 @@ waiting timing out, before returning. The function = will wait for $PostgreSQL::Test::Utils::timeout_default seconds before timing out. Unlike in C the value of the parameter is = discarded. =20 -=3Dover - -=3Ditem fast - -If set to C the checkpoint after disabling will be set to = immediate, else -it will be deferred. The default if no value is set is B. - =3Dback =20 =3Dcut @@ -217,14 +201,8 @@ sub disable_data_checksums my $postgresnode =3D shift; my %params =3D @_; =20 - # Set sane defaults for the parameters - $params{fast} =3D 'true' unless (defined($params{fast})); - - my $query =3D <<'EOQ'; -SELECT pg_disable_data_checksums(%s); -EOQ - - $postgresnode->safe_psql('postgres', sprintf($query, = $params{fast})); + $postgresnode->safe_psql('postgres', + 'SELECT pg_disable_data_checksums();'); =20 wait_for_checksum_state($postgresnode, 'off') if = (defined($params{wait})); } diff --git a/src/test/modules/test_checksums/test_checksums--1.0.sql = b/src/test/modules/test_checksums/test_checksums--1.0.sql index aa086d5c430..8322b64a6ae 100644 --- a/src/test/modules/test_checksums/test_checksums--1.0.sql +++ b/src/test/modules/test_checksums/test_checksums--1.0.sql @@ -7,22 +7,26 @@ CREATE FUNCTION dcw_inject_delay_barrier(attach = boolean DEFAULT true) RETURNS pg_catalog.void AS 'MODULE_PATHNAME' LANGUAGE C; =20 -CREATE FUNCTION dcw_inject_fail_database(attach boolean DEFAULT true) +CREATE FUNCTION dcw_inject_launcher_delay(attach boolean DEFAULT true) RETURNS pg_catalog.void AS 'MODULE_PATHNAME' LANGUAGE C; =20 -CREATE FUNCTION dcw_prune_dblist(attach boolean DEFAULT true) +CREATE FUNCTION dcw_inject_startup_delay(attach boolean DEFAULT true) RETURNS pg_catalog.void AS 'MODULE_PATHNAME' LANGUAGE C; =20 -CREATE FUNCTION dcw_fake_temptable(attach boolean DEFAULT true) +CREATE FUNCTION dcw_inject_fail_database(attach boolean DEFAULT true) RETURNS pg_catalog.void AS 'MODULE_PATHNAME' LANGUAGE C; =20 -CREATE FUNCTION dc_crash_before_checkpoint(attach boolean DEFAULT true) +CREATE FUNCTION dcw_prune_dblist(attach boolean DEFAULT true) + RETURNS pg_catalog.void + AS 'MODULE_PATHNAME' LANGUAGE C; + +CREATE FUNCTION dcw_fake_temptable(attach boolean DEFAULT true) RETURNS pg_catalog.void AS 'MODULE_PATHNAME' LANGUAGE C; =20 -CREATE FUNCTION dc_crash_before_xlog(attach boolean DEFAULT true) +CREATE FUNCTION dc_crash_after_checksum_enable(attach boolean DEFAULT = true) RETURNS pg_catalog.void AS 'MODULE_PATHNAME' LANGUAGE C; diff --git a/src/test/modules/test_checksums/test_checksums.c = b/src/test/modules/test_checksums/test_checksums.c index c182f2c868b..8e4c676df76 100644 --- a/src/test/modules/test_checksums/test_checksums.c +++ b/src/test/modules/test_checksums/test_checksums.c @@ -3,7 +3,7 @@ * test_checksums.c * Test data checksums * - * Copyright (c) 2025, PostgreSQL Global Development Group + * Copyright (c) 2026, PostgreSQL Global Development Group * * IDENTIFICATION * src/test/modules/test_checksums/test_checksums.c @@ -67,6 +67,50 @@ dcw_inject_delay_barrier(PG_FUNCTION_ARGS) PG_RETURN_VOID(); } =20 +PG_FUNCTION_INFO_V1(dcw_inject_launcher_delay); +Datum +dcw_inject_launcher_delay(PG_FUNCTION_ARGS) +{ +#ifdef USE_INJECTION_POINTS + bool attach =3D PG_GETARG_BOOL(0); + + if (attach) + = InjectionPointAttach("datachecksumsworker-launcher-delay", + = "test_checksums", + = "dc_delay_barrier", + NULL, + 0); + else + = InjectionPointDetach("datachecksumsworker-launcher-delay"); +#else + elog(ERROR, + "test is not working as intended when injection points = are disabled"); +#endif + PG_RETURN_VOID(); +} + +PG_FUNCTION_INFO_V1(dcw_inject_startup_delay); +Datum +dcw_inject_startup_delay(PG_FUNCTION_ARGS) +{ +#ifdef USE_INJECTION_POINTS + bool attach =3D PG_GETARG_BOOL(0); + + if (attach) + = InjectionPointAttach("datachecksumsworker-startup-delay", + = "test_checksums", + = "dc_delay_barrier", + NULL, + 0); + else + = InjectionPointDetach("datachecksumsworker-startup-delay"); +#else + elog(ERROR, + "test is not working as intended when injection points = are disabled"); +#endif + PG_RETURN_VOID(); +} + void dc_fail_database(const char *name, const void *private_data, void *arg) { @@ -180,19 +224,19 @@ crash(const char *name, const void *private_data, = void *arg) } =20 /* - * dc_crash_before_checkpoint + * dc_crash_after_checksum_enable * * Ensure that the server crashes just before the checkpoint is issued = after * enabling or disabling checksums. */ -PG_FUNCTION_INFO_V1(dc_crash_before_checkpoint); +PG_FUNCTION_INFO_V1(dc_crash_after_checksum_enable); Datum -dc_crash_before_checkpoint(PG_FUNCTION_ARGS) +dc_crash_after_checksum_enable(PG_FUNCTION_ARGS) { #ifdef USE_INJECTION_POINTS - = InjectionPointAttach("datachecksums-enable-checksums-pre-checkpoint", + InjectionPointAttach("datachecksums-enable-checksums-done", "test_checksums", = "crash", NULL, 0); - = InjectionPointAttach("datachecksums-disable-checksums-pre-checkpoint", + InjectionPointAttach("datachecksums-disable-checksums-done", "test_checksums", = "crash", NULL, 0); #else elog(ERROR, @@ -200,26 +244,3 @@ dc_crash_before_checkpoint(PG_FUNCTION_ARGS) #endif PG_RETURN_VOID(); } - -/* - * dc_crash_before_xlog - * - * Ensure that the server crashes right before it is about insert the = xlog - * record XLOG_CHECKSUMS. - */ -PG_FUNCTION_INFO_V1(dc_crash_before_xlog); -Datum -dc_crash_before_xlog(PG_FUNCTION_ARGS) -{ -#ifdef USE_INJECTION_POINTS - bool attach =3D PG_GETARG_BOOL(0); - - if (attach) - = InjectionPointAttach("datachecksums-xlogchecksums-pre-xloginsert", - = "test_checksums", "crash", NULL, 0); -#else - elog(ERROR, - "test is not working as intended when injection points = are disabled"); -#endif - PG_RETURN_VOID(); -} diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm = b/src/test/perl/PostgreSQL/Test/Cluster.pm index 1b510a294d5..9da65f2a55a 100644 --- a/src/test/perl/PostgreSQL/Test/Cluster.pm +++ b/src/test/perl/PostgreSQL/Test/Cluster.pm @@ -3899,15 +3899,6 @@ sub checksum_disable_offline return; } =20 -sub checksum_verify_offline -{ - my ($self) =3D @_; - - PostgreSQL::Test::Utils::system_or_bail('pg_checksums', '-D', - $self->data_dir, '-c'); - return; -} - =3Dpod =20 =3Dback diff --git a/src/test/regress/expected/rules.out = b/src/test/regress/expected/rules.out index 3ef1d5916d9..697e0329da4 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -2095,10 +2095,9 @@ pg_stat_progress_data_checksums| SELECT s.pid, CASE s.param1 WHEN 0 THEN 'enabling'::text WHEN 1 THEN 'disabling'::text - WHEN 2 THEN 'waiting'::text - WHEN 3 THEN 'waiting on temporary tables'::text - WHEN 4 THEN 'waiting on checkpoint'::text - WHEN 5 THEN 'done'::text + WHEN 2 THEN 'waiting on temporary tables'::text + WHEN 3 THEN 'waiting on barrier'::text + WHEN 4 THEN 'done'::text ELSE NULL::text END AS phase, CASE s.param2 diff --git a/src/tools/pgindent/typedefs.list = b/src/tools/pgindent/typedefs.list index 2b87e079aa9..f4cce543113 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -4321,6 +4321,7 @@ xl_btree_split xl_btree_unlink_page xl_btree_update xl_btree_vacuum +xl_checkpoint_redo xl_checksum_state xl_clog_truncate xl_commit_ts_truncate --=20 2.39.3 (Apple Git-146) --Apple-Mail=_B8DC27FC-A3EC-4551-9E38-E86153E9E5F4--