agora inbox for [email protected]
help / color / mirror / Atom feedRegarding the archive_command in "continuois archiving" chapter.
8+ messages / 4 participants
[nested] [flat]
* Regarding the archive_command in "continuois archiving" chapter.
@ 2026-05-05 14:37 PG Doc comments form <[email protected]>
2026-05-05 18:33 ` Re: Regarding the archive_command in "continuois archiving" chapter. David G. Johnston <[email protected]>
0 siblings, 1 reply; 8+ messages in thread
From: PG Doc comments form @ 2026-05-05 14:37 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]
The following documentation comment has been logged on the website:
Page: https://www.postgresql.org/docs/17/continuous-archiving.html
Description:
The example provided for archive_command for
https://www.postgresql.org/docs/18/continuous-archiving.html#BACKUP-ARCHIVING-WAL
should include commands to ensure that archiving has successfully written
the disk, even during an unexpected shutdown. Currently, PostgreSQL does not
force a write to the file system for archived WAL files as it does for data
files.
During periods of high write activity, PostgreSQL might lose files that were
confirmed as written but remained in the operating system cache. This loss
affects Point-in-Time Recovery (PITR) and the use of pg_rewind. While many
servers utilize power protection such as UPS or battery-backed RAID
controllers, the OS cache remains unprotected during a forced shutdown
without a sync call. This issue does not affect crash recovery or streaming
replication, but it impacts PITR and pg_rewind.
I recommend improving the documentation examples or including a visible
warning. The cp command in Linux should be treated as a stub; it should not
be used without an explicit sync call or by ensuring the destination
directory is mounted in an specific file system with the sync option in
/etc/fstab.
It could also include references to usage of other tools (rsync --fsync,
remote archiving or barman) or the options needed to use it with a nfs
mounted directory, but I might be too open.
This what would look like (I've used ai to help me that this gets written in
proper english)
-----
25.3.1. Setting Up WAL Archiving
In an abstract sense, a running PostgreSQL system produces an indefinitely
long sequence of WAL records. The system physically divides this sequence
into WAL segment files, which are normally 16MB apiece (although the segment
size can be altered during initdb). The segment files are given numeric
names that reflect their position in the abstract WAL sequence. When not
using WAL archiving, the system normally creates just a few segment files
and then “recycles” them by renaming no-longer-needed segment files to
higher segment numbers. It is assumed that segment files whose contents
precede the last checkpoint are no longer of interest and can be recycled.
When archiving WAL data, we need to capture the contents of each segment
file once it is filled and save that data somewhere before the segment file
is recycled for reuse. Depending on the application and the available
hardware, there could be many different ways of “saving the data somewhere”:
we could copy the segment files to an NFS-mounted directory on another
machine, write them onto a tape drive, or batch them together and burn them
onto CDs. To provide the database administrator with flexibility, PostgreSQL
does not make assumptions about how the archiving will be done. Instead,
PostgreSQL lets the administrator specify a shell command or an archive
library to be executed to copy a completed segment file to its destination.
This could be a shell command using cp, or it could invoke a complex C
function.
To enable WAL archiving, set the wal_level configuration parameter to
replica or higher, archive_mode to on, and specify the shell command to use
in the archive_command configuration parameter.
In archive_command, %p is replaced by the path name of the file to archive,
while %f is replaced by only the file name. The simplest useful command is
something like:
Plaintext
archive_command = 'test ! -f /mnt/server/archivedir/%f && cp %p
/mnt/server/archivedir/%f' # Unix
archive_command = 'copy "%p" "C:\\server\\archivedir\\%f"' # Windows
IMPORTANT: The simple cp or copy commands return success as soon as the data
is written to the operating system cache. However, PostgreSQL does not
automatically force these archived files to be flushed to physical storage.
In the event of an unexpected system shutdown or power failure, files
confirmed as archived might be lost from the OS cache, leading to data loss
in Point-in-Time Recovery (PITR) or issues with pg_rewind.
To ensure data persistence, the command should include a step to synchronize
the file to disk. On Unix/Linux systems, a more robust command is:
Plaintext
archive_command = 'test ! -f /mnt/server/archivedir/%f && cp %p
/mnt/server/archivedir/%f && sync /mnt/server/archivedir/%f'
The sync command must operate at the file level, and sync version should be
validated to not write the entire cache. Alternatively, the destination file
system can be mounted with the sync option in the system configuration
(e.g., /etc/fstab), though this may impact overall performance.
Administrators must validate the durability and behavior of the chosen
archival method on their specific operating system and storage architecture,
as file system synchronization guarantees vary significantly across
platforms.
After the %p and %f parameters have been replaced, the actual command
executed might look like this:
test ! -f /mnt/server/archivedir/00000001000000A900000065 && cp
pg_wal/00000001000000A900000065
/mnt/server/archivedir/00000001000000A900000065 && sync
/mnt/server/archivedir/00000001000000A900000065
A similar command will be generated for each new file to be archived.
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Regarding the archive_command in "continuois archiving" chapter.
2026-05-05 14:37 Regarding the archive_command in "continuois archiving" chapter. PG Doc comments form <[email protected]>
@ 2026-05-05 18:33 ` David G. Johnston <[email protected]>
0 siblings, 0 replies; 8+ messages in thread
From: David G. Johnston @ 2026-05-05 18:33 UTC (permalink / raw)
To: [email protected]; [email protected]
On Tue, May 5, 2026 at 11:19 AM PG Doc comments form <[email protected]>
wrote:
> The cp command in Linux should be treated as a stub;
The documentation says as much:
(This is an example, not a recommendation, and might not work on all
platforms.)
> This what would look like (I've used ai to help me that this gets written
> in
> proper english)
>
Getting it to update the sgml and produce a real diff file would be
considerably more helpful.
I do think the point here is to be so simple as to be obviously incomplete
and thus encourage the reader to find a proven backup system to plug into
here instead of writing their own script.
PostgreSQL consistently tries to avoid being inherently too platform and
process dependent and this is one of the key places where the community,
interested in their platforms and operating systems, are expected to
largely step in.
That all said, patching this part of the documentation isn't off the table.
(Though given prior comments like this exist and it hasn't been updated
suggests a low priority if nothing else - especially without a concrete
sgml patch.)
David J.
^ permalink raw reply [nested|flat] 8+ messages in thread
* pg_checksums docs mentioning online functions
@ 2026-06-04 18:55 Bruce Momjian <[email protected]>
2026-06-04 19:18 ` Re: pg_checksums docs mentioning online functions Daniel Gustafsson <[email protected]>
0 siblings, 1 reply; 8+ messages in thread
From: Bruce Momjian @ 2026-06-04 18:55 UTC (permalink / raw)
To: PostgreSQL-documentation <[email protected]>; +Cc: Daniel Gustafsson <[email protected]>
Looking at the pg_checksums docs, it says:
pg_checksums checks, enables or disables data checksums in a
PostgreSQL cluster. The server must be shut down cleanly before
running pg_checksums.
Do we want to mention the new functions here that can change the
checksum status without being shut down?
--
Bruce Momjian <[email protected]> https://momjian.us
EDB https://enterprisedb.com
Do not let urgent matters crowd out time for investment in the future.
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: pg_checksums docs mentioning online functions
2026-06-04 18:55 pg_checksums docs mentioning online functions Bruce Momjian <[email protected]>
@ 2026-06-04 19:18 ` Daniel Gustafsson <[email protected]>
2026-06-04 20:12 ` Re: pg_checksums docs mentioning online functions Bruce Momjian <[email protected]>
0 siblings, 1 reply; 8+ messages in thread
From: Daniel Gustafsson @ 2026-06-04 19:18 UTC (permalink / raw)
To: Bruce Momjian <[email protected]>; +Cc: PostgreSQL-documentation <[email protected]>
> On 4 Jun 2026, at 20:55, Bruce Momjian <[email protected]> wrote:
>
> Looking at the pg_checksums docs, it says:
>
> pg_checksums checks, enables or disables data checksums in a
> PostgreSQL cluster. The server must be shut down cleanly before
> running pg_checksums.
>
> Do we want to mention the new functions here that can change the
> checksum status without being shut down?
Not sure, but perhaps a link back to the main data checksums section (which
documents the online operation among other things) would be in order?
Something like:
--- a/doc/src/sgml/ref/pg_checksums.sgml
+++ b/doc/src/sgml/ref/pg_checksums.sgml
@@ -57,6 +57,10 @@ PostgreSQL documentation
rewritten in-place.
Disabling checksums only updates the file <filename>pg_control</filename>.
</para>
+
+ <para>
+ For more information on data checksums, see <xref linkend="checksums"/>.
+ </para>
</refsect1>
--
Daniel Gustafsson
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: pg_checksums docs mentioning online functions
2026-06-04 18:55 pg_checksums docs mentioning online functions Bruce Momjian <[email protected]>
2026-06-04 19:18 ` Re: pg_checksums docs mentioning online functions Daniel Gustafsson <[email protected]>
@ 2026-06-04 20:12 ` Bruce Momjian <[email protected]>
2026-06-04 20:39 ` Re: pg_checksums docs mentioning online functions Daniel Gustafsson <[email protected]>
0 siblings, 1 reply; 8+ messages in thread
From: Bruce Momjian @ 2026-06-04 20:12 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: PostgreSQL-documentation <[email protected]>
On Thu, Jun 4, 2026 at 09:18:34PM +0200, Daniel Gustafsson wrote:
> > On 4 Jun 2026, at 20:55, Bruce Momjian <[email protected]> wrote:
> >
> > Looking at the pg_checksums docs, it says:
> >
> > pg_checksums checks, enables or disables data checksums in a
> > PostgreSQL cluster. The server must be shut down cleanly before
> > running pg_checksums.
> >
> > Do we want to mention the new functions here that can change the
> > checksum status without being shut down?
>
> Not sure, but perhaps a link back to the main data checksums section (which
> documents the online operation among other things) would be in order?
> Something like:
>
> --- a/doc/src/sgml/ref/pg_checksums.sgml
> +++ b/doc/src/sgml/ref/pg_checksums.sgml
> @@ -57,6 +57,10 @@ PostgreSQL documentation
> rewritten in-place.
> Disabling checksums only updates the file <filename>pg_control</filename>.
> </para>
> +
> + <para>
> + For more information on data checksums, see <xref linkend="checksums"/>.
> + </para>
> </refsect1>
My big issue is that we explain the limitation of pg_checksums, but
don't point readers at the tool that doesn't have that offline
restriction. I think we need that explicitly mentioned, and at the top.
--
Bruce Momjian <[email protected]> https://momjian.us
EDB https://enterprisedb.com
Do not let urgent matters crowd out time for investment in the future.
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: pg_checksums docs mentioning online functions
2026-06-04 18:55 pg_checksums docs mentioning online functions Bruce Momjian <[email protected]>
2026-06-04 19:18 ` Re: pg_checksums docs mentioning online functions Daniel Gustafsson <[email protected]>
2026-06-04 20:12 ` Re: pg_checksums docs mentioning online functions Bruce Momjian <[email protected]>
@ 2026-06-04 20:39 ` Daniel Gustafsson <[email protected]>
2026-06-04 21:08 ` Re: pg_checksums docs mentioning online functions Bruce Momjian <[email protected]>
0 siblings, 1 reply; 8+ messages in thread
From: Daniel Gustafsson @ 2026-06-04 20:39 UTC (permalink / raw)
To: Bruce Momjian <[email protected]>; +Cc: PostgreSQL-documentation <[email protected]>
> On 4 Jun 2026, at 22:12, Bruce Momjian <[email protected]> wrote:
> My big issue is that we explain the limitation of pg_checksums, but
> don't point readers at the tool that doesn't have that offline
> restriction. I think we need that explicitly mentioned, and at the top.
How about something like the attached?
--
Daniel Gustafsson
Attachments:
[application/octet-stream] pg_checksums_docs.diff (2.1K, ../../[email protected]/2-pg_checksums_docs.diff)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_checksums.sgml b/doc/src/sgml/ref/pg_checksums.sgml
index 45890324075..d1048d43f11 100644
--- a/doc/src/sgml/ref/pg_checksums.sgml
+++ b/doc/src/sgml/ref/pg_checksums.sgml
@@ -38,17 +38,22 @@ PostgreSQL documentation
<para>
<application>pg_checksums</application> checks, enables or disables data
checksums in a <productname>PostgreSQL</productname> cluster. The server
- must be shut down cleanly before running
- <application>pg_checksums</application>. When verifying checksums, the exit
+ must be shut down cleanly before running <application>pg_checksums</application>.
+ Checksums can also be enabled while the cluster is running using
+ <xref linkend="checksums-online-enable-disable"/> processing, see
+ <xref linkend="checksums"/> for further details on the different approaches.
+ When verifying checksums, the exit
status is zero if there are no checksum errors, and nonzero if at least one
checksum failure is detected. When enabling or disabling checksums, the
exit status is nonzero if the operation failed.
</para>
<para>
- When enabling checksums, if checksums were in the process of being enabled
- when the cluster was shut down, <application>pg_checksums</application>
- will still process all relations regardless of the online processing.
+ When enabling checksums with <application>pg_checksums</application>, if
+ checksums were in the process of being enabled using
+ <xref linkend="checksums-online-enable-disable"/> when the cluster was shut
+ down, <application>pg_checksums</application> will still process all relation
+ files regardless of progress of the online processing.
</para>
<para>
diff --git a/doc/src/sgml/wal.sgml b/doc/src/sgml/wal.sgml
index c32931edde3..646076f7e39 100644
--- a/doc/src/sgml/wal.sgml
+++ b/doc/src/sgml/wal.sgml
@@ -276,7 +276,7 @@
</sect2>
- <sect2 id="checksums-online-enable-disable">
+ <sect2 id="checksums-online-enable-disable" xreflabel="Online Enabling of Checksums">
<title>Online Enabling of Checksums</title>
<para>
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: pg_checksums docs mentioning online functions
2026-06-04 18:55 pg_checksums docs mentioning online functions Bruce Momjian <[email protected]>
2026-06-04 19:18 ` Re: pg_checksums docs mentioning online functions Daniel Gustafsson <[email protected]>
2026-06-04 20:12 ` Re: pg_checksums docs mentioning online functions Bruce Momjian <[email protected]>
2026-06-04 20:39 ` Re: pg_checksums docs mentioning online functions Daniel Gustafsson <[email protected]>
@ 2026-06-04 21:08 ` Bruce Momjian <[email protected]>
2026-06-04 22:21 ` Re: pg_checksums docs mentioning online functions Daniel Gustafsson <[email protected]>
0 siblings, 1 reply; 8+ messages in thread
From: Bruce Momjian @ 2026-06-04 21:08 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: PostgreSQL-documentation <[email protected]>
On Thu, Jun 4, 2026 at 10:39:53PM +0200, Daniel Gustafsson wrote:
> > On 4 Jun 2026, at 22:12, Bruce Momjian <[email protected]> wrote:
>
> > My big issue is that we explain the limitation of pg_checksums, but
> > don't point readers at the tool that doesn't have that offline
> > restriction. I think we need that explicitly mentioned, and at the top.
>
> How about something like the attached?
Looks good, just one suggestion:
> + When enabling checksums with <application>pg_checksums</application>, if
> + checksums were in the process of being enabled using
> + <xref linkend="checksums-online-enable-disable"/> when the cluster was shut
> + down, <application>pg_checksums</application> will still process all relation
> + files regardless of progress of the online processing.
... pg_checksums will still process all relation files regardless of
the progress of online checksum processing.
-------------------------------
I think this idea is complex enough that we have to clearly state
"online _checksum_ processing."
--
Bruce Momjian <[email protected]> https://momjian.us
EDB https://enterprisedb.com
Do not let urgent matters crowd out time for investment in the future.
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: pg_checksums docs mentioning online functions
2026-06-04 18:55 pg_checksums docs mentioning online functions Bruce Momjian <[email protected]>
2026-06-04 19:18 ` Re: pg_checksums docs mentioning online functions Daniel Gustafsson <[email protected]>
2026-06-04 20:12 ` Re: pg_checksums docs mentioning online functions Bruce Momjian <[email protected]>
2026-06-04 20:39 ` Re: pg_checksums docs mentioning online functions Daniel Gustafsson <[email protected]>
2026-06-04 21:08 ` Re: pg_checksums docs mentioning online functions Bruce Momjian <[email protected]>
@ 2026-06-04 22:21 ` Daniel Gustafsson <[email protected]>
0 siblings, 0 replies; 8+ messages in thread
From: Daniel Gustafsson @ 2026-06-04 22:21 UTC (permalink / raw)
To: Bruce Momjian <[email protected]>; +Cc: PostgreSQL-documentation <[email protected]>
> On 4 Jun 2026, at 23:08, Bruce Momjian <[email protected]> wrote:
> On Thu, Jun 4, 2026 at 10:39:53PM +0200, Daniel Gustafsson wrote:
>> How about something like the attached?
>
> Looks good, just one suggestion:
Pushed with your suggestion added to the patch, thanks!
--
Daniel Gustafsson
^ permalink raw reply [nested|flat] 8+ messages in thread
end of thread, other threads:[~2026-06-04 22:21 UTC | newest]
Thread overview: 8+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-05-05 14:37 Regarding the archive_command in "continuois archiving" chapter. PG Doc comments form <[email protected]>
2026-05-05 18:33 ` David G. Johnston <[email protected]>
2026-06-04 18:55 pg_checksums docs mentioning online functions Bruce Momjian <[email protected]>
2026-06-04 19:18 ` Re: pg_checksums docs mentioning online functions Daniel Gustafsson <[email protected]>
2026-06-04 20:12 ` Re: pg_checksums docs mentioning online functions Bruce Momjian <[email protected]>
2026-06-04 20:39 ` Re: pg_checksums docs mentioning online functions Daniel Gustafsson <[email protected]>
2026-06-04 21:08 ` Re: pg_checksums docs mentioning online functions Bruce Momjian <[email protected]>
2026-06-04 22:21 ` Re: pg_checksums docs mentioning online functions Daniel Gustafsson <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox