From: Nagy László Zsolt <gandalf@shopzeus.com>
To: Rui DeSousa <rui.desousa@icloud.com>
To: scott ribe <scott_ribe@elevated-dev.com>
Cc: pgsql-admin@lists.postgresql.org
Subject: Re: Reliable WAL file shipping over unreliable network
Date: Wed, 28 Feb 2018 20:03:43 +0100
Message-ID: <f1629420-0d42-0ef9-fbca-a9cfca5e7a01@shopzeus.com> (raw)
In-Reply-To: <45F65094-CF9C-4BC4-AF56-2FE299F79EFF@icloud.com>
References: <dd116172-3ddf-ee56-57f3-6e18d49abf8e@shopzeus.com>
<400AC9CD-1E43-422C-A689-70BDBAFF231E@elevated-dev.com>
<ee3ae4be-ab24-8443-82c1-95bcd4f86a63@shopzeus.com>
<debd4205-e27a-cd9d-9169-6748e65083a2@shopzeus.com>
<BA4A1866-95B0-49C6-8CD1-3D1B60A7837E@elevated-dev.com>
<CF30D15A-BD42-42A7-9AD3-BB69AF9E8047@icloud.com>
<954e01b6-b240-eb9a-5feb-8efd202fef5a@shopzeus.com>
<9BE48599-5417-42BF-A7D3-554590878841@elevated-dev.com>
<45F65094-CF9C-4BC4-AF56-2FE299F79EFF@icloud.com>
>>
>>
>> There seem to be 2 fundamental misunderstandings here:
>>
>> 1) That other processes cannot see data written to a file until it is
>> flushed to disk; this is not true; while file data is still in file
>> cache, it is visible to other processes.
>>
>> 2) That rsync writes the file on the destination directly; it does
>> not; it writes into a temporary file and renames that file when it is
>> complete.
>>
>
> While you’re correct; I never made either of those assumptions
>
Number 1 is irrelevant because files are sent from the master and
received by the slave. Two different computers that don't share the same
cache. Number 2 blows away my obscurity completely, I just didn't know
that rsync works that way. It must be documented somewhere in the (very
long) rsync manual. I just opened it and I looked specifically for this
behaviour, but I cannot find it. There is a --temp-dir option that
suggests that data is written to temporary files first. But that is only
a suggestion. I don't see anything explicit about writting data to
temporary files and renaming them once they are complete. Although it
seems logical and I believe you, but I did not want to make such
assumptions either.
So, problem solved in theory. The next step is to do tests by simulating
network outages.
I also have a proposal: let's change the example in the PostgreSQL
documentation! The example archive_command presented in the docs
contains a simple cp command that does not write data to temporary
files. Anyone who tries to use that example on a production server will
shoot himself on the foot. At least we should have a note there, telling
that it is the user's responsibility to make sure that only complete WAL
files appear on the slave's archive directory, and that the cp command
is just a silly example that should never be used on a production server.
Thank you for your help!
Laszlo
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-admin@postgresql.org
Cc: gandalf@shopzeus.com, rui.desousa@icloud.com, scott_ribe@elevated-dev.com, pgsql-admin@lists.postgresql.org
Subject: Re: Reliable WAL file shipping over unreliable network
In-Reply-To: <f1629420-0d42-0ef9-fbca-a9cfca5e7a01@shopzeus.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox