From: Dianne Skoll <dfs@roaringpenguin.com>
Cc: pgsql-admin@lists.postgresql.org
Subject: Re: Reliable WAL file shipping over unreliable network
Date: Wed, 28 Feb 2018 14:16:43 -0500
Message-ID: <20180228141643.40a8eb8a@hydrogen.roaringpenguin.com> (raw)
In-Reply-To: <20180228135220.1cd46347@hydrogen.roaringpenguin.com>
References: <dd116172-3ddf-ee56-57f3-6e18d49abf8e@shopzeus.com>
<400AC9CD-1E43-422C-A689-70BDBAFF231E@elevated-dev.com>
<ee3ae4be-ab24-8443-82c1-95bcd4f86a63@shopzeus.com>
<debd4205-e27a-cd9d-9169-6748e65083a2@shopzeus.com>
<BA4A1866-95B0-49C6-8CD1-3D1B60A7837E@elevated-dev.com>
<CF30D15A-BD42-42A7-9AD3-BB69AF9E8047@icloud.com>
<D5B11CF5-E816-435B-BFB8-1226E23C046A@elevated-dev.com>
<37C5A42B-4CDD-4F01-8F1E-1B185C1819B9@icloud.com>
<20180228132620.2c882f84@hydrogen.roaringpenguin.com>
<33220469-5BB2-4A40-8F56-8F4F392C95B3@icloud.com>
<20180228135220.1cd46347@hydrogen.roaringpenguin.com>
So, I decided to run a test.
It turns out we are both wrong.
My test script is as follows:
#======================================================================
#!/bin/bash
# Remote machine
MACH=dianne.skoll.ca
# Create a 10MB pseudo-random file
echo "`date` Creating 10MB file"
FILE=pseudo-random
dd if=/dev/urandom bs=1k count=10k of=$FILE 2>/dev/null
# Rsync it over at 500kB per second in the background
echo "`date` Starting rsync"
rsync -c --verbose --bwlimit=500k $FILE $MACH:/tmp &
RSYNC_PID=$!
echo "`date` rsync PID=$RSYNC_PID; sleeping for 3 seconds"
# Sleep for 3 seconds
sleep 3
echo "`date` Stopping rsync"
# Stop the rsync
kill -STOP $RSYNC_PID
echo "`date` Changing source file"
# Change the source file
dd if=/dev/urandom bs=1k count=10k of=$FILE 2>/dev/null
echo "`date` Continuing rsync"
# Restart the rsync and bring it into the foreground
kill -CONT $RSYNC_PID
wait
echo "`date` Rsync done; comparing files"
ssh $MACH "cd /tmp && sha1sum $FILE" | sha1sum -c -
#======================================================================
Running it produces this output:
Wed Feb 28 14:14:32 EST 2018 Creating 10MB file
Wed Feb 28 14:14:32 EST 2018 Starting rsync
Wed Feb 28 14:14:32 EST 2018 rsync PID=24456; sleeping for 3 seconds
pseudo-random
Wed Feb 28 14:14:35 EST 2018 Stopping rsync
Wed Feb 28 14:14:35 EST 2018 Changing source file
Wed Feb 28 14:14:35 EST 2018 Continuing rsync
sent 10,488,419 bytes received 19,505 bytes 488,740.65 bytes/sec
total size is 10,485,760 speedup is 1.00
Wed Feb 28 14:14:53 EST 2018 Rsync done; comparing files
pseudo-random: FAILED
sha1sum: WARNING: 1 computed checksum did NOT match
So even *with* the -c flag, rsync didn't notice the file had been changed
while it was running. Interesting.
Regards,
Dianne.
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-admin@postgresql.org
Cc: dfs@roaringpenguin.com, pgsql-admin@lists.postgresql.org
Subject: Re: Reliable WAL file shipping over unreliable network
In-Reply-To: <20180228141643.40a8eb8a@hydrogen.roaringpenguin.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox