pg.ddx.io pgsql-admin@postgresql.org mailing list archive help / color / mirror / Atom feed From: Rui DeSousa <rui.desousa@icloud.com>
To: Dianne Skoll <dfs@roaringpenguin.com>
Cc: pgsql-admin@lists.postgresql.org
Subject: Re: Reliable WAL file shipping over unreliable network
Date: Wed, 28 Feb 2018 20:10:59 -0500
Message-ID: <21AF0CB4-0873-4960-BA14-E72FA08B352E@icloud.com> (raw )
In-Reply-To: <33220469-5BB2-4A40-8F56-8F4F392C95B3@icloud.com >
References: <dd116172-3ddf-ee56-57f3-6e18d49abf8e@shopzeus.com >
<400AC9CD-1E43-422C-A689-70BDBAFF231E@elevated-dev.com >
<ee3ae4be-ab24-8443-82c1-95bcd4f86a63@shopzeus.com >
<debd4205-e27a-cd9d-9169-6748e65083a2@shopzeus.com >
<BA4A1866-95B0-49C6-8CD1-3D1B60A7837E@elevated-dev.com >
<CF30D15A-BD42-42A7-9AD3-BB69AF9E8047@icloud.com >
<D5B11CF5-E816-435B-BFB8-1226E23C046A@elevated-dev.com >
<37C5A42B-4CDD-4F01-8F1E-1B185C1819B9@icloud.com >
<20180228132620.2c882f84@hydrogen.roaringpenguin.com >
<33220469-5BB2-4A40-8F56-8F4F392C95B3@icloud.com >
I’ve tested this and it seems that there is a still a bug in rsync (rsync version 3.1.2 protocol version 31). I used a 1GB archive filesytem to allow for an out of space test case. Not sure of the actual cause as it seems to work a few times; however, it then fails leaving a truncated file and returning a success code.
Example: 00000001000000590000003E - failed 4 fimes and on the fifth try rsync returned success and left a truncated file.
When there is actually no space left; rsync fails immidately and never returns a success code; i.e. 00000001000000590000003F. When freeing up space; archive resumes again. It seems that if rsync is already syncing when the files system fills up then there is a high risk the bug will occur; i.e. 000000010000005A00000001 is also truncated and with a rsync success code.
Since rsync is returning success on a failed sync; even the "-c" option will not help here.
Archive Script critical code:
OUTPUT=$(rsync -ac $XLOGFILE $ARCH_SERVER:$ARCH_DIR/$WALFILE)
if [ $? == 0 ]; then
STS=0
echo "Success: $WALFILE" >> /tmp/waltest.log
else
echo "Failed: $WALFILE" >> /tmp/waltest.log
fi
exit $STS
Archive Directory (Note: useing 64MB WALs):
[postgres@hades ~/arch/dbc1/wal]$ ls -al
total 1044351
drwxr-xr-x 2 postgres postgres 57 Feb 28 19:46 .
drwxr-xr-x 3 postgres postgres 3 Feb 28 17:33 ..
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 00000001000000590000000B
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 00000001000000590000000C
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 00000001000000590000000D
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 00000001000000590000000E
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 00000001000000590000000F
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 000000010000005900000010
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 000000010000005900000011
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 000000010000005900000012
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 000000010000005900000013
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 000000010000005900000014
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 000000010000005900000015
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 000000010000005900000016
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 000000010000005900000017
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 000000010000005900000018
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 000000010000005900000019
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 00000001000000590000001A
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 00000001000000590000001B
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 00000001000000590000001C
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 00000001000000590000001D
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 00000001000000590000001E
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 00000001000000590000001F
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 000000010000005900000020
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 000000010000005900000021
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 000000010000005900000022
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 000000010000005900000023
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 000000010000005900000024
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 000000010000005900000025
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 000000010000005900000026
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 000000010000005900000027
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 000000010000005900000028
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 000000010000005900000029
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 00000001000000590000002A
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 00000001000000590000002B
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 00000001000000590000002C
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 00000001000000590000002D
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 00000001000000590000002E
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 00000001000000590000002F
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 000000010000005900000030
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 000000010000005900000031
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 000000010000005900000032
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 000000010000005900000033
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 000000010000005900000034
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 000000010000005900000035
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 000000010000005900000036
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 000000010000005900000037
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 000000010000005900000038
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 000000010000005900000039
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 00000001000000590000003A
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 00000001000000590000003B
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 00000001000000590000003C
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 00000001000000590000003D
-rw------- 1 postgres postgres 3670016 Feb 28 17:33 00000001000000590000003E
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 00000001000000590000003F
-rw------- 1 postgres postgres 67108864 Feb 28 17:33 000000010000005A00000000
-rw------- 1 postgres postgres 12713984 Feb 28 17:33 000000010000005A00000001
Success/Failure Log:
[postgres@hades ~/arch/dbc1/wal]$ cat /tmp/waltest.log
Success: 000000010000005900000008
Success: 000000010000005900000009
Success: 00000001000000590000000A
Success: 00000001000000590000000B
Success: 00000001000000590000000C
Success: 00000001000000590000000D
Success: 00000001000000590000000E
Success: 00000001000000590000000F
Success: 000000010000005900000010
Success: 000000010000005900000011
Success: 000000010000005900000012
Success: 000000010000005900000013
Success: 000000010000005900000014
Success: 000000010000005900000015
Success: 000000010000005900000016
Success: 000000010000005900000017
Success: 000000010000005900000018
Success: 000000010000005900000019
Success: 00000001000000590000001A
Success: 00000001000000590000001B
Success: 00000001000000590000001C
Success: 00000001000000590000001D
Success: 00000001000000590000001E
Success: 00000001000000590000001F
Success: 000000010000005900000020
Success: 000000010000005900000021
Success: 000000010000005900000022
Success: 000000010000005900000023
Success: 000000010000005900000024
Success: 000000010000005900000025
Success: 000000010000005900000026
Success: 000000010000005900000027
Success: 000000010000005900000028
Success: 000000010000005900000029
Success: 00000001000000590000002A
Success: 00000001000000590000002B
Success: 00000001000000590000002C
Success: 00000001000000590000002D
Success: 00000001000000590000002E
Success: 00000001000000590000002F
Success: 000000010000005900000030
Success: 000000010000005900000031
Success: 000000010000005900000032
Success: 000000010000005900000033
Success: 000000010000005900000034
Success: 000000010000005900000035
Success: 000000010000005900000036
Success: 000000010000005900000037
Success: 000000010000005900000038
Success: 000000010000005900000039
Success: 00000001000000590000003A
Success: 00000001000000590000003B
Success: 00000001000000590000003C
Success: 00000001000000590000003D
Failed: 00000001000000590000003E
Failed: 00000001000000590000003E
Failed: 00000001000000590000003E
Failed: 00000001000000590000003E
Success: 00000001000000590000003E
Failed: 00000001000000590000003F
Failed: 00000001000000590000003F
Failed: 00000001000000590000003F
Failed: 00000001000000590000003F
Failed: 00000001000000590000003F
Failed: 00000001000000590000003F
Failed: 00000001000000590000003F
Failed: 00000001000000590000003F
Failed: 00000001000000590000003F
Failed: 00000001000000590000003F
Failed: 00000001000000590000003F
Failed: 00000001000000590000003F
Failed: 00000001000000590000003F
Failed: 00000001000000590000003F
Failed: 00000001000000590000003F
Failed: 00000001000000590000003F
Failed: 00000001000000590000003F
Failed: 00000001000000590000003F
Success: 00000001000000590000003F
Success: 000000010000005A00000000
Failed: 000000010000005A00000001
Failed: 000000010000005A00000001
Failed: 000000010000005A00000001
Failed: 000000010000005A00000001
Failed: 000000010000005A00000001
Failed: 000000010000005A00000001
Failed: 000000010000005A00000001
Success: 000000010000005A00000001
Failed: 000000010000005A00000002
Failed: 000000010000005A00000002
Failed: 000000010000005A00000002
Failed: 000000010000005A00000002
Failed: 000000010000005A00000002
Failed: 000000010000005A00000002
Failed: 000000010000005A00000002
Failed: 000000010000005A00000002
Failed: 000000010000005A00000002
view thread (59+ messages) latest in thread
Message-ID: <21AF0CB4-0873-4960-BA14-E72FA08B352E@icloud.com>
Permalink: ../21AF0CB4-0873-4960-BA14-E72FA08B352E@icloud.com/
Also on: postgresql.org/message-id/21AF0CB4-0873-4960-BA14-E72FA08B352E@icloud.com
copy link · copy postgr.es
reply Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-admin@postgresql.org
Cc: rui.desousa@icloud.com, dfs@roaringpenguin.com, pgsql-admin@lists.postgresql.org
Subject: Re: Reliable WAL file shipping over unreliable network
In-Reply-To: <21AF0CB4-0873-4960-BA14-E72FA08B352E@icloud.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox