Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1erCkV-000334-Mt for pgsql-admin@arkaria.postgresql.org; Thu, 01 Mar 2018 01:11:07 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1erCkU-0001Mu-DU for pgsql-admin@arkaria.postgresql.org; Thu, 01 Mar 2018 01:11:06 +0000 Received: from makus.postgresql.org ([2001:4800:1501:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1erCkT-0001Ml-TE for pgsql-admin@lists.postgresql.org; Thu, 01 Mar 2018 01:11:06 +0000 Received: from mr28p48im-tydg10113601.me.com ([17.110.88.45]) by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1erCkR-0008Kk-8h for pgsql-admin@lists.postgresql.org; Thu, 01 Mar 2018 01:11:04 +0000 Received: from process-dkim-sign-daemon.mr28p48im-tydg10113601.me.com by mr28p48im-tydg10113601.me.com (Oracle Communications Messaging Server 8.0.1.2.20170607 64bit (built Jun 7 2017)) id <0P4W00M000D6TJ00@mr28p48im-tydg10113601.me.com> for pgsql-admin@lists.postgresql.org; Thu, 01 Mar 2018 01:11:02 +0000 (GMT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=icloud.com; s=04042017; t=1519866662; bh=gKT/UgRlIkAR4z8ADnOJ3YpjUCzrk4w6PigBXso7g/0=; h=Content-type:MIME-version:Subject:From:Date:Message-id:To; b=ZmS4l9Exa6vWFUy8y39DG5z6VL4udGg1B0oaqrCbfsYbAvrt8ena3W4ztQw2iinhA aT1FvkvIreBVU6RaK9n1O8b+ZI3OZMVkobhW5Idy3lAEcbyOkFGGxBAERLEjUvDxF9 x/iDU1NBzNy0Jck8U+exbkuD1PUBQ7bYFkfQRMU3wM10EEHjvi6CtFmABc+mWVR/kZ x7c6h30q+aIKIrG+UIFakMgX6z8XCd/rGDvFWfKQfUy1hO6fNKIKgDBQxpCEOBtZwp Zsqq4jXtzohhFfER+MJW6/10mqXonW5H2dAbcpTfaLuVex9c2fNV/D4pVVcHJ7ABZy VlLCMFQuk5Aew== Received: from icloud.com ([127.0.0.1]) by mr28p48im-tydg10113601.me.com (Oracle Communications Messaging Server 8.0.1.2.20170607 64bit (built Jun 7 2017)) with ESMTPSA id <0P4W0027F0MCST20@mr28p48im-tydg10113601.me.com>; Thu, 01 Mar 2018 01:11:01 +0000 (GMT) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2018-02-28_13:,, signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 clxscore=1015 suspectscore=0 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1707230000 definitions=main-1803010014 Content-type: text/plain; charset=utf-8 MIME-version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: Re: Reliable WAL file shipping over unreliable network From: Rui DeSousa In-reply-to: <33220469-5BB2-4A40-8F56-8F4F392C95B3@icloud.com> Date: Wed, 28 Feb 2018 20:10:59 -0500 Cc: pgsql-admin@lists.postgresql.org Content-transfer-encoding: quoted-printable Message-id: <21AF0CB4-0873-4960-BA14-E72FA08B352E@icloud.com> References: <400AC9CD-1E43-422C-A689-70BDBAFF231E@elevated-dev.com> <37C5A42B-4CDD-4F01-8F1E-1B185C1819B9@icloud.com> <20180228132620.2c882f84@hydrogen.roaringpenguin.com> <33220469-5BB2-4A40-8F56-8F4F392C95B3@icloud.com> To: Dianne Skoll X-Mailer: Apple Mail (2.3445.5.20) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk I=E2=80=99ve tested this and it seems that there is a still a bug in = rsync (rsync version 3.1.2 protocol version 31). I used a 1GB archive = filesytem to allow for an out of space test case. Not sure of the = actual cause as it seems to work a few times; however, it then fails = leaving a truncated file and returning a success code. Example: 00000001000000590000003E - failed 4 fimes and on the fifth try = rsync returned success and left a truncated file. =20 When there is actually no space left; rsync fails immidately and never = returns a success code; i.e. 00000001000000590000003F. When freeing up = space; archive resumes again. It seems that if rsync is already syncing = when the files system fills up then there is a high risk the bug will = occur; i.e. 000000010000005A00000001 is also truncated and with a rsync = success code. Since rsync is returning success on a failed sync; even the "-c" option = will not help here. Archive Script critical code: OUTPUT=3D$(rsync -ac $XLOGFILE $ARCH_SERVER:$ARCH_DIR/$WALFILE) if [ $? =3D=3D 0 ]; then STS=3D0 echo "Success: $WALFILE" >> /tmp/waltest.log else echo "Failed: $WALFILE" >> /tmp/waltest.log fi exit $STS Archive Directory (Note: useing 64MB WALs): [postgres@hades ~/arch/dbc1/wal]$ ls -al total 1044351 drwxr-xr-x 2 postgres postgres 57 Feb 28 19:46 . drwxr-xr-x 3 postgres postgres 3 Feb 28 17:33 .. -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 00000001000000590000000B -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 00000001000000590000000C -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 00000001000000590000000D -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 00000001000000590000000E -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 00000001000000590000000F -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 000000010000005900000010 -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 000000010000005900000011 -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 000000010000005900000012 -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 000000010000005900000013 -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 000000010000005900000014 -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 000000010000005900000015 -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 000000010000005900000016 -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 000000010000005900000017 -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 000000010000005900000018 -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 000000010000005900000019 -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 00000001000000590000001A -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 00000001000000590000001B -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 00000001000000590000001C -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 00000001000000590000001D -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 00000001000000590000001E -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 00000001000000590000001F -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 000000010000005900000020 -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 000000010000005900000021 -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 000000010000005900000022 -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 000000010000005900000023 -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 000000010000005900000024 -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 000000010000005900000025 -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 000000010000005900000026 -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 000000010000005900000027 -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 000000010000005900000028 -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 000000010000005900000029 -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 00000001000000590000002A -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 00000001000000590000002B -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 00000001000000590000002C -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 00000001000000590000002D -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 00000001000000590000002E -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 00000001000000590000002F -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 000000010000005900000030 -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 000000010000005900000031 -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 000000010000005900000032 -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 000000010000005900000033 -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 000000010000005900000034 -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 000000010000005900000035 -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 000000010000005900000036 -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 000000010000005900000037 -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 000000010000005900000038 -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 000000010000005900000039 -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 00000001000000590000003A -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 00000001000000590000003B -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 00000001000000590000003C -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 00000001000000590000003D -rw------- 1 postgres postgres 3670016 Feb 28 17:33 = 00000001000000590000003E -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 00000001000000590000003F -rw------- 1 postgres postgres 67108864 Feb 28 17:33 = 000000010000005A00000000 -rw------- 1 postgres postgres 12713984 Feb 28 17:33 = 000000010000005A00000001 Success/Failure Log: [postgres@hades ~/arch/dbc1/wal]$ cat /tmp/waltest.log=20 Success: 000000010000005900000008 Success: 000000010000005900000009 Success: 00000001000000590000000A Success: 00000001000000590000000B Success: 00000001000000590000000C Success: 00000001000000590000000D Success: 00000001000000590000000E Success: 00000001000000590000000F Success: 000000010000005900000010 Success: 000000010000005900000011 Success: 000000010000005900000012 Success: 000000010000005900000013 Success: 000000010000005900000014 Success: 000000010000005900000015 Success: 000000010000005900000016 Success: 000000010000005900000017 Success: 000000010000005900000018 Success: 000000010000005900000019 Success: 00000001000000590000001A Success: 00000001000000590000001B Success: 00000001000000590000001C Success: 00000001000000590000001D Success: 00000001000000590000001E Success: 00000001000000590000001F Success: 000000010000005900000020 Success: 000000010000005900000021 Success: 000000010000005900000022 Success: 000000010000005900000023 Success: 000000010000005900000024 Success: 000000010000005900000025 Success: 000000010000005900000026 Success: 000000010000005900000027 Success: 000000010000005900000028 Success: 000000010000005900000029 Success: 00000001000000590000002A Success: 00000001000000590000002B Success: 00000001000000590000002C Success: 00000001000000590000002D Success: 00000001000000590000002E Success: 00000001000000590000002F Success: 000000010000005900000030 Success: 000000010000005900000031 Success: 000000010000005900000032 Success: 000000010000005900000033 Success: 000000010000005900000034 Success: 000000010000005900000035 Success: 000000010000005900000036 Success: 000000010000005900000037 Success: 000000010000005900000038 Success: 000000010000005900000039 Success: 00000001000000590000003A Success: 00000001000000590000003B Success: 00000001000000590000003C Success: 00000001000000590000003D Failed: 00000001000000590000003E Failed: 00000001000000590000003E Failed: 00000001000000590000003E Failed: 00000001000000590000003E Success: 00000001000000590000003E Failed: 00000001000000590000003F Failed: 00000001000000590000003F Failed: 00000001000000590000003F Failed: 00000001000000590000003F Failed: 00000001000000590000003F Failed: 00000001000000590000003F Failed: 00000001000000590000003F Failed: 00000001000000590000003F Failed: 00000001000000590000003F Failed: 00000001000000590000003F Failed: 00000001000000590000003F Failed: 00000001000000590000003F Failed: 00000001000000590000003F Failed: 00000001000000590000003F Failed: 00000001000000590000003F Failed: 00000001000000590000003F Failed: 00000001000000590000003F Failed: 00000001000000590000003F Success: 00000001000000590000003F Success: 000000010000005A00000000 Failed: 000000010000005A00000001 Failed: 000000010000005A00000001 Failed: 000000010000005A00000001 Failed: 000000010000005A00000001 Failed: 000000010000005A00000001 Failed: 000000010000005A00000001 Failed: 000000010000005A00000001 Success: 000000010000005A00000001 Failed: 000000010000005A00000002 Failed: 000000010000005A00000002 Failed: 000000010000005A00000002 Failed: 000000010000005A00000002 Failed: 000000010000005A00000002 Failed: 000000010000005A00000002 Failed: 000000010000005A00000002 Failed: 000000010000005A00000002 Failed: 000000010000005A00000002=