From: Rui DeSousa <rui.desousa@icloud.com>
To: Stephen Frost <sfrost@snowman.net>
Cc: pgsql-admin@lists.postgresql.org
Subject: Re: Reliable WAL file shipping over unreliable network
Date: Tue, 06 Mar 2018 12:55:20 -0500
Message-ID: <15F9D550-9909-40B3-A384-417AFCD745DF@icloud.com> (raw)
In-Reply-To: <20180305150218.GY2416@tamriel.snowman.net>
References: <6056907B-551F-4ED0-804B-0FCACD39EB23@icloud.com>
<ea674584-85bc-3f63-c182-0d45123bb3eb@catalyst.net.nz>
<988AB20C-9FDE-4197-BF3F-C54D80C2ED78@elevated-dev.com>
<E6AB850C-D05E-405B-8D4E-DE18E128F402@icloud.com>
<20180302180911.GO2416@tamriel.snowman.net>
<BB8E0E71-D65F-409F-B9AD-DDF2237CDF99@icloud.com>
<20180304150640.GR2416@tamriel.snowman.net>
<3D9CC2FB-57E1-45B2-B5FC-1B214F5DB068@icloud.com>
<20180305111521.GV2416@tamriel.snowman.net>
<ABC9F093-7249-498E-901B-6ADB7BBA3B22@icloud.com>
<20180305150218.GY2416@tamriel.snowman.net>
> On Mar 5, 2018, at 10:02 AM, Stephen Frost <sfrost@snowman.net> wrote:
>
> It doesn’t- but please don't encourage partial solutions which have ver clear issues.
Then problem is there are no good base utilities that is useful with archive_command; unless you’re writing directly to an NFS mount or a tape library. Even Barman recommends rsync with the archive_command; if you are unable to use pg_receivexlog solution. There are countless Postgres documentation out there that recommends use of rsync with the archive_command.
Here a solution that will fsync() file on the other end.
SSH_CMD="ssh -o ServerAliveInterval=20 $ARCH_SERVER"
STS=3
OUTPUT=$(cat $XLOGFILE | $SSH_CMD "(mkdir -p $ARCH_DIR && ~/bin/fwrite $ARCH_DIR/$WALFILE)")
if [ $? == 0 ]; then
STS=0
fi
exit $STS
fwrite code:
#include <sys/stat.h>
#include <err.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#define BUFSIZE 131072
int
main(int argc, char *argv[])
{
int fd, r, w;
char *buf;
char *name;
if (argc != 2) {
fprintf(stderr, "usage: fwrite [file]\n");
exit(1);
}
if ((buf = malloc(BUFSIZE)) == NULL)
err(1, "malloc");
++argv;
if ((name = (char *) malloc(strlen(*argv) + 10)) == NULL)
err(1, "malloc");
strcat(strcpy(name, *argv), ".XXXXXX");
if ((fd = mkstemp(name)) < 0)
err(1, "mkstemp");
while ((r = read(STDIN_FILENO, buf, BUFSIZE)) > 0)
if ((w = write(fd, buf, r)) == -1)
err(1, "write");
if (r < 0)
err(1, "read");
if (fsync(fd) != 0)
err(1, "fsync");
if (close(fd) != 0)
err(1, "close");
if (rename(name, *argv) != 0)
err(1, "rename");
free(name);
exit(0);
}
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-admin@postgresql.org
Cc: rui.desousa@icloud.com, sfrost@snowman.net, pgsql-admin@lists.postgresql.org
Subject: Re: Reliable WAL file shipping over unreliable network
In-Reply-To: <15F9D550-9909-40B3-A384-417AFCD745DF@icloud.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox