pg.ddx.io  pgsql-admin@postgresql.org mailing list archive  

help / color / mirror / Atom feed
From: Scott Ribe <scott_ribe@elevated-dev.com>
To: Jean-Armel Luce <jaluce06@gmail.com>
Cc: Cédric Villemain <cedric.villemain.debian@gmail.com>
Cc: [ADMIN] <pgsql-admin@postgresql.org>
Subject: Re: rsync and streaming replication
Date: Wed, 16 Nov 2011 10:23:28 -0700
Message-ID: <48B4CE29-7776-4D7A-8AA5-D45A4F77BB75@elevated-dev.com> (raw)
In-Reply-To: <CALnckSqvM7HrzBK2ETaDQh6OotRLZF=_enJvDfZ0wQZMoHMaHg@mail.gmail.com>
References: <CALnckSpEi-6JjfEFLUcp13nScHbbcywFcmhn51iK2sj_eXbEkQ@mail.gmail.com>
	<87pqgxf8p9.fsf@comcast.net>
	<CALnckSqDpZi=_w8XckMLBHu_V4kgtn+EqNsqFcNgLtbcYorzDA@mail.gmail.com>
	<CAF6yO=0UEB82UP2Lusd2xfrY+7TyVpn9pcFdDv1=_w9Rf=FqXg@mail.gmail.com>
	<CALnckSqdG0aQ09nUhUeGWZ8SC8J25WEHHe4wD0jnz7K2+Kov5A@mail.gmail.com>
	<639B3C52-BF6D-4CE9-A710-75D369E9607E@elevated-dev.com>
	<CAF6yO=3WQMsa5WQkHNbfW9URnto5M94U5U9ZEQ3LS=dLRri2dw@mail.gmail.com>
	<39A965E0-DE37-41F3-8D69-321419BB8B09@elevated-dev.com>
	<CAF6yO=2Cm_1sq7PZ=xRp5t+Oxu3e+bFWiRX+V2TOce4b5kauKg@mail.gmail.com>
	<640CEEBE-CB12-4FC9-9758-0319A49E7ACF@elevated-dev.com>
	<CALnckSqLVWff3x98k1sZcg09G8ARScQUPZPGkZ0ve1HjiFS6jw@mail.gmail.com>
	<CAF6yO=3G-P0KgSH=Eiqw3Se38Civ7QktKF8q2-Zs0Rgy47=6Ww@mail.gmail.com>
	<CALnckSp2Ygt7nJBwBX1gRkf6c7ZbEFHS=ydrLsn4cB79Yx_gvw@mail.gmail.com>
	<CAF6yO=3fjuokv9J80oW9oQWF9YdYxMcH_gSyfJ5KCKwiA5V_5w@mail.gmail.com>
	<CALnckSqvM7HrzBK2ETaDQh6OotRLZF=_enJvDfZ0wQZMoHMaHg@mail.gmail.com>

On Nov 16, 2011, at 10:11 AM, Jean-Armel Luce wrote:

> You are right. I used -a, and I was wanting to be more meaningful so I wrote --all in my post.
> Please read --archive insterad of --all

Oh, OK. Still seems odd that it took so much longer. Granted, for the files with different timestamps but identical contents, it then syncs them. But it does so by checksumming blocks, comparing checksums, and sending only blocks that are different over the network. Granted, it has to send some checksums over the network, but that's pretty minor traffic. I believe you said you'd seen 125Mb/s over the network? Is that actually accurate? Does the network connection have high latency?

Also, I believe you said -z seemed to slow it down? That has not been my experience at all with rsync'ing pg databases. Between all the values that are stored as plain text, and the redundancies in indexes, I usually see a good speed increase from compressing the data in transit.

I'm certainly glad that you've got a 3x speed increase--that's significant progress. But still, something seems odd about the performance you've reported, so I'm left wondering about what could cause that. Disk performance glitch at one end or the other, network performance, CPU load???

One thing worth doing I think is to use --stats on every test, so you can see every time how many files and how much data is actually transferred. Also, if you're sitting there watching, sometimes --progress can be informative...

-- 
Scott Ribe
scott_ribe@elevated-dev.com
http://www.elevated-dev.com/
(303) 722-0567 voice







view thread (26+ messages)  latest in thread

Message-ID: <48B4CE29-7776-4D7A-8AA5-D45A4F77BB75@elevated-dev.com>
Permalink:  ../48B4CE29-7776-4D7A-8AA5-D45A4F77BB75@elevated-dev.com/
Also on:    postgresql.org/message-id/48B4CE29-7776-4D7A-8AA5-D45A4F77BB75@elevated-dev.com

 · 

reply
Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-admin@postgresql.org
  Cc: scott_ribe@elevated-dev.com, jaluce06@gmail.com, cedric.villemain.debian@gmail.com
  Subject: Re: rsync and streaming replication
  In-Reply-To: <48B4CE29-7776-4D7A-8AA5-D45A4F77BB75@elevated-dev.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox