Subject: Problem with mirrorring

From: Devrim GUNDUZ <[email-masked]>
Date: Sat, 2 Oct 2004 18:14:05 +0300 (EEST)

Hi Dave,

I think we again have a stale lock file; my web mirror of PostgreSQL could 
not succeed on the last 2 tries.

Regards,
- --
Devrim GUNDUZ 
devrim~gunduz.org				devrim.gunduz~linux.org.tr
 			http://www.tdmsoft.com
 			http://www.gunduz.org

---

From: "Dave Page" <[email-masked]>
Date: Sat, 2 Oct 2004 21:24:12 +0100

> -----Original Message-----
> From: Devrim GUNDUZ [mailto:[email-masked]] 
> Sent: 02 October 2004 21:00
> To: Dave Page
> Cc: PostgreSQL WWW Mailing List
> Subject: Re: [pgsql-www] Problem with mirrorring
> 
> >> There was not an 'error' exactly...
> >
> > Ahh, yes. Of course, that's actually rsync working correctly! No 
> > changes on the server, so no transfers.
> 
> Umm, every cron job outputs an email of 259K, but when there 
> is a lockfile, then it decreases to 800 bytes. Rsync gets all 
> the files, even if there is no change!

It shouldn't do. Andrew Tridgell (sp?) of the Samba team wrote it for
his Phd thesis, designing it specifically to minimise the data
transferred. Iirc, it only even transfers the differences between files
rather than the whole thing. It's pretty clever stuff.

Anyway (I'll stop babbling now :-) ), if it's transferring everything,
even if it hasn't even been touched at out end, then something,
somewhere is broken.

> > This does mean that our mirroring is quite innefficient. We really 
> > should diff the results of each build and only update the 
> html files 
> > if there is a change. At the moment each mirror is probably pulling 
> > the whole site each time :-(
> 
> Agreed but... diffing all the files that we have might be 
> quite hard for us -- But I now don't have a better idea...
> 
> Umm... If we can keep a track of modified files in a db (a 
> new or updated FAQ, doc, news, etc...) maybe we could publish 
> only them...

It's more tricky than that, because files change even if a new user join
Gborg for example (because of the count in the left column of the
'portal' pages). Besides, thinking about it some more I don't think it
really is that much of a problem. If rsync does it's job as I remember
it should (bear in mind it's a while since I read the thesis, and it's
Saturday night!), then it should minimise the transfer even if the mod
time has been touched.

Regards, Dave.

---

From: "John Hansen" <[email-masked]>
Date: Sun, 3 Oct 2004 12:10:04 +1000

Ahemm,... 

> > 
> > http://archives.postgresql.org/pgsql-www/2004-09/msg00149.php
> > and your reply:
> > http://archives.postgresql.org/pgsql-www/2004-09/msg00152.php
> > 
> Ahh, yes. Of course, that's actually rsync working correctly! 
> No changes on the server, so no transfers.
> 
> This does mean that our mirroring is quite innefficient. We 
> really should diff the results of each build and only update 
> the html files if there is a change. At the moment each 
> mirror is probably pulling the whole site each time :-(

Don't you mean quite efficient... ?

I assume the email devrim receives is a list of files changed.
Rsync will do a diff on the files and copy the changes across.
If however the only difference is the timestamp, then the file 
will still be listed in said email, since it's timestamp was updated.

I hope this weeds out any confusion.

... John

---

From: "John Hansen" <[email-masked]>
Date: Sun, 3 Oct 2004 21:58:46 +1000

> Of course, how it does that is a mystery to me - without 
> fully comparing both versions, how can it create a diff? I 
> guess that's what made it PhD material...

IIRC, it does a chunk by chunk md5 checksum on each file and transfers
only the differing chunks.
Quite clever indeed.