pg.ddx.io pgsql-sql@postgresql.org mailing list archive
help / color / mirror / Atom feedComparing two (largish) tables on different servers
4+ messages / 4 participants
[nested] [flat]
* Comparing two (largish) tables on different servers
@ 2004-11-09 22:41 Gregory S. Williamson <gsw@globexplorer.com>
0 siblings, 2 replies; 4+ messages in thread
From: Gregory S. Williamson @ 2004-11-09 22:41 UTC (permalink / raw)
To: pgsql-sql
This is probably a silly question.
Our runtime deployment of database servers (7.4) involves some redundant/duplicate databases. In order to compare tables (about 5 gigs each) on different servers I unload the things (takes a while etc.), sort them with a UNIX sort and then do a cksum on them.
Is there any way to do this from inside postgres that anyone knows of ? I looked through the manual and the contrib stuff and didn't see much ...
Thanks,
Greg Williamson
DBA
GlobeXplorer LLC
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: Comparing two (largish) tables on different servers
@ 2004-11-10 00:12 Pierre-Frédéric Caillaud <lists@boutiquenumerique.com>
parent: Gregory S. Williamson <gsw@globexplorer.com>
1 sibling, 0 replies; 4+ messages in thread
From: Pierre-Frédéric Caillaud @ 2004-11-10 00:12 UTC (permalink / raw)
To: pgsql-sql
Idea :
Write a program which connects on the two databases, creates a cursor on
each to return the rows in order, then compare them as they come (row 1
from cursor 1 == row 1 from cursor 2, etc). Fetch in batchs. If there's a
difference you can then know which row.
I hope you have an index to sort on, to save you a huge disk sort.
On Tue, 9 Nov 2004 14:41:00 -0800, Gregory S. Williamson
<gsw@globexplorer.com> wrote:
> This is probably a silly question.
>
> Our runtime deployment of database servers (7.4) involves some
> redundant/duplicate databases. In order to compare tables (about 5 gigs
> each) on different servers I unload the things (takes a while etc.),
> sort them with a UNIX sort and then do a cksum on them.
>
> Is there any way to do this from inside postgres that anyone knows of ?
> I looked through the manual and the contrib stuff and didn't see much ...
>
> Thanks,
>
> Greg Williamson
> DBA
> GlobeXplorer LLC
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/docs/faqs/FAQ.html
>
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: Comparing two (largish) tables on different servers
@ 2004-11-10 09:18 Sam Mason <sam@samason.me.uk>
parent: Gregory S. Williamson <gsw@globexplorer.com>
1 sibling, 1 reply; 4+ messages in thread
From: Sam Mason @ 2004-11-10 09:18 UTC (permalink / raw)
To: pgsql-sql
Gregory S. Williamson wrote:
>Is there any way to do this from inside postgres that anyone knows of
>? I looked through the manual and the contrib stuff and didn't see
>much ...
Not really "inside postgres"; but could you do something like:
mkfifo db1
psql -h "db1" -t -q -c "$query" > db1
mkfifo db2
psql -h "db2" -t -q -c "$query" > db2
diff -u -0 db1 db2
That should work with most shells under Unix. . .
Have fun,
Sam
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: Comparing two (largish) tables on different servers
@ 2004-11-10 19:32 Michael Fuhr <mike@fuhr.org>
parent: Sam Mason <sam@samason.me.uk>
0 siblings, 0 replies; 4+ messages in thread
From: Michael Fuhr @ 2004-11-10 19:32 UTC (permalink / raw)
To: pgsql-sql
On Wed, Nov 10, 2004 at 09:18:21AM +0000, Sam Mason wrote:
>
> mkfifo db1
> psql -h "db1" -t -q -c "$query" > db1
> mkfifo db2
> psql -h "db2" -t -q -c "$query" > db2
> diff -u -0 db1 db2
This should work for small data sets, but the OP said the tables
were about 5G. Unless you use a cursor, psql will fetch the entire
result before writing anything. Also, some implementations of diff
might read all of the data from one file before reading much from
the other file, especially if the files have differences. Hope
you have lots of memory....
--
Michael Fuhr
http://www.fuhr.org/~mfuhr/
^ permalink raw reply [nested|flat] 4+ messages in thread
end of thread, other threads:[~2004-11-10 19:32 UTC | newest]
Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2004-11-09 22:41 Comparing two (largish) tables on different servers Gregory S. Williamson <gsw@globexplorer.com>
2004-11-10 00:12 ` Pierre-Frédéric Caillaud <lists@boutiquenumerique.com>
2004-11-10 09:18 ` Sam Mason <sam@samason.me.uk>
2004-11-10 19:32 ` Michael Fuhr <mike@fuhr.org>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox